Add support for transformer models

[marco-c]

Add a simple example model for detecting defects, both with finetuning and with using embeddings to train a xgboost model

Fixes #39

Train on Taskcluster: defect_finetuning

[marco-c]

It would also be nice to support LoRa (or at least add an example on how to use it). https://huggingface.co/blog/Lora-for-sequence-classification-with-Roberta-Llama-Mistral

[marco-c]

The docker build task is hanging at:

Successfully tagged registry.hub.docker.com/mozilla/bugbug-base:latest
b3cd317af722d52d61ce0214c7a95ecc046171da421bbce0732f09745d06bd7d
INFO:taskboot.docker:Built image registry.hub.docker.com/mozilla/bugbug-base:latest
INFO:taskboot.docker:Saving image with tags registry.hub.docker.com/mozilla/bugbug-base:latest to /images/bugbug-base.tar

Maybe the image is too big?

[suhaibmujahid]

The docker build task is hanging

It is moving now!

[marco-c]

It is super slow though!
I see:

/images/bugbug-base.tar : 45.96% ( 6.61 GiB => 3.04 GiB, /images/bugbug-base.tar.zst)

while on master:

/images/bugbug-base.tar : 37.84% ( 2.23 GiB => 864 MiB, /images/bugbug-base.tar.zst)

[marco-c]

I'm not sure why, but the situation seems far better now that we use uv.

[marco-c]

It looks like train on tc task is broken though, probably not due to this PR:

+ python -m scripts.trainer_extract_args
/usr/local/bin/python: Error while finding module specification for 'scripts.trainer_extract_args' (ModuleNotFoundError: No module named 'scripts')
+ python -m scripts.trainer
/usr/local/bin/python: Error while finding module specification for 'scripts.trainer' (ModuleNotFoundError: No module named 'scripts')

Maybe due to the uv migration?

[marco-c]

I'm not sure why, but the situation seems far better now that we use uv.

My bad, I didn't update uv.lock. It's still super slow.