What’s the Best Way to Fine-Tune a Transformer Model on a Custom Dataset Using the Transformers Library?

[Prakash-Hinduja]

Hi everyone, Prakash Hinduja, Swiss, I’m currently exploring fine-tuning a pre-trained Transformer model (like BERT or DistilBERT) on a custom text classification dataset, and I’m using the Hugging Face Transformers library along with the Datasets and Trainer API.

I’ve followed a few tutorials, but I’m still a bit unsure about best practices and would appreciate some guidance. Specifically:

What’s the recommended way to prepare and tokenize a CSV dataset with custom labels for classification?

Should I use Trainer or accelerate for training at scale (especially on Colab Pro or local GPU)?

How do I handle imbalanced datasets or apply weighted loss functions in the Trainer API?

What’s the best way to evaluate model performance (e.g. F1-score, precision, recall) after each epoch?

Any tips for saving and sharing the model back to the Hugging Face Hub?

I’d also love to see any code snippets, notebooks, or examples that helped you during your own fine-tuning projects.

Thanks a lot for your help!
Prakash Hinduja

[John6666]

The scope of the question is too broad, so I think it would be quicker to read the step-by-step tutorial and ask questions only about the parts you don’t understand…

Fine-tuning and General Info.

Introduction - Hugging Face LLM Course

Fine-tuning

Fine-tune ModernBERT for text classification using synthetic data

Evaluation

Adding accuracy, precision, recall and f1 score metrics during training

Hello! I am fine-tuning herbert-base model for token classification of named entities. Basically I am going through this tutorial with minor changes to data preprocessing, pretrained base model and datasets. I would love to see evaluation f1 score and accuracy throughout training. What would be most way to add those metrics after every epoch? I see option to load metrics from datasets. But I cannot find where should I put loaded objects. I know I could override Trainer class or…

Trainer is better for single GPU env. like Colab

Trainer and Accelerate

What are the differences and if Trainer can do multiple GPU work, why need Accelerate? Accelerate use only for custom code? (add or remove something)

Import CSV or other data to datasets

Quickstart