In https://alanbuxton.wordpress.com/2021/09/21/transformers-for-use-oriented-entity-extraction/ I wrote about an experience training a custom transformer-based model to do a type of entity extraction. I tried training from scratch because the source text happened to have been preprocessed / lemmatized in such a way to include over 20 custom tokens that the RoBERTa model wouldn’t know about. My assumption was that this text would be so different to normal English that you may as well treat it as its own language.
Once I saw the results I decided to test this hypothesis somewhat by comparing the results of the preprocessed/lemmatized text with the custom model vs a raw version of the text on a fine-tuned out of the box roberta-base model.
Turns out that, for me, the fine-tuned RoBERTa model always outperformed the model trained from scratch, though the difference in performance becomes pretty minimal once you’re in the millions of sentences.
Conclusion – when working in this space, don’t make assumptions. Stand on the shoulders of as many giants as possible.
| Approx number of sentences for fine-tuning | F1 Score – RoBERTa from scratch | F1 Score – Fine-tuned roberta-base |
| 1,500 | 0.39182 | 0.52233 |
| 15,000 | 0.75294 | 0.97764 |
| 40,000 | 0.92639 | 0.99494 |
| 65,000 | 0.97260 | 0.99627 |
| 125,000 | 0.99105 | 0.99776 |
| 300,000 | 0.99670 | 0.99797 |
| 600,000 | 0.99771 | 0.99866 |
| 960,000 | 0.99783 | 0.99865 |
| 1,400,000 | 0.99810 | 0.99888 |
Be good, work hard, get lucky 
One thought on “Is it worth training an NLP Transformer from scratch?”