Transformers vs Spend Classification

In recent posts I’ve written about the use of Transformers in Natural Language Processing.

A friend working in the procurement space asked about their application in combating decepticons unruly spend data. Specifically, could it help speed up classifying spend data.

So I fine-tuned a Distilbert model using publicly-available data from the TheyBuyForYou project to map text to CPV codes. It took a bit of poking around but the upshot is pretty promising. See the following classification results where the model can distinguish amongst the following types of spend items:

'mobile phone' => Radio, television, communication, telecommunication and related equipment (score = 0.9999891519546509)
'mobile app' => Software package and information systems (score = 0.9995172023773193)
'mobile billboard' => Advertising and marketing services (score = 0.5554304122924805)
'mobile office' => Construction work (score = 0.9570050835609436)

Usual disclaimers apply: this is a toy example that I played around with until it looked good for a specific use case. In reality you would need to apply domain expertise and understanding of the business. But the key point is that transformers are a lot more capable than older machine learning techniques that I’ve seen in spend classification.

The code is all on Github and made available under the Creative Commons BY-NC-SA 4.0 License. It doesn’t include the model itself as the model is too big for github and I haven’t had a chance to try out Git Large File Storage. If people are interested I’m more than happy to do so.