Comparison of Transformers vs older ML architectures in Spend Classification

I recently wrote a piece for my company blog about why Transformers are a better machine learning technology to use in your spend classification projects compared to older ML techniques.

Transformers: More Than Meets the Eye (in Spend Classification)

That was a theoretical post that discussed things like sub-word tokenization and self-attention and how these architectural features should be expected to deliver improvements over older ML approaches.

During the Jubilee Weekend, I thought I’d have a go at doing some real-world tests. I wanted to do a simple test to see how much of a difference this all really makes in the spend classification use case. The code is here: https://github.com/alanbuxton/tbfy-cpv-classifier-poc

TL;DR – Bidirectional LSTM is a world away from Support Vector Machines but Transformers have the edge over Bi-LSTM. In particular they are more tolerant of spelling inconsistencies.

This is an update of the code I did for this post: https://alanbuxton.wordpress.com/2021/10/25/transformers-vs-spend-classification/ in which I trained the Transformer for 20 epochs. In this case it was 15 epochs. FWIW the 20-epoch version was better at handling the ‘mobile office’ example. This does indicate that better results will be achieved with more training. But for the purposes of the current blog post there wasn’t any need to go further.

A digression on Software Patents

I wrote some software to make collaboration with Excel easier and have been writing up a few posts here about how it works. I’ve been struggling recently with how much of the details to describe on this public forum. Essentially I have this little niggling voice in the back of my head telling me: “Protect your IP, file a patent, protect your IP, file a patent”. Any sensible person would surely file a patent before going public with anything.

Yet I loathe software patents. And perhaps I’m not all that sensible.

Patents take a lot of time and distraction and cost lots of $$. The patent process moves at such a glacial pace that they are all but meaningless when you’re trying to get an idea off the ground. Interestingly it wasn’t always as bad as this. From “The Victorian Internet” by Tom Standage;

[Alexander Graham] Bell worked for several months to build a working prototype [of a “harmonic telegraph”, i.e. telephone]. On 14 February 1876, when it became clear that [a competitor] was pursuing the same goal, Bell filed for a patent, even though he had yet successfully to transmit speech. He was granted the patent on 3 March, and made the vital breakthrough a week later, when he succeeded in transmitting intelligible speech for the first time.

Now if I read that right it took less than a month to get a patent granted. Whereas now it takes years during which time who knows what could happen to your business. How many businesses do we take for granted today that barely existed five years ago?

If patents could be decided and genuinely only granted in non-obvious cases in a few weeks then I might see some value around the certainty they could provide. But they take forever and they seem to be as likely to be about something trivially obvious as they are about something genuinely ground-breaking.

And anyway. Even owning a patent doesn’t mean much unless you have the $$ to take someone to court over infringement.

So when it comes to the real world, at least where software is concerned, patents are a tax on innovation. They divert resources away from where they are best used. They are a brake on innovation.
Arguments I’ve heard in favour of patents include:

1. The revenue stream argument: If you own a patent you can license the IP.
2. The risk mitigation argument: If you are lucky enough to be successful and are about to IPO or similar then some fucker (technical term) somewhere will try to scupper you by claiming you are infringing a patent somewhere along the line. So better to have some patents that say you are allowed to do what you are doing, or failing that let you counter-sue the fucker by claiming that he is infringing on your patents.
3. The ego-trip argument: The idea that you must be cleverer than the other guys if you have more patents than them. Never mind that (as I am reliably informed) the number of patents you own is irrelevant – what is important is the number of claims across the patents. You could have one patent application with 1,000 claims in or you could file 10 patents each with 100 claims and you would be in an equivalent IP position. So why people insist on counting patents is surprising. Still they do.

Those are all valid arguments to a degree. If you want to build a business around selling some IP then you need to license it. If someone is going to value you based on a not very meaningful metric then, whatever, don’t hate the player, hate the game and all that. Just a bit of a depressing game.

So to cut a long story short I have now excised from my psyche any thinking about software patents. Not interested. I do hate the game and so will do my best not to play in it.

BTW – in case any lawyery types are reading – I don’t have a cavalier attitude to IP protection in general. I just think that in the world of software, copyright is the correct approach, not patents.

Lots of love

Alan

 

The Economist’s Fifth Annual Innovation Awards Ceremony

Happened today 9th Nov in London’s Science Museum. Some notable highlights:

Sam Pitroda, Chairman of WorldTel and The National Knowledge Commission, India, being recognised for his achievements as a telecoms inventor, entrepreneur and policymaker: giving a particularly impassioned and inspirational speech about his work with India’s telecoms under Rajiv Gandhi, the development of India over the last 20 years and about planting the seeds for a new explosion in India’s development for the next 20 years. And being refused his coat by the cloakroom attendant because he didn’t have his ticket. To be fair, she was only doing her job… but even so.

Janus Friis of Skype announcing that it is “really fucking cool to get this award from the Economist”. Can’t wait to see how they report this in the magazine.

Hernando De Soto, Chairman for the Institute of Liberty and Democracy explaining the need for co-operation in all innovative efforts: co-operation that has to be founded on the rule of law and strong property rights – things that first worlders take for granted.

And last, but not least, John Micklethwait and Tom Standage of the Economist looking incredibly young. You know how they say that you know you’re getting old when policemen start looking young …. you really know you’re getting old when the guys who run the Economist start looking young 🙂