Comparison of Transformers vs older ML architectures in Spend Classification

I recently wrote a piece for my company blog about why Transformers are a better machine learning technology to use in your spend classification projects compared to older ML techniques.

Transformers: More Than Meets the Eye (in Spend Classification)

That was a theoretical post that discussed things like sub-word tokenization and self-attention and how these architectural features should be expected to deliver improvements over older ML approaches.

During the Jubilee Weekend, I thought I’d have a go at doing some real-world tests. I wanted to do a simple test to see how much of a difference this all really makes in the spend classification use case. The code is here: https://github.com/alanbuxton/tbfy-cpv-classifier-poc

TL;DR – Bidirectional LSTM is a world away from Support Vector Machines but Transformers have the edge over Bi-LSTM. In particular they are more tolerant of spelling inconsistencies.

This is an update of the code I did for this post: https://alanbuxton.wordpress.com/2021/10/25/transformers-vs-spend-classification/ in which I trained the Transformer for 20 epochs. In this case it was 15 epochs. FWIW the 20-epoch version was better at handling the ‘mobile office’ example. This does indicate that better results will be achieved with more training. But for the purposes of the current blog post there wasn’t any need to go further.

Transformers vs Spend Classification

In recent posts I’ve written about the use of Transformers in Natural Language Processing.

A friend working in the procurement space asked about their application in combating decepticons unruly spend data. Specifically, could it help speed up classifying spend data.

So I fine-tuned a Distilbert model using publicly-available data from the TheyBuyForYou project to map text to CPV codes. It took a bit of poking around but the upshot is pretty promising. See the following classification results where the model can distinguish amongst the following types of spend items:

'mobile phone' => Radio, television, communication, telecommunication and related equipment (score = 0.9999891519546509)
'mobile app' => Software package and information systems (score = 0.9995172023773193)
'mobile billboard' => Advertising and marketing services (score = 0.5554304122924805)
'mobile office' => Construction work (score = 0.9570050835609436)

Usual disclaimers apply: this is a toy example that I played around with until it looked good for a specific use case. In reality you would need to apply domain expertise and understanding of the business. But the key point is that transformers are a lot more capable than older machine learning techniques that I’ve seen in spend classification.

The code is all on Github and made available under the Creative Commons BY-NC-SA 4.0 License. It doesn’t include the model itself as the model is too big for github and I haven’t had a chance to try out Git Large File Storage. If people are interested I’m more than happy to do so.

Nice weather for buyers

In England we have a phrase: nice weather for ducks. It means it’s miserable, rainy and wet. But still … nice if you’re a duck.

Reading me Economist this week (10th Jan issue) for the first time in a few months. First article in the Britain section is called Combating the Recession. Here’s paragraph 2:

The bank’s [latest rate cut] means that the base rate has now fallen by an extraordinary 3.5 percentage points since the start of October. It followed a clutch of closely watched business surveys of purchasing managers that painted a dismal picture of the economy in December. (my emphasis)

It’s a bit of a commonplace that the worse things get for a business the more it looks to departments such as procurement to dig it out of a hole. But even so: What an opportunity! If more and more people are sitting up and paying attention when buyers speak, perhaps now, finally, “the business” will act on all those things you’ve been hammering on about for goodness knows how long.

Of course the attention on buyers and the opportunity this represents only exist  because of the wider crisis going on. So this opportunity for buyers also represents a challenge: Develop the necessary credibility within your organisation so your views are taken equally seriously once we are out of this recession …. and so business don’t wait until the next economic slowdown to consult their procurement departments again.

An honest, open and transparent approach to supplier selection

I was doing some googling recently about Japanese auctions to see if there is anyone out there apart from TradingPartners who has anything useful to say about them in a procurement space. Turns out there doesn’t seem to be. But in the process I stumbled across this entertaining old article from the NY Times:

Takashi Hashiyama, president of Maspro Denkoh Corporation, an electronics company based outside of Nagoya, Japan, could not decide whether Christie’s or Sotheby’s should sell the company’s art collection, which is worth more than $20 million, at next week’s auctions in New York.  
 
He did not split the collection – which includes an important Cézanne landscape, an early Picasso street scene and a rare van Gogh view from the artist’s Paris apartment – between the two houses, as sometimes happens. Nor did he decide to abandon the auction process and sell the paintings through a private dealer.

Instead, he resorted to an ancient method of decision-making that has been time-tested on playgrounds around the world: rock breaks scissors, scissors cuts paper, paper smothers rock.
In Japan, resorting to such games of chance is not unusual. “I sometimes use such methods when I cannot make a decision,” Mr. Hashiyama said in a telephone interview. “As both companies were equally good and I just could not choose one, I asked them to please decide between themselves and suggested to use such methods as rock, paper, scissors.”

Well as a process it’s certainly honest, open and transparent.

Unite Members to Protest Against Council Tendering

See here.

Unite, Scotland’s largest trade union, will hold a demonstration [Wednesday, 4th June] outside South Lanarkshire Council offices in protest at the blind bidding process for the care and support services for adults with learning disabilities.

South Lanarkshire Council set up an e-auction for firms to tender for providing care at home by submitting charges by the hour.

Thus begins the press release. Looks like an e-auction is the root cause of the antipathy.

Well – look again, further down.

Bidders were given no information in the tender document on the current terms and conditions of the employees who would be transferred.

In other words: the union’s issue seems to be with the quality (or lack) of information that was given out to bidders rather than with the bidding process itself. Once again – to run a good sourcing process (whether it involves an e-auction or not) you still need to be clear with suppliers. And make sure they have full access to the information they need to place sensible bids.

Wordle: A bit of fun

Wordle.net is a fun utility that picks out the most significant words in blocks of text (e.g. documents or web pages or RSS feeds). Like an automated tag cloud generator.  FYI I picked it up from this posting: http://www.fastforwardblog.com/2008/07/07/transparent-and-explicit/

Here are Wordle pages for some supply chain blogs:

Spend Matters

SpendMatters. Lots of talk about Suppliers, Controls, Leveraged, Research but perhaps lay off the Metal Miner plugs for a while 🙂

Sourcing Innovation

Plenty of talk about Challenges, Sourcing, Execution and Johnson (again – see Spend Matters)

e-Sourcing Forum

And of course, mine:

e-Sourcing Place

Yikes! About time I started banging on about reverse auctions again methinks.

Procurement Blogs I’m reading

Here are the procurement blogs I have on my current reading list

• http://www.spendmatters.com Where I go for industry analysis. And some entertaining banter.
• http://blog.sourcinginnovation.com Where I go for in depth dissection of offerings in the market place.

These are without a doubt the top 2 blogs in the space. Top 2 in terms of quality and quantity. Read these blogs first; anything of burning significance will end up on one of these two sooner or later.

Blogs allied to a particular vendor/service provider.

• http://www.esourcingforum.com Iasta’s blog and definitely a good one. David Bush walks a fine line between making the blog a true forum for all things supply chain, and running a corporate mouthpiece
• http://www.purchasingcourses.com/purchasingcertificationblog.html Next Level Purchasing’s blog covers training – always a good read
• http://www.mfgx.com/blogs/mfgx MFG.com’s blog. Mainly manufacturing. (Incidentally they changed their servers recently so any old RSS subscriptions you have via http://www.mfgx.com/blog/index.cfm will now be broken)
• http://www.2sustain.com/ Aravo’s blog (Supplier Enablement company with a green streak)
• http://purchasingtransformation.ibxeurope.com/ IBX’s blog
• http://www.strategicsourceror.com/ SourceOne’s blog. These are the guys who provide WhyAbe.
• http://www.supplyexcellence.com Ariba’s blog with apparently pretty high traffic. Note to Tim: Please can you make it more obvious that this is the Ariba blog. To the casual observer it isn’t obvious that this is the Ariba blog. Purely in terms of Ariba logo usage on the screen the casual observer might think that SpendMatters is more closely allied to Ariba than Supply Excellence.
• http://www.esourcingplace.com And mine of course. Much as I would like to consider myself un-biased I do work for TradingPartners so that is bound to affect my views.

General supply chain commentary

• http://tcummins.wordpress.com/  Tim Cummins (IACCM) takes a broad view from both a buyer’s and supplier’s perspective
• http://www.theferrarigroup.com/blog1/
• http://www.supplychainer.com/
• http://www.supplychainnetwork.com/ 
• http://www.buyeranalytics.com/purchasingblogs/ 

Other specialist blogs (dare I call them niche?)

• http://vmo-blog.com/ Without a doubt this is one of the best blogs out there right now. Definitely, definitely worth a read for people in the procurement space.
• http://www.licensinghandbook.com/blog/ Contract negotation
• http://www.agmetalminer.com/  Metals
• http://at-scm.com/    Irregular but interesting
• http://muchosalsa.com/blog/ A gentleman from HP who is planning on building his own business in the supply chain space

Ones that I enjoyed but seem to have tailed off in recent months. Watch them if they come back

• http://360vendormanagement.com/ Outsourcing blog
• http://www.massin.nl/eSourcing  Occasional comments on the European sourcing industry

The various magazines also have their own blogs/RSS feeds. I distinguish between a blog which by its nature is more immediate and informal and an RSS feed of magazine articles. The blogs in my list right now are:

• Procurement Leaders  http://blog.procurementleaders.com/procurement-blog/
• Supply Management  http://www.supplymanagement.com/EDIT/BLOG/default.asp
• Purchasing.com  Has a range of blogs http://www.purchasing.com/blogs.html 

From what I can tell S&DC Exec and CPO Agenda don’t (yet) offer blogs.
   

 

Procurement Solutions Expo

At Procurement Solutions, in between presentations.

Some notes:

1. It’s a bewildering exhibition. From carpet tiles to legal services via tablet PCs. Too broad I think.

2. The TradingPartners blue bouncy balls are a big hit. With everyone picking them up for their kids/grandkids, I am predicting a surge in demand for TradingPartners services in about 30 years.

3. I was reading in Public Sector Procurement magazine about Zanzibar (the UK Public Sector eProcurement system) that Zanzibar has recently added RFQ processes. Before this, everything I have seen from the OGCbs has kept “procurement” separate from “sourcing”. Looks like this move by Zanzibar challenges that distinction. It will be interesting to see if ProcServe (the company behind Zanzibar) make a move on the eSourcing framework when it comes up for renewal.

Where next has changed.

Don’t adjust your browser settings. You’re in the right place. This is still Alan Buxton’s blog. But with a new look and focus.

(Note: The old URL and old Feeds, whether from WordPress or Feedburner, will continue to work)

I started this blog with the vague intention of blogging about eAuctions in particular and Business Technology in general. But the more I write the more I want to keep these areas distinct. So much so that I’ve now decided to start another blog, Golden Pebbles to get tech stuff off my chest.

In parallel, over the last two years I’ve realised that the distinction between eAuctions and eSourcing is more arbitrary and more complex than I originally thought. I’ll try to explain:

The vast majority of the questions/issues/challenges I hear about reverse auctions are not about reverse auctions at all. For example:

• Auctions aren’t able to handle quality – they just consider price
• Auctions lead to suppliers selling at unsustainable prices, which is bad for the supply market
• Auctions lead to poor decisions because they try to compare suppliers bidding apples with suppliers bidding pears
• etc

But hang on – auctions are a tool. Used by buyers. Some buyers are keen to hammer suppliers on price. They may or may not use auctions to achieve this. For sure they can use auctions to hammer suppliers. But there are also plenty other ways to hammer suppliers. Check the recent “flaming lamborghini” stories about some of the negotiation tactics used by retailers. So change the word “Auctions” in the bullet points above with “Some buyers” and you have a more accurate picture of the reality.

Additionally, from the reverse auctions I’ve seen, it’s clear that the better ones need a lot of up front work to deliver. What you see on auction day is just the cherry on the cake. Before the auction you need to select suppliers, agree specifications, make sure everyone is clear on what is being contracted etc etc. In other words all the same stuff buyers should be doing anyway when they are sourcing. Whether they do an auction or not.

So my epiphany is that auctions are just sourcing. On one level it’s abitrary where you put the line between them because really an auction is not a different way of sourcing. An auction is just the best way of harnessing competition in your sourcing projects.

But running a good auction is about more than just turning on a piece of software. There is a lot of thought and hard work to do up front to give you the biggest chance of success. Again this should be no surprise. If you send out RFPs then the challenge is more to do with putting together sensible questions in the RFP. The challenge is not really about which piece of software is used to issue the RFP.

So while I remain a staunch advocate of auctions (and use them myself when I can) I find myself taking a broader view of auctions & sourcing than perhaps I did 2 years ago. And I figured it’s time to grow up a bit and get a proper domain name. My trials and tribulations with domain names are a story for another day but I finally settled on www.esourcingplace.com. A place for sourcing. Still focussed on auctions because it’s what I’m closest too. But I’ll also drop in some more general sourcing points now and again.

 

What I learned from the conference season

An aside: Is conference “season” a North American thing? Or is a global phenomenon. I will readily admit to not being a conference junky. I only tend to go to conferences at which TradingPartners is exhibiting.

I’ve only been to one conference this “season” so far: eWorld Purchasing & Supply in Feb. I will be at Procurement Solutions next week, again speaking about eAuctions.

So what did I learn from eWorld Purchasing & Supply?

Briefly:

1. Hackett’s keynote fascinated me. It made the case that procurement as we know it (strategic sourcing, supply chain management etc) can only add so much value to the organisation. In order to advance beyond this then procurement will need to re-invent itself.

2. There seems to be a renaissance in eSourcing providers out there these days. I think we were in double digit identikit eSourcing software vendors exhibiting – many of whom I had never heard of before. How can vendors stand out in this kind of marketplace? Focus on key competencies.

3. But the real topic of conversation amongst the delegates, and the real reason most went, was to learn something about “green” and “sustainability”. And many felt let down by the advice that “you can be green and save money by using less packaging”. There is obviously a large gap in the market here. This is encouraging. But at the same time it’s worrying because the pent-up demand for green solutions of some shape or form is going to be an irresistible tempation for hucksters. Buyer of green solutions, beware. As ever.