Python script to handle company abbreviations

A while back I was doing some tasks to clean up company names. Wikipedia has a useful page but I couldn’t find a simple way to use this information in a python script. So after some downloading and wrangling data from this wikipedia page, here is a link to the code I ended up with. Posting it here in case it is of use to someone else out there.

Some useful factoids I picked up along the way:

  1. Take a lot of care with Unicode. Characters that look the same (depending on font) might be very different. For example: 'KT vs КТ'.lower() == 'kt vs кт'
  2. Some abbreviations might appear at the beginning of a name, not just at the end. For example ENEL RUSSIA PJSC vs PJSC “AEROFLOT”.

 

Advertisements

Comparing Python vs Ruby’s handling of empty Lists/Arrays

TIL Ruby is more forgiving when handling empty arrays that Python is with empty lists. If you try to access a non-existent index in Ruby you get a nil back. In Python you get an IndexError. I prefer the Python approach. It’s more straightforward.

Though Python also has a curious way that allows you to work around IndexError by trying to access a range of items in the List. And Ruby does also have an approach that will give you an appropriate error if the index is out of bounds. So who knows what the logic behind all of this is supposed to be ….

Python Version

Compare the following Python commands

empty = []
arr = ["here","are","some","words"]
empty[0]
arr[0]
empty[0:1]
arr[0:1]
" ".join(empty[0:1])
" ".join(arr[0:1])

Results of the Python commands:

>>> empty = []
>>> arr = ["here","are","some","words"]
>>> empty[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list index out of range
>>> arr[0]
'here'
>>> empty[0:0]
[]
>>> arr[0:0]
['here']
>>> " ".join(empty[0:0])
''
>>> " ".join(arr[0:0])
'here'

Ruby version

See the following Ruby commands

empty = []
arr = ["here","are","some","words"]
empty[0]
arr[0]
empty[0..0]
arr[0..0]
empty[0..0].join(" ")
arr[0..0].join(" ")

Results of the Ruby commands:

irb(main):007:0> empty = []
=> []
irb(main):008:0> arr = ["here","are","some","words"]
=> ["here", "are", "some", "words"]
irb(main):009:0> empty[0]
=> nil
irb(main):010:0> arr[0]
=> "here"
irb(main):011:0> empty[0..0]
=> []
irb(main):012:0> arr[0..0]
=> ["here"]
irb(main):013:0> empty[0..0].join(" ")
=> ""
irb(main):014:0> arr[0..0].join(" ")
=> "here"

Comparison

Python is pretty unforgiving if you try to access a non-existent element. Ruby is forgiving enough to give you a nil if the element is non-existent. Downside of the Ruby approach is that you don’t know if the nil means “there was a nil there” or “there was nothing there”.

If you want the less forgiving approach in Ruby you can use fetch like so:

irb(main):031:0> empty
=> []
irb(main):032:0> empty.fetch(0)
Traceback (most recent call last):
5: from /snap/ruby/132/bin/irb:23:in `<main>'
4: from /snap/ruby/132/bin/irb:23:in `load'
3: from /snap/ruby/132/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
2: from (irb):32
1: from (irb):32:in `fetch'
IndexError (index 0 outside of array bounds: 0...0)

 

Learning how Cypress interacts with LocalStorage

I asked some developer friends a while back about whether people still like Watir for testing or if people are going more wholesale down the Selenium route. Their answer: “None of the above, try Cypress instead.”

I love it when I ask a question and get an answer completely out of left field. Turns out they were absolutely right. Cypress has proved to be excellent. It makes writing browser automation tests fun. There was a bit of a learning curve, and so I want to share two things that it took a while for me to wrap my head around:

  1. Understanding Cypress’s async nature, and the need to rely on then for certain use cases.
  2. Understanding that if you follow along the tests in the UI then what you see isn’t always what Cypress sees.

To illustrate this, see this gist that accesess localStorage under different circumstances:

The first thing to note here is that Cypress’s async nature means that, for example, accessing localStorage makes sense from inside a then or inside an afterEach, but not from inside the main body of the test. See below log results from the first two tests in the gists which [a] open a page and clicks a link in it and [b] directly opens the same page.

You can see that the logging from the main body of the test doesn’t produce any output (row 7 from the first test and 4 from the second test).

The second thing to note is that what is seen is not always what is real.

In the third test, when following along with dev tools, I could see that localStorage was being set, but even the afterEach claimed there was no localStorage set (see row 1 of the afterEach).

Cypress, localStorage and redirection

 

I suspect this is because when an interactive user goes to https://docs.cypress.io, they end up redirected to a different page, and localStorage gets set during this redirection. What you are seeing in Dev tools is what is set as part of the redirection process (see how the URL in the image above is different to the one the test lands on), but what Cypress reports is what it found prior to the redirection.

 

 

Fun with True and False in Python

TIL: In Python, True = 1 and False = 0

>>> var0 = 0
>>> var1 = 1
>>> var0 == True
False
>>> var0 == False
True
>>> var1 == True
True
>>> var1 == False
False

The number 2, however, is neither True nor False. But it is truthy.

>>> var2 = 2
>>> var2 == True
False
>>> var2 == False
False
>>> bool(var2)
True

Which has the following interesting side effects when testing conditions

>>> if (var1 == True): print("***  var1 is true ***")
... else: print("*** var1 is not true ***")
... 
***  var1 is true ***

>>> if (var2 == True): print("*** var2 is true ***")
... else: print("*** var2 is not true ***")
... 
*** var2 is not true ***

>>> if (bool(var2)): print("*** var2 is truthy ***")
... else: print("*** var2 is not truthy ***")
... 
*** var2 is truthy ***

Admittedly, this is a bit artificial because in reality if you wanted to test for the existence of var2 you’d just do:

>>> if var2: print("*** var2 is truthy ***")
... else: print("*** var2 is not truthy ***")
...
*** var2 is truthy ***

But I did enjoy discovering this little nugget as it means you can easily count up the number of Trues in a list like so:

>>> sum([True,False,True,True,False,False,True])
4

Management and product development lessons from the 1950’s

2671775In 1955, Elihu Katz and Paul Lazarsfeld published “Personal Influence“. This studied how small-group dynamics moderate or influence mass media messaging. For example how people decide who to vote for, which brand of lipstick to use, or which movie to go and watch.

Reading this in 2018 it’s striking to see how much is still valid. I’m not posting this to provide tremendous new insights. Any insights here are over 60 years old. Apparently, human behaviour doesn’t change very much over the generations.

How people choose their leaders

In order to become a leader, one must share prevailing opinions and attitudes. (p52)

They cite a 1952 study on children in a day nursery in which kids with “leadership qualities” were separated from the other children who were then placed into groups of 3-6. These new groups created their own “traditions” (e.g. who sits where, group jargon, division of who plays with what objects). The original leaders were then re-introduced:

In every case, when an old leader attempted to assert authority which went contrary to a newly established “tradition” of the group, the group did not respond. Some of the leaders, as a matter of fact, never returned to power. Others, who were successful, did achieve leadership once more but only after they had completely identified with the new “tradition” and participated in it themselves. (p52)

Or another 1952 study amongst a religious interest group, a political group, a medical fraternity and a medical sorority:

[T]hose who had been chosen as leaders were much more accurate in judging group opinion … But this was so only on the matters which were relevant to the group’s interest – medicine for the medical group, politics for the political group, etc. It seems reasonable to conclude … that leaders of groups like this are chosen, in part at least, because of recognized qualities of ‘sensitivity’ to other members of the group. (p102)

A succinct argument as to why people who want to become leaders need to first spend time listening.

Group participation improves take-up

Here’s are some more 1952 studies that the authors cite:

  1. A study in a maternity hospital in which “some mothers were given individual instruction .. and others were formed into groups of six and guided in a discussion which culminated in a [group] ‘decision’ [to follow the instruction.” The participants in the group dicussion adhered “much more closely” to the child-care programme. (pp74-75).
  2. A study comparing a lecture approach vs a group discussion on “the nutritional and patriotic justifications for the wartime food campaign to buy and serve ‘unpopular’ cuts of meat. 3% of those involved in the lecture followed the desired course of action, vs 32% of those in the group discussion.

Worth bearing in mind in the next meeting you host, or the next corporate communication you send out.

How small groups construct their reality

So many things in the world are inaccessible to direct empirical observation that individuals must continually rely on each other for making sense out of things. (p55)

Apparently 1952 was a bumper year for social sciences. Here is another 1952 study in which individuals were asked to decide how far and in which direction a point of light was moving. The catch was that the point of light was static. The study found that:

  1. When people were shown the light individually, they would make their own judgment of how it was moving. When they were later put into small groups of 2 or 3, “[e]ach of the subjects based his first few estimates on his previously established standard, but confronted, this time, with the dissenting judgments of the others each gave way somewhat until a new, group standard became established.”
  2. If a group session came first, the group would achieve a consensus of how the light was moving, and each individual would adopt the group’s consensus as their own position.

The way reality is generated by social groups is something to bear in mind during user research activities.

How the make-up of a group affects quality of communication

You guessed it, it’s another 1952 study that found that:

  1. Rank in the group affects how people communicate. Specifically: “[P]-erson-to-person messaged are directed at the more popular group members and thus may be said to move upward in the hierarchy, while communication from one person to several others tends to flow down” (p89).
  2. As groups get larger (from 3 to 8) “more and more communication is directed to one member of the group, thus reducing the relative amount of interchange among all members with each other. At the same time the recipient of this increased attention begins to direct more and more of his remarks to the group as a whole, and proportionately less to specific individuals.” (pp89-90)

I’m sure these two findings will ring very true of many meetings you’ve been in. I suspect that the person who becomes the centralising leader in these communications might not even realise the role they are playing. Reading this makes me more keen to try out the kind of silent meetings approach they use at Square.

 

 

Using local settings in a Scrapy project

TL;DR Seeing a pattern used in one framework can help you address similar problems in a different framework.

I was working on a scrapy project that would save pages into a local database. We wanted to be able to test it using a test database, just as you would with Rails.

Wec could see how to configure a database connection string in the settings.py config file, but this didn’t help switch between development and test database

Stackoverflow wasn’t much help, so we ended up rolling our own:

  1. Edit the settings.py file so it would read from additional settings files depending on a SCRAPY_ENV environment variable
  2. Move all the settings files to a separate config directory (and change scrapy.cfg so it knew where to look
  3. Edit .gitignore so that local files wouldn’t get committed to the repo (and then added some .sample files)

Git repo is here.

The magic happens at the end of settings.py

from importlib import import_module
from scrapy.utils.log import configure_logging
import logging
import os

SCRAPY_ENV=os.environ.get('SCRAPY_ENV',None)
if SCRAPY_ENV == None:
    raise ValueError("Must set SCRAPY_ENV environment var")
logger = logging.getLogger(__name__)
configure_logging({'LOG_FORMAT': '%(levelname)s: %(message)s'})

# Load if file exists; incorporate any names started with an
# uppercase letter into globals()
def load_extra_settings(fname):
    if not os.path.isfile("config/%s.py" % fname):
        logger.warning("Couldn't find %s, skipping" % fname)
        return
    mdl=import_module("config.%s" % fname)
    names = [x for x in mdl.__dict__ if x[0].isupper()]
    globals().update({k: getattr(mdl,k) for k in names})

load_extra_settings("secrets")
load_extra_settings("secrets_%s" % SCRAPY_ENV)
load_extra_settings("settings_%s" % SCRAPY_ENV)

It feels a bit hacky, but it does the job, so I would love to learn a more pythonic way to address this issue.

Re-learning regexes to help with NLP

TL;DR Don’t forget about boring old-school solutions when you’re working with shiny new tech approaches. Dusty old tech might still have a part to play.

I’ve been experimenting with entity recognition. The use case is to identify company names in text. I’ve been using the fab service from dandelion.eu as my gold standard of what should be achievable.

Overall it is pretty impressive. Consider the following phrases:

Takeda is a gerbil

Dandelion recognises that this phrase is about a gerbil.

Takeda eats barley from Shire

Dandelion recognises that this is about barley

Takeda buys goods from Shire

A subtle change of words means this sentence is probably about a company called Takeda and a company called Shire.

Very cool.

But what about this sentence:

Polish media and telecoms group Cyfrowy Polsat (CPS.WA) said late Thursday it agreed to buy a controlling stake in sports content producer and distributor Eleven Sports Network Sp.z o.o. (ESN) for 38 million euros ($44.48 million).

Still pretty impressive but it has made one big fumble. It thinks that CPS stands for Canon, whereas in reality CPS.WA is the stock ticker for Cyfrowy Polsat.

Is there an alternative approach?

In this sort of document, company names are often abbreviated, or referenced by their stock ticker. When they are abbreviated they use the same kind of convention. Sounds like a job for a regex.

In case you think that something as old-school as regexes can’t possibly have a role to play with cutting edge code, bear in mind they are heavily used in NLP, so Dandelion must be using them somewhere under the covers. Or have a look at Scikit-learn’s CountVectorizer which uses a very simply regex for the token-pattern that it uses for splitting up text into different terms.

I can’t remember the last time I used a regex in anger. (See also this great stackoverflow question and answer on the topic). I don’t like just copy pasting from stack overflow so I broke the task down into a few steps to make sure I fully understood what was going on. The iterations I went through are below (paste this into a python console to see the results).

sent = "Polish media and telecoms group Cyfrowy Polsat (CPS.WA) said late Thursday it agreed to buy a controlling stake in sports content producer and distributor Eleven Sports Network Sp.z o.o. (ESN) for 38 million euros ($44.48 million)."
import regex as re
re.findall(r"\(.+\)",sent)
re.findall(r"\(.+?\)",sent)
re.findall(r"\([A-Z.]+?\)",sent)
re.findall(r"[A-Z]\w+\s\([A-Z.]+?\)",sent)
re.findall(r"(?:[A-Z]\w+\s)*\([A-Z.]+?\)",sent)
re.findall(r"((?:[A-Z]\w+\s*)*)\s\(([A-Z.]+?)\)",sent)
re.findall(r"((?:[A-Z]\w+\s(?:[A-Z][A-Za-z.\s]+\s?)*))\s\(([A-Z.]+)\)",sent)

To go through the iterations one by one with what I re-learnt at each stage.

>>> re.findall(r"\(.+\)",sent)
['(CPS.WA) said late Thursday it agreed to buy a controlling stake in sports content producer and distributor Eleven Sports Network Sp.z o.o. (ESN) for 38 million euros ($44.48 million)']

Find one or more characters inside a bracket. Brackets have special significance so need to be escaped. This simple regex finds the longest match (it’s “greedy”) so we need to change it to a “lazy” search.

>>> re.findall(r"\(.+?\)",sent)
['(CPS.WA)', '(ESN)', '($44.48 million)']

Better, but the only real initials are combinations of a capital and a dot, so specify that:

>>> re.findall(r"\([A-Z.]+?\)",sent)
['(CPS.WA)', '(ESN)']

Next I need to find the text before the initials. This would be a Capital letter followed by one or more other characters followed by a space:

>>> re.findall(r"[A-Z]\w+\s\([A-Z.]+?\)",sent)
['Polsat (CPS.WA)']

Getting somewhere, but really we need multiple words before the initials. So put brackets around the part of the regex that includes a capitalised word and match this one or more times

>>> re.findall(r"([A-Z]\w+\s)+\([A-Z.]+?\)",sent)
['Polsat ']

WTF? Makes no sense. Except it does: The brackets create a capture group. This changes findall’s behaviour. findall now returns the results of the capture group rather than the overall match. See below how using the captures() method returns the whole match.

>>> re.search(r"([A-Z]\w+\s)+\([A-Z.]+?\)",sent).captures()
['Cyfrowy Polsat (CPS.WA)']

Solution is to turn this new group that we created into a non-capturing group using ?:

>>> re.findall(r"(?:[A-Z]\w+\s)+\([A-Z.]+?\)",sent)
['Cyfrowy Polsat (CPS.WA)', '(ESN)']

A bit better, but it would be nice now if we can separate out the name from the abbreviation, so we return two capture groups in the regex:

>>> re.findall(r"([A-Z]\w+\s[A-Z](?:[\w.\s]+))\s\(([A-Z.]+?)\)",sent)
[('Cyfrowy Polsat', 'CPS.WA'), ('', 'ESN')]

At this point the regex is only looking for capitalised words before the abbreviation. Eleven Sports Network has some lower case terms in the name, so the regex needs a bit more tuning. The following example does the job in this particular case. It looks for a capitalised word then a space, a capital letter and then some other text until it gets to what looks like an abbreviation in brackets:

>>> re.findall(r"([A-Z]\w+\s[A-Z](?:[\w.\s]+))\s\(([A-Z.]+)\)",sent)
[('Cyfrowy Polsat', 'CPS.WA'), ('Eleven Sports Network Sp.z o.o.', 'ESN')]

You can see this regex in action on the fab regex101.com site. Let’s break this down:

(                           )\s\((       )\)
 [A-Z]\w+\s[A-Z]                  [A-Z.]+
                (?:[\w.\s]+)

  • Capturing Group 1
    1. [A-Z] a capital letter
    2. \w+ one or more letters (to complete a word)
    3. \s a space (either to start another word, or just before the abbreviation)
    4. [A-Z] then a capital letter, to start a 2nd capitalised word
      1. (?:[\w.\s]+) a non-capturing group of one or more letters or full stops (periods) or spaces.
  • \s\( a space and then an open bracket
  • Capturing Group 2:
    1. [A-Z.]+? one or more capital letters or full stops.
  • \) Close bracket

This regex does the job but it didn’t take long for it to become incomprehensible. If ever there were a use case for copious unit tests it’s using regexes.

Also, it doesn’t generalise well. It won’t identify “Acme & Bob Corp (ABC)” or “ABC and Partners Ltd (ABC)” or “X.Y.Z. Property Co (XYZ)” or “ABC Enterprises, Inc. (‘ABC’)” properly. Writing one regex to handle all of these types of string would quickly become very brittle and hard to understand. In reality I would end up using a serious of regexes rather than trying to code one regex to rule the all.

Nevertheless I hope it’s clear how a boring old piece of tech can help plug gaps in the cool new stuff. It’s never too late to (re-)learn the old ways. And it’s ok to put your pride aside and go back to basics.