Text classification with not much text

A quick overview of GoCardless's attempts to auto-classify users by industry segment - or "vertical" - based on short descriptions they provide about what their business does.

Presented at PyData London on 01 Sept 2015.

Transcript

1. Text classification with not much text Angus Bayley @angusbayley
2. Text classification with not much text Angus Bayley @angusbayley Narration Slides
3. Text classification with not much text Angus Bayley @angusbayley I work at GoCardless - the payments processor. If you go to gocardless.com you can sign up for an account for free in about a minute, and then start taking direct debit payments online immediately. We’re always trying to understand our users better, and a good place to start is to find out what sort of business each of them is. Are they accountants, sports clubs, or are they web hosting firms? We could Google each merchant that signs up, go to their website and figure it out, but GoCardless is growing quite fast now so this is a problem we should automate.
4. We have a source of information we can use to do this programatically. Everyone that signs up for an account with us fills in this form…
5. …and there’s one question in particular that can help us here.
7. “We are a small accounting firm. This is for invoicing clients” “club membership fees” “payments” Answers range from lucid, comprehensive descriptions to a single nondescript word…
8. “We are a small accounting firm. This is for invoicing clients” “club membership fees” “payments” 0 15 30 45 60 Accountants Sports clubs Charities Web dev ... …but nevertheless we want to boil this down to a breakdown of the types of users we have. The marketing term for this is the vertical: we are trying to find in what verticals our users are in. So this is a text classification problem.
9. Workflow 
 descriptions = [“We are a small business tax advisor...”,
 “club membership fees”,
 “payments”
 ...] Our workflow is based around Scikit Learn, and specifically its Naive Bayes capability - which is a standard choice for problems of this type. We’ve got an 8k large training data set that we’ve accumulated over the years manually that we can use for this. The first step is to get the descriptions into a list.
10. Workflow 
 descriptions = [“We are a small business tax advisor...”,
 “club membership fees”,
 “payments”
 ...] description_vectors = CountVectorizer().fit_transform(descriptions) …after which we convert them to feature vectors using Scikit Learn’s CountVectorizer module.
11. Workflow 
 descriptions = [“We are a small business tax advisor...”,
 “club membership fees”,
 “payments”
 ...] description_vectors = CountVectorizer().fit_transform(descriptions) classifier = BernoulliNB() At this point the features are ready to be fit into a classification model. We’ve gone for Bernoulli Naive Bayes.
12. Workflow 
 descriptions = [“We are a small business tax advisor...”,
 “club membership fees”,
 “payments”
 ...] description_vectors = CountVectorizer().fit_transform(descriptions) classifier = BernoulliNB() cross_val_score(classifier, description_vectors, y=labels,
 scoring=‘accuracy') …and we do the testing using another Scikit Learn built-in module: cross_val_score. We’re scoring based on accuracy here, which is a typical scoring method for many-class problems. Accuracy is defined as the % of users categorised correctly. We applied this workflow in a number of ways whilst searching for the optimal approach for the problem.
13. Don’t work so well: • Multinomial Naive Bayes • Maximum document frequency (max_df) • Stemming Whilst doing so we came across a few things that didn’t work. Multinomial Naive Bayes - which accounts for words that appear twice in a description - didn’t make much sense because the descriptions are so short that they rarely contain duplicate words. Maximum document frequency filtering was also of little use. This is usually used to filter out words that appear in most of the descriptions (words like ‘the’, ‘and’, and ‘if’), but in this case people tend to write in bullet point style, which reduces the frequency of use of these kinds of words. Similarly, stemming didn’t help either. People really only write these descriptions in the present tense - so no real need for it.
14. “payments” Now, three things that did work. These are pretty specific to this problem, but should at least give you some food for thought. First - dealing with the pesky one-word responses.
15. “payments” “payments Bill’s Gym” + “ Bill’s Gym” It turns out that if you concatenate the description with the user name, you’ll often introduce an important extra keyword. You’ll almost always add a really useless one too (“Bill’s” in this case), but that’s ok, because you can then use Minimum Document Frequency filtering to remove them, leaving you - in this example - with just “payments” and “Gym”, which is really easy to classify. This might seem like quite a contrived example, but it’s actually quite representative of what we see in a lot of these cases.
16. Class grouping Second: grouping classes together. Before going into this I want to point out what a useful tool Scikit Learn’s Confusion Matrix method is. It plots predicted vertical against true vertical, and with an accurate model all of the data should reside on the diagonal. If your model is struggling with any particular class it is eminently obvious, and it clearly identifies the one that is problematic. This drew our attention to several verticals that actually overlapped in definition, so much so that even humans were having a hard time deciding which of them a user should belong to. The solution here was to group those verticals together. We did this, and the model’s accuracy jumped.
17. Selective classification Finally, selective classification. In our case it was preferable to have high accuracies, and at the same time it was acceptable not to classify absolutely everyone. Therefore extra gains in accuracy could be achieved by only classifying the users whose classification certainty (as given by BernoulliNB’s predict_proba method) was above a threshold amount. There’s no built in Scikit Learn method to do this that I know of, so I wrote this one myself. This allowed us to trade off the accuracy score with the % of users that were classified, leaving us with a smaller number of more accurately classified subjects.
18. Thanks! @angusbayley angus@gocardless.com That’s it. Please get in touch if you’ve got any questions, or if you have any ideas for how to make this better!