Gain speed and space / precision with NLP in Solr

Transcript

1. Tobias Kässmann - @kaessmannt Gain speed and space with NLP in Solr precision
2. # Father, Husband, Hamburg # Software Engineer @shopping24 # Search Technology Hamburg
3. Tobias Kässmann - @kaessmannt
4. Tobias Kässmann - @kaessmannt < 3M 100M time Products #discspace (indexsize) #precision #nosharding
5. Tobias Kässmann - @kaessmannt 1. Reduce the indexed data / text! simple winnings:
 speed & space
6. Tobias Kässmann - @kaessmannt Product descriptions vs. usual text Solutions for:
7. Tobias Kässmann - @kaessmannt SEO / IRRELEVANT TEXT SEO / IRRELEVANT TEXT descriptions: lots of useless text / signals RELEVANT TEXT
8. Tobias Kässmann - @kaessmannt SEO text = evil = irrelevant
9. Tobias Kässmann - @kaessmannt …fits pretty well to… …it will give a really good feeling… …combine it with… …enjoy it at home on your couch… …will bring you a lot of fun on the road…
10. Tobias Kässmann - @kaessmannt "We can use fancy stuff like …" Keyword extraction Neural Networks OpenNLP Framework SVM's Deeplearning
11. Tobias Kässmann - @kaessmannt Named entity recognition Product pipeline Gathering / Preprocessing / Normalization … indexing
12. Tobias Kässmann - @kaessmannt
13. Tobias Kässmann - @kaessmannt Key assumption:
 "Sentences in product descriptions with useful information do not contain a lot of stop words"
 (from the view of a search engine) List sentences = splitToSentences(cleanUp(description)); for(s : sentences){ if(s contains a lot of signs){ // split again } int stopwordCount = countStopwords(s); int wordCount = countWords(s); if(stopwordCount / wordCount < threshold){ // sentence is interesting } } ~21%
14. Tobias Kässmann - @kaessmannt • Power of RegExp
 (libs with heuristics are not as fast)
 • Do not analyze text that is just one sentence • Split sentences again Learnings
15. Tobias Kässmann - @kaessmannt Word reduction of ~60-80%
16. Tobias Kässmann - @kaessmannt No SEO text or product description? Keywordextraction: RAKE-algorithm
17. Tobias Kässmann - @kaessmannt • RAKE-algorithm: • Uses stopwords and punctuation as boundaries • Calculates a score for each candidate • Returns 1/3 of the top candidates as keyword result Rake: How it works Solr is an open source enterprise search platform, written in Java, from the Apache Lucene project. candidate boundary
18. Tobias Kässmann - @kaessmannt • Enhancements: • Define domain specific "stop words": 
 additional word types, more signs, urls… • Propagate score from overlapping / related keywords • (Additional filtering) Rake:
19. Tobias Kässmann - @kaessmannt mail: work@tobiaskaessmann.de dev blog: developer.s24.com @github: shopping24/solr-analyzers