trrex: Efficient Keyword Extraction with Regular Expressions

This slides introduce the trrex package, for performing efficient keyword extraction with regular expressions. Additionally I'll show how to integrate it with pandas for text cleaning, how to use it with spacy to build a gazetteer and how to perform fuzzy matching with the regex library.

Transcript

1. trrex Efficient Keyword Extraction with Regular Expressions
2. Who I am • Hi, I’m Daniel • Research Engineer & Data Scientist • #SOReadyToHelp • Software Engineer (Java 😞) at Free Now
3. Who I am • Hi, I’m Daniel • Research Engineer & Data Scientist • #SOReadyToHelp • Software Engineer (Java 😞) at Free Now 🚕 Free Now 🚕
4. Why regular expressions? Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.
5. Why regular expressions? • Powerful • Ubiquitous • Fast
6. Why regular expressions? • Powerful • Ubiquitous • Fast?!!
7. Fast? 🤔 import re long_list_of_words = [...] pattern = "|".join(long_list_of_words) print(re.search(pattern, "Need to find this keyword fast"))
8. Fast? 🤔 import re long_list_of_words = [...] pattern = "|".join(long_list_of_words) print(re.search(pattern, "Need to find this keyword fast")) <re.Match object; span=(18, 25), match='keyword'>
9. Fast? 🤔
11. trrex pip install trrex trrex = trie + regex
12. trrex ['bank', 'bat', 'baby', 'air']
13. trrex ['bank', 'bat', 'baby', 'air']
14. trrex '\\b(?:ba(?:nk|by|t)|air)\\b' ['bank', 'bat', 'baby', 'air']
15. trrex import trrex as tx pattern = tx.make(["baby", "bat", "monkey", "monster"], prefix="", suffix="")
16. trrex import trrex as tx pattern = tx.make(["baby", "bat", "monkey", "monster"], prefix="", suffix="") '(?:mon(?:ster|key)|ba(?:by|t))'
17. trrex vs union-regex
18. trrex vs flashtext
19. RegEx Integrations (the 🍒 on the 🍰)
20. pandas import trrex as tx import pandas as pd df = pd.DataFrame(["The quick brown fox", "jumps over", "the lazy dog"], columns=["txt"]) pattern = tx.make(["dog", "fox"]) df["txt"].str.contains(pattern)
21. pandas import trrex as tx import pandas as pd df = pd.DataFrame(["The quick brown fox", "jumps over", "the lazy dog"], columns=["txt"]) pattern = tx.make(["dog", "fox"]) df["txt"].str.contains(pattern) 0 True 1 False 2 True Name: text, dtype: bool
22. pandas re pandas search contains, extract findall findall, extractall sub replace
23. regex import trrex as tx import regex words = ["cat", "dog", "monkey", "monster"] pattern = tx.make(, prefix="", suffix=r'{1<=e<=2}') regex.search(pattern, "This is really a master dag", regex.BESTMATCH)
24. regex import trrex as tx import regex words = ["cat", "dog", "monkey", "monster"] pattern = tx.make(, prefix="", suffix=r'{1<=e<=2}') regex.search(pattern, "This is really a master dag", regex.BESTMATCH) <regex.Match object; span=(24, 27), match='dag', fuzzy_counts=(1, 0, 0)>
25. spacy import trrex as tx from spacy.lang.en import English nlp = English() pattern = tx.make(["Amazon", "Apple", "Netflix", "Netlify"]) nlp.add_pipe("entity_ruler").add_patterns( [ { "label": "ORG", "pattern": [{"TEXT": {"REGEX": pattern}}], }, ] )
26. spacy doc = nlp("Netflix HQ is in Los Gatos.") [(ent.text, ent.label_) for ent in doc.ents]
27. spacy doc = nlp("Netflix HQ is in Los Gatos.") [(ent.text, ent.label_) for ent in doc.ents] [('Netflix', 'ORG')]
28. repo ⭐ https://github.com/mesejo/trex ⭐
29. THANKS! 🎉