Keywordfinder: automatic keyword extraction from text

As an Insight Data Science Fellow, I completed a 3-week project that involved building a keyword extraction algorithm. Given a block of text as input, my algorithm selects keywords that describe what the text is about.

Transcript

1. Extracting keywords from text Lavanya Sharan Insight Data Science Fellow with
3. matching ads to content
5. keywords
6. keywords matching ad
7. keywords matching ad
8. keywords matching ad ?
9. keywords matching ad ?
10. keywords matching ad ?
11. keywords matching ad ?
12. A machine learning model for keyword extraction Content Candidate keywords Feature extraction Keyword Classifier Keywords
13. A machine learning model for keyword extraction Content Candidate keywords Feature extraction Keyword Classifier Keywords
14. A machine learning model for keyword extraction Content Candidate keywords Feature extraction Keyword Classifier Keywords Brooklyn
15. A machine learning model for keyword extraction Content Candidate keywords Feature extraction Keyword Classifier Keywords Term  length   Wikipedia  freq   TF-­‐IDF  score   ... Brooklyn
16. A machine learning model for keyword extraction Content Candidate keywords Feature extraction Keyword Classifier Keywords Term  length   Wikipedia  freq   TF-­‐IDF  score   ... Brooklyn Logistic regression
17. A machine learning model for keyword extraction Crowd500 500 news articles Human-annotated keywords 9:1 training-test split Content Candidate keywords Feature extraction Keyword Classifier Keywords Term  length   Wikipedia  freq   TF-­‐IDF  score   ... Brooklyn Logistic regression
18. A machine learning model for keyword extraction Content Candidate keywords Feature extraction Keyword Classifier Keywords Term  length   Wikipedia  freq   TF-­‐IDF  score   ... Brooklyn Logistic regression
19. A machine learning model for keyword extraction P(keyword) = 0.81 Content Candidate keywords Feature extraction Keyword Classifier Keywords Term  length   Wikipedia  freq   TF-­‐IDF  score   ... Brooklyn Logistic regression
20. A machine learning model for keyword extraction P(keyword) = 0.81 Content Candidate keywords Feature extraction Keyword Classifier Keywords Term  length   Wikipedia  freq   TF-­‐IDF  score   ... Brooklyn Logistic regression
21. A machine learning model for keyword extraction P(keyword) = 0.81 nltk Content Candidate keywords Feature extraction Keyword Classifier Keywords Term  length   Wikipedia  freq   TF-­‐IDF  score   ... Brooklyn Logistic regression
23. State-of-the-art performance on Crowd500
28. About me glass Keywords: Lavanya, image recognition, foodie
29. Keyword classifier “Brooklyn” Term frequency TF-IDF score Wikipedia frequency Term length Capitalized? Position in page Spread in page Named entity? Noun phrase? Ngram? Logistic regression model In-sample: 65%, out-of-sample: 65%, chance: 50%
30. Simple heuristics beat complex features
31. All stages of model required for performance
32. Beats AlchemyAPI for a range of parameters
33. Beats AlchemyAPI for a range of parameters