Facebook Dev-C Meetup| NLP & Architecture

Transcript

1. NLP & Architecture Session
2. Topics Covered • Text Processing & Cleaning. • Sentence Tagging. • Classification VS Handmade Rules.
3. Text Processing & Cleaning
4. Word Tokenization From “my dog hasn't any fleas.” To [ ‘my’, ‘dog’, ‘has’, ’n’\t’, ‘any’, ‘fleas’, ‘.’ ]
5. Word Token vs Word Type • Word Token: Total number of words in a given sentence, regardless how often they are repeated. • Word Type: Total Number of Distinct words in a sentence.
6. Example The cushions are new and I can experience the comfort well. Tokens: 11 Types: 10
7. Different Tokenizing Algorithms
8. Stemming Stemmers: Identifying the root word from a token. (e.g fishing, fished, and fisher) their root is “fish”
9. Algorithms Porter. (Most commonly used, less aggressive, computational intensive) e.g “animal -> anim”, “any -> ani“ Lancaster. (Very aggressive) e.g “animal -> anim”, “any -> any”
10. String Distance • Jaro-Winkler • Levenshtein
11. Jaro-Winkler Example
12. Levenshtein Example
13. Other Techniques • Inflectors. • Phonetics. • N-Grams. • Spellcheck. • Part of speech tagging. • tf-idf
14. Sentence Taging
15. What’s Meant By • I want to read a book. • I want to eat sandwich. • I want to play football.
16. What’s Meant By • I want to read that book. • I read that book. • Did you read that book ? • Can you get me a book to read ? • I don’t want to read that book.
17. Identify Question Sentences • Do you have my pen? • How old are you? • What is your name?
18. Sentences With Numeric Values • I am 23 years old. • I am twenty three years old. • I am the twenty third.
19. Sentences With Names • Books by J.K Rowling. • My name is John.
20. Identify Sentence Pronouns • Can you help me. • What are you reading today. • Did you read thinking fast and slow. • Remind me tomorrow at 9 AM.
21. What’s Meant By • I want to read that book. • I read that book. • Did you read that book ? • Can you get me a book to read ? • I don’t want to read that book.
22. Classification & Handmade Rules
23. Supervised ML • Document. • Fixed Set of classes. • Training data
24. Classifier Example
25. Type of classifiers that are used for NLP • Naive Bayes. • Logistic regression. • Support-vector machines. • k-Nearest Neighbors.
26. Handmade Rules
27. What’s Meant By • I want to read a book. // Action => recommend Book • I want to eat sandwich. // Action => order sandwich • I want to play football. // Action => find suitable time
28. What is meant by • I want to read that book. • I read that book. • Did you read that book ? • Can you get me a book to read ? • I don’t want to read that book.
29. Tools & Libraries
30. Examples • wit.ai • api.ai (Dialogflow) • Facbook NLP Engine • Open source libraries for NLP Classifiers.
31. Topics Covered • Text Processing & Cleaning. • Sentence Tagging. • Classification VS Handmade Rules.
32. Architecture
34. Useful Links • Stanford Natural Language Processing Course: https://goo.gl/FFbyBd • Andrew NG, Machine Learning Course: https://goo.gl/AXkdyj • Recommended AI Books List: https://goo.gl/Sz9vyX