Building a Language Identifier

Use some classic machine learning techniques to build a document classifier for language identification.

http://langue.herokuapp.com

Transcript

1. Building a Language Identifier
2. • ઍཬ೭ߦ﹐࢝ԙ଍Լɻ • Беда ́ (никогда ́) не прихо ́ дит одна ́. • A buen entendedor, pocas palabras bastan
3. Training Examples
5. • Wikinews • EuroParl corpus • UN Corpora • Hand crawled web pages
6. Extracting Features
7. n-grams “On ne peut désirer ce qu'on ne connaît pas.” ! on, ne, peut, désirer, ce, qu, on, ne, connaît, pas ! on ne, ne peut, peut désirer, désirer ce, ce qu, qu on, on ne, ne connaît, connaît pas
8. the pourquoi a ⁋࢐ antidisest ablishmen tarianism … 10 0 7 0 2 … 0 8 9 0 0 … 0 0 0 1 0 … 6 0 3 0 0 …
9. N distinct features (n-grams) in corpus M example documents
10. M × N × sizeof(double) N distinct features (n-grams) in corpus M example documents
11. Original shape: M × N
12. Original shape: M × N value 4 0 … 0 5 0 … 0 3 0 … 6 0 …
13. Original shape: M × N index value 1 4 2 0 … … 666 0 667 5 668 0 … … 986 0 987 3 989 0 … … 1037 6 1038 0 … …
14. Original shape: M × N
15. Original shape: M × N Index Value 1 4 667 5 987 3 1037 6 1408 10 2867 2 5680 1 7896 1 11763 4 15879 9
16. 0 0 2 10 7 Hasher the pourquoi a ⁋࢐ antidisestablishmentarianism
17. Learning
18. Multinomial Naive Bayes Work from home today? Any meetings today? Is it raining? Am I out of coffee? What’s the temperature outside? % yes % no >
19. label features en zh ko fr en Classifier
20. Cross Validating
21. features Classifier predictions known en en ja zh ko ko fr fr nl en 60% accurate
22. The Pipeline
23. Feature Extractor Classifier Document Prediction
24. Feature Extractor Classifier Document Prediction Feature Extractor
25. Feature Extractor Classifier Document Feature Transformer
26. Making Predictions
27. ઍཬ೭ߦ﹐࢝ԙ଍ԼɻPipeline Chinese http://langue.herokuapp.com
28. Questions? (ask me to repeat them) @zacstewart