Collision Conference: Beginner's Guide to Natural Language Processing (NLP)

Natural Language Processing (NLP) is a field of artificial intelligence which focuses on the machines' ability to read, understand and derive meaning from human languages.It is with no doubt one of the fields that has been studied for very long times, with research publications dating back to the early 1900's. And fast-forwarding to 2020, we still experience ground-breaking NLP research with new language models emerging nearly on a monthly basis.In this session, we will introduce you to Amazon Comprehend, a fully-managed NLP service, to find insights and relationships in text without any machine learning experience required.

Transcript

1. © 2020, Amazon Web Services, Inc. or its Affiliates. Antje Barth Developer Advocate AI/ML Amazon Web Services Beginner’s Guide to Natural Language Processing (NLP)
2. © 2020, Amazon Web Services, Inc. or its Affiliates. King - Man + Woman = Queen = ??
3. © 2020, Amazon Web Services, Inc. or its Affiliates. Agenda Introduction to NLP BERT-family of models NLP with Amazon Comprehend Demo
4. © 2020, Amazon Web Services, Inc. or its Affiliates. Introduction to NLP
5. © 2020, Amazon Web Services, Inc. or its Affiliates. Problem statement • Natural Language Processing (NLP) is a major field in AI • NLP apps require a language model in order to predict the next word • Vocabulary size can be hundreds of thousands of words … in millions of documents • Can we build a compact mathematical representation of language, that will help with a variety of domain-specific NLP tasks?
6. © 2020, Amazon Web Services, Inc. or its Affiliates. « You shall know a word by the company it keeps », Firth (1957) • Word vectors are built from co-occurrence counts Also called word embeddings High dimensional: at least 50, up to 300 • Words with similar meanings should have similar vectors “car” ≈ “automobile” ≈ “sedan” • The distance between vectors for the same concepts should be similar distance (“Ottawa”, ”Canada”) ≈ distance(“Berlin”, ”Germany”) distance(“hot”, ”hotter”) ≈distance(“cold”, ”colder”)
7. © 2020, Amazon Web Services, Inc. or its Affiliates. King - Man + Woman = Queen Source: https://www.ed.ac.uk/informatics/news-events/stories/2019/king-man-woman-queen- the-hidden-algebraic-struct
8. © 2020, Amazon Web Services, Inc. or its Affiliates. High-level view 1. Start from a large text corpus (100s of millions of words, even billions) 2. Preprocess the corpus into tokens • Tokenize: « hello, world! » à « <BOS>hello<SP>world<SP>!<EOS>» • Multi-word entities: « Rio de Janeiro » à « rio_de_janeiro » 3. Build the vocabulary from the tokens 4. Learn vector representations for the vocabulary … or simply use pre-trained models with existing vector representations (more on this later)
9. © 2020, Amazon Web Services, Inc. or its Affiliates. Popular NLP use cases Representation learning Machine Translation Text Classification Language Modeling Sentiment Analysis Named Entity Recognition Question Answering
10. © 2020, Amazon Web Services, Inc. or its Affiliates. Algorithms & Concepts
11. © 2020, Amazon Web Services, Inc. or its Affiliates. Evolution of NLP algorithms Jan 2013 Jan 2014 Jul 2016 Jun 2017 Feb 2018 Oct 2018 Shallow neural network Continuous bag-of-words and continuous skip-gram Global Vectors for Word Representation Matrix factorization Extension of Word2Vec: Each word is treated as a set of sub-words (character n-grams)
12. © 2020, Amazon Web Services, Inc. or its Affiliates. Limitations of Word2Vec (and family) • Some words have different meanings « Kevin, stop throwing rocks! » vs. « Machine Learning rocks » Word2Vec encodes the different meanings of a word as the same vector • Bidirectional context is not taken into account Previous words (left-to-right) and next words (right-to-left)
13. © 2020, Amazon Web Services, Inc. or its Affiliates. Evolution of NLP algorithms Jan 2013 Jan 2014 Jul 2016 Jun 2017 Feb 2018 Oct 2018 “Embeddings from Language Models” (Pseudo-)bi-directional context using two uni-directional LSTMs “Attention Is All You Need” Replace LSTM with Transformers implementing true bidirectional attention
14. © 2020, Amazon Web Services, Inc. or its Affiliates. Attention on sentence “This movie is funny, it is great” The movie is funny , it is This movie is funny , it is great This is , it is great
15. © 2020, Amazon Web Services, Inc. or its Affiliates. BERT-family of models
16. © 2020, Amazon Web Services, Inc. or its Affiliates. BERT Bidirectional Encoder Representations from Transformers https://arxiv.org/abs/1810.04805 https://github.com/google-research/bert • BERT improves on ELMo • Replace LSTM with Transformers, which deal better with long-term dependencies • Truly bidirectional architecture: left-to-right and right-to-left contexts are learned by the same network • Words are randomly masked during training to improve learning • Sentences are randomly paired to improve Next Sentence Prediction (NSP) • Pre-trained models: BERT Base and BERT Large Layers Hidden Units Parameters BERT base 12 768 110M BERT large 24 1024 340M
17. © 2020, Amazon Web Services, Inc. or its Affiliates. BERT Pre-Training and Fine-Tuning
18. © 2020, Amazon Web Services, Inc. or its Affiliates. BERT MANIA!
19. © 2020, Amazon Web Services, Inc. or its Affiliates. NLP with Amazon Comprehend Discover insights and relationships in text – no ML experience required
20. © 2020, Amazon Web Services, Inc. or its Affiliates. AI services Pre-trained AI services that require no ML skills or training Easily add intelligence to your existing applications and workflows Quality and accuracy from continuously learning APIs Vision Speech Text Search Chatbots Personalization Forecasting Fraud Development Contact centers Amazon Rekognition Amazon Polly Amazon Transcribe +Medical Amazon Comprehend +Medical Amazon Translate Amazon Lex Amazon Personalize Amazon Forecast Amazon Fraud Detector Amazon CodeGuru Amazon Textract Amazon Kendra Contact Lens For Amazon Connect
21. © 2020, Amazon Web Services, Inc. or its Affiliates. Amazon Comprehend Discover insights and relationships in text Amazon Comprehend Entities Key phrases Language Sentiment Topic modeling
22. © 2020, Amazon Web Services, Inc. or its Affiliates. Finding insights in social data PubNub uses Amazon Comprehend to analyze twitter posts using election related hashtags, creating new insights on public reaction to political events.
23. © 2020, Amazon Web Services, Inc. or its Affiliates. Acting on customer feedback Bark & Co uses Amazon Comprehend to analyze their customers’ comments and feedback, allowing them to respond faster and improve the customer experience.
24. © 2020, Amazon Web Services, Inc. or its Affiliates. Finding new clinical insights Deloitte uses Amazon Comprehend Medical to help their customers uncover actionable insights from medical records.
25. © 2020, Amazon Web Services, Inc. or its Affiliates. Demo
26. © 2020, Amazon Web Services, Inc. or its Affiliates. Custom classification • Triage support tickets • Moderate forums • Organize customer feedback • Organize support calls Classification: PRICING Classification: CANCEL_ACCOUNT Classification: LOYALTY_PROGRAM Example
27. © 2020, Amazon Web Services, Inc. or its Affiliates. Automated custom classifier training Create a .csv file with training data Train the service Classify 1 2 3 • Automated algorithm selection • Automated tuning and testing • SDK or code-free console UX https://registry.opendata.aws/amazon-reviews/ 5 I loved it! 1 Really bad. 3 It’s ok.
28. © 2020, Amazon Web Services, Inc. or its Affiliates. Demo https://github.com/data-science-on- aws/workshop/blob/0c1b2a95f74794a756a55d1f4d7abc7ed4d76f86/ 02_automl/03_Train_Reviews_Comprehend.ipynb
29. © 2020, Amazon Web Services, Inc. or its Affiliates. Get started
30. © 2020, Amazon Web Services, Inc. or its Affiliates. Get started on AWS https://ml.aws https://aws.amazon.com/marketplace/solutions/machine- learning/natural-language-processing https://aws.amazon.com/comprehend/ https://aws.amazon.com/training/learning-paths/machine-learning/ https://www.coursera.org/learn/aws-machine-learning
31. © 2020, Amazon Web Services, Inc. or its Affiliates. Thank you! Antje Barth @anbarth data-science-on-aws/workshop linkedin.com/in/antje-barth/