BERT for Text Classification with Keras/TensorFlow 2

In this workshop, we'll learn how we can utilize BERT, a technique for natural language processing (NLP) pre-training to perform our own tasks. For this workshop we will use BERT in TensorFlow 2 for a text classification task.

Transcript

1. BERT for Text Classification with Keras/TensorFlow 2 Galuh Sahid Data Scientist, Gojek / ML GDE
2. What will we do today?
3. This movie is awesome! Positive
4. Positive Negative This movie is thrilling! Such a disappointing ending.
5. This movie is thrilling! Positive Model
6. 1. Train everything from scratch 2. Use a pre-trained model Ways to do training
7. A deep learning model is trained on a large dataset, then used to perform similar tasks on another dataset (e.g. text classification) Transfer learning
8. What is BERT?
9. BERT: Bidirectional Encoder Representations from Transformers
10. “...we train a general-purpose ‘language understanding’ model on a large text corpus (like Wikipedia), and then use that model for downstream NLP tasks that we care about (like question answering)” https://github.com/google-research/bert
11. “BERT outperforms previous methods because it is the first unsupervised, deeply bidirectional system for pre-training NLP.” https://github.com/google-research/bert
12. BERT was trained using only a plain text corpus Unsupervised
13. • Pre-trained representations can also either be context-free or contextual Bidirectional bank bank deposit river bank
14. • Contextual representations can further be unidirectional or bidirectional Bidirectional I made a bank deposit I made a bank deposit
15. • Starts from the very bottom of a deep neural network Deeply bidirectional
16. BERT Training Strategies
17. Positive Negative This movie is thrilling! Such a disappointing ending.
18. • Masked language model • Next sentence prediction Training strategies
19. Input: the man went to the [MASK1] . he bought a [MASK2] of milk. Labels: [MASK1] = store; [MASK2] = gallon Masked language model
20. Sentence A: the man went to the store . Sentence B: he bought a gallon of milk . Label: IsNextSentence Next sentence prediction
21. Sentence A: the man went to the store . Sentence B: penguins are flightless . Label: NotNextSentence Next sentence prediction
22. • https://github.com/google-research/bert References
23. Hands-on Practice
24. bit.ly/wtm-bert-colab