BERT for Text Classification with Keras/TensorFlow 2

In this workshop, we'll learn how we can utilize BERT, a technique for natural language processing (NLP) pre-training to perform our own tasks. For this workshop we will use BERT in TensorFlow 2 for a text classification task.

Transcript

1. BERT for Text
   Classification with
   Keras/TensorFlow 2
   Galuh Sahid
   Data Scientist, Gojek / ML GDE
   

   View Slide

2. What will we
   do today?
   

   View Slide

3. This movie is awesome! Positive
   

   View Slide

4. Positive
   Negative
   This movie is thrilling!
   Such a disappointing ending.
   

   View Slide

5. This movie is
   thrilling!
   Positive
   Model
   

   View Slide

6. 1. Train everything from scratch
   2. Use a pre-trained model
   Ways to do training
   

   View Slide

7. A deep learning model is trained on a
   large dataset, then used to perform
   similar tasks on another dataset (e.g.
   text classification)
   Transfer learning
   

   View Slide

8. What is BERT?
   

   View Slide

9. BERT: Bidirectional Encoder
   Representations from
   Transformers
   

   View Slide

10. “...we train a general-purpose
    ‘language understanding’ model
    on a large text corpus (like
    Wikipedia), and then use that
    model for downstream NLP tasks
    that we care about (like
    question answering)”
    https://github.com/google-research/bert
    

    View Slide

11. “BERT outperforms previous
    methods because it is the first
    unsupervised, deeply
    bidirectional system for
    pre-training NLP.”
    https://github.com/google-research/bert
    

    View Slide

12. BERT was trained using only a plain
    text corpus
    Unsupervised
    

    View Slide

13. ● Pre-trained representations can also
    either be context-free or contextual
    Bidirectional
    bank bank deposit river bank
    

    View Slide

14. ● Contextual representations can further
    be unidirectional or bidirectional
    Bidirectional
    I made a bank deposit
    I made a bank deposit
    

    View Slide

15. ● Starts from the very bottom of a deep
    neural network
    Deeply bidirectional
    

    View Slide

16. BERT Training
    Strategies
    

    View Slide

17. Positive
    Negative
    This movie is thrilling!
    Such a disappointing ending.
    

    View Slide

18. ● Masked language model
    ● Next sentence prediction
    Training strategies
    

    View Slide

19. Input: the man went to the [MASK1] .
    he bought a [MASK2] of milk.
    Labels: [MASK1] = store; [MASK2] =
    gallon
    Masked language model
    

    View Slide

20. Sentence A: the man went to the store .
    Sentence B: he bought a gallon of milk .
    Label: IsNextSentence
    Next sentence prediction
    

    View Slide

21. Sentence A: the man went to the store .
    Sentence B: penguins are flightless .
    Label: NotNextSentence
    Next sentence prediction
    

    View Slide

22. ● https://github.com/google-research/bert
    References
    

    View Slide

23. Hands-on
    Practice
    

    View Slide

24. bit.ly/wtm-bert-colab
    

    View Slide