Upgrade to Pro — share decks privately, control downloads, hide ads and more …

BERT for Text Classification with Keras/TensorFlow 2

Galuh Sahid
October 24, 2020

BERT for Text Classification with Keras/TensorFlow 2

In this workshop, we'll learn how we can utilize BERT, a technique for natural language processing (NLP) pre-training to perform our own tasks. For this workshop we will use BERT in TensorFlow 2 for a text classification task.

Galuh Sahid

October 24, 2020
Tweet

More Decks by Galuh Sahid

Other Decks in Technology

Transcript

  1. BERT for Text
    Classification with
    Keras/TensorFlow 2
    Galuh Sahid
    Data Scientist, Gojek / ML GDE

    View Slide

  2. What will we
    do today?

    View Slide

  3. This movie is awesome! Positive

    View Slide

  4. Positive
    Negative
    This movie is thrilling!
    Such a disappointing ending.

    View Slide

  5. This movie is
    thrilling!
    Positive
    Model

    View Slide

  6. 1. Train everything from scratch
    2. Use a pre-trained model
    Ways to do training

    View Slide

  7. A deep learning model is trained on a
    large dataset, then used to perform
    similar tasks on another dataset (e.g.
    text classification)
    Transfer learning

    View Slide

  8. What is BERT?

    View Slide

  9. BERT: Bidirectional Encoder
    Representations from
    Transformers

    View Slide

  10. “...we train a general-purpose
    ‘language understanding’ model
    on a large text corpus (like
    Wikipedia), and then use that
    model for downstream NLP tasks
    that we care about (like
    question answering)”
    https://github.com/google-research/bert

    View Slide

  11. “BERT outperforms previous
    methods because it is the first
    unsupervised, deeply
    bidirectional system for
    pre-training NLP.”
    https://github.com/google-research/bert

    View Slide

  12. BERT was trained using only a plain
    text corpus
    Unsupervised

    View Slide

  13. ● Pre-trained representations can also
    either be context-free or contextual
    Bidirectional
    bank bank deposit river bank

    View Slide

  14. ● Contextual representations can further
    be unidirectional or bidirectional
    Bidirectional
    I made a bank deposit
    I made a bank deposit

    View Slide

  15. ● Starts from the very bottom of a deep
    neural network
    Deeply bidirectional

    View Slide

  16. BERT Training
    Strategies

    View Slide

  17. Positive
    Negative
    This movie is thrilling!
    Such a disappointing ending.

    View Slide

  18. ● Masked language model
    ● Next sentence prediction
    Training strategies

    View Slide

  19. Input: the man went to the [MASK1] .
    he bought a [MASK2] of milk.
    Labels: [MASK1] = store; [MASK2] =
    gallon
    Masked language model

    View Slide

  20. Sentence A: the man went to the store .
    Sentence B: he bought a gallon of milk .
    Label: IsNextSentence
    Next sentence prediction

    View Slide

  21. Sentence A: the man went to the store .
    Sentence B: penguins are flightless .
    Label: NotNextSentence
    Next sentence prediction

    View Slide

  22. ● https://github.com/google-research/bert
    References

    View Slide

  23. Hands-on
    Practice

    View Slide

  24. bit.ly/wtm-bert-colab

    View Slide