Upgrade to Pro — share decks privately, control downloads, hide ads and more …

BERT for Text Classification with Keras/TensorFlow 2

BERT for Text Classification with Keras/TensorFlow 2

In this workshop, we'll learn how we can utilize BERT, a technique for natural language processing (NLP) pre-training to perform our own tasks. For this workshop we will use BERT in TensorFlow 2 for a text classification task.

2b6d7bdd43058e87f53866eb86538a59?s=128

Galuh Sahid

October 24, 2020
Tweet

Transcript

  1. BERT for Text Classification with Keras/TensorFlow 2 Galuh Sahid Data

    Scientist, Gojek / ML GDE
  2. What will we do today?

  3. This movie is awesome! Positive

  4. Positive Negative This movie is thrilling! Such a disappointing ending.

  5. This movie is thrilling! Positive Model

  6. 1. Train everything from scratch 2. Use a pre-trained model

    Ways to do training
  7. A deep learning model is trained on a large dataset,

    then used to perform similar tasks on another dataset (e.g. text classification) Transfer learning
  8. What is BERT?

  9. BERT: Bidirectional Encoder Representations from Transformers

  10. “...we train a general-purpose ‘language understanding’ model on a large

    text corpus (like Wikipedia), and then use that model for downstream NLP tasks that we care about (like question answering)” https://github.com/google-research/bert
  11. “BERT outperforms previous methods because it is the first unsupervised,

    deeply bidirectional system for pre-training NLP.” https://github.com/google-research/bert
  12. BERT was trained using only a plain text corpus Unsupervised

  13. • Pre-trained representations can also either be context-free or contextual

    Bidirectional bank bank deposit river bank
  14. • Contextual representations can further be unidirectional or bidirectional Bidirectional

    I made a bank deposit I made a bank deposit
  15. • Starts from the very bottom of a deep neural

    network Deeply bidirectional
  16. BERT Training Strategies

  17. Positive Negative This movie is thrilling! Such a disappointing ending.

  18. • Masked language model • Next sentence prediction Training strategies

  19. Input: the man went to the [MASK1] . he bought

    a [MASK2] of milk. Labels: [MASK1] = store; [MASK2] = gallon Masked language model
  20. Sentence A: the man went to the store . Sentence

    B: he bought a gallon of milk . Label: IsNextSentence Next sentence prediction
  21. Sentence A: the man went to the store . Sentence

    B: penguins are flightless . Label: NotNextSentence Next sentence prediction
  22. • https://github.com/google-research/bert References

  23. Hands-on Practice

  24. bit.ly/wtm-bert-colab