Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to NLP

Introduction to NLP

Ristek Data Science Internal Class

Galuh Sahid

August 04, 2021
Tweet

More Decks by Galuh Sahid

Other Decks in Technology

Transcript

  1. Introduction to


    NLP
    Galuh Sahid
    @galuhsahid | github.com/galuhsahid

    View full-size slide

  2. https://bit.ly/2Tplxbl
    @galuhsahid

    View full-size slide

  3. NLP tasks
    @galuhsahid

    View full-size slide

  4. @galuhsahid
    Text Classification

    View full-size slide

  5. @galuhsahid
    Text Summarisation
    Source

    View full-size slide

  6. @galuhsahid
    Text Clustering
    Source

    View full-size slide

  7. @galuhsahid
    Machine Translation

    View full-size slide

  8. @galuhsahid
    Text Generation
    Source

    View full-size slide

  9. @galuhsahid
    Speech Recognition
    Try out the model! Source code

    View full-size slide

  10. @galuhsahid
    Image Captioning
    Source

    View full-size slide

  11. @galuhsahid
    Question Answering
    Source

    View full-size slide

  12. Text Classification
    @galuhsahid

    View full-size slide

  13. @galuhsahid
    Text Classification
    INFO RESMI Tri Care SELAMAT Nomor Anda terpilih
    mendapatkan Hadiah 1 Unit MOBIL Dri Tri Care Dengan
    PIN Pemenang: br25h99 info: www.gebeyar-3care.tk
    Da sebenernya makan sama indomie ge cukup aku
    mah 😂
    RAWIT harganya miring, Internetan makin sering! AYO
    isi ulang kartu AXIS kamu sekarang, pastikan besok
    ikut promo Rabu RAWIT. Info838. LD328
    Fraud
    Normal
    Promo

    View full-size slide

  14. @galuhsahid
    Source

    View full-size slide

  15. @galuhsahid
    Source

    View full-size slide

  16. Approach #1: scikit-learn
    @galuhsahid

    View full-size slide

  17. @galuhsahid
    Overview
    Preprocess Transform Model fitting

    View full-size slide

  18. @galuhsahid
    Preprocessing
    There is no magic recipe, but generally you will need to do the following:


    • Tokenizing


    • Lowercasing


    • Lemmatizing


    • Stemming


    • Stopwords removal


    • … anything else?

    View full-size slide

  19. @galuhsahid
    Preprocessing
    There is no magic recipe, but generally you will need to do the following:


    • Remove HTML tags


    • Remove extra whitespaces


    • Convert accented characters to ASCII characters


    • Remove special characters, punctuations


    • Remove numbers


    • Convert slang words


    • Replace number with tag


    • … and many more

    View full-size slide

  20. @galuhsahid
    Transform
    Source

    View full-size slide

  21. @galuhsahid
    Transform
    Source
    Review 2: This movie is not scary and is slow
    Vocabulary: ‘This’, ‘movie’, ‘is’, ‘very’, ‘scary’, ‘and’, ‘long’, ‘not’,
    ‘slow’, ‘spooky’, ‘good’


    Number of words in Review 2 = 8


    TF for the word ‘this’ = (number of times ‘this’ appears in review 2)/
    (number of terms in review 2) = 1/8


    View full-size slide

  22. @galuhsahid
    Transform
    Source

    View full-size slide

  23. @galuhsahid
    Transform
    Source
    Review 2: This movie is not scary and is slow
    IDF(‘this’) = log(number of documents/number of documents containing
    the word ‘this’) = log(3/3) = log(1) = 0

    View full-size slide

  24. @galuhsahid
    Transform
    Source

    View full-size slide

  25. @galuhsahid
    Transform
    Source
    Review 2: This movie is not scary and is slow
    TF-IDF(‘this’, Review 2) = TF(‘this’, Review 2) * IDF(‘this’) = 1/8 * 0 = 0

    View full-size slide

  26. @galuhsahid
    Transform
    Source

    View full-size slide

  27. @galuhsahid
    Model Fitting
    Source
    There are many algorithms that we can experiment with, e.g.:


    • Random Forest


    • SVM


    • … and many more

    View full-size slide

  28. Approach #2: Transformer-based
    @galuhsahid

    View full-size slide

  29. @galuhsahid
    Ways to do training
    • Train everything from scratch


    • Use a pre-trained model

    View full-size slide

  30. @galuhsahid
    Transfer learning
    A deep learning model is trained on a large dataset, then used to perform
    similar tasks on another dataset (e.g. text classification)

    View full-size slide

  31. @galuhsahid
    BERT
    BERT: Bidirectional Encoder Representations from Transformers

    View full-size slide

  32. @galuhsahid
    “...we train a general-purpose ‘language understanding’ model on
    a large text corpus (like Wikipedia), and then use that model
    for downstream NLP tasks that we care about (like question
    answering)”
    https://github.com/google-research/bert

    View full-size slide

  33. @galuhsahid
    “BERT outperforms previous methods because it
    is the first unsupervised, deeply
    bidirectional system for pre-training NLP.”

    View full-size slide

  34. @galuhsahid
    Unsupervised?
    BERT was trained using only a plain text corpus

    View full-size slide

  35. @galuhsahid
    Bidirectional?
    Pre-trained representations can also either be context-free or contextual
    ban
    k

    bank deposit
    river bank

    View full-size slide

  36. @galuhsahid
    Bidirectional?
    Contextual representations can further be unidirectional or bidirectional

    View full-size slide

  37. @galuhsahid
    BERT Training Strategies
    • Masked language model


    • Next sentence prediction

    View full-size slide

  38. @galuhsahid
    Masked language model
    • Input: the man went to the [MASK1] . he bought a [MASK2] of milk.


    • Labels: [MASK1] = store; [MASK2] = gallon

    View full-size slide

  39. @galuhsahid
    Next sentence prediction
    • Sentence A: the man went to the store .


    • Sentence B: he bought a gallon of milk .


    • Label: IsNextSentence

    View full-size slide

  40. @galuhsahid
    Next sentence prediction
    • Sentence A: the man went to the store .


    • Sentence B: penguins are flightless .


    • Label: NotNextSentence

    View full-size slide

  41. @galuhsahid
    Demo

    View full-size slide

  42. @galuhsahid
    Pro cons between the two approaches

    View full-size slide