Next generation of word embeddings

Short talk on when to use word2vec, word rank, fast text. A small review of the theory of word2vec. Code in https://gist.github.com/tmylk/14f887f8585e9f89ab5896a10308447c

Transcript

1. Next generation of word embeddings
   Lev Konstantinovskiy
   Community Manager at Gensim
   @teagermylk
   http://rare-technologies.com/
   

   View Slide

2. Streaming
   Word2vec and Topic Modelling in Python
   

   View Slide

3. Gensim Open Source Package
   ● Numerous Industry Adopters
   ● 170 Code contributors, 4000 Github stars
   ● 200 Messages per month on the mailing list
   ● 150 People chatting on Gitter
   ● 500 Academic citations
   

   View Slide

4. Gensim coding sprint
   Date: April
   Location: somewhere in BH, To Be Announced
   Interested? Contact me on pydatabh.slack.com
   Twitter @teagermylk
   lev@rare-technologies.com
   Topic:
   Learn machine learning by improving our tutorials.
   

   View Slide

5. Credits
   Parul Sethi
   Undergraduate student
   University of Delhi, India
   RaReTech Incubator program
   Added WordRank to Gensim
   http://rare-technologies.com/incubator/
   

   View Slide

6. View Slide

7. Part 1. Different word embeddings
   Part 2. Theory of word2vec
   

   View Slide

8. Business Problems
   

   View Slide

9. Business Problems
   “What is Dona Flor like?”
   “List all female characters in
   “Dona Flor e seus dois maridos”?”
   

   View Slide

10. Two Different
    Business Problems
    1) What words are in the topic of “Dona
    Flor”?
    2) What are the Named Entities in the text?
    

    View Slide

11. DFDM is only 170k words so
    results are so-so
    

    View Slide

12. Teodoro
    

    View Slide

13. Closest word to “king”?
    Trained on Wikipedia 17m words
    Attribute Interchangeable Both
    

    View Slide

14. How to get the similarity you need
    My similar words must
    be
    Associated Interchangeable
    I want to describe the
    word’s
    Topic Function
    I want to Know what doc is about Recognize names
    Then I should run Wordrank (even on small
    corpus, 1m words)
    or
    Word2vec skipgram big
    window needs large corpus
    >5m words
    Word2vec skipgram small
    window
    or
    FastText
    or
    VarEmbed
    

    View Slide

15. Part 2. Theory of Word2vec
    

    View Slide

16. Word2vec is big victory of unsupervised learning
    Google ran word2vec on 100billion of unlabelled words.
    Then shared their trained model.
    Thanks to Google for cutting our training time to zero!. :)
    

    View Slide

17. Word embeddings can be used for:
    - automated text tagging
    - recommendation engines
    - synonyms and search query expansion
    - machine translation
    - plain feature engineering
    

    View Slide

18. What is a word embedding?
    ‘Word embedding’ = ‘word vectors’ = ‘distributed representations’
    It is a dense representation of words in a low-dimensional vector space.
    One-hot representation:
    king = [1 0 0 0.. 0 0 0 0 0]
    queen = [0 1 0 0 0 0 0 0 0]
    book = [0 0 1 0 0 0 0 0 0]
    king = [0.9457, 0.5774, 0.2224]
    Distributed representation:
    

    View Slide

19. Many other ways to get a vector for a word:
    - Factorise the co-occurence matrix (SVD/LSA)
    - GLoVe
    - EigenWords
    - WordRank
    - VarEmbed
    - FastText
    Disclaimer
    Word2vec is not the only word embedding in the world
    

    View Slide

20. View Slide

21. Use the “Distributional hypothesis”:
    “You shall know a word by the company it keeps”
    -J. R. Firth 1957
    Richard Socher’s NLP course http://cs224d.stanford.edu/lectures/CS224d-Lecture2.pdf
    How to come up with an embeddig?
    

    View Slide

22. Usual procedure
    1.Initialise random vectors
    2. Pick an objective function.
    3. Do gradient descent.
    

    View Slide

23. For the theory, take Richard Sochers’s CS224D free online class
    Richard Socher’s NLP course http://cs224d.stanford.edu/lectures/CS224d-Lecture2.pdf
    

    View Slide

24. “The fox jumped over the lazy dog”
    Maximize the likelihood of seeing the context words given the word over.
    P(the|over)
    P(fox|over)
    P(jumped|over)
    P(the|over)
    P(lazy|over)
    P(dog|over)
    word2vec algorithm
    Used with permission from @chrisemoody
    http://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec
    

    View Slide

25. Probability should depend on the word vectors.
    Used with permission from @chrisemoody
    http://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec
    P(fox|over)
    P(v
    fox
    |v
    over
    )
    

    View Slide

26. A twist: two vectors for every word
    Used with permission from @chrisemoody
    http://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec
    Should depend on whether it’s the input or the output.
    P(v
    OUT
    |v
    IN
    )
    “The fox jumped over the lazy dog”
    v
    IN
    

    View Slide

27. Twist: two vectors for every word
    Used with permission from @chrisemoody
    http://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec
    Should depend on whether it’s the input or the output.
    P(v
    OUT
    |v
    IN
    )
    “The fox jumped over the lazy dog”
    v
    IN
    v
    OUT
    = P(v
    THE
    |v
    OVER
    )
    

    View Slide

28. Twist: two vectors for every word
    Used with permission from @chrisemoody
    http://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec
    Should depend on whether it’s the input or the output.
    P(v
    OUT
    |v
    IN
    )
    “The fox jumped over the lazy dog”
    v
    IN
    v
    OUT
    

    View Slide

29. Twist: two vectors for every word
    Used with permission from @chrisemoody
    http://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec
    Should depend on whether it’s the input or the output.
    P(v
    OUT
    |v
    IN
    )
    “The fox jumped over the lazy dog”
    v
    IN
    v
    OUT
    

    View Slide

30. Twist: two vectors for every word
    Used with permission from @chrisemoody
    http://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec
    Should depend on whether it’s the input or the output.
    P(v
    OUT
    |v
    IN
    )
    “The fox jumped over the lazy dog”
    v
    IN
    v
    OUT
    

    View Slide

31. Twist: two vectors for every word
    Used with permission from @chrisemoody
    http://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec
    Should depend on whether it’s the input or the output.
    P(v
    OUT
    |v
    IN
    )
    “The fox jumped over the lazy dog”
    v
    IN
    v
    OUT
    

    View Slide

32. Twist: two vectors for every word
    Used with permission from @chrisemoody
    http://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec
    Should depend on whether it’s the input or the output.
    P(v
    OUT
    |v
    IN
    )
    “The fox jumped over the lazy dog”
    v
    IN
    v
    OUT
    

    View Slide

33. Twist: two vectors for every word
    Used with permission from @chrisemoody
    http://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec
    Should depend on whether it’s the input or the output.
    P(v
    OUT
    |v
    IN
    )
    “The fox jumped over the lazy dog”
    v
    IN
    v
    OUT
    

    View Slide

34. Twist: two vectors for every word
    Used with permission from @chrisemoody
    http://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec
    Should depend on whether it’s the input or the output.
    P(v
    OUT
    |v
    IN
    )
    “The fox jumped over the lazy dog”
    v
    IN
    v
    OUT
    

    View Slide

35. How to define P(v
    OUT
    |v
    IN
    )? First, define similarity.
    How similar are two vectors?
    Just dot product for unit length vectors
    v
    OUT
    * v
    IN
    Used with permission from @chrisemoody
    http://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec
    

    View Slide

36. Get a probability in [0,1] out of similarity in [-1, 1]
    Normalization term over all out words
    Used with permission from @chrisemoody
    http://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec
    

    View Slide

37. Word2vec is great!
    Vector arithmetic
    Slide from @chrisemoody
    http://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec
    -
    -
    

    View Slide

38. @datamusing Sudeep Das
    http://www.slideshare.net/SparkSummit/using-data-science-to-transform-opentable-into-delgado-das
    More directions
    

    View Slide

39. Consistent directions
    Mikolov et al. Distributed Representations of Words and Phrases and their Compositionality 2013
    

    View Slide

40. Explore word2vec yourself
    http://rare-technologies.com/word2vec-tutorial/#app
    

    View Slide

41. Thanks!
    Lev Konstantinovskiy
    github.com/tmylk
    @teagermylk
    Contact me if interested
    Gensim sprint in BH
    (full day coding event)
    

    View Slide