Gensim Open Source Package ● Numerous Industry Adopters ● 170 Code contributors, 4000 Github stars ● 200 Messages per month on the mailing list ● 150 People chatting on Gitter ● 500 Academic citations
Credits Parul Sethi Undergraduate student University of Delhi, India RaReTech Incubator program Added WordRank to Gensim http://rare-technologies.com/incubator/
What is a word embedding? ‘Word embedding’ = ‘word vectors’ = ‘distributed representations’ It is a dense representation of words in a low-dimensional vector space. One-hot representation: king = [1 0 0 0.. 0 0 0 0 0] queen = [0 1 0 0 0 0 0 0 0] book = [0 0 1 0 0 0 0 0 0] king = [0.9457, 0.5774, 0.2224] Distributed representation:
Co-occurence matrix ... and the cute kitten purred and then ... ... the cute furry cat purred and miaowed ... ... that the cute kitten miaowed and she ... ... the loud furry dog ran and bit ... kitten context words: [cute, purred, miaowed]. cat context words: [cute, furry, miaowed]. dog context words: [loud, furry, ran, bit]. Credit: Ed Grefenstette https://github.com/oxford-cs-deepnlp-2017/
Dimensionality reduction More precisely, in word2vec u*v approximates PMI(X) - log n, in Glove log(X) cute furry bit … kitten 2 0 0 ... cat 1 1 0 ... dog 0 1 1 ... ... ... ... ... ... X = = U * V Dims: vocab x vocab = (vocab x small) * (small x vocab) First row of U is the word embedding of “kitten”
Dimensionality reduction More precisely u*v approximates PMI(X) - log n, where n is the negative sampling parameter Co-occurence score in word2vec = U word * V context Dims: count = (small vector) * (small vector)
FastText: word is a sum of its parts Co-occurence score in FastText = U subword * V context over all subwords of w going = go + oi + in + ng + goi + oin + ing
FastText better than word2vec because morphology Credit: Takahiro Kubo http://qiita.com/icoxfog417/items/42a95b279c0b7ad26589 Slower because many more vectors to consider!
Same API as word2vec. Out-of-vocabulary words can also be used, provided they have at least one character n-gram present in the training data. FastText Gensim Wrapper
WordRank is a Ranking Algorithm Word2vec Input: Context Cute Output: Word Kitten Classification problem WordRank Input: Context Cute Output: Ranking 1. Kitten 2. Cat 3. Dog Robust: Mistake at the top of the rank costs more than mistake at the bottom.
Algorithm Train time (sec) Passes through corpus Related accuracy Related (WS-353) Interchange able accuracy Interchange able (SimLex-999 ) Word2Vec 18 6 4.69 0.37 2.77 0.17 FastText 50 6 6.57 0.36 36.95 0.13 WordRank 4hrs 91 15.26 0.39 4.23 0.09 Evaluation on 1 mln words corpus
How to get the similarity you need My similar words must be Related Interchangeable I want to describe the word’s Topic Function I want to Know what doc is about Recognize names Then I should run Wordrank (even on small corpus, 1m words) or Word2vec skipgram big window needs large corpus >5m words Word2vec skipgram small window or FastText or VarEmbed
How to get the similarity you need “Similar words” are Related Interchangeable Got a million words? FastText Word2vec small context VarEmbed WordRank Word2vec large context Yes No
Thanks for listening! Lev Konstantinovskiy github.com/tmylk @teagermylk Coding sprint today 13:30 - 15:30 “Learn NLP by running Gensim tutorials” “Turn your FullFact project into a Gensim tutorial” Fix an easy bug on github.
tmylk
javiergs
keitaro1020
ryunen344
itohiro73
meihei3
ivargrimstad
dicoding
tanakahisateru
kanamasa
mpyw
yuhei_fujita
nagworld
ammeep
lara
phodgson
smashingmag
roundedbygravity
geeforr
jensimmons
danielanewman
hatefulcrawdad
ddemaree
kastner
kneath