Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Poincare Embeddings で遊んでみた

ryuji0123
May 14, 2021
69

Poincare Embeddings で遊んでみた

CompML の発表資料です。

ryuji0123

May 14, 2021
Tweet

Transcript

  1. CompML Embedding ͱ͸ (2/2) • ೖྗ: • ΦϒδΣΫτͷू߹: • ೋͭͷΦϒδΣΫτ

    ͷڞىؔ܎΍਌ࢠؔ܎Λࣔ͢σʔλ: • ग़ྗ: • ֤ΦϒδΣΫτͷ࠲ඪू߹: X = {xi |1 ≤ i ≤ N} xi , xj ∈ X D Y = {yi |1 ≤ i ≤ N} 5
  2. CompML Embedding ͷྫ: Word2Vec (1/2) Skip-Gram [Mikolov et al., 2013[1]]:

    ֓ཁ • ฏۉର਺໬౓Λ࠷େԽ͢Δ͜ͱͰ จষ ͔Β 
 ୯ޠϕΫτϧΛܭࢉՄೳ 
 
 1 T T ∑ t=1 ∑ −c≤j≤c,j≠0 logp(wt+j |wt ) p(wO |wI ) = exp(v′  T wO vwI ) ∑W w=1 exp(v′  T w vwI ) 6
  3. CompML Embedding ͷྫ: Word2Vec (2/2) Skip-Gram [Mikolov et al., 2013[1]]:

    ࣮ݧ • ʮDNN ͷ embedding ʹར༻ͯ͠λεΫͷੑೳΛධՁʯͱ͍͏ྲྀΕͰ͸ͳ͍ • ఆੑධՁͷ໘ന͞ͱֶश଎౓ΛΞϐʔϧ 7
  4. CompML Poincare Embeddings ͱ͸ (1/4) Poincare Space • ࠲ඪ ؒͷڑ཭͕ҎԼͰఆ·Δ૒ۂۭؒ

    
 • த৺͔Β཭ΕΔ΄Ͳ఺ͷ਺͕૿͑Δٿঢ়ͷۭؒͰɺ 
 ֊૚ੑͷ͋ΔΦϒδΣΫτΛ embed ͠΍͍͢ u, v d(u, v) = arcosh(1 + 2 ||u − v||2 (1 − ||u||2 )(1 − ||v||2 ) ) 9
  5. CompML Poincare Embeddings ͱ͸ (2/4) Embed ݁Ռͷྫ [Nickel et al.,

    2017[2]] ಈ෺ͷ֊૚ੑΛ Wordnet ͔Βநग़ • Mammal -> Rodent • Rodent -> Squirrel 10
  6. CompML Poincare Embeddings ͱ͸ (3/4) ֶशํ๏ (WordNet ͷ৔߹) 1. ਌ࢠؔ܎ͷ͋ΔΦϒδΣΫτೋͭͷ૊ͷू߹

    Λੜ੒ 2. ͔ΒωΨςΟϒαϯϓϧ Λੜ੒ 3. ҎԼͷଛࣦؔ਺Λ࠷దԽ֤ͯ͠ΦϒδΣΫτʹ࠲ඪΛ༩͑Δ D = {(u, v)|u ∈ v} D N( ⋅ ) 11
  7. CompML Poincare Embeddings ͱ͸ (4/4) ࣮ݧ: 3 छྨͷσʔληοτʹର͠ఆྔධՁ • Network

    Reconstruction, Link Prediction (DNN ͸ؔ܎ͳ͘ɺ 
 Poincare Space ͰΦϒδΣΫτͷۙ๣ؔ܎Λอ͍ͯͯΔ͔ධՁ) • ௿࣍ݩͰߴ͍ੑೳΛग़ͤΔ͜ͱΛΞϐʔϧ 12
  8. CompML Poincare Embeddings ʹ͍ͭͯͷٙ໰ ҎԼࡾͭͷσʔληοτҎ֎ʹ΋ Citation Network Λ࢖͑ͦ͏ • WordNet:

    ਌ࢠؔ܎͕ࣗ໌ͳ༗޲άϥϑ • ໦ߏ଄ͷ਌ࢠؔ܎Λͦͷ··ೖྗʹ࢖༻Մೳ • Co-author NetWork: ਌ࢠؔ܎͕ඇࣗ໌ͳແ޲άϥϑ • ڞஶάϥϑ͔ΒΦϒδΣΫτؒͷۙ๣֬཰Λܭࢉ • Lexical Entailment: ΦϒδΣΫτؒͷؚҙؔ܎Λࣔ͢ू߹ • X ͕ Y ʹଐ͢Δఔ౓Λࣔ͢஋͔ΒείΞ(ͱ Spearman’s ρ) Λܭࢉ 14
  9. CompML Citation Network ͱ͸ • ֓ཁ • ࿦จͷҾ༻ / ඃҾ༻ͷؔ܎Λࣔͨ͠άϥϑ

    • Ҿ༻ -> ࢠ, ඃҾ༻ -> ਌ ʹม׵͢Ε͹ Poincare Embeddings Λద༻Ͱ͖ͦ͏ 15
  10. CompML Citation Network Λ༻͍ͨ Poincare Embeddings (1/5) • ࣮ݧઃఆ •

    ֶश • Ҿ༻ -> ࢠ, ඃҾ༻ -> ਌ ʹม׵͠ೖྗσʔλੜ੒ • ೖྗσʔλʹର͠ Poincare Embeddings • ධՁ • ՄࢹԽʹΑΔఆੑධՁ • Network Reconstruction Error ʹΑΔఆྔධՁ • Ծઆ: WordNet ͱҟͳΓෳ਺ͷ໦ߏ଄͕ଘࡏ͢ΔͷͰධՁ͸Լ͕Δ 16
  11. CompML Citation Network Λ༻͍ͨ Poincare Embeddings (2/5) ఆੑධՁ dim =

    2 ͷ Embed ݁ՌΛՄࢹԽ • σʔλ͕ଟ͍ͷͰີू͍ͯ͠Δ • ֊૚͝ͱʹ෼཭͍ͯ͠ͳ͍ 17
  12. CompML Citation Network Λ༻͍ͨ Poincare Embeddings (3/5) ఆྔධՁ (1/2) MAP

    (Mean Average Precision) (ߴ͍΄Ͳྑ͍) • dim = 2 ͱͦͷଞͰ͕ࠩ͋Δ • Network ͰͷੑೳΑΓ΋ѱ͍ 18
  13. CompML Citation Network Λ༻͍ͨ Poincare Embeddings (4/5) ఆྔධՁ (2/2): Mean

    Rank (௿͍΄Ͳྑ͍) • dim = 2 ͱͦͷଞͰ͕ࠩ͋Δ • WordNet ͰͷੑೳΑΓ΋ѱ͍ 
 (Network Ͱͷ Mean Rank ͸ݪஶະهࡌ) 19
  14. CompML Citation Network Λ༻͍ͨ Poincare Embeddings (5/5) ࣮ݧͷ·ͱΊ • Citation

    Network ΛφΠʔϒʹֶशͤͯ͞΋࿦จ΄Ͳͷੑೳ͸ग़ͳ͔ͬͨ • ਌ͱͯ͠Χ΢ϯτ͞Ε͍ͯΔΦϒδΣΫτͷݸ਺౳Λ෼ੳͯ͠σʔλͷҧ͍ Λௐ΂Δඞཁ͋Γ • ࠓճ͸ Citation Network ͱ gensim ͷ WordNet ·Ͱ͸෼ੳࡁΈ • gensim ͱ Facebook Research Ͱσʔληοτ͕ҟͳΔͷͰɺ·ͩޙऀ ͷσʔλΛ෼ੳͰ͖ͯͳ͍ 20
  15. CompML ࢀߟจݙ [1] Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado,

    Greg, and Dean, Jeffrey. Distributed representations of phrases and their compositionality. In Advances on Neural Information Processing Systems, 2013. [2] Maximillian Nickel and Douwe Kiela. Poincare embeddings for learning hierarchical representations. In Advances in Neural Information Processing Systems, 2017. 21