文献紹介 8月10日

Improving Distributional Similarity
with Lessons Learned from Word Embeddings

Transcript

1. ( Improving Distributional Similarity with Lessons Learned from Word Embeddings
2. 
   ▪  ▪ Omer Levy, Yoav Goldberg, Ido Dagan. ▪ Improving Distributional Similarity with Lessons Learned from Word Embeddings. ▪ Transactions of the Association for Computational Linguistics. 2015. ▪  ▪
   , Skip-gram, LSA 2
3. 
    ▪ , pi ur nu p i S a clulroug a S ▪ ) ▪ pi u tea , pi pi a a 3
4. 
    ▪ V ▪ 2 count predict ▪ Don’t count, predict! (Baroni et al., 2014) ▪ Word2Vec Skip-gram Shifted-PMI ▪ Neural word embeddings as implicit matrix factorization (Levy and Goldberg, 2014) 4
5. 
    ▪ 2 ▪ hyperparameter ▪ Word2Vec 5
6. 
    ▪ ))( (- 2 ▪ )( D L AW P ▪ 2 - - 2 ▪ - - 2 - 2 VI M W ▪ GW))( (- 2 V M P ▪ ▪ 6
7. 
   
    ▪ 2 ▪ C D 7 The quick brown fox jumps over the lazy dog 4 4 3 4 2 4 1 4 4 4 3 4 2 4 1 4 Word2Vec : 1 1 1 2 1 3 1 4 1 1 1 2 1 3 1 4 Glove :
8. 
   
    ▪
   
     ▪ '()&$,.-/+ ▪  #  # % # ▪ !"$) 8 ! = 1 − % & &: , %: 
   
9. 
   
    ▪ ▪ R G MR ▪ - - S ▪ Shifted PMI ▪ - 2 PID ▪ −log(&) W S NV 9 ( ) *( + = -./ 0, 2 − log(&) 3--./ 0, 2 = max(-./ 0, 2 − log(&), 0)
10. 
    
     ▪ 7 0 7 577 2 ▪ 7 0.SW 0 0 5 SV I ca db S W M b e P ▪ V S ▪ α WD C 10
11. 
    
     ▪ ▪ 2 G SV SV D ▪ G D AC 11
12. 
    
     ▪ ▪ E D ▪ E D C D 12
13. 
    
     ▪ ▪ ▪ 2 13
14. 
    
     14
15. 
    ▪
     ▪ PPMI-Matrix ▪ SVD ▪ SGNS (Word2Vec) ▪ Glove ▪  ▪   6$-"!#% ▪ &,-"  2$-"!#% ▪ '*(-")$+  ▪ 672)$+ 
16. 
    16 ▪  # "!# 
    ▪  SVD  
17. 
    17 ▪ )%+ '*(-$ 
    ▪   ▪ &,"-$# SGNS  
18. 
     ▪ - Se N ▪ - - - n y bc ▪ N smp n y i ▪ G tu d H uo a 18