Statistical Semantic入門 〜分布仮説からword2vecまで〜 @PFIセミナー

1. Statistical Semantic¨Ŭ ~ǔǮũ!Lword2vecC8~
   Ĥĸ™Preferred Infrastructure óµ ǧǺ (@unnonouno) 2014/02/06 PFIZg^k
   
2. œǕǿŽ
   !  óµǧǺ (@unnonouno) !  ƞŬ !  œƓüūŋÌ !  m[eo|VrŠ] !  ŸŒđúÌĔIBMċȀųPFI
   /
   
3. ĻƑ=g}q
   0
   
4. 1
   Semantics
   
5. ¨ŬœƓüūŋÌ [Bird+10]
   10Ǵ ī=ßƜ=ĐDz
   10.1 œƓüūÌĐ
   10.2 ŹøŅÌÒ 10.3 •ƳƐūŅÌ
   10.4 Əūī=ßƜŅ
   10.5 ŵõßƜŅ
   10.6 C9F
   10.7 ņěīǟ
   10.8 ǀǃáø
   2
   
6. œƓüūŋÌƬưǝƼhwoWXTƪÒ [–Ƃ +96]
   5. ßƜĐDz 5.1 ßƜĐDz9> 5.2 ÄǣÞŚ<ċ6%ßƜ°Ș ȗäĐŗ 5.3 ÊŸ¸Į<ċ6%ßƜ°Ș ȗäĐŗ 5.4 0=Ƹ=áø
   3
   
7. @9%3<ßƜŅ9ü47Gm|>ƅ
   !  Ü;ßƜŅWikipediaKM !  杝Dz !  ¤æßƜŅ !  žĸßƜŅ !  ȄŲßƜŅ !  ĒēßƜŅ
   4
   '=Ʀ=õ>*C.S
   
8. À‹=ÆƩ
   !  Statistical Semantics<¸Ê,Novk\<5 7ǭ%ę%C9FN !  ȋ5!=őů>•ǎ<Statistical Semantics= •Ǝ9*7>ūLO7;!G*OC.S !  •ǎ=ßƜŅ>Ǭȏ*C.S 5
   
9. Statistical Semantics9Distributional Semantics =Ū>
   !  (FS Q!LS!41 !  ľš>§+G=29Ġ47C, !  ľš=ŪRē47Nª1LĜ7½)
   6
   
10. ņǩÈį [Evert10]
    Ü<NAACL2010=Stefan EvertL=Distributional Semantic Models=j‚o…T†Rņě<*7C,
    .-
    
11. áø???=ƹū>Ŷ8*J
    ..
    [Evert10]KM
    aj]n=H#I& H(+P#?C8N
    
12. áø???=ƹū>Ŷ8*J
    ./
    [Evert10]KM
    aj]2ncatJpigE(&;C8C7knifeECF8
    
13. ìĐdog
    .0
    [Evert10]KM
    
14. ǔǮũ (Distributional Hypothesis)
    !  §+īȍ8”é,Nƹū>§+ßƜRû59'9 !  À‹=õ> Ń;!L-'=ǮũR÷Í*7N¸Ê *7NžÁÎǎRǦC, .1
    The Distributional Hypothesis is that words that occur in the same contexts tend to have similar meanings (Harris, 1954). (ACL wiki )
    
15. ŔĀ°ßƜŅ (Statistical Semantics)9>
    !  ū=ŔĀ°;äŨ!LßƜRěN
    !  Ȕ:= › " %=ŔĀ°;äŨR÷Í,N .2
    Statistical Semantics is the study of "how the statistical patterns of human word usage can be used to figure out what people mean, at least to a level sufficient for information access”(ACL wiki )
    
16. ū=ßƜ9 īȍ=¸Į
    .3
    &H I ) GKBC>ON
    &H GKBC ) ;>ON
    0L%ľªì*
    
17. ':G>:I47ßƜRÒS8N=! [Àč 13]
    !  ]‡xw†lG\ˆ‰k bŠG 5#)C !  ƌ9ƃ*!ēL; ²Ż<ǰƾR¡.1L !  ^†RēL;²Ż> ŕ8q_†9 .4
    Ƹ=`ot9=¸Į"Œ˜
    
18. ĝĽ°<8#0;'9
    !  §ŞūĢ©ūƱŞū !  ū9ū=ßƜ=¸ĮRǦNK<;N !  ąŞū !  īȍ<ʼn+7 :=ßƜ8ħQO7N=!"Q! N !  ƻēū´ū !  ƯƦīȍ!L ƻē=ƹū=ßƜ"ƱƘ8#N !  Ǿū !  ƿ;Nüū®8©ʼn,Nƹū=¸Į"¿L!< .5
    
19. ;/'=õR,N=!
    !  ļij;ĜƲđúRŀÝ9*;1F üūn i)O?ƷÍ8#N !  PFI=ŦĴ=Ǎ&7N³Rśƽ*7%ON! G*O; !  ćǶ‡`Šp=§ŞūƤŮūŞȘȗäĐŗ !  īĦƱ< &NĜƲni=Öŕ !  ąüūćǶ<ŀÝ;ǾūǛĦ !  '=1=®<ÆǂC*æĨ"úġ)O7N .6
    
20. ŒǪȉ<>35=ŋÌR—
    !  üūŋÌ°;­ŋÌR— !  ex: ĪÍRȖ, ŒīſËīſ ǚ›ū etc… !  ƹūRīȍ<K47Ôé,N !  ex: īĦ±ķŜ ī±ķŜ ǯƗǸ ƹªòǞªò ĮMî& ÓȌ-Æ°ū¸Į etc… !  ƹūÔéRC%Ôé*ĝ, !  ex: —Ǔye ĬŊye NNye etc… !  DZƛȐ ĬŊ€n† NN€n†Gƅ /-
    
21. ȈŁßƜĐDz: Latent Semantic Indexing (LSI), Latent Semantic Analysis (LSA) [Deerwester+90]
    !  đúćǶ=`}‚rmU8¤CO1 !  9 :'=ÈįRǘS8GŶƚ!ãÏ)O7 N%LćǶ9üūŋÌ>ǜ"ƍ !  ƹūR 0=ƹū"”é*7NīĦĿ›9* 7Ô, !  '=ƹūy\o†>#49ȓ–;=8ĚĎƭĄ 8#N /.
    
22. LSI=ƙĐ
    //
    ńƹū":=īĦ<Ŷ ¶”7N=!=—Ǔ
    U
    ∑
    V
    =
    x
    x
    (SVD)
    i
    k: KM>8
    ƹūiRkĚĎ8Ôé *7N
    &
    
    
23. LSI=[€j
    !  ƹū=ßƜ>īȍy\o†8ºCN !  ''8>īȍ9>īĦ±ȑÂ='9 !  ÇĈ>—Ǔ=ėMª>Ĕŭ=*:'P¬Ɛ !  īȍy\o†>ȓ–;=8SVDRħ47 G4 9ţĚĎ=y\o†8Ôé8#N>-
    /0
    
24. īȍy\o†=ėMªO'O
    !  ƹū-īĦ—Ǔ ƹū-ƹū—Ǔ etc. !  ī±ķŜ ǯƗǸbVf ƥƫ=İij etc. !  ĮMî&¸Į ÓȌ-Æ°ū¸Į Šīù’=Ɖ ²¸Į etc.
    /1
    
25. żīȍǫǐ=Ū<KNŐȁ
    /2
    E:,E:
    Vj_iE:`ckE:
    
26. Ü<5=ëÉ=ªòä
    /3
    LSI
    PLSI
    LDA
    NMF
    NNLM
    %&h\i
    "$
    NN RNNLM
    Skip- gram
    NTF
    
27. üū€n†Ƣ
    !  LSIƠ=ĬŊ°üū€n†90=ëÉƢ !  Ü<īǴ=ovk\9 ovk\!L=ƹū"¤æ)O N'9R€n†É,Novk\€n†9*7ëÉ Good !  ƟĖÿƩ"•ǹ*7 M ÇŴƟĖ*I, !  Ƹ=ĬŊ€n†9ýD›Q.I, Bad !  ĀIJ"þ1 /4
    
28. 0=­< ĬŊ°üū€n†9>
    !  `ot>NĬŊǔ!LbŠx…Š])O7N 9Ǯ¯,N !  ơǒ)O11%)S=m[eoni!L G9=`o t=ǔRƘ¯,N !  :=K;ĬŊǔRçĀ,N! :=K<ƀŊ°< ǔRƘ¯,N!"ǵ=¡.:'P
    /5
    ī
    
29. ĬŊ°ȈŁßƜĐDz Probabilistic Latent Semantic Indexing (PLSI) [Hofmann99]
    !  LSI9Ʊǽ=ßƜĵ&"8#NK;€n† !  ovk\<K47ƿ;Nƹū"”I, !  ex: e{l;LµĺbkZı¾ £ Ć;LŒŔŷÄťŽ;: !  Vd9*7> LSI8ƹūRţĚĎ<Ŏ9* 1=9§ƅ< ƹūGīĦGovk\=ĿCM 9Ġ47ĬŊ€n†<;NK<çĀ /6
    
30. PLSI=GŃ*3HS9*1ũ¿
    !  ńīĦ<>ovk\=ǔ"ºC47N !  ex: bkZ9óâ=õø"”I, D1;Ȋǐè !  ńƹū<©*7 !  ovk\ǔ<Ʈ475ovk\RºFN !  0=ovk\!LƹūR@95ºFN !  'ORǑMŢ*7īĦ"¤æ)O19Ġ
    0-
    
31. Latent Dirichlet Allocation (LDA) [Blei03]
    !  PLSI=ëÉƢ !  PLSI8>ńīĦ=ovk\ǔ"lj¯241" LDA8>'OG¤æ).3H 0.
    
32. LDA>:S:SëÉ*7%
    !  ovk\€n†9•ŒµRǖ# ÙGþ Ý;NLPT†a…f~=159*7ðLO7 N !  ĶǠ;: üūñâ=ni<GʼnÍ)O7 N !  ûŧäI ĥ¦)S=ÈįRņǩ*7½)
    0/
    
33. üū€n†ƢžÁ=Řå
    !  ĬŊ€n†;=8Ƹ=ĬŊ°žÁ9Ñä"Ř !  ĬŊƄ=dz&IJ<œƓ;ßƜĵ&"8#N !  ex: ŔĀ°đúćǶ ŔĀ°×ȆȒǾ etc. !  ƟĖªÁ"Ĭ¹)O7N !  ؝ǔ"Ƙ¯)O7N=8O? éÇ=œƓ ī=ĬŊ"¥%;47N>- !  ăï"ŀ-1.0<;NK<Ïš)O7N1F e `TĬŊ=Ǘ©Ƅ<ßƜ"N 00
    
34. —ǓĐƢ
    !  ƹū=īȍ—ǓN>mŠh†RėM ţ„Š\8ĕǽ,N !  Y…dq†=LSIGSVDR÷Í*7N Good !  —ǓĐ=m\rk\"0=CCħN Bad !  ßƜĵ&R,N="Ű* !  ĬŊ€n†9*7ħ;1F Ƹ=ʼnÍ9= Ɖïä"ţ
    01
    
35. Non-negative Matrix Factorization (NMF) [Lee +99]
    !  ¼ş=SVD8”7%Nœ=Įð=ßƜĵ&"Ű* !  ì=Ƅ<;NK;ÞÐ=G98—ǓRĐ,N !  ĊĨ"eue<;M ßƜGØMI,˜"ą
    02
    3HS9ŠæÝNj<Đ)O7N
    [Lee+99]KM
    
36. NMF = PLSI [Dinga+08]
    !  NMF9PLSI> Ç>§+Æ°¸ð8N'9"¬< !41 !  ij<¢¿)O1žÁ"Ç>§+ßƜRû471'9 "=3<;47Q!N'9>Ƶş<ą 03
    FQEF<CN
    NMF
    PLSI
    
37. Non-negative Tensor Factorization (NTF) [Cruys10]
    !  —Ǔ2935ñ“=¸ĮR¨OLO; !  2ĚĎ;S7ü47;83ĚĎ<*7 mŠh †Đ*K
    04
    
38. —ǓĐƢžÁ=Řå
    !  —Ǔm\rk\"0=CCħCQ.N !  ƋǗ¥ƇSVDT†a…f~"N9 0=CCĐDz "¥Ƈ<;N 05
    
39. r‚„†sko‰\Ƣ
    !  ƹū=y\o†ÔéRr‚„†sko8Òǃ !  Ùĕ=word2vecŅī8ŏ<õø<Üơ8, Good !  ƱǽÂñ“=ßƜ=Ny\o†Ôé"ÇŴ°<ďLO 1C2K%!47;)0 Bad !  ĀIJ"Œł ÇLJGŒł0 !  ƾK%Q!L;
    06
    
40. Neural Network Language Model (NNLM) [Bengio +03]
    !  N]„~üū€n†R NNÉ !  ÊêN-1īſ!L Ě =īſRÛ7NĬŊ€ n†=r‚„†sk oRŠǖ,N
    1-
    
41. Recurrent Neural Network Language Model (RNNLM) [Mikolov+10]
    !  t-1īſǘS29#=Œŝ Ry\o†É*7 tīſÆR 0=Œŝ!LÛ7N !  NNLM8>ĝ­Nīſ=y\o† !LĚRÛ771 !  ĝ­C8=īȍđú"ǡFĉ CO7NȊǐè !  http://rnnlm.org
    1.
    īſ ƹū
    «Ǚ =ǤOǏ
    ǤOǏ
    Ě=¨» =āǒ
    `v
    
42. RNNLM8Òǃ)O1Ôé"ŔūŠűIßƜŠűR Ģnj*7N'9"Q!47#1 [Mikolov+13a]
    !  ŶRĠ41! y\o†Rŕ*1Mí1M*1 L
    1/
    
43. RNNLM9Transition-based parser=¸Į
    !  RNNLM=€n†>Transition-based parser<ǽ7N K<¡NƁ¡ !  Stack"recurrent;y\o†<©ʼn*7NK<¡N !  Ŕū°;ļǢGÔé8#7N=>0=. !  ż? àƹé=Üū"#19đú"ǡFĉCO7 N=8> 10
     >Q'KM
    
44. Skip-gram€n† (word2vec) [Mikolov+13b]
    !  ƯƦƹūRāǒ,N€n† !  ƯƦƹū!Lāǒ,N€n† CBOWGĘČ*7N !  Analogical reasoning=NJÂ" ǻ°<ò“ !  ÇLJ"ÕÃ)ON G>IParser":'9!;S 241= 11
    
45. Skip-gram€n†[Mikolov+13b]=Æ°¸ð
    !  ¨»`ue: w 1 , w 2 , …, w T w i >ƹū
    12
    'ORÙ ŒÉ
    vw I&wP#@NK9FeU]il*FmD7 =OLP?A8
    cI WSYD5<L8
    
46. Ă9§+ÇŴ=ĊĨ"
    !  ƨƓNJ“"MĻ$8>
    13
    
47. ÇLJ“=m\rk\G:S:SëÉ [Mikolov+13c]
    !  Îƹū=ăïRØN9'PRC%bzN
    14
    bdTk
    Rd[k
    
48. word2vec";/ǼƔ°241!
    !  ƹū=y\o†Ôé>ÀC8G41 !  ƱǽDŽ=ļǢ>Ɲ1*71" ŕ*1Mí1M9 41ǥė>ÀC88#;!41 !  üū€n†Ƣ> ċ°<ī±ķŜR€n†É*7N1F ovk\9Œ#ȎÂ*!ȕLO7; !  —ǓĐƢ8 œ=e`T=ßƜĵ&<ƺƖ*7NMF"ƶÍ) O1'9RĠ”0 !  ßƜ=Šű>G49ǚǪ8 ¨Ų<ĹŇ)O1üūÈǁ 9ňDž";9Ôé8#;9Ġ471 15
    
49. ßƜ=ƆÂ"y\o†Ō®’<ǡFĉCON [Kim+13]
    !  “good”9”best”=ĞS’< C)!=”better”"ƕŁ
    16
    
50. üū®=ȒǾǛĦ"8#N [Mikolov+13d]
    !  ƹüū=`ue8ėLO1Ôéy\o†>ǽ7N !  Ń;©ǾǛĦ8ė41 Ôéy\o†Ō®=ŖžłƧ RėN 2-
    !&
    XfSj&
    
51. NNƢžÁ=''"Ř
    !  ș=äŨ"Ě9¿L!< !  ßƜ=ĝĽ°;äŨ" Ŷƚ!y\o†Ō®’<ǡ FĉCO7N !  ŕ*1Mí1MôLj941M !  B9S:"2013<úġ)O7N !  Mikolov>'=2&815ĕŅīR”*7N !  1AS Ŷ!"Q!N=>'O!L
    2.
    
52. ÆRdžF7DN9
    !  ŶRīȍ9,N! !  üū€n†Ƣī±ķŜ ĝ­Nƹū !  —ǓĐƢī±ķŜ !  NNƢĝ­Nƹū ǯƗǸ !  :łƧ,N! !  üū€n†ƢǤOłð<Ŏ9, !  —ǓĐƢţ„Š\<Ŏ9, !  NNƢÆ°¸ð"ٌÙË<;NK<Ïš 2/
    žÁ=Ū>O: ¢Ÿ>ǽ7N
    
53. À¬:;47%!
    !  NNƢžÁ=Ũ°<þÝ;³"ØM”)ON !  ;/ÀC8C%—!;!41=! Ũ°;þÝ ;Ū>Ŷ;=! !  ij=ßƜĵ&Iij=ÇéªÁ"”é,N !  —ǓĐƢ=ªÁ8ģé8#N9 ǻ°<Òǃ«® RĄL*1M Œňƒ<ĀIJ8#NK<;N!G !  ʼn͝µ=Ĭ¹ !  C2 ;S!ğƈ‡y† !  ÇĈ=Tx…^cƒŠ<ʼnÍ)O7% 20
    
54. C9F
    !  Statistical Semantics9>;<! !  ŔĀ°;äŨ Ü<ƹū=”éīȍ=äŨ9 0= ū=ßƜ=¸Į<!S,NųƊ !  Œ#%&735=ªòä !  —ǓĐƢ üū€n†Ƣ r‚„†skoƢ !  NNƢžÁ>ÖĠÅ !  ŕ*1M í1M ôLj941M !  Ȃź=NNüū€n†>C2ĭC41?!M2
    21
    
55. ņěīǟ1ř¨
    !  [Bird+10] Steven Bird, Ewan Klein, Edward Loper, ȇÚì‘, ’ ·Ƿę, öµƣ¿. . Y„V…duŠ, 2010. !  [–Ƃ+96] –ƂĞŤĩÌƴ. 
    !$#
     . ƬưĦō, 1996. !  [Evert10] Stefan Evert. Distributional Semantic Models. NAACL 2010 Tutorial. !  [Àč13] ÀčE5D. & . ȃȅĦǨ, 2013. !  [Deerwester+90] Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, Richard Harshman. Indexing by Latent Semantic Analysis. JASIS, 1990. 22
    
56. ņěīǟ2üū€n†Ƣ—ǓĐƢ
    !  [Hofmann99] Thomas Hofmann. Probabilistic Latent Semantic Indexing. SIGIR, 1999. !  [Blei+03] David M. Blei, Andrew Y. Ng, Michael I. Jordan. Latent Dirichlet Allocation. JMLR, 2003. !  [Lee+99] Daniel D. Lee, H. Sebastian Seung. Learning the parts of objects by non-negative matrix factorization. Nature, vol 401, 1999. !  [Ding+08] Chris Ding, Tao Li, Wei Peng. On the equivalence between Non-negative Matrix Factorization and Probabilistic Latent Semantic Indexing. Computational Statistics & Data Analysis, 52(8), 2008. !  [Cruys10] Tim Van de Cruys. A Non-negative Tensor Factorization Model for Selectional Preference Induction. Natural Language Engineering, 16(4), 2010. 23
    
57. ņěīǟ3NNƢ1
    !  [Bengio+03] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin. A Neural Probabilistic Language Model. JMLR, 2003. !  [Mikolov+10] Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan "Honza" Cernocky, Sanjeev Khudanpur. Recurrent neural network based language model. Interspeech, 2010. !  [Mikolov+13a] Tomas Mikolov, Wen-tau Yih, Geoffrey Zweig. Linguistic Regularities in Continuous Space Word Representations. HLT-NAACL, 2013. !  [Mikolov+13b] Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. CoRR, 2013. 24
    
58. ņěīǟ4NNƢ2
    !  [Mikolov+13c] Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. NIPS, 2013. !  [Kim+13] Joo-Kyung Kim, Marie-Catherine de Marneffe. Deriving adjectival scales from continuous space word representations. EMNLP , 2013. !  [Mikolov+13d] Tomas Mikolov, Quoc V. Le, Ilya Sutskever. Exploiting Similarities among Languages for Machine Translation. CoRR, 2013.
    25