自然言語処理と深層学習の最先端

第4回 JustTechTalk の発表資料

Transcript

1. ࣗવݴޠॲཧͱਂ૚ֶशͷ࠷ઌ୺ ಙӬ୓೭ +VTU5FDI5BML
   
2. ࣗવݴޠॲཧͱਂ૚ֶशͷ࠷ઌ୺ ͷҰ෦෼Λ঺հ͠·͢ ಙӬ୓೭ (@tkng) +VTU5FDI5BML
   
3. ࣗݾ঺հɿಙӬ୓೭ • Twitter ID: @tkng • εϚʔτχϡʔεגࣜձࣾͰࣗવݴޠॲཧ΍ ը૾ॲཧΛ΍ͬͯ·͢
5. ࣗવݴޠॲཧͱ͸ • ࣗવݴޠʢ≠ϓϩάϥϛϯάݴޠʣΛѻ͏෼໺ • ػց຋༁ • ࣭໰Ԡ౴ • จॻ෼ྨ • ߏจղੳɾ܎Γड͚ղੳ • ܗଶૉղੳɾ୯ޠ෼ׂ
6. ػց຋༁ͷྫ • Google຋༁ͷword lensػೳ IUUQHPPHMFUSBOTMBUFCMPHTQPUKQIBMMPIPMBPMBUPOFXNPSFQPXFSGVM@IUNM
7. ࣭໰Ԡ౴ͷྫ • IBM Watson • Jeopardy!Ͱਓؒʹউར IUUQXXXOZUJNFTDPNTDJFODFKFPQBSEZXBUTPOIUNM
8. ਂ૚ֶशͱ͸ • ≒ χϡʔϥϧωοτ • ۙ೥ͷྲྀߦ͸ɺҎԼͷཧ༝ʹΑΔ • ܭࢉػͷੑೳ޲্ • ֶशσʔλͷ૿Ճ • ࠷దԽख๏ͳͲͷݚڀͷਐల
9. ࣗવݴޠॲཧͱ ਂ૚ֶशͷ࠷ઌ୺
10. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (Xu+, 2015) • ը૾ʹର͢Δղઆจͷੜ੒ IUUQLFMWJOYVHJUIVCJPQSPKFDUTDBQHFOIUNM
11. Show, Attend and Tell͸ Ͳ͏͍͏ख๏͔ • ҎԼͷ3ͭͷ૊Έ߹Θͤ • Convolutional Neural Network • Long Short Term Memory • Attention
12. Generating Images from Captions with Attention (Mansimov+, 2015) • Ωϟϓγϣϯ͔Βը૾Λੜ੒͢Δ • ࡉ໨ͰݟΕ͹ඈߦػʹݟ͑ͳ͘΋ͳ͍
13. Effective Approaches to Attention- based Neural Machine Translation (Bahdanau+, 2015) • Deep LearningΛ༻͍ͯػց຋༁ • Local Attentionͱ͍͏৽͍͠ख๏ΛఏҊ • ͍͔ͭ͘ͷݴޠϖΞͰɺstate of the artΛୡ੒ ࠷ߴਫ४
14. Ask Me Anything: Dynamic Memory Networks for Natural Language Processing (Kumar+, 2015) • ৽͍͠ϞσϧʢDynamic Memory Networksʣ ΛఏҊͨ͠ • Recurrent Neural NetworkΛ૊Έ߹ΘͤͨΑ ͏ͳϞσϧʹͳ͍ͬͯΔ • ࣭໰Ԡ౴ɺ඼ࢺλά෇͚ɺڞࢀরղੳɺධ൑ ෼ੳͰstate of the art
15. ਂ૚ֶशͷNLPʹ͓͚Δݱঢ় • ਫ਼౓໘Ͱ͸ɺଞͷख๏ͱେࠩ͸͍͍ͭͯͳ͍ • ը૾ॲཧ΍Ի੠ೝࣝͱҧ͏ • ػց຋༁΍࣭໰Ԡ౴͕γϯϓϧͳख๏Ͱղ͚ ΔΑ͏ʹͳͬͨ • จͷੜ੒͕Ͱ͖ΔΑ͏ʹͳͬͨ
16. ࠓޙ͸Ͳ͏ͳΔͷ͔ʁ • ਖ਼௚ɺΑ͘Θ͔Βͳ͍…… • ը૾΍ಈըͱ૊Έ߹Θͤͨݚڀ͸૿͑ͦ͏
17. ࠷ઌ୺ʹ͍͍ͭͯͨ͘Ίʹ
18. 3ͭʹߜͬͯղઆ͠·͢ • Neural Networkͷجૅ • Recurrent Neural Network • ಛʹGated Recurrent Unit • Attention
19. χϡʔϥϧωοτϫʔΫ = ؔ਺ • χϡʔϥϧωοτϫʔΫ͸ɺ͋Δछͷؔ਺Ͱ ͋Δͱߟ͑Δ͜ͱ͕Ͱ͖Δ • ೖग़ྗ͸ϕΫτϧ • ඍ෼Մೳ
20. γϯϓϧͳྫ͔Β࢝ΊΔ y = f(x) = W x
21. ग़ྗΛ0ʙ1ʹਖ਼نԽ͢Δ • y = softmax(f(x))
22. ଟ૚Խͯ͠ΈΑ͏ • y = softmax(g(f(x)))
23. Ͳ͕͜ϨΠϠʔʁ
24. ౾஌ࣝ • ϨΠϠʔͱ͍͏ݴ༿ʹؾΛ͚ͭΑ͏ • ͲͬͪΛࢦͯ͠Δ͔ᐆດʢಡΉͱ͖ʹؾΛ ͚ͭΕ͹Θ͔Δ͕…ʣ • ϝδϟʔͳOSSͰ͸ɺؔ਺Λࢦ͢΋ͷ͕ଟ਺ ೿ʢCaffe, Torch, Chainer, TensorFlowʣ
25. Recurrent Neural Network • ࣌ܥྻʹฒͿཁૉΛ1ͭͣͭड͚औͬͯɺঢ়ଶ Λߋ৽͍ͯ͘͠ωοτϫʔΫͷ૯শ • ࠷ۙͱͯ΋ྲྀߦ͍ͯ͠Δ IUUQDPMBIHJUIVCJPQPTUT6OEFSTUBOEJOH-45.T
26. ͳͥRNN͕ྲྀߦ͍ͯ͠Δͷ͔ʁ • Մม௕ͷσʔλͷऔΓѻ͍͸೉͍͠ • RNNΛ࢖ͬͨseq2seqϞσϧʢEncoder/ DecoderϞσϧͱ΋ݺͿʣͰ͸Մม௕σʔλΛ ͏·͘औΓѻ͑Δࣄ͕Θ͔͖ͬͯͨ
27. Seq2seqϞσϧͱ͸ʁ • Մม௕ͷೖྗσʔλΛɺݻఆ௕ͷϕΫτϧʹ Τϯίʔυͯ͠ɺ͔ͦ͜Β຋༁ޙͷσʔλΛ σίʔυ͢Δ • ػց຋༁΍ࣗಈཁ໿ͳͲೖग़ྗͷ௕͕͞ҧ͏ λεΫͰۙ೥ݚڀ͕ਐΜͰ͍Δ
28. Seq2seqϞσϧͰͷ຋༁ 5IJT JT B QFO &04 ͜Ε ͸ ϖϯ Ͱ͢ &04 ͜Ε ͸ ϖϯ Ͱ͢
29. Seq2seqϞσϧͰͷ຋༁ 5IJT JT B QFO &04 ͜Ε ͸ ϖϯ Ͱ͢ &04 ͜Ε ͸ ϖϯ Ͱ͢ 5IJT
    JT
    B
    QFOΛݻఆ௕ʹ
    Τϯίʔυ͍ͯ͠Δʂ
30. Seq2seqϞσϧΛ຋༁ʹ࢖͏ͱʁ • ͔ͳΓ͏·͍͘͘ࣄ͕Θ͔͍ͬͯΔ • ͨͩ࣍͠ͷ༷ͳऑ఺͕͋Δ • ௕จʹऑ͍ • ݻ༗໊ࢺ͕ೖΕସΘΔ • ͜ΕΛղܾ͢Δͷ͕࣍ʹઆ໌͢ΔAttention
31. Attentionͱ͸ • σίʔυ࣌ʹΤϯίʔυ࣌ͷ৘ใΛগ͚ͩ͠ ࢀর͢ΔͨΊͷ࢓૊Έ • গ͚ͩ͠ = બ୒ͨ͠෦෼͚ͩΛݟΔ • Global AttentionͱLocal Attention͕͋Δ
32. Global Attention • ީิঢ়ଶͷॏΈ෇͖࿨ΛAttentionͱ͢Δ • ྺ࢙తʹ͸ͪ͜Βͷํ͕ͪΐͬͱݹ͍ 5IJT JT B QFO &04 ͜Ε ͸ ͜Ε ͸    
33. Local Attention • Τϯίʔυ࣌ͷঢ়ଶΛ͍͔ͭ͘બ୒ͯ͠࢖͏ 5IJT JT B QFO &04 ͜Ε ͸ ͜Ε ͸
34. Attentionͷॱং • ஫໨Λ౰ͯΔॱং͸ɺGlobal AttentionͰ΋ Local AttentionͰ΋೉͍͠໰୊Ͱ͋Δ • Attentionͷॱং΋RNNͰֶशͨ͠Γ͢Δ • લ͔ΒॱʹAttentionΛ౰͍͚ͯͯͩ͘Ͱ΋ੑ ೳ͸޲্͢Δ
35. ࣮ݧ݁ՌɿWMT'14
36. ࣮ݧ݁ՌɿWMT'15
37. ࣮ࡍͷ຋༁ͷྫ
38. ͜͜·Ͱͷ·ͱΊ • جૅతͳχϡʔϥϧωοτϫʔΫͷղઆ • Recurrent Neural Network • Attention
39. ࠓ೔࿩͞ͳ͔ͬͨ͜ͱ • ֶशʢback propagation, minibatchʣ • ଛࣦؔ਺ʢlog loss, cross entropy lossʣ • ਖ਼ଇԽͷςΫχοΫ • dropout, batch normalization • ࠷దԽͷςΫχοΫ • RMSProp, AdaGrad, Adam • ֤छ׆ੑԽؔ਺ • (Very) Leaky ReLU, Maxout
40. ࠓޙͷΦεεϝ • ࣗ෼Ͱͳʹ͔࣮ݧͯ͠ΈΑ͏ • γϯϓϧͳྫͰ͍͍͔Β·ͣಈ͔͢ • ಈ͍ͨΒ࣍ʹࣗ෼Ͱվ଄ͯ͠ΈΔ • ͱʹ͔͘खΛಈ͔͢͜ͱ͕େࣄ • ࠷ॳ͔Β೉͗͢͠Δ͜ͱʹ͸खΛग़͞ͳ͍
41. ࠷৽৘ใ΁ͷΞϯςφ (1) • TwitterͰػցֶशͳͲʹ͍ͭͯൃݴ͍ͯ͠Δ ਓΛϑΥϩʔ͢Δ • ͱΓ͋͑ͣ @hillbig • ͍͍ਓ͸ଞʹ΋ͨ͘͞Μ͍·͕͢ • ͋΍͍͠ਓ΋͍Δ͔Β஫ҙͯ͠Ͷ
42. ࠷৽৘ใ΁ͷΞϯςφ (2) • ࿦จΛಡ΋͏ • ಡΉ͚ͩ࣌ؒͷແବͳ࿦จ΋͋ΔͷͰ஫ҙ • ࠷ॳͷ͏ͪ͸ɺ༗໊ͳֶձʢACL, EMNLP, ICML, NIPS, KDD, etc.ʣʹ௨ͬͯΔ࿦จʹ ߜ͕ͬͨΑ͍
43. ࠷৽৘ใ΁ͷΞϯςφ (3) • ࿦จͷஶऀʹ஫໨͢Δ • ࿦จΛಡΜͰ͍Δ͏ͪʹɺࣗ෼͕໘ന͍ͱ ࢥ͏࿦จͷஶऀ͕Կਓ͔ग़ͯ͘Δ • ͦ͏͍͏ਓͷ৽͍͠࿦จ͸Ͳ͏ʹ͔ͯ͠ νΣοΫ͠Α͏
44. Take home messages • ؾ͕࣋ͪ੝Γ্͕ͬͯΔ͏ͪʹɺࣗ෼ͷखͰ ৭ʑ࣮ݧͯ͠ΈΑ͏ • ॳ৺ऀʹ͸Chainer͕Φεεϝ • ࠷৽৘ใ͸ωοτͰೖखͰ͖Δ • มͳํ޲ʹҙ͕ࣝߴ͍ਓʹ͸஫ҙ