Upgrade to Pro — share decks privately, control downloads, hide ads and more …

自然言語処理と深層学習の最先端

Cabd375086334aa7eb4abb4e317c3d14?s=47 tkng
January 15, 2016

 自然言語処理と深層学習の最先端

第4回 JustTechTalk の発表資料

Cabd375086334aa7eb4abb4e317c3d14?s=128

tkng

January 15, 2016
Tweet

More Decks by tkng

Other Decks in Technology

Transcript

  1. ࣗવݴޠॲཧͱਂ૚ֶशͷ࠷ઌ୺ ಙӬ୓೭ +VTU5FDI5BML

  2. ࣗવݴޠॲཧͱਂ૚ֶशͷ࠷ઌ୺ ͷҰ෦෼Λ঺հ͠·͢ ಙӬ୓೭ (@tkng) +VTU5FDI5BML

  3. ࣗݾ঺հɿಙӬ୓೭ • Twitter ID: @tkng • εϚʔτχϡʔεגࣜձࣾͰࣗવݴޠॲཧ΍ ը૾ॲཧΛ΍ͬͯ·͢

  4. None
  5. ࣗવݴޠॲཧͱ͸ • ࣗવݴޠʢ≠ϓϩάϥϛϯάݴޠʣΛѻ͏෼໺ • ػց຋༁ • ࣭໰Ԡ౴ • จॻ෼ྨ •

    ߏจղੳɾ܎Γड͚ղੳ • ܗଶૉղੳɾ୯ޠ෼ׂ
  6. ػց຋༁ͷྫ • Google຋༁ͷword lensػೳ IUUQHPPHMFUSBOTMBUFCMPHTQPUKQIBMMPIPMBPMBUPOFXNPSFQPXFSGVM@IUNM

  7. ࣭໰Ԡ౴ͷྫ • IBM Watson • Jeopardy!Ͱਓؒʹউར IUUQXXXOZUJNFTDPNTDJFODFKFPQBSEZXBUTPOIUNM

  8. ਂ૚ֶशͱ͸ • ≒ χϡʔϥϧωοτ • ۙ೥ͷྲྀߦ͸ɺҎԼͷཧ༝ʹΑΔ • ܭࢉػͷੑೳ޲্ • ֶशσʔλͷ૿Ճ

    • ࠷దԽख๏ͳͲͷݚڀͷਐల
  9. ࣗવݴޠॲཧͱ ਂ૚ֶशͷ࠷ઌ୺

  10. Show, Attend and Tell: Neural Image Caption Generation with Visual

    Attention (Xu+, 2015) • ը૾ʹର͢Δղઆจͷੜ੒ IUUQLFMWJOYVHJUIVCJPQSPKFDUTDBQHFOIUNM
  11. Show, Attend and Tell͸ Ͳ͏͍͏ख๏͔ • ҎԼͷ3ͭͷ૊Έ߹Θͤ • Convolutional Neural

    Network • Long Short Term Memory • Attention
  12. Generating Images from Captions with Attention (Mansimov+, 2015) • Ωϟϓγϣϯ͔Βը૾Λੜ੒͢Δ

    • ࡉ໨ͰݟΕ͹ඈߦػʹݟ͑ͳ͘΋ͳ͍
  13. Effective Approaches to Attention- based Neural Machine Translation (Bahdanau+, 2015)

    • Deep LearningΛ༻͍ͯػց຋༁ • Local Attentionͱ͍͏৽͍͠ख๏ΛఏҊ • ͍͔ͭ͘ͷݴޠϖΞͰɺstate of the artΛୡ੒ ࠷ߴਫ४
  14. Ask Me Anything: Dynamic Memory Networks for Natural Language Processing

    (Kumar+, 2015) • ৽͍͠ϞσϧʢDynamic Memory Networksʣ ΛఏҊͨ͠ • Recurrent Neural NetworkΛ૊Έ߹ΘͤͨΑ ͏ͳϞσϧʹͳ͍ͬͯΔ • ࣭໰Ԡ౴ɺ඼ࢺλά෇͚ɺڞࢀরղੳɺධ൑ ෼ੳͰstate of the art
  15. ਂ૚ֶशͷNLPʹ͓͚Δݱঢ় • ਫ਼౓໘Ͱ͸ɺଞͷख๏ͱେࠩ͸͍͍ͭͯͳ͍ • ը૾ॲཧ΍Ի੠ೝࣝͱҧ͏ • ػց຋༁΍࣭໰Ԡ౴͕γϯϓϧͳख๏Ͱղ͚ ΔΑ͏ʹͳͬͨ • จͷੜ੒͕Ͱ͖ΔΑ͏ʹͳͬͨ

  16. ࠓޙ͸Ͳ͏ͳΔͷ͔ʁ • ਖ਼௚ɺΑ͘Θ͔Βͳ͍…… • ը૾΍ಈըͱ૊Έ߹Θͤͨݚڀ͸૿͑ͦ͏

  17. ࠷ઌ୺ʹ͍͍ͭͯͨ͘Ίʹ

  18. 3ͭʹߜͬͯղઆ͠·͢ • Neural Networkͷجૅ • Recurrent Neural Network • ಛʹGated

    Recurrent Unit • Attention
  19. χϡʔϥϧωοτϫʔΫ = ؔ਺ • χϡʔϥϧωοτϫʔΫ͸ɺ͋Δछͷؔ਺Ͱ ͋Δͱߟ͑Δ͜ͱ͕Ͱ͖Δ • ೖग़ྗ͸ϕΫτϧ • ඍ෼Մೳ

  20. γϯϓϧͳྫ͔Β࢝ΊΔ y = f(x) = W x

  21. ग़ྗΛ0ʙ1ʹਖ਼نԽ͢Δ • y = softmax(f(x))

  22. ଟ૚Խͯ͠ΈΑ͏ • y = softmax(g(f(x)))

  23. Ͳ͕͜ϨΠϠʔʁ

  24. ౾஌ࣝ • ϨΠϠʔͱ͍͏ݴ༿ʹؾΛ͚ͭΑ͏ • ͲͬͪΛࢦͯ͠Δ͔ᐆດʢಡΉͱ͖ʹؾΛ ͚ͭΕ͹Θ͔Δ͕…ʣ • ϝδϟʔͳOSSͰ͸ɺؔ਺Λࢦ͢΋ͷ͕ଟ਺ ೿ʢCaffe, Torch,

    Chainer, TensorFlowʣ
  25. Recurrent Neural Network • ࣌ܥྻʹฒͿཁૉΛ1ͭͣͭड͚औͬͯɺঢ়ଶ Λߋ৽͍ͯ͘͠ωοτϫʔΫͷ૯শ • ࠷ۙͱͯ΋ྲྀߦ͍ͯ͠Δ IUUQDPMBIHJUIVCJPQPTUT6OEFSTUBOEJOH-45.T

  26. ͳͥRNN͕ྲྀߦ͍ͯ͠Δͷ͔ʁ • Մม௕ͷσʔλͷऔΓѻ͍͸೉͍͠ • RNNΛ࢖ͬͨseq2seqϞσϧʢEncoder/ DecoderϞσϧͱ΋ݺͿʣͰ͸Մม௕σʔλΛ ͏·͘औΓѻ͑Δࣄ͕Θ͔͖ͬͯͨ

  27. Seq2seqϞσϧͱ͸ʁ • Մม௕ͷೖྗσʔλΛɺݻఆ௕ͷϕΫτϧʹ Τϯίʔυͯ͠ɺ͔ͦ͜Β຋༁ޙͷσʔλΛ σίʔυ͢Δ • ػց຋༁΍ࣗಈཁ໿ͳͲೖग़ྗͷ௕͕͞ҧ͏ λεΫͰۙ೥ݚڀ͕ਐΜͰ͍Δ

  28. Seq2seqϞσϧͰͷ຋༁ 5IJT JT B QFO &04 ͜Ε ͸ ϖϯ Ͱ͢

    &04 ͜Ε ͸ ϖϯ Ͱ͢
  29. Seq2seqϞσϧͰͷ຋༁ 5IJT JT B QFO &04 ͜Ε ͸ ϖϯ Ͱ͢

    &04 ͜Ε ͸ ϖϯ Ͱ͢ 5IJTJTBQFOΛݻఆ௕ʹ Τϯίʔυ͍ͯ͠Δʂ
  30. Seq2seqϞσϧΛ຋༁ʹ࢖͏ͱʁ • ͔ͳΓ͏·͍͘͘ࣄ͕Θ͔͍ͬͯΔ • ͨͩ࣍͠ͷ༷ͳऑ఺͕͋Δ • ௕จʹऑ͍ • ݻ༗໊ࢺ͕ೖΕସΘΔ •

    ͜ΕΛղܾ͢Δͷ͕࣍ʹઆ໌͢ΔAttention
  31. Attentionͱ͸ • σίʔυ࣌ʹΤϯίʔυ࣌ͷ৘ใΛগ͚ͩ͠ ࢀর͢ΔͨΊͷ࢓૊Έ • গ͚ͩ͠ = બ୒ͨ͠෦෼͚ͩΛݟΔ • Global

    AttentionͱLocal Attention͕͋Δ
  32. Global Attention • ީิঢ়ଶͷॏΈ෇͖࿨ΛAttentionͱ͢Δ • ྺ࢙తʹ͸ͪ͜Βͷํ͕ͪΐͬͱݹ͍ 5IJT JT B QFO

    &04 ͜Ε ͸ ͜Ε ͸    
  33. Local Attention • Τϯίʔυ࣌ͷঢ়ଶΛ͍͔ͭ͘બ୒ͯ͠࢖͏ 5IJT JT B QFO &04 ͜Ε

    ͸ ͜Ε ͸
  34. Attentionͷॱং • ஫໨Λ౰ͯΔॱং͸ɺGlobal AttentionͰ΋ Local AttentionͰ΋೉͍͠໰୊Ͱ͋Δ • Attentionͷॱং΋RNNͰֶशͨ͠Γ͢Δ • લ͔ΒॱʹAttentionΛ౰͍͚ͯͯͩ͘Ͱ΋ੑ

    ೳ͸޲্͢Δ
  35. ࣮ݧ݁ՌɿWMT'14

  36. ࣮ݧ݁ՌɿWMT'15

  37. ࣮ࡍͷ຋༁ͷྫ

  38. ͜͜·Ͱͷ·ͱΊ • جૅతͳχϡʔϥϧωοτϫʔΫͷղઆ • Recurrent Neural Network • Attention

  39. ࠓ೔࿩͞ͳ͔ͬͨ͜ͱ • ֶशʢback propagation, minibatchʣ • ଛࣦؔ਺ʢlog loss, cross entropy

    lossʣ • ਖ਼ଇԽͷςΫχοΫ • dropout, batch normalization • ࠷దԽͷςΫχοΫ • RMSProp, AdaGrad, Adam • ֤छ׆ੑԽؔ਺ • (Very) Leaky ReLU, Maxout
  40. ࠓޙͷΦεεϝ • ࣗ෼Ͱͳʹ͔࣮ݧͯ͠ΈΑ͏ • γϯϓϧͳྫͰ͍͍͔Β·ͣಈ͔͢ • ಈ͍ͨΒ࣍ʹࣗ෼Ͱվ଄ͯ͠ΈΔ • ͱʹ͔͘खΛಈ͔͢͜ͱ͕େࣄ •

    ࠷ॳ͔Β೉͗͢͠Δ͜ͱʹ͸खΛग़͞ͳ͍
  41. ࠷৽৘ใ΁ͷΞϯςφ (1) • TwitterͰػցֶशͳͲʹ͍ͭͯൃݴ͍ͯ͠Δ ਓΛϑΥϩʔ͢Δ • ͱΓ͋͑ͣ @hillbig • ͍͍ਓ͸ଞʹ΋ͨ͘͞Μ͍·͕͢

    • ͋΍͍͠ਓ΋͍Δ͔Β஫ҙͯ͠Ͷ
  42. ࠷৽৘ใ΁ͷΞϯςφ (2) • ࿦จΛಡ΋͏ • ಡΉ͚ͩ࣌ؒͷແବͳ࿦จ΋͋ΔͷͰ஫ҙ • ࠷ॳͷ͏ͪ͸ɺ༗໊ͳֶձʢACL, EMNLP, ICML,

    NIPS, KDD, etc.ʣʹ௨ͬͯΔ࿦จʹ ߜ͕ͬͨΑ͍
  43. ࠷৽৘ใ΁ͷΞϯςφ (3) • ࿦จͷஶऀʹ஫໨͢Δ • ࿦จΛಡΜͰ͍Δ͏ͪʹɺࣗ෼͕໘ന͍ͱ ࢥ͏࿦จͷஶऀ͕Կਓ͔ग़ͯ͘Δ • ͦ͏͍͏ਓͷ৽͍͠࿦จ͸Ͳ͏ʹ͔ͯ͠ νΣοΫ͠Α͏

  44. Take home messages • ؾ͕࣋ͪ੝Γ্͕ͬͯΔ͏ͪʹɺࣗ෼ͷखͰ ৭ʑ࣮ݧͯ͠ΈΑ͏ • ॳ৺ऀʹ͸Chainer͕Φεεϝ • ࠷৽৘ใ͸ωοτͰೖखͰ͖Δ

    • มͳํ޲ʹҙ͕ࣝߴ͍ਓʹ͸஫ҙ