Upgrade to Pro — share decks privately, control downloads, hide ads and more …

LSPC deep-people for music processing #06 RNN

LSPC deep-people for music processing #06 RNN

筑波大学人と音の情報学研究室で行われた,
深層学習×音楽データの勉強会の資料を公開しています.
誤りなどがあるかもしれません.その場合,ご指摘お願いします.

#06 再帰型ニューラルネットワーク

Ae87a6075a4bc957a6775c6d70c8be90?s=128

Yuya Yamamoto

May 05, 2022
Tweet

More Decks by Yuya Yamamoto

Other Decks in Research

Transcript

  1. Recurrent Neural Network (RNN) Deep-people #6

  2. લճͷ͓͞Β͍ • CNN • ہॴੑͱҐஔෆมੑ • ৞ΈࠐΈ૚ɼϓʔϦϯά૚ɼશ݁߹૚ • ༗໊Ϟσϧͷ঺հ 2

  3. ࠓ೔ͷࢀߟࢿྉ • Juhanͷࢿྉɹ • https://mac.kaist.ac.kr/~juhan/gct634/Slides/ [week9-1]%20recurrent%20neural%20network.pdf • Under standing LSTM

    Networks • http://colah.github.io/posts/2015-08-Understanding-LSTMs/ 3
  4. ࠓճͷ͓࿩ • ࠶ؼܕχϡʔϥϧωοτϫʔΫʢҎ߱ɼRNNʣ • ࣗવݴޠॲཧͰ੒ޭΛ਱͛ͨχϡʔϥϧωοτϫʔΫ • ಛʹܥྻσʔλͷॲཧʹڧ͍ͱ͞Ε͍ͯΔ 4

  5. ܥྻσʔλ • Իͷσʔλ͸͠͹͠͹ܥྻσʔλΛग़ྗ͢Δ͜ͱ͕ٻΊΒΕΔ • CNN΋࢖͑Δ͕ɼ͋͘·Ͱ1ϑϨʔϜ෼ͷॲཧΛಠཱʹߦ͏ʢԼਤʣ • →ଞͷϑϨʔϜͷӨڹʹίϯςΩετ৘ใΛ׆༻Ͱ͖ͳ͍͔ʁ 5 ex. ϐονਪఆ

    by CNN ಉ͡ϐονΛ৳͹͍ͯͯ͠΋ɼͦͷ࣌఺Ͱͷ ৘ใ͚ͩͰϐονͷਪఆ஋ΛܾΊͯ͠·͏ ʹଞͷ࣌఺ͷਪఆ݁ՌΛ׆༻Ͱ͖͍ͯͳ͍
  6. ଞͷϑϨʔϜ৘ใΛܨ͛Δ 6 • ண໨࣌఺ͷೖྗʹՃ͑ɼաڈ or ະདྷͷӅΕ૚ͷঢ়ଶΛ׆༻͢Δ • ӅΕ૚ͷঢ়ଶΛจ຺৘ใͱͯ͠ೖྗ͢Δ • →ΑΓ޿͍ൣғͷӨڹΛߟྀͰ͖ΔΑ͏ʹʂ

  7. Recurrent Neural Network ʢRNNʣ • લ࣌ࠁͷӅΕ૚ͷঢ়ଶͱɼݱࡏͷঢ়ଶΛೖྗ͢ΔχϡʔϥϧωοτϫʔΫ 7

  8. Vanilla RNN • ͍ΘΏΔ࠷΋γϯϓϧͳRNN • લ࣌ࠁͷӅΕঢ়ଶͷॏΈWt ͱݱ࣌ࠁͷೖྗͷॏΈWh ʢόΠΞεb΋ʣͷ2छྨ͕ଘࡏ • ׆ੑԽؔ਺g(ɾ)ʹ͸tanh͕Α͘༻͍ΒΕΔ

    • ग़ྗ૚ʹ͸ΞϑΟϯม׵+׆ੑԽؔ਺fΛ͔͚Δʢgͱ۠ผ͠ͳ͍͜ͱ΋ʣ 8
  9. ॱ఻ൖʢforwardܭࢉʣ • ֤࣌ࠁ͝ͱͷঢ়ଶΛԣʹల։͢Δ • ֤࣌ࠁͰॏΈʢWh (i),Wt (i)ʣΛڞ༗͢ΔɼͲͰ͔͍1ͭͷNNͱଊ͑Δ 9

  10. Back-propagation through time (BPTT)ͱΑ͹ΕΔ 10 ٯ఻ൖʢBackwardʣ ࣌ࠁ͕ޙͷํ͔Βɼ࣌ࠁ͕લͷํʹ఻ൖͤ͞Δ

  11. BPTTͷ໰୊఺ • ޯ഑ফࣦ/രൃͷ໰୊ • ܥྻ௕͕௕͘ͳΔ΄Ͳޯ഑ͷ஋͕ෆ҆ఆʹͳΔ • ޯ഑രൃʹ͸ޯ഑ΫϦοϐϯάʢᮢ஋Λ௒͑ͨΒͦͷᮢ஋ʹ͢Δʣ͕࢖ΘΕΔ • Ұํɼޯ഑ফࣦʹ͸ରԠͰ͖ͳ͍ͷͰɼߏ଄ࣗମͷվળΛߟ͑Δ 11

    ޯ഑ΫϦοϐϯά https://medium.com/@ayushch612/vanishing-gradient-and- exploding-gradient-problems-7737c0aa535f
  12. ήʔτ෇͖RNN • Gated Recurrent Unit (GRU)ͱ Long Short-Term Memory (LSTM)

    12 ܥྻ௕ͷ௕͍σʔλͰ΋ֶशՄೳ
  13. LSTMͷ͘͠Έ • 4ͭͷϞδϡʔϧʹ͍ͭͯղઆ 13

  14. ෆཁͳաڈͷ৘ใΛ”๨ΕΔ” 14 • ft ɿ͕͜͜[0,1]ͷൣғΛͱΓɼલ࣌ࠁͷهԱηϧͷӨڹྗΛ੍ޚ • 0ʹ͍ۙ΄Ͳલͷ৘ใΛغ٫͠ɼ1ʹ͍ۙ΄Ͳอ࣋͢Δ ᶃ๨٫ήʔτ

  15. ݱ࣌఺ͷ৽͍͠هԱΛ௥Ճ͢Δ 15 • Ci ɿݱ࣌ࠁͷ৘ใΛՃ͑Δ ~ • it ɿCi ͷ৘ใͷՁ஋Λ൑அ͢ΔͨΊͷॏΈ

    ~ ᶄݱ࣌ࠁͷೖྗΛར༻͢Δʮ৽͍͠هԱηϧʯ
  16. ࣍ͷ࣌ࠁʹ౉͢هԱηϧΛܭࢉ͢Δ 16 • ˒ɿ๨٫͞Ε࢒ͬͨલ࣌ࠁͷهԱ • ˒ɿ֮͑Δ΂͖ݱ࣌ࠁͷهԱ ᶅهԱηϧͷߋ৽ ˎ͸ཁૉੵ

  17. ϝΠϯ෦෼ 17 • Ot ɿલ࣌ࠁͷӅΕঢ়ଶht-1 ͱݱ࣌ࠁͷೖྗXt ͷΞϑΟϯ&׆ੑԽؔ਺ • ht ɿOt

    ͱ׆ੑԽؔ਺Λ௨ͨ͠هԱηϧCt ͷཁૉੵ → ࣍ͷ࣌ࠁ΁ ᶆग़ྗήʔτ
  18. શ෦Ͱ6ͭͷࣜ 18 ·ͱΊ ᶃ ᶄ ᶅ ᶆ ᶇ ᶈ ᶃ

    ᶄ ᶅ ᶆ ᶇ ᶈ ύϥϝʔλ͸4छྨ (36͸লུ͠·͕͢ େମ͓Μͳ͡Ͱ͢
  19. ໰୊ઃఆʹΑͬͯม͑Α͏ 19 RNNͷೖग़ྗ৭ʑ One-to-Many ܥྻσʔλΛ0͔Β ੜ੒͢Δ ʢೖྗx͸։࢝τʔΫϯʣ ʢԻָੜ੒౳ʣ Many-to-One ࣌ܥྻͷσʔλ͔Β

    άϩʔόϧͳ৘ใΛਪఆ ʢจষײ৘ਪఆ౳ʣ Many-to-ManyʢରԠ͋Γʣ ೖྗͱग़ྗͷ࣌ࠁతͳ ରԠΛਪఆ͢Δ ʢ୯ޠͷ඼ࢺਪఆ౳ʣ Many-to-ManyʢରԠͳ͠ʣ Seq2Seqͱ΋ɽ ௕͞ͷҟͳΔܥྻؒͷม׵ ʢػց຋༁౳ʣ
  20. աڈˠະདྷʹՃ͑ɼະདྷˠաڈͷ৘ใ఻೻΋ߟྀ 20 • ະདྷͷ৘ใ͕ࣝผʹ໾ཱͭ৔߹͸RNNΛ૒ํ޲ʹͯ͠΋͍͍ • ʢϦΞϧλΠϜॲཧͰ͸࢖Θͳ͍ํ͕͍͍͔΋͠Εͳ͍…?ʣ • ͚Ͳ࢖ͬͯΔ࿦จ͸βϥʹݟΔ ૒ํ޲ʢBi-DirectionalʣRNN

  21. CNNͱ૊Έ߹ΘͤͯԻλεΫͰ੒ޭ 21 • ಈըɾԻͳͲɼͦͷ࣌ࠁͷಛ௃நग़ΛCNNͰ΍Δ • ʴͦͷ࣌ؒมԽʹؔ͢Δಛ௃ΛRNNʹΑͬͯଊ͑Δ ৞ΈࠐΈ࠶ؼܕNNʢCRNNʣ https://ys0510.hatenablog.com/entry/crnn ΑΓ

  22. CRNN͸CNNͱRNNͷ͍͍ͱ͜औΓ • CNN͸प೾਺৘ใΛɼRNN͸࣌ ؒ৘ใΛΑ͘ଊ͑Δ • εϖΫτϩάϥϜΛ࢖ͬͨॲཧ ͱ૬ੑ˕ 22 $BL’S FUBM$POWPMVUJPOBMSFDVSSFOUOFVSBMOFUXPSLTGPSQPMZQIPOJDTPVOEFWFOUEFUFDUJPO5"4-1

  23. ࠶ؼߏ଄Λͱ͍ͬͯΔͨΊֶश͕஗Ίɿ଎͍୅ସϞσϧ͕͋Δ 23 • Temporal Convolutional NetworkʢTCNʣ • ࣌ࠁํ޲ʹ1࣍ݩ৞ΈࠐΈΛߦ͏ + ड༰໺Λdilated֦ͯ͠େ

    • ֶश͕଎͍͠ɼਫ਼౓΋LSTMΑΓΑ͘ͳΔ৔߹΋͋Δʢࢁຊͷײ૝ʣ ࣮͸RNN͸ۦஞ͞Εؾຯɽɽɽ #BJFUBM"O&NQJSJDBM&WBMVBUJPOPG(FOFSJD$POWPMVUJPOBMBOE3FDVSSFOU/FUXPSLTGPS4FRVFODF.PEFMJOH
  24. ࠶ؼߏ଄Λͱ͍ͬͯΔͨΊֶश͕஗Ίɿ଎͍୅ସϞσϧ͕͋Δ 24 • Transformer • ʮͳΜ͔࠷͖ۙͯΔ΍ͭʯ • ஫ҙػߏʢAttentionʣʹجͮ͘Ϟσϧ • CNN͸Χʔωϧͷൣғ಺ɼɹɹɹɹ

    RNN͸ܥྻͷҰͭҰͭΛண໨͢Δ͕ɼ Transformer͸ܥྻશͯΛҰؾʹண໨ ͯ͠஫ҙ͢ΔϙΠϯτΛܾΊΔ ࣮͸RNN͸ۦஞ͞Εؾຯɽɽɽ ৄ͘͠͸ະདྷͷTransformerճʹ 7BTXBOJFUBM"UUFOUJPOJTBMMZPVOFFE/FVS*14
  25. ·ͱΊ • ܥྻσʔλʹ͸RNN͕ޮՌత • ௕͍ܥྻσʔλʹ͓͚Δޯ഑ফࣦͷ໰୊Λղܾ͢Δήʔτ෇͖RNN • ೖग़ྗʹΑΔRNNͷηοςΟϯά • CNNΛ૊Έ߹ΘͤͨCRNN •

    ࠷ۙͷܥྻσʔλॲཧͰRNNΛۦஞͭͭ͋͠ΔϞσϧͷ঺հ 25