Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Deep Learning輪読会#4

Deep Learning輪読会#4

書籍「Deep Learning」の輪読会4/5回目の資料 (7章後半)です。

Avatar for Takayuki Maeda

Takayuki Maeda

January 21, 2018
Tweet

Other Decks in Technology

Transcript

  1. ઢܗճؼͰͷεύʔεදݱ εύʔεͳύϥϝʔλɿ εύʔεͳදݱɿ جຊతʹӅΕϢχοτΛ࣋ͭϞσϧΛεύʔεʹͰ͖Δ 2 6 6 6 6 4

    18 5 15 9 3 3 7 7 7 7 5 = 2 6 6 6 6 4 4 0 0 2 0 0 0 0 1 0 3 0 0 5 0 0 0 0 1 0 0 1 0 4 1 0 0 0 5 0 3 7 7 7 7 5 2 6 6 6 6 6 6 4 2 3 1 5 1 4 3 7 7 7 7 7 7 5 2 6 6 6 6 4 14 1 19 2 23 3 7 7 7 7 5 = 2 6 6 6 6 4 3 1 2 5 4 1 4 2 3 1 1 3 1 5 4 2 3 2 3 1 2 3 0 3 5 4 2 2 5 1 3 7 7 7 7 5 2 6 6 6 6 6 6 4 0 2 0 0 3 0 3 7 7 7 7 7 7 5 y 2 Rm x 2 Rn A 2 Rm⇥n B 2 Rm⇥n h 2 Rn y 2 Rm
  2. දݱͷϊϧϜϖφϧςΟਖ਼ଇԽ • දݱͷϊϧϜϖφϧςΟਖ਼ଇԽɿ • εύʔεදݱΛ΋ͨΒ͢ϖφϧςΟͷྫɿ • L1ϖφϧςΟ • ενϡʔσϯτͷtࣄલ෼෍͔Βಋ͔ΕͨϖφϧςΟ •

    KLμΠόʔδΣϯεϖφϧςΟ • ෳ਺ࣄྫʹΘͨΔ׆ੑԽฏۉʹΑΔਖ਼ଇԽɻ
 ϕΫτϧͷ֤੒෼͕0.01ͷΑ͏ͳ໨ඪ஋ʹۙͮ͘ ˜ J(✓; X, y) = J(✓; X, y) + ↵⌦(h) ↵ 2 [0, 1] ⌦(h) = khk1 = X i |hi |
  3. ׆ੑԽ஋ʹݫ੍͍͠໿Λ࣋ͭɹ εύʔεදݱ • ௚ަϚονϯά௥੻ (OMP-k: Orthogonal Matching Pursuit): • W͕௚ަ͢Δͱ͍͏੍໿ͷԼͰޮ཰తʹղ͚Δ

    • OMP-1͸ਂ͍ΞʔΩςΫνϟͰඇৗʹಛ௃తʹಛ ௃ྔΛநग़Ͱ͖Δ arg min h,khk0 k kx W hk2 khk0 ඇͷཁૉͷ਺
  4. E 2 4 1 k X i ✏i !2 3

    5 = 1 k2 E 2 4 X i 0 @✏2 i + X j6=i ✏i✏j 1 A 3 5 = 1 k v + k 1 k c = ( 1 k v (c = 0) v (c = v) ϞσϧฏۉԽ • ෳ਺ͷϞσϧΛผʑʹ܇࿅ͤͨ͞ޙɺͦΕΒͷϞσϧ͢΂ ͔ͯΒςετࣄྫʹର͢Δग़ྗΛ౤ථ • ௨ৗϞσϧ͕ҟͳΕ͹ɺಉ͡ςετࣄྫͰ͋ͬͯ΋͢΂͕ͯ ಉؒ͡ҧ͍Λ͠ͳ͍ͨΊɺ͏·͘ಇ͘ • ޡ͕ࠩฏۉ0ͷଟมྔਖ਼ن෼෍ʹै͏kݸͷճؼϞσϧͷྫ: ෼ࢄ E[✏2 i ] ڞ෼ࢄ E[✏i✏j] ֤Ϟσϧͷޡࠩ Ξϯαϯϒϧ༧ଌثͷ ظ଴ೋ৐ޡࠩ
  5. υϩοϓΞ΢τͷֶश • ϛχόον͝ͱʹೋ஋ϚεΫμΛ ແ࡞ҝʹαϯϓϦϯά • μ͕1ʹͳΔ֬཰ (ϋΠύʔύϥ ϝʔλ) • ӅΕ૚:

    0.5 • ೖྗ૚: 0.8 • ޙ͸௨ৗ௨Γֶश • ࢦ਺ؔ਺తʹେ͖ͳ਺ͷωοτϫʔ Ϋʹର͢ΔόΪϯάͷۙࣅΛ࣮ݱ
  6. όΪϯάͱυϩοϓΞ΢τͷҧ͍ όΪϯά υϩοϓΞ΢τ ֤Ϟσϧ ಠཱ ύϥϝʔλू߹Λڞ༗͠ɺ਌ͷχϡʔϥϧω οτϫʔΫ͔Β෦෼ू߹Λܧঝ ܇࿅ ֤Ϟσϧ͸ͦΕ ͧΕͷ܇࿅ू߹

    Ͱ܇࿅ ΄ͱΜͲͷϞσϧ͸໌ࣔతʹ܇࿅͞Εͳ͍ɻ Մೳͳ෦෼ωοτϫʔΫͷখ͞ͳ෦෼͕ε ςοϓͰ܇࿅͞Εɺύϥϝʔλڞ༗ʹΑΓ࢒ Γͷ෦෼ωοτϫʔΫ͕ྑ͍ઃఆͱͳΔ
  7. ॏΈεέʔϦϯάਪ࿦نଇ • ҎԼͷ৔߹ɺॏΈεέʔϦϯάਪ࿦نଇ͕ݫີ • ඇઢܗͷӅΕϢχοτΛ࣋ͨͳ͍ϞσϧΫϥεʢԼࣜʣ • ৚݅෇͖ਖ਼ن෼෍ग़ྗΛ࣋ͭճؼωοτϫʔΫ • ඇઢܗੑͷͳ͍ӅΕ૚Λ࣋ͭਂ͍ωοτϫʔΫ P(y

    = y|v) = softmax(W >v + b)y · · · ˜ Pensemble(y = y|v) / 2n v u u t Y d2{0,1}n exp ⇣ W > y,: (d v) + by ⌘ = exp 0 @ 1 2n X d2{0,1}n W > y,: (d v) + by 1 A = exp ✓ 1 2 W > y,: v + by ◆
  8. (ࢀߟ) The definition of “distributed representation” • Each neuron must

    represent something, so this must be a local representation. • “Distributed representation” means a many-to- many relationship between two types of representation (such as concepts and neurons). • Each concept is represented by many neurons • Each neuron participates in the representation of many concepts IUUQXXXDTUPSPOUPFEVdCPOOFSDPVSTFTTDTDMFDUVSFTMFDQEG
  9. υϩοϓΞ΢τʹ͓͚Δ஫ҙ఺ • ܭࢉίετ͕େ͖͘ͳΔՄೳੑ͕͋Δ • ਖ਼ଇԽʹΑΔϞσϧͷදݱྗ࡟ݮΛ૬ࡴ͢ΔͨΊɺϞσϧα Πζͷ֦େ΍ɺ܇࿅ͷ൓෮਺ͷ૿Ճ͕ඞཁ • σʔλू߹͕େ͖͍৔߹ɺਖ਼ଇԽʹΑΔ൚Խޡࠩͷݮগ͕ಘ ΒΕʹ͍͘ •

    ϥϕϧ͋Γ܇࿅ࣄྫ͕ۃ୺ʹগͳ͍৔߹ɺଞͷख๏ͷํ͕༏Ґ ͳ৔߹΋͋Δ • ϕΠδΞϯχϡʔϥϧωοτϫʔΫ • ௥ՃͰϥϕϧͳ͠σʔλ͕ར༻Մೳͳ৔߹ɺڭࢣͳ͠ͷಛ௃ ྔֶश
  10. ύϥϝʔλڞ༗ͱͯ͠ͷυϩοϓΞ΢τ • υϩοϓΞ΢τ͸ɺόΪϯάͨ͠ϞσϧͷΞϯαϯϒϧ͚ͩ Ͱͳ͘ɺӅΕϢχοτΛڞ༗͢ΔϞσϧͷΞϯαϯϒϧ΋܇ ࿅ • Ϟσϧͷ֤ӅΕϢχοτ͸ɺଞͷӅΕϢχοτʹؔΘΒ ͣྑ͍ੑೳΛൃش • ֤ӅΕϢχοτ͸ϞσϧؒͰަ׵΍ஔ׵͕Մೳ

    • ֤ӅΕϢχοτ͕୯ʹྑ͍ಛ௃ྔͱ͍͏͚ͩͰͳ͘ɺଟ ͘ͷ؍఺Ͱྑ͍ಛ௃ྔͱͳΔΑ͏ʹਖ਼ଇԽ • ൚Խޡࠩͷվળ͸ɺಠཱͨ͠ϞσϧͷΞϯαϯϒϧΑΓ΋υ ϩοϓΞ΢τͷํ͕େ͖͍
  11. ఢରతֶश • ܇࿅ू߹ʹఢରతͳՃ޻Λͨ͠ࣄྫֶशʹΑΓɺݩͷi.i.d.ςετू߹ʹ͓ ͚ΔޡΓ཰Λ࡟ݮͰ͖Δ • ఢରతࣄྫ͸ա౓ͷઢܗੑ͕ओͳݪҼͷ1ͭ • χϡʔϥϧωοτϫʔΫ͸ओʹઢܗੑʹؔ࿈ͨ͠ߏ੒ཁૉΛݩʹߏங • શମతͳؔ਺΋ɺ݁Ռతʹߴ͍ઢܗੑΛ࣋ͭ

    • ࠷దԽ͕༰қͳҰํɺೖྗͷ਺͕๲େͰ͋Ε͹ɺઢܗؔ਺ͷ஋͕ٸܹʹ มԽ͢ΔՄೳੑ͕͋Δ (֤ೖྗ͕ มԽͨ͠৔߹ɺ࠷େͰ มԽ) • ఢରతֶशʹΑΓ܇࿅σʔλۙ๣ͰͷણࡉͰہॴతʹઢܗͳڍಈΛ๦֐Մ • ہॴෆมੑͷࣄલ஌ࣝΛڭࢣ͋Γχϡʔϥϧωοτʹ໌ࣔతʹಋೖ ✏ ✏kwk1
  12. Ծ૝ఢରతࣄྫʹΑΔ൒ڭࢣ͋Γֶश • σʔλͳ͠ͷσʔλ͕ଘࡏ͢Δଟ༷ମʹԊͬͯɺͲ ͜ʹ͓͍ͯ΋খ͞ͳมԽʹରͯ͠ؤ݈ͳؔ਺Λֶश • σʔλू߹ͷதͰϥϕϧ͕෇༩͞Ε͍ͯͳ͍఺ ʹ ͓͍ͯɺϞσϧͰϥϕϧ ΛׂΓ౰ͯΔ •

    ෼ྨثʹ Ͱ͋Δϥϕϧ Λग़ྗͤ͞Δఢର తࣄྫ Λ୳͢(Ծ૝తఢରࣄྫ) • ෼ྨث͕ ͱ ʹಉ͡ϥϕϧΛׂΓ౰ͯΔΑ͏ʹ ֶशͤ͞Δ x ˆ y y0 6= ˆ y y0 x x0 x0
  13. ઀ڑ཭ΞϧΰϦζϜ • ଟ༷ମԾઆΛ׆༻ͨ͠ॳظͷࢼߦͷ1ͭ • ϊϯύϥϝτϦοΫͳ࠷ۙ๣ΞϧΰϦζϜ • ϢʔΫϦουڑ཭Ͱ͸ͳ͘ɺͦͷۙ๣Ͱ֬཰͕ूத͍ͯ͠Δͱ͍͏ଟ ༷ମͷ஌͔ࣝΒಘΒΕͨࢦඪΛར༻ • ఺

    ͱ ؒͷ࠷ۙ๣ڑ཭ͱͯ͠ɺͦΕͧΕ͕ଐ͢Δଟ༷ମ ͱ ؒͷڑ཭Λ࢖͏ͷ͕ଥ౰ • ෼ྨث͸ଟ༷ମ্ͷಈ͖ʹରԠ͢ΔہॴతཁҼʹରͯ͠ෆมͷͨΊ • ܭࢉ্ࠔ೉͔΋͠Εͳ͍ͨΊɺ Λ Ͱͷ઀ฏ໘Ͱۙࣅ͠ɺ2ͭͷ ฏ໘ؒͷڑ཭ΛଌΔ͔ɺ1ͭͷ઀ฏ໘ͱ1ͭͷ఺ͷؒͷڑ཭ΛଌΔํ ๏͕͋Δ x1 x2 xi M1 M2 Mi
  14. ઀ઢ఻೻ΞϧΰϦζϜ • χϡʔϥϧωοτϫʔΫͷ֤ग़ྗ Λط஌ͷมಈཁҼʢը ૾ͷҠಈɺճసɺ֦େॖখͳͲʣʹରͯ͠ہॴతʹෆมͳ෼ ྨثΛ܇࿅ • ͕ ʹ͓͚Δط஌ͷଟ༷ମͷ઀ϕΫτϧ ʹର

    ͯ͠௚ަ͍ͯ͠Δ • ҎԼͷΑ͏ͳਖ਼ଇԽϖφϧςΟΛ௥Ճ͢Δ͜ͱͰɺ ํ ޲ʹରͯ͠ Ͱͷ ͷํ޲ඍ෼͕খ͘͞ͳΔ • ڭࢣ͋Γֶश͚ͩͰͳ͘ɺڧԽֶशͷ؍఺Ͱ΋࢖ΘΕ͖ͯͨ ⌦(f) = X i ⇣ (rxf(x))>v(i) ⌘2 rxf(x) x v(i) f v(i) x f(x)
  15. ઀ઢ఻೻๏ͱσʔλू߹֦ுͱͷؔ܎ • ڞ௨఺ • ωοτϫʔΫͷग़ྗΛม͑ͳ͍ม׵ͷू߹Λࢦఆ͢Δ͜ ͱͰɺλεΫʹؔ͢Δࣄલ஌ࣝΛූ߸Խ • ҧ͍ • σʔλू߹֦ு:

    গͳ͘ͳ͍ճ਺ͷม׵Λࢪͯ͠ੜ੒ͨ͠ ݸผͷೖྗΛਖ਼͘͠෼ྨ͢ΔΑ͏ʹ໌ࣔతʹ܇࿅ • ઀ઢ఻೻๏: ໌ࣔతʹ৽͍͠ೖྗ఺Λௐ΂Δඞཁ͸ͳ͘ɺ ղੳతʹਖ਼ଇԽ͢Δ͜ͱͰɺಛఆͷม׵ʹରԠ͢Δํ޲ ͷઁಈʹ఍߅͢ΔΑ͏ʹ͢Δ
  16. ೋॏٯ఻೻๏ͱఢରతֶश ઀ઢ఻೻๏ ೋॏٯ఻೻๏ ʢϠίϏߦྻ͕খ͘͞ͳΔΑ͏ ʹਖ਼ଇԽʣ σʔλू߹֦ு ఢରతֶश ʢޓ͍ʹ͍ۙೖྗͷग़ྗ͕ಉ͡ ʹͳΔΑ͏ʹ܇࿅ʣ ಛఆํ޲ͷೖྗͷ

    มԽʹରͯ͠ Ϟσϧ͕ෆม ͢΂ͯͷํ޲ͷೖྗͷ มԽʹରͯ͠ Ϟσϧ͕ෆม ඍখͳઁಈʹ ͷΈ఍߅Մೳ ඍখͰͳ͍ ઁಈʹ఍߅Մೳ
  17. ଟ༷ମ઀෼ྨث • ଟ༷ମ෼ྨثͰ͸ɺࣗݾූ߸Խثʢ14ষʣΛ࢖ͬͯ઀ϕΫτϧΛਪ ఆ͢ΔͨΊɺϢʔβ͕઀ϕΫτϧΛࢦఆ͢Δඞཁ͕ͳ͍ 1.ࣗݾූ߸ԽثΛ࢖ͬͯڭࢣͳֶ͠शͰଟ༷ମͷߏ଄ (ଟ༷ମͷ઀ϕ Ϋτϧ) Λֶश 2.͜ΕΒͷ઀ϕΫτϧΛ࢖ͬͯχϡʔϥϧωοτϫʔΫ෼ྨثΛਖ਼ ଇԽ

    (઀ઢ఻೻๏) • ࣗݾූ߸ԽثͰਪఆ͞Εͨ઀ϕΫτϧ͸ର৅ʹݻ༗ͳཁૉΛؚΉ • ը૾ͷҠಈɺճసɺ֦େॖখͱ͍ͬͨزԿֶత഑ஔ͔Βੜ͡Δෆ มੑΛ௒͑Δ • ྫ͑͹ɺ಄΍٭ͳͲର৅෺ͷ෦ҐͷҠಈ΍มԽʹ૬౰͢Δ