Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
Deep Learning Chapter 6
mocobt
August 06, 2019
Technology
1
170
Deep Learning Chapter 6
Japanese Explanation of Chapter 6 of Deep Learning, including the outline of book reading
mocobt
August 06, 2019
Tweet
Share
More Decks by mocobt
See All by mocobt
j20232
0
120
j20232
0
120
j20232
0
890
j20232
0
1.2k
j20232
0
110
j20232
0
120
j20232
0
460
j20232
1
160
j20232
0
470
Other Decks in Technology
See All in Technology
finengine
0
320
fortkle
0
450
yuzoiwasaki
0
170
nnstt1
2
120
akabekobeko
0
130
sat
1
260
picardparis
4
2.4k
katsumataryo
3
820
kenichimunezawa
0
820
hmatsu47
1
160
oracle4engineer
2
480
kawaguti
0
380
Featured
See All Featured
jakevdp
776
200k
sachag
445
36k
tenderlove
54
3.5k
phodgson
88
4k
jensimmons
208
10k
addyosmani
1346
200k
tanoku
258
24k
hatefulcrawdad
257
17k
erikaheidi
14
4.5k
malarkey
392
61k
hursman
107
9.3k
smashingmag
232
18k
Transcript
ਂֶश ୈⅡ෦ྠಡձ #01 ୈ6ষ ਂॱܕωοτϫʔΫ @mocobt
Agenda ։࠵֓ཁ (7min) ࣗݾհ (8min) ୈ6ষղઆ (30min) ୈ7ষղઆ (60min) ୲ܾΊ
(10min) ఫऩ (5min)
ྠಡձ ։࠵֓ཁ ಈػ • ਂֶशΛجૅ͔ΒԠ༻·Ͱ͑ͨ༗໊ຊͳͷͰͬͺΓಡΈ͍ͨ • ෳਓͰճ͠ಡΈͯ͠ύύͬͱऴΘΒͤͪΌ͓͏ʂʂ త: ຊͷྲྀΕΛ௫ΈɼҰਓͰಡΜͩͱ͖ʹԿ͕ॻ͍ͯ͋Δ͔ཧղͰ͖ΔΑ͏ʹͳΔ ର
• ػցֶशʹڵຯ͕͋ΔਓɼਂֶशͷཧΛֶͼ͍ͨਓ • ॳֶऀͱަྲྀ͢Δ͜ͱͰ৽ͨͳؾ͖ͮΛಘ͍ͨਓ (ܴ) ࢀՃඅ༻ʹ͍ͭͯ • ॴ͕େମ10,000ԁͳͷͰɼࢀՃਓ͕10ਓͷͱ͖ۉʹͳΔΑ͏ʹ͍ͯ͠·͢ (ඇӦར)
ྠಡձ ਐΊํ • 1ճʹ͖ͭ1ষ͔Β2ষΛਐΊ͍ͯ͘ • 1ষ͕ͦͦ͜͜ॏ͍ͷͰɼඞཁʹԠͯ͡ෳਓͰ୲͢Δ • ୲ऀ͕ॏཁͱࢥͬͨՕॴΛϐοΫΞοϓ͢Δ - ֤ϖʔδͰԿ͕ॻ͍͔ͯ͋ͬͨΛ؆୯ʹ·ͱΊΔ
- ແཧʹશͯΘͳͯ͘OK - Θ͔Βͳ͍͜ͱʮΘ͔Γ·ͤΜͰͨ͠ʯͰOK - ͪΖΜᘳʹ͍͚ͬͯͨͩΔͱେมॿ͔Γ·͢ʂ • Α΄Ͳͷࣄ͕ͳ͍ݶΓɼ୲ऀͷυλΩϟϯ͓߇͍͑ͩ͘͞
• ୈ6ষ: ਂॱܕωοτϫʔΫ • ୈ7ষ: ਂֶशͷͨΊͷਖ਼ଇԽ • ୈ8ষ: ਂϞσϧͷ܇࿅ͷͨΊͷ࠷దԽ •
ୈ9ষ: ΈࠐΈωοτϫʔΫ • ୈ10ষ: ܥྻϞσϦϯά: ճؼ݁߹ܕχϡʔϥϧωοτϫʔΫͱ࠶ؼܕωοτϫʔΫ • ୈ11ষ: ࣮༻తͳํ๏ • ୈ12ষ: ΞϓϦέʔγϣϯ ୈⅡষͷུ֓
ࣗݾհλΠϜ
• OpticsدΓͷCV & CGܥR&D - ۙʑത࢜՝ఔߦ͖͍ͨ# • Kaggle Expert •
NN࠶࣮ͱएׯͷվྑ͕Ͱ͖Δఔ About me @mocobt
ҎԼɼ୲ୈ6ষ
ୈ6ষ ਂॱܕωοτϫʔΫ • ͋Β͢͡ • 6.1 ྫ: XORͷֶश • 6.2
ޯʹجֶͮ͘श • 6.3 ӅΕϢχοτ • 6.4 ΞʔΩςΫνϟͷઃܭ • 6.5 ޡࠩٯ๏͓ΑͼͦͷଞͷඍΞϧΰϦζϜ • 6.6 ྺ࢙ϊʔτ !9
ୈ6ষ ਂॱܕωοτϫʔΫ • ͋Β͢͡ • 6.1 ྫ: XORͷֶश • 6.2
ޯʹجֶͮ͘श • 6.3 ӅΕϢχοτ • 6.4 ΞʔΩςΫνϟͷઃܭ • 6.5 ޡࠩٯ๏͓ΑͼͦͷଞͷඍΞϧΰϦζϜ • 6.6 ྺ࢙ϊʔτ !10 লུ ͻͨ͢Βఆٛ ͪΐͬͱ಄͏ (ͱ͍͏͔͍͖Εͳ͍)
ਂॱܕωοτϫʔΫ (feedforward NNs) • ͋Δؔͷۙࣅ͢ΔϞσϧ - ex. ೖྗը૾Λ͋ΔΧςΰϦࣸ૾͢Δؔͷۙࣅ - ଟύʔηϓτϩϯ
(multilayer perceptrons, MLP)ͱݺΕΔ • ॱ: ೖྗ͔Βग़ྗͱதؒతͳܭࢉΛධՁ͢Δ͜ͱ • ؔͷΈ߹ΘͤΛ༗ඇ८ճάϥϑͰදݱͰ͖ΔͨΊɼωοτϫʔΫͱ͍͏ ͋Β͢͡ !11
(Layer)ͱϢχοτ(Units) ͋Β͢͡ Layer Unit (≒ԋࢉࢠ) Output layer ؔࣗମΛLayerͱࢦ͢ྲྀ͋Δ༷ (͜͜Ͱؔͷग़ྗ) !12
Hidden layer
XORΛNNͰදݱ 6.1 ྫ: XORͷֶश !13 x0 x1 y=XOR(X0, X1) 0
0 0 0 1 1 1 0 1 1 1 0 NNͰඇઢܗͳXORΛදݱͰ͖͍ͯΔ͜ͱ͕Θ͔Δ
NNͷඇઢܗੑ 6.1 ྫ: XORͷֶश !14 ΞϑΟϯมͰඇઢܗੑΛදݱͰ͖ͳ͍ͷͰɼ׆ੑԽؔΛಋೖ
ޯʹجֶͮ͘श • NNඇઢܗੑΏ͑ʹଛࣦ͕ؔඇತʹͳΓɼ࠷దղΛղੳతʹಋग़Ͱ͖ͳ͍ - ଛࣦؔ: ϥϕϧ(ਅ)ͱग़ྗͷޡࠩΛදؔ͢ • ޯΛ༻͍ͯܭࢉతʹಋग़͢Δͷ͕Ұൠత (ͨͩ͠ऩଋੑͷอূͳ͠) 6.2
ޯʹجֶͮ͘श !15 ্ʹತ Լʹತ ඇತ
6.2.1 ίετؔ • NNͰ͏ίετؔଞͷύϥϝτϦοΫϞσϧͷͷͱେମಉ͡ ࠷ਪఆʹΑΔ͖݅ͷֶश • ΄ͱΜͲͷNN࠷๏ϕʔε • e.g.) ܇࿅σʔλͱϞσϧͷؒͷަࠩΤϯτϩϐʔΛίετؔͱͯ͠ఆٛ
• ɹɹ ͳΒɼฏۉೋޡࠩͷίετؔҎԼͷΑ͏ʹͳΔ 6.2 ޯʹجֶͮ͘श !16
6.2.2 ઢܗϢχοτ • ΞϑΟϯมʹجͮ͘ઢܗͳϢχοτ • ͖݅ΨεͷฏۉΛग़ྗ͢Δ͜ͱ͕ଟ͍ 6.2 ޯʹجֶͮ͘श !17 ೖྗಛྔ
6.2.2 γάϞΠυϢχοτ • 2Ϋϥεྨʹ͓͍ͯ1Ͱ͋Δ֬Λग़ྗ͢ΔϢχοτ • γάϞΠυมʹΑͬͯϕϧψʔΠ(ίΠϯ͛ͷ)͕ಘΒΕΔ • ରۭؒͰͷ֬Λ༧ଌ͢ΔͨΊɼ࠷ֶशͱΈ߹ΘͤΔ͜ͱ͕ଟ͍ - ίετؔதͷରͱsigmoidͷࢦ͕ޓ͍ʹଧͪফ͠߹͏ར͕͋Δ
(ޯফࣦ͠ʹ͍͘) - ฏۉೋޡࠩͷΑ͏ͳଞͷଛࣦؔͱΈ߹ΘͤΔͱɼֶश͕ਐΈʹ͘͘ͳͬͯ͠·͏ 6.2 ޯʹجֶͮ͘श !18 ϩδεςΟοΫγάϞΠυؔ → 3.10અ
6.2.2 ιϑτϚοΫεϢχοτ • nݸͷΫϥεʹର͢Δ֬(ΧςΰϦΧϧ)Λදݱ͢ΔϢχοτ • γάϞΠυϢχοτΛ2Ϋϥε͔ΒnΫϥεҰൠԽ͍ͯ͠Δ • ͋ΔΫϥεͷग़ྗ͕େ͖͘ͳΔͱଞΫϥεͷग़ྗখ͘͞ͳΔ (૯1) •
γάϞΠυϢχοτͱಉ༷ʹίετؔʹରΛ༻͍Δͱྑ͍ 6.2 ޯʹجֶͮ͘श !19
ӅΕϢχοτ ࠔͬͨΒ׆ੑԽؔReLUͬͱ͚OKʂ 6.3 ӅΕϢχοτ !20
ReLUͱͦͷҰൠԽ • ReLUͷಛ • ઢܗؔͱΑ͘ࣅ͍ͯΔͨΊ࠷దԽ͍͢͠ • ޯফࣦ͕ੜ͡ʹ͍͘ • ReLuͷҰൠԽ •
Absolute value rectification: → ମೝࣝʹར༻ • Leaky ReLU: • Parametric ReLU (PReLU): ΛֶशՄೳͳύϥϝλͱͯ͠ѻ͏ 6.3 ӅΕϢχοτ !21
ϚοΫεΞτϢχοτ • ReLUΛ͞ΒʹҰൠԽͨ͠Ϣχοτ • ೖྗzΛkݸͷͷάϧʔϓʹׂ͠ɼ֤ϢχοτͰάϧʔϓຖͷ࠷େΛग़ྗ • k=1ͷͱ͖ReLUͱಉ • ҙͷತؔΛۙࣅՄೳͰɼաڈͷֶश༰Λ٫͢Δ͜ͱʹରͯ͠ੑ͋Γ •
ͨͩͦͷաֶशͦ͢͠͏…ʁ 6.3 ӅΕϢχοτ !22 άϧʔϓi (֤άϧʔϓʹkݸͷཁૉ͕ଘࡏ)
ϩδεςΟοΫγάϞΠυͱϋΠύϘϦοΫλϯδΣϯτ • ReLUొҎલʹΘΕ͍ͯͨ׆ੑԽؔ • ࠷๏ͱΈ߹ΘͤΔͷࢦআͷΛ͠ͳ͍ͱޯ͕ফ͑ΔͨΊඇਪ • ϋΠύϘϦοΫλϯδΣϯτ߃ؔʹ͍ۙͷͰ·͍͍ͩ͢ • Ͱɼ͋Μ·Γݟͳ͍… 6.3
ӅΕϢχοτ !23 ϩδεςΟοΫγάϞΠυ ϋΠύϘϦοΫλϯδΣϯτ 6.3.3Ͱଞʹ৭ʑհ͞Ε͍͕ͯͨɼ͋·Γݟ͔͚ͳ͍ͷͰলུ
ΞʔΩςΫνϟͷઃܭ • ΞʔΩςΫνϟ: Ϣχοτͷ߹ͤɼωοτϫʔΫશମͷߏͷҙ • ສೳۙࣅఆཧ(universal approximation theorem): • ׆ੑԽ͕ؔগͳ͘ͱ1ؚͭ·ΕΔेʹਂ͍NNҙؔΛۙࣅՄೳ
• දݱͰ͖Δ͜ͱอূ͞Ε͍ͯΔ͕ɼֶशͰ͖Δ͔Ͳ͏͔อূ͞Ε͍ͯͳ͍ • ͦͦͲΕ͚ͩਂ͚Εྑ͍͔ෆ໌ • ࿈తʹͭͳ͛Δͷ͕Ұൠత͕ͩɼεΩοϓ͢ΔΞʔΩςΫνϟଟ͍ • …ͱ͍͏͔࠷ۙͷΞʔΩςΫνϟجຊͦΜͳײ͡… 6.4 ΞʔΩςΫνϟͷઃܭ !24 ʮਂ͚Εදݱͷ෯·Δ͕͋Δʯ͜ͱΛʹड़͍ͯΔ
ޡࠩٯ๏ • ॱ: ೖྗ͕࠷ॳͷใͱͳΓɼӅΕʹΘͬͯ࠷ऴతʹग़ྗΛੜ͢Δ͜ͱ • NNɼεΧϥʔͷଛࣦΛಘΔ·Ͱॱ͠ଓ͚Δ • ޡࠩٯ๏: ଛࣦ͔Βٯ͖ʹޯΛܭࢉ͢Δ͜ͱ •
͋͘·Ͱޯܭࢉख๏ͰɼֶशΞϧΰϦζϜΛࢦ͢Θ͚Ͱͳ͍ 6.5 ޡࠩٯ๏͓ΑͼͦͷଞͷඍΞϧΰϦζϜ !25 ޯਖ਼ ޯෛ ޯ0 ޯ͕0ʹݶΓͳ͍ۙ͘ → ࠷దղ͕ಘΒΕΔ ਤ͋͘·ͰΠϝʔδͰ͢
४උ: ܭࢉάϥϑ • ޡࠩٯ๏Λઆ໌͢Δʹ༗ܭࢉάϥϑ͕ศར • ล͕ԋࢉʹରԠ 6.5 ޡࠩٯ๏͓ΑͼͦͷଞͷඍΞϧΰϦζϜ !26
ඍੵͷ࿈ • Ͱ Ͱ͋Δͱ͖ɼ ҎԼཱ͕ • ҰൠԽ: Ͱ ͷͱ͖ҎԼཱ͕ •
ҰൠܗҎԼͷΑ͏ʹॻ͚Δ 6.5 ޡࠩٯ๏͓ΑͼͦͷଞͷඍΞϧΰϦζϜ !27
ޡࠩٯͷͨΊͷ࿈ͷ࠶ؼతͳద༻ 6.5 ޡࠩٯ๏͓ΑͼͦͷଞͷඍΞϧΰϦζϜ !28 ஞ࣍ޯܭࢉ → Algorithm 6.1 ಉ͡ܭࢉ݁ՌΛอ࣋ͯ͠దٓ׆͔͢ →
Algorithm 6.2
Algorithm 6.1 6.5 ޡࠩٯ๏͓ΑͼͦͷଞͷඍΞϧΰϦζϜ !29 ෳࡶʹݟ͑Δ͕ɼ˘෦ΛҰൠԽ͍ͯ͠Δ͚ͩ
Algorithm 6.2 6.5 ޡࠩٯ๏͓ΑͼͦͷଞͷඍΞϧΰϦζϜ !30 ෳࡶʹݟ͑Δ͕ɼ˘෦ΛҰൠԽ͍ͯ͠Δ͚ͩ
શ݁߹MLPͰॱͱଛࣦؔͷܭࢉ Algorithm 6.3 6.5 ޡࠩٯ๏͓ΑͼͦͷଞͷඍΞϧΰϦζϜ !31 ઢܗϢχοτ & ׆ੑԽؔ ଛࣦؔͷܭࢉ
શ݁߹MLPͰͷޡࠩٯ๏ Algorithm 6.4 6.5 ޡࠩٯ๏͓ΑͼͦͷଞͷඍΞϧΰϦζϜ !32 ֤εςοϓͰޯܭࢉͯ͠Δ͚ͩ
γϯϘϧؒͷඍ 6.5 ޡࠩٯ๏͓ΑͼͦͷଞͷඍΞϧΰϦζϜ !33 • γϯϘϦοΫදݱ: άϥϑʹجͮ͘දݱ • NNͰɼಛఆͷΛ࣋ͨͳ͍γϯϘϧมʹରͯ͠ԋࢉΛ࣮ࢪ •
ҎԼͰඍͷදݱΛ༩͑ΔϊʔυΛάϥϑʹՃ͢Δํ๏Ͱޡࠩٯ๏ΛҰൠԽ
Ұൠతͳޡࠩٯ๏ Algorithm 6.5 6.5 ޡࠩٯ๏͓ΑͼͦͷଞͷඍΞϧΰϦζϜ !34 ܭࢉάϥϑʹҰൠԽ ஞ࣍తʹඍͷγϯϘϦοΫදݱΛάϥϑʹՃ
ྫ: ޡࠩٯ๏ʹΑΔMLPͷ܇࿅ 6.5 ޡࠩٯ๏͓ΑͼͦͷଞͷඍΞϧΰϦζϜ !35
ྫ: ޡࠩٯ๏ʹΑΔMLPͷ܇࿅ 6.5 ޡࠩٯ๏͓ΑͼͦͷଞͷඍΞϧΰϦζϜ !36 ܇࿅ͷͨΊʹWͷޯΛಘ͍ͨ
ྫ: ޡࠩٯ๏ʹΑΔMLPͷ܇࿅ 6.5 ޡࠩٯ๏͓ΑͼͦͷଞͷඍΞϧΰϦζϜ !37 ܇࿅ͷͨΊʹWͷޯΛಘ͍ͨ ॏΈݮਰํͷW(i)ͷޯͷد༩ ਖ਼ଇԽ߲(ॏΈݮਰ߲)
ྫ: ޡࠩٯ๏ʹΑΔMLPͷ܇࿅ 6.5 ޡࠩٯ๏͓ΑͼͦͷଞͷඍΞϧΰϦζϜ !38 ܇࿅ͷͨΊʹWͷޯΛಘ͍ͨ ॏΈݮਰํͷW(i)ͷޯͷد༩ ަࠩΤϯτϩϐʔํͷW(i)ͷޯͷد༩ (cross_entropyͰఏڙ͞ΕΔޯΛGͱ͢Δ) ਖ਼ଇԽ߲(ॏΈݮਰ߲)
ReLUͷٯΛߟྀͨ͠ޯ
·ͱΊ • ઢܗϢχοτ+׆ੑԽؔΛΈ߹ΘͤΕNNͰҙͷؔΛදݱՄೳ - ઢܗϢχοτ: Ψεʹجͮ͘ग़ྗ - γάϞΠυϢχοτ: ϕϧψʔΠʹجͮ͘ग़ྗ -
ιϑτϚοΫεϢχοτ: ΧςΰϦΧϧʹجͮ͘ग़ྗ - ׆ੑԽؔࠔͬͨΒReLUΛ͑OK • NNͰදݱ͞ΕΔؔඇઢܗؔͳͷͰɼղੳతʹ࠷దղΛಋग़Ͱ͖ͳ͍ • ͦ͜ͰɼޯΛ༻͍ͯܭࢉతʹಋग़͢Δ (ͨͩ͠ہॴ࠷దղʹؕΔՄೳੑ͋Γ) • ޯΛٻΊΔࡍʹඍͷ࿈ʹج͍ͮͨޡࠩٯ๏Λར༻͢Δͱྑ͍ !39