Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Deep Learning Book 10その2 / deep learning book 1...
Search
himkt
January 29, 2018
Research
2
200
Deep Learning Book 10その2 / deep learning book 10 vol2
himkt
January 29, 2018
Tweet
Share
More Decks by himkt
See All by himkt
Linformer: paper reading
himkt
0
550
RoBERTa: paper reading
himkt
1
350
NLP SoTA 勉強会 / ner_2019
himkt
2
1.4k
自然言語処理 @ クックパッド / nlp at cookpad
himkt
1
530
Interpretable Machine Learning 6.3 - Prototypes and Criticisms
himkt
2
170
ニューラル固有表現抽出 / Neural Named Entity Recognition
himkt
3
760
ニューラル固有表現抽出器を実装してみる / PyNER
himkt
6
2.1k
Spacyでお手軽NLP / NLP with spacy
himkt
0
1k
ふわふわ系列ラベリング / ner 2018
himkt
5
850
Other Decks in Research
See All in Research
アニメにおける宇宙猫ミームとその表現
yttrium173340
0
100
長期・短期メモリを活用したエージェントの個別最適化
isidaitc
0
190
PhD Defense 2025: Visual Understanding of Human Hands in Interactions
tkhkaeio
1
250
論文紹介:Not All Tokens Are What You Need for Pretraining
kosuken
0
200
AI in Enterprises - Java and Open Source to the Rescue
ivargrimstad
0
740
Nullspace MPC
mizuhoaoki
1
200
なめらかなシステムと運用維持の終わらぬ未来 / dicomo2025_coherently_fittable_system
monochromegane
0
3.9k
Time to Cash: The Full Stack Breakdown of Modern ATM Attacks
ratatata
0
160
超高速データサイエンス
matsui_528
1
150
A scalable, annual aboveground biomass product for monitoring carbon impacts of ecosystem restoration projects
satai
4
360
Creation and environmental applications of 15-year daily inundation and vegetation maps for Siberia by integrating satellite and meteorological datasets
satai
3
400
Remote sensing × Multi-modal meta survey
satai
4
500
Featured
See All Featured
Context Engineering - Making Every Token Count
addyosmani
7
260
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
9
910
BBQ
matthewcrist
89
9.8k
4 Signs Your Business is Dying
shpigford
185
22k
GraphQLの誤解/rethinking-graphql
sonatard
73
11k
Git: the NoSQL Database
bkeepers
PRO
431
66k
Faster Mobile Websites
deanohume
310
31k
Thoughts on Productivity
jonyablonski
70
4.9k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Build The Right Thing And Hit Your Dates
maggiecrowley
37
2.9k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
21
1.2k
Transcript
&DIP4UBUF/FUXPSLT&YQMJDJU.FNPSZ IJNLU!य़ΤϦΞ DEEP LEARNING BOOK 4FRVFODF.PEFMJOH3FDVSSFOUBOE3FDVSTJWF/FUT
&DIP4UBUF/FUXPSLT w 3//ʹֶ͓͍ͯश͕େมͳύϥϝʔλ w ӅΕӅΕ SFDVSSFOUXFJHIUT w ೖྗӅΕ JOQVUXFJHIUT
w &DIP4UBUF/FUXPSL w ӅΕӅΕॏΈΛݻఆ w ֶश͢Δͷʜ w ೖྗӅΕ JOQVUXFJHIUT w ӅΕग़ྗ PVUQVUXFJHIUT
&DIP4UBUF/FUXPSLT IUUQXXXTDIPMBSQFEJBPSHBSUJDMF&DIP@TUBUF@OFUXPSL
,FSOFMNBDIJOFͱͷྨࣅੑ w Χʔωϧ͕ͬͯΔ͜ͱͬͯʁ w ҙͷ͞ͷܥྻΛݻఆͷϕΫτϧࣸ͢ w ݻఆͷϕΫτϧΛ༻͍ͯྨث͕Λղ͘ w ͜ͷܗͷ߹ɼֶशͷج४ͷઃܭ͕༰қͰ͋Δ w
ग़ྗઢܗճؼͷ߹.4&ͰֶशͰ͖Δ w &4/TೖྗΛԿΒ͔ͷϕΫτϧʹࣸ͢ૢ࡞Λ͍ͯ͠Δ w தͷॏΈݻఆ͍ͯ͠Δ ͍͔ʹաڈͷใΛ๛ʹؚΉදݱ͕ಘΒΕΔ ॏΈΛઃఆ͢ΕΑ͍͔ʁ શવҙຯ͕Θ͔Βͣʜ 3//ΛಈతγεςϜͱΈͳ͢ γεςϜ͕҆ఆ͢ΔΑ͏ͳॏΈΛઃఆ͢Δ
-FBLZ6OJUTBOE0UIFS4USBUFHJFTGPS.VMUJQMF5JNF4DBMF w աڈͷใΛ͑ΔͨΊͷ "EEJOH4LJQ$POOFDUJPOTUISPVHI5JNF w ޯͷফࣦͷ͕͘ͳΔ w രൃݩͷ3//ͱಉ͡Ͱൃੜ͢Δ
-FBLZ6OJUTBOEB4QFDUSVNPG%J⒎FSFOU5JNF4DBMFT w աڈͷใΛͲͷఔ͔͢Λ੍ޚ͢Δ 3FNPWJOH$POOFDUJPOT w ͍࣌ࠁͰͷґଘΛ͍࣌ࠁͰͷґଘʹஔ͖͑Δ
-FBLZ6OJUT w աڈͷใΛͲͷ͘Β͍͔͢Λௐ͢Δ w ҠಈฏۉͷΑ͏ͳ;Δ·͍Λ͢Δ w Ћ͕େ͖͍ ʹ͍ۙ աڈͷใΛΑΓอଘ͢Δ w
Ћ͕খ͍͞ ʹ͍ۙ աڈͷใΛ͙͢ʹࣺͯΔ w Ћదʹܾఆ͢ΔϋΠύʔύϥϝʔλ µ(t) ↵µ(t 1) + (1 ↵)v(t)
-POH4IPSU5FSN.FNPSZ w ࣗݾϧʔϓΛಋೖ͢Δ͜ͱͰޯ͕ফ͑ʹ͘͘͢Δ IUUQDPMBIHJUIVCJPQPTUT6OEFSTUBOEJOH-45.T 3// -45.
(BUFE3FDVSSFOU6OJUT w ٙ-45.ෳࡶ͗͢ΔͷͰͳ͍͔ʁ w (36-45.ΑΓߴɾ-45.ͱಉͷੑೳ w ͲͪΒ͕ྑ͍͔λεΫʹΑΔ -45. (36
IUUQTJTBBDDIBOHIBVHJUIVCJP-45.BOE(36'PSNVMB4VNNBSZ
ࣜతʹൺֱ͢ΔʢόΠΞεΛແࢹʣ -45. (36 zt = (xtUz + ht 1Wz)
rt = (xtUr + ht 1Wr) ˜ ht = tanh ⇣ xt + Uh + (rt ht 1)Wh ⌘ ht = (1 zt) ht 1 + zt ˜ ht it = (xtUi + ht 1Wi) ft = (xtUf + ht 1Wf ) ot = (xtUo + ht 1Wg) ˜ Ct = tanh (xtUg + ht 1Wg) Ct = (ft Ct 1 + it ˜ Ct) ht = tanh (Ct) ot (36Ͱೖྗήʔτͱ٫ήʔτ͕౷߹͞Ε͍ͯΔ
0QUJNJ[BUJPOGPS-POH5FSN%FQFOEFODJFT w 3//Λϕʔεͱͨ͠χϡʔϥϧωοτϫʔΫͷඍ w ඇৗʹେ͖ͳΛͱΔPS w ඇৗʹখ͞ͳΛͱΔ w ಛʹɼޯ͕ඇৗʹେ͖ͳͱ͖ʹͲ͏͢Εྑ͍͔ʁ
ޯͷΫϦοϐϯά ޯͷਖ਼نԽ
$MJQQJOH(SBEJFOU w ޯ͕ඇৗʹ େ͖͍cখ͍͞ ͱʁ w ͍͍ͩͨฏΒ͚ͩͲͱ͖Ͳ͖֑͕͋Δ IUUQXXXEFFQMFBSOJOHCPPLPSHMFDUVSF@TMJEFTIUNM
$MJQQJOH(SBEJFOU w ޯ๏ϕʔεͷख๏ʹΑΔͱʜ w ֑ͷपΓͰ͕ਧ͖ඈΜͰ͠·͏ ޯരൃ w ޯ͕େ͖͘ͳΓ͗ͨ͢ΒޯͷϊϧϜͰׂΔ w
ޯΛHͱͯ͠ʜ w WϋΠύʔύϥϝʔλ ࣗવݴޠॲཧͩͱ͕ଟ͍ g ( gv ||g|| (||g|| > v) g (otherwise)
3FHVMBSJ[JOHUP&ODPVSBHF*OGPSNBUJPO'MPX w ਖ਼ଇԽ߲Λಋೖ͢Δ͜ͱͰʮJOGPSNBUJPOqPXʯΛଅਐ w ͜ͷ߲ͷܭࢉ͍͕͠ɼۙࣅ͕ఏҊ͞Ε͍ͯΔ w $MJQQJOHͱΈ߹ΘͤΔ͜ͱͰهԱͰ͖Δڑ͕৳ͼΔ ⌦ =
X t ⇣||(rh(t) L) @h(t) @h(t 1) || ||(rh(t) L)|| 1 ⌘2
&YQMJDJU.FNPSZ w χϡʔϥϧωοτϫʔΫʜ w ҉తͳใͷอ࣋ಘҙ w ໌ࣔతͳใ ࣄ࣮ ͷอ࣋ۤख w
໌ࣔతͳใΛอ࣋͠ɼਪʹ׆༻͢Δߏ ʢϫʔΩϯάϝϞϦͷಋೖʣ w .FNPSZ/FUXPSLT w /FVSBM5VSJOH.BDIJOF
"TDIFNBUJDPGBOFUXPSLXJUIBOFYQMJDJUNFNPSZ IUUQXXXEFFQMFBSOJOHCPPLPSHMFDUVSF@TMJEFTIUNM
"TDIFNBUJDPGBOFUXPSLXJUIBOFYQMJDJUNFNPSZ w ਖ਼֬ͳϝϞϦͷΞυϨεΛग़ྗ͢Δͷ͍͠ w ଟ͘ͷϝϞϦηϧͷॏΈ͖ฏۉΛͱΔ w ॏΈιϑτϚοΫεͳͲͰ࡞Δ ʢͰ͖Δ͚ͩҰՕॴͷϝϞϦΛࢀর͢ΔΑ͏ʹʣ w ϝϞϦηϧεΧϥΑΓϕΫτϧͷํ͕ྑ͍
w ίϯςϯπϕʔεΞυϨογϯά͕ՄೳʹͳΔ w ʮl8FBMMMJWFJOBZFMMPXTVCNBSJOFzΛؚΉՎࢺΛݟ͚ͭΔʯ w ʢϩέʔγϣϯϕʔεΞυϨογϯάͱʁʣ w ʮεϩοτ347ʹ֨ೲ͞Ε͍ͯΔՎࢺΛऔಘ͢Δʯ w ʢΞυϨογϯάΞςϯγϣϯͱಉ͡ܗࣜʣ