Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Deep Learning Book 10その2 / deep learning book 1...
Search
himkt
January 29, 2018
Research
2
160
Deep Learning Book 10その2 / deep learning book 10 vol2
himkt
January 29, 2018
Tweet
Share
More Decks by himkt
See All by himkt
Linformer: paper reading
himkt
0
350
RoBERTa: paper reading
himkt
1
300
NLP SoTA 勉強会 / ner_2019
himkt
2
1.3k
自然言語処理 @ クックパッド / nlp at cookpad
himkt
1
480
Interpretable Machine Learning 6.3 - Prototypes and Criticisms
himkt
2
130
ニューラル固有表現抽出 / Neural Named Entity Recognition
himkt
3
640
ニューラル固有表現抽出器を実装してみる / PyNER
himkt
6
2k
Spacyでお手軽NLP / NLP with spacy
himkt
0
960
ふわふわ系列ラベリング / ner 2018
himkt
5
840
Other Decks in Research
See All in Research
LLM時代の半導体・集積回路
kentaroy47
1
430
熊本から日本の都市交通政策を立て直す~「車1割削減、渋滞半減、公共交通2倍」の実現へ~@公共交通マーケティング研究会リスタートセミナー
trafficbrain
0
100
Weekly AI Agents News!
masatoto
22
20k
Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve
eumesy
PRO
6
1.1k
snlp2024_multiheadMoE
takase
0
380
Isotropy, Clusters, and Classifiers
hpprc
3
560
[2024.08.30] Gemma-Ko, 오픈 언어모델에 한국어 입히기 @ 머신러닝부트캠프2024
beomi
0
590
SSII2024 [OS1] 画像認識におけるモデル・データの共進化
ssii
PRO
0
500
Kaggle役立ちアイテム紹介(入門編)
k951286
13
4.3k
SSII2024 [OS2] 大規模言語モデルとVision & Languageのこれから
ssii
PRO
5
1.4k
20240918 交通くまもとーく 未来の鉄道網編(太田恒平)
trafficbrain
0
120
DiscordにおけるキャラクターIPを活用したUGCコンテンツ生成サービスの ラピッドプロトタイピング ~国際ハッカソンでの事例研究
o_ob
0
230
Featured
See All Featured
Designing for humans not robots
tammielis
249
25k
Infographics Made Easy
chrislema
239
18k
The Mythical Team-Month
searls
218
43k
The World Runs on Bad Software
bkeepers
PRO
65
11k
Reflections from 52 weeks, 52 projects
jeffersonlam
346
20k
What's in a price? How to price your products and services
michaelherold
243
11k
Designing on Purpose - Digital PM Summit 2013
jponch
114
6.9k
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3.5k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
48k
Producing Creativity
orderedlist
PRO
341
39k
Raft: Consensus for Rubyists
vanstee
136
6.6k
VelocityConf: Rendering Performance Case Studies
addyosmani
325
23k
Transcript
&DIP4UBUF/FUXPSLT&YQMJDJU.FNPSZ IJNLU!य़ΤϦΞ DEEP LEARNING BOOK 4FRVFODF.PEFMJOH3FDVSSFOUBOE3FDVSTJWF/FUT
&DIP4UBUF/FUXPSLT w 3//ʹֶ͓͍ͯश͕େมͳύϥϝʔλ w ӅΕӅΕ SFDVSSFOUXFJHIUT w ೖྗӅΕ JOQVUXFJHIUT
w &DIP4UBUF/FUXPSL w ӅΕӅΕॏΈΛݻఆ w ֶश͢Δͷʜ w ೖྗӅΕ JOQVUXFJHIUT w ӅΕग़ྗ PVUQVUXFJHIUT
&DIP4UBUF/FUXPSLT IUUQXXXTDIPMBSQFEJBPSHBSUJDMF&DIP@TUBUF@OFUXPSL
,FSOFMNBDIJOFͱͷྨࣅੑ w Χʔωϧ͕ͬͯΔ͜ͱͬͯʁ w ҙͷ͞ͷܥྻΛݻఆͷϕΫτϧࣸ͢ w ݻఆͷϕΫτϧΛ༻͍ͯྨث͕Λղ͘ w ͜ͷܗͷ߹ɼֶशͷج४ͷઃܭ͕༰қͰ͋Δ w
ग़ྗઢܗճؼͷ߹.4&ͰֶशͰ͖Δ w &4/TೖྗΛԿΒ͔ͷϕΫτϧʹࣸ͢ૢ࡞Λ͍ͯ͠Δ w தͷॏΈݻఆ͍ͯ͠Δ ͍͔ʹաڈͷใΛ๛ʹؚΉදݱ͕ಘΒΕΔ ॏΈΛઃఆ͢ΕΑ͍͔ʁ શવҙຯ͕Θ͔Βͣʜ 3//ΛಈతγεςϜͱΈͳ͢ γεςϜ͕҆ఆ͢ΔΑ͏ͳॏΈΛઃఆ͢Δ
-FBLZ6OJUTBOE0UIFS4USBUFHJFTGPS.VMUJQMF5JNF4DBMF w աڈͷใΛ͑ΔͨΊͷ "EEJOH4LJQ$POOFDUJPOTUISPVHI5JNF w ޯͷফࣦͷ͕͘ͳΔ w രൃݩͷ3//ͱಉ͡Ͱൃੜ͢Δ
-FBLZ6OJUTBOEB4QFDUSVNPG%J⒎FSFOU5JNF4DBMFT w աڈͷใΛͲͷఔ͔͢Λ੍ޚ͢Δ 3FNPWJOH$POOFDUJPOT w ͍࣌ࠁͰͷґଘΛ͍࣌ࠁͰͷґଘʹஔ͖͑Δ
-FBLZ6OJUT w աڈͷใΛͲͷ͘Β͍͔͢Λௐ͢Δ w ҠಈฏۉͷΑ͏ͳ;Δ·͍Λ͢Δ w Ћ͕େ͖͍ ʹ͍ۙ աڈͷใΛΑΓอଘ͢Δ w
Ћ͕খ͍͞ ʹ͍ۙ աڈͷใΛ͙͢ʹࣺͯΔ w Ћదʹܾఆ͢ΔϋΠύʔύϥϝʔλ µ(t) ↵µ(t 1) + (1 ↵)v(t)
-POH4IPSU5FSN.FNPSZ w ࣗݾϧʔϓΛಋೖ͢Δ͜ͱͰޯ͕ফ͑ʹ͘͘͢Δ IUUQDPMBIHJUIVCJPQPTUT6OEFSTUBOEJOH-45.T 3// -45.
(BUFE3FDVSSFOU6OJUT w ٙ-45.ෳࡶ͗͢ΔͷͰͳ͍͔ʁ w (36-45.ΑΓߴɾ-45.ͱಉͷੑೳ w ͲͪΒ͕ྑ͍͔λεΫʹΑΔ -45. (36
IUUQTJTBBDDIBOHIBVHJUIVCJP-45.BOE(36'PSNVMB4VNNBSZ
ࣜతʹൺֱ͢ΔʢόΠΞεΛແࢹʣ -45. (36 zt = (xtUz + ht 1Wz)
rt = (xtUr + ht 1Wr) ˜ ht = tanh ⇣ xt + Uh + (rt ht 1)Wh ⌘ ht = (1 zt) ht 1 + zt ˜ ht it = (xtUi + ht 1Wi) ft = (xtUf + ht 1Wf ) ot = (xtUo + ht 1Wg) ˜ Ct = tanh (xtUg + ht 1Wg) Ct = (ft Ct 1 + it ˜ Ct) ht = tanh (Ct) ot (36Ͱೖྗήʔτͱ٫ήʔτ͕౷߹͞Ε͍ͯΔ
0QUJNJ[BUJPOGPS-POH5FSN%FQFOEFODJFT w 3//Λϕʔεͱͨ͠χϡʔϥϧωοτϫʔΫͷඍ w ඇৗʹେ͖ͳΛͱΔPS w ඇৗʹখ͞ͳΛͱΔ w ಛʹɼޯ͕ඇৗʹେ͖ͳͱ͖ʹͲ͏͢Εྑ͍͔ʁ
ޯͷΫϦοϐϯά ޯͷਖ਼نԽ
$MJQQJOH(SBEJFOU w ޯ͕ඇৗʹ େ͖͍cখ͍͞ ͱʁ w ͍͍ͩͨฏΒ͚ͩͲͱ͖Ͳ͖֑͕͋Δ IUUQXXXEFFQMFBSOJOHCPPLPSHMFDUVSF@TMJEFTIUNM
$MJQQJOH(SBEJFOU w ޯ๏ϕʔεͷख๏ʹΑΔͱʜ w ֑ͷपΓͰ͕ਧ͖ඈΜͰ͠·͏ ޯരൃ w ޯ͕େ͖͘ͳΓ͗ͨ͢ΒޯͷϊϧϜͰׂΔ w
ޯΛHͱͯ͠ʜ w WϋΠύʔύϥϝʔλ ࣗવݴޠॲཧͩͱ͕ଟ͍ g ( gv ||g|| (||g|| > v) g (otherwise)
3FHVMBSJ[JOHUP&ODPVSBHF*OGPSNBUJPO'MPX w ਖ਼ଇԽ߲Λಋೖ͢Δ͜ͱͰʮJOGPSNBUJPOqPXʯΛଅਐ w ͜ͷ߲ͷܭࢉ͍͕͠ɼۙࣅ͕ఏҊ͞Ε͍ͯΔ w $MJQQJOHͱΈ߹ΘͤΔ͜ͱͰهԱͰ͖Δڑ͕৳ͼΔ ⌦ =
X t ⇣||(rh(t) L) @h(t) @h(t 1) || ||(rh(t) L)|| 1 ⌘2
&YQMJDJU.FNPSZ w χϡʔϥϧωοτϫʔΫʜ w ҉తͳใͷอ࣋ಘҙ w ໌ࣔతͳใ ࣄ࣮ ͷอ࣋ۤख w
໌ࣔతͳใΛอ࣋͠ɼਪʹ׆༻͢Δߏ ʢϫʔΩϯάϝϞϦͷಋೖʣ w .FNPSZ/FUXPSLT w /FVSBM5VSJOH.BDIJOF
"TDIFNBUJDPGBOFUXPSLXJUIBOFYQMJDJUNFNPSZ IUUQXXXEFFQMFBSOJOHCPPLPSHMFDUVSF@TMJEFTIUNM
"TDIFNBUJDPGBOFUXPSLXJUIBOFYQMJDJUNFNPSZ w ਖ਼֬ͳϝϞϦͷΞυϨεΛग़ྗ͢Δͷ͍͠ w ଟ͘ͷϝϞϦηϧͷॏΈ͖ฏۉΛͱΔ w ॏΈιϑτϚοΫεͳͲͰ࡞Δ ʢͰ͖Δ͚ͩҰՕॴͷϝϞϦΛࢀর͢ΔΑ͏ʹʣ w ϝϞϦηϧεΧϥΑΓϕΫτϧͷํ͕ྑ͍
w ίϯςϯπϕʔεΞυϨογϯά͕ՄೳʹͳΔ w ʮl8FBMMMJWFJOBZFMMPXTVCNBSJOFzΛؚΉՎࢺΛݟ͚ͭΔʯ w ʢϩέʔγϣϯϕʔεΞυϨογϯάͱʁʣ w ʮεϩοτ347ʹ֨ೲ͞Ε͍ͯΔՎࢺΛऔಘ͢Δʯ w ʢΞυϨογϯάΞςϯγϣϯͱಉ͡ܗࣜʣ