Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
城ヶ崎美嘉で学ぶRNNLM
Kento Nozawa
June 05, 2016
Programming
2
2.7k
城ヶ崎美嘉で学ぶRNNLM
オタク機械学習勉強会#0 のLT
Kento Nozawa
June 05, 2016
Tweet
Share
More Decks by Kento Nozawa
See All by Kento Nozawa
[NeurIPS Japan meetup 2021 talk] Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
nzw0301
0
49
[IBIS2021] 対照的自己教師付き表現学習おける負例数の解析
nzw0301
0
46
Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
nzw0301
0
130
Introduction of PAC-Bayes and its Application for Contrastive Unsupervised Representation Learning
nzw0301
2
480
NLP Tutorial; word representation learning
nzw0301
0
82
Analyzing Centralities of Embedded Nodes
nzw0301
0
86
Paper Reading: Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics
nzw0301
2
790
Deep Learning book 18. Confront the Partition Function
nzw0301
0
250
Deep Learning book 11. Practical Methodology
nzw0301
0
310
Other Decks in Programming
See All in Programming
書籍『良いコード/悪いコードで学ぶ設計入門』でエンジニアリングの当たり前を変える
minodriven
3
1.2k
Unboxing Rails 7
claudiob
1
120
Quartoを使ってみませんか / quarto_get_started
s_uryu
2
420
SRE NEXT 2022: Sensible Incident Management for Software Startups
takanabe
2
770
あなたの会社の古いシステム、なんとかしませんか?~システム刷新から考えるDX化への道筋とバリエーション~/webinar20220420-systems
grapecity_dev
0
130
Your Test Suite is Making Too Many Database Calls!
joelq
0
200
不具合に立ち向かう テスト戦略 ~ NestJSで作るCI環境 ~
naoki_haba
0
130
TechFeed Conference 2022 - Kotlin Experimental
jmatsu
0
860
スモールチームがAmazon Cognitoでコスパよく作るサービス間連携認証
tacke_jp
2
920
You CANt teach an old dog new tricks
michaelbukachi
0
120
Reactive Microservices with Spring Boot and JHipster - Spring I/O 2022
mraible
PRO
2
520
既存画面の Jetpack Composeでの書き換え: FAANSでの事例紹介 / Case study of rewriting existing screens with Jetpack Compose
horie1024
0
350
Featured
See All Featured
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
37
3.2k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
151
12k
Stop Working from a Prison Cell
hatefulcrawdad
261
17k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
498
130k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
15
920
Designing Dashboards & Data Visualisations in Web Apps
destraynor
224
49k
Fashionably flexible responsive web design (full day workshop)
malarkey
396
62k
In The Pink: A Labor of Love
frogandcode
130
21k
Fantastic passwords and where to find them - at NoRuKo
philnash
25
1.5k
Gamification - CAS2011
davidbonilla
75
3.9k
YesSQL, Process and Tooling at Scale
rocio
157
12k
Producing Creativity
orderedlist
PRO
333
37k
Transcript
ϲ࡚ඒՅ Λը૾ݕࡧ͓ͯͪ͠Լ͍͞
ϲ࡚ඒՅͰֶͿ RNNLM 2016/6/5 ΦλΫػցֶशษڧձ #0 @nzw0301
Ϟνϕʔγϣϯ ϲ࡚ඒՅͷηϦϑੜ
Recurrent Neural Network Language Model • ηϦϑੜ: લ·Ͱͷ୯ޠ͔Β࣍ͷ1୯ޠΛ༧ଌ͠ଓ͚Δ • ྫɿΊΔΊΔʜᣦՅʹϝʔϧૹ৴ͬ˒
• ୯ޠׂ: <BOS> ΊΔΊΔʜᣦՅʹϝʔϧૹ৴ͬ˒&04 • ֶश: Q ΊΔΊΔc#04 ͱ͔ Q ᣦՅc<BOS>, ΊΔΊΔ ʜ
RNNLMͷߏ ޠኮV࣍ݩͷϕΫτϧ softmax ؔ 1ͭલͷதؒͷϕΫτϧ RNNͷ༝ԑ h࣍ݩͷதؒ
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿೖྗ w #04ͷPOFPG,දݱΛೖྗ w ࣍ݩͰີͳϕΫτϧʹม <BOS> ΊΔΊΔ 0 B
B B B B @ 0 1 0 . . . 0 1 C C C C C A
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿதؒ • ີͳϕΫτϧΛதؒʹ͢ • ଟύʔηϓτϩϯͱಉ͡ <BOS> ΊΔΊΔ
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿग़ྗ • ग़ྗʹதؒͷϕΫτϧΛ͢ • ݱࡏͷதؒͷΛอ࣋ <BOS> ΊΔΊΔ
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿॏΈߋ৽ • SoftmaxؔͰ֬Λܭࢉ • Backpropagation Ͱ ΊΔΊΔ ͷ͕֬େ͖͘ͳΔΑ͏ʹߋ৽ <BOS>
ΊΔΊΔ
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿೖྗ ૄΊΔΊΔϕΫτϧΛೖྗ͠ɼີͳΊΔΊΔϕΫτϧʹม p(ΊΔΊΔ|<BOS>)Ͱܭࢉͨ͠தؒͷϕΫτϧ ʜ ΊΔΊΔ 0 B B
B B B B B B B B @ 0 . . . 0 1 0 . . . 0 1 C C C C C C C C C C A
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿதؒ ີͳΊΔΊΔϕΫτϧͱલʹܭࢉͨ͠தؒͷϕΫτϧΛதؒ p(ΊΔΊΔ|<BOS>)Ͱܭࢉͨ͠தؒͷϕΫτϧ ʜ ΊΔΊΔ
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿग़ྗ • ग़ྗʹதؒͷϕΫτϧΛͯ͠ɼݱࡏͷதؒͷϕΫτϧΛอ࣋ p(ʜ|<BOS>, ΊΔΊΔ)Ͱܭࢉͨ͠தؒͷϕΫτϧ ʜ ΊΔΊΔ
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿॏΈߋ৽ • SoftmaxؔͰ֬Λܭࢉ • Backpropagation Ͱ ʜ ͷ͕֬େ͖͘ͳΔΑ͏ʹߋ৽
ʜ ΊΔΊΔ
࣮ݧ
࣮ݧ֓ཁ • SCRNΛ༻ • LSTM GRU ΛΘͳ͍ • Keras
Ͱ࣮ • લॲཧ • ܗଶૉղੳͤͣʹจࣈ୯ҐͰֶश • /。|★|?|!|♪/ ͰηϦϑΛׂ • 900ηϦϑ (Վࢺ) Λ༻ • ϞόϚε • σϨες • TOKIMEKIΤεΧϨʔτ
݁Ռ
10epochޙɿϓϩσϡʔαʔͷҰ෦͕ͱΕͯΔ ϓϩσϩσϡʔͯͳͪʙʹෲΞλ γ΄ϡʔαʔΒతͳʔɺͨ͜ͳ
40epochޙɿΪϟϧޠʁ ϓϩσϡʔαʔʹ͍ͪΌΜɺ ݟ͘ͳ͍ʔ͘ͱԿߴͩ͠ʔͬ̇
80epochޙɿݺΕͨؾ͕ͨ͠ ϓϩσϡʔαʔ!
“<BOS> ϓ” ͔Β࠷ਪఆɿϧʔϓ ϓϩσϡʔαʔɺΞλγͷ͜ͱ͔Βɺ ϓϩσϡʔαʔɺΞλγͷ͜ͱ
ϥϯμϜʹηϦϑੜ
ॴײ • ηϦϑΛͲ͜ͰΔ͖͔ • ྫɿ͝Μʹ͢Δ?͓෩࿊ʹ͢Δ?…͜ΕͪΐͬͱϕλͬΆ͍ͳ͊ • ? Ͱ۠Δ͖͔൱͔ • …લޙͲͬͪͰ۠Δ͔൱͔ʁͦΕͱͳ͘͢ʁ
• ήʔϜը໘ͷͨΊ͔1ηϦϑܥྻ͕΄΅Ұఆʢֶͼʣ
ࢀߟจݙͳͲ • http://keras.io/ • DLͷϥΠϒϥϦ • ָ͍͢͝ʹॻ͚Δ • Mikolov at.el.
Recurrent neural network based language model. 2010. • RNNͷը૾͜ͷจͷͷΛ༻ • Mikolov at.el Learning Longer Memory in Recurrent Neural Networks. 2014. • ࠓճ༻ͨ͠Ϟσϧ