Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
城ヶ崎美嘉で学ぶRNNLM
Search
Kento Nozawa
June 05, 2016
Programming
2
3k
城ヶ崎美嘉で学ぶRNNLM
オタク機械学習勉強会#0 のLT
Kento Nozawa
June 05, 2016
Tweet
Share
More Decks by Kento Nozawa
See All by Kento Nozawa
Analysis on Negative Sample Size in Contrastive Unsupervised Representation Learning
nzw0301
0
160
[IJCAI-ECAI 2022] Evaluation Methods for Representation Learning: A Survey
nzw0301
0
610
[NeurIPS Japan meetup 2021 talk] Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
nzw0301
0
190
[IBIS2021] 対照的自己教師付き表現学習おける負例数の解析
nzw0301
0
180
Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
nzw0301
0
480
Introduction of PAC-Bayes and its Application for Contrastive Unsupervised Representation Learning
nzw0301
2
810
NLP Tutorial; word representation learning
nzw0301
0
210
Analyzing Centralities of Embedded Nodes
nzw0301
0
160
Paper Reading: Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics
nzw0301
2
1.2k
Other Decks in Programming
See All in Programming
Webの外へ飛び出せ NativePHPが切り拓くPHPの未来
takuyakatsusa
2
470
Is Xcode slowly dying out in 2025?
uetyo
1
250
A2A プロトコルを試してみる
azukiazusa1
2
1.3k
プロダクト志向ってなんなんだろうね
righttouch
PRO
0
180
0626 Findy Product Manager LT Night_高田スライド_speaker deck用
mana_takada
0
140
Flutterで備える!Accessibility Nutrition Labels完全ガイド
yuukiw00w
0
140
XP, Testing and ninja testing
m_seki
3
220
AIエージェントはこう育てる - GitHub Copilot Agentとチームの共進化サイクル
koboriakira
0
490
エンジニア向け採用ピッチ資料
inusan
0
180
PHP 8.4の新機能「プロパティフック」から学ぶオブジェクト指向設計とリスコフの置換原則
kentaroutakeda
2
720
Google Agent Development Kit でLINE Botを作ってみた
ymd65536
2
220
「Cursor/Devin全社導入の理想と現実」のその後
saitoryc
0
700
Featured
See All Featured
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Mobile First: as difficult as doing things right
swwweet
223
9.7k
4 Signs Your Business is Dying
shpigford
184
22k
Build The Right Thing And Hit Your Dates
maggiecrowley
36
2.8k
Embracing the Ebb and Flow
colly
86
4.7k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Code Review Best Practice
trishagee
69
18k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
130
19k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
107
19k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
Transcript
ϲ࡚ඒՅ Λը૾ݕࡧ͓ͯͪ͠Լ͍͞
ϲ࡚ඒՅͰֶͿ RNNLM 2016/6/5 ΦλΫػցֶशษڧձ #0 @nzw0301
Ϟνϕʔγϣϯ ϲ࡚ඒՅͷηϦϑੜ
Recurrent Neural Network Language Model • ηϦϑੜ: લ·Ͱͷ୯ޠ͔Β࣍ͷ1୯ޠΛ༧ଌ͠ଓ͚Δ • ྫɿΊΔΊΔʜᣦՅʹϝʔϧૹ৴ͬ˒
• ୯ޠׂ: <BOS> ΊΔΊΔʜᣦՅʹϝʔϧૹ৴ͬ˒&04 • ֶश: Q ΊΔΊΔc#04 ͱ͔ Q ᣦՅc<BOS>, ΊΔΊΔ ʜ
RNNLMͷߏ ޠኮV࣍ݩͷϕΫτϧ softmax ؔ 1ͭલͷதؒͷϕΫτϧ RNNͷ༝ԑ h࣍ݩͷதؒ
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿೖྗ w #04ͷPOFPG,දݱΛೖྗ w ࣍ݩͰີͳϕΫτϧʹม <BOS> ΊΔΊΔ 0 B
B B B B @ 0 1 0 . . . 0 1 C C C C C A
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿதؒ • ີͳϕΫτϧΛதؒʹ͢ • ଟύʔηϓτϩϯͱಉ͡ <BOS> ΊΔΊΔ
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿग़ྗ • ग़ྗʹதؒͷϕΫτϧΛ͢ • ݱࡏͷதؒͷΛอ࣋ <BOS> ΊΔΊΔ
p(ΊΔΊΔ|<BOS>) ͷܭࢉྫɿॏΈߋ৽ • SoftmaxؔͰ֬Λܭࢉ • Backpropagation Ͱ ΊΔΊΔ ͷ͕֬େ͖͘ͳΔΑ͏ʹߋ৽ <BOS>
ΊΔΊΔ
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿೖྗ ૄΊΔΊΔϕΫτϧΛೖྗ͠ɼີͳΊΔΊΔϕΫτϧʹม p(ΊΔΊΔ|<BOS>)Ͱܭࢉͨ͠தؒͷϕΫτϧ ʜ ΊΔΊΔ 0 B B
B B B B B B B B @ 0 . . . 0 1 0 . . . 0 1 C C C C C C C C C C A
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿதؒ ີͳΊΔΊΔϕΫτϧͱલʹܭࢉͨ͠தؒͷϕΫτϧΛதؒ p(ΊΔΊΔ|<BOS>)Ͱܭࢉͨ͠தؒͷϕΫτϧ ʜ ΊΔΊΔ
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿग़ྗ • ग़ྗʹதؒͷϕΫτϧΛͯ͠ɼݱࡏͷதؒͷϕΫτϧΛอ࣋ p(ʜ|<BOS>, ΊΔΊΔ)Ͱܭࢉͨ͠தؒͷϕΫτϧ ʜ ΊΔΊΔ
p(ʜc#04 ΊΔΊΔ) ͷܭࢉྫɿॏΈߋ৽ • SoftmaxؔͰ֬Λܭࢉ • Backpropagation Ͱ ʜ ͷ͕֬େ͖͘ͳΔΑ͏ʹߋ৽
ʜ ΊΔΊΔ
࣮ݧ
࣮ݧ֓ཁ • SCRNΛ༻ • LSTM GRU ΛΘͳ͍ • Keras
Ͱ࣮ • લॲཧ • ܗଶૉղੳͤͣʹจࣈ୯ҐͰֶश • /。|★|?|!|♪/ ͰηϦϑΛׂ • 900ηϦϑ (Վࢺ) Λ༻ • ϞόϚε • σϨες • TOKIMEKIΤεΧϨʔτ
݁Ռ
10epochޙɿϓϩσϡʔαʔͷҰ෦͕ͱΕͯΔ ϓϩσϩσϡʔͯͳͪʙʹෲΞλ γ΄ϡʔαʔΒతͳʔɺͨ͜ͳ
40epochޙɿΪϟϧޠʁ ϓϩσϡʔαʔʹ͍ͪΌΜɺ ݟ͘ͳ͍ʔ͘ͱԿߴͩ͠ʔͬ̇
80epochޙɿݺΕͨؾ͕ͨ͠ ϓϩσϡʔαʔ!
“<BOS> ϓ” ͔Β࠷ਪఆɿϧʔϓ ϓϩσϡʔαʔɺΞλγͷ͜ͱ͔Βɺ ϓϩσϡʔαʔɺΞλγͷ͜ͱ
ϥϯμϜʹηϦϑੜ
ॴײ • ηϦϑΛͲ͜ͰΔ͖͔ • ྫɿ͝Μʹ͢Δ?͓෩࿊ʹ͢Δ?…͜ΕͪΐͬͱϕλͬΆ͍ͳ͊ • ? Ͱ۠Δ͖͔൱͔ • …લޙͲͬͪͰ۠Δ͔൱͔ʁͦΕͱͳ͘͢ʁ
• ήʔϜը໘ͷͨΊ͔1ηϦϑܥྻ͕΄΅Ұఆʢֶͼʣ
ࢀߟจݙͳͲ • http://keras.io/ • DLͷϥΠϒϥϦ • ָ͍͢͝ʹॻ͚Δ • Mikolov at.el.
Recurrent neural network based language model. 2010. • RNNͷը૾͜ͷจͷͷΛ༻ • Mikolov at.el Learning Longer Memory in Recurrent Neural Networks. 2014. • ࠓճ༻ͨ͠Ϟσϧ