Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ゼロから作るDeep Learning 2 3章 word2vec 3.1〜3.2
Search
Sponsored
·
SiteGround - Reliable hosting with speed, security, and support you can count on.
→
ota42y
May 29, 2019
Programming
1
560
ゼロから作るDeep Learning 2 3章 word2vec 3.1〜3.2
ゼロから作るDeep Learning 2 自然言語編 読書会 第5回
の資料です!
https://retrieva.connpass.com/event/131746/
ota42y
May 29, 2019
Tweet
Share
More Decks by ota42y
See All by ota42y
バックログを導入し やっぱやめた話
ota42y
1
370
PFNにある2つのKubernetes
ota42y
10
5.8k
Q&A for How to use OpenAPI3 for API developer
ota42y
0
2.8k
How to use OpenAPI3 for API developer (RubyKaigi 2019)
ota42y
5
22k
How should we face with microservices (我々はマイクロサービスとどう向き合うべきか)
ota42y
20
4.9k
DeepLearningの本番環境にSageMakerを利用してる話
ota42y
1
6.7k
検索結果の良さを計測して定量的に改善していく
ota42y
3
2.6k
Flutterを広めるために技術同人誌を作った話
ota42y
1
1.8k
何も考えずにCIや継続的デリバリーしたら辛くなった話.pdf
ota42y
0
3.2k
Other Decks in Programming
See All in Programming
今から始めるClaude Code超入門
448jp
8
9.1k
責任感のあるCloudWatchアラームを設計しよう
akihisaikeda
3
180
AtCoder Conference 2025
shindannin
0
1.1k
そのAIレビュー、レビューしてますか? / Are you reviewing those AI reviews?
rkaga
6
4.6k
CSC307 Lecture 05
javiergs
PRO
0
500
Oxlint JS plugins
kazupon
1
1k
副作用をどこに置くか問題:オブジェクト指向で整理する設計判断ツリー
koxya
1
610
Gemini for developers
meteatamel
0
100
Package Management Learnings from Homebrew
mikemcquaid
0
230
Claude Codeと2つの巻き戻し戦略 / Two Rewind Strategies with Claude Code
fruitriin
0
140
The Past, Present, and Future of Enterprise Java
ivargrimstad
0
620
例外処理とどう使い分ける?Result型を使ったエラー設計 #burikaigi
kajitack
16
6.1k
Featured
See All Featured
The Pragmatic Product Professional
lauravandoore
37
7.1k
What the history of the web can teach us about the future of AI
inesmontani
PRO
1
440
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
162
16k
HDC tutorial
michielstock
1
390
SEOcharity - Dark patterns in SEO and UX: How to avoid them and build a more ethical web
sarafernandez
0
120
Understanding Cognitive Biases in Performance Measurement
bluesmoon
32
2.8k
jQuery: Nuts, Bolts and Bling
dougneiner
65
8.4k
Deep Space Network (abreviated)
tonyrice
0
65
Speed Design
sergeychernyshev
33
1.5k
How to build an LLM SEO readiness audit: a practical framework
nmsamuel
1
650
SEO in 2025: How to Prepare for the Future of Search
ipullrank
3
3.3k
Groundhog Day: Seeking Process in Gaming for Health
codingconduct
0
98
Transcript
θϩ͔Β࡞ΔDeep Learning 2 ̏ষ word2vec 3.1ʙ3.2 ota42y θϩ͔Β࡞ΔDeep Learning
2 ࣗવݴޠฤ ಡॻձ ୈ5ճ
͜ͷষͰΔ͜ͱ • word2vecΛ࣮͢Δ • ਪϕʔεͰ୯ޠΛϕΫτϧͰද͢ํ๏ • γϯϓϧ͕ͩແବଟ͍࣮ • ࣍ͷষͰରԠ
3.1 ਪϕʔεͷख๏ͱ χϡʔϥϧωοτϫʔΫ
ਪϕʔεͷϕΫτϧԽ • ୯ޠΛϕΫτϧʹ͢Δ̎ͭͷख๏ • Χϯτϕʔεʢ̎ষʣ • ਪϕʔεʢ̏ষʣ • ͲͪΒԾઃΛϕʔεʹͯ͠Δ͕Ξϓϩʔνશ͘ผ •
Ծઃɿ୯ޠͷҙຯपғͷ୯ޠ͔Βܗ͞ΕΔ (p.67)
3.1.1ɹΧϯτϕʔεͷख๏ͷ • Χϯτϕʔεपғͷ୯ޠͷසΛܭࢉ͢Δ • ޠኮ͕nͩͱn*nͷڊେͳڞىߦྻ͕ඞཁʹͳΔ • ࣍ݩݮͷͨΊͷSVDO(n^3)ͷܭࢉྔɺ͍
ਪϕʔεͷར • Χϯτϕʔείʔύεશମͷ౷ܭσʔλΛҰؾʹར༻͢Δ • ਪϕʔε(χϡʔϥϧωοτ)ίʔύεͷҰ෦Ͱֶश͢Δ • GPUͷฒྻܭࢉฉ͘ • খ͚ʹͰ͖ɺߴʹฒྻॲཧͰ͖ΔͷͰڊେσʔλͰରԠͰ͖Δ •
ଞʹັྗతͳ͕͋Δ(Β͍͠ɺৄ͘͠3.5.3)
3.1.2ɹਪϕʔεͷख๏ͷ֓ཁ
पғͷ୯ޠ͔Β୯ޠΛʮਪʯ͢Δ • `?`ʹԿ͕ೖΔ͔Λલޙ͔Βਪ • ίϯςΩετ͔ΒλʔήοτΛਪ • ίϯςΩετɿपғͷ୯ޠ(you, goodby) • λʔήοτɿରͷ୯ޠ(`?`)
ਪ݁Ռ • ֤୯ޠ͕ͦ͜ʹݱΕΔ֬Λग़ྗ • ίϯςΩετΛϞσϧʹ༩͑Δͱ୯ޠͷ͕֬ಘΒΕΔ
3.1.3 χϡʔϥϧωοτϫʔΫʹ͓͚Δ୯ ޠͷॲཧํ๏ • χϡʔϥϧωοτϫʔΫ(NN)ͷೖྗݻఆϕΫτϧ • ୯ޠΛͦͷ··ೖΕΔͷ͍͠ • ୯ޠΛone-hotදݱ(one-hotϕΫτϧ)ʹม͢Δ
one-hotදݱ • ޠኮͷ͞Λ࣋ͪɺ୯ޠIDͱ֘͢Δ෦͕1ɺͦΕҎ֎͕0 ͷϕΫτϧ • ͯ͢ͷ୯ޠΛಉ͡͞ͷϕΫτϧͱͯ͠දݱ
one-hotදݱ • શ݁߹Ͱม͢ΔͳΒ؆୯(ྫதؒ=3)
αϯϓϧίʔυ(p.99) • np.dot(c, W)୯ޠʹରԠ͢ΔॏΈΛऔΓग़ͯ͠Δ͚ͩ • W[0]ͷσʔλΛऔΓग़ͯ͠Δ͚ͩ • ແବͬΆ͍͕࣍ͷষͰ࣏͢Β͍͠
ϨΠϠදݱ • MatMulϨΠϠ(p.30)Ͱಉ͜͡ͱ͕Ͱ͖Δ • np.dot͢Δ͚ͩͷϨΠϠͳͷͰ
3.2ɹγϯϓϧͳword2vec
word2vecΛ࣮͢Δ • word2vecͰΘΕΔϞσϧCROWϞσϧͱskip-gramϞσϧ • "word2vec"͕͜ΕΒͷϞσϧΛࢦ͢߹͋Δ • ຊདྷͷҙຯͱζϨͯΔ
3.2.1 CBOWϞσϧͷਪॲཧ • ίϯςΩετ͔ΒλʔήοτΛਪଌ͢ΔNN • ίϯςΩετʹपғͷ୯ޠ • λʔήοτʹରͷ୯ޠ
୯ޠͷࢄදݱ • CBOWϞσϧΛ܇࿅͢Δ͜ͱͰ୯ޠͷࢄදݱΛಘΒΕΔ • Ϟσϧͷύϥϝʔλ͕ࢄදݱʹରԠ͢Δ
CBOWϞσϧͷશମ૾ • ίϯςΩετʹ̎ɺӅΕʹ̏ͷ߹
CBOWϞσϧͷશମ૾ • ೖྗෳݸͷone-hotදݱͷ୯ޠ • ग़ྗ֤୯ޠͷείΞ • softmaxΛ͏ͱ͕֬ಘΒΕΔ • தؒೖྗ͔Βͷͷฏۉ
• ࢄදݱͷਖ਼ମ • [$ W_{in}]7*3ͷॏΈ • ͜Ε͕୯ޠͷࢄදݱ • ֶशʹΑͬͯྑ͍ࢄදݱʹ͍ͯ͘͠
CBOWϞσϧͷϨΠϠදݱ
CBOWϞσϧͷϨΠϠදݱ • ̎ͭͷMatMulϨΠϠ • ୯ޠʹରԠ͢ΔॏΈΛऔΓग़ͭ͢(P.99) • ̎ͭͷฏۉΛऔΔ(=ͯ͠0.5Λ͔͚Δ) • scoreͷશ݁߹ •
׆ੑԽؔແ͍ͷͰΘΓͱγϯϓϧ
3.2.2 CBOWϞσϧͷֶश • χϡʔϥϧωοτϫʔΫͷηΦϦʔ௨Γ • CBOWଞΫϥεྨΛ͢ΔNN • Ϋϥεʹone-hotͰද͞Εͨ୯ޠ • είΞ͔Β֬ΛٻΊͯɺਖ਼ղͱͷࠩΛֶश͢Δ
• Softmaxؔʹ͔͚ͯ֬ʹ͢Δ • ڭࢣϥϕϧ͔ΒަࠩΤϯτϩϐʔޡࠩΛٻΊΔ
ϨΠϠදݱ • Softmax with lossΛ͚Ճ͑Δ
ίʔυϦʔσΟϯά • ch03/cbow_predict.py • https://github.com/oreilly-japan/deep-learning-from- scratch-2/blob/master/ch03/cbow_predict.py
3.2.3 word2vecͷॏΈͱࢄදݱ • ͱɹɹͷҧ͍ • ྆ํͱ୯ޠͷҙຯ͕Τϯίʔυ͞Ε͍ͯΔ • ܗঢ়͕ҧ͏ • ɹɹ7x3
• ɹɹ3x7 Win Wout Win Wout
ࢄදݱɹɹΛ͏ • ɹɹ શ͘Θͳ͍ɹ • ɹɹʹର͢Δskip-ngramͰͷ༗༻ੑ࣮ݧ • https://arxiv.org/abs/1611.01462 • ɹɹ͏͜ͱͰΑ͍݁Ռ͕ಘΒΕΔͱ͍͏ใࠂ
• https://nlp.stanford.edu/projects/glove/ • word2vecͱࣅ͍ͯΔͭͷख๏ Win Win Wout Wout