Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ゼロから作るDeep Learning 2 3章 word2vec 3.1〜3.2
Search
ota42y
May 29, 2019
Programming
1
550
ゼロから作るDeep Learning 2 3章 word2vec 3.1〜3.2
ゼロから作るDeep Learning 2 自然言語編 読書会 第5回
の資料です!
https://retrieva.connpass.com/event/131746/
ota42y
May 29, 2019
Tweet
Share
More Decks by ota42y
See All by ota42y
バックログを導入し やっぱやめた話
ota42y
1
350
PFNにある2つのKubernetes
ota42y
10
5.7k
Q&A for How to use OpenAPI3 for API developer
ota42y
0
2.8k
How to use OpenAPI3 for API developer (RubyKaigi 2019)
ota42y
5
22k
How should we face with microservices (我々はマイクロサービスとどう向き合うべきか)
ota42y
20
4.8k
DeepLearningの本番環境にSageMakerを利用してる話
ota42y
1
6.6k
検索結果の良さを計測して定量的に改善していく
ota42y
3
2.6k
Flutterを広めるために技術同人誌を作った話
ota42y
1
1.8k
何も考えずにCIや継続的デリバリーしたら辛くなった話.pdf
ota42y
0
3.2k
Other Decks in Programming
See All in Programming
How Software Deployment tools have changed in the past 20 years
geshan
0
28k
30分でDoctrineの仕組みと使い方を完全にマスターする / phpconkagawa 2025 Doctrine
ttskch
3
730
ハイパーメディア駆動アプリケーションとIslandアーキテクチャ: htmxによるWebアプリケーション開発と動的UIの局所的適用
nowaki28
0
340
AIエージェントを活かすPM術 AI駆動開発の現場から
gyuta
0
230
20251127_ぼっちのための懇親会対策会議
kokamoto01_metaps
2
400
CloudNative Days Winter 2025: 一週間で作る低レイヤコンテナランタイム
ternbusty
7
1.9k
AIコーディングエージェント(Manus)
kondai24
0
120
React Native New Architecture 移行実践報告
taminif
1
130
Building AI Agents with TypeScript #TSKaigiHokuriku
izumin5210
6
1.2k
All(?) About Point Sets
hole
0
260
Reactive Thinking with Signals and the new Resource API
manfredsteyer
PRO
0
160
[SF Ruby Conf 2025] Rails X
palkan
0
440
Featured
See All Featured
Keith and Marios Guide to Fast Websites
keithpitt
413
23k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
46
2.6k
Code Review Best Practice
trishagee
73
19k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.5k
A designer walks into a library…
pauljervisheath
210
24k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
55
3.1k
We Have a Design System, Now What?
morganepeng
54
7.9k
4 Signs Your Business is Dying
shpigford
186
22k
Build The Right Thing And Hit Your Dates
maggiecrowley
38
3k
Embracing the Ebb and Flow
colly
88
4.9k
Why You Should Never Use an ORM
jnunemaker
PRO
60
9.6k
YesSQL, Process and Tooling at Scale
rocio
174
15k
Transcript
θϩ͔Β࡞ΔDeep Learning 2 ̏ষ word2vec 3.1ʙ3.2 ota42y θϩ͔Β࡞ΔDeep Learning
2 ࣗવݴޠฤ ಡॻձ ୈ5ճ
͜ͷষͰΔ͜ͱ • word2vecΛ࣮͢Δ • ਪϕʔεͰ୯ޠΛϕΫτϧͰද͢ํ๏ • γϯϓϧ͕ͩແବଟ͍࣮ • ࣍ͷষͰରԠ
3.1 ਪϕʔεͷख๏ͱ χϡʔϥϧωοτϫʔΫ
ਪϕʔεͷϕΫτϧԽ • ୯ޠΛϕΫτϧʹ͢Δ̎ͭͷख๏ • Χϯτϕʔεʢ̎ষʣ • ਪϕʔεʢ̏ষʣ • ͲͪΒԾઃΛϕʔεʹͯ͠Δ͕Ξϓϩʔνશ͘ผ •
Ծઃɿ୯ޠͷҙຯपғͷ୯ޠ͔Βܗ͞ΕΔ (p.67)
3.1.1ɹΧϯτϕʔεͷख๏ͷ • Χϯτϕʔεपғͷ୯ޠͷසΛܭࢉ͢Δ • ޠኮ͕nͩͱn*nͷڊେͳڞىߦྻ͕ඞཁʹͳΔ • ࣍ݩݮͷͨΊͷSVDO(n^3)ͷܭࢉྔɺ͍
ਪϕʔεͷར • Χϯτϕʔείʔύεશମͷ౷ܭσʔλΛҰؾʹར༻͢Δ • ਪϕʔε(χϡʔϥϧωοτ)ίʔύεͷҰ෦Ͱֶश͢Δ • GPUͷฒྻܭࢉฉ͘ • খ͚ʹͰ͖ɺߴʹฒྻॲཧͰ͖ΔͷͰڊେσʔλͰରԠͰ͖Δ •
ଞʹັྗతͳ͕͋Δ(Β͍͠ɺৄ͘͠3.5.3)
3.1.2ɹਪϕʔεͷख๏ͷ֓ཁ
पғͷ୯ޠ͔Β୯ޠΛʮਪʯ͢Δ • `?`ʹԿ͕ೖΔ͔Λલޙ͔Βਪ • ίϯςΩετ͔ΒλʔήοτΛਪ • ίϯςΩετɿपғͷ୯ޠ(you, goodby) • λʔήοτɿରͷ୯ޠ(`?`)
ਪ݁Ռ • ֤୯ޠ͕ͦ͜ʹݱΕΔ֬Λग़ྗ • ίϯςΩετΛϞσϧʹ༩͑Δͱ୯ޠͷ͕֬ಘΒΕΔ
3.1.3 χϡʔϥϧωοτϫʔΫʹ͓͚Δ୯ ޠͷॲཧํ๏ • χϡʔϥϧωοτϫʔΫ(NN)ͷೖྗݻఆϕΫτϧ • ୯ޠΛͦͷ··ೖΕΔͷ͍͠ • ୯ޠΛone-hotදݱ(one-hotϕΫτϧ)ʹม͢Δ
one-hotදݱ • ޠኮͷ͞Λ࣋ͪɺ୯ޠIDͱ֘͢Δ෦͕1ɺͦΕҎ֎͕0 ͷϕΫτϧ • ͯ͢ͷ୯ޠΛಉ͡͞ͷϕΫτϧͱͯ͠දݱ
one-hotදݱ • શ݁߹Ͱม͢ΔͳΒ؆୯(ྫதؒ=3)
αϯϓϧίʔυ(p.99) • np.dot(c, W)୯ޠʹରԠ͢ΔॏΈΛऔΓग़ͯ͠Δ͚ͩ • W[0]ͷσʔλΛऔΓग़ͯ͠Δ͚ͩ • ແବͬΆ͍͕࣍ͷষͰ࣏͢Β͍͠
ϨΠϠදݱ • MatMulϨΠϠ(p.30)Ͱಉ͜͡ͱ͕Ͱ͖Δ • np.dot͢Δ͚ͩͷϨΠϠͳͷͰ
3.2ɹγϯϓϧͳword2vec
word2vecΛ࣮͢Δ • word2vecͰΘΕΔϞσϧCROWϞσϧͱskip-gramϞσϧ • "word2vec"͕͜ΕΒͷϞσϧΛࢦ͢߹͋Δ • ຊདྷͷҙຯͱζϨͯΔ
3.2.1 CBOWϞσϧͷਪॲཧ • ίϯςΩετ͔ΒλʔήοτΛਪଌ͢ΔNN • ίϯςΩετʹपғͷ୯ޠ • λʔήοτʹରͷ୯ޠ
୯ޠͷࢄදݱ • CBOWϞσϧΛ܇࿅͢Δ͜ͱͰ୯ޠͷࢄදݱΛಘΒΕΔ • Ϟσϧͷύϥϝʔλ͕ࢄදݱʹରԠ͢Δ
CBOWϞσϧͷશମ૾ • ίϯςΩετʹ̎ɺӅΕʹ̏ͷ߹
CBOWϞσϧͷશମ૾ • ೖྗෳݸͷone-hotදݱͷ୯ޠ • ग़ྗ֤୯ޠͷείΞ • softmaxΛ͏ͱ͕֬ಘΒΕΔ • தؒೖྗ͔Βͷͷฏۉ
• ࢄදݱͷਖ਼ମ • [$ W_{in}]7*3ͷॏΈ • ͜Ε͕୯ޠͷࢄදݱ • ֶशʹΑͬͯྑ͍ࢄදݱʹ͍ͯ͘͠
CBOWϞσϧͷϨΠϠදݱ
CBOWϞσϧͷϨΠϠදݱ • ̎ͭͷMatMulϨΠϠ • ୯ޠʹରԠ͢ΔॏΈΛऔΓग़ͭ͢(P.99) • ̎ͭͷฏۉΛऔΔ(=ͯ͠0.5Λ͔͚Δ) • scoreͷશ݁߹ •
׆ੑԽؔແ͍ͷͰΘΓͱγϯϓϧ
3.2.2 CBOWϞσϧͷֶश • χϡʔϥϧωοτϫʔΫͷηΦϦʔ௨Γ • CBOWଞΫϥεྨΛ͢ΔNN • Ϋϥεʹone-hotͰද͞Εͨ୯ޠ • είΞ͔Β֬ΛٻΊͯɺਖ਼ղͱͷࠩΛֶश͢Δ
• Softmaxؔʹ͔͚ͯ֬ʹ͢Δ • ڭࢣϥϕϧ͔ΒަࠩΤϯτϩϐʔޡࠩΛٻΊΔ
ϨΠϠදݱ • Softmax with lossΛ͚Ճ͑Δ
ίʔυϦʔσΟϯά • ch03/cbow_predict.py • https://github.com/oreilly-japan/deep-learning-from- scratch-2/blob/master/ch03/cbow_predict.py
3.2.3 word2vecͷॏΈͱࢄදݱ • ͱɹɹͷҧ͍ • ྆ํͱ୯ޠͷҙຯ͕Τϯίʔυ͞Ε͍ͯΔ • ܗঢ়͕ҧ͏ • ɹɹ7x3
• ɹɹ3x7 Win Wout Win Wout
ࢄදݱɹɹΛ͏ • ɹɹ શ͘Θͳ͍ɹ • ɹɹʹର͢Δskip-ngramͰͷ༗༻ੑ࣮ݧ • https://arxiv.org/abs/1611.01462 • ɹɹ͏͜ͱͰΑ͍݁Ռ͕ಘΒΕΔͱ͍͏ใࠂ
• https://nlp.stanford.edu/projects/glove/ • word2vecͱࣅ͍ͯΔͭͷख๏ Win Win Wout Wout