Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ゼロから作るDeep Learning 2 3章 word2vec 3.1〜3.2
Search
ota42y
May 29, 2019
Programming
1
470
ゼロから作るDeep Learning 2 3章 word2vec 3.1〜3.2
ゼロから作るDeep Learning 2 自然言語編 読書会 第5回
の資料です!
https://retrieva.connpass.com/event/131746/
ota42y
May 29, 2019
Tweet
Share
More Decks by ota42y
See All by ota42y
バックログを導入し やっぱやめた話
ota42y
0
250
PFNにある2つのKubernetes
ota42y
10
5.4k
Q&A for How to use OpenAPI3 for API developer
ota42y
0
2.5k
How to use OpenAPI3 for API developer (RubyKaigi 2019)
ota42y
5
21k
How should we face with microservices (我々はマイクロサービスとどう向き合うべきか)
ota42y
20
4.7k
DeepLearningの本番環境にSageMakerを利用してる話
ota42y
1
6.3k
検索結果の良さを計測して定量的に改善していく
ota42y
3
2.4k
Flutterを広めるために技術同人誌を作った話
ota42y
1
1.6k
何も考えずにCIや継続的デリバリーしたら辛くなった話.pdf
ota42y
0
2.9k
Other Decks in Programming
See All in Programming
menu基盤チームによるGoogle Cloudの活用事例~Application Integration, Cloud Tasks編~
yoshifumi_ishikura
0
110
Spatial Rendering for Apple Vision Pro
warrenm
0
110
PHPUnitしか使ってこなかった 一般PHPerがPestに乗り換えた実録
mashirou1234
0
220
良いユニットテストを書こう
mototakatsu
8
2.6k
Асинхронность неизбежна: как мы проектировали сервис уведомлений
lamodatech
0
820
php-conference-japan-2024
tasuku43
0
320
テストケースの名前はどうつけるべきか?
orgachem
PRO
0
140
PHPで作るWebSocketサーバー ~リアクティブなアプリケーションを知るために~ / WebSocket Server in PHP - To know reactive applications
seike460
PRO
2
470
Amazon S3 NYJavaSIG 2024-12-12
sullis
0
100
たのしいparse.y
ydah
3
120
Effective Signals in Angular 19+: Rules and Helpers @ngbe2024
manfredsteyer
PRO
0
140
Effective Signals in Angular 19+: Rules and Helpers
manfredsteyer
PRO
0
110
Featured
See All Featured
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
132
33k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
38
1.9k
Designing on Purpose - Digital PM Summit 2013
jponch
116
7k
Building Applications with DynamoDB
mza
91
6.1k
Rebuilding a faster, lazier Slack
samanthasiow
79
8.7k
RailsConf 2023
tenderlove
29
940
The Power of CSS Pseudo Elements
geoffreycrofte
73
5.4k
KATA
mclloyd
29
14k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
2
290
Imperfection Machines: The Place of Print at Facebook
scottboms
266
13k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
Mobile First: as difficult as doing things right
swwweet
222
9k
Transcript
θϩ͔Β࡞ΔDeep Learning 2 ̏ষ word2vec 3.1ʙ3.2 ota42y θϩ͔Β࡞ΔDeep Learning
2 ࣗવݴޠฤ ಡॻձ ୈ5ճ
͜ͷষͰΔ͜ͱ • word2vecΛ࣮͢Δ • ਪϕʔεͰ୯ޠΛϕΫτϧͰද͢ํ๏ • γϯϓϧ͕ͩແବଟ͍࣮ • ࣍ͷষͰରԠ
3.1 ਪϕʔεͷख๏ͱ χϡʔϥϧωοτϫʔΫ
ਪϕʔεͷϕΫτϧԽ • ୯ޠΛϕΫτϧʹ͢Δ̎ͭͷख๏ • Χϯτϕʔεʢ̎ষʣ • ਪϕʔεʢ̏ষʣ • ͲͪΒԾઃΛϕʔεʹͯ͠Δ͕Ξϓϩʔνશ͘ผ •
Ծઃɿ୯ޠͷҙຯपғͷ୯ޠ͔Βܗ͞ΕΔ (p.67)
3.1.1ɹΧϯτϕʔεͷख๏ͷ • Χϯτϕʔεपғͷ୯ޠͷසΛܭࢉ͢Δ • ޠኮ͕nͩͱn*nͷڊେͳڞىߦྻ͕ඞཁʹͳΔ • ࣍ݩݮͷͨΊͷSVDO(n^3)ͷܭࢉྔɺ͍
ਪϕʔεͷར • Χϯτϕʔείʔύεશମͷ౷ܭσʔλΛҰؾʹར༻͢Δ • ਪϕʔε(χϡʔϥϧωοτ)ίʔύεͷҰ෦Ͱֶश͢Δ • GPUͷฒྻܭࢉฉ͘ • খ͚ʹͰ͖ɺߴʹฒྻॲཧͰ͖ΔͷͰڊେσʔλͰରԠͰ͖Δ •
ଞʹັྗతͳ͕͋Δ(Β͍͠ɺৄ͘͠3.5.3)
3.1.2ɹਪϕʔεͷख๏ͷ֓ཁ
पғͷ୯ޠ͔Β୯ޠΛʮਪʯ͢Δ • `?`ʹԿ͕ೖΔ͔Λલޙ͔Βਪ • ίϯςΩετ͔ΒλʔήοτΛਪ • ίϯςΩετɿपғͷ୯ޠ(you, goodby) • λʔήοτɿରͷ୯ޠ(`?`)
ਪ݁Ռ • ֤୯ޠ͕ͦ͜ʹݱΕΔ֬Λग़ྗ • ίϯςΩετΛϞσϧʹ༩͑Δͱ୯ޠͷ͕֬ಘΒΕΔ
3.1.3 χϡʔϥϧωοτϫʔΫʹ͓͚Δ୯ ޠͷॲཧํ๏ • χϡʔϥϧωοτϫʔΫ(NN)ͷೖྗݻఆϕΫτϧ • ୯ޠΛͦͷ··ೖΕΔͷ͍͠ • ୯ޠΛone-hotදݱ(one-hotϕΫτϧ)ʹม͢Δ
one-hotදݱ • ޠኮͷ͞Λ࣋ͪɺ୯ޠIDͱ֘͢Δ෦͕1ɺͦΕҎ֎͕0 ͷϕΫτϧ • ͯ͢ͷ୯ޠΛಉ͡͞ͷϕΫτϧͱͯ͠දݱ
one-hotදݱ • શ݁߹Ͱม͢ΔͳΒ؆୯(ྫதؒ=3)
αϯϓϧίʔυ(p.99) • np.dot(c, W)୯ޠʹରԠ͢ΔॏΈΛऔΓग़ͯ͠Δ͚ͩ • W[0]ͷσʔλΛऔΓग़ͯ͠Δ͚ͩ • ແବͬΆ͍͕࣍ͷষͰ࣏͢Β͍͠
ϨΠϠදݱ • MatMulϨΠϠ(p.30)Ͱಉ͜͡ͱ͕Ͱ͖Δ • np.dot͢Δ͚ͩͷϨΠϠͳͷͰ
3.2ɹγϯϓϧͳword2vec
word2vecΛ࣮͢Δ • word2vecͰΘΕΔϞσϧCROWϞσϧͱskip-gramϞσϧ • "word2vec"͕͜ΕΒͷϞσϧΛࢦ͢߹͋Δ • ຊདྷͷҙຯͱζϨͯΔ
3.2.1 CBOWϞσϧͷਪॲཧ • ίϯςΩετ͔ΒλʔήοτΛਪଌ͢ΔNN • ίϯςΩετʹपғͷ୯ޠ • λʔήοτʹରͷ୯ޠ
୯ޠͷࢄදݱ • CBOWϞσϧΛ܇࿅͢Δ͜ͱͰ୯ޠͷࢄදݱΛಘΒΕΔ • Ϟσϧͷύϥϝʔλ͕ࢄදݱʹରԠ͢Δ
CBOWϞσϧͷશମ૾ • ίϯςΩετʹ̎ɺӅΕʹ̏ͷ߹
CBOWϞσϧͷશମ૾ • ೖྗෳݸͷone-hotදݱͷ୯ޠ • ग़ྗ֤୯ޠͷείΞ • softmaxΛ͏ͱ͕֬ಘΒΕΔ • தؒೖྗ͔Βͷͷฏۉ
• ࢄදݱͷਖ਼ମ • [$ W_{in}]7*3ͷॏΈ • ͜Ε͕୯ޠͷࢄදݱ • ֶशʹΑͬͯྑ͍ࢄදݱʹ͍ͯ͘͠
CBOWϞσϧͷϨΠϠදݱ
CBOWϞσϧͷϨΠϠදݱ • ̎ͭͷMatMulϨΠϠ • ୯ޠʹରԠ͢ΔॏΈΛऔΓग़ͭ͢(P.99) • ̎ͭͷฏۉΛऔΔ(=ͯ͠0.5Λ͔͚Δ) • scoreͷશ݁߹ •
׆ੑԽؔແ͍ͷͰΘΓͱγϯϓϧ
3.2.2 CBOWϞσϧͷֶश • χϡʔϥϧωοτϫʔΫͷηΦϦʔ௨Γ • CBOWଞΫϥεྨΛ͢ΔNN • Ϋϥεʹone-hotͰද͞Εͨ୯ޠ • είΞ͔Β֬ΛٻΊͯɺਖ਼ղͱͷࠩΛֶश͢Δ
• Softmaxؔʹ͔͚ͯ֬ʹ͢Δ • ڭࢣϥϕϧ͔ΒަࠩΤϯτϩϐʔޡࠩΛٻΊΔ
ϨΠϠදݱ • Softmax with lossΛ͚Ճ͑Δ
ίʔυϦʔσΟϯά • ch03/cbow_predict.py • https://github.com/oreilly-japan/deep-learning-from- scratch-2/blob/master/ch03/cbow_predict.py
3.2.3 word2vecͷॏΈͱࢄදݱ • ͱɹɹͷҧ͍ • ྆ํͱ୯ޠͷҙຯ͕Τϯίʔυ͞Ε͍ͯΔ • ܗঢ়͕ҧ͏ • ɹɹ7x3
• ɹɹ3x7 Win Wout Win Wout
ࢄදݱɹɹΛ͏ • ɹɹ શ͘Θͳ͍ɹ • ɹɹʹର͢Δskip-ngramͰͷ༗༻ੑ࣮ݧ • https://arxiv.org/abs/1611.01462 • ɹɹ͏͜ͱͰΑ͍݁Ռ͕ಘΒΕΔͱ͍͏ใࠂ
• https://nlp.stanford.edu/projects/glove/ • word2vecͱࣅ͍ͯΔͭͷख๏ Win Win Wout Wout