Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
トピックモデルによる分散表現獲得手法の提案
Search
Kento Nozawa
March 09, 2016
Research
230
1
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
トピックモデルによる分散表現獲得手法の提案
2016年の言語処理学会の発表スライド
Kento Nozawa
March 09, 2016
More Decks by Kento Nozawa
See All by Kento Nozawa
[最先端NLP勉強会2026] Checklists Are Better Than Reward Models For Aligning Language Models
nzw0301
1
350
Analysis on Negative Sample Size in Contrastive Unsupervised Representation Learning
nzw0301
0
250
[IJCAI-ECAI 2022] Evaluation Methods for Representation Learning: A Survey
nzw0301
0
740
[NeurIPS Japan meetup 2021 talk] Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
nzw0301
0
290
[IBIS2021] 対照的自己教師付き表現学習おける負例数の解析
nzw0301
0
240
Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
nzw0301
0
610
Introduction of PAC-Bayes and its Application for Contrastive Unsupervised Representation Learning
nzw0301
2
940
NLP Tutorial; word representation learning
nzw0301
0
280
Analyzing Centralities of Embedded Nodes
nzw0301
0
240
Other Decks in Research
See All in Research
COMETAを用いたデータ民主化運動の歴史
sazimai
0
300
[webinaire du Ciirpo] Agnelages plein air : des éleveurs partagent leurs pratiques
institutdelelevage
PRO
0
320
山中市政が招いた組織崩壊:よこはま安心ボックスを例に
sishi
0
370
SigLIP2 Explanation
auranightsky
0
130
幾何学的深層学習入門 ~3次元構造と対称性を学習するニューラルネットワーク~
taniai
3
1k
Evaluating LLM Reliability Across Facts, Evidence, and Cultures
yukiar
0
230
Santé et fertilité des sols : ce que nous disent les pratiques dans les systèmes de polyculture-élevage
institutdelelevage
PRO
0
220
PHTalks Bengaluru - SSRF When All Else Fails
dk999
0
1.2k
Physical AIでモデリングはどう変わるか/How Physical AI Will Change Modeling
stktu
0
340
[SNLP2026] Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
wataruuuuu
0
370
モデル評価を効率的にする方法
ftakahashi
0
240
Pesée et pointage par imagerie 3D : PHENO3D entre en phase de test terrains
institutdelelevage
PRO
0
110
Featured
See All Featured
Optimizing for Happiness
mojombo
378
71k
What’s in a name? Adding method to the madness
productmarketing
PRO
24
4.2k
HTML-Aware ERB: The Path to Reactive Rendering @ RubyCon 2026, Rimini, Italy
marcoroth
5
910
The innovator’s Mindset - Leading Through an Era of Exponential Change - McGill University 2025
jdejongh
PRO
1
390
From π to Pie charts
rasagy
1
440
Gemini Prompt Engineering: Practical Techniques for Tangible AI Outcomes
mfonobong
2
630
RailsConf 2023
tenderlove
30
1.6k
Are puppies a ranking factor?
jonoalderson
2
4.2k
Agile Leadership in an Agile Organization
kimpetersen
PRO
0
310
Reality Check: Gamification 10 Years Later
codingconduct
0
2.4k
Building the Perfect Custom Keyboard
takai
2
950
Building a Scalable Design System with Sketch
lauravandoore
464
34k
Transcript
τϐοΫϞσϧʹΑΔ ࢄදݱͷ֫ಘख๏ͷఏҊ ݈ਓ एྛ ܒ (ஜେֶ)
୯ޠͷࢄදݱ • ୯ޠΛີͳϕΫτϧۭؒͷ1ͱͯ͠දݱ • ୯ޠؒͷԋࢉ͕Մೳʢking - man + woman =
queenʣ ࣍ݩΛಛ͚Δ୯ޠΛ֫ಘͰ͖ΕදݱΑΓϦονʹ 2 ਤ1ɿ2࣍ݩۭؒʹ͓͚Δࢄදݱͷྫ
طଘݚڀ ࢄදݱͷ֫ಘϞσϧ • χϡʔϥϧωοτϫʔΫʹΑΔ༧ଌϞσϧ • ڞىߦྻͱߦྻղʹجͮ͘Ϟσϧ ࢄදݱͷղऍ [Faruqui+, 2015] •
ߴ࣍ݩͰૄϕΫτϧʹม • ಛྔͱͯ͠ղऍ͍͢͠ 3
ݚڀత 1. τϐοΫϞσϧʹΑΔ୯ޠͷࢄදݱͷ֫ಘ • ୯ޠΛ֬ͱϕΫτϧͱͯ͠ѻ͏ 2. ୯ޠϕΫτϧͱτϐοΫͷؔੑͷ֬ೝ • ୯ޠΛ֬Ͱදݱ͍ͯ͠ΔͨΊղऍ͍͢͠ 4
ఏҊख๏ʹ͓͚Δࢄදݱ֫ಘͷྲྀΕ 1. จॻσʔλͷಡΈࠐΈ 2. ୯ޠλΠϓΛڞى͢Δ୯ޠͷଟॏू߹Ͱදݱ 3. 1ଟॏू߹Λ1จॻͱΈͳ͠ɼLDAΛֶश 4. ୯ޠͷτϐοΫͱτϐοΫͷ୯ޠΛग़ྗ 5
୯ޠλΠϓΛڞى͢Δ୯ޠͷଟॏू߹Ͱදݱ 1୯ޠλΠϓΛͦͷલޙn୯ޠ͔ΒͳΔଟॏू߹Ͱදݱ จॻ1: ੨ ͷ Ξοϓϧ δϡʔε ͓͍͍͠ จॻ2:
ΦϥΫϧ ͱ Ξοϓϧ ΞϝϦΧ ͷ اۀ ͩ จॻ3:[ࢁས ͷ ͿͲ͏ δϡʔε ҿΈ ͍͢] ↓ Ξοϓϧ = [੨, ͷ, δϡʔε, , ΦϥΫϧ, ͱ, , ΞϝϦΧ] ྫɿલޙ2୯ޠͷ߹ 6
୯ޠλΠϓΛڞى͢Δ୯ޠͷଟॏू߹Ͱදݱ Ծઆ͔Βࣅͨ୯ޠࣅͨଟॏू߹ʹͳΓ͍͢ จॻ1: ੨ ͷ Ξοϓϧ δϡʔε ͓͍͍͠ จॻ2:
ΦϥΫϧ ͱ Ξοϓϧ ΞϝϦΧ ͷ اۀ ͩ จॻ3:[ࢁས ͷ ͿͲ͏ δϡʔε ҿΈ ͍͢] ↓ Ξοϓϧ = [੨, ͷ, δϡʔε, , ΦϥΫϧ, ͱ, , ΞϝϦΧ] ͿͲ͏ = [ࢁས, ͷ,δϡʔε, ] ྫɿલޙ2୯ޠͷ߹ 7
Latent Dirichlet Allocation • จॻͷ֬తੜϞσϧ [Blei+, 2003] • จॻͷ୯ޠͷڞىใ͔ΒτϐοΫΛൃݟ •
τϐοΫͰ͚݅ͮΒΕΔશ୯ޠͷ֬ • จॻͷ୯ޠͰ͚݅ͮΒΕΔτϐοΫͷ֬ 8 ػցֶश ػցֶशͱɺਓ ೳʹ͓͚Δݚ ڀ՝ͷҰͭͰɺ ਓ͕ؒࣗવʹߦͬ ͍ͯΔֶशೳྗͱ ಉ༷ͷػೳΛίϯ ϐϡʔλͰ࣮ݱʜ ݚڀ ՝ ࣝ Պֶऀ ʜ ػցֶश ਓೳ Ϟσϧ αϯϓϧ ʜ จॻσʔλ τϐοΫ τϐοΫͷ֬
LDAͷֶशʢτϐοΫ3ͷ߹ʣ ଟॏू߹Ͱදݱͨ͠1୯ޠΛ1จॻͱΈͳ͠ɼLDAΛֶश • ྫɿΞοϓϧ = [੨, ͷ, δϡʔε, , ΦϥΫϧ,
ͱ, , ΞϝϦΧ] 1. ୯ޠʢଟॏू߹ʣ͝ͱͷτϐοΫͷଟ߲ • ֬ͱϕΫτϧͱͯ͠ѻ͏ ྫɿΞοϓϧ = [0.3, 0.6, 0.1], ΦϥΫϧ = [0.6, 0.1, 0.3], ͿͲ͏ = [0.1, 0.7, 0.2] 2. τϐοΫ͝ͱͷ୯ޠͷଟ߲ ྫɿτϐοΫ1 = [ιϑτΣΞ:0.3, PC:0.2, ϓϩάϥϜ:0.1, …] τϐοΫ2 = [δϡʔε:0.4,ύΠ:0.2, ϑϧʔπ:0.1, …] τϐοΫ3 = [͕:0.2, ͷ:0.2, ʹ:0.1, …] 9
࣍ݩͷղऍ • ୯ޠͷτϐοΫͷ͕ߴ͍΄ͲɼͦͷτϐοΫΛͭ ྫɿΞοϓϧ = [0.3, 0.6, 0.1] τϐοΫͷ୯ޠͷ͕ߴ͍΄ͲɼτϐοΫΛಛ͚Δ୯ޠ τϐοΫ1
= [ιϑτΣΞ:0.3, PC:0.2, ϓϩάϥϜ:0.1, …] τϐοΫ2 = [δϡʔε:0.4, ύΠ:0.2, ϑϧʔπ:0.1, …] τϐοΫ3 = [͕:0.2, ͷ:0.2, ʹ:0.1, …] Ξοϓϧδϡʔε, ύΠ, ϑϧʔπͳͲͷ୯ޠͰಛ͚ΒΕΔ ͿͲ͏ = [0.1, 0.7, 0.2]ಉ༷ͷτϐοΫͰಛ͚ΒΕΔ ͿͲ͏ɼτϐοΫ2ͷҙຯͰΞοϓϧͱྨࣅ 10
ఏҊख๏ͷ·ͱΊ • ୯ޠλΠϓΛपғn୯ޠ͔ΒͳΔଟॏू߹ͱΈͳ͢ • ୯ޠλΠϓͷτϐοΫͷ֬ΛϕΫτϧͱͯ֫͠ಘ • τϐοΫ͔Β୯ޠλΠϓΛಛ͚Δ୯ޠΛநग़Մೳ 11
࣮ݧ
ֶशσʔλͱલॲཧ ֶशσʔλ 2010ͷӳޠ൛Wikipediaͷຊจهࣄͷ1/3 ҎԼͷલॲཧͷ݁Ռ112,635୯ޠλΠϓΛ༻ લॲཧ • ස͕100Ҏ্ͷ୯ޠͷΈΛ༻ • ඇӳࣈআ •
ࣈରԠ͢Δӳ୯ޠʹม • ߴසޠͷαϒαϯϓϦϯά [Mikilov+, 2013] 13
LDAͷֶशΞϧΰϦζϜͱύϥϝʔλ ֬తมϕΠζ๏Λ༻ֶ͍ͨशΞϧΰϦζϜ • ֬తޯ๏ʹجֶͮ͘श[Mimno+, 12] • 1ճͷ෮ܭࢉͰҰ෦ͷจॻ͚ͩΛ༻͍Δ σΟϦΫϨͷύϥϝʔλ α ͷֶशͷ༗ແ࣮ݧ
• α Λֶश͢ΔͱτϐοΫͷ·ͱ·Γ͕Α͘ͳΔ 14
ࢄදݱͷධՁํ๏ ධՁํ๏ • word similarityɿॱҐ૬ؔʹΑΔධՁ • apple tree 0.2 •
apple orange 0.7 • analogyɿਖ਼ղʹΑΔධՁ • man king woman queen ൺֱख๏ CBoWͱSkip–gram [Mikolov+, 2013] 15
ൺֱ࣮ݧͷ݁Ռ • word similarityͰ0.2~0.3ͷࠩ • analogyͰ0.4~0.5ͷࠩ cos: ίαΠϯྨࣅ js: δΣϯηϯγϟϊϯμΠόʔδΣϯε
16 XPSETJNJMBSJUZ BOBMPHZ EBUBTFU 84 844 843 .&/ .5 38 (PPHMF .43 $#P8 4LJQrHSBN ఏҊ๏ DPT ఏҊ๏ KT ද1ɿൺֱ࣮ݧͷ݁Ռ
τϐοΫͷղऍɿpython • pythonͷτϐοΫͷ֬θͷߴ্͍Ґ3τϐοΫʹ • ͦΕͧΕͷτϐοΫͷ͏ͪ֬ͷߴ্͍Ґ10୯ޠ • 880ɿऄͱʮMonty Pythonʯ • 145ɿϓϩάϥϛϯά
• 732ɿߴසޠ 17 UPQJD*% В Ћ DJSDVT BSDIJWF UIF TOBLF TPGUXBSF J NPOUZ XFC JU DPCSB QSPHSBNNJOH ZPV TLFUDI EBUBCBTF CF QBMJO CBTFE IBWF MJ[BSE WJEFP B FWFOJOH TFSWFS CVU HSBJM MJOVY JG WJQFS JOUFSGBDF DBO
τϐοΫͷղऍɿbow • bowͷτϐοΫͰα<0.2͔ͭ֬θͷߴ্͍Ґ3τϐοΫʹ • ͦΕͧΕͷτϐοΫͷ͏ͪ֬ͷߴ্͍Ґ10୯ޠ • 389ɿધɼඋͳͲ • 547ɿધ •
919ɿҥମͷ෦Ґ 18 UPQJD*% В Ћ BJSDSBGU TIJQ GBDF TQFFE TIJQT XPSO XFJHIU NFSDIBOU DBQ HVO QBUSPM TIPFT CVJMU CPBUT XFBST TIJQ OBWBM UJF NBDIJOF WFTTFMT XPSF QPXFS DSFX XFBS ESJWF DBSHP TIJSU TUFBN WFTTFM TIPVMEFS
·ͱΊ LDAʹΑΔࢄදݱͷ֫ಘख๏ • ֬ΛϕΫτϧͱΈͳ͢ • ϕΫτϧͱͯ͠ѻ͏͜ͱɼຊධՁͰෆ͖ • ֬ͰՃࢉݮࢉʹ૬͢Δૢ࡞͕ඞཁ • ϕΫτϧͷ࣍ݩͱτϐοΫ͕ରԠ
• ޠٛʹ͍ۙτϐοΫͷ֫ಘ • Word Sense InductionͷԠ༻ 19