Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Autoencoding Variational Inference for Topic Modelsの解説スライド
Kento Nozawa
June 15, 2017
Research
3
29k
Autoencoding Variational Inference for Topic Modelsの解説スライド
ICLR2017読み会のスライド
https://connpass.com/event/57631/
Kento Nozawa
June 15, 2017
Tweet
Share
More Decks by Kento Nozawa
See All by Kento Nozawa
[IJCAI-ECAI 2022] Evaluation Methods for Representation Learning: A Survey
nzw0301
0
9
[NeurIPS Japan meetup 2021 talk] Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
nzw0301
0
52
[IBIS2021] 対照的自己教師付き表現学習おける負例数の解析
nzw0301
0
48
Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
nzw0301
0
140
Introduction of PAC-Bayes and its Application for Contrastive Unsupervised Representation Learning
nzw0301
2
510
NLP Tutorial; word representation learning
nzw0301
0
83
Analyzing Centralities of Embedded Nodes
nzw0301
0
87
Paper Reading: Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics
nzw0301
2
790
Deep Learning book 18. Confront the Partition Function
nzw0301
0
260
Other Decks in Research
See All in Research
第14回チャンピオンズミーティング・ジェミニ杯ラウンド1集計 / Umamusume Gemini 2022 Round1
kitachan_black
0
720
Stack-chanで始めるROS音声対話ロボット
yoshipon
1
230
Collaborative editing through a databases lens
ept
0
420
第11回チャンピオンズミーティング・ピスケス杯ラウンド2集計 / Umamusume Pisces 2022 Round2
kitachan_black
0
910
素人発想玄人実行2.0
hf149
0
1.2k
論述リビジョンのためのメタ評価基盤
chemical_tree
0
150
ゼロからのスタートアップ立ち上げにおけるリサーチ事例 by @takejune
takejune
2
790
PLDI '21論文読み会: Provable Repair of Deep Neural Networks
ideininc
0
720
再帰化への認知的転回/the-turn-to-recursive-system
monochromegane
0
160
Celebrate UTIG: Staff and Student Awards 2022
utig
0
190
[IR Reading 2022春 論文紹介] Personalized Transfer of User Preferences for Cross-domain Recommendation (WSDM 2022) /IR-Reading-2022-spring
koheishinden
PRO
0
130
大学研究者による事業提案制度(大学提案) 募集概要
tocho_zaiseika
0
3.7k
Featured
See All Featured
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
498
130k
Making the Leap to Tech Lead
cromwellryan
113
7.4k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
236
1M
Fashionably flexible responsive web design (full day workshop)
malarkey
396
62k
Embracing the Ebb and Flow
colly
73
3.4k
VelocityConf: Rendering Performance Case Studies
addyosmani
316
22k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
37
3.3k
Learning to Love Humans: Emotional Interface Design
aarron
261
37k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
349
27k
Pencils Down: Stop Designing & Start Developing
hursman
112
9.8k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
119
28k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
5
510
Transcript
Autoencoding Variational Inference For Topic Models Akash Srivastava and Charles
Sutton ICLR2017ಡΈձ ಡΉਓ: @nzw0301
֓ཁ 1. Latent Dirichlet Allocation (LDA) ΛNeural Variational Inference (NVI)
Ͱ • Dirichlet ͷ reparameterization trick 2. ৽ϞσϧͷఏҊ 3. ѱ͍ہॴղʹϋϚΔͷΛ༧ 2
ࣄલࣝɿLDAͱVAEͷ֓ཁ 3
LDA จॻͷ֬తੜϞσϧ [Blei et al., 2003]
จॻͷτϐοΫQ [cВ ݚڀ ՝ ࣝ Պֶऀ ʜ ػցֶश ਓೳ Ϟσϧ αϯϓϧ ʜ τϐοΫͷ୯ޠ p(w|β) Ќ Ќ ػցֶश ػցֶशݚڀ ਓೳ՝ Ϟσϧ-%" Պֶֶण࢘ ίʔύε 4
VAE: Encoder • NNΛͬͨੜϞσϧ • Encoder: • σʔλ͔Β֬ͷύϥϝʔλͷม • ֬જࡏมΛੜ
• Decoder: • જࡏม͔Βσʔλੜ • Reparameterization trick • BPʹαϯϓϧΛؚΊΔ • ඪ४ਖ਼نͷαϯϓϧͱͷ ύϥϝʔλ͔ΒαϯϓϧΛߏ 5
VAE: Decoder • NNΛͬͨੜϞσϧ • Encoder: • σʔλ͔Β֬ͷύϥϝʔλͷม • ֬જࡏมΛੜ
• Decoder: • જࡏม͔Βσʔλੜ • Reparameterization trick • BPʹαϯϓϧΛؚΊΔ • ඪ४ਖ਼نͷαϯϓϧͱͷ ύϥϝʔλ͔ΒαϯϓϧΛߏ 6
VAE: Reparameterization trick • NNΛͬͨੜϞσϧ • Encoder: • σʔλ͔Β֬ͷύϥϝʔλͷม •
֬જࡏมΛੜ • Decoder: • જࡏม͔Βσʔλੜ • Reparameterization trick • BPʹαϯϓϧΛؚΊΔ • ඪ४ਖ਼نͷαϯϓϧͱͷ ύϥϝʔλ͔ΒαϯϓϧΛߏ 7
VAE: ϩεؔ 8 L (⇥) = D X d=1 (
1 2 ⇣ tr (⌃0) + µT 0 µ0 K log | ⌃0 | ⌘ + E ✏⇠N (0,1) ⇣ log p xd |f ( µ0 + ⌃ 1/2 0 ✏ ) ⌘ ) (Ⅰ) ࣄલͱͷKLμΠόʔδΣϯε (Ⅱ) ର ࣜશମ: Evidence Lower Bound (I) (Ⅱ)
ຊ 9
Reparameterization trick for Dirichlet Distribution • LDAͷθ: Dirichlet͔Βαϯϓϧ • Scale
family DistributionͰͳ͍ͨΊɼߏͰ͖ͳ͍ 10 จॻͷτϐοΫQ [cВ
Reparameterization trick for Dirichlet Distribution • LDAͷθ: Dirichlet͔Βαϯϓϧ • Scale
family DistributionͰͳ͍ͨΊɼߏͰ͖ͳ͍ • Laplace approximation • ਖ਼نͷαϯϓϧʹsoftmaxؔΛద༻ͯ͠༻ • ࣄલͷύϥϝʔλɿ µk = log( ↵k) 1 K K X i=1 log ↵i ⌃k,k = 1 ↵k (1 2 K ) + 1 K2 K X i=1 1 ↵k 11
ωοτϫʔΫͱϩεؔ 12 X encoder µ( X ) ⌃ ( X
) KL {N( z ; µ( X ) , ⌃ ( X ))||N( z ; µ1, ⌃1)} ✏ ⇠ N(✏; 0, I ) + decoder: f ( Z ) loss ( x, f ( Z )) • σ: softmaxؔ • β : DecoderͷॏΈʢunnormalizedʣ • σ(β): ୯ޠͷDiriclet͔ΒͷαϯϓϧʹରԠ L ( ⇥ ) = D X d=1 ( 1 2 ⇣ tr ( ⌃ 1 1 ⌃0) + ( µ1 µ0) T ⌃ 1 1 ( µ1 µ0) K + log |⌃1 | |⌃0 | ⌘ + E ✏⇠N (0,1) wt d log ⇣ ( µ0 + ⌃1/2 0 ✏ ) ⌘ !) θ සϕΫτϧ
prodLDA: ఏҊϞσϧ • Products of Experts • βͱθͷੵʹsoftmaxؔ 13 L
( ⇥ ) = D X d=1 ( 1 2 ⇣ tr ( ⌃ 1 1 ⌃0) + ( µ1 µ0) T ⌃ 1 1 ( µ1 µ0) K + log |⌃1 | |⌃0 | ⌘ + E ✏⇠N (0,1) wt d log ⇣ ( µ0 + ⌃1/2 0 ✏ ) ⌘ !) ( ✓)
࠷దԽͱωοτϫʔΫͷ NVIͷɿ ֶशͷॳظஈ֊Ͱlocal optimumʹߦ͖͍͢ • AdamͷύϥϝʔλΛௐ • ηͱβ1 ͷͷߴΊʹઃఆ •
Batch NormalizationͱDropoutΛ༻ 14
࣮ݧ 1. CoherenceͱPerplexity • ޙड़ 2. ֶशͱࣄલΛม͑ͨͱ͖ͷޮՌ • ߴֶ͍श &
Dirichlet͕ϕλʔ 3. ςετσʔλʹର͢Δ࠷దԽͷ༗ແ • ͠ͳ͍͍ͯ͘ 4. p(w|β)ͷϦετ • লུ 15
Coherence 16 දจ͔ΒҾ༻ • LDA VAE: ఏҊਪ๏ • prodLDA: ఏҊਪ๏+ఏҊϞσϧ
• LDA DMFVI: Online Mean-Field Variational Inference • NVDM: VAEϕʔεͷจॻϞσϦϯά දͷ: 40ճ࣮ߦͯ͠ࢉग़
Perplexity 17 දจ͔ΒҾ༻
ϨϏϡʔ: ؾʹͳͬͨͷΛ͍͔ͭ͘ Q1. NVDMͰadamͷֶशΛม͑ͨํ͕ެฏ A1. จʹө Q2. ϋΠύʔύϥϝʔλ࠷దԽ͔ͨ͠ A2. ൺֱख๏͍ͯ͠ΔɼఏҊख๏BO
Rating: 6-7-6-5 18
ͦͷଞ • ஶऀ࣮: TensorFlow • NVDMͷஶऀΒͷ৽Ϟσϧ͕ICML2017ʹ࠾ 19