Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Autoencoding Variational Inference for Topic Modelsの解説スライド

Autoencoding Variational Inference for Topic Modelsの解説スライド

ICLR2017読み会のスライド https://connpass.com/event/57631/

2ab3dc02a9448f246bab64174b19dc1e?s=128

Kento Nozawa

June 15, 2017
Tweet

More Decks by Kento Nozawa

Other Decks in Research

Transcript

  1. Autoencoding Variational Inference For Topic Models Akash Srivastava and Charles

    Sutton ICLR2017ಡΈձ ಡΉਓ: @nzw0301
  2. ֓ཁ 1. Latent Dirichlet Allocation (LDA) ΛNeural Variational Inference (NVI)

    Ͱ • Dirichlet ෼෍ͷ reparameterization trick 2. ৽ϞσϧͷఏҊ 3. ѱ͍ہॴղʹϋϚΔͷΛ༧๷ 2
  3. ࣄલ஌ࣝɿLDAͱVAEͷ֓ཁ 3

  4. LDA จॻͷ֬཰తੜ੒Ϟσϧ [Blei et al., 2003]    

     จॻͷτϐοΫ෼෍Q [cВ ݚڀ ՝୊ ஌ࣝ Պֶऀ ʜ ػցֶश ਓ޻஌ೳ Ϟσϧ αϯϓϧ ʜ τϐοΫͷ୯ޠ෼෍ p(w|β) Ќ Ќ ػցֶश ػցֶशݚڀ ਓ޻஌ೳ՝୊ Ϟσϧ-%" Պֶ਺ֶण࢘ ίʔύε 4
  5. VAE: Encoder • NNΛ࢖ͬͨੜ੒Ϟσϧ • Encoder: • σʔλ͔Β֬཰෼෍ͷύϥϝʔλ΁ͷม׵ • ֬཰෼෍͸જࡏม਺Λੜ੒

    • Decoder: • જࡏม਺͔Βσʔλੜ੒ • Reparameterization trick • BPʹαϯϓϧΛؚΊΔ޻෉ • ඪ४ਖ਼ن෼෍ͷαϯϓϧͱ෼෍ͷ
 ύϥϝʔλ͔ΒαϯϓϧΛߏ੒ 5
  6. VAE: Decoder • NNΛ࢖ͬͨੜ੒Ϟσϧ • Encoder: • σʔλ͔Β֬཰෼෍ͷύϥϝʔλ΁ͷม׵ • ֬཰෼෍͸જࡏม਺Λੜ੒

    • Decoder: • જࡏม਺͔Βσʔλੜ੒ • Reparameterization trick • BPʹαϯϓϧΛؚΊΔ޻෉ • ඪ४ਖ਼ن෼෍ͷαϯϓϧͱ෼෍ͷ
 ύϥϝʔλ͔ΒαϯϓϧΛߏ੒ 6
  7. VAE: Reparameterization trick • NNΛ࢖ͬͨੜ੒Ϟσϧ • Encoder: • σʔλ͔Β֬཰෼෍ͷύϥϝʔλ΁ͷม׵ •

    ֬཰෼෍͸જࡏม਺Λੜ੒ • Decoder: • જࡏม਺͔Βσʔλੜ੒ • Reparameterization trick • BPʹαϯϓϧΛؚΊΔ޻෉ • ඪ४ਖ਼ن෼෍ͷαϯϓϧͱ෼෍ͷ
 ύϥϝʔλ͔ΒαϯϓϧΛߏ੒ 7
  8. VAE: ϩεؔ਺ 8 L (⇥) = D X d=1 (

    1 2 ⇣ tr (⌃0) + µT 0 µ0 K log | ⌃0 | ⌘ + E ✏⇠N (0,1) ⇣ log p xd |f ( µ0 + ⌃ 1/2 0 ✏ ) ⌘ ) (Ⅰ) ࣄલ෼෍ͱͷKLμΠόʔδΣϯε (Ⅱ) ର਺໬౓ ࣜશମ: Evidence Lower Bound (I) (Ⅱ)
  9. ຊ୊ 9

  10. Reparameterization trick for Dirichlet Distribution • LDAͷθ: Dirichlet෼෍͔Βαϯϓϧ • Scale

    family DistributionͰͳ͍ͨΊɼߏ੒Ͱ͖ͳ͍ 10      จॻͷτϐοΫ෼෍Q [cВ
  11. Reparameterization trick for Dirichlet Distribution • LDAͷθ: Dirichlet෼෍͔Βαϯϓϧ • Scale

    family DistributionͰͳ͍ͨΊɼߏ੒Ͱ͖ͳ͍ • Laplace approximation • ਖ਼ن෼෍ͷαϯϓϧʹsoftmaxؔ਺Λద༻ͯ͠୅༻ • ࣄલ෼෍ͷύϥϝʔλɿ µk = log( ↵k) 1 K K X i=1 log ↵i ⌃k,k = 1 ↵k (1 2 K ) + 1 K2 K X i=1 1 ↵k 11
  12. ωοτϫʔΫͱϩεؔ਺ 12 X encoder µ( X ) ⌃ ( X

    ) KL {N( z ; µ( X ) , ⌃ ( X ))||N( z ; µ1, ⌃1)} ✏ ⇠ N(✏; 0, I ) + decoder: f ( Z ) loss ( x, f ( Z )) • σ: softmaxؔ਺ • β : DecoderͷॏΈʢunnormalizedʣ • σ(β): ୯ޠͷDiriclet෼෍͔ΒͷαϯϓϧʹରԠ L ( ⇥ ) = D X d=1 ( 1 2 ⇣ tr ( ⌃ 1 1 ⌃0) + ( µ1 µ0) T ⌃ 1 1 ( µ1 µ0) K + log |⌃1 | |⌃0 | ⌘ + E ✏⇠N (0,1) wt d log ⇣ ( µ0 + ⌃1/2 0 ✏ ) ⌘ !) θ ස౓ϕΫτϧ
  13. prodLDA: ఏҊϞσϧ • Products of Experts • βͱθͷੵʹsoftmaxؔ਺ 13 L

    ( ⇥ ) = D X d=1 ( 1 2 ⇣ tr ( ⌃ 1 1 ⌃0) + ( µ1 µ0) T ⌃ 1 1 ( µ1 µ0) K + log |⌃1 | |⌃0 | ⌘ + E ✏⇠N (0,1) wt d log ⇣ ( µ0 + ⌃1/2 0 ✏ ) ⌘ !) ( ✓)
  14. ࠷దԽͱωοτϫʔΫͷ޻෉ NVIͷ໰୊఺ɿ ֶशͷॳظஈ֊Ͱlocal optimumʹߦ͖΍͍͢ • AdamͷύϥϝʔλΛௐ੔ • ηͱβ1 ͷ஋ͷߴΊʹઃఆ •

    Batch NormalizationͱDropoutΛ࢖༻ 14
  15. ࣮ݧ 1. CoherenceͱPerplexity • ޙड़ 2. ֶश཰ͱࣄલ෼෍Λม͑ͨͱ͖ͷޮՌ • ߴֶ͍श཰ &

    Dirichlet෼෍͕ϕλʔ 3. ςετσʔλʹର͢Δ࠷దԽͷ༗ແ • ͠ͳͯ͘΋͍͍ 4. p(w|β)ͷϦετ • লུ 15
  16. Coherence 16 ද͸౰࿦จ͔ΒҾ༻ • LDA VAE: ఏҊਪ࿦๏ • prodLDA: ఏҊਪ࿦๏+ఏҊϞσϧ

    • LDA DMFVI: Online Mean-Field Variational Inference • NVDM: VAEϕʔεͷจॻϞσϦϯά දͷ஋: 40ճ࣮ߦͯ͠ࢉग़
  17. Perplexity 17 ද͸౰࿦จ͔ΒҾ༻

  18. ϨϏϡʔ: ؾʹͳͬͨ΋ͷΛ͍͔ͭ͘ Q1. NVDMͰ΋adamͷֶश཰Λม͑ͨํ͕ެฏ A1. ࿦จʹ൓ө Q2. ϋΠύʔύϥϝʔλ͸࠷దԽ͔ͨ͠ A2. ൺֱख๏͸͍ͯ͠ΔɼఏҊख๏͸BO

    Rating: 6-7-6-5 18
  19. ͦͷଞ • ஶऀ࣮૷: TensorFlow • NVDMͷஶऀΒͷ৽Ϟσϧ͕ICML2017ʹ࠾࿥ 19