トピックモデル - AS^2 LT

トピックモデル - AS^2 LT

At BrainPad, Inc.

C6b97a47d5406cfdef50a5c755751c16?s=128

Sorami Hisamoto

April 17, 2015
Tweet

Transcript

  1. τϐοΫϞσϧ Sorami Hisamoto AS^2 LT April 18, 2015

  2. ๏ Modeling latent “topics” of each data. ๏ Originally a

    method for text,
 but not limited to text. What is topic modeling? 2 Figure from [Blei+ 2003] Data e.g. document Topics e.g. word distribution
  3. http://developer.smartnews.com/blog/2013/08/19/lda-based-channel-categorization-in-smartnews/

  4. http://aial.shiroyagi.co.jp/2014/12/τϐοΫϞσϧʹجͮ͘ଟ༷ੑͷఆྔԽ

  5. http://aial.shiroyagi.co.jp/2014/12/τϐοΫϞσϧʹجͮ͘ଟ༷ੑͷఆྔԽ

  6. http://smrmkt.hatenablog.jp/entry/2014/12/25/205630

  7. http://smrmkt.hatenablog.jp/entry/2014/12/25/205630 a-mp.jp/article/568

  8. http://smrmkt.hatenablog.jp/entry/2014/12/25/205630

  9. http://smrmkt.hatenablog.jp/entry/2014/12/25/205630

  10. http://smrmkt.hatenablog.jp/entry/2014/12/25/205630

  11. http://smrmkt.hatenablog.jp/entry/2014/12/25/205630

  12. http://mrorii.github.io/blog/2013/12/27/analyzing-dazai-osamu-literature-using-topic-models/

  13. http://mrorii.github.io/blog/2013/12/27/analyzing-dazai-osamu-literature-using-topic-models/

  14. http://mrorii.github.io/blog/2013/12/27/analyzing-dazai-osamu-literature-using-topic-models/

  15. ๏ Matrix Decompositions: LSI, SVD, … ๏ 1999: pLSI ๏

    2003: LDA
 Same method independently found in population genetics [Pritchard+ 200] ๏ 2003-: Extensions of LDA ๏ 2007-: Scalable algorithms 7 History of the topic models
  16. latent Dirichlet allocation (LDA) [Blei+ 2003] 8 ๏ “Document” is

    a set of “Words”. ๏ “Document” consists of multiple “Topics”. ๏ “Topic” is a distribution over a vocabulary (all possible words). ๏ “Words” are generated by “Topics”.
  17. latent Dirichlet allocation (LDA) [Blei+ 2003] 8 Two-stage generation process

    for each document 1. Randomly choose a distribution over topics. 2. For each word in the document a) Randomly choose a topic from the distribution over topic in step #1. b) Randomly choose a word from the corresponding topic. ๏ “Document” is a set of “Words”. ๏ “Document” consists of multiple “Topics”. ๏ “Topic” is a distribution over a vocabulary (all possible words). ๏ “Words” are generated by “Topics”.
  18. latent Dirichlet allocation (LDA) [Blei+ 2003] 8 Two-stage generation process

    for each document 1. Randomly choose a distribution over topics. 2. For each word in the document a) Randomly choose a topic from the distribution over topic in step #1. b) Randomly choose a word from the corresponding topic. ๏ “Document” is a set of “Words”. ๏ “Document” consists of multiple “Topics”. ๏ “Topic” is a distribution over a vocabulary (all possible words). ๏ “Words” are generated by “Topics”.
  19. latent Dirichlet allocation (LDA) [Blei+ 2003] 8 Two-stage generation process

    for each document 1. Randomly choose a distribution over topics. 2. For each word in the document a) Randomly choose a topic from the distribution over topic in step #1. b) Randomly choose a word from the corresponding topic. ๏ “Document” is a set of “Words”. ๏ “Document” consists of multiple “Topics”. ๏ “Topic” is a distribution over a vocabulary (all possible words). ๏ “Words” are generated by “Topics”.
  20. latent Dirichlet allocation (LDA) [Blei+ 2003] 8 Two-stage generation process

    for each document 1. Randomly choose a distribution over topics. 2. For each word in the document a) Randomly choose a topic from the distribution over topic in step #1. b) Randomly choose a word from the corresponding topic. ๏ “Document” is a set of “Words”. ๏ “Document” consists of multiple “Topics”. ๏ “Topic” is a distribution over a vocabulary (all possible words). ๏ “Words” are generated by “Topics”.
  21. 9 Figure from [Blei 2011]

  22. 9 Figure from [Blei 2011] Topic: distribution over vocabulary

  23. 9 Figure from [Blei 2011] Topic: distribution over vocabulary Step

    1: Choose a distribution over topics
  24. 9 Figure from [Blei 2011] Topic: distribution over vocabulary Step

    1: Choose a distribution over topics Step 2a: Choose a topic from distribution
  25. 9 Figure from [Blei 2011] Topic: distribution over vocabulary Step

    1: Choose a distribution over topics Step 2a: Choose a topic from distribution Step 2b: Choose a word from topic
  26. 10 Figures from [Blei 2011] Graphical model representation

  27. 10 Figures from [Blei 2011] topic Graphical model representation

  28. 10 Figures from [Blei 2011] topic proportion topic Graphical model

    representation
  29. 10 Figures from [Blei 2011] topic assignment topic proportion topic

    Graphical model representation
  30. 10 Figures from [Blei 2011] observed word topic assignment topic

    proportion topic Graphical model representation
  31. 10 Figures from [Blei 2011] Joint probability of hidden and

    observed variables observed word topic assignment topic proportion topic Graphical model representation
  32. 10 Figures from [Blei 2011] Joint probability of hidden and

    observed variables observed word topic assignment topic proportion topic Graphical model representation
  33. 10 Figures from [Blei 2011] Joint probability of hidden and

    observed variables observed word topic assignment topic proportion topic Graphical model representation
  34. Geometric interpretation 11 Figure from [Blei+ 2003]

  35. Geometric interpretation 11 Figure from [Blei+ 2003] Topic: in word

    simplex
  36. Geometric interpretation 11 Figure from [Blei+ 2003] Step 1: Choose

    a distribution over topics Topic: in word simplex В
  37. Geometric interpretation 11 Figure from [Blei+ 2003] Step 1: Choose

    a distribution over topics Topic: in word simplex Step 2a: Choose a topic from distribution В ;
  38. Geometric interpretation 11 Figure from [Blei+ 2003] Step 1: Choose

    a distribution over topics Step 2b: Choose a word from topic Topic: in word simplex 8 Step 2a: Choose a topic from distribution В ;
  39. Geometric interpretation 11 Figure from [Blei+ 2003] Step 1: Choose

    a distribution over topics Step 2b: Choose a word from topic Topic: in word simplex 8 Step 2a: Choose a topic from distribution В ; LDA: finding the optimal sub-simplex to represent documents.
  40. Geometric interpretation 11 Figure from [Blei+ 2003] Step 1: Choose

    a distribution over topics Step 2b: Choose a word from topic Topic: in word simplex 8 Step 2a: Choose a topic from distribution В ; LDA: finding the optimal sub-simplex to represent documents. sub-simplex
  41. “reverse” the generation process ๏ We are interested in the

    posterior distribution. ๏ latent topic structure, given the observed documents. ๏ But it is difficult … → approximate: ๏ 1. Sampling-based methods (e.g. Gibbs sampling) ๏ 2. Variational methods (e.g. variational Bayes) ๏ etc… 12
  42. “reverse” the generation process ๏ We are interested in the

    posterior distribution. ๏ latent topic structure, given the observed documents. ๏ But it is difficult … → approximate: ๏ 1. Sampling-based methods (e.g. Gibbs sampling) ๏ 2. Variational methods (e.g. variational Bayes) ๏ etc… 12
  43. “reverse” the generation process ๏ We are interested in the

    posterior distribution. ๏ latent topic structure, given the observed documents. ๏ But it is difficult … → approximate: ๏ 1. Sampling-based methods (e.g. Gibbs sampling) ๏ 2. Variational methods (e.g. variational Bayes) ๏ etc… 12
  44. “reverse” the generation process ๏ We are interested in the

    posterior distribution. ๏ latent topic structure, given the observed documents. ๏ But it is difficult … → approximate: ๏ 1. Sampling-based methods (e.g. Gibbs sampling) ๏ 2. Variational methods (e.g. variational Bayes) ๏ etc… 12
  45. ๏ Hierarchical Dirichlet Process [Teh+ 2005] ๏ Correlated Topic Models

    [Blei+ 2006] ๏ Supervised Topic Models [Blei+ 2007] ๏ Topic Models with Power-law using Pitman-Yor process [Sato+ 2010] ๏ Time-series: ๏ Dynamic Topic Models [Blei+ 2006] ๏ Continuous Time Dynamic Topic Models [Wang+ 2008] ๏ Online Multiscale Dynamic Topic Models [Iwata+ 2010] ๏ Various learning methods ๏ Various scaling algorithms ๏ Various applications ๏ … 13 Extensions of LDA
  46. ๏ Text analysis
 Papers, Blogs, Classical texts … ๏ Video

    analysis ๏ Audio analysis ๏ Bioinformatics ๏ Network analysis ๏ … 14 Applications
  47. ๏ Gensim
 Python-based, Radim Řehůřek
 ๏ Mallet
 Java-based, UMass
 ๏

    Stanford Topic Modeling Toolbox
 Java-based, Stanford
 15 Tools
  48. References (1): books 16 “τϐοΫϞσϧʹΑΔ౷ܭతજࡏҙຯղੳ” ࠤ౻Ұ੣, 2015 “τϐοΫϞσϧ (ػցֶशϓϩϑΣογϣφϧγϦʔζ) ”

    ؠా۩࣏, 2015
  49. ๏ [Blei&Lafferty 2009] Topic Models
 http://www.cs.princeton.edu/~blei/papers/BleiLafferty2009.pdf ๏ [Blei 2011] Introduction

    to Probabilistic Topic Models
 https://www.cs.princeton.edu/~blei/papers/Blei2011.pdf ๏ [Blei 2012] Review Articles: Probabilistic Topic Models
 Communications of The ACM
 http://www.cs.princeton.edu/~blei/papers/Blei2012.pdf ๏ [Blei 2012] Probabilistic Topic Models
 Machine Learning Summer School
 http://www.cs.princeton.edu/~blei/blei-mlss-2012.pdf ๏ Topic Models by David Blei (video)
 https://www.youtube.com/watch?v=DDq3OVp9dNA 17 References (2): papers, videos, and articles ๏ What is a good explanation of Latent Dirichlet Allocation? - Quora
 http://www.quora.com/What-is-a-good-explanation-of-Latent-Dirichlet-Allocation ๏ The LDA Buffet is Now Open by Matthew L. Jockers
 http://www.matthewjockers.net/2011/09/29/ ๏ [ࠤ౻ 2012] ࢲͷϒοΫϚʔΫ Latent Topic Model (જࡏతτϐοΫϞσϧ)
 http://www.ai-gakkai.or.jp/my-bookmark_vol27-no3/ ๏ [࣋ڮ&ੴࠇ 2013] ֬཰తτϐοΫϞσϧ
 ౷ܭ਺ཧݚڀॴ H24೥౓ެ։ߨ࠲
 http://www.ism.ac.jp/~daichi/lectures/ISM-2012-TopicModels-daichi.pdf ๏ Links to the Papers Related to Topic Models by Tomonori Masada
 http://tmasada.wikispaces.com/Links+to+the+Papers+Related+to+Topic+Models