Upgrade to Pro — share decks privately, control downloads, hide ads and more …

トピックモデル - AS^2 LT

トピックモデル - AS^2 LT

At BrainPad, Inc.

Sorami Hisamoto

April 17, 2015
Tweet

More Decks by Sorami Hisamoto

Other Decks in Technology

Transcript

  1. τϐοΫϞσϧ
    Sorami Hisamoto
    AS^2 LT
    April 18, 2015

    View full-size slide

  2. ๏ Modeling latent “topics” of each data.
    ๏ Originally a method for text,

    but not limited to text.
    What is topic modeling?
    2
    Figure from [Blei+ 2003]
    Data
    e.g. document
    Topics
    e.g. word distribution

    View full-size slide

  3. http://developer.smartnews.com/blog/2013/08/19/lda-based-channel-categorization-in-smartnews/

    View full-size slide

  4. http://aial.shiroyagi.co.jp/2014/12/τϐοΫϞσϧʹجͮ͘ଟ༷ੑͷఆྔԽ

    View full-size slide

  5. http://aial.shiroyagi.co.jp/2014/12/τϐοΫϞσϧʹجͮ͘ଟ༷ੑͷఆྔԽ

    View full-size slide

  6. http://smrmkt.hatenablog.jp/entry/2014/12/25/205630

    View full-size slide

  7. http://smrmkt.hatenablog.jp/entry/2014/12/25/205630
    a-mp.jp/article/568

    View full-size slide

  8. http://smrmkt.hatenablog.jp/entry/2014/12/25/205630

    View full-size slide

  9. http://smrmkt.hatenablog.jp/entry/2014/12/25/205630

    View full-size slide

  10. http://smrmkt.hatenablog.jp/entry/2014/12/25/205630

    View full-size slide

  11. http://smrmkt.hatenablog.jp/entry/2014/12/25/205630

    View full-size slide

  12. http://mrorii.github.io/blog/2013/12/27/analyzing-dazai-osamu-literature-using-topic-models/

    View full-size slide

  13. http://mrorii.github.io/blog/2013/12/27/analyzing-dazai-osamu-literature-using-topic-models/

    View full-size slide

  14. http://mrorii.github.io/blog/2013/12/27/analyzing-dazai-osamu-literature-using-topic-models/

    View full-size slide

  15. ๏ Matrix Decompositions: LSI, SVD, …
    ๏ 1999: pLSI
    ๏ 2003: LDA

    Same method independently found in population genetics [Pritchard+ 200]
    ๏ 2003-: Extensions of LDA
    ๏ 2007-: Scalable algorithms
    7
    History of the topic models

    View full-size slide

  16. latent Dirichlet allocation (LDA) [Blei+ 2003]
    8
    ๏ “Document” is a set of “Words”.
    ๏ “Document” consists of multiple “Topics”.
    ๏ “Topic” is a distribution over a vocabulary (all possible words).
    ๏ “Words” are generated by “Topics”.

    View full-size slide

  17. latent Dirichlet allocation (LDA) [Blei+ 2003]
    8
    Two-stage generation process for each document
    1. Randomly choose a distribution over topics.
    2. For each word in the document
    a) Randomly choose a topic from the distribution over topic in step #1.
    b) Randomly choose a word from the corresponding topic.
    ๏ “Document” is a set of “Words”.
    ๏ “Document” consists of multiple “Topics”.
    ๏ “Topic” is a distribution over a vocabulary (all possible words).
    ๏ “Words” are generated by “Topics”.

    View full-size slide

  18. latent Dirichlet allocation (LDA) [Blei+ 2003]
    8
    Two-stage generation process for each document
    1. Randomly choose a distribution over topics.
    2. For each word in the document
    a) Randomly choose a topic from the distribution over topic in step #1.
    b) Randomly choose a word from the corresponding topic.
    ๏ “Document” is a set of “Words”.
    ๏ “Document” consists of multiple “Topics”.
    ๏ “Topic” is a distribution over a vocabulary (all possible words).
    ๏ “Words” are generated by “Topics”.

    View full-size slide

  19. latent Dirichlet allocation (LDA) [Blei+ 2003]
    8
    Two-stage generation process for each document
    1. Randomly choose a distribution over topics.
    2. For each word in the document
    a) Randomly choose a topic from the distribution over topic in step #1.
    b) Randomly choose a word from the corresponding topic.
    ๏ “Document” is a set of “Words”.
    ๏ “Document” consists of multiple “Topics”.
    ๏ “Topic” is a distribution over a vocabulary (all possible words).
    ๏ “Words” are generated by “Topics”.

    View full-size slide

  20. latent Dirichlet allocation (LDA) [Blei+ 2003]
    8
    Two-stage generation process for each document
    1. Randomly choose a distribution over topics.
    2. For each word in the document
    a) Randomly choose a topic from the distribution over topic in step #1.
    b) Randomly choose a word from the corresponding topic.
    ๏ “Document” is a set of “Words”.
    ๏ “Document” consists of multiple “Topics”.
    ๏ “Topic” is a distribution over a vocabulary (all possible words).
    ๏ “Words” are generated by “Topics”.

    View full-size slide

  21. 9
    Figure from [Blei 2011]

    View full-size slide

  22. 9
    Figure from [Blei 2011]
    Topic:
    distribution
    over vocabulary

    View full-size slide

  23. 9
    Figure from [Blei 2011]
    Topic:
    distribution
    over vocabulary
    Step 1:
    Choose a
    distribution over topics

    View full-size slide

  24. 9
    Figure from [Blei 2011]
    Topic:
    distribution
    over vocabulary
    Step 1:
    Choose a
    distribution over topics
    Step 2a:
    Choose a topic
    from distribution

    View full-size slide

  25. 9
    Figure from [Blei 2011]
    Topic:
    distribution
    over vocabulary
    Step 1:
    Choose a
    distribution over topics
    Step 2a:
    Choose a topic
    from distribution
    Step 2b:
    Choose a word
    from topic

    View full-size slide

  26. 10
    Figures from [Blei 2011]
    Graphical model representation

    View full-size slide

  27. 10
    Figures from [Blei 2011]
    topic
    Graphical model representation

    View full-size slide

  28. 10
    Figures from [Blei 2011]
    topic proportion topic
    Graphical model representation

    View full-size slide

  29. 10
    Figures from [Blei 2011]
    topic
    assignment
    topic proportion topic
    Graphical model representation

    View full-size slide

  30. 10
    Figures from [Blei 2011]
    observed word
    topic
    assignment
    topic proportion topic
    Graphical model representation

    View full-size slide

  31. 10
    Figures from [Blei 2011]
    Joint probability of hidden and observed variables
    observed word
    topic
    assignment
    topic proportion topic
    Graphical model representation

    View full-size slide

  32. 10
    Figures from [Blei 2011]
    Joint probability of hidden and observed variables
    observed word
    topic
    assignment
    topic proportion topic
    Graphical model representation

    View full-size slide

  33. 10
    Figures from [Blei 2011]
    Joint probability of hidden and observed variables
    observed word
    topic
    assignment
    topic proportion topic
    Graphical model representation

    View full-size slide

  34. Geometric interpretation
    11
    Figure from [Blei+ 2003]

    View full-size slide

  35. Geometric interpretation
    11
    Figure from [Blei+ 2003]
    Topic:
    in word simplex

    View full-size slide

  36. Geometric interpretation
    11
    Figure from [Blei+ 2003]
    Step 1:
    Choose a
    distribution over topics
    Topic:
    in word simplex
    В

    View full-size slide

  37. Geometric interpretation
    11
    Figure from [Blei+ 2003]
    Step 1:
    Choose a
    distribution over topics
    Topic:
    in word simplex
    Step 2a:
    Choose a topic
    from distribution
    В
    ;

    View full-size slide

  38. Geometric interpretation
    11
    Figure from [Blei+ 2003]
    Step 1:
    Choose a
    distribution over topics
    Step 2b:
    Choose a word
    from topic
    Topic:
    in word simplex
    8
    Step 2a:
    Choose a topic
    from distribution
    В
    ;

    View full-size slide

  39. Geometric interpretation
    11
    Figure from [Blei+ 2003]
    Step 1:
    Choose a
    distribution over topics
    Step 2b:
    Choose a word
    from topic
    Topic:
    in word simplex
    8
    Step 2a:
    Choose a topic
    from distribution
    В
    ;
    LDA:

    finding the optimal sub-simplex

    to represent documents.

    View full-size slide

  40. Geometric interpretation
    11
    Figure from [Blei+ 2003]
    Step 1:
    Choose a
    distribution over topics
    Step 2b:
    Choose a word
    from topic
    Topic:
    in word simplex
    8
    Step 2a:
    Choose a topic
    from distribution
    В
    ;
    LDA:

    finding the optimal sub-simplex

    to represent documents.
    sub-simplex

    View full-size slide

  41. “reverse” the generation process
    ๏ We are interested in the posterior distribution.
    ๏ latent topic structure, given the observed documents.
    ๏ But it is difficult … → approximate:
    ๏ 1. Sampling-based methods (e.g. Gibbs sampling)
    ๏ 2. Variational methods (e.g. variational Bayes)
    ๏ etc…
    12

    View full-size slide

  42. “reverse” the generation process
    ๏ We are interested in the posterior distribution.
    ๏ latent topic structure, given the observed documents.
    ๏ But it is difficult … → approximate:
    ๏ 1. Sampling-based methods (e.g. Gibbs sampling)
    ๏ 2. Variational methods (e.g. variational Bayes)
    ๏ etc…
    12

    View full-size slide

  43. “reverse” the generation process
    ๏ We are interested in the posterior distribution.
    ๏ latent topic structure, given the observed documents.
    ๏ But it is difficult … → approximate:
    ๏ 1. Sampling-based methods (e.g. Gibbs sampling)
    ๏ 2. Variational methods (e.g. variational Bayes)
    ๏ etc…
    12

    View full-size slide

  44. “reverse” the generation process
    ๏ We are interested in the posterior distribution.
    ๏ latent topic structure, given the observed documents.
    ๏ But it is difficult … → approximate:
    ๏ 1. Sampling-based methods (e.g. Gibbs sampling)
    ๏ 2. Variational methods (e.g. variational Bayes)
    ๏ etc…
    12

    View full-size slide

  45. ๏ Hierarchical Dirichlet Process [Teh+ 2005]
    ๏ Correlated Topic Models [Blei+ 2006]
    ๏ Supervised Topic Models [Blei+ 2007]
    ๏ Topic Models with Power-law using Pitman-Yor process [Sato+ 2010]
    ๏ Time-series:
    ๏ Dynamic Topic Models [Blei+ 2006]
    ๏ Continuous Time Dynamic Topic Models [Wang+ 2008]
    ๏ Online Multiscale Dynamic Topic Models [Iwata+ 2010]
    ๏ Various learning methods
    ๏ Various scaling algorithms
    ๏ Various applications
    ๏ …
    13
    Extensions of LDA

    View full-size slide

  46. ๏ Text analysis

    Papers, Blogs, Classical texts …
    ๏ Video analysis
    ๏ Audio analysis
    ๏ Bioinformatics
    ๏ Network analysis
    ๏ …
    14
    Applications

    View full-size slide

  47. ๏ Gensim

    Python-based, Radim Řehůřek

    ๏ Mallet

    Java-based, UMass

    ๏ Stanford Topic Modeling Toolbox

    Java-based, Stanford

    15
    Tools

    View full-size slide

  48. References (1): books
    16
    “τϐοΫϞσϧʹΑΔ౷ܭతજࡏҙຯղੳ”

    ࠤ౻Ұ੣, 2015
    “τϐοΫϞσϧ (ػցֶशϓϩϑΣογϣφϧγϦʔζ) ”

    ؠా۩࣏, 2015

    View full-size slide

  49. ๏ [Blei&Lafferty 2009] Topic Models

    http://www.cs.princeton.edu/~blei/papers/BleiLafferty2009.pdf
    ๏ [Blei 2011] Introduction to Probabilistic Topic Models

    https://www.cs.princeton.edu/~blei/papers/Blei2011.pdf
    ๏ [Blei 2012] Review Articles: Probabilistic Topic Models

    Communications of The ACM

    http://www.cs.princeton.edu/~blei/papers/Blei2012.pdf
    ๏ [Blei 2012] Probabilistic Topic Models

    Machine Learning Summer School

    http://www.cs.princeton.edu/~blei/blei-mlss-2012.pdf
    ๏ Topic Models by David Blei (video)

    https://www.youtube.com/watch?v=DDq3OVp9dNA
    17
    References (2): papers, videos, and articles
    ๏ What is a good explanation of Latent Dirichlet Allocation? - Quora

    http://www.quora.com/What-is-a-good-explanation-of-Latent-Dirichlet-Allocation
    ๏ The LDA Buffet is Now Open by Matthew L. Jockers

    http://www.matthewjockers.net/2011/09/29/
    ๏ [ࠤ౻ 2012] ࢲͷϒοΫϚʔΫ Latent Topic Model (જࡏతτϐοΫϞσϧ)

    http://www.ai-gakkai.or.jp/my-bookmark_vol27-no3/
    ๏ [࣋ڮ&ੴࠇ 2013] ֬཰తτϐοΫϞσϧ

    ౷ܭ਺ཧݚڀॴ H24೥౓ެ։ߨ࠲

    http://www.ism.ac.jp/~daichi/lectures/ISM-2012-TopicModels-daichi.pdf
    ๏ Links to the Papers Related to Topic Models by Tomonori Masada

    http://tmasada.wikispaces.com/Links+to+the+Papers+Related+to+Topic+Models

    View full-size slide