Faceting analyzed fields with some sprinkles of probability theory

Talk given at Berlin buzzwords 2013

Transcript

1. Faceting analyzed fields with some sprinkles of probability theory conjures trending topic analysis and other interesting insights Boaz Leskes Elasticsearch @bleskes work done for Buzzcapture
2. Trending?
3. © Buzzcapture
4. © Buzzcapture
5. reference reference topic © Buzzcapture
6. topic reference ≠
7. topic reference P(w|T) = kDt |w 2 Dt k kDt k
9. topic reference P(w|T) = kDt |w 2 Dt k kDt k
10. P(w|T) = kDt |w 2 Dt k kDt k
11. brown dog fox quick 2 5 10 12 5 6 12 13 2 5 6 10 12 13 brown dog fox quick
12. In our index. • Terms = 12GB • “Arrows” = 41GB
13. { tweet: { type: "string", analyzer: "whitespace" fielddata: { filter: { regex: "^#.*", frequency: { min: 10 } } } } } Drop terms which occur too little
14. Drop docs with too many terms
15. reference reference topic © Buzzcapture
16. iculture 10,122 floor 8,998 cover 6,874 toy 4,402 ground 3,841 4.0 7,878 4.1 4,292 rtacties 4,078 jelly 2,905 bean 2,857