Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Faceting analyzed fields with some sprinkles of probability theory

Faceting analyzed fields with some sprinkles of probability theory

Talk given at Berlin buzzwords 2013

Boaz Leskes

June 04, 2013
Tweet

More Decks by Boaz Leskes

Other Decks in Technology

Transcript

  1. Faceting analyzed fields with some sprinkles of probability theory conjures

    trending topic analysis and other interesting insights Boaz Leskes Elasticsearch @bleskes work done for Buzzcapture
  2. Trending?

  3. © Buzzcapture

  4. © Buzzcapture

  5. reference reference topic © Buzzcapture

  6. topic reference ≠

  7. topic reference P(w|T) = kDt |w 2 Dt k kDt

    k
  8. None
  9. topic reference P(w|T) = kDt |w 2 Dt k kDt

    k
  10. P(w|T) = kDt |w 2 Dt k kDt k

  11. brown dog fox quick 2 5 10 12 5 6

    12 13 2 5 6 10 12 13 brown dog fox quick
  12. In our index. • Terms = 12GB • “Arrows” =

    41GB
  13. { tweet: { type: "string", analyzer: "whitespace" fielddata: { filter:

    { regex: "^#.*", frequency: { min: 10 } } } } } Drop terms which occur too little
  14. Drop docs with too many terms

  15. reference reference topic © Buzzcapture

  16. iculture 10,122 floor 8,998 cover 6,874 toy 4,402 ground 3,841

    4.0 7,878 4.1 4,292 rtacties 4,078 jelly 2,905 bean 2,857