Faceting analyzed fields with some sprinkles of probability theory

Talk given at Berlin buzzwords 2013

Transcript

1. Faceting analyzed fields
   with some sprinkles of
   probability theory
   conjures trending topic analysis and other
   interesting insights
   Boaz Leskes
   Elasticsearch
   @bleskes
   work done for
   Buzzcapture
   

   View Slide

2. Trending?
   

   View Slide

3. © Buzzcapture
   

   View Slide

4. © Buzzcapture
   

   View Slide

5. reference reference
   topic
   © Buzzcapture
   

   View Slide

6. topic reference
   ≠
   

   View Slide

7. topic reference
   P(w|T) =
   kDt
   |w 2 Dt
   k
   kDt
   k
   

   View Slide

8. View Slide

9. topic reference
   P(w|T) =
   kDt
   |w 2 Dt
   k
   kDt
   k
   

   View Slide

10. P(w|T) =
    kDt
    |w 2 Dt
    k
    kDt
    k
    

    View Slide

11. brown
    dog
    fox
    quick
    2 5 10 12
    5 6 12 13
    2
    5
    6
    10
    12
    13
    brown
    dog
    fox
    quick
    

    View Slide

12. In our index.
    • Terms = 12GB
    • “Arrows” = 41GB
    

    View Slide

13. {
    tweet: {
    type: "string",
    analyzer: "whitespace"
    fielddata: {
    filter: {
    regex: "^#.*",
    frequency: {
    min: 10
    }
    }
    }
    }
    }
    Drop terms which occur too little
    

    View Slide

14. Drop docs with too many terms
    

    View Slide

15. reference reference
    topic
    © Buzzcapture
    

    View Slide

16. iculture
    10,122
    floor
    8,998
    cover
    6,874
    toy
    4,402
    ground
    3,841
    4.0
    7,878
    4.1
    4,292
    rtacties
    4,078
    jelly
    2,905
    bean
    2,857
    

    View Slide