$30 off During Our Annual Pro Sale. View Details »

Faceting analyzed fields with some sprinkles of probability theory

Faceting analyzed fields with some sprinkles of probability theory

Talk given at Berlin buzzwords 2013

Boaz Leskes

June 04, 2013
Tweet

More Decks by Boaz Leskes

Other Decks in Technology

Transcript

  1. Faceting analyzed fields
    with some sprinkles of
    probability theory
    conjures trending topic analysis and other
    interesting insights
    Boaz Leskes
    Elasticsearch
    @bleskes
    work done for
    Buzzcapture

    View Slide

  2. Trending?

    View Slide

  3. © Buzzcapture

    View Slide

  4. © Buzzcapture

    View Slide

  5. reference reference
    topic
    © Buzzcapture

    View Slide

  6. topic reference

    View Slide

  7. topic reference
    P(w|T) =
    kDt
    |w 2 Dt
    k
    kDt
    k

    View Slide

  8. View Slide

  9. topic reference
    P(w|T) =
    kDt
    |w 2 Dt
    k
    kDt
    k

    View Slide

  10. P(w|T) =
    kDt
    |w 2 Dt
    k
    kDt
    k

    View Slide

  11. brown
    dog
    fox
    quick
    2 5 10 12
    5 6 12 13
    2
    5
    6
    10
    12
    13
    brown
    dog
    fox
    quick

    View Slide

  12. In our index.
    • Terms = 12GB
    • “Arrows” = 41GB

    View Slide

  13. {
    tweet: {
    type: "string",
    analyzer: "whitespace"
    fielddata: {
    filter: {
    regex: "^#.*",
    frequency: {
    min: 10
    }
    }
    }
    }
    }
    Drop terms which occur too little

    View Slide

  14. Drop docs with too many terms

    View Slide

  15. reference reference
    topic
    © Buzzcapture

    View Slide

  16. iculture
    10,122
    floor
    8,998
    cover
    6,874
    toy
    4,402
    ground
    3,841
    4.0
    7,878
    4.1
    4,292
    rtacties
    4,078
    jelly
    2,905
    bean
    2,857

    View Slide