Common Terms Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Have your cake and eat it too Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited @ZacharyTong polyfractal on IRC Developing - Writing - Training ಠ_ಠ (amoeba) Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Stop Words Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited or, it, be, a, and, to Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Often empty of meaning or, it, be, a, and, to Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Often empty of meaning or, it, be, a, and, to “the quick and brown fox jumped over a ledge” Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Often empty of meaning or, it, be, a, and, to “the quick and brown fox jumped over a ledge” Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Often empty of meaning Used frequently “the quick and brown fox jumped over a ledge” 33% or, it, be, a, and, to Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Often empty of meaning Used frequently Bloat inverted index or, it, be, a, and, to Friday, November 22, 13 

Stop Words... Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ...frequent ...little discriminatory value ...hurts performance Friday, November 22, 13 

Stop Words... Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Stop Filter Friday, November 22, 13 

“To be or not to be” Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Friday, November 22, 13 

“To be or not to be” Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Multi-field mapping - with stop filter - without stop filter Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Multi-field mapping { “query” : { “bool” : { “should” : [ “match” : { “body” : { “query” : “quick fox”, “boost” : 3 } }, “match” : { “body.without_stop” : { “query” : “quick fox”, “boost” : 1 } } ] }}} Boost stop-removed match But check stop-words too Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Bloat inverted index Remember, stop-words: ? Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Bloat inverted index Multi-field mapping: 2x! Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Other general problems: - manually maintain stop-list - language dependent - domain dependent - makes query scoring tricky Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Common Terms Intelligent stop-word removal Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Overview - identify “important” terms in query - find documents with “important” terms - score those matching docs with entire query Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited “unimportant” “important” - quick - brown - fox - jumped - over - ledge - the - and - a Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited high low - the - and - a Document Frequency: - quick - brown - fox - jumped - over - ledge Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited { "common": { "body": { "query": "the quick and brown fox jumped over the ledge", "cutoff_frequency": 0.001 } } } The Query Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited { "query": { "bool": { "must": [ { "term": { "body": "quick"}}, { "term": { "body": "brown"}}, { "term": { "body": "fox"}}, { "term": { "body": "jumped"}}, { "term": { "body": "over"}}, { "term": { "body": "ledge"}}, ] }}} internal execution (roughly) step 1: find docs w/ “important” Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited { "query": { "bool": { "must": [ { "term": { "body": "quick"}}, { "term": { "body": "brown"}}, { "term": { "body": "fox"}}, { "term": { "body": "jumped"}}, { "term": { "body": "over"}}, { "term": { "body": "ledge"}}, ], "should": [ { "term": { "body": "the"}}, { "term": { "body": "and"}}, { "term": { "body": "a"}}, ] }}} step 2: score matching docs internal execution (roughly) Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited { "common": { "body": { "query": "the quick and brown fox jumped over the ledge", "cutoff_frequency": 0.001, "low_freq_operator": "or", "high_freq_operator": "or", "minimum_should_match": { "low_freq" : "60%", "high_freq" : "20%" } } } } Controlling Leniency use “or” for low-freq terms Friday, November 22, 13 

{ "common": { "body": { "query": "the quick and brown fox jumped over the ledge", "cutoff_frequency": 0.001, "low_freq_operator": "or", "high_freq_operator": "or", "minimum_should_match": { "low_freq" : "60%", "high_freq" : "20%" } } } } Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Controlling Leniency how many clauses should match Friday, November 22, 13 

{ "query": { "bool": { "should": [ { "term": { "body": "quick"}}, { "term": { "body": "brown"}}, { "term": { "body": "fox"}}, { "term": { "body": "jumped"}}, { "term": { "body": "over"}}, { "term": { "body": "ledge"}}, ], "should": [ { "term": { "body": "the"}}, { "term": { "body": "and"}}, { "term": { "body": "a"}}, ] }}} Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Controlling Leniency internal execution (roughly) (use “or”) Friday, November 22, 13 

{ "query": { "bool": { "should": [ { "term": { "body": "quick"}}, { "term": { "body": "brown"}}, { "term": { "body": "fox"}}, { "term": { "body": "jumped"}}, { "term": { "body": "over"}}, { "term": { "body": "ledge"}}, ], "should": [ { "term": { "body": "the"}}, { "term": { "body": "and"}}, { "term": { "body": "a"}}, ] }}} Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Controlling Leniency internal execution (roughly) 4 clauses must match 1 clause must match (60%) (20%) Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited { "common": { "body": { "query": "the quick and brown fox jumped over the ledge", "cutoff_frequency": 0.001, "low_freq_operator": "or", "high_freq_operator": "or", "minimum_should_match": { "low_freq" : "60%", "high_freq" : "20%" } } } } Controlling Importance adjust the high/low cutoff (0.1%) Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited high low - the - and - a Document Frequency: - quick - brown - fox - jumped - over - ledge Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited { "common": { "body": { "query": "the quick and brown fox jumped over the ledge", "cutoff_frequency": 0.10, "low_freq_operator": "or", "high_freq_operator": "or", "minimum_should_match": { "low_freq" : "60%", "high_freq" : "20%" } } } } Controlling Importance adjust the high/low cutoff (10.0%) Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited high low - the - and - a - over - quick Document Frequency: - brown - fox - jumped - ledge Friday, November 22, 13 

“To be or not to be” Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited { "common": { "body": { "query": "to be or not to be", "cutoff_frequency": 0.001 } } } All high-frequency terms Friday, November 22, 13 

{ "query": { "bool": { "must": [ { "term": { "body": "to"}}, { "term": { "body": "be"}}, { "term": { "body": "or"}}, { "term": { "body": "not"}}, { "term": { "body": "be"}}, ] }}} Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited internal execution (roughly) automatically a “must” All high-frequency terms Friday, November 22, 13 

{ "common": { "body": { "query": "to be or not to be", "cutoff_frequency": 0.001, "minimum_should_match": { "low_freq" : "60%" } } } } Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Controlling Leniency how many clauses should match Friday, November 22, 13 

{ "query": { "bool": { "should": [ { "term": { "body": "to"}}, { "term": { "body": "be"}}, { "term": { "body": "or"}}, { "term": { "body": "not"}}, { "term": { "body": "be"}}, ] }}} Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited internal execution (roughly) becomes a “should” All high-frequency terms 3 clauses must match (60%) Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Adaptive Stop-lists Are these stop words? - “video” - “movie” - “film” Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Adaptive Stop-lists Are these stop words? - “video” - “movie” - “film” For YouTube, they might be! Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Adaptive Stop-lists - Common Terms uses your index for frequency - Adapts to your domain - No manual stop-list creation/maintenance - Adapts to language, etc Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Limitations - Frequencies are per-index, not per-type Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Limitations - Frequencies are per-index, not per-type - No good way to pick cutoff frequency Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Limitations - Frequencies are per-index, not per-type - No good way to pick cutoff frequency - Takes data to “warm” the query Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Limitations - Frequencies are per-index, not per-type - No good way to pick cutoff frequency - Takes data to “warm” the query - Some advanced behavior missing (fuzzy, etc) Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Questions? ಠ_ಠ @ZacharyTong polyfractal on IRC Friday, November 22, 13 

Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited Resources Common Terms Docs : http://bit.ly/1an7NOd “Stop Stopping Stopwords” : http://bit.ly/17hE2uq Friday, November 22, 13