Full-Text Search Philipp Krenn @xeraa 

ViennaDB Papers We Love Vienna 

Infrastructure | Developer Advocate 

Full-Text Search (FTS) Databases vs Full-Text Search 

No content

Before FTS Regular expressions ! 

Before FTS Arrays with relevant terms ! https:/ /docs.mongodb.com/manual/tutorial/model-data-for- keyword-search/ 

{ title : "Moby-Dick" , author : "Herman Melville" , published : 1851 , ISBN : 0451526996 , topics : [ "whaling" , "allegory" , "revenge" , "American" , "novel" , "nautical" , "voyage" , "Cape Cod" ] } db.volumes.createIndex({ topics: 1 }) db.volumes.findOne({ topics : "voyage" }, { title: 1 }) 

No Stemming Fuzziness Synonyms Ranking 

Finally FTS 230+ votes Created 2009 Resolved 2013 https:/ /jira.mongodb.org/browse/SERVER-380 

FTS in MongoDB Beta since 2.4 Stable since 3.0 80% solution — for more Elasticsearch 

FTS in MongoDB In Latin alphabets Case insensitive (default in 3.2) [A-z] other characters removed (default in 3.2) 

$text Updated version in MongoDB 3.2 { $text: { $search: "", $language: "", $caseSensitive: , $diacriticSensitive: } } 

No content

Example text These are not the droids you are looking for. 

Tokenizer these̴are̴not̴the̴droids̴you̴ are̴looking̴for 

Stop Words droids̴looking 

Stemming droid̴look 

Let's try it > db.starwars.ensureIndex({ quote: "text" }) { "createdCollectionAutomatically": true, "numIndexesBefore": 1, "numIndexesAfter": 2, "ok": 1 } 

Let's try it > db.starwars.getIndices() [ { "v": 1, "key": { "_id": 1 }, "name": "_id_", "ns": "starwars.starwars" }, { "v": 1, "key": { "_fts": "text", "_ftsx": 1 }, "name": "quote_text", "ns": "starwars.starwars", "weights": { "quote": 1 }, "default_language": "english", "language_override": "language", "textIndexVersion": 3 } ] 

Let's try it > db.starwars.insert({ quote: "These are not the droids you are looking for." }) Inserted 1 record(s) in 39ms WriteResult({ "nInserted": 1 }) 

Let's try it > db.starwars.find({ $text: { $search: "droid" }}) { "_id": ObjectId("574c50c3920246255ce2ad81"), "quote": "These are not the droids you are looking for." } Fetched 1 record(s) in 35ms > db.starwars.find({ $text: { $search: "look" }}) { "_id": ObjectId("574c50c3920246255ce2ad81"), "quote": "These are not the droids you are looking for." } Fetched 1 record(s) in 4ms > db.starwars.find({ $text: { $search: "you" }}) Fetched 0 record(s) in 1ms 

Let's try it > db.starwars.find({ $text: { $search: "look" }}).explain() { "queryPlanner": { "plannerVersion": 1, "namespace": "starwars.starwars", "indexFilterSet": false, "parsedQuery": { "$text": { "$search": "look", "$language": "english", "$caseSensitive": false, "$diacriticSensitive": false } }, ... 

Let's try it ... "parsedTextQuery": { "terms": [ "look" ], "negatedTerms": [ ], "phrases": [ ], "negatedPhrases": [ ] }, ... 

Let's try it > db.starwars.find({ $text: { $search: "-look" } }).explain() ... "parsedTextQuery": { "terms": [ ], "negatedTerms": [ "look" ], "phrases": [ ], "negatedPhrases": [ ] }, ... 

Let's try it > db.starwars.find({ $text: { $search: "look" }}, {score: {$meta: "textScore"}}) { "_id": ObjectId("574c50c3920246255ce2ad81"), "quote": "These are not the droids you are looking for.", "score": 0.75 } Fetched 1 record(s) in 8ms > db.starwars.find({ $text: { $search: "looks" }}, {score: {$meta: "textScore"}}) .sort({ score: { $meta: "textScore" } }).limit(1) { "_id": ObjectId("574c50c3920246255ce2ad81"), "quote": "These are not the droids you are looking for.", "score": 0.75 } Fetched 1 record(s) in 5ms 

Indexing String or array of strings only Optional language or translations Optional weighting if multiple fields indexed 

Indexing > db.allFields.ensureIndex({ "$**": "text"}, { default_language: "german" }) 

Queries // OR > db.starwars.find({ $text: { $search: "look droid" } }) // AND but without input stemming > db.starwars.find({ $text: { $search: "\"look\" \"droid\"" } }) // Negation > db.starwars.find({ $text: { $search: "look -droid" } }) // Phrase > db.starwars.find({ $text: { $search: "\"look droid\"" } }) // Translation > db.starwars.find({ $text: { $search: "buscar", $language: "es" } }) 

Score > db.starwars.find({ $text: { $search: "father look" } }, { score: { $meta: "textScore" } }) { "_id": ObjectId("574c83c9920246255ce2ad82"), "quote": "These are not the droids you are looking for", "score": 0.75 } { "_id": ObjectId("574c8712920246255ce2ad83"), "quote": "I am your father", "score": 1 } { "_id": ObjectId("574c8763920246255ce2ad84"), "quote": "Look at me father", "score": 1.5 } 

Score https:/ /github.com/mongodb/mongo/blob/v3.2/src/mongo/db/fts/fts_spec.cpp#L219 double coeff = (0.5 * data.count / numTokens) + 0.5; data.count: matches numTokens: stemmed words 

Score father look "These are not the droids you are looking for" droid look == 1 match, 2 tokens coeff: 

Score father look "I am your father" father == 1 match, 1 token coeff: 

Score https:/ /github.com/mongodb/mongo/blob/v3.2/src/mongo/db/fts/fts_spec.cpp#L228 score += (weight * data.freq * coeff * adjustment); weight: method parameter data.freq, adjustment: 1 

Score father look "Look at me father" look father == 1 match, 2 tokens look father == 1 match, 2 tokens coeff: 

Score father look each score: Sum: 

Limitations B-tree vs inverted index 

Missing features Fuzziness Suggestions Highlighting Synonyms 

Thanks! Questions? @xeraa