Full-Text Search in MongoDB

Taking a look at the inner workings of full-text search in MongoDB 3.2

Transcript

1. Full-Text Search Philipp Krenn @xeraa
2. ViennaDB Papers We Love Vienna
3. Infrastructure | Developer Advocate
4. Full-Text Search (FTS) Databases vs Full-Text Search
6. Before FTS Regular expressions !
7. Before FTS Arrays with relevant terms ! https:/ /docs.mongodb.com/manual/tutorial/model-data-for- keyword-search/
8. { title : "Moby-Dick" , author : "Herman Melville" , published : 1851 , ISBN : 0451526996 , topics : [ "whaling" , "allegory" , "revenge" , "American" , "novel" , "nautical" , "voyage" , "Cape Cod" ] } db.volumes.createIndex({ topics: 1 }) db.volumes.findOne({ topics : "voyage" }, { title: 1 })
9. No Stemming Fuzziness Synonyms Ranking
10. Finally FTS 230+ votes Created 2009 Resolved 2013 https:/ /jira.mongodb.org/browse/SERVER-380
11. FTS in MongoDB Beta since 2.4 Stable since 3.0 80% solution — for more Elasticsearch
12. FTS in MongoDB In Latin alphabets Case insensitive (default in 3.2) [A-z] other characters removed (default in 3.2)
13. $text Updated version in MongoDB 3.2 { $text: { $search: "<string>", $language: "<string>", $caseSensitive: <boolean>, $diacriticSensitive: <boolean> } }
15. Example text These are not the droids you are looking for.
16. Tokenizer these̴are̴not̴the̴droids̴you̴ are̴looking̴for
17. Stop Words droids̴looking
18. Stemming droid̴look
19. Let's try it > db.starwars.ensureIndex({ quote: "text" }) { "createdCollectionAutomatically": true, "numIndexesBefore": 1, "numIndexesAfter": 2, "ok": 1 }
20. Let's try it > db.starwars.getIndices() [ { "v": 1, "key": { "_id": 1 }, "name": "_id_", "ns": "starwars.starwars" }, { "v": 1, "key": { "_fts": "text", "_ftsx": 1 }, "name": "quote_text", "ns": "starwars.starwars", "weights": { "quote": 1 }, "default_language": "english", "language_override": "language", "textIndexVersion": 3 } ]
21. Let's try it > db.starwars.insert({ quote: "These are not the droids you are looking for." }) Inserted 1 record(s) in 39ms WriteResult({ "nInserted": 1 })
22. Let's try it > db.starwars.find({ $text: { $search: "droid" }}) { "_id": ObjectId("574c50c3920246255ce2ad81"), "quote": "These are not the droids you are looking for." } Fetched 1 record(s) in 35ms > db.starwars.find({ $text: { $search: "look" }}) { "_id": ObjectId("574c50c3920246255ce2ad81"), "quote": "These are not the droids you are looking for." } Fetched 1 record(s) in 4ms > db.starwars.find({ $text: { $search: "you" }}) Fetched 0 record(s) in 1ms
23. Let's try it > db.starwars.find({ $text: { $search: "look" }}).explain() { "queryPlanner": { "plannerVersion": 1, "namespace": "starwars.starwars", "indexFilterSet": false, "parsedQuery": { "$text": { "$search": "look", "$language": "english", "$caseSensitive": false, "$diacriticSensitive": false } }, ...
24. Let's try it ... "parsedTextQuery": { "terms": [ "look" ], "negatedTerms": [ ], "phrases": [ ], "negatedPhrases": [ ] }, ...
25. Let's try it > db.starwars.find({ $text: { $search: "-look" } }).explain() ... "parsedTextQuery": { "terms": [ ], "negatedTerms": [ "look" ], "phrases": [ ], "negatedPhrases": [ ] }, ...
26. Let's try it > db.starwars.find({ $text: { $search: "look" }}, {score: {$meta: "textScore"}}) { "_id": ObjectId("574c50c3920246255ce2ad81"), "quote": "These are not the droids you are looking for.", "score": 0.75 } Fetched 1 record(s) in 8ms > db.starwars.find({ $text: { $search: "looks" }}, {score: {$meta: "textScore"}}) .sort({ score: { $meta: "textScore" } }).limit(1) { "_id": ObjectId("574c50c3920246255ce2ad81"), "quote": "These are not the droids you are looking for.", "score": 0.75 } Fetched 1 record(s) in 5ms
27. Indexing String or array of strings only Optional language or translations Optional weighting if multiple fields indexed
28. Indexing > db.allFields.ensureIndex({ "$**": "text"}, { default_language: "german" })
29. Queries // OR > db.starwars.find({ $text: { $search: "look droid" } }) // AND but without input stemming > db.starwars.find({ $text: { $search: "\"look\" \"droid\"" } }) // Negation > db.starwars.find({ $text: { $search: "look -droid" } }) // Phrase > db.starwars.find({ $text: { $search: "\"look droid\"" } }) // Translation > db.starwars.find({ $text: { $search: "buscar", $language: "es" } })
30. Score > db.starwars.find({ $text: { $search: "father look" } }, { score: { $meta: "textScore" } }) { "_id": ObjectId("574c83c9920246255ce2ad82"), "quote": "These are not the droids you are looking for", "score": 0.75 } { "_id": ObjectId("574c8712920246255ce2ad83"), "quote": "I am your father", "score": 1 } { "_id": ObjectId("574c8763920246255ce2ad84"), "quote": "Look at me father", "score": 1.5 }
31. Score https:/ /github.com/mongodb/mongo/blob/v3.2/src/mongo/db/fts/fts_spec.cpp#L219 double coeff = (0.5 * data.count / numTokens) + 0.5; data.count: matches numTokens: stemmed words
32. Score father look "These are not the droids you are looking for" droid look == 1 match, 2 tokens coeff:
33. Score father look "I am your father" father == 1 match, 1 token coeff:
34. Score https:/ /github.com/mongodb/mongo/blob/v3.2/src/mongo/db/fts/fts_spec.cpp#L228 score += (weight * data.freq * coeff * adjustment); weight: method parameter data.freq, adjustment: 1
35. Score father look "Look at me father" look father == 1 match, 2 tokens look father == 1 match, 2 tokens coeff:
36. Score father look each score: Sum:
37. Limitations B-tree vs inverted index
38. Missing features Fuzziness Suggestions Highlighting Synonyms
39. Thanks! Questions? @xeraa