Elasticsearch: From Keyword Search To Data Science

Elasticsearch serves as the product search engine on the Kaufland Marketplace, powering search across several european storefronts and serving millions of queries and updates a day. While keyword based search has served us well for over a decade, we're working hard on the next evolution of search. How can we keep the speed of a keyword based search, while improving the search experience for the user?

Scouring the internet for interesting approaches, we teamed up with our Data Science team for query rewriting and query expansion topics and came up with a few interesting query preprocessing steps that involve scripted queries and even ESQL.

Let's take a look!

Transcript

1. From Keyword Search to Data Science
2. Introduction
3. Kau and e-commerce
4. Architecture BigQuery Kafka Indexer Elasticsearch Searcher Monolith reads reads reads writes reads
5. Searches
6. Phase: Analysis { "title": "Modernes Wandbild ... Kunstdruck New York", "search_stats": [ { "id_item": 306593629, "rank": 0.00012203718164007487, "term": "modernes wandbild new york" } ] }
7. Phase: Analysis - Category Guessing
8. Phase: Analysis - Shorter queries
9. Phase: Search
10. Next steps
11. Keyword search
12. ELSER
13. Query Expansion
14. Query Expansion
15. Query expansion steps
16. Query Example
17. Retrieve candidates PUT query-expansion-phrases { "mappings": { "properties": { "candidate": { "type": "text", "similarity" : "boolean", "fields": { "keyword" : { "type" : "keyword" } } } } } } POST query-expansion-phrases/_doc { "candidate": "nintendo switch controller" }
18. Retrieve candidates GET query_expansion_phrases/_search { "size": 500, "query": { "bool": { "must": [ { "match": { "candidate": "nintendo switch controller grün" } } ], "filter": [ { "script": { "script": { "source": ... } } } ] } } }
19. Retrieve candidates // params.query = "nintendo switch controller grün" if (params.query.indexOf(doc['candidate.keyword'].value) < 0) { return false; } def asList = Arrays.asList(/ /.split(params.query)); def tokenizer = new StringTokenizer(doc['candidate.keyword'].value, " "); while (tokenizer.hasMoreElements()) { def term = tokenizer.nextElement(); def isTermInQuery = asList.contains(term); // if the query does not contain the current term, we bail if (!isTermInQuery) { return false } } return true;
20. Retrieve candidates
21. Retrieve scores PUT query_expansion_cohesion_scores { "mappings": { "properties": { "item_term": { "type": "text", "fields": { "keyword": { "type": "keyword" } } }, "query_term": { "type": "text", "fields": { "keyword": { "type": "keyword" } } }, "score": { "type": "float" } } } }
22. Retrieve scores: Sample document POST query_expansion_cohesion_scores/_doc { "query_term": "nintendo switch controller", "item_term": "joy-con", "score": 0.023316383085096128 }
23. Retrieve scores: Query
24. Correlate query & document terms FROM query_expansion_cohesion_scores | WHERE ( query_term.keyword LIKE "nintendo switch controller" OR query_term.keyword LIKE "switch controller" OR query_term.keyword LIKE "nintendo switch" ) | STATS final_score=SUM(score) * POW( TO_DOUBLE( COUNT(item_term.keyword) )/3, 0.01 ), query_terms=VALUES(query_term.keyword) BY item_term.keyword | SORT final_score DESC | LIMIT 10
25. Correlate query & document terms "nintendo switch" "controller" "joy-con" { 1 "columns": [ 2 { "name": "final_score", "type": "double" }, 3 { "name": "query_terms", "type": "keyword" }, 4 { "name": "item_term.keyword", "type": "keyword" } 5 ], 6 "values": [ 7 [ 8 0.07394268180954819, 9 [ "switch controller", "nintendo switch controller" ], 10 11 ], 12 [ 13 0.073837760835886, 14 [ "switch controller", "nintendo switch", "nintendo switch controller" ], 15 16 ], 17 [ 18 0.05886813160032034, 19 [ "switch controller", "nintendo switch", "nintendo switch controller" ], 20 21 ], 22 23
26. Results
27. Next steps
28. Summary
29. Thank you
30. Thank you