Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Efficient top-k query processing in Lucene 8

8261d04bf57a042c8eab6757c386f7b2?s=47 Tomoko Uchida
February 26, 2019
Technology
3
2.8k

Efficient top-k query processing in Lucene 8

Talk at Search Engineering Tech Talk #1
https://search-tech.connpass.com/event/112866/

8261d04bf57a042c8eab6757c386f7b2?s=128

Tomoko Uchida

February 26, 2019
Tweet

More Decks by Tomoko Uchida

See All by Tomoko Uchida
8261d04bf57a042c8eab6757c386f7b2?s=48 mocobeta
4
2k
8261d04bf57a042c8eab6757c386f7b2?s=48 mocobeta
0
360
8261d04bf57a042c8eab6757c386f7b2?s=48 mocobeta
5
1.5k

Other Decks in Technology

See All in Technology
35f1e35d8c02c612e281c6c7a9d22341?s=48 hirosys
0
140
23f4d5d797a91b6d17d627b90b5a42d9?s=48 kentaro
1
430
0f4a27389318ca7087ae12b1593171c1?s=48 saoritakita
0
350
9cdb1d21375062bc2cd0d1358c2fa1f0?s=48 noir_neo
0
130
D7f27502ce4dfef29afae545df767b27?s=48 hsano
0
130
9e4ea8b3f81c2acc322cc6ed0db32be9?s=48 satoshirobatofujimoto
0
110
A61fc58218907d6778a6cbf0fe7611da?s=48 redhatopenshift
0
360
A5fed8c15d016c2a42d7d9f830326045?s=48 takuros
3
530
34d5dd227dad6526e026f43b42b269b6?s=48 kekeke_47
0
380
19fc8f6113c5c3d86e6176362ff29479?s=48 kanaugust
PRO
0
180
928b1395afedebb5ee48d44ab917c5f1?s=48 ryusa
2
240
E39a62a5c1f1829cc6ce0ec642424947?s=48 cmwatanabeseigo
0
360

Featured

See All Featured
A60068bce2e73de3a37ca9d2dbe36092?s=48 bryan
30
3.3k
A9179349dd2bdc67f377719f56d85656?s=48 garrettdimon
287
110k
9cde37f47e4a800ea081ea42de8d749a?s=48 dougneiner
119
7.8k
0fe2973fa8d27d96547e43322341183a?s=48 swwweet
206
6.8k
Ae14cc4491ac334f9cd23f9f93b4305e?s=48 orderedlist
PRO
329
35k
7653c7c449918ff6542880a8c284f5e7?s=48 jponch
103
5k
462dad0dd24c568a32dcdb0ae895ae1b?s=48 rasmusluckow
318
18k
A0517b08532aec8ab2aae3f65d40ad86?s=48 hannesfritz
27
930
C86443373fbfe92996aa882d0d7a59f8?s=48 imathis
478
150k
1519587b1a0febb658c36c94f6272b0d?s=48 roundedbygravity
84
7.8k
027d1ebf66cc039a0bd3b55eeadbe75d?s=48 sachag
446
36k
D09fad3e36ae6841f54820be0e907f7a?s=48 rocio
155
11k

Transcript

  1. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 1/32

    EFFICIENT TOP-K QUERY EFFICIENT TOP-K QUERY PROCESSING IN LUCENE 8 PROCESSING IN LUCENE 8 Tomoko Uchida 2019/02/26 @ Roppongi Hills

  2. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 2/32

    WHO AM I WHO AM I Twitter: @moco_beta 5+ years of experience w/ Solr and Elasticsearch Software Engineer @ Developing patent search w/ AI technologies developer co-mainteiner lead author AI Samurai Inc. Janome Luke: Lucene Toolbox Project 改訂3版 Apache Solr 入門

  3. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 3/32

    Lucene/Solr 8.0 and Elasticsearch 7.0 are coming...

  4. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 4/32

    SUMMARY OF THIS TALK SUMMARY OF THIS TALK Top-k query processing / scoring will be much faster! Especially effective in disjunction (OR) query Also works for complex queries such as PhraseQuery, WildcardQuery and their combinations Exact total hits count will not be returned (in default)

  5. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 5/32

    AND THERE IS A LONG VERSION... AND THERE IS A LONG VERSION... This talk is a short version of my survey. Please see this post (in Japanese) for more details :) Lucene 8 の Top-k クエリプロセッシング最適化

  6. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 6/32

    REFERENCES REFERENCES Magic WAND: Faster Retrieval of Top Hits in Elasticsearch (FOSDEM 2019) Super-speedy scoring in Lucene 8 (FOSDEM 2019) Apache Lucene and Apache Solr 8 (Berlin Buzzwords 2012) Ef cient Scoring in Lucene 転置インデックスと Top k-query

  7. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 7/32

    PAPERS PAPERS [1] T. Strohman, H. Turtle, and B. Croft. Optimization strategies for complex queries. In Proceedings of ACM SIGIR conference, 2005. [2] K. Chakrabarti, S. Chaudhuri, V. Ganti. Interval- Based Pruning for Top-k Processing over Compressed Lists, in Proc. of ICDE, 2011. [3] A. Z. Broder, D. Carmel, M. Herscovici, A. Soffer, J. Y. Zien. Ef cient Query Evaluation using a Two-Level Retrieval Process, in Proc. of CIKM, 2003. [4] S. Ding and T. Suel. Faster top-k document retrieval using block-max indexes. SIGIR, 2011.

  8. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 8/32

    HOW MUCH FASTER? - AND QUERY HOW MUCH FASTER? - AND QUERY http://people.apache.org/~mikemccand/lucenebench/An

  9. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 9/32

    HOW MUCH FASTER? - OR QUERY (1) HOW MUCH FASTER? - OR QUERY (1) http://people.apache.org/~mikemccand/lucenebench/Or

  10. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 10/32

    HOW MUCH FASTER? - OR QUERY (2) HOW MUCH FASTER? - OR QUERY (2) http://people.apache.org/~mikemccand/lucenebench/Or

  11. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 11/32

    HOW MUCH FASTER? - TERM QUERY HOW MUCH FASTER? - TERM QUERY http://people.apache.org/~mikemccand/lucenebench/Te

  12. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 12/32

    ALGORITHMS ALGORITHMS

  13. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 13/32

    POSTING LIST RETRIEVAL AND THE POSTING LIST RETRIEVAL AND THE CHALLENGE ON DISJUNCTION CHALLENGE ON DISJUNCTION Query "search OR engine"

  14. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 14/32

    MAXSCORE MAXSCORE Introduced by H.R.Turtle and J.Flood in 1995

  15. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 15/32

    INTERVAL-BASED PRUNING INTERVAL-BASED PRUNING MaxScore variant adopted to block compressed indexes [2]

  16. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 16/32

    WAND WAND Special operator proposed in [3] "WAND" is the abbreviation for "Week AND" or "Weighted AND" OR is being close to AND when a document contains a large enough subset of the query terms Score of a document having a large subset of the query terms is higher than the ones of documents with a few of them

  17. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 17/32

    SOUNDS FAMILIAR? SOUNDS FAMILIAR? Lucene already has similar concept : "Minimum Should Match"

  18. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 18/32

    WAND WAND Query "the OR search OR engine OR lucene"

  19. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 19/32

    WAND WAND Steps 1. Assume current threshold (kth highest score) is 12. 2. Sort postings by current pointer. 3. Find "pivot" term and docid - here, that is "search" and id=486. 4. Calculate the partial score for doc 486 if it also contains "the" and "engine".

  20. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 20/32

    BLOCK-MAX WAND BLOCK-MAX WAND WAND variant working with block compressed indexes [4] Finally come in Lucene!

  21. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 21/32

    BLOCK-MAX WAND BLOCK-MAX WAND

  22. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 22/32

    DIVE INTO IMPLEMENTATION DIVE INTO IMPLEMENTATION

  23. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 23/32

    DISCLAIMER DISCLAIMER This is about low-level, complex part of Lucene. Could include mistakes... Lucene API can be rapidly changed. This is based on branch_8_0 branch.

  24. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 24/32

    REVIEW: LUCENE SCORING ARCHITECTURE REVIEW: LUCENE SCORING ARCHITECTURE Ex. TermQuery

  25. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 25/32

    REVIEW: LUCENE SCORING ARCHITECTURE REVIEW: LUCENE SCORING ARCHITECTURE Ex. BooleanQuery

  26. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 26/32

    BLOCK-MAX WAND IMPLEMENTATION BLOCK-MAX WAND IMPLEMENTATION Changes in indexing o.a.l.index.Impact o.a.l.codecs.CompetitiveImpactAccumulator o.a.l.codecs.lucene50.Lucene50SkipWriter#writeImpa ...

  27. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 27/32

    BLOCK-MAX WAND IMPLEMENTATION BLOCK-MAX WAND IMPLEMENTATION Changes in retrieving posting list o.a.l.codecs.lucene50.Lucene50ScoreSkipReader o.a.l.index.ImpactsSource o.a.l.search.MaxScoreCache o.a.l.search.ImpactsDISI ...

  28. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 28/32

    BLOCK-MAX WAND IMPLEMENTATION BLOCK-MAX WAND IMPLEMENTATION Changes in scoring Ex. TermQuery

  29. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 29/32

    BLOCK-MAX WAND IMPLEMENTATION BLOCK-MAX WAND IMPLEMENTATION Changes in scoring Ex. BooleanQuery

  30. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 30/32

    [ANN] LUKE HAS BEEN REVISED! [ANN] LUKE HAS BEEN REVISED! GUI tool for introspecting and debugging your Lucene/Solr/Elasticsearch index. https://github.com/DmitryKey/luke

  31. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 31/32

    [ANN] LUKE HAS BEEN REVISED! [ANN] LUKE HAS BEEN REVISED! Eventually rewritten on top of Swing ... in 2019? :) Licenced under ALv2 and works ne with JDK11+ Popular in US, Europe and China Still big growth potential in Japan It's a long story

  32. 2019/2/25 Efficient top-k query processing in Lucene 8 http://mocobeta.github.io/slides-html/search-tech-talk-1/search-tech-talk-1.html?print-pdf 32/32

    THANK YOU THANK YOU Happy (paper | code) reading!