Apache Lucene/Solr Internals for JEEConf

Transcript

1. Apache Lucene/Solr
   Internals
2. 100GB disk space 18 066 980 records Indexing took 1,5 hours 
 in 200 threads
 1000 batch 4 CPU cores 16 GB memory
3. 1999 2010 2003 2004
6. Data Model • document oriented • flat • store • index
7. Data Model • document oriented • flat • store • index score:1 tag:java type:answer Document boost = 1.1 docID = 23
8. Showcase
9. Basic Flow Lucene Directory Index Writer Index Searcher Analyzer Index Reader
10. Basic Flow score:1 tag:java type:answer Document boost = 1.1 Lucene Directory Index Writer Index Searcher addDocument Analyzer Index Reader
11. Basic Flow score:1 tag:java type:answer Document boost = 1.1 Lucene Directory Index Writer Index Searcher addDocument query tag:java Analyzer Index Reader
12. Basic Flow score:1 tag:java type:answer Document boost = 1.1 Lucene Directory Index Writer Index Searcher addDocument query tag:java score:1 tag:java type:answer Document boost = 1.1 search Analyzer Index Reader
13. Lucene Index Structure
14. score:0 score:1 score:5 ... tag:java tag:mysql tag:css ... type:answer type:question 3 4 2 2 3 4 3 2 Term Infos
15. score:0 score:1 score:5 ... tag:java tag:mysql tag:css ... type:answer type:question 3 +1 +2 10 +3 +1 4 +11 5 +2 6 +52 +1 1 +30 +27 3 +7 +1 5 +2 3 +7 +2 4 2 2 3 4 3 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 5 2 1 Term Infos Term Frequencies
16. score:0 ... tag:mysql ... score:0 score:1 score:5 ... tag:java tag:mysql tag:css ... type:answer type:question type:question 3 +1 +2 10 +3 +1 4 +11 5 +2 6 +52 +1 1 +30 +27 3 +7 +1 5 +2 3 +7 +2 4 2 2 3 4 3 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 5 2 1 Term Infos Term Info Index Term Frequencies 3 3 2
17. Directory
18. Directory Segment
 α A B C D E 1 1 1 1 1
19. Directory Segment
 α Segment
 β A B C D E F G H I 1 1 1 1 1 1 1 1 1
20. Directory Segment
 α Segment
 β Segment
 ɣ A B C D E F G H I J K L 1 1 1 1 1 1 1 1 1 1 1 1
21. Directory Segment
 α Segment
 β Segment
 ɣ A B C D E F G H I J K L K L 1 1 1 1 1 1 1 1 1 1 1 1
22. Directory Segment
 α Segment
 β Segment
 ɣ Segment
 δ A B C D E F G H I J K L K L 0 0 1 1 1 1 1 1 1 1 1 1 1
23. Directory Segment
 α Segment
 β Segment
 ɣ Segment
 δ A B C D E F G H I J L L 0 K 0 1 K 1 1 1 1 1 1 1 1 1 1
24. Directory Segment
 α Segment
 β Segment
 ɣ Segment
 δ A B C D E F G H I J 1 K 1 1 1 1 1 1 1 1 1 1
25. Directory Segment
 α Segment
 β Segment
 ɣ Segment
 δ A B C D E F G H I J 1 K 1 1 1 1 1 1 1 1 Segment
 ε 1 1
26. Showcase
27. NLP
28. Natural Language Processing
29. Analyzer Tokenizer Filter Char filter
30. Analyzer Tokenizer Filter Char filter Index time Query time
31. Analyzer <strong>There are no pointers in Java!</strong> Tokenizer Filter Char filter Index time Query time
32. Analyzer <strong>There are no pointers in Java!</strong> Tokenizer Filter There are no pointers in Java! Char filter Index time Query time
33. Analyzer <strong>There are no pointers in Java!</strong> Tokenizer There are no pointers in Java Filter There are no pointers in Java! Char filter Index time Query time
34. Analyzer <strong>There are no pointers in Java!</strong> Tokenizer There are no pointers in Java Filter There are no pointers in Java! Char filter Index time Query time ? ? ? pointer ? java
35. Analyzer <strong>There are no pointers in Java!</strong> Tokenizer There are no pointers in Java Filter There are no pointers in Java! Char filter pointers in Java Index time Query time ? ? ? pointer ? java
36. Analyzer <strong>There are no pointers in Java!</strong> Tokenizer There are no pointers in Java Filter There are no pointers in Java! Char filter pointers in Java Index time Query time pointers in Java ? ? ? pointer ? java
37. Analyzer <strong>There are no pointers in Java!</strong> Tokenizer There are no pointers in Java Filter There are no pointers in Java! Char filter pointers in Java Index time Query time pointers in Java pointers Java in ? ? ? pointer ? java
38. Analyzer <strong>There are no pointers in Java!</strong> Tokenizer There are no pointers in Java Filter There are no pointers in Java! Char filter pointers in Java Index time Query time pointers in Java pointers Java in ? ? ? pointer ? java pointer java ?
39. Showcase
40. Keyword Tokenizer <strong>There are no pointers in Java!</strong>
41. Keyword Tokenizer <strong>There are no pointers in Java!</strong> <strong>There are no pointers in Java!</strong>
42. Standard Tokenizer <strong>There are no pointers in Java!</strong>
43. Standard Tokenizer <strong>There are no pointers in Java!</strong> strong strong There are no pointers in Java
44. Stemming pointer pointers pointing
45. Stemming pointer pointers pointing pointer pointer point
46. Stop Words There are no pointers in Java
47. Stop Words There are no pointers in Java ? ? ? pointers ? Java
48. Edge N-Gram pointers
49. Edge N-Gram pointers p po poi poin point pointe pointer pointers
50. Showcase
51. Shingle There are no pointers in Java
52. Shingle There There are There are no There are no pointers There are no pointers in Java are are no are no pointers are no pointers in no no pointers no pointers in no pointers in Java pointers pointers in pointers in Java in in Java Java
53. Showcase
54. Path Hierarchy /Technology/Java/Virtual Machine
55. Path Hierarchy /Technology/Java/Virtual Machine /Technology /Technology/Java/ /Technology/Java/Virtual Machine
56. Showcase
57. Faceted Navigation
58. Showcase
59. Algorithm tag:java tag:mysql tag:css 5 +2 6 +52 +1 1 +30 +27 +2 7 31 58 59 Query Result Index
60. Algorithm tag:java tag:mysql tag:css 5 +2 6 +52 +1 1 +30 +27 +2 7 31 58 59 Query Result Index 5 7 6 58 59 1 31 58 60
61. Algorithm tag:java tag:mysql tag:css 5 +2 6 +52 +1 1 +30 +27 +2 7 31 58 59 Query Result Index 5 7 6 58 59 1 31 58 60 7 58 59 31 58
62. Algorithm tag:java tag:mysql tag:css 5 +2 6 +52 +1 1 +30 +27 +2 7 31 58 59 Query Result Index Facet 1 2 2 5 7 6 58 59 1 31 58 60 7 58 59 31 58
63. Showcase
64. Lucene Index vs RDBMS Lucene • Text search • Updates are expensive • Requests N-times faster • Facets are cheap • Values modifications DB • Value search • Updates are cheap • Requests N-time slower • Facets are expensive • Values are stored as is
65. Q&A @AnatolSokolenko
66. The End