Apache Lucene/Solr Internals

Transcript

1. Apache Lucene/Solr
   Internals
2. 4 nodes 
 ✕ 12GB disk space June 2013 database 14 630 209 records Indexing took 2,5 hours 
 in 200 threads
 1000 batch VM 
 16 CPU cores 16 GB memory
3. lightweight performant search library
4. Data Model • document oriented • flat • store • index
5. Data Model • document oriented • flat • store • index score:1 tag:java type:answer Document boost = 1.1 docID = 23
6. Showcase
7. Basic Flow Lucene Directory Index Writer Index Searcher Analyzer Index Reader
8. Basic Flow score:1 tag:java type:answer Document boost = 1.1 Lucene Directory Index Writer Index Searcher addDocument Analyzer Index Reader
9. Basic Flow score:1 tag:java type:answer Document boost = 1.1 Lucene Directory Index Writer Index Searcher addDocument query tag:java Analyzer Index Reader
10. Basic Flow score:1 tag:java type:answer Document boost = 1.1 Lucene Directory Index Writer Index Searcher addDocument query tag:java score:1 tag:java type:answer Document boost = 1.1 search Analyzer Index Reader
11. Lucene Index Structure
12. Directory
13. Directory Segment
 α A B C D E 1 1 1 1 1
14. Directory Segment
 α Segment
 β A B C D E F G H I 1 1 1 1 1 1 1 1 1
15. Directory Segment
 α Segment
 β Segment
 ɣ A B C D E F G H I J K L 1 1 1 1 1 1 1 1 1 1 1 1
16. Directory Segment
 α Segment
 β Segment
 ɣ A B C D E F G H I J K L K L 1 1 1 1 1 1 1 1 1 1 1 1
17. Directory Segment
 α Segment
 β Segment
 ɣ Segment
 δ A B C D E F G H I J K L K L 0 0 1 1 1 1 1 1 1 1 1 1 1
18. Directory Segment
 α Segment
 β Segment
 ɣ Segment
 δ A B C D E F G H I J L L 0 K 0 1 K 1 1 1 1 1 1 1 1 1 1
19. Directory Segment
 α Segment
 β Segment
 ɣ Segment
 δ A B C D E F G H I J 1 K 1 1 1 1 1 1 1 1 1 1
20. Directory Segment
 α Segment
 β Segment
 ɣ Segment
 δ A B C D E F G H I J 1 K 1 1 1 1 1 1 1 1 Segment
 ε 1 1
21. score:0 score:1 score:5 ... tag:java tag:mysql tag:css ... type:answer type:question 3 4 2 2 3 4 3 2 Term Infos
22. score:0 score:1 score:5 ... tag:java tag:mysql tag:css ... type:answer type:question 3 +1 +2 10 +3 +1 4 +11 5 +2 6 +52 +1 1 +30 +27 3 +7 +1 5 +2 3 +7 +2 4 2 2 3 4 3 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 5 2 1 Term Infos Term Frequencies
23. score:0 ... tag:mysql ... score:0 score:1 score:5 ... tag:java tag:mysql tag:css ... type:answer type:question type:question 3 +1 +2 10 +3 +1 4 +11 5 +2 6 +52 +1 1 +30 +27 3 +7 +1 5 +2 3 +7 +2 4 2 2 3 4 3 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 5 2 1 Term Infos Term Info Index Term Frequencies 3 3 2
24. Showcase
25. scalable
 enterprise search server

26. Request Handlers Data Import Handler SolrCloud solrconfig.xml
 schema.xml
27. Text Analysis
28. Analyzer Tokenizer Filter Char filter
29. Analyzer Tokenizer Filter Char filter Index time
30. Analyzer <strong>There are no pointers in Java!</strong> Tokenizer Filter Char filter Index time
31. Analyzer <strong>There are no pointers in Java!</strong> Tokenizer Filter There are no pointers in Java! Char filter Index time
32. Analyzer <strong>There are no pointers in Java!</strong> Tokenizer There are no pointers in Java Filter There are no pointers in Java! Char filter Index time
33. Analyzer <strong>There are no pointers in Java!</strong> Tokenizer There are no pointers in Java Filter There are no pointers in Java! Char filter Index time ? ? ? pointer ? java
34. Analyzer <strong>There are no pointers in Java!</strong> Tokenizer There are no pointers in Java Filter There are no pointers in Java! Char filter Index time Query time ? ? ? pointer ? java
35. Analyzer <strong>There are no pointers in Java!</strong> Tokenizer There are no pointers in Java Filter There are no pointers in Java! Char filter pointers in Java Index time Query time ? ? ? pointer ? java
36. Analyzer <strong>There are no pointers in Java!</strong> Tokenizer There are no pointers in Java Filter There are no pointers in Java! Char filter pointers in Java Index time Query time pointers in Java ? ? ? pointer ? java
37. Analyzer <strong>There are no pointers in Java!</strong> Tokenizer There are no pointers in Java Filter There are no pointers in Java! Char filter pointers in Java Index time Query time pointers in Java pointers Java in ? ? ? pointer ? java
38. Analyzer <strong>There are no pointers in Java!</strong> Tokenizer There are no pointers in Java Filter There are no pointers in Java! Char filter pointers in Java Index time Query time pointers in Java pointers Java in ? ? ? pointer ? java pointer java ?
39. Showcase
40. Faceted Navigation
41. Showcase
42. Algorithm tag:java tag:mysql tag:css 5 +2 6 +52 +1 1 +30 +27 +2 7 31 58 59 Query Result Index
43. Algorithm tag:java tag:mysql tag:css 5 +2 6 +52 +1 1 +30 +27 +2 7 31 58 59 Query Result Index 5 7 6 58 59 1 31 58 60
44. Algorithm tag:java tag:mysql tag:css 5 +2 6 +52 +1 1 +30 +27 +2 7 31 58 59 Query Result Index 5 7 6 58 59 1 31 58 60 7 58 59 31 58
45. Algorithm tag:java tag:mysql tag:css 5 +2 6 +52 +1 1 +30 +27 +2 7 31 58 59 Query Result Index Facet 1 2 2 5 7 6 58 59 1 31 58 60 7 58 59 31 58
46. Showcase
47. Spell Suggestions
48. Levenshtein Distance html htmm Levenshtein
 distance = 1 hlmz html Levenshtein
 distance = 2 tag:php tag:jquery tag:json tag:java tag:c# tag:apache tag:osx tag:html
49. Levenshtein Automaton html Levenshtein
 distance = 1 H t t m m l l t m l H t t m m l l m l t t m H l m
50. Showcase
51. SolrCloud
52. Join Cluster
53. Shard 3 Shard 2 Shard 1 Join Cluster
54. Indexing Shard 1 Shard 2 Shard 3
55. Indexing Shard 1 Shard 2 Shard 3
56. Indexing Shard 1 Shard 2 Shard 3
57. Indexing Shard 1 Shard 2 Shard 3
58. Indexing Shard 1 Shard 2 Shard 3
59. Indexing Shard 1 Shard 2 Shard 3
60. Query Shard 1 Shard 2 Shard 3
61. Query Shard 1 Shard 2 Shard 3 query tag:java
62. Query Shard 1 Shard 2 Shard 3 query tag:java
63. Query Shard 1 Shard 2 Shard 3 query tag:java
64. Query Shard 1 Shard 2 Shard 3 query tag:java
65. Failure Shard 1 Shard 2 Shard 3
66. Failure Shard 1 Shard 2 Shard 3
67. Failure Shard 1 Shard 2 Shard 3
68. CAP Model C A P SolrCloud Solr
69. Showcase
70. Solr is... • enterprise level search engine • vertically scalable • horizontally scalable, but... • tunable • poorly documented • with active community
71. Q&A @AnatolSokolenko
72. The End