Apache Solr: Lessons Learned

Lessons learned when working with (a custom version of) Solr for 3 years

Transcript

1. lessons learned Solr @jeroenrosenberg
2. Frontend of Lucene
3. Lucene xml/json api + field types + caching + faceting + grouping +
4. Indexing
5. Indexing
6. Lucene's inverted index
7. Efficient when many docs share the same value
8. Field types
9. <field name="id" type="string" indexed="true" stored="true" required=" true" multiValued="false"/> <field name="name" type="string" indexed="false" stored="true" required="true" multiValued="false"/> Field type definition
10. <field name="id" type="string" indexed="true" stored="true" required=" true" multiValued="false"/> <field name="name" type="string" indexed="false" stored="true" required="true" multiValued="false"/> Field type definition
11. ... <fieldtype name="pdate" class="solr.DateField" sortMissingLast="true"/> ... <field name="date" type="pdate" indexed="false" stored="true"/> <field name="range_date" type="pdate" indexed="true" stored="false"/> <copyField source="date" dest="range_date"/> Field type definition
12. ... <fieldtype name="pdate" class="solr.DateField" sortMissingLast="true"/> ... <field name="date" type="pdate" indexed="false" stored="true"/> <field name="range_date" type="pdate" indexed="true" stored="false"/> <copyField source="date" dest="range_date"/> Field type definition
13. <dynamicField name="*_s" type="string" indexed="true" stored="true"/> Schemaless
14. Segments
15. Tune the merge factor
16. Max. # of segments Faster search, but slower indexing Faster indexing, but slower search
17. Don't commit. Ever.
18. Don't commit often.
19. Sharding
20. Manual distribution
21. foo foo foo core1 core2 core3 Index distributor replication
22. Look Ma, no downtime!
23. q=name:hotel1& shards=solr2:7070/solr/foo,solr3: 7070/solr/foo& partialResults=true Distributed search
24. <requestHandler name="distributedSearch" class="solr.SearchHandler" default="false"> <lst name="defaults"> <int name="rows">10</int> <str name="fl">*</str> <bool name="partialResults">true</bool> <str name="shards">solr2:7070/solr/foo,solr3:7070/solr/foo</str> </lst> </requestHandler> Distributed search config
25. <requestHandler name="distributedSearch" class="solr.SearchHandler" default="false"> <lst name="defaults"> <int name="rows">10</int> <str name="fl">*</str> <bool name="partialResults">true</bool> <str name="shards">solr2:7070/solr/foo,solr3:7070/solr/foo</str> </lst> </requestHandler> Distributed search config
26. <requestHandler name="distributedSearch" class="solr.SearchHandler" default false"> <lst name="defaults"> <int name="rows">10</int> <str name="fl">*</str> <bool name="partialResults">true</bool> <str name="shards">solr2:7070/solr/foo,solr3:7070/solr/foo</str> </lst> </requestHandler> Distributed search config
27. q=name:hotel1&qt=distributedSearch Distributed search
28. Caching
29. Field value Filter Document Query result
30. Document Field value Query result Filter Doc ids of results per filter query
31. Query result Document Filter Field value Field names (facets) mapped to mapping of doc ids to terms
32. Field value Filter Document Query result Ordered set of doc ids of top N results
33. Field value Filter Query result Document Stored fields for each doc
34. Autowarming
36. q=*:*&fq=country:AN&fq=duration:[1 TO *]& fq=date:[NOW TO 2013-07-01T00:00:00Z] Filter queries...
37. q=*:*&fq=country:AN&fq=duration:[1 TO *]& fq=date:[NOW TO 2013-07-01T00:00:00Z] Match all documents q=*:*
38. q=*:*&fq=country:AN&fq=duration:[1 TO *]& fq=date:[NOW TO 2013-07-01T00:00:00Z] Filter by field value fq=country:AN
39. q=*:*&fq=country:AN&fq=duration:[1 TO *]& fq=date:[NOW TO 2013-07-01T00:00:00Z] Range query with wildcard fq=duration:[1 TO *] range query using DateMath syntax fq=date:[NOW TO 2013-07-01T00:00:00Z]
40. q=*:*&rows=10000000 Getting all results
41. Faceting
42. rows=0&facet=true&facet.field=departureairport& facet.field=touroperator&facet.limit=-1& facet.mincount=1&f.touroperator.facet.limit=2 A facet query...
43. rows=0&facet=true&facet.field=departureairport& facet.field=touroperator&facet.limit=-1& facet.mincount=1&f.touroperator.facet.limit=2 Enable faceting facet=true
44. rows=0&facet=true&facet.field=departureairport& facet.field=touroperator&facet.limit=-1& facet.mincount=1&f.touroperator.facet.limit=2 rows=0 Suppress document results
45. rows=0&facet=true&facet.field=departureairport& facet.field=touroperator&facet.limit=-1& facet.mincount=1&f.touroperator.facet.limit=2 facet.field=departureairport Specify a field name ...and another one facet.field=touroperator
46. rows=0&facet=true&facet.field=departureairport& facet.field=touroperator&facet.limit=-1& facet.mincount=1&f.touroperator.facet.limit=2 Unlimited field values (globally) facet.limit=-1
47. rows=0&facet=true&facet.field=departureairport& facet.field=touroperator&facet.limit=-1& facet.mincount=1&f.touroperator.facet.limit=2 Unlimited field values (globally) facet.limit=-1 Basically, always a good idea
48. rows=0&facet=true&facet.field=departureairport& facet.field=touroperator&facet.limit=-1& facet.mincount=1&f.touroperator.facet.limit=2 Override global limit for specific field names f.touroperator.facet.limit=2
49. rows=0&facet=true&facet.field=departureairport& facet.field=touroperator&facet.limit=-1& facet.mincount=1&f.touroperator.facet.limit=2 At least 1 document per field value facet.mincount=1
50. q=*:*&fq={!tag=country}country:AN&facet=true& facet.field={!ex=country}country&facet.limit=-1& facet.mincount=1 Multi-select faceting...
51. q=*:*&fq={!tag=country}country:AN&facet=true& facet.field={!ex=country}country&facet.limit=-1& facet.mincount=1 fq={!tag=country}country:AN Tag a filter query... ...and exclude it for a field value facet.field={!ex=country}country
52. FACET ALL THE THINGS! FACET ALL THE THINGS!
53. Grouping
54. group=true&group.field=accoid& group.sort=price asc&sort=popularity asc& group.facets=UNGROUPED A grouping query...
55. group=true&group.field=accoid& group.sort=price asc&sort=popularity asc& group.facets=UNGROUPED Enable grouping group=true
56. group=true&group.field=accoid& group.sort=price asc&sort=popularity asc& group.facets=UNGROUPED Specify the field name group.field=accoid
57. group=true&group.field=accoid& group.sort=price asc&sort=popularity asc& group.facets=UNGROUPED Determines group head group.sort=price asc Determine order of document results sort=popularity asc
58. group=true&group.field=accoid& group.sort=price asc&sort=popularity asc& group.facets=UNGROUPED Determines group head group.sort=price asc Determine order of document results sort=popularity asc Only group heads are returned!
59. ONE DOES NOT SIMPLY EXPLAIN SOLR QUERIES ONE DOES NOT SIMPLY EXPLAIN SOLR QUERIES
60. debugQuery=true
61. Solr 4.3 is coming http://docs.lucidworks.com/display/solr/Major+Changes+from+Solr+3+to+Solr+4
62. Queries?