Open Source RESTful Search: Solr

Solr is an open source search platform built on top of the Apache Lucene project. Solr wraps Lucene with a nice RESTful API, and adds other features like faceted search, grouping, field types, caching, xml configuration, an administration interface, and the ability to scale with distributed search. This session will start with the basics of Solr and Lucene. We’ll then touch on some of the advanced/awesome features of Solr and also look at how to extend Solr with additional pluggable functionality. This will of course be supplemented with working examples and demos.

Transcript

1. Open Source RESTful Search Scott Smerchek @smerchek scottsmerchek.com
2. Lucene Lucene Core: – indexing and search technology – ranking/scoring results – spellchecking – hit highlighting – advanced analysis/tokenization – long history (est. 2000)
3. Inverted Index
4. Lucene, but better • Solr: – built on Lucene Core – REST API (xml/json) – explicit field types – caching – faceting/grouping – highly configurable/extensible – replication/distribution (scalable)
5. Indexing Just a POST curl “http://localhost:8983/solr/update/json?commit=true” --data-binary @listings.json -H “Content-type:application/json”
6. Query Syntax: Term test “You need to test your code.” test* “Your code should be testable.”
7. Query Syntax: Field title:"The Right Way" AND text:go { id: 1, title: “The Right Way”, description: “Go the right way, or else.” } +title:"The Right Way" +text:go
8. Query Syntax: Range quantity:[2 TO *] { id: 2, title: “Things”, quantity: 2 }
9. Query Syntax: Range quantity:[2 TO *] { id: 2, title: “Things”, quantity: 4 }
10. Query Syntax: Range quantity:[2 TO *] { id: 2, title: “Things”, quantity: 8 }
11. Query Syntax: Range quantity:[2 TO *] { id: 2, title: “Things”, quantity: 1 }
12. Query Syntax: Range creation:[NOW/DAY-30DAY TO *] { id: 2, title: “Things”, quantity: 2, creation: "2012-04-26T02:42:07Z" }
13. Query Syntax: Range creation:[NOW/DAY-30DAY TO *] { id: 2, title: “Things”, quantity: 2, creation: "2012-03-29T05:22:06Z" }
14. Query Syntax: Range creation:[NOW/DAY-30DAY TO *] { id: 2, title: “Things”, quantity: 2, creation: "2012-04-21T11:36:52Z" }
15. Query Syntax: Range creation:[NOW/DAY-30DAY TO *] { id: 2, title: “Things”, quantity: 2, creation: "2012-03-15T11:36:52Z" }
16. A Solr Query http://localhost:8983/solr/select? q=*:*& start=0& rows=10& fq=tags:Jewelry& debugQuery=true& wt=json
17. DebugQuery: Explaining http://localhost:8983/solr/select? q=beautiful& debugQuery=true 71407194: 0.6276805 = (MATCH) weight(text:beautiful in 3948), result of: 0.6276805 = fieldWeight in 3948, product of: 1.4142135 = tf(freq=2.0), with freq of: 2.0 = termFreq=2.0 2.8405576 = idf(docFreq=1972, maxDocs=12430) 0.15625 = fieldNorm(doc=3948)
18. A Solr DisMax Query http://localhost:8983/solr/select? q=beautiful& defType=edismax& qf=title^2,description& tie=.1
19. Faceting Source: newegg.com - Internal SSD listings
20. A Solr Query: Facets http://localhost:8983/solr/select? q=*:*& facet=true& facet.field=who_made& facet.sort=count& f.who_made.facet.method=enum
21. A Solr Query: Date Facets http://localhost:8983/solr/select? q=*:*& facet=true& facet.date=original_creation_tsz& facet.date.start=NOW/YEAR-4YEARS& facet.date.gap=+1YEAR& facet.date.end=NOW/YEAR& facet.date.other=before
22. A Solr Query: Term Suggest http://localhost:8983/solr/select? q=tags:Clutch& facet=true& rows=0& facet.field=text& facet.limit=10& facet.mincount=1& facet.prefix=be
23. A Solr Query: Highlighting http://localhost:8983/solr/select? q=beautiful& hl=true& hl.fl=title,description& hl.simple.pre=<em>& hl.simple.post=</em>
24. A Solr Query: Grouping http://localhost:8983/solr/select? q=beautiful& group=true& group.field=user_id& group.limit=3
25. A Solr Query: Statistics http://localhost:8983/solr/select? q=*:*& rows=0& stats=true& stats.field=quantity
26. Solr Cores *Use multiple cores* • Rebuild an index without downtime • Test configuration changes • Merge indexes of two cores into a 3rd
27. Solr Caching • Filter Cache – Caches unordered sets of doc ids that match a key (query) – Used for results of fq filter queries and faceting • Field Value Cache – Primarily used by faceting • Query Result Cache – Stores ordered sets of doc ids • Document Cache – Stores Lucene Document objects • User/Generic Caches – Generic object cache for custom Solr plugins
28. Solr in applications • Ruby • .NET • Java • Python • Javascript • Anything that make a HTTP request…
29. Solr in production • Best on linux • Lots of RAM • Manage config.xml with git • Restrict access by IP • Servlet monitoring solution • Localize fields with field alias (dismax) • Increase memory allocated to the JVM – java -Xms512M -Xmx1024M -jar start.jar
30. Other Features • NearRealtime Searching • Payloads • SolrCloud (distributed) • Custom plugins • (Geo)Spatial Search • Joins • Indexing Rich Documents (PDF, etc) • Clustering
31. Solr Resources • Apache Solr 3 Enterprise Search Server • Lucene in Action (2nd Edition) • http://wiki.apache.org/solr/ • http://wiki.apache.org/solr/SolrResources