Open Source RESTful Search: Solr

Transcript

1. Open Source RESTful Search Scott Smerchek @smerchek scottsmerchek.com

2. Lucene Lucene Core: – indexing and search technology – ranking/scoring

   results – spellchecking – hit highlighting – advanced analysis/tokenization – long history (est. 2000)

3. Inverted Index

4. Lucene, but better • Solr: – built on Lucene Core

   – REST API (xml/json) – explicit field types – caching – faceting/grouping – highly configurable/extensible – replication/distribution (scalable)

5. Indexing Just a POST curl “http://localhost:8983/solr/update/json?commit=true” --data-binary @listings.json -H “Content-type:application/json”

6. Query Syntax: Term test “You need to test your code.”

   test* “Your code should be testable.”

7. Query Syntax: Field title:"The Right Way" AND text:go { id:

   1, title: “The Right Way”, description: “Go the right way, or else.” } +title:"The Right Way" +text:go

8. Query Syntax: Range quantity:[2 TO *] { id: 2, title:

   “Things”, quantity: 2 }

9. Query Syntax: Range quantity:[2 TO *] { id: 2, title:

   “Things”, quantity: 4 }

10. Query Syntax: Range quantity:[2 TO *] { id: 2, title:

    “Things”, quantity: 8 }

11. Query Syntax: Range quantity:[2 TO *] { id: 2, title:

    “Things”, quantity: 1 }

12. Query Syntax: Range creation:[NOW/DAY-30DAY TO *] { id: 2, title:

    “Things”, quantity: 2, creation: "2012-04-26T02:42:07Z" }

13. Query Syntax: Range creation:[NOW/DAY-30DAY TO *] { id: 2, title:

    “Things”, quantity: 2, creation: "2012-03-29T05:22:06Z" }

14. Query Syntax: Range creation:[NOW/DAY-30DAY TO *] { id: 2, title:

    “Things”, quantity: 2, creation: "2012-04-21T11:36:52Z" }

15. Query Syntax: Range creation:[NOW/DAY-30DAY TO *] { id: 2, title:

    “Things”, quantity: 2, creation: "2012-03-15T11:36:52Z" }

16. A Solr Query http://localhost:8983/solr/select? q=*:*& start=0& rows=10& fq=tags:Jewelry& debugQuery=true& wt=json

17. DebugQuery: Explaining http://localhost:8983/solr/select? q=beautiful& debugQuery=true 71407194: 0.6276805 = (MATCH) weight(text:beautiful

    in 3948), result of: 0.6276805 = fieldWeight in 3948, product of: 1.4142135 = tf(freq=2.0), with freq of: 2.0 = termFreq=2.0 2.8405576 = idf(docFreq=1972, maxDocs=12430) 0.15625 = fieldNorm(doc=3948)

18. A Solr DisMax Query http://localhost:8983/solr/select? q=beautiful& defType=edismax& qf=title^2,description& tie=.1

19. Faceting Source: newegg.com - Internal SSD listings

20. A Solr Query: Facets http://localhost:8983/solr/select? q=*:*& facet=true& facet.field=who_made& facet.sort=count& f.who_made.facet.method=enum

21. A Solr Query: Date Facets http://localhost:8983/solr/select? q=*:*& facet=true& facet.date=original_creation_tsz& facet.date.start=NOW/YEAR-4YEARS&

    facet.date.gap=+1YEAR& facet.date.end=NOW/YEAR& facet.date.other=before

22. A Solr Query: Term Suggest http://localhost:8983/solr/select? q=tags:Clutch& facet=true& rows=0& facet.field=text&

    facet.limit=10& facet.mincount=1& facet.prefix=be

23. A Solr Query: Highlighting http://localhost:8983/solr/select? q=beautiful& hl=true& hl.fl=title,description& hl.simple.pre=<em>& hl.simple.post=</em>

24. A Solr Query: Grouping http://localhost:8983/solr/select? q=beautiful& group=true& group.field=user_id& group.limit=3

25. A Solr Query: Statistics http://localhost:8983/solr/select? q=*:*& rows=0& stats=true& stats.field=quantity

26. Solr Cores *Use multiple cores* • Rebuild an index without

    downtime • Test configuration changes • Merge indexes of two cores into a 3rd

27. Solr Caching • Filter Cache – Caches unordered sets of

    doc ids that match a key (query) – Used for results of fq filter queries and faceting • Field Value Cache – Primarily used by faceting • Query Result Cache – Stores ordered sets of doc ids • Document Cache – Stores Lucene Document objects • User/Generic Caches – Generic object cache for custom Solr plugins

28. Solr in applications • Ruby • .NET • Java •

    Python • Javascript • Anything that make a HTTP request…

29. Solr in production • Best on linux • Lots of

    RAM • Manage config.xml with git • Restrict access by IP • Servlet monitoring solution • Localize fields with field alias (dismax) • Increase memory allocated to the JVM – java -Xms512M -Xmx1024M -jar start.jar

30. Other Features • NearRealtime Searching • Payloads • SolrCloud (distributed)

    • Custom plugins • (Geo)Spatial Search • Joins • Indexing Rich Documents (PDF, etc) • Clustering

31. Solr Resources • Apache Solr 3 Enterprise Search Server •

    Lucene in Action (2nd Edition) • http://wiki.apache.org/solr/ • http://wiki.apache.org/solr/SolrResources