Elasticsearch DSL

Slides from DjangoCon US 2014 by Honza Král, presenting on the Python DSL for Elasticsearch.

Transcript

1. Honza Král @honzakral Elasticsearch DSL
2. Elasticsearch
3. Distributed Search Engine Open Source
 
 Distributed
 
 Document-based
 
 Based on Lucene 
 JSON over HTTP
4. Document based JSON
 Dynamic Schema
 Some Relationships Nested Parent/Child
5. {! "id": 7635,! "accepted_answer_id": 7641,! "answer_count": 9,! "title": "Are you able to close your eyes and focus/think just on your code?",! "body": "How do I ......?",! "creation_date": "2010-09-27T19:16:57.757",! "closed_date": "2011-11-13T12:12:05.937",! "comment_count": 2,! "comments": [{! "creation_date": "2010-09-27T19:31:27.200",! "id": 9372,! "owner": { "display_name": "sange", "id": 3092 },! "post_id": 7635,! "text": "I sometimes close my eyes or stare at something ....."! }, {......}],! "favorite_count": 2,! "last_activity_date": "2010-09-28T00:28:08.393",! "owner": { "display_name": "flow", "id": 3761 },! "rating": 6,! "tags": [ "focus", "concentration" ],! "view_count": 368! } StackOverflow Question
6. Query DSL
7. queries & filters Abstract Syntax Tree
8. Queries (unstructured) Core Queries match, multi_match, phrase fuzzy, regexp, wildcard Compound Queries filtered bool function score ! Relies on analysis, produce score (relevancy)
9. Filters (structured) Core filters term, range, exists geo_distance, geo_bbox script Compound filters bool and/or/not ! Fast, cacheable
10. ! {! "query": {! "filtered": {! "query": {! "bool": {! "must": [! {"multi_match": {"fields": ["title^10", "body"], "query": "php"}},! {"has_child": { "child_type": "answer", "query": {"match": {"body": "python"}}}}! ],! "must_not": {"multi_match": {"fields": ["title", "body"], "query": "python"}}! }! },! "filter": {"range": {"creation_date": {"from": "2012-01-01"}}}! }! },! "aggs": {! "tags": {! "terms": {"field": "tags"},! "aggs": {! "comment_avg": {"avg": {"field": "comment_count"}}! }! },! "frequency": {"date_histogram": {"field": "creation_date", "interval": "month"}}! }! } Example
11. Python
12. how hard can it be? HTTP
13. Elasticsearch Distributed load balancing, node failure, node discovery ! Different deployment environments nginx, thrift, PaaS ! REST API 96 API endpoints, 672 parameters, escaping, encoding
14. elasticsearch-py Low Level 1-to-1 to REST No Excuses No opinions Extensible/Modular All APIs
15. dict -> JSON from elasticsearch import Elasticsearch! ! es = Elasticsearch()! result = es.search(body={! "query": {! "filtered": {! "query": {! "bool": {! "must": [{"match": {"title": "python"}}],! "must_not": [{"match": {"title": "ruby"}}]! ! }! },! "filter": {! "range": {"creation_date": {"from": "2012-01-01"}}! }! }! }! })
16. elasticsearch-dsl from elasticsearch_dsl import Search, Q! ! # create Search, bind it to client! s = Search(using=es)! ! # querying twice will combine queries! # or combine manually: s.query(Q() & ~Q())! s = s.query('match', title='python').query(~Q('match', title='ruby'))! ! # filter will turn it to filtered query! s = s.filter('range', creation_date={"from": date(2012, 1, 1)})! ! # get a fancy result object!! result = s.execute()
17. Design
18. Say no to { [ ] }!
19. Automatic composition
20. We are not SQL Stay true to Query DSL
21. elasticsearch-dsl from elasticsearch_dsl import Search, Q! ! # create Search, bind it to client! s = Search(using=es)! ! # querying twice will combine queries! # or combine manually: s.query(Q() & ~Q())! s = s.query('match', title='python').query(~Q('match', title='ruby'))! ! # filter will turn it to filtered query! s = s.filter('range', creation_date={"from": date(2012, 1, 1)})! ! # get a fancy result object!! result = s.execute()
22. Q/F/A Shortcut for creating Queries/Filters/Aggregations ! Can use dicts or name + params ! Has elementary boolean logic Q('match', title='python') == Q({'match': {'title': 'python'}}) Q(1) & Q(2) == Q('bool', must=[Q(1), Q(2)])! Q(1) | (Q2) | Q(3) == Q('bool', should=[Q(1), Q(2), Q(3)])! ~F(1) == F('bool', must=[F(1)]) Q('match', title='python') == Match(title='python')
23. .query().filter() chaining Copy is made on each change ! Same for other methods ! Except Aggs! s[0:10], s.using(es), s.index('today', 'yesterday'), ... s.aggs.bucket('per_tag').metric('avg').metric('max')! s.aggs.bucket('per_country').bucket('per_tag').metric('avg')! s.aggs['per_country'].metric(...) s2 = s1.query(Q(1))! s1 != s2
24. Response Response object is returned:
 
 
 You can iterate over it and get hits:
 
 
 Aggregations can be accessed: for h in response:! print(h._meta.id, h.title) top_tag = response.aggregations.per_tag.buckets[0] response = s.execute()! if not response.success(): print("Partial results!")
25. Migration Path query = {! "query": {! "filtered": {! "query": {! "bool": {! "must": [{"match": {"title": "python"}}],! "must_not": [{"match": {"title": "ruby"}}]! ! }! },! "filter": {! "range": {"creation_date": {"from": date(2012, 1, 1)}}! }! }! }! }! ! s = Search.from_dict(query)! s = ...! query = s.to_dict()
26. Demo time! STOP
27. Django
28. Put your data in.... def sync_to_es(instance, **kwargs):! es.index(! index=settings.ES_INDEX,! doc_type=str(instance._meta),! id=instance.pk,! body=instance.to_json()) from elasticsearch.helpers import bulk! ! es.indices.put_mapping(index=settings.ES_INDEX, body={...})! ! bulk(es,! map(methodcaller('to_dict'), Model.objects.iterator()),! index=settings.ES_INDEX,! doc_type=str(Model._meta)) Bulk load - mgmt command Sync after change - signals
29. ...query as usual
30. Future
31. Mapping DSL
32. Persistence layer
33. Django integration
34. @robhudson @willcage Thanks!
35. Thank You! 
 Honza Král twitter: @honzakral email: honza.kral@elasticsearch.com ! Support: http://elasticsearch.com/support Training: http://training.elasticsearch.com/ We are hiring: http://elasticsearch.com/about/jobs/