Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Python DSL

Python DSL

Slides for Honza's talk at Elastic{on}

Elasticsearch Inc

March 11, 2015
Tweet

More Decks by Elasticsearch Inc

Other Decks in Technology

Transcript

  1. Python DSL
    Honza Král
    @honzakral

    View full-size slide

  2. { }
    DSL ?
    Don't you mean ORM?
    3

    View full-size slide

  3. { }
    Current State
    {
    "query": {
    "filtered": {
    "query": {
    "function_score": {
    "query": {
    "bool": {
    "must": [
    {"multi_match": {
    "fields": ["title^10", "body"], "query": "php"}},
    {"has_child": {
    "child_type": "answer",
    "query": {"match": {"body": "python"}}}}
    ],
    "must_not": [
    {"multi_match": {
    "fields": ["title", "body"],
    "query": "python"}}
    ]
    }
    },
    "field_value_factor": {"field": "rating"}
    }
    },
    "filter": {"range": {"creation_date": {"from": "2010-01-01"}}}
    }},
    4
    "highlight": {
    "fields": {
    "title": {"fragment_size" : 50},
    "body": {"fragment_size" : 50}
    }
    },
    "aggs": {
    "tags": {
    "terms": {"field": "tags"},
    "aggs": {
    "comment_avg": {
    "avg": {"field": "comment_count"}
    }
    }
    },
    "frequency": {
    "date_histogram": {
    "field": "creation_date",
    "interval": "month"
    }
    }
    }
    }
    JSON DSL

    View full-size slide

  4. { }
    Now add a filter to it!
    5

    View full-size slide

  5. { }
    Search Object
    s = Search(doc_type='question')
    6

    View full-size slide

  6. { }
    Simple Query
    s = s.query('multi_match',
    fields=['title^10', 'body'],
    query='php')
    7

    View full-size slide

  7. { }
    Compound Query
    s = s.query('has_child',
    child_type='answer',
    query=Q('match', body='python'))
    8

    View full-size slide

  8. { }
    Q shortcut
    {"has_child": {
    "child_type": "answer',
    "query": {"match": {"body": "python"}}}}
    Q({'has_child': {
    'child_type': 'answer',
    'query': {'match': {'body': 'python'}}}})
    Q('has_child', child_type='answer', query=Q('match', body='python'))
    HasChild(child_type='answer', query=Match(body='python'))
    9

    View full-size slide

  9. { }
    Query expressions
    Q(...) & Q(...) == Bool(must=[Q(...), Q(...)])
    Q(...) | Q(...) == Bool(should=[Q(...), Q(...)])
    ~Q(...) == Bool(must_not=[Q(..)])
    10

    View full-size slide

  10. { }
    Filter
    s = s.filter('range',
    creation_date={'from': date(2010, 1, 1)})
    11

    View full-size slide

  11. { }
    Exclude
    s = s.query(~Q('multi_match',
    fields=['title^10', 'body'],
    query='python'))
    12

    View full-size slide

  12. { }
    Manual query
    s.query = Q('function_score',
    query=s.query,
    field_value_factor={'field': 'rating'})
    13

    View full-size slide

  13. { }
    Aggregations
    s.aggs.bucket('tags', 'terms', field='tags')\
    .metric('comment_avg', 'avg', field='comment_count')
    s.aggs.bucket('frequency', 'date_histogram',
    field='creation_date',
    interval='month')
    14

    View full-size slide

  14. { }
    Highlight ...
    s = s.highlight('title', 'body',
    fragment_size=50)
    15

    View full-size slide

  15. { }
    Migration path
    s = Search.from_dict(my_glorious_query)
    s = s.filter('term', tag='published')
    my_glorious_query = s.to_dict()
    16
    query at a time

    View full-size slide

  16. { }
    Response
    response = s.execute()
    for hit in response:
    print(hit.meta.score, hit.title)
    for tag in response.aggregations.tags.buckets:
    print(tag.key, tag.avg_comments.value)
    17
    No more brackets!

    View full-size slide

  17. { }
    Persistence
    From Mapping to Model-like DocTypes
    18

    View full-size slide

  18. { }
    Mapping DSL
    m = Mapping('article')
    m.field('published_from', Date())
    m.field('title',
    String(fields={'raw': String(index='not_analyzed')}))
    m.field('comments', Nested())
    m['comments'].property('author', String())
    m.save('index-name')
    m.update_from_es('index-name')
    19

    View full-size slide

  19. { }
    DocType
    class Article(DocType):
    title = String()
    created_date = Date()
    comments = Nested(properties={'author': String()})
    class Meta:
    index = 'blog'

    def save(self, **kwargs):
    self.created_date = now()
    super().save(**kwargs)
    Article.init()
    Article.search()...
    Search(doc_type=Article)
    20

    View full-size slide

  20. { }
    Configuration
    21

    View full-size slide

  21. { }
    Connections
    connections.configure(
    default={'hosts': ['localhost'], 'sniff_on_start': True},
    logging={
    'hosts': ['log1:9200', 'log2:9200'],
    'timeout': 30,
    'sniff_timeout': 120})
    Search(using='logging')
    es = connections.get_connection()
    es.indices.delete(index='_all')
    22

    View full-size slide

  22. { }
    More DSLs!
    Index
    Analyzers
    Settings
    ...
    24

    View full-size slide

  23. { }
    FacetedSearch ?
    class MySiteSearch(FacetedSearch):
    doc_type = [Article, Comment]
    fields = ['title', 'body']
    published = DateHistogram(
    interval='week',
    field='published_date')
    category = Term(field='category')
    25
    Definition ???

    View full-size slide

  24. { }
    FacetedSearch ?
    s = MySiteSearch('python', category='blog')
    for hit in s:
    print(h.meta.score, h.title)
    cat_facet = s.facets['category']
    for name, count in cat_facet:
    mask = '%s: %d'
    if name == cat_facet.selected:
    mask = '%s: %d'
    print(mask % (name, count))
    26
    Usage ????

    View full-size slide

  25. { }
    Django integration ?
    Model -> DocType
    signal handlers to update
    management command to sync
    FacetedSearch -> Form
    view + template pattern
    27

    View full-size slide

  26. { }
    Thank you!
    @honzakral

    View full-size slide

  27. { }
    This work is licensed under the Creative Commons
    Attribution-NoDerivatives 4.0 International License.
    To view a copy of this license, visit:
    http://creativecommons.org/licenses/by-nd/4.0/
    or send a letter to:
    Creative Commons
    PO Box 1866
    Mountain View, CA 94042
    USA
    CC-BY-ND 4.0 29

    View full-size slide