Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Profiling queries for fun and profit

Profiling queries for fun and profit

Elasticsearch 2.2 received a query profiler. This meetup talk discussed how Lucene executes queries and then showed a demo of the new profiler.

Zachary Tong

March 24, 2016
Tweet

More Decks by Zachary Tong

Other Decks in Technology

Transcript

  1. ‹#›
    @ZacharyTong
    Profiling queries for
    fun and profit

    View full-size slide

  2. 2
    Elasticsearch 2.2 received a profiler!
    • Attempts to time execution of
    query components
    • Best-effort profiling
    • Expensive! Verbose!
    “SQL Explain for ES”
    https://www.elastic.co/guide/en/elasticsearch/reference/current/search-profile.html

    View full-size slide

  3. 3
    Lucene search at 20,000 feet
    Documents
    Shard 1

    View full-size slide

  4. 4
    Search request arrives
    Search
    Request
    Shard 1
    Documents

    View full-size slide

  5. 5
    Shard builds a Collector
    Shard 1
    Search
    Request
    Collector
    Documents

    View full-size slide

  6. 6
    Collector builds Scorer + Weight
    Shard 1
    Search
    Request
    Collector
    S W
    Documents

    View full-size slide

  7. 7
    Collector walks over documents
    Shard 1
    Search
    Request
    Collector
    S W
    Documents

    View full-size slide

  8. 8
    Shard 1
    Search
    Request
    Collector
    S W
    Documents
    Collector walks over documents

    View full-size slide

  9. 9
    Shard 1
    Search
    Request
    Collector
    S W
    Documents
    Collector walks over documents

    View full-size slide

  10. 10
    Shard 1
    Search
    Request
    Collector
    S W
    ….
    Documents
    Collector walks over documents

    View full-size slide

  11. 11
    Shard 1
    Search
    Request
    Collector 2
    S W
    Documents
    A request can use multiple collectors

    View full-size slide

  12. 12
    Shard 1
    Search
    Request
    Collector 2
    S W
    Documents
    A request can use multiple collectors

    View full-size slide

  13. 13
    Shard 1
    Search
    Request
    Collector 2
    S W
    Documents
    A request can use multiple collectors

    View full-size slide

  14. 14
    A query often has multiple scorers
    Shard 1
    Search
    Request
    Collector
    S W
    Documents
    S W

    View full-size slide

  15. 15
    A query often has multiple scorers
    Shard 1
    Search
    Request
    Collector
    S W
    Documents
    S W

    View full-size slide

  16. 16
    A query often has multiple scorers
    Shard 1
    Search
    Request
    Collector
    S W
    Documents
    S W

    View full-size slide

  17. 17
    A query often has multiple scorers
    Shard 1
    Search
    Request
    Collector
    S W
    Documents
    S W

    View full-size slide

  18. 18
    A query often has multiple scorers
    Shard 1
    Search
    Request
    Collector
    S W
    Documents
    S W


    View full-size slide

  19. 19
    A query often has multiple scorers
    Shard 1
    Search
    Request
    Collector
    S W
    Documents
    S W


    View full-size slide

  20. 20
    A query often has multiple scorers
    Shard 1
    Search
    Request
    Collector
    S W
    Documents
    S W



    Match!

    View full-size slide

  21. 21
    Shard 1
    Search
    Request
    Collector 2
    S W
    Documents
    A request can use multiple collectors

    View full-size slide

  22. 22
    Shard 1
    Search
    Request
    Collector 2
    S W
    Documents
    A request can use multiple collectors

    View full-size slide

  23. 23
    Shard 1
    Search
    Request
    Collector 2
    S W
    Documents
    A request can use multiple collectors

    View full-size slide

  24. 24
    And a request can hit multiple shards
    Search
    Request
    Shard 1
    Documents
    Shard 2
    Documents

    View full-size slide

  25. 25
    Which means the
    output is verbose

    View full-size slide

  26. 26
    But maybe some day…

    View full-size slide

  27. Demo Time!
    27

    View full-size slide

  28. ‹#›
    Please attribute Elastic with a link to elastic.co
    Except where otherwise noted, this work is licensed under
    http://creativecommons.org/licenses/by-nd/4.0/
    Creative Commons and the double C in a circle are
    registered trademarks of Creative Commons in the United States and other countries.
    Third party marks and brands are the property of their respective holders.
    28

    View full-size slide