Profiling queries for fun and profit

Elasticsearch 2.2 received a query profiler. This meetup talk discussed how Lucene executes queries and then showed a demo of the new profiler.

Transcript

1. ‹#›
   @ZacharyTong
   Profiling queries for
   fun and profit
   

   View Slide

2. 2
   Elasticsearch 2.2 received a profiler!
   • Attempts to time execution of
   query components
   • Best-effort profiling
   • Expensive! Verbose!
   “SQL Explain for ES”
   https://www.elastic.co/guide/en/elasticsearch/reference/current/search-profile.html
   

   View Slide

3. 3
   Lucene search at 20,000 feet
   Documents
   Shard 1
   

   View Slide

4. 4
   Search request arrives
   Search
   Request
   Shard 1
   Documents
   

   View Slide

5. 5
   Shard builds a Collector
   Shard 1
   Search
   Request
   Collector
   Documents
   

   View Slide

6. 6
   Collector builds Scorer + Weight
   Shard 1
   Search
   Request
   Collector
   S W
   Documents
   

   View Slide

7. 7
   Collector walks over documents
   Shard 1
   Search
   Request
   Collector
   S W
   Documents
   

   View Slide

8. 8
   Shard 1
   Search
   Request
   Collector
   S W
   Documents
   Collector walks over documents
   

   View Slide

9. 9
   Shard 1
   Search
   Request
   Collector
   S W
   Documents
   Collector walks over documents
   

   View Slide

10. 10
    Shard 1
    Search
    Request
    Collector
    S W
    ….
    Documents
    Collector walks over documents
    

    View Slide

11. 11
    Shard 1
    Search
    Request
    Collector 2
    S W
    Documents
    A request can use multiple collectors
    

    View Slide

12. 12
    Shard 1
    Search
    Request
    Collector 2
    S W
    Documents
    A request can use multiple collectors
    

    View Slide

13. 13
    Shard 1
    Search
    Request
    Collector 2
    S W
    Documents
    A request can use multiple collectors
    

    View Slide

14. 14
    A query often has multiple scorers
    Shard 1
    Search
    Request
    Collector
    S W
    Documents
    S W
    

    View Slide

15. 15
    A query often has multiple scorers
    Shard 1
    Search
    Request
    Collector
    S W
    Documents
    S W
    ✓
    

    View Slide

16. 16
    A query often has multiple scorers
    Shard 1
    Search
    Request
    Collector
    S W
    Documents
    S W
    ✓
    

    View Slide

17. 17
    A query often has multiple scorers
    Shard 1
    Search
    Request
    Collector
    S W
    Documents
    S W
    ✓
    

    View Slide

18. 18
    A query often has multiple scorers
    Shard 1
    Search
    Request
    Collector
    S W
    Documents
    S W
    ✓
    ✓
    

    View Slide

19. 19
    A query often has multiple scorers
    Shard 1
    Search
    Request
    Collector
    S W
    Documents
    S W
    ✓
    ✓
    

    View Slide

20. 20
    A query often has multiple scorers
    Shard 1
    Search
    Request
    Collector
    S W
    Documents
    S W
    ✓
    ✓
    ✓
    Match!
    

    View Slide

21. 21
    Shard 1
    Search
    Request
    Collector 2
    S W
    Documents
    A request can use multiple collectors
    

    View Slide

22. 22
    Shard 1
    Search
    Request
    Collector 2
    S W
    Documents
    A request can use multiple collectors
    

    View Slide

23. 23
    Shard 1
    Search
    Request
    Collector 2
    S W
    Documents
    A request can use multiple collectors
    

    View Slide

24. 24
    And a request can hit multiple shards
    Search
    Request
    Shard 1
    Documents
    Shard 2
    Documents
    

    View Slide

25. 25
    Which means the
    output is verbose
    

    View Slide

26. 26
    But maybe some day…
    

    View Slide

27. Demo Time!
    27
    

    View Slide

28. ‹#›
    Please attribute Elastic with a link to elastic.co
    Except where otherwise noted, this work is licensed under
    http://creativecommons.org/licenses/by-nd/4.0/
    Creative Commons and the double C in a circle are
    registered trademarks of Creative Commons in the United States and other countries.
    Third party marks and brands are the property of their respective holders.
    28
    

    View Slide