Profiling Elasticsearch Queries for Fun and Profit

Writing a good query can require experience, intuition, and sometimes, divination. This talk will cover how the Elasticsearch DSL translates to Lucene queries, the stages of query execution, and how the new query profiler can be used to identify slow components and improve performance.

Transcript

1. ‹#› @ZacharyTong Profiling queries for fun and profit
2. 2 Elasticsearch 2.2 received a profiler! • Attempts to time execution of query components • Best-effort profiling • Expensive! Verbose! “SQL Explain for ES” https://www.elastic.co/guide/en/elasticsearch/reference/current/search-profile.html
3. 3 Lucene search at 20,000 feet Documents Shard 1
4. 4 Search request arrives Search Request Shard 1 Documents
5. 5 Shard builds a Collector Shard 1 Search Request Collector Documents
6. 6 Collector builds Scorer + Weight Shard 1 Search Request Collector S W Documents
7. 7 Collector walks over documents Shard 1 Search Request Collector S W Documents
8. 8 Shard 1 Search Request Collector S W Documents Collector walks over documents
9. 9 Shard 1 Search Request Collector S W Documents Collector walks over documents
10. 10 Shard 1 Search Request Collector S W …. Documents Collector walks over documents
11. 11 Shard 1 Search Request Collector 2 S W Documents A request can use multiple collectors
12. 12 Shard 1 Search Request Collector 2 S W Documents A request can use multiple collectors
13. 13 Shard 1 Search Request Collector 2 S W Documents A request can use multiple collectors
14. 14 And a request can hit multiple shards Search Request Shard 1 Documents Shard 2 Documents
15. 15 Which means the output is verbose
16. 16 But maybe some day…
17. Demo Time! 17
18. ‹#› Please attribute Elastic with a link to elastic.co Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 18