Profiling queries for fun and profit

Elasticsearch 2.2 received a query profiler. This meetup talk discussed how Lucene executes queries and then showed a demo of the new profiler.

Transcript

1. ‹#› @ZacharyTong Profiling queries for fun and profit
2. 2 Elasticsearch 2.2 received a profiler! • Attempts to time execution of query components • Best-effort profiling • Expensive! Verbose! “SQL Explain for ES” https://www.elastic.co/guide/en/elasticsearch/reference/current/search-profile.html
3. 3 Lucene search at 20,000 feet Documents Shard 1
4. 4 Search request arrives Search Request Shard 1 Documents
5. 5 Shard builds a Collector Shard 1 Search Request Collector Documents
6. 6 Collector builds Scorer + Weight Shard 1 Search Request Collector S W Documents
7. 7 Collector walks over documents Shard 1 Search Request Collector S W Documents
8. 8 Shard 1 Search Request Collector S W Documents Collector walks over documents
9. 9 Shard 1 Search Request Collector S W Documents Collector walks over documents
10. 10 Shard 1 Search Request Collector S W …. Documents Collector walks over documents
11. 11 Shard 1 Search Request Collector 2 S W Documents A request can use multiple collectors
12. 12 Shard 1 Search Request Collector 2 S W Documents A request can use multiple collectors
13. 13 Shard 1 Search Request Collector 2 S W Documents A request can use multiple collectors
14. 14 A query often has multiple scorers Shard 1 Search Request Collector S W Documents S W
15. 15 A query often has multiple scorers Shard 1 Search Request Collector S W Documents S W ✓
16. 16 A query often has multiple scorers Shard 1 Search Request Collector S W Documents S W ✓
17. 17 A query often has multiple scorers Shard 1 Search Request Collector S W Documents S W ✓
18. 18 A query often has multiple scorers Shard 1 Search Request Collector S W Documents S W ✓ ✓
19. 19 A query often has multiple scorers Shard 1 Search Request Collector S W Documents S W ✓ ✓
20. 20 A query often has multiple scorers Shard 1 Search Request Collector S W Documents S W ✓ ✓ ✓ Match!
21. 21 Shard 1 Search Request Collector 2 S W Documents A request can use multiple collectors
22. 22 Shard 1 Search Request Collector 2 S W Documents A request can use multiple collectors
23. 23 Shard 1 Search Request Collector 2 S W Documents A request can use multiple collectors
24. 24 And a request can hit multiple shards Search Request Shard 1 Documents Shard 2 Documents
25. 25 Which means the output is verbose
26. 26 But maybe some day…
27. Demo Time! 27
28. ‹#› Please attribute Elastic with a link to elastic.co Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 28