Profiling queries for fun and profit

Profiling queries for fun and profit

Elasticsearch 2.2 received a query profiler. This meetup talk discussed how Lucene executes queries and then showed a demo of the new profiler.

Deca372d0ae79d8d84f90e1e68bd1618?s=128

Zachary Tong

March 24, 2016
Tweet

Transcript

  1. ‹#› @ZacharyTong Profiling queries for fun and profit

  2. 2 Elasticsearch 2.2 received a profiler! • Attempts to time

    execution of query components • Best-effort profiling • Expensive! Verbose! “SQL Explain for ES” https://www.elastic.co/guide/en/elasticsearch/reference/current/search-profile.html
  3. 3 Lucene search at 20,000 feet Documents Shard 1

  4. 4 Search request arrives Search Request Shard 1 Documents

  5. 5 Shard builds a Collector Shard 1 Search Request Collector

    Documents
  6. 6 Collector builds Scorer + Weight Shard 1 Search Request

    Collector S W Documents
  7. 7 Collector walks over documents Shard 1 Search Request Collector

    S W Documents
  8. 8 Shard 1 Search Request Collector S W Documents Collector

    walks over documents
  9. 9 Shard 1 Search Request Collector S W Documents Collector

    walks over documents
  10. 10 Shard 1 Search Request Collector S W …. Documents

    Collector walks over documents
  11. 11 Shard 1 Search Request Collector 2 S W Documents

    A request can use multiple collectors
  12. 12 Shard 1 Search Request Collector 2 S W Documents

    A request can use multiple collectors
  13. 13 Shard 1 Search Request Collector 2 S W Documents

    A request can use multiple collectors
  14. 14 A query often has multiple scorers Shard 1 Search

    Request Collector S W Documents S W
  15. 15 A query often has multiple scorers Shard 1 Search

    Request Collector S W Documents S W ✓
  16. 16 A query often has multiple scorers Shard 1 Search

    Request Collector S W Documents S W ✓
  17. 17 A query often has multiple scorers Shard 1 Search

    Request Collector S W Documents S W ✓
  18. 18 A query often has multiple scorers Shard 1 Search

    Request Collector S W Documents S W ✓ ✓
  19. 19 A query often has multiple scorers Shard 1 Search

    Request Collector S W Documents S W ✓ ✓
  20. 20 A query often has multiple scorers Shard 1 Search

    Request Collector S W Documents S W ✓ ✓ ✓ Match!
  21. 21 Shard 1 Search Request Collector 2 S W Documents

    A request can use multiple collectors
  22. 22 Shard 1 Search Request Collector 2 S W Documents

    A request can use multiple collectors
  23. 23 Shard 1 Search Request Collector 2 S W Documents

    A request can use multiple collectors
  24. 24 And a request can hit multiple shards Search Request

    Shard 1 Documents Shard 2 Documents
  25. 25 Which means the output is verbose

  26. 26 But maybe some day…

  27. Demo Time! 27

  28. ‹#› Please attribute Elastic with a link to elastic.co Except

    where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 28