Scale Your Metrics with Elasticsearch

Scale Your Metrics with Elasticsearch

"Only accept features that scale" is one of Elasticsearch's engineering principles. So how do we scale metrics stored in Elasticsearch? And is that even possible on a full-text search engine? This talk explores:
* How are metrics stored in Elasticsearch and how does this translate to disk use as well as query performance?
* What does an efficient multi-tier architecture look like to balance speed for today's data against density for older metrics?
* How can you compress old data and what does the mathematical model look like for different metrics?

We are trying all of this hands-on during the talk, since this has become much simpler recently.

Ce4685da897c912aa41a815435b40a5a?s=128

Philipp Krenn

May 05, 2019
Tweet

Transcript

  1. 2.
  2. 3.

    $ curl http://localhost:9200 { "name" : "elasticsearch-hot", "cluster_name" : "metrics-cluster",

    "cluster_uuid" : "06nHPLLgTrmZEpYli6JW5w", "version" : { "number" : "6.5.0", "build_flavor" : "default", "build_type" : "tar", "build_hash" : "c53b7d3", "build_date" : "2018-11-08T21:28:50.577384Z", "build_snapshot" : false, "lucene_version" : "7.5.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }
  3. 4.
  4. 5.
  5. 6.
  6. 7.
  7. 8.
  8. 9.
  9. 10.
  10. 11.
  11. 13.
  12. 22.

    Append-Only Optimization IDs assigned by coordinating node Fast add instead

    of the slow update https://github.com/elastic/elasticsearch/issues/19813
  13. 28.
  14. 29.

    https://github.com/elastic/beats/blob/master/metricbeat/module/system/load/_meta/ fields.yml - name: load type: group description: > CPU

    load averages. release: ga fields: - name: "1" type: scaled_float scaling_factor: 100 description: > Load average for the last minute. - name: "5" type: scaled_float scaling_factor: 100 description: > Load average for the last 5 minutes. ...
  15. 35.
  16. 36.
  17. 42.

    Features & Order https://github.com/elastic/elasticsearch/blob/7.0/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/TimeseriesLifecycleType.java static final List<String> ORDERED_VALID_HOT_ACTIONS = Arrays.asList(

    SetPriorityAction.NAME, UnfollowAction.NAME, RolloverAction.NAME ); static final List<String> ORDERED_VALID_WARM_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, ReadOnlyAction.NAME, AllocateAction.NAME, ShrinkAction.NAME, ForceMergeAction.NAME ); static final List<String> ORDERED_VALID_COLD_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, AllocateAction.NAME, FreezeAction.NAME ); static final List<String> ORDERED_VALID_DELETE_ACTIONS = Arrays.asList( DeleteAction.NAME );
  18. 47.