Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scale Your Metrics with Elasticsearch

Scale Your Metrics with Elasticsearch

"Only accept features that scale" is one of Elasticsearch's engineering principles. So how do we scale metrics stored in Elasticsearch? And is that even possible on a full-text search engine? This talk explores:
* How are metrics stored in Elasticsearch and how does this translate to disk use as well as query performance?
* What does an efficient multi-tier architecture look like to balance speed for today's data against density for older metrics?
* How can you compress old data and what does the mathematical model look like for different metrics?

We are trying all of this hands-on during the talk, since this has become much simpler recently.

Philipp Krenn

May 05, 2019
Tweet

More Decks by Philipp Krenn

Other Decks in Programming

Transcript

  1. Scale Your Metrics
    with Elasticsearch
    Philipp Krenn@xeraa

    View full-size slide

  2. $ curl http://localhost:9200
    {
    "name" : "elasticsearch-hot",
    "cluster_name" : "metrics-cluster",
    "cluster_uuid" : "06nHPLLgTrmZEpYli6JW5w",
    "version" : {
    "number" : "6.5.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "c53b7d3",
    "build_date" : "2018-11-08T21:28:50.577384Z",
    "build_snapshot" : false,
    "lucene_version" : "7.5.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
    },
    "tagline" : "You Know, for Search"
    }

    View full-size slide

  3. I'm not going to use a search engine
    for metrics.
    — Too often

    View full-size slide

  4. Agenda
    Building Blocks
    Architecture
    Demo

    View full-size slide

  5. Building Blocks

    View full-size slide

  6. Only accept features that scale.
    — https://github.com/elastic/engineering/blob/master/
    development_constitution.md

    View full-size slide

  7. Horizontal Scaling
    Shards
    Replication
    Writes & Reads

    View full-size slide

  8. Cluster, Node, Index, Shard

    View full-size slide

  9. Write
    Coordinating Node, Hash, Primary, Replica(s)

    View full-size slide

  10. Get
    Coordinating Node, Hash, Shard

    View full-size slide

  11. Search
    Coordinating Node, Query then Fetch

    View full-size slide

  12. Append-Only
    Optimization
    IDs assigned by coordinating node
    Fast add instead of the slow update
    https://github.com/elastic/elasticsearch/issues/19813

    View full-size slide

  13. Lucene Segments
    index.refresh_interval: 1s
    index.search.idle.after: 30s
    Iff default refresh (added in 7.0)

    View full-size slide

  14. Storage Compression
    LZ4 (default)
    DEFLATE (best_compression)

    View full-size slide

  15. BKD Trees
    Points in Lucene (added in 6.0)

    View full-size slide

  16. Integer (1D 4 byte point) vs legacy IntField

    View full-size slide

  17. Half & Scaled Floats

    View full-size slide

  18. https://github.com/elastic/beats/blob/master/metricbeat/module/system/load/_meta/
    fields.yml
    - name: load
    type: group
    description: >
    CPU load averages.
    release: ga
    fields:
    - name: "1"
    type: scaled_float
    scaling_factor: 100
    description: >
    Load average for the last minute.
    - name: "5"
    type: scaled_float
    scaling_factor: 100
    description: >
    Load average for the last 5 minutes.
    ...

    View full-size slide

  19. _all Removal
    https://www.elastic.co/guide/en/elasticsearch/reference/
    current/mapping-all-field.html

    View full-size slide

  20. Doc Values
    Replaced Fielddata
    https://www.elastic.co/guide/en/elasticsearch/guide/
    current/_deep_dive_on_doc_values.html

    View full-size slide

  21. Architecture

    View full-size slide

  22. Time Based Indices
    index: "metricbeat-%{[beat.version]}-%{+yyyy.MM.dd}"

    View full-size slide

  23. Rollover Indices
    Condition when to switch

    View full-size slide

  24. $ bin/elasticsearch
    -Enode.attr.rack=rack1
    -Enode.attr.size=hot
    PUT /metricbeat/_settings
    {
    "index.routing.allocation.include.size": "hot"
    }

    View full-size slide

  25. Frozen Indices
    Ratio Heap : Storage
    Read-only
    No memory

    View full-size slide

  26. Frozen Indices
    Throttled Thread Pool
    1 parallel search / node
    100 in queue

    View full-size slide

  27. Index Lifecycle
    Management
    https://github.com/elastic/curator for snapshots

    View full-size slide

  28. Features & Order
    https://github.com/elastic/elasticsearch/blob/7.0/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/TimeseriesLifecycleType.java
    static final List ORDERED_VALID_HOT_ACTIONS = Arrays.asList(
    SetPriorityAction.NAME, UnfollowAction.NAME, RolloverAction.NAME
    );
    static final List ORDERED_VALID_WARM_ACTIONS = Arrays.asList(
    SetPriorityAction.NAME, UnfollowAction.NAME, ReadOnlyAction.NAME,
    AllocateAction.NAME, ShrinkAction.NAME, ForceMergeAction.NAME
    );
    static final List ORDERED_VALID_COLD_ACTIONS = Arrays.asList(
    SetPriorityAction.NAME, UnfollowAction.NAME, AllocateAction.NAME, FreezeAction.NAME
    );
    static final List ORDERED_VALID_DELETE_ACTIONS = Arrays.asList(
    DeleteAction.NAME
    );

    View full-size slide

  29. Demo
    Code: https://github.com/xeraa/scale-elasticsearch

    View full-size slide

  30. Agenda
    Building Blocks
    Architecture
    Demo

    View full-size slide

  31. Benchmarks
    Fair
    Reproducible
    Close to Production

    View full-size slide

  32. Problems
    Counters (incremental)
    Single values instead of JSON docs

    View full-size slide

  33. Questions?
    Philipp Krenn@xeraa

    View full-size slide