Scale Your Metrics with Elasticsearch

"Only accept features that scale" is one of Elasticsearch's engineering principles. So how do we scale metrics stored in Elasticsearch? And is that even possible on a full-text search engine? This talk explores:
* How are metrics stored in Elasticsearch and how does this translate to disk use as well as query performance?
* What does an efficient multi-tier architecture look like to balance speed for today's data against density for older metrics?
* How can you compress old data and what does the mathematical model look like for different metrics?

We are trying all of this hands-on during the talk, since this has become much simpler recently.

Transcript

1. Scale Your Metrics
   with Elasticsearch
   Philipp Krenn@xeraa
   

   View Slide

2. View Slide

3. $ curl http://localhost:9200
   {
   "name" : "elasticsearch-hot",
   "cluster_name" : "metrics-cluster",
   "cluster_uuid" : "06nHPLLgTrmZEpYli6JW5w",
   "version" : {
   "number" : "6.5.0",
   "build_flavor" : "default",
   "build_type" : "tar",
   "build_hash" : "c53b7d3",
   "build_date" : "2018-11-08T21:28:50.577384Z",
   "build_snapshot" : false,
   "lucene_version" : "7.5.0",
   "minimum_wire_compatibility_version" : "5.6.0",
   "minimum_index_compatibility_version" : "5.0.0"
   },
   "tagline" : "You Know, for Search"
   }
   

   View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. View Slide

12. I'm not going to use a search engine
    for metrics.
    — Too often
    

    View Slide

13. Developer
    

    View Slide

14. Agenda
    Building Blocks
    Architecture
    Demo
    

    View Slide

15. Building Blocks
    

    View Slide

16. Only accept features that scale.
    — https://github.com/elastic/engineering/blob/master/
    development_constitution.md
    

    View Slide

17. Horizontal Scaling
    Shards
    Replication
    Writes & Reads
    

    View Slide

18. Cluster, Node, Index, Shard
    

    View Slide

19. Write
    Coordinating Node, Hash, Primary, Replica(s)
    

    View Slide

20. Get
    Coordinating Node, Hash, Shard
    

    View Slide

21. Search
    Coordinating Node, Query then Fetch
    

    View Slide

22. Append-Only
    Optimization
    IDs assigned by coordinating node
    Fast add instead of the slow update
    https://github.com/elastic/elasticsearch/issues/19813
    

    View Slide

23. Lucene Segments
    index.refresh_interval: 1s
    index.search.idle.after: 30s
    Iff default refresh (added in 7.0)
    

    View Slide

24. Storage Compression
    LZ4 (default)
    DEFLATE (best_compression)
    

    View Slide

25. BKD Trees
    Points in Lucene (added in 6.0)
    

    View Slide

26. Integer (1D 4 byte point) vs legacy IntField
    

    View Slide

27. Half & Scaled Floats
    

    View Slide

28. View Slide

29. https://github.com/elastic/beats/blob/master/metricbeat/module/system/load/_meta/
    fields.yml
    - name: load
    type: group
    description: >
    CPU load averages.
    release: ga
    fields:
    - name: "1"
    type: scaled_float
    scaling_factor: 100
    description: >
    Load average for the last minute.
    - name: "5"
    type: scaled_float
    scaling_factor: 100
    description: >
    Load average for the last 5 minutes.
    ...
    

    View Slide

30. _all Removal
    https://www.elastic.co/guide/en/elasticsearch/reference/
    current/mapping-all-field.html
    

    View Slide

31. Doc Values
    Replaced Fielddata
    https://www.elastic.co/guide/en/elasticsearch/guide/
    current/_deep_dive_on_doc_values.html
    

    View Slide

32. Architecture
    

    View Slide

33. Time Based Indices
    index: "metricbeat-%{[beat.version]}-%{+yyyy.MM.dd}"
    

    View Slide

34. Rollover Indices
    Condition when to switch
    

    View Slide

35. View Slide

36. Rollups
    

    View Slide

37. Nodes!
    
    "
    
    

    View Slide

38. $ bin/elasticsearch
    -Enode.attr.rack=rack1
    -Enode.attr.size=hot
    PUT /metricbeat/_settings
    {
    "index.routing.allocation.include.size": "hot"
    }
    

    View Slide

39. Frozen Indices
    Ratio Heap : Storage
    Read-only
    No memory
    

    View Slide

40. Frozen Indices
    Throttled Thread Pool
    1 parallel search / node
    100 in queue
    

    View Slide

41. Index Lifecycle
    Management
    https://github.com/elastic/curator for snapshots
    

    View Slide

42. Features & Order
    https://github.com/elastic/elasticsearch/blob/7.0/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/TimeseriesLifecycleType.java
    static final List ORDERED_VALID_HOT_ACTIONS = Arrays.asList(
    SetPriorityAction.NAME, UnfollowAction.NAME, RolloverAction.NAME
    );
    static final List ORDERED_VALID_WARM_ACTIONS = Arrays.asList(
    SetPriorityAction.NAME, UnfollowAction.NAME, ReadOnlyAction.NAME,
    AllocateAction.NAME, ShrinkAction.NAME, ForceMergeAction.NAME
    );
    static final List ORDERED_VALID_COLD_ACTIONS = Arrays.asList(
    SetPriorityAction.NAME, UnfollowAction.NAME, AllocateAction.NAME, FreezeAction.NAME
    );
    static final List ORDERED_VALID_DELETE_ACTIONS = Arrays.asList(
    DeleteAction.NAME
    );
    

    View Slide

43. Demo
    Code: https://github.com/xeraa/scale-elasticsearch
    

    View Slide

44. Conclusion
    

    View Slide

45. Agenda
    Building Blocks
    Architecture
    Demo
    

    View Slide

46. Benchmarks
    Fair
    Reproducible
    Close to Production
    

    View Slide

47. View Slide

48. Problems
    Counters (incremental)
    Single values instead of JSON docs
    

    View Slide

49. Questions?
    Philipp Krenn@xeraa
    

    View Slide