Scale Your Metrics with Elasticsearch

Slide 1

Slide 1 text

Scale Your Metrics with Elasticsearch Philipp Krenn@xeraa

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

$ curl http://localhost:9200 { "name" : "elasticsearch-hot", "cluster_name" : "metrics-cluster", "cluster_uuid" : "06nHPLLgTrmZEpYli6JW5w", "version" : { "number" : "6.5.0", "build_ﬂavor" : "default", "build_type" : "tar", "build_hash" : "c53b7d3", "build_date" : "2018-11-08T21:28:50.577384Z", "build_snapshot" : false, "lucene_version" : "7.5.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

I'm not going to use a search engine for metrics. — Too often

Slide 13

Slide 13 text

Developer

Slide 14

Slide 14 text

Agenda Building Blocks Architecture Demo

Slide 15

Slide 15 text

Building Blocks

Slide 16

Slide 16 text

Only accept features that scale. — https://github.com/elastic/engineering/blob/master/ development_constitution.md

Slide 17

Slide 17 text

Horizontal Scaling Shards Replication Writes & Reads

Slide 18

Slide 18 text

Cluster, Node, Index, Shard

Slide 19

Slide 19 text

Write Coordinating Node, Hash, Primary, Replica(s)

Slide 20

Slide 20 text

Get Coordinating Node, Hash, Shard

Slide 21

Slide 21 text

Search Coordinating Node, Query then Fetch

Slide 22

Slide 22 text

Append-Only Optimization IDs assigned by coordinating node Fast add instead of the slow update https://github.com/elastic/elasticsearch/issues/19813

Slide 23

Slide 23 text

Lucene Segments index.refresh_interval: 1s index.search.idle.after: 30s Iff default refresh (added in 7.0)

Slide 24

Slide 24 text

Storage Compression LZ4 (default) DEFLATE (best_compression)

Slide 25

Slide 25 text

BKD Trees Points in Lucene (added in 6.0)

Slide 26

Slide 26 text

Integer (1D 4 byte point) vs legacy IntField

Slide 27

Slide 27 text

Half & Scaled Floats

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

https://github.com/elastic/beats/blob/master/metricbeat/module/system/load/_meta/ fields.yml - name: load type: group description: > CPU load averages. release: ga fields: - name: "1" type: scaled_float scaling_factor: 100 description: > Load average for the last minute. - name: "5" type: scaled_float scaling_factor: 100 description: > Load average for the last 5 minutes. ...

Slide 30

Slide 30 text

_all Removal https://www.elastic.co/guide/en/elasticsearch/reference/ current/mapping-all-field.html

Slide 31

Slide 31 text

Doc Values Replaced Fielddata https://www.elastic.co/guide/en/elasticsearch/guide/ current/_deep_dive_on_doc_values.html

Slide 32

Slide 32 text

Architecture

Slide 33

Slide 33 text

Time Based Indices index: "metricbeat-%{[beat.version]}-%{+yyyy.MM.dd}"

Slide 34

Slide 34 text

Rollover Indices Condition when to switch

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

Rollups

Slide 37

Slide 37 text

Nodes! "

Slide 38

Slide 38 text

$ bin/elasticsearch -Enode.attr.rack=rack1 -Enode.attr.size=hot PUT /metricbeat/_settings { "index.routing.allocation.include.size": "hot" }

Slide 39

Slide 39 text

Frozen Indices Ratio Heap : Storage Read-only No memory

Slide 40

Slide 40 text

Frozen Indices Throttled Thread Pool 1 parallel search / node 100 in queue

Slide 41

Slide 41 text

Index Lifecycle Management https://github.com/elastic/curator for snapshots

Slide 42

Slide 42 text

Features & Order https://github.com/elastic/elasticsearch/blob/7.0/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifecycle/TimeseriesLifecycleType.java static final List ORDERED_VALID_HOT_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, RolloverAction.NAME ); static final List ORDERED_VALID_WARM_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, ReadOnlyAction.NAME, AllocateAction.NAME, ShrinkAction.NAME, ForceMergeAction.NAME ); static final List ORDERED_VALID_COLD_ACTIONS = Arrays.asList( SetPriorityAction.NAME, UnfollowAction.NAME, AllocateAction.NAME, FreezeAction.NAME ); static final List ORDERED_VALID_DELETE_ACTIONS = Arrays.asList( DeleteAction.NAME );