Monitor your containers with the Elastic Stack

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Monitor your containers with the Elastic Stack Monica Sarbu

Slide 3

Slide 3 text

3 Monica Sarbu Team lead, Beats team [email protected] @monicasarbu

Slide 4

Slide 4 text

Monitor your containers with the Elastic Stack

Slide 5

Slide 5 text

@monicasarbu Elastic Stack 5

Slide 6

Slide 6 text

Beats are lightweight shippers that collect and ship all kinds of operational data to Elasticsearch

Slide 7

Slide 7 text

@monicasarbu Multiple data types, one place 7 •Docker metrics •flows •MySQL logs •diskIO •HTTP transactions •MySQL transactions •Redis metrics •Apache logs •Redis logs •CPU % •Docker metrics •Docker logs •memory % •filesystem •Redis transactions •flows

Slide 8

Slide 8 text

@monicasarbu Central point for your distributed infrastructure 8

Slide 9

Slide 9 text

@monicasarbu The Beats 9 30+ other community Beats shipping

Slide 10

Slide 10 text

Filebeat 10

Slide 11

Slide 11 text

• Tails log files, without parsing them • “At least once” guarantees, handles backpressure • Extra powers: • Multiline • JSON logs • Filtering 11 Filebeat

Slide 12

Slide 12 text

@monicasarbu Parse log lines with Ingest Node 12 I N G E S T

Slide 13

Slide 13 text

@monicasarbu Parse log lines with Logstash 13 I N G E S T

Slide 14

Slide 14 text

Filebeat Back pressure handling 14

Slide 15

Slide 15 text

@monicasarbu Why back-pressure is key? 15

Slide 16

Slide 16 text

@monicasarbu Synchronous sending 16 batch of messages ack stream of log lines read read acked registry file

Slide 17

Slide 17 text

• Filebeat adapts its speed automatically to as much as the next stage can process • But: be aware when benchmarking 17 This means..

Slide 18

Slide 18 text

• Filebeat patiently waits • Log lines are not lost • It doesn’t allocate memory, it doesn’t buffer things on disk 18 When the next stage is down..

Slide 19

Slide 19 text

Filebeat Collect container logs 19

Slide 20

Slide 20 text

@monicasarbu Docker logging drivers 20 https://docs.docker.com/engine/admin/logging/overview/

Slide 21

Slide 21 text

@monicasarbu Centralize Docker logs: option 1/522 • Use the Docker gelf driver and the Logstash-gelf-input • Pros: • No shipper to install, send directly to Logstash • Cons: • UDP based, no delivery guarantees, no congestion control 21

Slide 22

Slide 22 text

@monicasarbu Centralize Docker logs: option 2/522 • Use the Docker JSON driver, use Filebeat with the JSON support • Pros: • Simple (default driver) • Easy to add container metadata (name, labels, etc.) • `docker logs` works • Cons: • JSON driver can slow down Docker 22

Slide 23

Slide 23 text

@monicasarbu Centralize Docker logs: option 3/522 • Use the Docker syslog driver, and a local syslog server, then Filebeat for shipping • Pros: • Good control over the path where the files are written, rotation strategies, etc. • Cons: • you need to manage the syslog server • metadata is serialized as string, needs to be de- serialized again (opportunity for mistakes) • multiline is difficult because data from containers can be mixed 23

Slide 24

Slide 24 text

@monicasarbu Centralize Docker logs: option 4/522 • Use the Docker journald driver then Filebeat for shipping • Pros: • journald is often already available • convenient support for metadata • `docker logs` works • Cons: • Filebeat doesn’t yet support journald (a Journalbeat exists, however) 24

Slide 25

Slide 25 text

@monicasarbu Centralize Docker logs: option 5/522 • Mount a volume and have your app write logs into the volume • Pros: • If your app can rotate it’s own logs, it’s very easy to setup • Scales well • Cons: • Difficult to pass metadata 25

Slide 26

Slide 26 text

@monicasarbu Centralize Docker logs: conclusion • json driver, syslog driver, and shared volume are pretty good options today • journald driver might be better options in the future 26

Slide 27

Slide 27 text

Metricbeat 27 new in 5.0

Slide 28

Slide 28 text

@monicasarbu One Metricbeat module for each service 28 + Add your own

Slide 29

Slide 29 text

@monicasarbu Metricbeat system module 29 CPU Mem diskIO filesystem processes load network cores

Slide 30

Slide 30 text

Metricbeat Collect container metrics 30

Slide 31

Slide 31 text

@monicasarbu Querying the Docker API • Dedicated Docker module • Has access to container names and labels • Easy to setup • Offers: • CPU and memory • Docker container information • network (in/out bytes, dropped) • diskIO (reads/writes) • status of containers (# of stopped, running, etc) 31 in progress

Slide 32

Slide 32 text

@monicasarbu Reading cgroup data from /proc/ • Doesn’t require access to the Docker API (can be a security issue) • Works for any container runtime (Docker, rkt, runC, LXD, etc.) • Part of the system module • Automatically enhances process data with cgroup information • Cannot get the container name and labels 32

Slide 33

Slide 33 text

@monicasarbu Run as a container 33 App1 App2 App3 Host

Slide 34

Slide 34 text

34 Elasticsearch as time series DB

Slide 35

Slide 35 text

#velo @monicasarbu Elasticsearch BKD trees 35 • Added for Geo-points • faster to index • faster to query • more disk-efficient • more memory efficient

Slide 36

Slide 36 text

@monicasarbu 0 10000 20000 30000 40000 50000 60000 70000 80000 float half float scaled float (factor = 4000) scaled float (factor = 100) On Disk Usage in kb Points disk usage (kb) docs_values disk usage (kb) Float values 36 • half floats • scaled floats (using a scaling factor) - great for things like percentage points

Slide 37

Slide 37 text

#velo @monicasarbu Why Elasticsearch for time series • Horizontal scalability. Mature and battle tested cluster support. • Flexible aggregations (incl moving averages & Holt Winters) • One system for both logs and metrics • Timelion UI, Grafana • Great ecosystem: e.g. alerting tools 37

Slide 38

Slide 38 text

Packetbeat 38

Slide 39

Slide 39 text

@monicasarbu Supported traffic decoders 39 + Add your own http:// Thrift DNS ICMP AMQP

Slide 40

Slide 40 text

@monicasarbu Unknown traffic, use flows •Look into data for which we don’t understand the application layer protocol •TLS •Protocols we don’t yet support •Get data about IP / TCP / UDP layers •number of packets & bytes •retransmissions •inter-arrival time 40

Slide 41

Slide 41 text

Packetbeat Monitor traffic exchanged between your containers 41

Slide 42

Slide 42 text

@monicasarbu Monitor outside containers 42 App1 Host App2 App3 Packetbeat traffic exchanged between your containers

Slide 43

Slide 43 text

43 Demo: Metricbeat, Filebeat, Packetbeat Multiple data types, one view in Kibana

Slide 44

Slide 44 text

Thank you • github.com/elastic/beats • discuss.elastic.co • @elastic #elasticbeats • #beats on freenode 44

Slide 45

Slide 45 text

No content