!1 Thomas Watson Mar 2019 Logging, Metrics, and APM: The Holy Trinity of Operations @wa7son 

Logs Metrics APM @wa7son 

Who am I? • Thomas Watson • Open Source developer at github.com/watson • Principal Software Engineer at Elastic • Node.js Core Member • Tweets as @wa7son @wa7son 

!4 Benefits of Logs + Metrics + APM in one stack @wa7son 

!5 Unified Dashboards Same UI for KPI summaries and root cause analysis @wa7son 

!6 Unified Alerting Trigger off any operational data to provide unified SLA monitoring @wa7son 

!7 Unified Machine Learning Correlate multiple data sources for more intelligent anomaly detection @wa7son 

!8 Operational gains Single technology for operational data saves on administrative costs @wa7son 

!9 Elastic Stack for logs @wa7son 

Logs 64.242.88.10 - - [07/Mar/2017:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291 64.242.88.10 - - [07/Mar/2017:16:11:58 -0800] "POST /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 404 7352 64.242.88.10 - - [07/Mar/2017:16:20:55 -0800] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 5253 For each event, print out what happened. Logs are chronological records of events @wa7son 

Making logging more turnkey with ‘modules’ • Turnkey experience for specific data types • Data to dashboard in just one step • Automated parsing and enrichment • Default dashboards, alerts, ML jobs @wa7son 

Logging modules System • Linux / MacOS • Windows Events Containers • Docker • Kubernetes Databases • MySQL • PostgreSQL Queues • Kafka • Redis Web servers • Apache • Nginx Audit data • Filesystem • System calls WINLOGBEAT FILEBEAT AUDITBEAT Infrastructure Applications @wa7son 

!13 Ad-hoc log search and visualization Kibana Discover, Visualize, Dashboard @wa7son 

!14 Elastic Stack for metrics @wa7son 

Metrics vs Logs 64.242.88.10 - - [07/Mar/2017:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291 64.242.88.10 - - [07/Mar/2017:16:11:58 -0800] "POST /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 404 7352 64.242.88.10 - - [07/Mar/2017:16:20:55 -0800] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 5253 For each event, print out what happened. Logs are chronological records of events 07/Mar/2017 16:10:00 all 2.58 0.00 0.70 1.12 0.05 95.55 server1 containerX regionA
 07/Mar/2017 16:20:00 all 2.56 0.00 0.69 1.05 0.04 95.66 server2 containerY regionB
 07/Mar/2017 16:30:00 all 2.64 0.00 0.65 1.15 0.05 95.50 server2 containerZ regionC
 
 Every x minutes, measure the CPU load and print it out, and annotate with meta-data.
 Metrics are periodic measurements of numeric KPIs @wa7son 

!16 Evolution of Elasticsearch into a Metrics Store @wa7son 

Elasticsearch beginnings Primarily used for application search Search engine Inverted index primary data structure, and is great for search 2010 @wa7son 

Source: Computer Graphics - Principles and Practice @wa7son 

Source: Computer Graphics - Principles and Practice 

@wa7son 

Elasticsearch beginnings Primarily used for application search Search engine Inverted index primary data structure, and is great for search 2010 @wa7son 

2012 Columnar storage Structured data storage, resulting in compact storage and faster analytics Elasticsearch evolves to support analytics https://www.elastic.co/blog/elasticsearch-as-a-column-store Columnar Store, Built on Lucene "doc values" Search engine Inverted index primary data structure, and is great for search 2010 @wa7son 

2014 Aggregation Framework Analytics features to slice and dice data along various dimensions Aggregation Framework Out-of-this-world aggregations https://www.elastic.co/blog/out-of-this-world-aggregations Search engine Inverted index primary data structure, and is great for search 2010 2012 Columnar storage Structured data storage, resulting in compact storage and faster analytics @wa7son 

BKD trees and sparse fields Data structures optimized for numbers. Faster analytics, lower storage footprint 2016 2014 Aggregation Framework Analytics features to slice and dice data along various dimensions Elasticsearch storage efficiencies BKD Trees & Sparse Fields https://www.elastic.co/blog/searching-numb3rs-in-5.0 1-Dimension 2-Dimensions Sparse Data Search engine Inverted index primary data structure, and is great for search 2010 2012 Columnar storage Structured data storage, resulting in compact storage and faster analytics @wa7son 

Rollups Roll up or aggregate older data into bigger time buckets and save on disk space 2018 Rollup support for long-term retention Added in Elasticsearch 6.3 https://www.elastic.co/blog/data-rollups-in-elasticsearch-you-know-for-saving-space Search engine Inverted index primary data structure, and is great for search 2010 BKD trees and sparse fields Data structures optimized for numbers. Faster analytics, lower storage footprint 2016 2014 Aggregation Framework Analytics features to slice and dice data along various dimensions 2012 Columnar storage Structured data storage, resulting in compact storage and faster analytics @wa7son 

!26 Elastic Stack as a Metrics Solution @wa7son 

Metrics modules System • Linux • MacOS • Windows • Perfmon Infrastructure Cloud • AWS • GCP • Azure • DigitalOcean • Alibaba Containers • Docker • Kubernetes Virtualization • vSphere PACKETBEAT METRICBEAT Network • Netflow • Packets • TLS Envelope Storage • Ceph HEARTBEAT @wa7son 

Applications Datastores • MySQL • PostgreSQL • MongoDB • Couchbase • Aerospike • Graphite Web servers • Apache • Nginx Other • HAProxy • Zookeeper Queues • Kafka • Redis • RabbitMQ Caches • Memcached Uptime • Heartbeat Custom apps • JMX/Jolokia • PHP-FPM • Golang Metrics modules PACKETBEAT METRICBEAT HEARTBEAT @wa7son 

Heartbeat: Uptime Monitoring @wa7son 

Heartbeat: Uptime Monitoring 

Functionbeat: Serverless data shipper Cloudwatch Cloudwatch Logs @wa7son 

Functionbeat: Serverless data shipper @wa7son 

• Correlate data from different sources • Ability to re-use analysis content • Ability to re-use Elastic-provided content Correlation between logs, metrics, and APM Benefits • v1.0.0 published: github.com/elastic/ecs • Integrating into Elastic products in progress • Community feedback welcome! Status Elastic Common Schema @wa7son 

Visualizing time series data Time Series Visual Builder @wa7son 

Visualizing time series data Annotations @wa7son 

!36 Elastic Stack for APM @wa7son 

What is APM? Example 08:32:10 Request "/api/checkout" 08.32:11 Response "/api/checkout 500 ERROR" @wa7son 

What is APM? Example 08:32:10 Request "/api/products/top" 08.32:17 Response "/api/products/top 200 OK" 7 seconds - zZzzZZz @wa7son 

How does APM work? Data processor apm-server Data storage elasticsearch Browser Agent Web server Agent Web server Agent Web server Agent UI kibana Browser Agent Browser Agent @wa7son 

• Focuses on search experience on top of APM data • ‘Just another index’ in Elastic Stack Elastic APM APM adds end-user experience and application-level monitoring to the stack Language support ● Python
 ● Node.js
 ● Ruby
 ● RUM 
 ● Java ● Go ● .NET (in dev) @wa7son 

APM is another index in Elasticsearch Need another visualization? Build a dashboard, no need to wait for your vendor @wa7son 

Single transaction Distributed Tracing Transaction Span Span Span HTTP request Response @wa7son 

Distributed tracing example Distributed Tracing Trace A Transaction 1 Span Span Span Transaction 2 Span Transaction 3 Span Span @wa7son 

Distributed Tracing Trace and map across multiple services
 • See the end-to-end view and navigate to individual transactions • Based on the notion of a end-to- end Trace ID across services • Investigating compatibility with OpenTracing API and aligning with W3C trace context spec @wa7son 

!45 DEMO @wa7son 

What now? Try it yourself! @wa7son 

What now? Try it yourself! @wa7son 

!48 You can always send me a tweet at @wa7son Questions?