Observability from 0 to 100

We all know that our systems need to be more observable, but how do you get to valuable insights? Can you really get up and running as fast as vendors let you believe? And what are the places where it is worth spending extra time to get things right? Observability capabilities have been growing rapidly over the last years. Many vendors and tools exist in this field, with offerings varying in customizability and out-of-the-box capabilities. In this talk we'll take a look at what you need to do to get observability going for your platform and applications.

We go beyond the marketing-talk and see what it is like to build up proper observability. To make it concrete we'll do that for a Kubernetes cluster running JVM applications for which we will gather logs, metrics and traces to an Elastic stack. We'll manage all the configuration as Infrastructure-as-code with the CDK for Terraform. Along the way we will share some key ingredients that we discovered that make your observability setup more effective and downright simpler. Expect to discover what you need to do, not just on infrastructure level, but also in application code, and most important: on organizational level, so you can expose everything to the right people.

Transcript

1. Observability from 0 to 100 Nico Krijnen
2. Observability – Ingredients – Make it work – OpenTelemetry – Takeaway Observability
3. Observability "The ability to measure the current state of a system by examining its outputs."
4. Observability Metrics Traces Logs & Events
5. Observability Monitoring APM Security Analytics / SIEM Property of a system, like testability Act of observing a system over time Monitoring the performance and availability of an application Threat detection and proactive security measures
6. @nicokrijnen nkrijnen nicokrijnen @nicokrijnen nkrijnen nicokrijnen Nico Krijnen Observability from 0 to 100
7. Observability – Ingredients – Make it work – OpenTelemetry – Takeaway Ingredients
8. Use structured logging!
10. Logs Process Logstash Filebeat Fluentd FluentBit Vector Elastic Agent ... Store & Visualize Elasticsearch + Kibana Datadog Splunk ...
11. Logs Process Logstash Filebeat Fluentd FluentBit Vector Elastic Agent ... Store & Visualize Elasticsearch + Kibana Datadog Splunk ...
13. Logs Process Store & Visualize
14. Logs Process Store & Visualize
15. Logs Store & Visualize
16. {"@timestamp":"2019-08-06T12:09:12.375Z", "log.level": "INFO", "message":"Tomcat started
17. {"@timestamp":"2019-08-06T12:09:12.375Z", "log.level": "INFO", "message":"Tomcat started {"level":30,"time":1643365551573,"pid":5673,"hostname":"Kreig","msg":"Hello, world!"}
18. Use standardized JSON logs!
19. Elastic Common Schema (ECS)
20. Use the tooling from your vendor
21. Collectors - so many options Logstash FileBeat, MetricBeat FluentD, FluentBit Vector Elastic Agent
22. Vendor Agents • Log & Metrics collection on host tailing log files and collecting host metrics • Metrics & Trace collection in-proces, adding instrumentation inside your application In-process agent Process Agent Host
23. Use alerts and health checks No need to put dashboards on the wall
24. Tune your alerts alert threshold
25. Tune your alerts alert threshold
26. alert threshold Tune your alerts alert threshold
27. alert threshold Tune your alerts alert threshold
28. alert threshold Tune your alerts alert threshold
29. alert threshold Tune your alerts alert threshold
30. Know what to log and what not to log
31. What to log? https://youtu.be/Ak1hGQuGBhY
32. What to log? https://youtu.be/Ak1hGQuGBhY
33. Observability – Ingredients – Make it work – OpenTelemetry – Takeaway Make it work Understanding & Buy-in
34. Know your needs and budget • Easy to produce terrabytes of data • Cost rises with amount of data & retention • Know your budget & what that gets you • Know what your organization needs • Make sure you have support
35. Start small • Spike: whole chain working for one service • Validate and fine-tune • Scale out, implement blue-print for all services
36. Expect to be surprised
37. Observability – Ingredients – Make it work – OpenTelemetry – Takeaway Make it work Collection
38. Collecting Logs & Metrics
39. configuration for agents observability data Elastic Agents Elastic Fleet
40. Elastic Cloud [Azure Deployment in West Europe (NL) managed by Elastic] Developer / Operator [Person] Analyze logs, metrics, traces. Quickly find cause of issues. Admin [Person] Manage configuration NON-PROD [k8s cluster] PROD [k8s cluster] Elastic Agent [Container: DaemonSet Pod = one pod per worker node] Run FileBeat, MetricBeat, APM server with configuration as defined in Elastic Fleet Policy. Exposes APM Server port 8200 on host. aks-agentpool-vmss0 [k8s worker node] some-service [Container: Pod] Elastic APM JVM Agent is embedded in Docker image other-server [Container: Pod] Elastic APM JVM Agent is embedded in Docker image aks-agentpool-vmss1 [k8s worker node] aks-agentpool-vmss2 [k8s worker node] Elastic Agent Elastic Agent Elastic Agent [DaemonSet] Send JVM metrics and traces to APM Server [HTTP @ HOST-IP:8200] Send JVM metrics and traces to APM Server [HTTP @ HOST-IP:8200] Ingest logs, metrics, traces [HTTPS] Configure log, metrics, trace collection policies [HTTPS] Search & analyze logs, metrics, traces [HTTPS] Idem Elastic Fleet [Software System] Central management of configuration for log & metric collection agents. Elasticsearch [Software System] Search & analytics engine. Also stores configuration data. Kibana [Software System] Centralized logs, metrics, traces & Management UI Legend Person External Software System Infra Container Application Container Fleet enrollment token for PROD agent policy Fleet enrollment token for NON-PROD agent policy Elastic Agent Business / PO [Person] Analyze usage. Gather data to make decisions. Elastic Agent Elastic Agent Fetch desired config & Send logs, metrics, traces to Fleet Server [HTTPS]
41. Elastic Cloud [Azure Deployment in West Europe (NL) managed by Elastic] Developer / Operator [Person] Analyze logs, metrics, traces. Quickly find cause of issues. Admin [Person] Manage configuration NON-PROD [k8s cluster] PROD [k8s cluster] Elastic Agent [Container: DaemonSet Pod = one pod per worker node] Run FileBeat, MetricBeat, APM server with configuration as defined in Elastic Fleet Policy. Exposes APM Server port 8200 on host. aks-agentpool-vmss0 [k8s worker node] some-service [Container: Pod] Elastic APM JVM Agent is embedded in Docker image other-server [Container: Pod] Elastic APM JVM Agent is embedded in Docker image aks-agentpool-vmss1 [k8s worker node] aks-agentpool-vmss2 [k8s worker node] Elastic Agent Elastic Agent Elastic Agent [DaemonSet] Send JVM metrics and traces to APM Server [HTTP @ HOST-IP:8200] Send JVM metrics and traces to APM Server [HTTP @ HOST-IP:8200] Ingest logs, metrics, traces [HTTPS] Configure log, metrics, trace collection policies [HTTPS] Search & analyze logs, metrics, traces [HTTPS] Idem Elastic Fleet [Software System] Central management of configuration for log & metric collection agents. Elasticsearch [Software System] Search & analytics engine. Also stores configuration data. Kibana [Software System] Centralized logs, metrics, traces & Management UI Legend Person External Software System Infra Container Application Container Fleet enrollment token for PROD agent policy Fleet enrollment token for NON-PROD agent policy Elastic Agent Business / PO [Person] Analyze usage. Gather data to make decisions. Elastic Agent Elastic Agent Fetch desired config & Send logs, metrics, traces to Fleet Server [HTTPS]
42. Elastic Agent [Container] Elastic Agent [Component: Process] Manage collectors & shippers Elastic Filebeat [Component: Process] Lightweight log-shipper Starts & provides configuration Elastic Metricbeat [Component: Process] Lightweight metric-collector & shipper Elastic APM Server [Component: Process] Lightweight trace-collector & shipper Elastic Fleet [Software System] Central management of configuration for log & metric collection agents. Elastic Cloud [Azure Deployment managed by Elastic] Ship logs [HTTPS] Loads policy config & poll for policy changes [HTTPS] Ship metrics [HTTPS] Ship metrics & traces [HTTPS] some-service [Container] Elastic APM Agent [Component: Java Agent] Wraps Java application and automatically instruments JVM, Spring, Logging, etc to enrich log output and collect traces & metrics. Java VM [Process] Application [Component: Spring Boot Application] Tail pod container logfiles [Volume mount] Collect metrics [JSON/HTTP] /var/log/containers/*${kubernetes.container.id}.log K8s API Send JVM metrics and traces to APM Server [HTTP @ HOST-IP:8200] aks-agentpool-vmss* [k8s worker node (host)] Legend External Software System Infra Component Application Component
43. Elastic Agent [Component: Process] Manage collectors & shippers Starts & provides configuration Elastic Fleet [Software System] Central management of configuration for log & metric collection agents. Elastic Cloud [Azure Deployment managed by Elastic] Loads policy config & poll for policy changes [HTTPS]
44. Elastic Agent [Container] Elastic Agent [Component: Process] Manage collectors & shippers Elastic Filebeat [Component: Process] Lightweight log-shipper Starts & provides configuration Elastic Metricbeat [Component: Process] Lightweight metric-collector & shipper Elastic APM Server [Component: Process] Lightweight trace-collector & shipper Ship logs [HTTPS] Ship metrics [HTTPS] Ship metrics & traces [HTTPS] rvice Elastic APM Agent [Component: Java Agent] Wraps Java application and automatically instruments JVM, Spring, Logging, etc to enrich log output and collect traces & metrics. a VM ess] Application [Component: Spring Boot Application] Tail pod container logfiles [Volume mount] Collect metrics [JSON/HTTP] /var/log/containers/*${kubernetes.container.id}.log K8s API Send JVM metrics and traces to APM Server [HTTP @ HOST-IP:8200]
45. Collecting metrics
47. Collecting Traces
48. Elastic Agent [Container] Elastic Agent [Component: Process] Manage collectors & shippers Elastic Filebeat [Component: Process] Lightweight log-shipper Starts & provides configuration Elastic Metricbeat [Component: Process] Lightweight metric-collector & shipper Elastic APM Server [Component: Process] Lightweight trace-collector & shipper Elastic Fleet [Software System] Central management of configuration for log & metric collection agents. Elastic Cloud [Azure Deployment managed by Elastic] Ship logs [HTTPS] Loads policy config & poll for policy changes [HTTPS] Ship metrics [HTTPS] Ship metrics & traces [HTTPS] some-service [Container] Elastic APM Agent [Component: Java Agent] Wraps Java application and automatically instruments JVM, Spring, Logging, etc to enrich log output and collect traces & metrics. Java VM [Process] Application [Component: Spring Boot Application] Tail pod container logfiles [Volume mount] Collect metrics [JSON/HTTP] /var/log/containers/*${kubernetes.container.id}.log K8s API Send JVM metrics and traces to APM Server [HTTP @ HOST-IP:8200] aks-agentpool-vmss* [k8s worker node (host)] Legend External Software System Infra Component Application Component
49. Elastic Agent [Container] Elastic Filebeat [Component: Process] Lightweight log-shipper Elastic Metricbeat [Component: Process] Lightweight metric-collector & shipper Elastic APM Server [Component: Process] Lightweight trace-collector & shipper Ship logs [HTTPS] Ship metrics [HTTPS] Ship metrics & traces [HTTPS] some-service [Container] Elastic APM Agent [Component: Java Agent] Wraps Java application and automatically instruments JVM, Spring, Logging, etc to enrich log output and collect traces & metrics. Java VM [Process] Application [Component: Spring Boot Application] Tai Send JVM metrics and traces to APM Server [HTTP @ HOST-IP:8200] aks-agentpool-vmss* [k8s worker node (host)] Elastic Agent [Container] Elastic Filebeat [Component: Process] Lightweight log-shipper Elastic Metricbeat [Component: Process] Lightweight metric-collector & shipper Elastic APM Server [Component: Process] Lightweight trace-collector & shipper Ship logs [HTTPS] Ship metrics [HTTPS] Ship metrics & trac [HTTPS] some-service [Container] Elastic APM Agent [Component: Java Agent] Wraps Java application and automatically instruments JVM, Spring, Logging, etc to enrich log output and collect traces & metrics. Java VM [Process] Application [Component: Spring Boot Application] Send JVM metrics and traces to APM Server [HTTP @ HOST-IP:8200] aks-agentpool-vmss* [k8s worker node (host)]
50. Instrumentation Elastic Agent [Container] Elastic Agent [Component: Process] Manage collectors & shippers Elastic Filebeat [Component: Process] Lightweight log-shipper Starts & provides configuration Elastic Metricbeat [Component: Process] Lightweight metric-collector & shipper Elastic APM Server [Component: Process] Lightweight trace-collector & shipper Ship logs [HTTPS] Ship metrics [HTTPS] Ship metrics & traces [HTTPS] some-service [Container] Elastic APM Agent [Component: Java Agent] Wraps Java application and automatically instruments JVM, Spring, Logging, etc to enrich log output and collect traces & metrics. Java VM [Process] Application [Component: Spring Boot Application] Tail pod container logfiles [Volume mount] Collect metrics [JSON/HTTP] /var/log/containers/*${kubernetes.container.id}.log K8s API Send JVM metrics and traces to APM Server [HTTP @ HOST-IP:8200] aks-agentpool-vmss* [k8s worker node (host)]
53. Observability – Ingredients – Make it work – OpenTelemetry – Takeaway Make it work Finetuning
54. JSON + ECS
55. JSON + ECS ELASTIC_APM_LOG_FORMAT_SOUT=JSON ELASTIC_APM_LOG_ECS_REFORMATTING=REPLACE
56. Optimize query results • Add metadata to help you filter • Service name • Environment (dev, staging, prod) • Kubernetes metadata • Cloud metadata
57. Make sure no data is dropped object mapping for [json.data] tried to parse field [data] as object, but found a concrete value"}, dropping event! {"json": {"data": {"gtin": "...", "product":"..." } } } {"json": {"data": "..."} }
58. Add index mappings github:elastic/ecs elasticsearch
59. Tune capacity - retention policies • Not all data is equally valuable • Your apps vs system and platform • Logs, Metrics, Traces
60. Tune capacity - rollup jobs • Drop what you don't need • Reduce metric density after some time • Capture all failure traces, but only sample % of normal traces
61. Tune capacity - data tiering • Benefit of observability data: it's all timeseries • Lifecycle policy • Hot, expensive storage for latest data • Warm, cold and frozen for data as it gets older
62. Observability – Ingredients – Make it work – OpenTelemetry – Takeaway Make it work Finetuning
63. Observability – Ingredients – Make it work – OpenTelemetry – Takeaway OpenTelemetry
67. OpenTelemetry maturity https://opentelemetry.io/docs/reference/specification/status/ Tracing • API: stable, feature-freeze • SDK: stable • Protocol: stable Metrics • API: stable • SDK: mixed • Protocol: stable Logging • API: draft • SDK: draft • Protocol: stable
68. Observability – Ingredients – Make it work – OpenTelemetry – Takeaway Key takeaway
69. Key takeaway
70. Key takeaway • Observability can be complex, but it doesn't have to be.
71. Key takeaway • Observability can be complex, but it doesn't have to be. • Uniform JSON logging will help you keep things simple.
72. Key takeaway • Observability can be complex, but it doesn't have to be. • Uniform JSON logging will help you keep things simple. • Don't forget who you're doing this for: increase understanding of your systems so you can provide better experiences.
73. Nico Krijnen @nicokrijnen nkrijnen nicokrijnen Observability from 0 to 100