Beyond Dashboarding The Grafana Observability Stack by Steve Caron

Transcript

1. Beyond Dashboarding The Grafana Observability Stack Steve Caron Staff Solutions Engineer
2. How most people started with Grafana
3. Loki for logs Grafana for visualizations Tempo for traces Mimir for metrics
4. Grafana Mimir
5. Running Prometheus at scale Prometheus is great but… Out of the box, Prometheus scales only vertically. Centralised view of metrics can only be achieved using hierarchical federation or cross-service federation. Prometheus use local storage. Traditionally retention is set to 15 days and rarely above 30 days. No authentication mechanism or role based access controls for protecting your data. Limited horizontal scalability No robust federation Not designed for long term retention No security model
6. Prometheus on steroids: Mimir = Mimir + Durable storage Blazing fast query performance Production-proven dashboards, alerts, and playbooks High availability Horizontal scalability Real multi-tenancy Prometheus
7. Running Prometheus at scale Application 1 Application 2 Application N Region A Application 1 Application 2 Application N Region B Remote write Remote write Remote write Remote write For Prometheus users • Leverage your existing investment by using Prometheus as a Metrics forwarder. • 100% compatible with your existing queries, alerts and recording rules are . For all • Get started in a few clicks using the Grafana agent (embeds the Prometheus agent). • Query your Mimir metrics using Grafana. Query
8. Grafana Loki
9. Who did we make Loki for? Effective Debugging and troubleshooting of applications Visualise and alert on services/apps performance metrics Build actionable insights from log data and other supported data sources DevOps SRE DataEng
10. Format agnostic Efficient at scale Why do they like Loki? Built for correlation Logs as metrics
11. Under the hood 2019-12-11T10:01:02.123456789Z {app=”nginx”, env=”dev”} Timestamp with nanosecond precision Log content JSON, logfmt, custom, etc. Labels/Selectors key-value pairs Indexed Unindexed GET /about 1034 Debug “page not found”
12. ` Get the most out of your logs with LogQL • Inspired from PromQL syntax for effortless correlations between Metrics and Logs. • Build Metrics from Logs and unlock new use cases. • Use your LogQL queries for creating advanced alerting rules. {app=”nginx”,instance=”1.1.1.1”} Label matchers != "Googlebot/" Line filters | json Parser | request_time >= 100 and status == 200 Label filters *Successful requests with a latency superior to 100ms (Googlebot requests excluded)
13. Promtail Makes logs collection easy with... • Targets discovery for Kubernetes, Syslog, files and more • Automatically attach labels to your log lines • Advanced pipeline mechanism for parsing, transforming and filtering your logs • Build and expose custom metrics from your logs data But Loki is open. logstash Lambda
14. Grafana Tempo
15. What is distributed tracing? A way to observe requests as they propagate through a distributed system
16. How to get started with distributed tracing? Instrument your code using agents and libraries to generate spans for your services. Use tracing pipelines to collect, transform and enrich spans. Store all the traces for querying and building more insights. Use Grafana to detect and investigate service issues. Correlate your traces with metrics and log data. Instrument Collect Store Visualize
17. • traces_spanmetrics_calls_total - Counter, Total count of the span (Rate, Error) • traces_spanmetrics_latency - Histogram, Duration of the span (Duration) (includes Exemplars) • traces_spanmetrics_size_total - Counter, Total size of spans ingested (Volume) Metrics Generation
18. | {} TraceQL { .namespace = “prod” } > { .service.name = “auth” && { .http.status_code = 500 } { .http.status_code = 500 } | count() > 1 Inspired by PromQL and LogQL Extract insights from traces interactively Analyze traces based on their structure >
19. Monolithic mode • Simplest deployment mode • All components in one single process • Great for testing Microservices mode • Maximum scalability • Separate Read/Write paths • Recommended for production deployments and large volumes How to run the Grafana stack? Grafana Cloud • Fully managed by Grafana • Available in 7 regions • Free-forever tier (50GB logs and traces per month, 10K active series, 3 users.
20. Simplified architecture
21. How to collect your telemetry data The community way Or The Grafana way
22. Anatomy of the Grafana Agent Metrics - Shares the same codebase as the Prometheus Agent. Logs - Embeds Promtail, the log forwarder built by Grafana, for Loki. Traces - Based on OpenTelemetry Collector.
23. Demo
24. + An open source, highly scalable and cost efficient continuous profiling database
25. An open source web SDK for frontend application observability 1.5M+ NPM Downloads
26. Open source eBPF auto-instrumentation for application observability
27. Have more questions? Join us at community.grafana.com or Grafana public slack: slack.grafana.com #channel grafana/ community.grafana.com Get involved: Thank you!