Improve Monitoring and Observability for Kubernetes with OSS tools

Slide deck related to the presentation at the KubeDay Singapore event. The session covered 3 pillars of Observability and how to use Jaeger for Distribute Tracing, Loki for Log Aggregation and Prometheus and Grafana for Metrics in a distributed application. Azure Kubernetes Service AKS cluster was used for live demo.

https://events.linuxfoundation.org/kubeday-singapore/

Transcript

1. @nileshgule Improve Monitoring and Observability for Kubernetes with OSS tools
2. Nilesh Gule ARCHITECT | MICROSOFT MVP | First Docker Captain in Singapore “Code with Passion and Strive for Excellence” nileshgule @nileshgul e Nilesh Gule NileshGule www.handsonarchitect.co m https://www.youtube.com/@nilesh-gule
3. @nileshgule
4. @nileshgule CNCF cloud trail https://github.com/cncf/trailmap
5. @nileshgule CNCF Observability landscape https://landscape.cncf.io
6. @nileshgule CNCF Observability Radar https://radar.cncf.io/2020-09-observability
7. @nileshgule 3 Pillars of Observability Logs Metrics Traces
8. @nileshgule Centralized Logging
9. @nileshgule ❑ Application specific ❖ Long term log retention for compliance reasons ❖ Workloads scheduled on different nodes during application restarts / updates ❖ Autoscaling workloads ❑ Kubernetes upgrades ❖ Auto healing can reschedule workloads ❖ Underlying nodes added / deleted during cluster scaling ❖ Underlying nodes replaced during cluster upgrades Container based workloads Why centralized logging ❖ Not much control over underlying infra ❖ Relies on cloud prover specific logging and monitoring solution PaaS / Serverless services
10. @nileshgule Financial Services App Loki integration Log collector Log storage Log search, visualise, dashboards backend-service account-service authentication-service forex-service transaction-service
11. @nileshgule Demo 1 – Log Aggregation with Loki
12. @nileshgule Metrics
13. @nileshgule • Application specific • Monitor resource usage • Monitor scaling needs • Monitor anomalies / outliers • Kubernetes platform level • Monitor cluster resources (CPU / RAM) • API health • Autoscaling Container based workloads Why Metrics • Monitor resource usage • Scaling • Bottlenecks PaaS / Serverless services
14. @nileshgule Prometheus Architecture
15. @nileshgule Demo 2 – Metrics using Prometheus & Grafana
16. @nileshgule Financial Services App Prometheus integration Scrape Metrics Metrics storage visualise, dashboards backend-service account-service authentication-service forex-service transaction-service service-monitor
17. @nileshgule Distributed Tracing
18. @nileshgule • Distributed Tracing • Understanding complex systems • Performance monitoring and optimizations • Debugging and problem resolution Why Distributed Tracing
19. @nileshgule Financial Services App Jaeger integration Distributed Traces Visualise Traces backend-service account-service authentication-service forex-service transaction-service Jaeger Operator
20. @nileshgule Demo 3 – Distributed Tracing using Jaeger
21. @nileshgule End to End Observability backend-service account-service authentication-service forex-service transaction-service
22. @nileshgule Analogy - Use right tool for right purpose
23. @nileshgule Summary Modern day cloud native applications need new ways to address observability & monitoring ✓ Use best-of-class for given use case ✓ Rely on open standards (e.g. OpenTelemetry) ✓ Build portable observability systems (e.g. hybrid cloud migration) Log Aggregation ✓ Loki helps in centralized logging ✓ Grafana is used to visualize logs and build dashboards Metrics ✓ Prometheus provides easy to use metrics for platforms, applications ✓ Grafana provides visualization capabilities to build intuitive dashboards Distributed Tracing ✓ Jaeger provides distributed tracing capabilities
24. @nileshgule Some Recommendations ♣ Too many agents ♣ Instrumentation, vendor lock-in ♣ Cloud native logs ♣ Cloud native metrics ♣ Cloud native traces ♣ Single pane of glass, correlation ∞ OpenTelemetry collector ∞ OpenTelemetry, OpenMetrics ∞ Fluent Bit / Fluentd, OpenSearch, Loki ∞ Prometheus, Cortex, Thanos ∞ OpenTelemetry, Jaeger, Grafana ∞ Grafana Challenges Tools
25. @nileshgule References Log Aggregation ❖ Grafana Loki Monitoring & Alerting ❖ Prometheus ❖ Grafana ❖ Kube Prometheus stack ❖ Houssem Dellai – Prometheus & Grafana for monitoring Kubernetes Distributed Tracing ❖ Jaeger Tracing
26. @nileshgule Source Code & slide deck Financial Services Demo https://github.com/infofractionalservices/microservices/tree/do cker_build_fixes https://speakerdeck.com/nileshgule/ https://www.slideshare.net/nileshgule/
27. Q&A