@nileshgule Improve Monitoring and Observability for Kubernetes with OSS tools

Nilesh Gule ARCHITECT | MICROSOFT MVP | First Docker Captain in Singapore “Code with Passion and Strive for Excellence” nileshgule @nileshgul e Nilesh Gule NileshGule www.handsonarchitect.co m https://www.youtube.com/@nilesh-gule

@nileshgule

@nileshgule CNCF cloud trail https://github.com/cncf/trailmap

@nileshgule CNCF Observability landscape https://landscape.cncf.io

@nileshgule CNCF Observability Radar https://radar.cncf.io/2020-09-observability

@nileshgule 3 Pillars of Observability Logs Metrics Traces

@nileshgule Centralized Logging

@nileshgule ❑ Application specific ❖ Long term log retention for compliance reasons ❖ Workloads scheduled on different nodes during application restarts / updates ❖ Autoscaling workloads ❑ Kubernetes upgrades ❖ Auto healing can reschedule workloads ❖ Underlying nodes added / deleted during cluster scaling ❖ Underlying nodes replaced during cluster upgrades Container based workloads Why centralized logging ❖ Not much control over underlying infra ❖ Relies on cloud prover specific logging and monitoring solution PaaS / Serverless services

@nileshgule Financial Services App Loki integration Log collector Log storage Log search, visualise, dashboards backend-service account-service authentication-service forex-service transaction-service

@nileshgule Demo 1 – Log Aggregation with Loki

@nileshgule Metrics

@nileshgule • Application specific • Monitor resource usage • Monitor scaling needs • Monitor anomalies / outliers • Kubernetes platform level • Monitor cluster resources (CPU / RAM) • API health • Autoscaling Container based workloads Why Metrics • Monitor resource usage • Scaling • Bottlenecks PaaS / Serverless services

@nileshgule Prometheus Architecture

@nileshgule Demo 2 – Metrics using Prometheus & Grafana

@nileshgule Financial Services App Prometheus integration Scrape Metrics Metrics storage visualise, dashboards backend-service account-service authentication-service forex-service transaction-service service-monitor

@nileshgule Distributed Tracing

@nileshgule • Distributed Tracing • Understanding complex systems • Performance monitoring and optimizations • Debugging and problem resolution Why Distributed Tracing

@nileshgule Financial Services App Jaeger integration Distributed Traces Visualise Traces backend-service account-service authentication-service forex-service transaction-service Jaeger Operator

@nileshgule Demo 3 – Distributed Tracing using Jaeger

@nileshgule End to End Observability backend-service account-service authentication-service forex-service transaction-service

@nileshgule Analogy - Use right tool for right purpose

@nileshgule Summary Modern day cloud native applications need new ways to address observability & monitoring ✓ Use best-of-class for given use case ✓ Rely on open standards (e.g. OpenTelemetry) ✓ Build portable observability systems (e.g. hybrid cloud migration) Log Aggregation ✓ Loki helps in centralized logging ✓ Grafana is used to visualize logs and build dashboards Metrics ✓ Prometheus provides easy to use metrics for platforms, applications ✓ Grafana provides visualization capabilities to build intuitive dashboards Distributed Tracing ✓ Jaeger provides distributed tracing capabilities

@nileshgule Some Recommendations ♣ Too many agents ♣ Instrumentation, vendor lock-in ♣ Cloud native logs ♣ Cloud native metrics ♣ Cloud native traces ♣ Single pane of glass, correlation ∞ OpenTelemetry collector ∞ OpenTelemetry, OpenMetrics ∞ Fluent Bit / Fluentd, OpenSearch, Loki ∞ Prometheus, Cortex, Thanos ∞ OpenTelemetry, Jaeger, Grafana ∞ Grafana Challenges Tools

@nileshgule References Log Aggregation ❖ Grafana Loki Monitoring & Alerting ❖ Prometheus ❖ Grafana ❖ Kube Prometheus stack ❖ Houssem Dellai – Prometheus & Grafana for monitoring Kubernetes Distributed Tracing ❖ Jaeger Tracing

@nileshgule Source Code & slide deck Financial Services Demo https://github.com/infofractionalservices/microservices/tree/do cker_build_fixes https://speakerdeck.com/nileshgule/ https://www.slideshare.net/nileshgule/

Q&A