Improve Monitoring and Observability of Kubernetes with OSS tools

Deck used for the Surati Tech Talks 2022 event on 11 January. The demo covers end to end Monitoring and Observability for Kubernetes using Elasticsearch, Fluentbit and Kibana for log aggregation, Prometheus & Grafana for Monitoring & Alerting and Sentry for Exception handling. The target environment is Azure Kubernetes Service (AKS) cluster.

Transcript

1. @nileshgule Improve Monitoring and Observability for Kubernetes with OSS tools
2. $whoami { “name” : “Nilesh Gule”, “website” : “https://www.HandsOnArchitect.com", “github” : “https://GitHub.com/NileshGule" “twitter” : “@nileshgule”, “linkedin” : “https://www.linkedin.com/in/nileshgule”, “likes” : “Technical Evangelism, Cricket”, “co-organizer” : “Azure Singapore UG” } https://bit.ly/AzureSingaporeSessions
3. @nileshgule Pre-requisites Self contained application with all its dependencies Docker ❖ Orchestrates containers ❖ Self healing ❖ Service discovery ❖ Scaling Kubernetes ❖ Scalable apps in dynamic environments (public / private / hybrid clouds) ❖ Exemplified by Containers, service meshes, microservices, immutable infrastructure & declarative APIs ❖ Loosely coupled systems, resilient, observable & manageable ❖ Robust automation Cloud Native Applications
4. @nileshgule
5. @nileshgule CNCF cloud trail https://github.com/cncf/trailmap
6. @nileshgule Centralized Logging
7. @nileshgule ❑ Application specific ❖ Long term log retention for compliance reasons ❖ Workloads scheduled on different nodes during application restarts / updates ❖ Autoscaling workloads ❑ Kubernetes upgrades ❖ Auto healing can reschedule workloads ❖ Underlying nodes added / deleted during cluster scaling ❖ Underlying nodes replaced during cluster upgrades Container based workloads Why centralized logging ❖ Not much control over underlying infra ❖ Relies on cloud prover specific logging and monitoring solution PaaS / Serverless services
8. @nileshgule Tech Talks EFK integration Log collector Log storage Log search, visualise, dashboards rabbitmq-producer-service rabbitmq-consumer-deployment
9. @nileshgule Demo 1 – Log Aggregation with EFK
10. @nileshgule Monitoring and Alerting
11. @nileshgule • Application specific • Monitor resource usage • Monitor scaling needs • Monitor anomalies / outliers • Kubernetes platform level • Monitor cluster resources (CPU / RAM) • API health • Autoscaling Container based workloads Why Monitoring & Alerting • Monitor resource usage • Scaling • Bottlenecks PaaS / Serverless services
12. @nileshgule Prometheus Architecture
13. @nileshgule Demo 2 – Metrics using Prometheus & Grafana
14. @nileshgule Spring Boot Conference App integration https://github.com/NileshGule/spring-boot-conference-app/tree/mssql-server conference-demo-service-monitor conference-demo-service
15. @nileshgule Exception Handling
16. @nileshgule Sentry Architecture https://develop.sentry.dev/architecture/
17. @nileshgule Spring Boot Sentry integration conference-demo-service Managed Kubernetes cluster
18. @nileshgule Demo 3 – Exception aggregation using Sentry
19. @nileshgule End to End Observability
20. @nileshgule Summary Modern day cloud native applications need new ways to address observability & monitoring Log Aggregation ✓ EFK stack helps in centralized logging ✓ Kibana is used to visualize logs and build dashboards Monitoring & Alerting ✓ Prometheus provides easy to use metrics for platforms, applications ✓ Grafana provides visualization capabilities to build intuitive dashboards Exception Aggregation ✓ Sentry provides Exception Aggregation capabilities ✓ Excellent telemetry data captured by Sentry to help diagnose problems
21. @nileshgule References Log Aggregation ❖ Elastic stack ❖ Kibana ❖ Fluentbit Monitoring & Alerting ❖ Prometheus ❖ Grafana ❖ Kube Prometheus stack ❖ Houssem Dellai – Prometheus & Grafana for monitoring Kubernetes Sentry ❖ Sentry docs
22. @nileshgule Source Code & slide deck Tech Talks https://github.com/NileshGule/pd-tech-fest-2019 Observability & Monitoring markdown Conference app https://github.com/NileshGule/spring-boot-conference-app/tree/mssql-server https://speakerdeck.com/nileshgule/ https://www.slideshare.net/nileshgule/
23. Nilesh Gule ARCHITECT | MICROSOFT MVP “Code with Passion and Strive for Excellence” nileshgule @nileshgule Nilesh Gule NileshGule www.handsonarchitect.com https://bit.ly/youtube-nileshgule
24. Q&A