Introduction to Chaos Engineering with Kubernetes

Transcript

1. Chaos Engineering with Kubernetes 10 Jun 2021 Spiros Economakis Site Reliability Engineering Lead @ Mattermost Cloud
2. 2 Resilience Ability of a system to adapt to changes, failures and disturbances
3. Using chaos to bring resiliency to your infrastructure and applications 3
4. 4 “ Chaos doesn’t cause problems. It reveals them. ” Nora Jones, CEO of Jeli, (ex) Head of Chaos Engineering @Slack, @Netflix
5. 5 Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. https://principlesofchaos.org/
6. Phases of Chaos Engineering 6
7. 7 Fault Injection • Infrastructure Level • Application Level • Network Level
8. Kubernetes Cluster 8
9. K8s failures 9
10. Managed K8s Failures (AWS EKS) 10
11. 11 Bring me Thanos (Chaos) in kubernetes with the right tooling
12. Chaos Mesh 12
13. Chaos Mesh 13
14. Chaos Toolkit 14
15. Chaos Toolkit 15
16. Chaos Toolkit 16 version: 1.0.0 title: Pod should be automatically killed and restarted when unhealthy description: Can we trust Kubernetes to restart our microservice when it detects it is unhealthy? tags: - microservice - kubernetes - python configuration: webapp_service_url: type: env key: WEBAPP_SERVICE_ADDR prometheus_base_url: type: env key: PROMETHEUS_ADDR
17. Chaos Toolkit 17 steady-state-hypothesis : title: Services are all available and healthy probes: - type: probe name: all-services-are-healthy tolerance : true provider : type: python module: chaosk8s.probes func: all_microservices_healthy - type: probe name: webapp-is-available tolerance : true provider : type: python module: chaosk8s.probes func: microservice_available_and_healthy arguments : name: webapp-app
18. Chaos Toolkit 18 method: - type: action name: talk-to-webapp background : true provider: type: process path: vegeta timeout: 63 arguments: attack: '' "-duration" : 60s "-connections" : '1' "-rate": '1' "-output": report.bin "-targets" : urls.txt - type: action name: confirm-purchase provider: type: http url: "${webapp_service_url}/purchase/confirm"
19. Chaos Toolkit 19 - type: probe name: collect-how-many-times-our-service-container-restarted-in-the-last-minute provider: type: python module: chaosprometheus.probes func: query_interval arguments: query: kube_pod_container_status_restarts{container="webapp-app"} start: 2 minutes ago end: now pauses: before: 45 - type: probe name: read-webapp-logs-for-the-pod-that-was-killed provider: type: python module: chaosk8s.probes func: read_microservices_logs arguments: name: webapp-app from_previous : true
20. 20 DON’T blame anyone but learn and run post mortems (COE)
21. 21 Fix & improve, otherwise Thanos will find you in an infinity loop of time travel
22. 22 Start small and increase the blast radius
23. 23 Chaos Gamedays are an ideal way to ease into Chaos Engineering
24. Thank you!