Automating Chaos Attacks

Deck presented at Chaos Conf 2020: https://www.chaosconf.io/chaosconf/session/311566/automating-chaos-attacks

Transcript

1. 1 Automating Chaos Attacks Daniel Albuquerque Nikos Katirtzis @worldtiki @nikos912000
2. 2 Expedia Group Proprietary and Confidential Expedia Group
3. 3 Expedia Group Proprietary and Confidential Validating fallbacks Application CI Type Environment
4. 4 Expedia Group Proprietary and Confidential Reviews on the Website ML-sorted reviews Date-sorted reviews
5. 5 Expedia Group Proprietary and Confidential Reviews Architecture Service A fallback Service B Service C Sorts reviews based on ML Sorts reviews based on date
6. 6 Expedia Group Proprietary and Confidential Steady State Service A fallback Service B Service C Sorts reviews based on ML Sorts reviews based on date When Service B is accessible reviews are sorted based on usefulness (ML)
7. 7 Expedia Group Proprietary and Confidential Validating fallback using Resilience Tests Service A fallback Service B Service C Sorts reviews based on ML Sorts reviews based on date When Service B is not accessible we fall back to Service C and sort reviews by date Testcontainers Toxiproxy
8. 8 Expedia Group Proprietary and Confidential Validating fallback using Resilience Tests Service A fallback Service B Service C Sorts reviews based on ML Sorts reviews based on date When Service B is not accessible we fall back to Service C and sort reviews by date
9. 9 Expedia Group Proprietary and Confidential Automated Tests in CI Post-merge Pre-merge Deploy to test environment
10. 10 Expedia Group Proprietary and Confidential Attacking Kubernetes Pods Platform Production Type Environment
11. 11 Expedia Group Proprietary and Confidential Manual Pod Termination https://expediadotcom.github.io/haystack/
12. 12 Expedia Group Proprietary and Confidential Manual Pod Termination https://expediadotcom.github.io/haystack/ !
13. 13 Expedia Group Proprietary and Confidential Manual Pod Termination https://expediadotcom.github.io/haystack/ ! ! ! !
14. 14 Expedia Group Proprietary and Confidential Automated Pod Termination using Kube-Monkey Kill one pod please! !
15. 15 Expedia Group Proprietary and Confidential Kube-Monkey - Integration A base helm chart (configuration template) that all our services use Percentage or a fixed number instead Opt-in expedia-parent-chart: kubemonkey: enabled: true killMode: "percentage" killValue: 50
16. 16 Expedia Group Proprietary and Confidential Kube-Monkey - Notifications Schedule Report Attack Report
17. 17 Expedia Group Proprietary and Confidential Learnings https://expediagroup.github.io/mittens/ Cold start issues
18. 18 Expedia Group Proprietary and Confidential The Chaos Engineering Iceberg Culture Observability Pre-experiment analysis Post-experiment analysis Automation Learning
19. 19 Expedia Group Proprietary and Confidential Daniel Albuquerque Nikos Katirtzis @worldtiki @nikos912000 Thank you!