Automating Chaos Attacks

Slide 1

Slide 1 text

1 Automating Chaos Attacks Daniel Albuquerque Nikos Katirtzis @worldtiki @nikos912000

Slide 2

Slide 2 text

2 Expedia Group Proprietary and Confidential Expedia Group

Slide 3

Slide 3 text

3 Expedia Group Proprietary and Confidential Validating fallbacks Application CI Type Environment

Slide 4

Slide 4 text

4 Expedia Group Proprietary and Confidential Reviews on the Website ML-sorted reviews Date-sorted reviews

Slide 5

Slide 5 text

5 Expedia Group Proprietary and Confidential Reviews Architecture Service A fallback Service B Service C Sorts reviews based on ML Sorts reviews based on date

Slide 6

Slide 6 text

6 Expedia Group Proprietary and Confidential Steady State Service A fallback Service B Service C Sorts reviews based on ML Sorts reviews based on date When Service B is accessible reviews are sorted based on usefulness (ML)

Slide 7

Slide 7 text

7 Expedia Group Proprietary and Confidential Validating fallback using Resilience Tests Service A fallback Service B Service C Sorts reviews based on ML Sorts reviews based on date When Service B is not accessible we fall back to Service C and sort reviews by date Testcontainers Toxiproxy

Slide 8

Slide 8 text

8 Expedia Group Proprietary and Confidential Validating fallback using Resilience Tests Service A fallback Service B Service C Sorts reviews based on ML Sorts reviews based on date When Service B is not accessible we fall back to Service C and sort reviews by date

Slide 9

Slide 9 text

9 Expedia Group Proprietary and Confidential Automated Tests in CI Post-merge Pre-merge Deploy to test environment

Slide 10

Slide 10 text

10 Expedia Group Proprietary and Confidential Attacking Kubernetes Pods Platform Production Type Environment

Slide 11

Slide 11 text

11 Expedia Group Proprietary and Confidential Manual Pod Termination https://expediadotcom.github.io/haystack/

Slide 12

Slide 12 text

12 Expedia Group Proprietary and Confidential Manual Pod Termination https://expediadotcom.github.io/haystack/ !

Slide 13

Slide 13 text

13 Expedia Group Proprietary and Confidential Manual Pod Termination https://expediadotcom.github.io/haystack/ ! ! ! !

Slide 14

Slide 14 text

14 Expedia Group Proprietary and Confidential Automated Pod Termination using Kube-Monkey Kill one pod please! !

Slide 15

Slide 15 text

15 Expedia Group Proprietary and Confidential Kube-Monkey - Integration A base helm chart (configuration template) that all our services use Percentage or a fixed number instead Opt-in expedia-parent-chart: kubemonkey: enabled: true killMode: "percentage" killValue: 50

Slide 16

Slide 16 text

16 Expedia Group Proprietary and Confidential Kube-Monkey - Notifications Schedule Report Attack Report

Slide 17

Slide 17 text

17 Expedia Group Proprietary and Confidential Learnings https://expediagroup.github.io/mittens/ Cold start issues

Slide 18

Slide 18 text

18 Expedia Group Proprietary and Confidential The Chaos Engineering Iceberg Culture Observability Pre-experiment analysis Post-experiment analysis Automation Learning

Slide 19

Slide 19 text

19 Expedia Group Proprietary and Confidential Daniel Albuquerque Nikos Katirtzis @worldtiki @nikos912000 Thank you!