Chaos Engineering - AWS Community Days 2018

Chaos Engineering 101 introduction incl. Live Demo about injecting latency at runtime on Application Level via Chaos Monkey for Spring Boot.

Transcript

1. WITHSTANDING TURBULENT CONDITIONS IN PRODUCTION
2. BENJAMIN WILMS CODECENTRIC @MRBWILMS GITHUB.COM/MRBW
5. FROM A DEVELOPER'S POINT OF VIEW
9. 80% - 90% ...
10. HOW OUR BABY BEHAVES IN PRODUCTION?
30. JEPSEN IS AN EFFORT TO IMPROVE THE SAFETY OF DISTRIBUTED DATABASES, QUEUES, CONSENSUS SYSTEMS, ETC.
33. CHAOS ENGINEERING IS NOT... ...TO CAUSE CHAOS!
34. CHAOS ENGINEERING IS NOT... ...BREAKING THINGS JUST TO BREAK THEM!
35. CHAOS ENGINEERING IS NOT... ...TO RUN A CHAOS MONKEY!
36. CHAOS ENGINEERING IS NOT... ...TO LET THE WHOLE SIMIAN ARMY OUT OF THE CAGE!
37. CHAOS ENGINEERING IS NOT... ...TO USE IT IN PRODUCTION!
38. CHAOS ENGINEERING IS NOT... ...TO USE IT IN PRODUCTION, FIRST!
39. CHAOS ENGINEERING IS NOT... ...TO REPLACE OTHER KINDS OF TESTS!
40. CHAOS ENGINEERING IS NOT... ...TO DO IT ALONE AND WITHOUT ANY ARRANGEMENT!
43. THIS IS BOB
44. HE IS RESPONSIBLE FOR SERVICE A
45. THESE ARE BOB'S TEAMMATES
46. THEY DEPEND ON BOB'S SERVICE
47. THEY BLAME BOB FOR THE BAD PERFORMANCE OF SERVICE A
48. BE SOCIAL AND COMMUNICATIVE SHARE YOUR EXPERIENCES AND THOUGHTS STOP BLAMING EACH OTHER WORK TOGETHER
51. CHAOS ENGINEERING IS THE DISCIPLINE OF EXPERIMENTING ON A DISTRIBUTED SYSTEM IN ORDER TO BUILD CONFIDENCE IN THE SYSTEM’S CAPABILITY TO WITHSTAND TURBULENT CONDITIONS IN PRODUCTION.
55. WHAT SHOULD HAPPEN WHEN...
58. IF YOU KNOW YOUR CHAOS EXPERIMENT WILL FAIL... ...DON'T DO IT!!!
60. CPU BURNING - INSPIRED BY TAMMY BUTOW # burn.zsh while true; do openssl speed; done EOF # cpu_burning.zsh for i in {1..32} do nohup /bin/zsh burn.zsh & done
61. NET DELAY - INSPIRED BY JOCHEN MADER # tc Linux traffic control, package iproute2 # add 1000ms to eth0 tc qdisc add dev eth0 root netem delay 1000
63. PLATTFORMS
64. CHAOS TOOLKIT FELT IN LOVE WITH CHAOS MONKEY FOR SPRING BOOT
65. HOW DOES IT WORK <dependency> <groupid>de.codecentric</groupid> <artifactid>chaos­monkey­spring­boot</artifactid> <version>X.X.X</version> </dependency>
66. SIDECAR PATTERN java ­cp your­app.jar ­Dloader.path=chaos­monkey­spring­boot­X.X.X.jar org.springframework.boot.loader.PropertiesLauncher ­­spring.profiles.active=chaos­monkey
67. ACTIVATE SPRING PROFILE CHAOS-MONKEY
68. WATCHER
69. ASSAULTS
70. SPRING BOOT ACTUATOR ENDPOINT CONTROL VIA REST ENDPOINT AT RUNTIME
82. FINDINGS?
84. BENJAMIN WILMS @MRBWILMS BENJAMIN.WILMS@CODECENTRIC.DE SHOPPING DEMO (DOCKER-COMPOSE FILE) SHOPPING DEMO DOCKER HUB