Chaos Engineering - W-JAX 2018

Chaos Engineering @ W-JAX 2018
- ChaosToolkit
- ChaosHub
- Chaos Monkey for Spring Boot

Transcript

1. WITHSTANDING TURBULENT CONDITIONS IN PRODUCTION
2. BENJAMIN WILMS CODECENTRIC @MRBWILMS GITHUB.COM/MRBW
5. FROM A DEVELOPER'S POINT OF VIEW
9. 80% - 90% ...
10. HOW OUR BABY BEHAVES IN PRODUCTION?
31. JEPSEN IS AN EFFORT TO IMPROVE THE SAFETY OF DISTRIBUTED DATABASES, QUEUES, CONSENSUS SYSTEMS, ETC.
34. CHAOS ENGINEERING IS NOT... ...TO CAUSE CHAOS!
35. CHAOS ENGINEERING IS NOT... ...BREAKING THINGS JUST TO BREAK THEM!
36. CHAOS ENGINEERING IS NOT... ...TO RUN A CHAOS MONKEY!
37. CHAOS ENGINEERING IS NOT... ...TO LET THE WHOLE SIMIAN ARMY OUT OF THE CAGE!
38. CHAOS ENGINEERING IS NOT... ...TO USE IT IN PRODUCTION!
39. CHAOS ENGINEERING IS NOT... ...TO USE IT IN PRODUCTION, FIRST!
40. CHAOS ENGINEERING IS NOT... ...TO REPLACE OTHER KINDS OF TESTS!
41. CHAOS ENGINEERING IS NOT... ...TO DO IT ALONE AND WITHOUT ANY ARRANGEMENT!
44. THIS IS BOB
45. HE IS RESPONSIBLE FOR SERVICE A
46. THESE ARE BOB'S TEAMMATES
47. THEY DEPEND ON BOB'S SERVICE
48. THEY BLAME BOB FOR THE BAD PERFORMANCE OF SERVICE A
49. BE SOCIAL AND COMMUNICATIVE SHARE YOUR EXPERIENCES AND THOUGHTS STOP BLAMING EACH OTHER WORK TOGETHER
52. CHAOS ENGINEERING IS THE DISCIPLINE OF EXPERIMENTING ON A DISTRIBUTED SYSTEM IN ORDER TO BUILD CONFIDENCE IN THE SYSTEM’S CAPABILITY TO WITHSTAND TURBULENT CONDITIONS IN PRODUCTION.
56. STEADY STATE ORDERS PER MINUTE ON A TYPICAL MONDAY MORNING
57. STEADY STATE ORDERS PER MINUTE
58. STEADY STATE EXPERIMENT WAS CANCELED
59. WHAT SHOULD HAPPEN WHEN...
62. IF YOU KNOW YOUR CHAOS EXPERIMENT WILL FAIL... ...DON'T DO IT!!!
64. CPU BURNING - INSPIRED BY TAMMY BUTOW # burn.zsh while true; do openssl speed; done EOF # cpu_burning.zsh for i in {1..32} do nohup /bin/zsh burn.zsh & done
65. STRESS CPU stress ­­cpu 2 ­­io 1 ­­vm 1 ­­vm­bytes 128M ­­timeout 10s ­­verbose
66. TRAFFIC CONTROL (TC) PACKAGE IPROUTE2
68. PLATFORMS
69. CHAOS TOOLKIT FELT IN LOVE WITH CHAOS MONKEY FOR SPRING BOOT
70. HOW DOES IT WORK <dependency> <groupid>de.codecentric</groupid> <artifactid>chaos­monkey­spring­boot</artifactid> <version>2.0.1</version> </dependency>
71. (SIDECAR PATTERN) java ­cp your­app.jar ­Dloader.path=chaos­monkey­spring­boot­2.0.1­jar­with­dependencies.jar org.springframework.boot.loader.PropertiesLauncher ­­spring.profiles.active=chaos­monkey ­­spring.config.location=file:./chaos­monkey.properties
72. WATCHER
73. ASSAULTS
74. SPRING BOOT ACTUATOR ENDPOINT CONTROL VIA REST ENDPOINT AT RUNTIME
86. FINDINGS?
91. BENJAMIN WILMS @MRBWILMS BENJAMIN.WILMS@CODECENTRIC.DE SHOPPING DEMO CHAOS MONKEY FOR SPRING BOOT CHAOSTOOLKIT & CHAOSHUB