Chaos Engineering @ JUG Bonn

Transcript

1. Chaos Engineering Chaos Engineering Wie lerne ich möglichst schnell all meine Kollegen kennen? Wie lerne ich möglichst schnell all meine Kollegen kennen?
2. About me About me Benjamin Wilms Benjamin Wilms codecentric codecentric @MrBWilms @MrBWilms github.com/MrBW github.com/MrBW
3. On the way On the way to the most beautiful to the most beautiful place in the world place in the world
4. PRODUCTION PRODUCTION
5. PRODUCTION PRODUCTION From a developer's point of view From a developer's point of view
6. Unit Tests Unit Tests
7. Integration Tests Integration Tests
8. Test Pyramid dilemma Test Pyramid dilemma
9. Test Coverage Test Coverage 80% - 90% ... 80% - 90% ...
10. this unpleasant feeling remains... this unpleasant feeling remains... how our baby behaves in production? how our baby behaves in production?
12. but... but...
13. it's release day! it's release day!
14. Production Production
15. PRODUCTION PRODUCTION hates you! hates you!
16. Dying app cycle Dying app cycle
17. 8 fallacies of distributed computing 8 fallacies of distributed computing
18. The network is reliable. The network is reliable. Latency is zero. Latency is zero. Bandwidth is infinite. Bandwidth is infinite. The network is secure. The network is secure. Topology doesn't change. Topology doesn't change. There is one administrator. There is one administrator. Transport cost is zero. Transport cost is zero. The network is homogeneous. The network is homogeneous.
19. The network is reliable. The network is reliable. Latency is zero. Latency is zero. Bandwidth is infinite. Bandwidth is infinite. The network is secure. The network is secure. Topology doesn't change. Topology doesn't change. There is one administrator. There is one administrator. Transport cost is zero. Transport cost is zero. The network is homogeneous. The network is homogeneous.
20. The network is reliable. The network is reliable. Latency is zero. Latency is zero. Bandwidth is infinite. Bandwidth is infinite. The network is secure. The network is secure. Topology doesn't change. Topology doesn't change. There is one administrator. There is one administrator. Transport cost is zero. Transport cost is zero. The network is homogeneous. The network is homogeneous.
21. The network is reliable. The network is reliable. Latency is zero. Latency is zero. Bandwidth is infinite. Bandwidth is infinite. The network is secure. The network is secure. Topology doesn't change. Topology doesn't change. There is one administrator. There is one administrator. Transport cost is zero. Transport cost is zero. The network is homogeneous. The network is homogeneous.
22. The network is reliable. The network is reliable. Latency is zero. Latency is zero. Bandwidth is infinite. Bandwidth is infinite. The network is secure. The network is secure. Topology doesn't change. Topology doesn't change. There is one administrator. There is one administrator. Transport cost is zero. Transport cost is zero. The network is homogeneous. The network is homogeneous.
23. The network is reliable. The network is reliable. Latency is zero. Latency is zero. Bandwidth is infinite. Bandwidth is infinite. The network is secure. The network is secure. Topology doesn't change. Topology doesn't change. There is one administrator. There is one administrator. Transport cost is zero. Transport cost is zero. The network is homogeneous. The network is homogeneous.
24. The network is reliable. The network is reliable. Latency is zero. Latency is zero. Bandwidth is infinite. Bandwidth is infinite. The network is secure. The network is secure. Topology doesn't change. Topology doesn't change. There is one administrator. There is one administrator. Transport cost is zero. Transport cost is zero. The network is homogeneous. The network is homogeneous.
25. The network is reliable. The network is reliable. Latency is zero. Latency is zero. Bandwidth is infinite. Bandwidth is infinite. The network is secure. The network is secure. Topology doesn't change. Topology doesn't change. There is one administrator. There is one administrator. Transport cost is zero. Transport cost is zero. The network is homogeneous. The network is homogeneous.
26. Architectures Architectures
27. Architectures Architectures
28. Problem Problem
29. Problem Problem
30. Jepsen Jepsen Jepsen is an effort to improve the safety of distributed databases, queues, Jepsen is an effort to improve the safety of distributed databases, queues, consensus systems, etc. consensus systems, etc.
31. Welcome to Welcome to Chaos Engineering Chaos Engineering
32. Chaos Engineering is NOT Chaos Engineering is NOT
33. Chaos Engineering is NOT... Chaos Engineering is NOT... ...to cause Chaos! ...to cause Chaos!
34. Chaos Engineering is NOT... Chaos Engineering is NOT... ...breaking things just to break them! ...breaking things just to break them!
35. Chaos Engineering is NOT... Chaos Engineering is NOT... ...to run a Chaos Monkey! ...to run a Chaos Monkey!
36. Chaos Engineering is NOT... Chaos Engineering is NOT... ...to let the whole Simian Army out of the cage! ...to let the whole Simian Army out of the cage!
37. Chaos Engineering is NOT... Chaos Engineering is NOT... ...to use it in PRODUCTION! ...to use it in PRODUCTION!
38. Chaos Engineering is NOT... Chaos Engineering is NOT... ...to use it in PRODUCTION, first! ...to use it in PRODUCTION, first!
39. Chaos Engineering is NOT... Chaos Engineering is NOT... ...to replace other kinds of tests! ...to replace other kinds of tests!
40. Chaos Engineering is NOT... Chaos Engineering is NOT... ...to do it alone and without any arrangement! ...to do it alone and without any arrangement!
41. Chaos is not just tools Chaos is not just tools
42. it's culture it's culture
43. this is bob this is bob
44. he is responsible for service A he is responsible for service A
45. These are Bob's teammates These are Bob's teammates
46. They depend on Bob's service They depend on Bob's service
47. They blame Bob for the bad performance of Service A They blame Bob for the bad performance of Service A
48. Stop it!!! Stop it!!!
49. be social and communicative be social and communicative share your experiences and thoughts share your experiences and thoughts stop blaming each other stop blaming each other work together work together
50. What is Chaos Engineering? What is Chaos Engineering?
51. Chaos Engineering book Chaos Engineering book
52. Chaos Engineering is the discipline of experimenting on a distributed system Chaos Engineering is the discipline of experimenting on a distributed system in order to build in order to build confidence confidence in the system’s capability in the system’s capability to withstand to withstand turbulent conditions in turbulent conditions in production production. .
53. Failure is everywhere Failure is everywhere
54. it's all about availability it's all about availability
55. Chaos Engineering 101 Chaos Engineering 101
56. Levels of Chaos Engineering Levels of Chaos Engineering
57. Steady State Steady State
58. Hypothesis Hypothesis What should happen when... What should happen when...
59. Blast Radius Blast Radius
60. Chaos Engineering Cycle Chaos Engineering Cycle
61. If you know your chaos experiment will fail... If you know your chaos experiment will fail... ...don't do it!!! ...don't do it!!!
62. Chaos Engineering Tools Chaos Engineering Tools
63. Basics Basics CPU burning - inspired by Tammy Butow CPU burning - inspired by Tammy Butow # burn.zsh while true; do openssl speed; done EOF # cpu_burning.zsh for i in {1..32} do nohup /bin/zsh burn.zsh & done
64. Basics Basics Disk delay - inspired by Jochen Mader Disk delay - inspired by Jochen Mader device='/dev/ram0' #create a RAM disk modprobe brd rd_nr=1 rd_size=131027 readdelay=100 writedelay=100 size=$(blockdev --getsize $device) # Size in 512-bytes sectors mkfs.ext4 $device echo "0 $size delay $device 0 $readdelay $device 0 $writedelay" | dmsetup create delay
65. Chaos Engineering Tools Chaos Engineering Tools
66. Chaos Toolkit Chaos Toolkit felt in love with felt in love with Chaos Monkey for Spring Boot Chaos Monkey for Spring Boot
67. Chaos Monkey for Spring Boot Chaos Monkey for Spring Boot how does it work how does it work <dependency> <groupid>de.codecentric</groupid> <artifactid>chaos-monkey-spring-boot</artifactid> <version>X.X.X</version> </dependency>
69. Chaos Monkey for Spring Boot Chaos Monkey for Spring Boot Activate Spring Profile Activate Spring Profile chaos-monkey chaos-monkey
70. Chaos Monkey for Spring Boot Chaos Monkey for Spring Boot Watcher Watcher
71. Chaos Monkey for Spring Boot Chaos Monkey for Spring Boot Assaults Assaults
72. Chaos Monkey for Spring Boot Chaos Monkey for Spring Boot Spring Boot Actuator Endpoint Spring Boot Actuator Endpoint Controll via REST Endpoint at runtime Controll via REST Endpoint at runtime
73. Shopping Demo Shopping Demo
74. Shopping Demo Shopping Demo
75. Chaos Experiment Chaos Experiment
76. preparations preparations
84. Result Result Findings? Findings?
85. Advanced topics Advanced topics
86. What else I can offer you What else I can offer you
87. Thanks for having me Thanks for having me Benjamin Wilms Benjamin Wilms @MrBWilms @MrBWilms benjamin.wilms@codecentric.de benjamin.wilms@codecentric.de Shopping demo (docker-compose file) Shopping demo (docker-compose file) Shopping Demo Docker Hub Shopping Demo Docker Hub