Slide 1
Slide 1 text
WITHSTANDING TURBULENT CONDITIONS IN PRODUCTION
Slide 2
Slide 2 text
BENJAMIN WILMS
CODECENTRIC
@MRBWILMS
GITHUB.COM/MRBW
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
No content
Slide 5
Slide 5 text
FROM A DEVELOPER'S POINT OF VIEW
Slide 6
Slide 6 text
No content
Slide 7
Slide 7 text
No content
Slide 8
Slide 8 text
No content
Slide 9
Slide 9 text
80% - 90% ...
Slide 10
Slide 10 text
HOW OUR BABY BEHAVES IN PRODUCTION?
Slide 11
Slide 11 text
No content
Slide 12
Slide 12 text
No content
Slide 13
Slide 13 text
No content
Slide 14
Slide 14 text
No content
Slide 15
Slide 15 text
No content
Slide 16
Slide 16 text
No content
Slide 17
Slide 17 text
No content
Slide 18
Slide 18 text
No content
Slide 19
Slide 19 text
No content
Slide 20
Slide 20 text
No content
Slide 21
Slide 21 text
No content
Slide 22
Slide 22 text
No content
Slide 23
Slide 23 text
No content
Slide 24
Slide 24 text
No content
Slide 25
Slide 25 text
No content
Slide 26
Slide 26 text
No content
Slide 27
Slide 27 text
No content
Slide 28
Slide 28 text
No content
Slide 29
Slide 29 text
No content
Slide 30
Slide 30 text
JEPSEN IS AN EFFORT TO IMPROVE THE SAFETY OF DISTRIBUTED DATABASES, QUEUES, CONSENSUS SYSTEMS, ETC.
Slide 31
Slide 31 text
No content
Slide 32
Slide 32 text
No content
Slide 33
Slide 33 text
CHAOS ENGINEERING IS NOT...
...TO CAUSE CHAOS!
Slide 34
Slide 34 text
CHAOS ENGINEERING IS NOT...
...BREAKING THINGS JUST TO BREAK THEM!
Slide 35
Slide 35 text
CHAOS ENGINEERING IS NOT...
...TO RUN A CHAOS MONKEY!
Slide 36
Slide 36 text
CHAOS ENGINEERING IS NOT...
...TO LET THE WHOLE SIMIAN ARMY OUT OF THE CAGE!
Slide 37
Slide 37 text
CHAOS ENGINEERING IS NOT...
...TO USE IT IN PRODUCTION!
Slide 38
Slide 38 text
CHAOS ENGINEERING IS NOT...
...TO USE IT IN PRODUCTION, FIRST!
Slide 39
Slide 39 text
CHAOS ENGINEERING IS NOT...
...TO REPLACE OTHER KINDS OF TESTS!
Slide 40
Slide 40 text
CHAOS ENGINEERING IS NOT...
...TO DO IT ALONE AND WITHOUT ANY ARRANGEMENT!
Slide 41
Slide 41 text
No content
Slide 42
Slide 42 text
No content
Slide 43
Slide 43 text
THIS IS BOB
Slide 44
Slide 44 text
HE IS RESPONSIBLE FOR SERVICE A
Slide 45
Slide 45 text
THESE ARE BOB'S TEAMMATES
Slide 46
Slide 46 text
THEY DEPEND ON BOB'S SERVICE
Slide 47
Slide 47 text
THEY BLAME BOB FOR THE BAD PERFORMANCE OF SERVICE A
Slide 48
Slide 48 text
BE SOCIAL AND COMMUNICATIVE
SHARE YOUR EXPERIENCES AND THOUGHTS
STOP BLAMING EACH OTHER
WORK TOGETHER
Slide 49
Slide 49 text
No content
Slide 50
Slide 50 text
No content
Slide 51
Slide 51 text
CHAOS ENGINEERING IS THE DISCIPLINE OF EXPERIMENTING ON A DISTRIBUTED SYSTEM
IN ORDER TO BUILD CONFIDENCE IN THE SYSTEM’S CAPABILITY
TO WITHSTAND TURBULENT CONDITIONS IN PRODUCTION.
Slide 52
Slide 52 text
No content
Slide 53
Slide 53 text
No content
Slide 54
Slide 54 text
No content
Slide 55
Slide 55 text
WHAT SHOULD HAPPEN WHEN...
Slide 56
Slide 56 text
No content
Slide 57
Slide 57 text
No content
Slide 58
Slide 58 text
IF YOU KNOW YOUR CHAOS EXPERIMENT WILL FAIL...
...DON'T DO IT!!!
Slide 59
Slide 59 text
No content
Slide 60
Slide 60 text
CPU BURNING - INSPIRED BY TAMMY BUTOW
# burn.zsh
while true;
do openssl speed;
done
EOF
# cpu_burning.zsh
for i in {1..32}
do
nohup /bin/zsh burn.zsh &
done
Slide 61
Slide 61 text
NET DELAY - INSPIRED BY JOCHEN MADER
# tc Linux traffic control, package iproute2
# add 1000ms to eth0
tc qdisc add dev eth0 root netem delay 1000
Slide 62
Slide 62 text
No content
Slide 63
Slide 63 text
PLATTFORMS
Slide 64
Slide 64 text
CHAOS TOOLKIT
FELT IN LOVE WITH
CHAOS MONKEY FOR SPRING BOOT
Slide 65
Slide 65 text
HOW DOES IT WORK
de.codecentric
chaosmonkeyspringboot
X.X.X
Slide 66
Slide 66 text
SIDECAR PATTERN
java cp yourapp.jar
Dloader.path=chaosmonkeyspringbootX.X.X.jar
org.springframework.boot.loader.PropertiesLauncher
spring.profiles.active=chaosmonkey
Slide 67
Slide 67 text
ACTIVATE SPRING PROFILE
CHAOS-MONKEY
Slide 68
Slide 68 text
WATCHER
Slide 69
Slide 69 text
ASSAULTS
Slide 70
Slide 70 text
SPRING BOOT ACTUATOR ENDPOINT
CONTROL VIA REST ENDPOINT AT RUNTIME
Slide 71
Slide 71 text
No content
Slide 72
Slide 72 text
No content
Slide 73
Slide 73 text
No content
Slide 74
Slide 74 text
No content
Slide 75
Slide 75 text
No content
Slide 76
Slide 76 text
No content
Slide 77
Slide 77 text
No content
Slide 78
Slide 78 text
No content
Slide 79
Slide 79 text
No content
Slide 80
Slide 80 text
No content
Slide 81
Slide 81 text
No content
Slide 82
Slide 82 text
FINDINGS?
Slide 83
Slide 83 text
No content
Slide 84
Slide 84 text
BENJAMIN WILMS
@MRBWILMS
[email protected]
SHOPPING DEMO (DOCKER-COMPOSE FILE)
SHOPPING DEMO DOCKER HUB