What is suitable for Chaos Engineering? ~ChaosConf 2019 recap~

Slide 1

Slide 1 text

What is suitable for Chaos Engineering? ChaosConf2019 recap @AWS LOFT TOKYO Fumihiko Hata | Solutions Architect, AWS Japan | 2019.11.11

Slide 2

Slide 2 text

Fumihiko Hata Role: Solutions Architect, AWS Japan Customer: Game companies Favorite: Elm, Clojure, Tutorial D Who am I

Slide 3

Slide 3 text

Agenda Chaos Engineering (* &) &) ChaosConf 2019.9'-% @SanFrancisco/ !"+$#, What is suitable for Chaos Engineering?

Slide 4

Slide 4 text

Agenda 1. Industries 2. Microservices or Monorith 3. Server-based or Server-less 4. Cloud or on-prem 5. Production or Non-production env What is suitable for Chaos Engineering?

Slide 5

Slide 5 text

Industries

Slide 6

Slide 6 text

Chaos Engineering

Slide 7

Slide 7 text

Chaos Engineering Chaos Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf

Slide 8

Slide 8 text

Chaos Engineering Chaos Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf

Slide 9

Slide 9 text

Chaos Engineering Chaos Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf

Slide 10

Slide 10 text

ChaosConf2019 EC https://speakerdeck.com/chaosconf/humans-are-fallible https://speakerdeck.com/chaosconf/embracing-chaos https://speakerdeck.com/chaosconf/a-roadmap-towards-chaos-engineering

Slide 11

Slide 11 text

EC 7 • .40 , • $/( ,#12)0'6 • +5 !"3%-*&

Slide 12

Slide 12 text

Uber EMBRACING CHAOS! Paul Osman, Under Armour Ana Medina, Gremlin • uDestroy: Uber Chaos Experiment • Uber Gremlin Ana https://speakerdeck.com/chaosconf/ embracing-chaos

Slide 13

Slide 13 text

Uber https://speakerdeck.com/chaosconf/ embracing-chaos

Slide 14

Slide 14 text

9513 • ChaosConf &+ Gremlin 4$8 Chaos Engineering .( %1

Slide 15

Slide 15 text

:624? National Australia Bank • ChaosConf &+ Gremlin 5$9 Chaos Engineering .' %2=0,>-:624 • ;7:63< ' FinTech ) 1 :6 "#!#' • /*8( @ NAB deploys Chaos Monkey to kill servers 24/7 - Software – Cloud - iTnews 2014 https://www.itnews.com.au/news/nab-deploys-chaos-monkey-to-kill-servers-24-7-382285

Slide 16

Slide 16 text

9513> Capital One • ChaosConf %* Gremlin 4#8 Chaos Engineering -& $1

Slide 17

Slide 17 text

NI@CQ TLPT (Threat-Led Penetration Test) H5$,%" *,(+ 2018710>NI8NI.M #,')9/2 0F =O

Slide 18

Slide 18 text

NHBES TLPT (Threat-Led Penetration Test) G:(.#)'%,.!+-$#% M86G:(.#)'%,. !+-$#%QTLPTRO 74A 2IL TLPT Chaos Experiment ?81 =>@CD 0 PF 5 9 /;JKNH BE -"&3< ,*.% ! TLPT # " https://www.fsa.go.jp/common/about/research/20180516.html

Slide 19

Slide 19 text

^XIRc TLPT (Threat-Led Penetration Test) “W;(/")&%-/!,.$"% O<0RB Y]^XK_ '/ ED>BC8[1 >BV6b ML$"%” 2.2.1. $"%JZ “TLPT O<0R4ED>B V6bML$"% ?$"%

Slide 20

Slide 20 text

^XIRc TLPT (Threat-Led Penetration Test) “W;(/")&%-/!,.$"% O<0RB Y]^XK_ '/ ED>BC8[1 >BV6b ML$"%” 2.2.1. $"%JZ “TLPT O<0R4ED>B V6bML$"% ?$"%

Slide 21

Slide 21 text

Microservices or Monorith

Slide 22

Slide 22 text

Microservices

Slide 23

Slide 23 text

Microservices )/ 1'" • FH N@< 1'"T) / 1'"U(.1!-0B> 1%$,4IK; !"%*SG(/1$ • 6 1'" API R5 • 1'"

Slide 24

Slide 24 text

Microservices • 7T( .U"1&$Q Failure BS9R Chaos Experiment @HPF"1 &$;5A%1) J><? • Netflix Amazon ( ."1&$ DKI= • 2G Netflix "1&$Q ',!1#+/0-* Failure Context 4E3L O"1&$Q 8 6M:CN

Slide 25

Slide 25 text

Monorith THINK BIG: CHAOS TESTING A MONOLITH https://speakerdeck.com/chaosconf/think-big-chaos-testing-a-monolith • 3;8JC$;-&/6(*. ;2 Mailchimp ZD Mailchimp RMI SRE Caroline '(%5: • 2000 • 1!9$;-&%&)2 OB[FE A#:0;+ :* Experiment GP @?LQK G=N VSUW

Slide 26

Slide 26 text

Monorith THINK BIG: CHAOS TESTING A MONOLITH https://speakerdeck.com/chaosconf/think-big-chaos-testing-a-monolith

Slide 27

Slide 27 text

Monorith THINK BIG: CHAOS TESTING A MONOLITH https://speakerdeck.com/chaosconf/think-big-chaos-testing-a-monolith

Slide 28

Slide 28 text

Server-based or Serverless

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Server-based or Server-less Server-based • )!&*($938 Failure Injection /; • shutdown now • tc qdisc add dev eth0 root netem delay 2000ms • "'+%)*($ Injection 26/; Serverless • )!&*($938 4. 5>,? • )!&($931 =0< + +( #' • 078"'+%)($ Injection -:0<

Slide 36

Slide 36 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. | | | | | | | | | | | | | | | 56 Microservices & % 7-2+ • 2014.101 FIT: Failure Injection Testing (Netflix TechBlog) • Request Context (Failure Context, FIT Context) #(/,)0 '$("'!(' 3 !' Failure *4 https://medium.com/netflix-techblog/fit-failure-injection-testing-35d8e2a9bb2

Slide 37

Slide 37 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. | | | | | | | | | | | | | | | 9: Microservices ("'<06. • 20142105 FIT: Failure Injection Testing (Netflix TechBlog) • Request Context (Failure Context, FIT Context) %+3/,4 *&+$*#+!* 7 #* Failure -8 https://medium.com/netflix-techblog/fit-failure-injection-testing-35d8e2a9bb2 a little too wild 1)' ;

Slide 38

Slide 38 text

Slide 39

Slide 39 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S FIT NBV_9 Chaos Monkey, Simian Army ; EC2 Instance API terminate tc NW ^E \>"5,264"/ WJVI Blast Radius: PU?G [d • :TY@ QB9aB • FcY@ D FIT ; UA #2"!5+ XL bCO=*(+ ,2$07&]< DB !#)( ^E !-3%7'1564"/ HA 8K !-3%7'1 564"/HA Z` • R."5+ • MaBY@

Slide 40

Slide 40 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. OSS Tools: Serverless aws-lambda-chaos-injection Chaos Injection library for AWS Lambda Python I?< α .6$04J 2/$*2!6,6 Adrian Hornsby AWS Principal Evangelist Adrian Gremlin Ana &(#04 Breaking Containers: Chaos Engineering for Modern Applications on AWS (CON310) ;@ %1 + BF https://medium.com/@adhorn/failure-injection-gain-confidence-in-your-serverless-application-ce6c0060f586 7D latency injection )"36' Lambda function -4+19 >C555 ↑3%/4%8G= ↓ Duration >H Delay :AE

Slide 41

Slide 41 text

Cloud or on-prem

Slide 42

Slide 42 text

Cloud or on-prem • )$'&=5 Chaos Experiment ,-D9 • Netflix )$' AWS @C G Chaos Monkey F?*/>7 #(EA.F Chaos Engineering 6 +")%)! H 3<;01B2:84 I

Slide 43

Slide 43 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. | | | | | | | | | | | | | | | Netflix AWS -. "+( • 2010$12'14& Four Reasons We Choose Amazon’s Cloud as Our Computing Platform (Netflix TechBlog) ,/ Amazon 0%4)* • 2010$12'16& 5 Lessons We’ve Learned Using AWS (Netflix TechBlog) AWS !#5 https://medium.com/netflix-techblog/four-reasons-we-choose-amazons-cloud-as-our-computing-platform-4aceb692afec https://medium.com/netflix-techblog/5-lessons-weve-learned-using-aws-1f2a28588e4c

Slide 44

Slide 44 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. | | | | | | | | | | | | | | | Netflix AWS FK 04D@ • 2010812>14< Four Reasons We Choose Amazon’s Cloud as Our Computing Platform (Netflix TechBlog) EM.'+/#.),"$(/* Amazon ,%N:4AB • 2010812>16< 5 Lessons We’ve Learned Using AWS (Netflix TechBlog) AWS 2 6 5-" . https://medium.com/netflix-techblog/four-reasons-we-choose-amazons-cloud-as-our-computing-platform-4aceb692afec https://medium.com/netflix-techblog/5-lessons-weve-learned-using-aws-1f2a28588e4c P7 (failure) O=J;? HICP7L (fail) Chaos Monkey 5 /& ,.!*93G1

Slide 45

Slide 45 text

AWS Chaos Experiment API Chaos Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf

Slide 46

Slide 46 text

Production or Non-production

Slide 47

Slide 47 text

https://principlesofchaos.org (en), https://principlesofchaos.org/?lang=JAcontent (ja) $!#$ 30 ":8') 9%61* 4 .2,5 7& "/- /+ (;

Slide 48

Slide 48 text

https://principlesofchaos.org (en), https://principlesofchaos.org/?lang=JAcontent (ja) $!#$ 1. " 6%4/( 2 ,0*3 5& "-+ -) '7

Slide 49

Slide 49 text

GameDay game-day: 8F1= • !(")G !(")&+! !(")G >%+59 I • ?B2@A3C6-:59ED • AWS GameDay: <.+%0AWS ,;/ 4HA3!(")&+! *$ '*87#*! https://eow.alc.co.jp/search?q=game-day https://aws.amazon.com/gameday/

Slide 50

Slide 50 text

AWS GameDay

Slide 51

Slide 51 text

Wrap-up

Slide 52

Slide 52 text

! • Chaos Engineering UH#]PQXZ% S>"e 3-/25+.*-W_#8;"aB • cIK bc& j gA`h C<&:R''#%#(#^Y ,-.4TJ?[L$D M)9ENF • fG& 1706 iOdP Dev V@# GameDay C<#=%\ $!

Slide 53

Slide 53 text

Thank you!