What is suitable for Chaos Engineering? ChaosConf2019 recap @AWS LOFT TOKYO Fumihiko Hata | Solutions Architect, AWS Japan | 2019.11.11

Fumihiko Hata Role: Solutions Architect, AWS Japan Customer: Game companies Favorite: Elm, Clojure, Tutorial D Who am I

Agenda Chaos Engineering (* &) &) ChaosConf 2019.9'-% @SanFrancisco/ !"+$#, What is suitable for Chaos Engineering?

Agenda 1. Industries 2. Microservices or Monorith 3. Server-based or Server-less 4. Cloud or on-prem 5. Production or Non-production env What is suitable for Chaos Engineering?

Industries

Chaos Engineering

Chaos Engineering Chaos Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf

Chaos Engineering Chaos Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf

Chaos Engineering Chaos Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf

ChaosConf2019 EC https://speakerdeck.com/chaosconf/humans-are-fallible https://speakerdeck.com/chaosconf/embracing-chaos https://speakerdeck.com/chaosconf/a-roadmap-towards-chaos-engineering

EC 7 • .40 , • $/( ,#12)0'6 • +5 !"3%-*&

Uber EMBRACING CHAOS! Paul Osman, Under Armour Ana Medina, Gremlin • uDestroy: Uber Chaos Experiment • Uber Gremlin Ana https://speakerdeck.com/chaosconf/ embracing-chaos

Uber https://speakerdeck.com/chaosconf/ embracing-chaos

9513 • ChaosConf &+ Gremlin 4$8 Chaos Engineering .( %1 ,=-9513 • :6952; ( FinTech * 0 95!" "( • #'7)>

:624? National Australia Bank • ChaosConf &+ Gremlin 5$9 Chaos Engineering .' %2=0,>-:624 • ;7:63< ' FinTech ) 1 :6 "#!#' • /*8( @ NAB deploys Chaos Monkey to kill servers 24/7 - Software – Cloud - iTnews 2014 https://www.itnews.com.au/news/nab-deploys-chaos-monkey-to-kill-servers-24-7-382285

9513> Capital One • ChaosConf %* Gremlin 4#8 Chaos Engineering -& $1 +=,9513 • :6952; & FinTech ( 0 95!" "& • .)7'? https://twitter.com/CapitalOneTech/status/ 1194995848062623744 20191115

NI@CQ TLPT (Threat-Led Penetration Test) H5$,%" *,(+ 2018710>NI8NI.M #,')9/2 0F =O

NHBES TLPT (Threat-Led Penetration Test) G:(.#)'%,.!+-$#% M86G:(.#)'%,. !+-$#%QTLPTRO 74A 2IL TLPT Chaos Experiment ?81 =>@CD 0 PF 5 9 /;JKNH BE -"&3< ,*.% ! TLPT # " https://www.fsa.go.jp/common/about/research/20180516.html

^XIRc TLPT (Threat-Led Penetration Test) “W;(/")&%-/!,.$"% O<0RB Y]^XK_ '/ ED>BC8[1 >BV6b ML$"%” 2.2.1. $"%JZ “TLPT O<0R4ED>B V6bML$"% ?$"%

^XIRc TLPT (Threat-Led Penetration Test) “W;(/")&%-/!,.$"% O<0RB Y]^XK_ '/ ED>BC8[1 >BV6b ML$"%” 2.2.1. $"%JZ “TLPT O<0R4ED>B V6bML$"% ?$"%

Microservices or Monorith

Microservices

Microservices )/ 1'" • FH N@< 1'"T) / 1'"U(.1!-0B> 1%$,4IK; !"%*SG(/1$ • 6 1'" API R5 • 1'"

Microservices • 7T( .U"1&$Q Failure BS9R Chaos Experiment @HPF"1 &$;5A%1) J><? • Netflix Amazon ( ."1&$ DKI= • 2G Netflix "1&$Q ',!1#+/0-* Failure Context 4E3L O"1&$Q 8 6M:CN

Monorith THINK BIG: CHAOS TESTING A MONOLITH https://speakerdeck.com/chaosconf/think-big-chaos-testing-a-monolith • 3;8JC$;-&/6(*. ;2 Mailchimp ZD Mailchimp RMI SRE Caroline '(%5: • 2000 • 1!9$;-&%&)2 OB[FE A#:0;+ :* Experiment GP @?LQK G=N VSUW

Monorith THINK BIG: CHAOS TESTING A MONOLITH https://speakerdeck.com/chaosconf/think-big-chaos-testing-a-monolith

Monorith THINK BIG: CHAOS TESTING A MONOLITH https://speakerdeck.com/chaosconf/think-big-chaos-testing-a-monolith

Server-based or Serverless

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Server-based or Server-less Server-based • )!&*($938 Failure Injection /; • shutdown now • tc qdisc add dev eth0 root netem delay 2000ms • "'+%)*($ Injection 26/; Serverless • )!&*($938 4. 5>,? • )!&($931 =0< + +( #' • 078"'+%)($ Injection -:0<

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. | | | | | | | | | | | | | | | 56 Microservices & % 7-2+ • 2014.101 FIT: Failure Injection Testing (Netflix TechBlog) • Request Context (Failure Context, FIT Context) #(/,)0 '$("'!(' 3 !' Failure *4 https://medium.com/netflix-techblog/fit-failure-injection-testing-35d8e2a9bb2

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. | | | | | | | | | | | | | | | 9: Microservices ("'<06. • 20142105 FIT: Failure Injection Testing (Netflix TechBlog) • Request Context (Failure Context, FIT Context) %+3/,4 *&+$*#+!* 7 #* Failure -8 https://medium.com/netflix-techblog/fit-failure-injection-testing-35d8e2a9bb2 a little too wild 1)' ;

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. / FIT ,%28 Chaos Monkey, Simian Army ! EC2 Instance API terminate tc NW 7' 6" 3+2* Blast Radius: -1#) 5; • 04$.%9% • (:4$&

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S FIT NBV_9 Chaos Monkey, Simian Army ; EC2 Instance API terminate tc NW ^E \>"5,264"/ WJVI Blast Radius: PU?G [d • :TY@ QB9aB • FcY@ D FIT ; UA #2"!5+ XL bCO=*(+ ,2$07&]< DB !#)( ^E !-3%7'1564"/ HA 8K !-3%7'1 564"/HA Z` • R."5+ • MaBY@

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. OSS Tools: Serverless aws-lambda-chaos-injection Chaos Injection library for AWS Lambda Python I?< α .6$04J 2/$*2!6,6 Adrian Hornsby AWS Principal Evangelist Adrian Gremlin Ana &(#04 Breaking Containers: Chaos Engineering for Modern Applications on AWS (CON310) ;@ %1 + BF https://medium.com/@adhorn/failure-injection-gain-confidence-in-your-serverless-application-ce6c0060f586 7D latency injection )"36' Lambda function -4+19 >C555 ↑3%/4%8G= ↓ Duration >H Delay :AE

Cloud or on-prem

Cloud or on-prem • )$'&=5 Chaos Experiment ,-D9 • Netflix )$' AWS @C G Chaos Monkey F?*/>7 #(EA.F Chaos Engineering 6 +")%)! H 3<;01B2:84 I

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. | | | | | | | | | | | | | | | Netflix AWS -. "+( • 2010$12'14& Four Reasons We Choose Amazon’s Cloud as Our Computing Platform (Netflix TechBlog) ,/ Amazon 0%4)* • 2010$12'16& 5 Lessons We’ve Learned Using AWS (Netflix TechBlog) AWS !#5 https://medium.com/netflix-techblog/four-reasons-we-choose-amazons-cloud-as-our-computing-platform-4aceb692afec https://medium.com/netflix-techblog/5-lessons-weve-learned-using-aws-1f2a28588e4c

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. | | | | | | | | | | | | | | | Netflix AWS FK 04D@ • 2010812>14< Four Reasons We Choose Amazon’s Cloud as Our Computing Platform (Netflix TechBlog) EM.'+/#.),"$(/* Amazon ,%N:4AB • 2010812>16< 5 Lessons We’ve Learned Using AWS (Netflix TechBlog) AWS 2 6 5-" . https://medium.com/netflix-techblog/four-reasons-we-choose-amazons-cloud-as-our-computing-platform-4aceb692afec https://medium.com/netflix-techblog/5-lessons-weve-learned-using-aws-1f2a28588e4c P7 (failure) O=J;? HICP7L (fail) Chaos Monkey 5 /& ,.!*93G1

AWS Chaos Experiment API Chaos Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf

Production or Non-production

https://principlesofchaos.org (en), https://principlesofchaos.org/?lang=JAcontent (ja) $!#$ 30 ":8') 9%61* 4 .2,5 7& "/- /+ (;

https://principlesofchaos.org (en), https://principlesofchaos.org/?lang=JAcontent (ja) $!#$ 1. " 6%4/( 2 ,0*3 5& "-+ -) '7

GameDay game-day: 8F1= • !(")G !(")&+! !(")G >%+59 I • ?B2@A3C6-:59ED • AWS GameDay: <.+%0AWS ,;/ 4HA3!(")&+! *$ '*87#*! https://eow.alc.co.jp/search?q=game-day https://aws.amazon.com/gameday/

AWS GameDay

Wrap-up

! • Chaos Engineering UH#]PQXZ% S>"e 3-/25+.*-W_#8;"aB • cIK bc& j gA`h C<&:R''#%#(#^Y ,-.4TJ?[L$D M)9ENF • fG& 1706 iOdP Dev V@# GameDay C<#=%\ $!

Thank you!