Slide 1

Slide 1 text

What is suitable for Chaos Engineering? ChaosConf2019 recap @AWS LOFT TOKYO Fumihiko Hata | Solutions Architect, AWS Japan | 2019.11.11

Slide 2

Slide 2 text

Fumihiko Hata Role: Solutions Architect, AWS Japan Customer: Game companies Favorite: Elm, Clojure, Tutorial D Who am I

Slide 3

Slide 3 text

Agenda Chaos Engineering  (* &) &)   ChaosConf 2019.9'-% @SanFrancisco/ !"+$#,  What is suitable for Chaos Engineering?

Slide 4

Slide 4 text

Agenda 1. Industries 2. Microservices or Monorith 3. Server-based or Server-less 4. Cloud or on-prem 5. Production or Non-production env What is suitable for Chaos Engineering?

Slide 5

Slide 5 text

Industries

Slide 6

Slide 6 text

  Chaos Engineering     

Slide 7

Slide 7 text

  Chaos Engineering      Chaos Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf

Slide 8

Slide 8 text

  Chaos Engineering      Chaos Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf

Slide 9

Slide 9 text

  Chaos Engineering      Chaos Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf

Slide 10

Slide 10 text

ChaosConf2019   EC   https://speakerdeck.com/chaosconf/humans-are-fallible https://speakerdeck.com/chaosconf/embracing-chaos https://speakerdeck.com/chaosconf/a-roadmap-towards-chaos-engineering

Slide 11

Slide 11 text

 EC 7 • .40 , • $/( , #1 2)0 '6 • +5  !" 3%-*&

Slide 12

Slide 12 text

Uber EMBRACING CHAOS! Paul Osman, Under Armour Ana Medina, Gremlin • uDestroy: Uber  Chaos Experiment •  Uber  Gremlin  Ana  https://speakerdeck.com/chaosconf/ embracing-chaos

Slide 13

Slide 13 text

Uber https://speakerdeck.com/chaosconf/ embracing-chaos

Slide 14

Slide 14 text

9513 • ChaosConf &+ Gremlin 4$8  Chaos Engineering .(  %1

Slide 15

Slide 15 text

:624? National Australia Bank • ChaosConf &+ Gremlin 5$9  Chaos Engineering .'  %2=0,>-:624 •  ;7:63< ' FinTech ) 1 :6 "#!#' • /*8( @ NAB deploys Chaos Monkey to kill servers 24/7 - Software – Cloud - iTnews 2014  https://www.itnews.com.au/news/nab-deploys-chaos-monkey-to-kill-servers-24-7-382285

Slide 16

Slide 16 text

9513> Capital One • ChaosConf %* Gremlin 4#8  Chaos Engineering -&  $1

Slide 17

Slide 17 text

NI@CQ TLPT (Threat-Led Penetration Test) H5$,%" *,(+ 2018710>NI8NI.M #,')9/2 0F =O

Slide 18

Slide 18 text

NHBES TLPT (Threat-Led Penetration Test) G:(.#)'%,.!+-$#% M86G:(.#)'%,. !+-$#%QTLPTRO 74A 2IL TLPT  Chaos Experiment ?81 =>@CD  0 PF 5 9 /;JK NH BE -"&3< ,*.% !  TLPT  # "   https://www.fsa.go.jp/common/about/research/20180516.html

Slide 19

Slide 19 text

^XIRc TLPT (Threat-Led Penetration Test) “W;(/")&%-/!,.$"% O<0RB Y]^XK_ '/ ED>BC8[1 >BV6b  ML$"%” 2.2.1. $"%JZ  “TLPT O<0R4ED>B V6b ML$"% ?$"%

Slide 20

Slide 20 text

^XIRc TLPT (Threat-Led Penetration Test) “W;(/")&%-/!,.$"% O<0RB Y]^XK_ '/ ED>BC8[1 >BV6b  ML$"%” 2.2.1. $"%JZ  “TLPT O<0R4ED>B V6b ML$"% ?$"%

Slide 21

Slide 21 text

Microservices or Monorith

Slide 22

Slide 22 text

Microservices

Slide 23

Slide 23 text

Microservices )/ 1'" • FH N@< 1'"T) / 1'"U (.1!-0B> 1%$,4IK; !"%*SG(/1$ • 6 1'" API R 5 •   1'"

Slide 24

Slide 24 text

Microservices • 7T( .U"1&$Q Failure BS9R  Chaos Experiment @HPF"1 &$;5A%1) J><? • Netflix  Amazon ( ."1&$ DK I= • 2G Netflix "1&$Q  ',!1#+/0-*  Failure Context 4E3L O"1&$Q 8 6M:CN 

Slide 25

Slide 25 text

Monorith THINK BIG: CHAOS TESTING A MONOLITH https://speakerdeck.com/chaosconf/think-big-chaos-testing-a-monolith • 3;8JC$;-&/6(*. ;2 Mailchimp ZD  Mailchimp RMI  SRE Caroline '(%5: • 2000  • 1!9$;-&%&)2 OB[FE  A#:0;+ :* Experiment GP   @?LQK G=N VSUW

Slide 26

Slide 26 text

Monorith THINK BIG: CHAOS TESTING A MONOLITH https://speakerdeck.com/chaosconf/think-big-chaos-testing-a-monolith

Slide 27

Slide 27 text

Monorith THINK BIG: CHAOS TESTING A MONOLITH https://speakerdeck.com/chaosconf/think-big-chaos-testing-a-monolith

Slide 28

Slide 28 text

Server-based or Serverless

Slide 29

Slide 29 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.  

Slide 30

Slide 30 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.  

Slide 31

Slide 31 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.  

Slide 32

Slide 32 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.  

Slide 33

Slide 33 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.  

Slide 34

Slide 34 text

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Slide 35

Slide 35 text

Server-based or Server-less Server-based • )!&*($938 Failure  Injection /; • shutdown now • tc qdisc add dev eth0 root netem delay 2000ms •  "'+%)*($  Injection 26/; Serverless • )!&*($938 4. 5>,? •  )!&($931 =0<  + +( #' • 078"'+%)($  Injection -:0<

Slide 36

Slide 36 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. | | | | | | | | | | | | | | | 56 Microservices & % 7-2+ • 2014.101 FIT: Failure Injection Testing (Netflix TechBlog) • Request Context (Failure Context, FIT Context) #(/,)0  '$("'!(' 3 !' Failure *4 https://medium.com/netflix-techblog/fit-failure-injection-testing-35d8e2a9bb2

Slide 37

Slide 37 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. | | | | | | | | | | | | | | | 9: Microservices ("' <06. • 20142105 FIT: Failure Injection Testing (Netflix TechBlog) • Request Context (Failure Context, FIT Context) %+3/,4   *&+$*#+!* 7 #* Failure -8  https://medium.com/netflix-techblog/fit-failure-injection-testing-35d8e2a9bb2 a little too wild 1)' ;

Slide 38

Slide 38 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. /  FIT ,%28 Chaos Monkey, Simian Army ! EC2 Instance  API terminate  tc NW 7' 6"  3+2* Blast Radius: -1#) 5; • 04$.%9%  • (:4$& 

Slide 39

Slide 39 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S FIT NBV_9 Chaos Monkey, Simian Army ; EC2 Instance  API  terminate  tc  NW ^E \> "5,264"/ WJVI Blast Radius: PU?G [d • :TY@ QB9aB  • FcY@ D   FIT ; UA #2"!5+ XL bCO= *(+ ,2$07&]<  DB !#)( ^E  !-3%7'1564"/ HA 8K !-3%7'1 564"/HA Z`  • R."5+ • MaB Y@

Slide 40

Slide 40 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. OSS Tools: Serverless aws-lambda-chaos-injection Chaos Injection library for AWS Lambda Python I?< α .6$04J 2/$*2!6,6 Adrian Hornsby  AWS  Principal Evangelist Adrian  Gremlin  Ana &(#04  Breaking Containers: Chaos Engineering for Modern Applications on AWS (CON310)  ;@  %1 + BF   https://medium.com/@adhorn/failure-injection-gain-confidence-in-your-serverless-application-ce6c0060f586 7D latency injection )"36' Lambda function -4+19 >C 555 ↑3%/4%8G= ↓ Duration >H Delay : AE 

Slide 41

Slide 41 text

Cloud or on-prem

Slide 42

Slide 42 text

Cloud or on-prem • )$'&=5  Chaos Experiment ,-D9   •  Netflix )$' AWS @C G Chaos Monkey F?*/>7  #( EA.F  Chaos Engineering 6 +")%)! H  3<;01B2:84  I

Slide 43

Slide 43 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. | | | | | | | | | | | | | | | Netflix  AWS -. "+( • 2010$12'14& Four Reasons We Choose Amazon’s Cloud as Our Computing Platform (Netflix TechBlog) ,/  Amazon   0%4)* • 2010$12'16& 5 Lessons We’ve Learned Using AWS (Netflix TechBlog) AWS !# 5 https://medium.com/netflix-techblog/four-reasons-we-choose-amazons-cloud-as-our-computing-platform-4aceb692afec https://medium.com/netflix-techblog/5-lessons-weve-learned-using-aws-1f2a28588e4c

Slide 44

Slide 44 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. | | | | | | | | | | | | | | | Netflix  AWS FK 04D@ • 2010812>14< Four Reasons We Choose Amazon’s Cloud as Our Computing Platform (Netflix TechBlog) EM.'+/#.),"$(/*  Amazon ,%N:4 AB • 2010812>16< 5 Lessons We’ve Learned Using AWS (Netflix TechBlog) AWS 2 6 5 -" . https://medium.com/netflix-techblog/four-reasons-we-choose-amazons-cloud-as-our-computing-platform-4aceb692afec https://medium.com/netflix-techblog/5-lessons-weve-learned-using-aws-1f2a28588e4c P7 (failure) O=J;? HICP7L (fail)   Chaos Monkey 5  /& ,.!*93G1

Slide 45

Slide 45 text

AWS  Chaos Experiment  API Chaos Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf

Slide 46

Slide 46 text

Production or Non-production

Slide 47

Slide 47 text

https://principlesofchaos.org (en), https://principlesofchaos.org/?lang=JAcontent (ja) $!#$ 30  ":8') 9%61* 4  .2,5   7& "/-  /+ (; 

Slide 48

Slide 48 text

https://principlesofchaos.org (en), https://principlesofchaos.org/?lang=JAcontent (ja) $!#$ 1.  " 6%4/( 2  ,0*3   5& "-+  -) '7 

Slide 49

Slide 49 text

GameDay game-day: 8F1= • !(")G  !(")&+!    !(")G >%+59  I • ?B2@A3C6-:59ED • AWS GameDay: <.+%0AWS ,;/  4HA3!(")&+!  *$ '*87#*! https://eow.alc.co.jp/search?q=game-day https://aws.amazon.com/gameday/

Slide 50

Slide 50 text

AWS GameDay

Slide 51

Slide 51 text

Wrap-up

Slide 52

Slide 52 text

! • Chaos Engineering UH#]PQXZ% S>"e 3-/25+.*-W_#8;"aB • cIK bc& j gA`h C<&:R''#%# (#^Y ,-.4TJ?[L$D M)9 ENF • fG& 1706 iOdP Dev V@# GameDay C<#=%\ $!

Slide 53

Slide 53 text

Thank you!