What is suitable for Chaos Engineering? ~ChaosConf 2019 recap~

Chaos Engineering → Chaos Monkey → 「本番環境でサーバを落とす」 → うちでは無理そう、、
というよくある誤解を解きたいと思って喋ったスライドです。 ChaosConf 2019 におけるセッションのスライドの紹介も交えながら説明しています。

AWS DevDay Tokyo 2019 の『Chaos Engineering ～入門と実例～』
https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf

から、スライドをいくつか引用しています。こちらは Netflix 社の Chaos Engineering の取り組みを時系列でまとめるかたちで Chaos Engineering とは何かを説明しているので、 Chaos Engineering 自体に馴染みのない方はまずはこちらの資料をどうぞ。

Transcript

1. What is suitable for Chaos Engineering? ChaosConf2019 recap @AWS LOFT TOKYO Fumihiko Hata | Solutions Architect, AWS Japan | 2019.11.11
2. Fumihiko Hata Role: Solutions Architect, AWS Japan Customer: Game companies Favorite: Elm, Clojure, Tutorial D Who am I
3. Agenda Chaos Engineering  (* &) 
   &)   ChaosConf 2019.9'-% @SanFrancisco/ !"+$#,  What is suitable for Chaos Engineering?
4. Agenda 1. Industries 2. Microservices or Monorith 3. Server-based or Server-less 4. Cloud or on-prem 5. Production or Non-production env What is suitable for Chaos Engineering?
5. Industries
6.   Chaos Engineering   
    
7.   Chaos Engineering   
    
   Chaos Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf
8.   Chaos Engineering   
    
   Chaos Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf
9.   Chaos Engineering   
    
   Chaos Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf
10. ChaosConf2019  
     EC   https://speakerdeck.com/chaosconf/humans-are-fallible https://speakerdeck.com/chaosconf/embracing-chaos https://speakerdeck.com/chaosconf/a-roadmap-towards-chaos-engineering
11.  EC 7 • .40 , • $/( , #1 2)0 '6 • +5
    !" 3%-*&
12. Uber
    EMBRACING CHAOS! Paul Osman, Under Armour Ana Medina, Gremlin • uDestroy: Uber  Chaos Experiment •  Uber  Gremlin  Ana  https://speakerdeck.com/chaosconf/ embracing-chaos
13. Uber https://speakerdeck.com/chaosconf/ embracing-chaos
14. 9513 • ChaosConf &+ Gremlin 4$8  Chaos Engineering .(  %1 </,=-
    9513 •
    :6952; (
     FinTech *
    0 95!" "( • #'7) >
15. :624? National Australia Bank • ChaosConf &+ Gremlin 5$9  Chaos Engineering .'  %2=0,>-:624 •  ;7:63< ' FinTech ) 1 :6 "#!#' • /*8( @
    NAB deploys Chaos Monkey to kill servers 24/7 - Software – Cloud - iTnews
    
    2014  https://www.itnews.com.au/news/nab-deploys-chaos-monkey-to-kill-servers-24-7-382285
16. 9513> Capital One • ChaosConf %* Gremlin 4#8  Chaos Engineering -&  $1 </+=,
    9513 •
    :6952; &
     FinTech (
    0 95!" "& • .)7' ? https://twitter.com/CapitalOneTech/status/ 1194995848062623744 201911
    15
17. NI@CQ TLPT (Threat-Led Penetration Test) H5$,%" *,(+ 2018710>NI8
    NI.M #,')9/2 0F =O<K
    4:NIAP6  TLPT ;L  -J *!,&
    *!,&BG
    3?D1E ! 
     
    TLPT  # "   https://www.fsa.go.jp/common/about/research/20180516.html
18. NHBES TLPT (Threat-Led Penetration Test) G:(.#)'%,.!+-$#% M86G:(.#)'%,. !+-$#%QTLPTRO 74A 2IL
    TLPT  Chaos Experiment
    ?81 =>@CD  0 PF 5 9 /;JK 
    NH BE -"&3< ,*.% ! 
     
    TLPT  # "   https://www.fsa.go.jp/common/about/research/20180516.html
19. ^XIRc TLPT (Threat-Led Penetration Test) “W;(/")&%-/!,.$"%
    O<0R<a\  '/ED $"#/7T !*+-/!,.
    `@3^XK_`@
    HU
    >B Y]^XK_ '/ ED>BC8[1
    >BV6b  ML$"%” 2.2.1. $"%JZ  “TLPT 
    O<0R<a\ E D7T!*+-/!,.
    '/ ED2= >4ED>B V6b ML$"% ?
    $"%<K Q
    9 5 GSP:<F ” 2.2.3. TLPT NA  20181
    31  
    
     TLPT   https://www.fsa.go.jp/common/about/research/20180516/TLPT.pdf
20. ^XIRc TLPT (Threat-Led Penetration Test) “W;(/")&%-/!,.$"%
    O<0R<a\  '/ED $"#/7T !*+-/!,.
    `@3^XK_`@
    HU
    >B Y]^XK_ '/ ED>BC8[1
    >BV6b  ML$"%” 2.2.1. $"%JZ  “TLPT 
    O<0R<a\ E D7T!*+-/!,.
    '/ ED2= >4ED>B V6b ML$"% ?
    $"%<K Q
    9 5 GSP:<F ” 2.2.3. TLPT NA  20181 31  
    
     TLPT   https://www.fsa.go.jp/common/about/research/20180516/TLPT.pdf Chaos Engineering    Gameday
      
21. Microservices or Monorith
22. Microservices
23. Microservices )/ 1'" • FH N@< 1'"T) / 1'"U (.1!-0B> 
    1%$,4IK; !"%*SG(/1$ • 6 1'" API R 5 • 
     1'"<OC M= 9JFH $1*?A • ;E
    N@ 2 DLV  1'"T(/#"U3 :J7 (.1!-0QP+&." 8
24. Microservices • 7T( .U"1&$Q Failure BS9R 
     Chaos Experiment @HPF"1 &$;5A%1) J><? • Netflix  Amazon ( ."1&$ DK I= • 2G Netflix "1&$Q 
    ',!1#+/0-*  Failure Context 4E3L O
    "1&$Q 8 6M:CN 
25. Monorith
    THINK BIG: CHAOS TESTING A MONOLITH https://speakerdeck.com/chaosconf/think-big-chaos-testing-a-monolith • 3;8JC$;-&/6(*. ;2 Mailchimp ZD  Mailchimp RMI  SRE Caroline '(%5: • 2000<TX4,7& PHP /7 ";%5:
     Chaos Engineering GY H>  • 1!9$;-&
    %&)2 OB[FE
    A#:0;+ :* Experiment GP  
    @?LQK G=N VSUW
26. Monorith
    THINK BIG: CHAOS TESTING A MONOLITH https://speakerdeck.com/chaosconf/think-big-chaos-testing-a-monolith
27. Monorith
    THINK BIG: CHAOS TESTING A MONOLITH https://speakerdeck.com/chaosconf/think-big-chaos-testing-a-monolith
28. Server-based or Serverless
29. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.  
    
30. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.  
    
31. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.  
    
32. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.  
    
33. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.  
    
34. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
35. Server-based or Server-less Server-based • )!&*($938 Failure  Injection /; • shutdown now • tc qdisc add dev eth0 root netem delay 2000ms •  "'+%)*($  Injection 26/; Serverless • )!&*($938 4.
    5>,? • 
    )!&($931 =0<  + +( #' • 078"'+%)($  Injection -:0<
36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. | | | | | | | | | | | | | | | 56 Microservices & % 7-2+ • 2014.101 FIT: Failure Injection Testing (Netflix TechBlog) • Request Context (Failure Context, FIT Context) #(/,)0
    '$("'!(' 3 !' Failure *4 https://medium.com/netflix-techblog/fit-failure-injection-testing-35d8e2a9bb2
37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. | | | | | | | | | | | | | | | 9: Microservices ("' <06. • 20142105 FIT: Failure Injection Testing (Netflix TechBlog) • Request Context (Failure Context, FIT Context) %+3/,4 
    *&+$*#+!* 7 #* Failure -8  https://medium.com/netflix-techblog/fit-failure-injection-testing-35d8e2a9bb2 a little too wild 1)' ;
38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. /  FIT ,%28 Chaos Monkey, Simian Army !
    EC2 Instance  API terminate 
    tc NW 7' 6" 
     3+2* Blast Radius: -1#) 5; • 04$.%9%  • (:4$& 
39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S FIT NBV_9 Chaos Monkey, Simian Army ;
    EC2 Instance  API  terminate 
    tc  NW ^E \> 
    "5,264"/ WJVI Blast Radius: PU?G [d • :TY@ QB9aB  • FcY@ D   FIT ;
     UA #2"!5+ XL bCO= 
    *(+ ,2$07&]<  DB !#)( ^E 
    !-3%7'1564"/ HA 8K
    !-3%7'1 564"/HA Z`  • R."5+ • MaB Y@
40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. OSS Tools: Serverless aws-lambda-chaos-injection Chaos Injection library for AWS Lambda Python I?< α .6$04J 2/$*2!6,6 Adrian Hornsby  AWS  Principal Evangelist Adrian  Gremlin  Ana &(#04  Breaking Containers: Chaos Engineering for Modern Applications on AWS (CON310)  ;@  %1 + BF   https://medium.com/@adhorn/failure-injection-gain-confidence-in-your-serverless-application-ce6c0060f586 7D latency injection )"36' Lambda function -4+19 >C 555 ↑3%/4%8G=
    ↓ Duration >H Delay : AE 
41. Cloud or on-prem
42. Cloud or on-prem • )$'&=5  Chaos Experiment ,-D9   •  Netflix )$' AWS @C G Chaos Monkey F?*/>7  #( EA.F  Chaos Engineering 6 +")%)! H
     3<;01B2:84  I
43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. | | | | | | | | | | | | | | | Netflix  AWS -. "+( • 2010$12'14& Four Reasons We Choose Amazon’s Cloud as Our Computing Platform (Netflix TechBlog) ,/ 
     Amazon   0%
    4)* • 2010$12'16& 5 Lessons We’ve Learned Using AWS (Netflix TechBlog) AWS !# 5 https://medium.com/netflix-techblog/four-reasons-we-choose-amazons-cloud-as-our-computing-platform-4aceb692afec https://medium.com/netflix-techblog/5-lessons-weve-learned-using-aws-1f2a28588e4c
44. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. | | | | | | | | | | | | | | | Netflix  AWS FK 04D@ • 2010812>14< Four Reasons We Choose Amazon’s Cloud as Our Computing Platform (Netflix TechBlog) EM.'+/#.),"$(/*  Amazon ,%N:4 AB • 2010812>16< 5 Lessons We’ve Learned Using AWS (Netflix TechBlog) AWS 2 6 5 -" . https://medium.com/netflix-techblog/four-reasons-we-choose-amazons-cloud-as-our-computing-platform-4aceb692afec https://medium.com/netflix-techblog/5-lessons-weve-learned-using-aws-1f2a28588e4c P7 (failure) O=J
    ;? HICP7L (fail)   Chaos Monkey 5  /& ,.!*93G1
45. AWS  Chaos Experiment 
     API
    Chaos Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf
46. Production or Non-production
47. https://principlesofchaos.org (en), https://principlesofchaos.org/?lang=JAcontent (ja) $!#$ 30  ":8') 9%61* 4  .2,5   7& "/-  /+ (;
    
48. https://principlesofchaos.org (en), https://principlesofchaos.org/?lang=JAcontent (ja) $!#$ 1.  "
     6%4/( 2  ,0*3   5& "-+  -) '7
    
49. GameDay game-day: 8F1= • !(")G  !(")&+!  
     !(")G >%+59  I • ?B2@A3
    C6-:59ED • AWS GameDay: <.+%0
    AWS ,;/  4HA3!(")&+!  *$ '*87#*! https://eow.alc.co.jp/search?q=game-day https://aws.amazon.com/gameday/
50. AWS GameDay
51. Wrap-up
52. ! • Chaos Engineering 
    UH#]PQXZ%
    S>"e 3-/25+.*-W_#8;"aB • cIK bc& j gA`h C<&:R
    ''#%# (#^Y ,-.4TJ?[L$D M)9 ENF • fG& 1706 iOdP
     Dev V@# GameDay C<#=%\ $!
53. Thank you!