ある SRE チームの挑戦 運用 6 年目の大規模ゲームを AWS 移設後に安定運用するための技術と今後の展望

ある SRE チームの挑戦 運用 6 年目の大規模ゲームを AWS 移設後に安定運用するための技術と今後の展望

AWS Summit 2019 の発表資料です。

Ec2fcdc4ea7905b289967a2c4c43e154?s=128

CyberAgent SGE Engineer

June 14, 2019
Tweet

Transcript

  1. E 1 0 2 9 9A 1669 91A 9 A

    / . , RL T LSah R 0 c U IMLbd WULgf i . 1 1 1 1 , . 1 e l . 1 1 , . 1
  2. 9 ,1. 1 /1 / 9 22 1 1 1

    10 I a , W U SA , T M
  3. 9 ,1. 1 /1 / 9 22 1 1 1

    10 I a , W U SA , T M
  4. 3E 2 4 5 A 5 A 3 3 A

    9 A A 6 1 0 r l h f . h 0 3 3 39 n s Wd b u h f m M . h M hgc i b e R ah z n o L 2 e S 3E 3 n s Wd b u h f m M 2 R I zI t T t z vR ah M U L 2 e S . l h f . h 0 5 , 36
  5. 1 0 2 A9 9 1669 91 9 8 A

    4 . fvglsT , v fvglsTd v g TWR S S s Ef iT I r T ETbt hz T e n b e o c T ra i o c r ms o c T d vT U R M T ra i S RM U
  6. A . 9 2 2 02 90 . . 2

    5 2 2 21 , l h TM W 2 29 a g SC c CTI U di fe b
  7. 3 1 4 . 5 A 5 A 3 3

    A 9 A A .0,, ce ceW g l t im iMzrNS Zbe a sh 2- g l t shn v KU d a f o im iMcSd Ta d s I
  8. ( , - K 5 2 9 . 9 I

    A I -AA I 2 00.3 fe /.61- • + T ZM db • ( , ( W MUS N -52 ca • ( I 9 • () 9
  9. 9 ,1. 1 1 9 22 1 3 1 1

    10 I M A
  10. , 0 09 .0 . 9 ,11 , 0 9

    2 90 09 0 A
  11. 9 ,1. 1 /1 / 9 22 1 1 1

    10 I a , W U SA , T M
  12. C 0 1 , 2 2 0 9 0 99

    ,. c aE cW r t b mIM A1 c aE cW g Ee dE n lvOST f h r t i f Osz y U o L
  13. B 9 A . 0 1 1 2 , .

    cM b UD T DW gd fe ih D D S a I
  14. 9 ,1. 1 /1 / 9 22 1 1 1

    10 I a , W U SA , T M
  15. , 0 09 .0 . 9 ,11 , 0 9

    2 90 09 0 ➢ S ab M ➢ TA I ➢ UTA W
  16. , 0 09 .0 . 9 ,11 , 0 9

    2 90 09 0 ➢ S ab M ➢ TA I ➢ UTA W
  17. , 0 09 .0 . 9 ,11 , 0 9

    2 90 09 0 WI a T S I M A U I W 0 T
  18. ( 2 1 3 . A 2 2A 9A .0,,

    / ➢c g d MP g g v ➢ S a z M lo T ➢ We M g g v I i M M r ht n 1 3 U htm I T b loPf W r 1 3 U s
  19. I 4 A 3 0 C - AC E 499

    4E C E C C 02//-1 30 b ST L AF 34E E Uz mre L0E ,F E A b u .4 4 v a L dtkltk b L FEA 0 4 fs oW bkg p rtf L 4 b L a dtkltk Uh nkdt Lcorg i t T M
  20. I 4 A 3 0 C - AC E 499

    4E C E C C 02//-1 30 b ST L AF 34E E Uz mre L0E ,F E A b u .4 4 v a L dtkltk b L FEA 0 4 fs oW bkg p rtf L 4 b L a dtkltk Uh nkdt Lcorg i t T M
  21. I 4 A 3 0 C - AC E 499

    4E C E C C 02//-1 30 b ST L AF 34E E Uz mre L0E ,F E A b u .4 4 v a L dtkltk b L FEA 0 4 fs oW bkg p rtf L 4 b L a dtkltk Uh nkdt Lcorg i t T M
  22. I 4 A 3 0 C - AC E 499

    4E C E C C 02//-1 30 b ST L AF 34E E Uz mre L0E ,F E A b u .4 4 v a L dtkltk b L FEA 0 4 fs oW bkg p rtf L 4 b L a dtkltk Uh nkdt Lcorg i t T M
  23. I 4 A 3 0 C - AC E 499

    4E C E C C 02//-1 30 b ST L AF 34E E Uz mre L0E ,F E A b u .4 4 v a L dtkltk b L FEA 0 4 fs oW bkg p rtf L 4 b L a dtkltk Uh nkdt Lcorg i t T M
  24. I 4 A 3 0 C - AC E 499

    4E C E C C 02//-1 30 b ST L AF 34E E Uz mre L0E ,F E A b u .4 4 v a L dtkltk b L FEA 0 4 fs oW bkg p rtf L 4 b L a dtkltk Uh nkdt Lcorg i t T M
  25. I 4 A 3 0 C - AC E 499

    4E C E C C 02//-1 30 b ST L AF 34E E Uz mre L0E ,F E A b u .4 4 v a L dtkltk b L FEA 0 4 fs oW bkg p rtf L 4 b L a dtkltk Uh nkdt Lcorg i t T M
  26. , 0 09 .0 . 9 ,11 , 0 9

    2 90 09 0 A MI
  27. , 0 0 .0 . ,11 , 0 2 0

    0 0 9 , .0 ebM T gvf S W pa A in s r d 0 U Iot Ie h mpalc
  28. , 0 09 .0 . 9 ,11 , 0 9

    2 90 09 0 ➢ S ab M ➢ TA I ➢ UTA W
  29. L B8 D 7 9 I G / D G

    8 A 8 G AA G G I 611/5 Sn lM f ➢ , h M ◦ 2 D 8 I D 1 30 ◦ n e h ◦ h o sy ➢ t g iv f ◦ S ., 8 ◦ t a oe bp ds ceo oz ◦ fr ay mb pP ◦ M TU QW
  30. , 9M D 8 5 G A 0D G AI

    9 ABA9I ,BB GA I G G 572206 , G G9 fW ➢ . tQ Qa ◦ 3 G D9 5 G G G 2L541 a P . i S ◦ yl eftQ Q yl b 6 c ◦ tQ Qz a b 9AB G U hgf ➢ aTTf sQu r ◦ MA / 9L ◦ m Qz n ap vo z z Qv © ◦ a r m a n Q U ◦ e b TTfW d
  31. ) 2 4 A A 88 A 9 A A

    6 1 0 ., Mh e i Q Ua Mglf McdIb M TQI S aL ◦ 64 ( 9 (QW S S ◦ 64 9 QW S S
  32. 9 A . 0 1 ) 1 2 ), I

    e ca ( g ( . ( 0 1 hf T b d SM W D U
  33. 9 ,1. 1 1 9 22 1 1 1 10

    WaM 9 M SU ID A T
  34. 9 ,1. 1 1 9 22 1 1 1 10

    DI A
  35. 2 16 .6 C 46 , 4 A 2 2A6

    89A 6 6 C6 .0 , . ub l lv i • oI M oI M • . ida m a • cxIM lv h l s eIg b i i Witmd • B 12A49 a M itmd • 4 D 2 86 D 2 86 D S l . id • fI nIrz U . id W T • ida
  36. , D M 9 6 B I 1 B I

    B B I B A I I 683317 TUsPp i ➢ /. PvP ◦ 6 3L652 ( ( ort )ort ◦ scnh PvP ◦ PvPt ➢ QQ by Pl i ◦ MBG TU ) 0. L ◦ y dzPth e g mfhtnt Pm ◦ i ad e P ◦ W QQ S
  37. A 9 .30 3 13 1 3 3 3 32

    , tlc W gv M 3 im Q im bd nr n r W LM im Wo IMSe Qim afsh U T
  38. , 0 09 .0 . 9 ,11 , 0 9

    2 90 09 0 ➢ S ab M ➢ TA I ➢ UTA W
  39. 0 1 , A 2 2 0 9 0 99

    A ,. W gom TJ I . 2 019 acrn MWJN WMS 0 0 htf selO b d WM p O T v i U
  40. , 0 09 .0 . 9 ,11 , 0 9

    2 90 09 0 b g sm v T d et i lfU hz W v arI nA M S c o
  41. , 0 09 .0 . 9 ,11 , 0 9

    2 90 09 0 Ø M Ø U S M Ø IW T A
  42. , 0 09 .0 . 9 ,11 , 0 9

    2 90 09 0 ➢ M ➢U S M ➢IW T A
  43. , 0 09 .0 . 9 ,11 , 0 9

    2 90 09 0 SW SW A M SU IT
  44. 9 ,1. 1 1 9 22 1 3 1 1

    10 A IM SUT IM SUT AM IM WT IM SUT
  45. /,, . ( 1 0 2 93 A 3 9

    A 1 9 91 A 9 A A U adgrep M A) 3A 9 3 :1 3 ,1 1 U adgrepA U adgrep nr W y u jnadh v U ad lcp b m j fi c oaj rts S U WihTWp I
  46. , 0 09 .0 . 9 ,11 , 0 9

    2 90 09 0 W c a b d i T c W c a e M r l smgA W c a b f d cS M smhA T c U cI W c a U c e on T c r
  47. 1 0 2 D93 A 3 9BA 1 9 91B

    A 9 BA A D . adfmekchlW C9 D I 9 A gk e m r n sgke v bah S ahl eMT S ahl eUdj N hj im adfmek o f im U Wm d hj im o f imU u z N , t ml e N s
  48. 1 0 2 D93 A 3 9BA 1 9 91B

    A 9 BA A D . adfmekchlW C9 D I 9 A gk e m r n sgke v bah S ahl eMT S ahl eUdj N hj im adfmek o f im U Wm d hj im o f imU u z N , t ml e N s
  49. 1 0 2 D93 A 3 9BA 1 9 91B

    A 9 BA A D . adfmekchlW C9 D I 9 A gk e m r n sgke v bah S ahl eMT S ahl eUdj N hj im adfmek o f im U Wm d hj im o f imU u z N , t ml e N s
  50. 1 0 2 D93 A 3 9BA 1 9 91B

    A 9 BA A D . adfmekchlW C9 D I 9 A gk e m r n sgke v bah S ahl eMT S ahl eUdj N hj im adfmek o f im U Wm d hj im o f imU u z N , t ml e N s
  51. 9 ,1. 1 1 9 22 1 3 1 1

    10 A SUT IM SUT AM IM WT IM SUT
  52. 1 0 2 - C9 9A 1 9 91A 9

    A C - ,, . c vl S to M I W 9A > 9A 12 9A 12 M 1 2 1 1 2 1 m lugorL n n horugorW to T I W T to Uzfdlr a to i p r b to U i p r b ez s
  53. , 0 09 .0 . 9 ,11 , 0 9

    2 90 09 0 a W c T U M S AI b
  54. 9 ,1. 1 1 9 22 1 1 1 10

    b d SU W ATAI M C c a 1
  55. , 0 09 .0 . 9 ,11 , 0 9

    2 90 09 0 a W c T U M S AI b
  56. .A 9 2 2 02 90 .33 . 2 2

    2 21 , smh pt n c d ITMoi a fU l r U e C 992 a fUW bgC S
  57. A 9 .30 3 13 1 3 3 3 32

    , d nso C af TLle WbUgiIM M 902 Ur c 902 ns WbU hm S
  58. 0A 1 , 2 2 0 9 0 99 ,.

    rlf ot m c CT IS dh ng b eW M DM 0 1 0 otb eW 0 M N LMDUa is U
  59. A 9 . 0 1 1 2 , oi s

    c CT S d Nr a gWln D vM mt bf eh T lMUI U
  60. 9 ,1. 1 1 9 22 1 3 1 1

    10 A SUT IM SUT AM IM WT IM SUT
  61. . 9 2/ 2 02 90 . . 2 2

    2 21 , sI eg v eg p d T ny eg ru mru S gi ah c beg eg vU eg mru zl TS tI eg v 0 U A90 eg p oy ibfT eg ru MS W mru
  62. A 9 ,1. 1 1 9 22 1 3 1

    1 10 m f S U bW a v 3 igs p a M ctI ba u M h T unr Ub b lo hT deI
  63. A 9 ,1. 1 1 9 22 1 3 1

    1 10 m f S U bW a v 3 igs p a M ctI ba u M h T unr Ub b lo hT deI
  64. A 9 ,1. 1 1 9 22 1 3 1

    1 10 m f S U bW a v 3 igs p a M ctI ba u M h T unr Ub b lo hT deI
  65. A 9 ,1. 1 1 9 22 1 3 1

    1 10 m f S U bW a v 3 igs p a M ctI ba u M h T unr Ub b lo hT deI
  66. , 0 09 .0 . 9 ,11 , 0 9

    2 90 09 0 ➢ M ➢U S M ➢IW T A
  67. A . 9 2 2 02 90 . . 2

    2 2 21 , aP I 2 0.0 2 e ci s UT P z r s wo M 21 P S 2 bP f g e ci 0 9 29 . 9 lmn 0 9 29 . 9 M Rtv w aP e ci aP lmn dhWI U
  68. 1E 0 2 C9 9A 1 9 91A 9 A

    C . rt nlpm 1 9 1A9 9 T R rws , 9 T 1 9 9 A A 1 9 Tz 0 o 1 A9 1 cews v ,1A 9 9A MS LT , 9 T ifb P , 9 dcP cg uU WT 2 a hU 1 1 I T ifb P
  69. , 0 09 .0 . 9 ,11 , 0 9

    2 90 09 0 ➢ M ➢U S M ➢IW T A
  70. 9 ,1. 1 5 1 9 5 225 5 1

    5 1 1 10 l o a e A lUa U T fhW A vS fhU T g A d i t b l t A l r A S t S S a e A n I S t A i c e g A U g t b m l t A I n I s A S t e n T S z Ui cU fh e M
  71. 9 ,1. 1 1 9 22 1 1 1 10

    g f e A M T UMIS g A .0 W d c a L b
  72. . 9 2 2 02 90 . . 2 2

    2 21 , S E BI T A M
  73. . 9 ,2 2 02 90 . . 2 2

    2 21 ve ds b T ve ds gf l bt A MoS mn a W irT Zd M hI dUc s  
  74. , A . 9 2 2 02 90 . .

    2 2 2 21 9 . 0 2 29 .9. 2 29 e f v ru cbh f naogbm dlEi M W s I zt U IST S I S
  75. 9 ,1. 1 /1 / 9 22 1 1 1

    10 I a , W U SA , T M
  76. 5 A 9 19 F 9C A DC 5 5D9C

    DC 9C9 F9 13.. 2 ab d l ab d l 0A D9 Uze RN h I M d u Wzeo bc 1 5 AF9 A , A D . T f S r t n gm i vs
  77. IA 3 5 0 + A 99 02,,+1 phn io

    g M f URTW vytc ph m 0.- O 0 5 5 F m k a SP U re lU 0.- m l M lO 9 A z s bud
  78. ) 2 = 1 . = A 2 2A )

    9A .0,, a T b .+ ( z .+ ( .+ iz .+) ( .+ d ntSlghO WvmOs Uo IMiz I c LM efWlg r
  79. / . 1 052 5 935 3 9A 1 9

    91A5 9 A 5 5 5 W ohn c 0 W MUtU U • nciocsu T • acf T • U tU • , A5 ,5 9 SRU • lIgb efT mc A A1 35 Up rSR z dI v T W
  80. 1 0 2 9 A 9 A 1 9 91

    A 9 A A / . , a olb n gdje f i a olRU S h W c Mm T I A EEE A 1 : 9