ある SRE チームの挑戦 運用 6 年目の大規模ゲームを AWS 移設後に安定運用するための技術と今後の展望

AWS Summit 2019 の発表資料です。

Transcript

1. E 1 0 2 9 9A 1669 91A 9 A / . , RL T LSah R 0 c U IMLbd WULgf i . 1 1 1 1 , . 1 e l . 1 1 , . 1
2. 9 ,1. 1 /1 / 9 22 1 1 1 10 I a , W U SA , T M
3. 9 ,1. 1 /1 / 9 22 1 1 1 10 I a , W U SA , T M
4. 3E 2 4 5 A 5 A 3 3 A 9 A A 6 1 0 r l h f . h 0 3 3 39 n s Wd b u h f m M . h M hgc i b e R ah z n o L 2 e S 3E 3 n s Wd b u h f m M 2 R I zI t T t z vR ah M U L 2 e S . l h f . h 0 5 , 36
5. 1 0 2 A9 9 1669 91 9 8 A 4 . fvglsT , v fvglsTd v g TWR S S s Ef iT I r T ETbt hz T e n b e o c T ra i o c r ms o c T d vT U R M T ra i S RM U
6. A . 9 2 2 02 90 . . 2 5 2 2 21 , l h TM W 2 29 a g SC c CTI U di fe b
7. 3 1 4 . 5 A 5 A 3 3 A 9 A A .0,, ce ceW g l t im iMzrNS Zbe a sh 2- g l t shn v KU d a f o im iMcSd Ta d s I
8. ( , - K 5 2 9 . 9 I A I -AA I 2 00.3 fe /.61- • + T ZM db • ( , ( W MUS N -52 ca • ( I 9 • () 9
9. 9 ,1. 1 1 9 22 1 3 1 1 10 I M A
10. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0 A
11. 9 ,1. 1 /1 / 9 22 1 1 1 10 I a , W U SA , T M
12. C 0 1 , 2 2 0 9 0 99 ,. c aE cW r t b mIM A1 c aE cW g Ee dE n lvOST f h r t i f Osz y U o L
13. B 9 A . 0 1 1 2 , . cM b UD T DW gd fe ih D D S a I
14. 9 ,1. 1 /1 / 9 22 1 1 1 10 I a , W U SA , T M
15. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0 ➢ S ab M ➢ TA I ➢ UTA W
16. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0 ➢ S ab M ➢ TA I ➢ UTA W
17. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0 WI a T S I M A U I W 0 T
18. ( 2 1 3 . A 2 2A 9A .0,, / ➢c g d MP g g v ➢ S a z M lo T ➢ We M g g v I i M M r ht n 1 3 U htm I T b loPf W r 1 3 U s
19. I 4 A 3 0 C - AC E 499 4E C E C C 02//-1 30 b ST L AF 34E E Uz mre L0E ,F E A b u .4 4 v a L dtkltk b L FEA 0 4 fs oW bkg p rtf L 4 b L a dtkltk Uh nkdt Lcorg i t T M
20. I 4 A 3 0 C - AC E 499 4E C E C C 02//-1 30 b ST L AF 34E E Uz mre L0E ,F E A b u .4 4 v a L dtkltk b L FEA 0 4 fs oW bkg p rtf L 4 b L a dtkltk Uh nkdt Lcorg i t T M
21. I 4 A 3 0 C - AC E 499 4E C E C C 02//-1 30 b ST L AF 34E E Uz mre L0E ,F E A b u .4 4 v a L dtkltk b L FEA 0 4 fs oW bkg p rtf L 4 b L a dtkltk Uh nkdt Lcorg i t T M
22. I 4 A 3 0 C - AC E 499 4E C E C C 02//-1 30 b ST L AF 34E E Uz mre L0E ,F E A b u .4 4 v a L dtkltk b L FEA 0 4 fs oW bkg p rtf L 4 b L a dtkltk Uh nkdt Lcorg i t T M
23. I 4 A 3 0 C - AC E 499 4E C E C C 02//-1 30 b ST L AF 34E E Uz mre L0E ,F E A b u .4 4 v a L dtkltk b L FEA 0 4 fs oW bkg p rtf L 4 b L a dtkltk Uh nkdt Lcorg i t T M
24. I 4 A 3 0 C - AC E 499 4E C E C C 02//-1 30 b ST L AF 34E E Uz mre L0E ,F E A b u .4 4 v a L dtkltk b L FEA 0 4 fs oW bkg p rtf L 4 b L a dtkltk Uh nkdt Lcorg i t T M
25. I 4 A 3 0 C - AC E 499 4E C E C C 02//-1 30 b ST L AF 34E E Uz mre L0E ,F E A b u .4 4 v a L dtkltk b L FEA 0 4 fs oW bkg p rtf L 4 b L a dtkltk Uh nkdt Lcorg i t T M
26. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0 A MI
27. , 0 0 .0 . ,11 , 0 2 0 0 0 9 , .0 ebM T gvf S W pa A in s r d 0 U Iot Ie h mpalc
28. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0 ➢ S ab M ➢ TA I ➢ UTA W
29. L B8 D 7 9 I G / D G 8 A 8 G AA G G I 611/5 Sn lM f ➢ , h M ◦ 2 D 8 I D 1 30 ◦ n e h ◦ h o sy ➢ t g iv f ◦ S ., 8 ◦ t a oe bp ds ceo oz ◦ fr ay mb pP ◦ M TU QW
30. , 9M D 8 5 G A 0D G AI 9 ABA9I ,BB GA I G G 572206 , G G9 fW ➢ . tQ Qa ◦ 3 G D9 5 G G G 2L541 a P . i S ◦ yl eftQ Q yl b 6 c ◦ tQ Qz a b 9AB G U hgf ➢ aTTf sQu r ◦ MA / 9L ◦ m Qz n ap vo z z Qv © ◦ a r m a n Q U ◦ e b TTfW d
31. ) 2 4 A A 88 A 9 A A 6 1 0 ., Mh e i Q Ua Mglf McdIb M TQI S aL ◦ 64 ( 9 (QW S S ◦ 64 9 QW S S
32. 9 A . 0 1 ) 1 2 ), I e ca ( g ( . ( 0 1 hf T b d SM W D U
33. 9 ,1. 1 1 9 22 1 1 1 10 WaM 9 M SU ID A T
34. 9 ,1. 1 1 9 22 1 1 1 10 DI A
35. 2 16 .6 C 46 , 4 A 2 2A6 89A 6 6 C6 .0 , . ub l lv i • oI M oI M • . ida m a • cxIM lv h l s eIg b i i Witmd • B 12A49 a M itmd • 4 D 2 86 D 2 86 D S l . id • fI nIrz U . id W T • ida
36. , D M 9 6 B I 1 B I B B I B A I I 683317 TUsPp i ➢ /. PvP ◦ 6 3L652 ( ( ort )ort ◦ scnh PvP ◦ PvPt ➢ QQ by Pl i ◦ MBG TU ) 0. L ◦ y dzPth e g mfhtnt Pm ◦ i ad e P ◦ W QQ S
37. A 9 .30 3 13 1 3 3 3 32 , tlc W gv M 3 im Q im bd nr n r W LM im Wo IMSe Qim afsh U T
38. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0 ➢ S ab M ➢ TA I ➢ UTA W
39. 0 1 , A 2 2 0 9 0 99 A ,. W gom TJ I . 2 019 acrn MWJN WMS 0 0 htf selO b d WM p O T v i U
40. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0 b g sm v T d et i lfU hz W v arI nA M S c o
41. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0 Ø M Ø U S M Ø IW T A
42. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0 ➢ M ➢U S M ➢IW T A
43. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0 SW SW A M SU IT
44. 9 ,1. 1 1 9 22 1 3 1 1 10 A IM SUT IM SUT AM IM WT IM SUT
45. /,, . ( 1 0 2 93 A 3 9 A 1 9 91 A 9 A A U adgrep M A) 3A 9 3 :1 3 ,1 1 U adgrepA U adgrep nr W y u jnadh v U ad lcp b m j fi c oaj rts S U WihTWp I
46. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0 W c a b d i T c W c a e M r l smgA W c a b f d cS M smhA T c U cI W c a U c e on T c r
47. 1 0 2 D93 A 3 9BA 1 9 91B A 9 BA A D . adfmekchlW C9 D I 9 A gk e m r n sgke v bah S ahl eMT S ahl eUdj N hj im adfmek o f im U Wm d hj im o f imU u z N , t ml e N s
48. 1 0 2 D93 A 3 9BA 1 9 91B A 9 BA A D . adfmekchlW C9 D I 9 A gk e m r n sgke v bah S ahl eMT S ahl eUdj N hj im adfmek o f im U Wm d hj im o f imU u z N , t ml e N s
49. 1 0 2 D93 A 3 9BA 1 9 91B A 9 BA A D . adfmekchlW C9 D I 9 A gk e m r n sgke v bah S ahl eMT S ahl eUdj N hj im adfmek o f im U Wm d hj im o f imU u z N , t ml e N s
50. 1 0 2 D93 A 3 9BA 1 9 91B A 9 BA A D . adfmekchlW C9 D I 9 A gk e m r n sgke v bah S ahl eMT S ahl eUdj N hj im adfmek o f im U Wm d hj im o f imU u z N , t ml e N s
51. 9 ,1. 1 1 9 22 1 3 1 1 10 A SUT IM SUT AM IM WT IM SUT
52. 1 0 2 - C9 9A 1 9 91A 9 A C - ,, . c vl S to M I W 9A > 9A 12 9A 12 M 1 2 1 1 2 1 m lugorL n n horugorW to T I W T to Uzfdlr a to i p r b to U i p r b ez s
53. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0 a W c T U M S AI b
54. 9 ,1. 1 1 9 22 1 1 1 10 b d SU W ATAI M C c a 1
55. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0 a W c T U M S AI b
56. .A 9 2 2 02 90 .33 . 2 2 2 21 , smh pt n c d ITMoi a fU l r U e C 992 a fUW bgC S
57. A 9 .30 3 13 1 3 3 3 32 , d nso C af TLle WbUgiIM M 902 Ur c 902 ns WbU hm S
58. 0A 1 , 2 2 0 9 0 99 ,. rlf ot m c CT IS dh ng b eW M DM 0 1 0 otb eW 0 M N LMDUa is U
59. A 9 . 0 1 1 2 , oi s c CT S d Nr a gWln D vM mt bf eh T lMUI U
60. 9 ,1. 1 1 9 22 1 3 1 1 10 A SUT IM SUT AM IM WT IM SUT
61. . 9 2/ 2 02 90 . . 2 2 2 21 , sI eg v eg p d T ny eg ru mru S gi ah c beg eg vU eg mru zl TS tI eg v 0 U A90 eg p oy ibfT eg ru MS W mru
62. A 9 ,1. 1 1 9 22 1 3 1 1 10 m f S U bW a v 3 igs p a M ctI ba u M h T unr Ub b lo hT deI
63. A 9 ,1. 1 1 9 22 1 3 1 1 10 m f S U bW a v 3 igs p a M ctI ba u M h T unr Ub b lo hT deI
64. A 9 ,1. 1 1 9 22 1 3 1 1 10 m f S U bW a v 3 igs p a M ctI ba u M h T unr Ub b lo hT deI
65. A 9 ,1. 1 1 9 22 1 3 1 1 10 m f S U bW a v 3 igs p a M ctI ba u M h T unr Ub b lo hT deI
66. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0 ➢ M ➢U S M ➢IW T A
67. A . 9 2 2 02 90 . . 2 2 2 21 , aP I 2 0.0 2 e ci s UT P z r s wo M 21 P S 2 bP f g e ci 0 9 29 . 9 lmn 0 9 29 . 9 M Rtv w aP e ci aP lmn dhWI U
68. 1E 0 2 C9 9A 1 9 91A 9 A C . rt nlpm 1 9 1A9 9 T R rws , 9 T 1 9 9 A A 1 9 Tz 0 o 1 A9 1 cews v ,1A 9 9A MS LT , 9 T ifb P , 9 dcP cg uU WT 2 a hU 1 1 I T ifb P
69. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0 ➢ M ➢U S M ➢IW T A
70. 9 ,1. 1 5 1 9 5 225 5 1 5 1 1 10 l o a e A lUa U T fhW A vS fhU T g A d i t b l t A l r A S t S S a e A n I S t A i c e g A U g t b m l t A I n I s A S t e n T S z Ui cU fh e M
71. 9 ,1. 1 1 9 22 1 1 1 10 g f e A M T UMIS g A .0 W d c a L b
72. . 9 2 2 02 90 . . 2 2 2 21 , S E BI T A M
73. . 9 ,2 2 02 90 . . 2 2 2 21 ve ds b T ve ds gf l bt A MoS mn a W irT Zd M hI dUc s 
    
     
    
    
74. , A . 9 2 2 02 90 . . 2 2 2 21 9 . 0 2 29 .9. 2 29 e f v ru cbh f naogbm dlEi M W s I zt U IST S I S
75. 9 ,1. 1 /1 / 9 22 1 1 1 10 I a , W U SA , T M
76. 5 A 9 19 F 9C A DC 5 5D9C DC 9C9 F9 13.. 2 ab d l ab d l 0A D9 Uze RN h I M d u Wzeo bc 1 5 AF9 A , A D . T f S r t n gm i vs
77. IA 3 5 0 + A 99 02,,+1 phn io g M f URTW vytc ph m 0.- O 0 5 5 F m k a SP U re lU 0.- m l M lO 9 A z s bud
78. ) 2 = 1 . = A 2 2A ) 9A .0,, a T b .+ ( z .+ ( .+ iz .+) ( .+ d ntSlghO WvmOs Uo IMiz I c LM efWlg r
79. / . 1 052 5 935 3 9A 1 9 91A5 9 A 5 5 5 W ohn c 0 W MUtU U • nciocsu T • acf T • U tU • , A5 ,5 9 SRU • lIgb efT mc A A1 35 Up rSR z dI v T W
80. 1 0 2 9 A 9 A 1 9 91 A 9 A A / . , a olb n gdje f i a olRU S h W c Mm T I A EEE A 1 : 9