Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ある SRE チームの挑戦 運用 6 年目の大規模ゲームを AWS 移設後に安定運用するための技術と今後の展望

ある SRE チームの挑戦 運用 6 年目の大規模ゲームを AWS 移設後に安定運用するための技術と今後の展望

AWS Summit 2019 の発表資料です。

CyberAgent SGE Engineer

June 14, 2019
Tweet

More Decks by CyberAgent SGE Engineer

Other Decks in Technology

Transcript

  1. E 1 0 2 9 9A 1669 91A 9 A
    / .
    , RL
    T LSah R 0 c U
    IMLbd WULgf
    i
    . 1 1 1
    1 , . 1
    e l
    . 1
    1 , . 1

    View full-size slide

  2. 9 ,1. 1 /1 / 9 22 1 1 1 10
    I a
    , W U
    SA , T
    M

    View full-size slide

  3. 9 ,1. 1 /1 / 9 22 1 1 1 10
    I a
    , W U
    SA , T
    M

    View full-size slide

  4. 3E 2 4 5 A 5 A 3 3 A 9 A A 6
    1 0
    r
    l h f
    . h
    0 3 3 39
    n
    s Wd b u
    h f m M . h M hgc
    i b e R ah z n o
    L 2 e S 3E 3
    n
    s Wd b u
    h f m M 2 R I zI t
    T t z vR ah M U
    L 2 e S .
    l h f
    . h
    0 5 , 36

    View full-size slide

  5. 1 0 2 A9 9 1669 91 9 8 A 4
    .
    fvglsT , v
    fvglsTd v g TWR S S
    s Ef iT
    I
    r T ETbt hz T
    e n b e o c T
    ra i o c r
    ms o c T
    d vT U R M T
    ra i S RM U

    View full-size slide

  6. A . 9 2 2 02 90 . . 2 5 2 2 21
    ,
    l
    h TM
    W 2 29 a
    g
    SC c CTI U di fe b

    View full-size slide

  7. 3 1 4 . 5 A 5 A 3 3 A 9 A A
    .0,,
    ce ceW
    g l t
    im iMzrNS Zbe a
    sh 2-
    g l t
    shn v KU d a
    f o im iMcSd Ta d
    s I

    View full-size slide

  8. ( , - K 5 2 9 . 9 I A I -AA I
    2 00.3
    fe /.61-
    • + T ZM db
    • ( , (
    W MUS N
    -52 ca
    • ( I 9
    • () 9

    View full-size slide

  9. 9 ,1. 1 1 9 22 1 3 1 1 10
    I
    M A

    View full-size slide

  10. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0
    A

    View full-size slide

  11. 9 ,1. 1 /1 / 9 22 1 1 1 10
    I a
    , W U
    SA , T
    M

    View full-size slide

  12. C 0 1 , 2 2 0 9 0 99
    ,.
    c aE cW r t
    b mIM A1 c aE cW
    g Ee dE n lvOST
    f h r t i
    f Osz y U o L

    View full-size slide

  13. B 9 A . 0 1 1 2
    ,
    .
    cM b UD T DW gd
    fe ih D D S a I

    View full-size slide

  14. 9 ,1. 1 /1 / 9 22 1 1 1 10
    I a
    , W U
    SA , T
    M

    View full-size slide

  15. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0
    ➢ S ab M
    ➢ TA I
    ➢ UTA W

    View full-size slide

  16. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0
    ➢ S ab M
    ➢ TA I
    ➢ UTA W

    View full-size slide

  17. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0
    WI a T
    S I M A U I W 0 T

    View full-size slide

  18. ( 2 1 3 . A 2 2A 9A
    .0,, /
    ➢c g d MP g g v
    ➢ S a z M lo T
    ➢ We M g g v I i
    M M
    r ht n 1 3 U
    htm I T
    b loPf W r
    1 3 U s

    View full-size slide

  19. I 4 A 3 0 C - AC E 499 4E C E C C
    02//-1
    30 b ST
    L AF 34E E Uz mre
    L0E ,F E A b u .4 4
    v a
    L dtkltk b
    L FEA 0 4 fs
    oW bkg p rtf
    L 4 b
    L a dtkltk
    Uh nkdt
    Lcorg i t T
    M

    View full-size slide

  20. I 4 A 3 0 C - AC E 499 4E C E C C
    02//-1
    30 b ST
    L AF 34E E Uz mre
    L0E ,F E A b u .4 4
    v a
    L dtkltk b
    L FEA 0 4 fs
    oW bkg p rtf
    L 4 b
    L a dtkltk
    Uh nkdt
    Lcorg i t T
    M

    View full-size slide

  21. I 4 A 3 0 C - AC E 499 4E C E C C
    02//-1
    30 b ST
    L AF 34E E Uz mre
    L0E ,F E A b u .4 4
    v a
    L dtkltk b
    L FEA 0 4 fs
    oW bkg p rtf
    L 4 b
    L a dtkltk
    Uh nkdt
    Lcorg i t T
    M

    View full-size slide

  22. I 4 A 3 0 C - AC E 499 4E C E C C
    02//-1
    30 b ST
    L AF 34E E Uz mre
    L0E ,F E A b u .4 4
    v a
    L dtkltk b
    L FEA 0 4 fs
    oW bkg p rtf
    L 4 b
    L a dtkltk
    Uh nkdt
    Lcorg i t T
    M

    View full-size slide

  23. I 4 A 3 0 C - AC E 499 4E C E C C
    02//-1
    30 b ST
    L AF 34E E Uz mre
    L0E ,F E A b u .4 4
    v a
    L dtkltk b
    L FEA 0 4 fs
    oW bkg p rtf
    L 4 b
    L a dtkltk
    Uh nkdt
    Lcorg i t T
    M

    View full-size slide

  24. I 4 A 3 0 C - AC E 499 4E C E C C
    02//-1
    30 b ST
    L AF 34E E Uz mre
    L0E ,F E A b u .4 4
    v a
    L dtkltk b
    L FEA 0 4 fs
    oW bkg p rtf
    L 4 b
    L a dtkltk
    Uh nkdt
    Lcorg i t T
    M

    View full-size slide

  25. I 4 A 3 0 C - AC E 499 4E C E C C
    02//-1
    30 b ST
    L AF 34E E Uz mre
    L0E ,F E A b u .4 4
    v a
    L dtkltk b
    L FEA 0 4 fs
    oW bkg p rtf
    L 4 b
    L a dtkltk
    Uh nkdt
    Lcorg i t T
    M

    View full-size slide

  26. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0
    A MI

    View full-size slide

  27. , 0 0 .0 . ,11 , 0 2 0 0 0
    9 , .0
    ebM T gvf S W
    pa A
    in s r d 0 U Iot Ie h
    mpalc

    View full-size slide

  28. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0
    ➢ S ab M
    ➢ TA I
    ➢ UTA W

    View full-size slide

  29. L B8 D 7 9 I G / D G 8 A 8 G AA G G I
    611/5
    Sn lM f
    ➢ , h M
    ○ 2 D 8 I D 1 30
    ○ n e h
    ○ h o sy
    ➢ t g iv f
    ○ S ., 8
    ○ t a oe bp ds ceo oz
    ○ fr ay mb pP
    ○ M TU QW

    View full-size slide

  30. , 9M D 8 5 G A 0D G AI 9 ABA9I ,BB GA I G G
    572206
    , G G9 fW
    ➢ . tQ Qa
    ○ 3 G D9 5 G G G 2L541 a P . i S
    ○ yl eftQ Q yl b 6 c
    ○ tQ Qz a b 9AB G U hgf
    ➢ aTTf sQu r
    ○ MA / 9L
    ○ m Qz n ap vo z z Qv ©
    ○ a r m a n Q U
    ○ e b TTfW d

    View full-size slide

  31. ) 2 4 A A 88 A 9 A A 6
    1 0
    ., Mh e
    i Q Ua Mglf McdIb M TQI
    S aL

    64 ( 9 (QW S S

    64 9 QW S S

    View full-size slide

  32. 9 A . 0 1 ) 1 2
    ),
    I e ca ( g
    ( . ( 0 1
    hf T b d SM W
    D U

    View full-size slide

  33. 9 ,1. 1 1 9 22 1 1 1 10
    WaM
    9 M SU ID A T

    View full-size slide

  34. 9 ,1. 1 1 9 22 1 1 1 10
    DI A

    View full-size slide

  35. 2 16 .6 C 46 , 4 A 2 2A6 89A 6 6 C6
    .0 ,
    . ub
    l
    lv i

    oI M oI M

    . ida m a

    cxIM lv h
    l
    s eIg b i i Witmd

    B 12A49 a M itmd

    4 D 2 86 D 2 86 D S
    l
    . id

    fI nIrz U . id W T

    ida

    View full-size slide

  36. , D M 9 6 B I 1 B I B B I B A I I
    683317
    TUsPp i
    ➢ /. PvP
    ○ 6 3L652 ( ( ort )ort
    ○ scnh PvP
    ○ PvPt
    ➢ QQ by Pl i
    ○ MBG TU ) 0. L
    ○ y dzPth e g mfhtnt Pm
    ○ i ad e P
    ○ W QQ S

    View full-size slide

  37. A 9 .30 3 13 1 3 3 3 32
    ,
    tlc W gv M 3 im
    Q
    im bd nr n r
    W LM im
    Wo IMSe Qim
    afsh U T

    View full-size slide

  38. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0
    ➢ S ab M
    ➢ TA I
    ➢ UTA W

    View full-size slide

  39. 0 1 , A 2 2 0 9 0 99 A
    ,.
    W gom TJ
    I . 2 019
    acrn MWJN WMS
    0 0 htf selO
    b d WM p O T v i U

    View full-size slide

  40. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0
    b g sm v
    T d et i
    lfU hz
    W v
    arI nA M
    S c o

    View full-size slide

  41. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0
    Ø M
    Ø U S M
    Ø IW T
    A

    View full-size slide

  42. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0
    ➢ M
    ➢U S M
    ➢IW T
    A

    View full-size slide

  43. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0
    SW
    SW A
    M SU IT

    View full-size slide

  44. 9 ,1. 1 1 9 22 1 3 1 1 10
    A IM SUT
    IM SUT
    AM IM WT
    IM SUT

    View full-size slide

  45. /,, . ( 1 0 2 93 A 3 9 A 1 9 91 A 9 A A
    U adgrep M
    A) 3A 9 3 :1 3 ,1 1 U adgrepA
    U adgrep nr W y u jnadh v
    U ad lcp b m j fi c oaj
    rts S U WihTWp I

    View full-size slide

  46. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0
    W c a b d i
    T c W c a e M r l
    smgA
    W c a b f
    d cS M
    smhA
    T c U cI
    W c a U c
    e on
    T c r

    View full-size slide

  47. 1 0 2 D93 A 3 9BA 1 9 91B A 9 BA A D
    .
    adfmekchlW
    C9 D I 9 A gk
    e m r
    n sgke v bah
    S ahl eMT
    S ahl eUdj
    N hj im
    adfmek o f im
    U Wm d
    hj im o
    f imU u z N
    , t ml e N s

    View full-size slide

  48. 1 0 2 D93 A 3 9BA 1 9 91B A 9 BA A D
    .
    adfmekchlW
    C9 D I 9 A gk
    e m r
    n sgke v bah
    S ahl eMT
    S ahl eUdj
    N hj im
    adfmek o f im
    U Wm d
    hj im o
    f imU u z N
    , t ml e N s

    View full-size slide

  49. 1 0 2 D93 A 3 9BA 1 9 91B A 9 BA A D
    .
    adfmekchlW
    C9 D I 9 A gk
    e m r
    n sgke v bah
    S ahl eMT
    S ahl eUdj
    N hj im
    adfmek o f im
    U Wm d
    hj im o
    f imU u z N
    , t ml e N s

    View full-size slide

  50. 1 0 2 D93 A 3 9BA 1 9 91B A 9 BA A D
    .
    adfmekchlW
    C9 D I 9 A gk
    e m r
    n sgke v bah
    S ahl eMT
    S ahl eUdj
    N hj im
    adfmek o f im
    U Wm d
    hj im o
    f imU u z N
    , t ml e N s

    View full-size slide

  51. 9 ,1. 1 1 9 22 1 3 1 1 10
    A SUT
    IM SUT
    AM IM WT
    IM SUT

    View full-size slide

  52. 1 0 2 - C9 9A 1 9 91A 9 A C
    - ,, .
    c
    vl S to M
    I
    W
    9A > 9A 12
    9A 12 M 1 2 1
    1 2 1 m lugorL
    n
    n horugorW
    to T
    I
    W T
    to Uzfdlr
    a to
    i p r b
    to U
    i p r b
    ez s

    View full-size slide

  53. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0
    a W c T
    U M S AI b

    View full-size slide

  54. 9 ,1. 1 1 9 22 1 1 1 10
    b d SU
    W ATAI M C c a
    1

    View full-size slide

  55. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0
    a W c T
    U M S AI b

    View full-size slide

  56. .A 9 2 2 02 90 .33 . 2 2 2 21
    ,
    smh pt n c
    d ITMoi a fU l r
    U e C 992
    a fUW bgC S

    View full-size slide

  57. A 9 .30 3 13 1 3 3 3 32
    ,
    d nso
    C af TLle WbUgiIM M
    902 Ur c
    902 ns WbU hm S

    View full-size slide

  58. 0A 1 , 2 2 0 9 0 99
    ,.
    rlf ot m c
    CT IS dh ng b eW M DM
    0 1 0 otb eW 0 M
    N LMDUa is U

    View full-size slide

  59. A 9 . 0 1 1 2
    ,
    oi s c
    CT S d Nr a gWln D vM
    mt bf eh T lMUI U

    View full-size slide

  60. 9 ,1. 1 1 9 22 1 3 1 1 10
    A SUT
    IM SUT
    AM IM WT
    IM SUT

    View full-size slide

  61. . 9 2/ 2 02 90 . . 2 2 2 21
    ,
    sI
    eg v
    eg p
    d T ny
    eg ru
    mru S
    gi ah c beg
    eg vU eg mru zl TS
    tI
    eg v
    0 U A90
    eg p
    oy ibfT
    eg ru
    MS W
    mru

    View full-size slide

  62. A 9 ,1. 1 1 9 22 1 3 1 1 10
    m f S U bW a
    v 3 igs
    p a M
    ctI ba
    u M
    h T unr
    Ub b lo
    hT deI

    View full-size slide

  63. A 9 ,1. 1 1 9 22 1 3 1 1 10
    m f S U bW a
    v 3 igs
    p a M
    ctI ba
    u M
    h T unr
    Ub b lo
    hT deI

    View full-size slide

  64. A 9 ,1. 1 1 9 22 1 3 1 1 10
    m f S U bW a
    v 3 igs
    p a M
    ctI ba
    u M
    h T unr
    Ub b lo
    hT deI

    View full-size slide

  65. A 9 ,1. 1 1 9 22 1 3 1 1 10
    m f S U bW a
    v 3 igs
    p a M
    ctI ba
    u M
    h T unr
    Ub b lo
    hT deI

    View full-size slide

  66. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0
    ➢ M
    ➢U S M
    ➢IW T
    A

    View full-size slide

  67. A . 9 2 2 02 90 . . 2 2 2 21
    ,
    aP I 2 0.0 2 e ci
    s UT P
    z r s
    wo
    M
    21 P S
    2 bP f g
    e ci
    0 9 29 . 9
    lmn
    0 9 29 . 9 M
    Rtv
    w
    aP
    e ci
    aP
    lmn
    dhWI U

    View full-size slide

  68. 1E 0 2 C9 9A 1 9 91A 9 A C
    .
    rt nlpm
    1 9 1A9 9 T R rws , 9 T
    1 9 9 A A 1 9 Tz
    0 o 1 A9 1 cews v
    ,1A 9 9A MS LT , 9 T ifb P
    , 9 dcP cg uU WT 2 a hU
    1 1 I T ifb P

    View full-size slide

  69. , 0 09 .0 . 9 ,11 , 0 9 2 90 09 0
    ➢ M
    ➢U S M
    ➢IW T
    A

    View full-size slide

  70. 9 ,1. 1 5 1 9 5 225 5 1 5 1 1 10
    l
    o
    a e
    A lUa U T fhW
    A vS fhU T
    g
    A d i t b
    l t
    A l r
    A S t S S
    a e
    A n I S t
    A i c e
    g
    A U g t b m
    l t
    A I n I s
    A S t e n T
    S z Ui cU fh e M

    View full-size slide

  71. 9 ,1. 1 1 9 22 1 1 1 10
    g f
    e
    A M
    T UMIS g
    A .0 W
    d c a
    L b

    View full-size slide

  72. . 9 2 2 02 90 . . 2 2 2 21
    ,
    S E BI
    T A M

    View full-size slide

  73. . 9 ,2 2 02 90 . . 2 2 2 21
    ve ds
    b T
    ve ds
    gf
    l bt A
    MoS mn
    a W irT
    Zd M
    hI dUc s


    View full-size slide

  74. , A . 9 2 2 02 90 . . 2 2 2 21
    9 . 0 2 29 .9. 2 29
    e f v ru cbh f naogbm dlEi M W
    s I zt U IST S
    I S

    View full-size slide

  75. 9 ,1. 1 /1 / 9 22 1 1 1 10
    I a
    , W U
    SA , T
    M

    View full-size slide

  76. 5 A 9 19 F 9C A DC 5 5D9C DC 9C9 F9
    13.. 2
    ab d l
    ab d l
    0A D9 Uze RN
    h I M d
    u Wzeo
    bc
    1 5 AF9
    A , A D . T f S
    r t n gm i vs

    View full-size slide

  77. IA 3 5 0 + A 99
    02,,+1
    phn
    io
    g M f URTW vytc
    ph
    m 0.- O
    0 5 5 F m k a
    SP U
    re lU 0.-
    m l M lO
    9 A z s bud

    View full-size slide

  78. ) 2 = 1 . = A 2 2A ) 9A
    .0,,
    a T b
    .+ ( z
    .+ ( .+ iz
    .+) ( .+ d
    ntSlghO WvmOs Uo IMiz I
    c LM efWlg r

    View full-size slide

  79. / . 1 052 5 935 3 9A 1 9 91A5 9 A 5 5 5
    W
    ohn c 0 W
    MUtU U
    ● nciocsu T
    ● acf T
    ● U
    tU
    ● , A5 ,5 9 SRU
    ● lIgb efT mc
    A A1 35 Up rSR z dI v T W

    View full-size slide

  80. 1 0 2 9 A 9 A 1 9 91 A 9 A A
    / .
    , a olb
    n gdje f i
    a olRU S
    h W c Mm T I
    A EEE A 1 : 9

    View full-size slide