サービスダウンから生まれたSWATチームが手がけるクラウド移行への道 / JAWS DAYS 2019 Hybrid Cloud Session

KDDIコマースフォワードではECモールWowma!を運営しています。

前身サービスから引き継いできたシステムは、日常的にDBのCPU使用率が90%近く、限界が近づいていました。そんな中、前代未聞の高負荷イベントの開催が決定。

15年ものの秘伝のタレ満載のシステムに対して

・DMSでのRDS(Oracle)へのDB移行
・S3とCloudFrontを利用したLPサイトのキャッシュ化
・既存APIのCloudFrontによるキャッシュ化　など

地道な改善を行うも、あっけなく落ちてしまったシステム。そこから、負荷対策チームSWATが誕生しました。

終わりの見えない負荷対策に始まり、AWSへの移行を行いながらもオンプレミスにオラクルDBを構築した経緯、数年後のシステム全体リプレイスを見据えながらマルチクラウド移行を目指すエンジニアの泥臭い戦いについてお話します。

Wowma! ”https://wowma.jp”

Transcript

1. A w K
   02 1 A
   w
   3GD 63 D ) (0
   ) (0 ) )
   66:
   b
   

   View Slide

2. View Slide

3.  
    
   b GMJ a
   66:
   e a
   3GD ) w
   Solutions Architect - Associate
   

   View Slide

4. IS P
   

   View Slide

5. IS P
    
   S P
    
   c
   S
   

   View Slide

6. A A
   A b
   

   View Slide

7. A A A
   75 GT RI
   

   View Slide

8. 2 3
   S ✋
   

   View Slide

9. A
   S ✋
   

   View Slide

10. 2 3
      6M 3 s 66:
      66:
      GT RI
    6M 3 t
    1999೥ 2016೥
    

    View Slide

11.   t a
      s •
     
    GMJ x )
    D15MSW D , 4
    , s
    IYI( ) s t w a
      b A A Sa e a
    

    View Slide

12. b
    bK t Sm
    ) (. s t r t e
     
    w t A a k d
     
    2 4 S k d
    

    View Slide

13. T m W
    

    View Slide

14. or S A d
    o
    t s ss
    s 9
    ×
    

    View Slide

15. w B A
    s
    ○
    

    View Slide

16. T
    

    View Slide

17. S i
     
    s t s w t zw
    w
     
    t t o
    •
     
    t o aaa
    

    View Slide

18. i
     
    D t
     
     
    u t
    w A
    

    View Slide

19. w
    20 w w
    

    View Slide

20. w
    6OUMKW5TSSMKW 3GD
    

    View Slide

21. 3GD
    3GDn n3GD
    

    View Slide

22. w
     
    64 t 5AF / l
     
    

    View Slide

23.  
    )
     
    w
     
    A
    C6D UIKQM
    75) UIKQM or
    

    View Slide

24. A
    C6D UIKQM
     
    LJ U / QIUNMt
     
    UIKQM
     
    D D D DE7 C6D
    

    View Slide

25. b
    

    View Slide

26. C
    A
    

    View Slide

27. D w
    Kao
    b K a
    

    View Slide

28.  
    D 5QT L8UTSW A
    

    View Slide

29.  
    3A: 5QT L8UTSW
    

    View Slide

30.  
    DB
    UIKQM DWIW AIKPs DB
    s
    

    View Slide

31.  
    GMJ 3A 3A:
     
     
    tGMJ
    DA A5 GMJ
    

    View Slide

32.  
    6 D i
    64 t v 6 D s
    u s a
    

    View Slide

33. G:
    A
    or w A w
    to z t uw v
    P m ba P b
    "
    D7g
    #
    VS
    

    View Slide

34. 02 1
    

    View Slide

35. a K
    

    View Slide

36. a K
    GMJ
    

    View Slide

37. P 02 1 A
     
    t t
    t t
     
    s )
    w
     
    A a
    02 1
    

    View Slide

38. A S
    A
    A
    A
    2 4 A A w
    

    View Slide

39. A
    A
    

    View Slide

40. A
     
    t
     
    64 )
    w u
     
     
    m u
    

    View Slide

41. C6D UIKQM
    bbb
     
    75) UIKQM
    a
    C35t3GD a
    to t
     
    UIKQM
    C6D UIKQM
     
    LJ U / QIUNMt
     
    UIKQM
     
    D D D DE7
    t r
    

    View Slide

42. A A
    S A A
     
    3GD 6OUMKW5TSSMKW G3
    u
     
    95A 3 UM
    u
    ϚϧνΫϥ΢υ؀ڥʹҰาۙͮ͘ʂ
    

    View Slide

43. wo
    6OUMKW5TSSMKW
    

    View Slide

44. A
    K A
    ) (/
    ) (/ /
    b
    s
    64
    

    View Slide

45. A
    

    View Slide

46. View Slide

47. bHU
    

    View Slide

48. "B(
    •  Wowma! MOVhı­
    • 
    GerMOVhAWSü|
    •  KCF-AWS >EArXenM
    •  Wowma! AWSü|
    • 
    ªŒ*
    
    

    View Slide

49. Wowma! RTYhĉ¦
    • 
    ġs)war`=@oWowma!+-.£&
    • 
    GerMOVh+0‘'ÜMOVh(&IJď
    Web/AP
    Oracle
    MySQL
    RoR
    Ger
    Wowma! MOVh
    Batch
    ìį¢MOVh
    ÈĭCrNr ”ÃMOVh
    ¯‘MOVh
    Web/AP
    Batch
    

    View Slide

50. Wowma! RTYh2¨³¯
    • 
    °ÖñÂ
    •  OS,32bitJava,1.6
    • 
    ĝ§*57CPUJ>½ě)8
    • 
    WSQrS+ü|
    •  2022t/'*WSQrS;z&á­7
    •  HW+Čđç>Ub
    •  FAN+Čđƅ
    • 
    ÔĔ+FAN;ˆƒ8+1ý)$&!
    

    View Slide

51. é’!,>
    MfrRTYh2ã|
    <à‚
    

    View Slide

52. MfrRTYh2®×…œ
    •  Wowma!MOVh(+•¹,API'+37¨7
    • 
    K\ó²,Chef'ù¡9&8
    •  DBWS+K@P, <)*Æ)
    •  DBMySQL'¥&8
    

    View Slide

53. Wowma! .MfrRTYh2ݔ
    9 9ü|' 2)
    
    Web/AP
    Oracle
    Wowma! MOVh
    Batch
    GerMOVh
    RoR
    HTTPS
    Batch
    

    View Slide

54. &B0.0*'
    RTYhž˜ċĂx­,0*'
    ¦¿Ô)
    

    View Slide

55. Wowma! .MfrRTYh2›î(¦¿Ô)
    Web/AP
    Oracle
    Wowma! MOVh
    Batch
    GerMOVh
    RoR
    HTTPS
    Batch
    4$1Ͱ
    
    ϑΝΠϧड৴
    
    ͜ͷαʔόʔ(*1
    
    ࣋ͬͯͳ͍
    
    • 
    GerMOVh,č¢ðą¶;Ñ&)$!
    • 
    č¢ð+„DrbpSW+³ŽäĞ1á­()7ē»‹)
    •  BatchK\*EIP%8+,QFjnV?*37!)
    

    View Slide

56. Wowma! .MfrRTYh2›î(¦¿Ô)–À“õ
    • 
    xSSHK\;ϲ +K\'`=@o37¨7
    •  S3¤â1Ñ!ĮĊ+Batch+ē»ª ñ*)85*!
    • 
    ÆĉSSH1Ñ!DrbpÙ+batchK\ÆĉSSH*‰æ&)$!
    Web/AP
    Oracle
    Batch
    GerMOVh
    HTTPS
    xSSHK\
    Þ¢AWS
    

    View Slide

57. ChefßOS‹Ã(*'2-Õ?0*'
    Chef-™3ª?,0*'
    \Ljir[0Chef(*'
    0/ó18ö=5!'
    Ó}1mmT
    
    
    
    !(Õ$!60.01%6Cm
    

    View Slide

58. `OX[|‰ •¨³„…
    

    View Slide

59. Wowma! .MfrRTYh2›î `OX[|‰ •
    •  ALB+ij´®Á(Õ$!ú$!
    •  ELB'1Cmõ&)
    • 
    顋Ģ:'1)$!
    Web/AP
    Oracle
    GerMOVh
    HTTPS
    λΠϜΞ΢τΤϥʔ
    ϩάʹ͸Կ΋Ͱ͍ͯͳ͍
    
    

    View Slide

60. Wowma! .MfrRTYh2›î `OX[|‰ •ˆÑ
    • 
    Þ¢AWS*ELBϲVPC_>¤â'GerMOVh(•
    ¹
    • 
    «Ě,¦—"ê}*]IUX|Š¦—®Á,˔
    Web/AP
    Oracle
    GerMOVh
    HTTPS
    Þ¢AWS
    

    View Slide

61. KCF – AWS
    DKGr[fmR
    

    View Slide

62. dpWM[AWSDKGr[
    •  1bqTGX'™…vþ+2>EArX
    • 
    ™…Āû(vþĀû+•¹ėß
    • 
    ™…Āû,ģ¸ñ
    • 
    vþĀû,á­ªö+¸ñ
    ™… vþ
    

    View Slide

63. DirectConnect«¸
    •  DirectConnect,Þ¢AWS*ϲ
    •  DirectConnectá­)>EArX*,ĩā@rS`B@O¤â
    • 
    À¢'8CIDR,ñ69&8+'DirectConnect¦­‘‚,ÜVPC
    Þ¢AWS
    
    

    View Slide

64. VLjmYEðpN«¸
    GurdDuty CloudTrail
    Þ¢AWS
    GurdDuty
    CloudTrail
    GurdDuty
    EC2
    EC2
    syslog
    AWSĨÏqH
    ëº
    Amazon Inspector
    Inspector
    Inspector
    Inspector
    

    View Slide

65. VLjmYEðpN3.=$5.7,>oen
    ¨³3Òݚ
    
    
    SplunkA²!,VLjmYEAÒݚ1ā¢w
    

    View Slide

66. ^FcmX[MlG\0AÐ!ćè
    

    View Slide

67. ˝AWSDKGr[-:*,>.
    •  SSH+Ĝ0ÓK\
    •  syslogK\
    • 
    ëøK\Zabbix)
    •  SES(+Ÿċ¢SMTP relayK\(Postfix)
    • 
    Drbp¢+NTPK\
    • 
    Drbp¢+DNSK\
    • 
    Drbp¢\UG>Ub¢NFSK\
    •  Linux+kLù¡¢K\FreeIPA)
    • 
    etc
    ϋΠϒϦου
    
    Ϋϥ΢υʁ
    
    
    

    View Slide

68. ˝AWSDKGr[-Jrdo2NTP/DNSQ_
    •  DB,à*‹ĥ‡¬Ä­
    • 
    ’Oracle8±,NTP/DNSK\)
    •  DirectConnect'Ķ$&8Þ¢AWS>EArX'ϲ8
    Oracle
    Oracle Enterprise Manager
    NTP Amazon Time Sync Service
    DNS(np+0 Route53(Private)
    

    View Slide

69. ˝AWSDKGr[- SMTP relayQ_
    • 
    >mXĐ,SES'ioò¹
    •  Oracle Enterprise Manager+io1SES'ò7!
    •  SMTP relayK\¤â'SESò¹
    Oracle Enterprise Manager
    SMTP relay Amazon
    SES
    SES
    Zabbix
    

    View Slide

70. Wowma! u˜2mdoFT1ñ‡
    

    View Slide

71. ‹Ä2·¦fFr[ì=Ü=
    • 
    u~zµ!‹CrN[>1w1)
    • 
    žĈ&9&8crT<1Ï$&8+,{‘
    ĵ1»Ú)ćĄ;Í6)
    

    View Slide

72. Wowma! AWSš1ÿ½ufmR
    •  Java3OS+\Nlr>Ub*58>bn+ē»,8
    • 
    +S@grH'n`=GSnrH,œÛ)
    • 
    RO+SVNù¡;Gitù¡*äĞ
    

    View Slide

73. ýùMlG\02-°”1ā¢
    • 
    ^oYWbq@+ħƒ¥
    •  Ant;¢!ƒ¥^oYJenkins*58^oY
    • 
    ĴWbq@Jenkins;ˢ!Wbq@
    •  SSHĪİÒ¥
    •  Tail&qH;ÚÌCloudWatchLogs'ÚÌ
    •  OS/MW+Ž¬)\Nlr>Ub
    • 
    -.Ğ’)1dO+Ğ’InspectorÈÍ
    

    View Slide

74. an\ ZdpF2þ‚Ÿš – ®×ڜ
    •  WebK\,3ķÓ½Ċã8
    • 
    K\ęÇ'>bn+³Ž`=@oú
    • 
    ïī‚Ĥ›
    • 
    ³Ž`=@o+ĆĖĕÅ+©
    • 
    ˜/',ă€MBo'³Ž`=@o;ϲ
    • 
    åK\'ă€MBo'³Ž`=@oϲ&!
    • 
    ޕK\'{Ĭ'³Ž`=@oϲ8©
    

    View Slide

75. §2úí30!
    †µ1«ŒbCFn2Àû
    †µ0Âz
    
    /s}
    
    
    

    View Slide

76. ®Î
    0B.çƒ'
    {¢18Í$Ê@=­>
    

    View Slide

77. Deploy
    • 
    ³Ž`=@o®Á8+'CodeDeploy(BeansTalk×)
    • 
    ă€MBo(AWS System Manager'‰æ
    Čan\¼ÛÉ
    čSSM-deploy™|
    ψÞQ_-RunCommnd™|
    ďÍ¦0warbCFn.«ŒbCFn¡¾
    

    View Slide

78. SSHăĈŝ
    ˜“,#4$(²Ø,y!
    Ĵí
    !"96Īİ*¿&ğ·8+0
    

    View Slide

79. OS/MW2Œ¤0_SkrDXd
    • 
    ƒ÷BlueGreenWbq@'‰æ
    • 
    ÿđ腆‹,n\bq+¿;Ê*Ġ
    ČELB/EC2.!VX[ª
    čDdmZdpF
    m_dp
    Ďm_dp2±¹AÏ÷ Route531,
    

    View Slide

80. £Š1
    

    View Slide

81. –‘35$ØĄˆ´¥Ć
    –‘ˆÑ!,>}Ê@*'Š1â+23
    Ȍ
    
    gFMpQaTš
    
    

    View Slide

82. æURLôº-™3ádpWM[ yĊï¶6
    • 
    kL6,1%+K^O"œÉ,Ħ½bqTGX
    • 
    £š;irVZrO*)&15)8
    n\bq
    /abc/ô– ĒÎ+URL¾¯
    /xyz/ô–
    

    View Slide

83. æodmKDBª
    •  DMS;À¢&{‘WS+¼$Ď
    • 
    91`BP1'Oraclenbp@O!6zµ8}
    •  Oracle fV^j;À¢&Vaoó²;ªĘ&DMS
    •  Aurora Serverless'APIÐĂîÝWS(,DynamoDB(
    Oracle
    Drbp
    DMS
    Aurora EC2 ELB
    Lambda
    DynamoDB API Gateway
    

    View Slide

84. '(¬AÈ0.0}
    ê')3gFMpQaTš!'@"90
    PrY]š1!'@"90
    Ϛ1©»ò!:#Wowma!AÂ='(
    
    

    View Slide

85. KDDIPgTbIq\3Öä(-0
    QaTA8*.Ù!,.Ç*,?>vA
    sĀÌw-#
    %4yą1Wowma! AÙ!,5!;
    ÁvŽÆ
    https://hrmos.co/pages/kddi-cf/jobs
    —„cpN
    https://kcf-developers.hatenablog.jp
    
    

    View Slide

86. View Slide

87. KDDIPgTbIq\3LUXA.~ø!
    2019t4€<
    <1!0=5#
    

    View Slide

88. ëü=. 5!'
    

    View Slide