$30 off During Our Annual Pro Sale. View Details »

サービスダウンから生まれたSWATチームが手がけるクラウド移行への道 / JAWS DAYS 2019 Hybrid Cloud Session

サービスダウンから生まれたSWATチームが手がけるクラウド移行への道 / JAWS DAYS 2019 Hybrid Cloud Session

KDDIコマースフォワードではECモールWowma!を運営しています。

前身サービスから引き継いできたシステムは、日常的にDBのCPU使用率が90%近く、限界が近づいていました。そんな中、前代未聞の高負荷イベントの開催が決定。

15年ものの秘伝のタレ満載のシステムに対して

・DMSでのRDS(Oracle)へのDB移行
・S3とCloudFrontを利用したLPサイトのキャッシュ化
・既存APIのCloudFrontによるキャッシュ化 など

地道な改善を行うも、あっけなく落ちてしまったシステム。そこから、負荷対策チームSWATが誕生しました。

終わりの見えない負荷対策に始まり、AWSへの移行を行いながらもオンプレミスにオラクルDBを構築した経緯、数年後のシステム全体リプレイスを見据えながらマルチクラウド移行を目指すエンジニアの泥臭い戦いについてお話します。

Wowma! ”https://wowma.jp

Takuya Tatsumi

February 23, 2019
Tweet

Other Decks in Technology

Transcript

  1. A w K
    02 1 A
    w
    3GD 63 D ) (0
    ) (0 ) )
    66:
    b

    View Slide

  2. View Slide



  3. b GMJ a
    66:
    e a
    3GD ) w
    Solutions Architect - Associate

    View Slide

  4. IS P

    View Slide

  5. IS P

    S P

    c
    S

    View Slide

  6. A A
    A b

    View Slide

  7. A A A
    75 GT RI

    View Slide

  8. 2 3
    S ✋

    View Slide

  9. A
    S ✋

    View Slide

  10. 2 3
      6M 3 s 66:
      66:
      GT RI
    6M 3 t
    1999೥ 2016೥

    View Slide

  11.   t a
      s •

    GMJ x )
    D15MSW D , 4
    , s
    IYI( ) s t w a
      b A A Sa e a

    View Slide

  12. b
    bK t Sm
    ) (. s t r t e

    w t A a k d

    2 4 S k d

    View Slide

  13. T m W

    View Slide

  14. or S A d
    o
    t s ss
    s 9
    ×

    View Slide

  15. w B A
    s

    View Slide

  16. T

    View Slide

  17. S i

    s t s w t zw
    w

    t t o


    t o aaa

    View Slide

  18. i

    D t


    u t
    w A

    View Slide

  19. w
    20 w w

    View Slide

  20. w
    6OUMKW5TSSMKW 3GD

    View Slide

  21. 3GD
    3GDn n3GD

    View Slide

  22. w

    64 t 5AF / l

    View Slide


  23. )

    w

    A
    C6D UIKQM
    75) UIKQM or

    View Slide

  24. A
    C6D UIKQM

    LJ U / QIUNMt

    UIKQM

    D D D DE7 C6D

    View Slide

  25. b

    View Slide

  26. C
    A

    View Slide

  27. D w
    Kao
    b K a

    View Slide


  28. D 5QT L8UTSW A

    View Slide


  29. 3A: 5QT L8UTSW

    View Slide


  30. DB
    UIKQM DWIW AIKPs DB
    s

    View Slide


  31. GMJ 3A 3A:


    tGMJ
    DA A5 GMJ

    View Slide


  32. 6 D i
    64 t v 6 D s
    u s a

    View Slide

  33. G:
    A
    or w A w
    to z t uw v
    P m ba P b
    "
    D7g
    #
    VS

    View Slide

  34. 02 1

    View Slide

  35. a K

    View Slide

  36. a K
    GMJ

    View Slide

  37. P 02 1 A

    t t
    t t

    s )
    w

    A a
    02 1

    View Slide

  38. A S
    A
    A
    A
    2 4 A A w

    View Slide

  39. A
    A

    View Slide

  40. A

    t

    64 )
    w u


    m u

    View Slide

  41. C6D UIKQM
    bbb

    75) UIKQM
    a
    C35t3GD a
    to t

    UIKQM
    C6D UIKQM

    LJ U / QIUNMt

    UIKQM

    D D D DE7
    t r

    View Slide

  42. A A
    S A A

    3GD 6OUMKW5TSSMKW G3
    u

    95A 3 UM
    u
    ϚϧνΫϥ΢υ؀ڥʹҰาۙͮ͘ʂ

    View Slide

  43. wo
    6OUMKW5TSSMKW

    View Slide

  44. A
    K A
    ) (/
    ) (/ /
    b
    s
    64

    View Slide

  45. A

    View Slide

  46. View Slide

  47. bHU

    View Slide

  48. "B(
    •  Wowma! MOVhı­
    • 
    GerMOVhAWSü|
    •  KCF-AWS >EArXenM
    •  Wowma! AWSü|
    • 
    ªŒ*

    View Slide

  49. Wowma! RTYhĉ¦
    • 
    ġs)war`=@oWowma!+-.£&
    • 
    GerMOVh+0‘'ÜMOVh(&IJď
    Web/AP
    Oracle
    MySQL
    RoR
    Ger
    Wowma! MOVh
    Batch
    ìį¢MOVh
    ÈĭCrNr ”ÃMOVh
    ¯‘MOVh
    Web/AP
    Batch

    View Slide

  50. Wowma! RTYh2¨³¯
    • 
    °ÖñÂ
    •  OS,32bitJava,1.6
    • 
    ĝ§*57CPUJ>½ě)8
    • 
    WSQrS+ü|
    •  2022t/'*WSQrS;z&á­7
    •  HW+Čđç>Ub
    •  FAN+Čđƅ
    • 
    ÔĔ+FAN;ˆƒ8+1ý)$&!

    View Slide

  51. é’!,>
    MfrRTYh2ã|
    <à‚

    View Slide

  52. MfrRTYh2®×…œ
    •  Wowma!MOVh(+•¹,API'+37¨7
    • 
    K\ó²,Chef'ù¡9&8
    •  DBWS+K@P, <)*Æ)
    •  DBMySQL'¥&8

    View Slide

  53. Wowma! .MfrRTYh2ݔ
    9 9ü|' 2)

    Web/AP
    Oracle
    Wowma! MOVh
    Batch
    GerMOVh
    RoR
    HTTPS
    Batch

    View Slide

  54. &B0.0*'
    RTYhž˜ċĂx­,0*'
    ¦¿Ô)

    View Slide

  55. Wowma! .MfrRTYh2›î(¦¿Ô)
    Web/AP
    Oracle
    Wowma! MOVh
    Batch
    GerMOVh
    RoR
    HTTPS
    Batch
    4$1Ͱ

    ϑΝΠϧड৴

    ͜ͷαʔόʔ(*1

    ࣋ͬͯͳ͍

    • 
    GerMOVh,č¢ðą¶;Ñ&)$!
    • 
    č¢ð+„DrbpSW+³ŽäĞ1á­()7ē»‹)
    •  BatchK\*EIP%8+,QFjnV?*37!)

    View Slide

  56. Wowma! .MfrRTYh2›î(¦¿Ô)–À“õ
    • 
    xSSHK\;ϲ +K\'`=@o37¨7
    •  S3¤â1Ñ!ĮĊ+Batch+ē»ª ñ*)85*!
    • 
    ÆĉSSH1Ñ!DrbpÙ+batchK\ÆĉSSH*‰æ&)$!
    Web/AP
    Oracle
    Batch
    GerMOVh
    HTTPS
    xSSHK\
    Þ¢AWS

    View Slide

  57. ChefßOS‹Ã(*'2-Õ?0*'
    Chef-™3ª?,0*'
    \Ljir[0Chef(*'
    0/ó18ö=5!'
    Ó}1mmT



    !(Õ$!60.01%6Cm

    View Slide

  58. `OX[|‰ •¨³„…

    View Slide

  59. Wowma! .MfrRTYh2›î `OX[|‰ •
    •  ALB+ij´®Á(Õ$!ú$!
    •  ELB'1Cmõ&)
    • 
    顋Ģ:'1)$!
    Web/AP
    Oracle
    GerMOVh
    HTTPS
    λΠϜΞ΢τΤϥʔ
    ϩάʹ͸Կ΋Ͱ͍ͯͳ͍

    View Slide

  60. Wowma! .MfrRTYh2›î `OX[|‰ •ˆÑ
    • 
    Þ¢AWS*ELBϲVPC_>¤â'GerMOVh(•
    ¹
    • 
    «Ě,¦—"ê}*]IUX|Š¦—®Á,˔
    Web/AP
    Oracle
    GerMOVh
    HTTPS
    Þ¢AWS

    View Slide

  61. KCF – AWS
    DKGr[fmR

    View Slide

  62. dpWM[AWSDKGr[
    •  1bqTGX'™…vþ+2>EArX
    • 
    ™…Āû(vþĀû+•¹ėß
    • 
    ™…Āû,ģ¸ñ
    • 
    vþĀû,á­ªö+¸ñ
    ™… vþ

    View Slide

  63. DirectConnect«¸
    •  DirectConnect,Þ¢AWS*ϲ
    •  DirectConnectá­)>EArX*,ĩā@rS`B@O¤â
    • 
    À¢'8CIDR,ñ69&8+'DirectConnect¦­‘‚,ÜVPC
    Þ¢AWS

    View Slide

  64. VLjmYEðpN«¸
    GurdDuty CloudTrail
    Þ¢AWS
    GurdDuty
    CloudTrail
    GurdDuty
    EC2
    EC2
    syslog
    AWSĨÏqH
    ëº
    Amazon Inspector
    Inspector
    Inspector
    Inspector

    View Slide

  65. VLjmYEðpN3.=$5.7,>oen
    ¨³3Òݚ


    SplunkA²!,VLjmYEAÒݚ1ā¢w

    View Slide

  66. ^FcmX[MlG\0AÐ!ćè

    View Slide

  67. ˝AWSDKGr[-:*,>.
    •  SSH+Ĝ0ÓK\
    •  syslogK\
    • 
    ëøK\Zabbix)
    •  SES(+Ÿċ¢SMTP relayK\(Postfix)
    • 
    Drbp¢+NTPK\
    • 
    Drbp¢+DNSK\
    • 
    Drbp¢\UG>Ub¢NFSK\
    •  Linux+kLù¡¢K\FreeIPA)
    • 
    etc
    ϋΠϒϦου

    Ϋϥ΢υʁ


    View Slide

  68. ˝AWSDKGr[-Jrdo2NTP/DNSQ_
    •  DB,à*‹ĥ‡¬Ä­
    • 
    ’Oracle8±,NTP/DNSK\)
    •  DirectConnect'Ķ$&8Þ¢AWS>EArX'ϲ8
    Oracle
    Oracle Enterprise Manager
    NTP Amazon Time Sync Service
    DNS(np+0 Route53(Private)

    View Slide

  69. ˝AWSDKGr[- SMTP relayQ_
    • 
    >mXĐ,SES'ioò¹
    •  Oracle Enterprise Manager+io1SES'ò7!
    •  SMTP relayK\¤â'SESò¹
    Oracle Enterprise Manager
    SMTP relay Amazon
    SES
    SES
    Zabbix

    View Slide

  70. Wowma! u˜2mdoFT1ñ‡

    View Slide

  71. ‹Ä2·¦fFr[ì=Ü=
    • 
    u~zµ!‹CrN[>1w1)
    • 
    žĈ&9&8crT<1Ï$&8+,{‘
    ĵ1»Ú)ćĄ;Í6)

    View Slide

  72. Wowma! AWSš1ÿ½ufmR
    •  Java3OS+\Nlr>Ub*58>bn+ē»,8
    • 
    +S@grH'n`=GSnrH,œÛ)
    • 
    RO+SVNù¡;Gitù¡*äĞ

    View Slide

  73. ýùMlG\02-°”1ā¢
    • 
    ^oYWbq@+ħƒ¥
    •  Ant;¢!ƒ¥^oYJenkins*58^oY
    • 
    ĴWbq@Jenkins;ˢ!Wbq@
    •  SSHĪİÒ¥
    •  Tail&qH;ÚÌCloudWatchLogs'ÚÌ
    •  OS/MW+Ž¬)\Nlr>Ub
    • 
    -.Ğ’)1dO+Ğ’InspectorÈÍ

    View Slide

  74. an\ ZdpF2þ‚Ÿš – ®×ڜ
    •  WebK\,3ķÓ½Ċã8
    • 
    K\ęÇ'>bn+³Ž`=@oú
    • 
    ïī‚Ĥ›
    • 
    ³Ž`=@o+ĆĖĕÅ+©
    • 
    ˜/',ă€MBo'³Ž`=@o;ϲ
    • 
    åK\'ă€MBo'³Ž`=@oϲ&!
    • 
    ޕK\'{Ĭ'³Ž`=@oϲ8©

    View Slide

  75. §2úí30!
    †µ1«ŒbCFn2Àû
    †µ0Âz

    /s}


    View Slide

  76. ®Î
    0B.çƒ'
    {¢18Í$Ê@=­>

    View Slide

  77. Deploy
    • 
    ³Ž`=@o®Á8+'CodeDeploy(BeansTalk×)
    • 
    ă€MBo(AWS System Manager'‰æ
    Čan\¼ÛÉ
    čSSM-deploy™|
    ψÞQ_-RunCommnd™|
    ďÍ¦0warbCFn.«ŒbCFn¡¾

    View Slide

  78. SSHăĈŝ
    ˜“,#4$(²Ø,y!
    Ĵí
    !"96Īİ*¿&ğ·8+0

    View Slide

  79. OS/MW2Œ¤0_SkrDXd
    • 
    ƒ÷BlueGreenWbq@'‰æ
    • 
    ÿđ腆‹,n\bq+¿;Ê*Ġ
    ČELB/EC2.!VX[ª
    čDdmZdpF
    m_dp
    Ďm_dp2±¹AÏ÷ Route531,

    View Slide

  80. £Š1

    View Slide

  81. –‘35$ØĄˆ´¥Ć
    –‘ˆÑ!,>}Ê@*'Š1â+23
    Ȍ

    gFMpQaTš

    View Slide

  82. æURLôº-™3ádpWM[ yĊï¶6
    • 
    kL6,1%+K^O"œÉ,Ħ½bqTGX
    • 
    £š;irVZrO*)&15)8
    n\bq
    /abc/ô– ĒÎ+URL¾¯
    /xyz/ô–

    View Slide

  83. æodmKDBª
    •  DMS;À¢&{‘WS+¼$Ď
    • 
    91`BP1'Oraclenbp@O!6zµ8}
    •  Oracle fV^j;À¢&Vaoó²;ªĘ&DMS
    •  Aurora Serverless'APIÐĂîÝWS(,DynamoDB(
    Oracle
    Drbp
    DMS
    Aurora EC2 ELB
    Lambda
    DynamoDB API Gateway

    View Slide

  84. '(¬AÈ0.0}
    ê')3gFMpQaTš!'@"90
    PrY]š1!'@"90
    Ϛ1©»ò!:#Wowma!AÂ='(

    View Slide

  85. KDDIPgTbIq\3Öä(-0
    QaTA8*.Ù!,.Ç*,?>vA
    sĀÌw-#
    %4yą1Wowma! AÙ!,5!;
    ÁvŽÆ
    https://hrmos.co/pages/kddi-cf/jobs
    —„cpN
    https://kcf-developers.hatenablog.jp

    View Slide

  86. View Slide

  87. KDDIPgTbIq\3LUXA.~ø!
    2019t4€<
    <1!0=5#

    View Slide

  88. ëü=. 5!'

    View Slide