Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cookpad's deployment and auto scaling

Cookpad's deployment and auto scaling

クックパッドのデプロイとオートスケール
2014.03.15 JAWS DAYS 2014 Immutable Infrastructure track
http://jawsdays2014.jaws-ug.jp/speaker/i_naruta/

Issei Naruta

March 15, 2014
Tweet

More Decks by Issei Naruta

Other Decks in Technology

Transcript

  1. ΫοΫύουͷ
    σϓϩΠͱ
    Φʔτεέʔϧ
    ΫοΫύουגࣜձࣾ੒ాҰੜ
    +"84%":4

    View Slide

  2. KBXTEBZTJOGSB

    View Slide

  3. ੒ాҰੜ
    ͳΔͨ ͍͍ͬͤ
    !NJSBLVJ
    ΫοΫύουגࣜձࣾ
    ΠϯϑϥετϥΫνϟʔ෦௕

    View Slide

  4. ࠓ೔ͷ࿩
    ΫοΫύουʹ͍ͭͯ
    ϓϩϏδϣχϯάʢαʔϏεσΟεΧόϦʣ
    ΞϓϦέʔγϣϯͷσϓϩΠ
    Φʔτεέʔϧ
    !

    View Slide

  5. ΫοΫύουʹ͍ͭͯ

    View Slide

  6. 'VMM"84
    2011೥3݄ ౦ژϦʔδϣϯΦʔϓϯ
    2011೥8݄ DC͔ΒAWS΁Ҡߦ

    View Slide

  7. .6#NP
    .14VTFST
    .DPPLJOHSFDJQFT
    ˞೥݄ݱࡏ
    !"
    #
    #

    View Slide

  8. &$JOTUBODFT
    &-#T
    4#VDLFUT
    3PVUF3FDPSET

    View Slide

  9. 3FETIJGUGPS#*
    UBCFNJSVDPN
    4/4GPS.PCJMF1VTI/PUJpDBUJPO

    View Slide

  10. 0QT
    %FWFMPQFST
    %FQMPZTEBZ

    #

    View Slide

  11. ΞϓϦέʔγϣϯίʔυͷσϓϩΠ

    View Slide

  12. $
    QVMMSFRVFTU
    ()&
    #
    EFWFMPQFS

    View Slide

  13. $
    QVMMSFRVFTU
    NFOUJPO
    ()&
    #
    EFWFMPQFS
    #
    SFWJFXFS

    View Slide

  14. $
    -(5.
    QVMMSFRVFTU
    NFOUJPO
    ()&
    #
    EFWFMPQFS
    #
    SFWJFXFS

    View Slide

  15. $
    -(5.
    QVMMSFRVFTU
    NFOUJPO
    ()&
    #
    EFWFMPQFS
    #
    SFWJFXFS
    NFSHF

    View Slide

  16. $
    -(5.
    QVMMSFRVFTU
    NFOUJPO
    ()&
    #
    EFWFMPQFS
    #
    SFWJFXFS
    $*
    +FOLJOT
    NFSHF

    View Slide

  17. $
    TVDDFTT
    -(5.
    QVMMSFRVFTU
    NFOUJPO
    ()&
    #
    EFWFMPQFS
    #
    SFWJFXFS
    $*
    +FOLJOT
    NFSHF

    View Slide

  18. $
    TVDDFTT
    -(5.
    QVMMSFRVFTU
    NFOUJPO
    EFQMPZ
    &$
    ()&
    #
    EFWFMPQFS
    #
    SFWJFXFS
    $*
    +FOLJOT
    NFSHF

    View Slide

  19. σϓϩΠϧʔϧʢҰ෦ʣ
    • CI Λύεͨ͠ϦϏδϣϯͷΈσϓϩΠͯ͠Α͍
    • σϓϩΠ͸ίʔυΛ push ͨ͠։ൃऀࣗ਎͕ߦ͏
    • Ӧۀ࣌ؒ಺ͷΈσϓϩΠՄೳ
    • σϓϩΠޙ͸։ൃऀ͕ಈ࡞֬ೝ͠ɺ

    ෆ۩߹Λݟ͚ͭͨΒ͙͢ʹϩʔϧόοΫ͢Δ

    View Slide

  20. σϓϩΠޙʹΤϥʔ΍ϨεϙϯελΠϜྼԽ͕ແ͍͜ͱΛ֬ೝ
    ϦΞϧλΠϜϞχλϦϯάπʔϧlJNPOz

    View Slide

  21. ΫοΫύουͷ
    αʔόϓϩϏδϣχϯά

    View Slide

  22. αʔϏεσΟεΧόϦ໰୊
    αʔό͕ಈతʹ૿͑ͨΓݮͬͨΓ͢Δঢ়گͰ͸

    ੩తͳ”αʔό؅ཧ୆ா”͸ӡ༻Ͱ͖ͳ͍
    !
    αʔό͕௥Ճ͞ΕͨΒݕ஌͢Δ࢓૊Έ͕ඞཁ
    →Serf, Zookeeper ౳
    %

    View Slide

  23. &$λάʹΑΔσΟεΧόϦ
    Name λά
    Role λά
    Status λά
    &

    View Slide

  24. &$ͷλάػೳʹ͍ͭͯ
    key: value ͷϖΞ

    View Slide

  25. ΫοΫύουͰͷ&$λά
    ͦͷαʔόͷଐੑ΍ঢ়ଶΛهड़
    !
    Πϯελϯε͸ࣗ෼ࣗ਎΍ɺଞΠϯελϯεͷλάΛݟͯ
    ࣗ཯ڠௐతʹಈ͘

    View Slide

  26. /BNFλά
    ͦͷΠϯελϯεͷϗετ໊
    DNSαʔό͕ࣗಈݕ஌͠ɺAϨίʔυΛ௥Ճ
    !
    ྫ:
    Name: app-001
    &

    View Slide



  27. View Slide

  28. ' /BNFEC


    View Slide

  29. ' /BNFEC
    ' /BNFEC


    View Slide

  30. ' /BNFEC
    ' /BNFEC


    1PXFS%/4

    View Slide

  31. ' /BNFEC
    ' /BNFEC


    1PXFS%/4

    (

    View Slide

  32. ' /BNFEC
    ' /BNFEC


    1PXFS%/4

    (
    EC"
    EC"
    %/4αʔό͕ఆظతʹશ/BNFλάΛ૸ࠪ͠ɺ
    %/4ϨίʔυΛߋ৽

    View Slide

  33. 3PMFλά
    Πϯελϯεͷ໾ׂΛهड़
    !
    ྫ:
    Role: app
    &

    View Slide

  34. QVQQFU
    NBTUFS )

    View Slide

  35. ' 3PMFBQQ
    QVQQFU
    NBTUFS )

    View Slide

  36. ' 3PMFBQQ
    BQQMZlBQQz
    QVQQFU
    NBTUFS )

    View Slide

  37. ' 3PMFBQQ
    ' 3PMFEC
    BQQMZlBQQz
    QVQQFU
    NBTUFS )

    View Slide

  38. ' 3PMFBQQ
    ' 3PMFEC
    BQQMZlBQQz
    QVQQFU
    NBTUFS
    BQQMZlECz
    )

    View Slide

  39. ' 3PMFBQQ
    ' 3PMFEC
    BQQMZlBQQz
    QVQQFU
    NBTUFS
    BQQMZlECz
    QVQQFUNBTUFS͸ద༻ઌͷ3PMFλάΛݟͯ

    ద༻͢ΔΫϥεΛࣗಈͰܾఆ
    )

    View Slide

  40. 4UBUVTλά
    ͦͷΠϯελϯε͕όϥϯαʔʹ

    ௥Ճ͍͍ͯ͠΋ͷ͔Ͳ͏͔Λఆٛ
    !
    ྫ:
    Status: working
    &

    View Slide

  41. 3PMFBQQ
    *
    IBQSPYZDGH
    BQQIBQSPYZ
    3PMFBQQ

    View Slide

  42. 3PMFBQQ
    ' 4UBUVTXPSLJOH
    *
    IBQSPYZDGH
    BQQIBQSPYZ
    3PMFBQQ

    View Slide

  43. 3PMFBQQ
    ' 4UBUVTXPSLJOH

    *
    IBQSPYZDGH
    BQQIBQSPYZ
    3PMFBQQ

    View Slide

  44. 3PMFBQQ
    ' 4UBUVTXPSLJOH

    *
    IBQSPYZDGH
    BQQIBQSPYZ
    3PMFBQQ

    View Slide

  45. 3PMFBQQ
    ' 4UBUVTXPSLJOH

    *
    IBQSPYZDGH
    BQQIBQSPYZ
    3PMFBQQ
    ' 4UBUVTXPSLJOH

    View Slide

  46. 3PMFBQQ
    ' 4UBUVTXPSLJOH

    *
    IBQSPYZDGH
    BQQIBQSPYZ
    3PMFBQQ
    ' 4UBUVTXPSLJOH

    View Slide

  47. 3PMFBQQ
    ' 4UBUVTXPSLJOH

    *
    IBQSPYZDGH
    BQQIBQSPYZ
    3PMFBQQ
    ' 4UBUVTXPSLJOH

    View Slide

  48. 3PMFBQQ
    ' 4UBUVTXPSLJOH

    *
    IBQSPYZDGH
    BQQIBQSPYZ
    3PMFBQQ
    ' 4UBUVTXPSLJOH
    XPSLJOHঢ়ଶͷΠϯελϯεΛࣗಈͰαʔϏεΠϯ

    View Slide

  49. '
    ͳͥλάͰશ෦΍Δͷ͔
    Zookeeper, Serf ౳ͷಛผͳϛυϧ΢ΣΞΛඞཁͱͤͣʹ

    αʔϏεσΟεΧόϦ͕Մೳ
    !
    ඇৗʹγϯϓϧ͔ͭϓϩάϥϚϒϧ

    View Slide

  50. λάͷϋϚϦͲ͜Ζ
    DescribeTags ͷ API Limit
    1 Ϧιʔεʹλά͸10ݸ·Ͱ
    %
    -JNJU
    DBDIF
    (

    View Slide

  51. πʔϧ
    λάϑϨϯυϦʔͳ؅ཧπʔϧ͕ඞཁ
    ElasticFox-ec2tag

    View Slide

  52. Φʔτεέʔϧ

    View Slide

  53. τϥϑΟοΫ
    &$Πϯελϯε਺
    ࣌ؒ
    τϥϑΟοΫͷมԽʹԠͯ͡Πϯελϯε਺Λ૿ݮ

    View Slide

  54. ͳͥΦʔτεέʔϧͳͷ͔
    '

    View Slide

  55. *OTUBODFIPVS
    EC2ͷ࠷খ՝ۚ୯Ґ
    ىಈதͷΠϯελϯε਺ x ىಈ࣌ؒʢ1࣌ؒ୯Ґʣ
    ΦϯσϚϯυɺ౦ژϦʔδϣϯ೥݄೔ݱࡏIUUQBXTBNB[PODPNKQFDQSJDJOH

    View Slide

  56. &$Πϯελϯε਺
    I
    Πϯελϯε࣌
    ( JOTUBODFIPVS
    Φʔτεέʔϧ
    ͳ͠
    Φʔτεέʔϧ
    ͋Γ

    View Slide

  57. "NB[PO"VUP4DBMJOH

    View Slide

  58. $MPVE8BUDI
    "VUP
    4DBMJOH
    "VUP4DBMJOH(SPVQ
    ".*

    View Slide

  59. ͨͱ͑͹
    "NB[PO"VUP4DBMJOH
    Λ΍ΊΔ

    View Slide

  60. ͳͥ"NB[PO"VUP4DBMJOHͰ͸ͳ͍ͷ͔
    CloudWatch 1෼ͷน
    Graceful Terminate ໰୊
    ELB Ͱ͸ͳ͘ haproxy Λ࢖͍͍ͨ
    σϓϩΠͱͷഉଞ੍ޚΛεϜʔζʹߦ͏ͨΊ

    View Slide

  61. $MPVE8BUDIͷੑೳ΁ͷෆຬ
    ෼ͷนɹϝτϦΫε͕औಘͰ͖Δͷ͸࠷଎Ͱ΋1෼ޙ
    +

    View Slide

  62. (SBDFGVM5FSNJOBUF໰୊
    desired_capacity ͕ݮΔͱ͖ɺ

    Πϯελϯε͸໰౴ແ༻ͰTerminate ͞Εͯ͠·͏
    ҆શͳαʔϏεΞ΢τΛߦ͏ʹ͸޻෉͕ඞཁ

    View Slide

  63. *OUFSOBM&-#Ͱ͸ͳ͘IBQSPYZΛ࢖͍͍ͨ
    ELB ͷ IP ΞυϨεͷಈతͳมԽʹ

    ϛυϧ΢ΣΞ͕ରԠ͖͠Εͳ͍
    !
    ϒϥοΫϘοΫε౓͕ߴ͘ɺ໰୊͕೺Ѳͮ͠Β͍
    ˞&-#ͳ͠Ͱ΋"NB[PO"VUP4DBMJOH͸࢖͑·͢

    View Slide

  64. "NB[PO"VUP4DBMJOH
    WT
    EFQMPZTEBZ


    View Slide

  65. W W W
    W
    #

    View Slide

  66. W W W
    W
    W

    #

    View Slide

  67. W W W
    W
    W

    #

    View Slide

  68. W W W
    W
    W

    W W W
    #

    View Slide

  69. W W W W W W
    W
    W

    W W W
    #

    View Slide

  70. W W W W W W
    W W W
    ݹ͍όʔδϣϯͷΠϯελϯε͕࡞ΒΕͯ͠·͏

    View Slide

  71. "NB[PO"VUP4DBMJOHͰഉଞ੍ޚ
    σϓϩΠத͸ AutoScaling ΛࢭΊɺ
    AutoScaling த͸σϓϩΠΛېࢭ͍ͨ͠
    !
    AutoScaling ͕ΠϯελϯεΛ૿΍͢ͱ͖ɾݮΒ͢ͱ͖ʹ
    ॲཧΛׂΓ͜·ͤΔඞཁ͕͋Δ
    →ࠔ೉

    View Slide

  72. CloudWatch 1෼ͷน
    Graceful Terminate ໰୊
    ELB Ͱ͸ͳ͘ haproxy Λ࢖͍͍ͨ
    σϓϩΠͱͷഉଞ੍ޚΛεϜʔζʹߦ͍͍ͨ
    ࣗ෼Ͱશ෦ॻ͍ͯ΋͋·Γखؒ͸มΘΒͳͦ͏
    "NB[PO"VUP4DBMJOHͰղܾෆՄೳͰ͸ͳ͍͚Ͳɺ
    ଟ͘ͷίʔυΛॻ͖ɺΧελϚΠζ͢Δඞཁ͕͋Δ
    +
    +

    View Slide

  73. $PPLQBE"VUP4DBMJOHͷ࣮૷
    ,

    View Slide

  74. JNPO "4
    #
    W
    W
    EFWFMPQFS
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH

    View Slide

  75. JNPO "4
    #

    W
    W
    EFWFMPQFS
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH

    View Slide

  76. W
    W
    JNPO "4
    #

    W
    W
    EFWFMPQFS
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH

    View Slide

  77. W
    W
    JNPO "4
    #
    W
    W
    EFWFMPQFS
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH

    View Slide

  78. W
    W
    JNPO "4
    #
    W
    W
    EFWFMPQFS
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH

    View Slide

  79. W
    W
    JNPO "4
    #
    W
    W
    EFWFMPQFS
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH

    View Slide

  80. W
    W
    JNPO "4
    #
    W
    W
    EFWFMPQFS
    W
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH

    View Slide

  81. W
    W
    JNPO "4
    #
    W
    W
    EFWFMPQFS
    W
    W
    W
    W
    W
    W
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH

    View Slide

  82. W
    W
    JNPO "4
    #
    W
    W
    EFWFMPQFS
    W
    W
    W
    W
    W
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH

    View Slide

  83. Φʔτεέʔϧͷ͋Δ฻Β͠

    View Slide

  84. ͱ͋ΔΠϯελϯεͷ
    ୆਺

    Φʔτεέʔϧ

    View Slide

  85. ΦʔτεέʔϧʹΑΔσϓϩΠϩοΫத
    σϓϩΠத

    View Slide

  86. (
    (
    ΦʔτεέʔϧʹΑΔσϓϩΠϩοΫத
    σϓϩΠத

    View Slide

  87. Φʔτεέʔϧத
    σϓϩΠத

    View Slide

  88. View Slide

  89. View Slide

  90. View Slide

  91. - ೥Ҏ্ӡ༻Ͱ͖ͯΔ

    View Slide

  92. ೥ͷӡ༻࣮੷Λ௨ͯ͠
    ۤ࿑ͨ͠఺ɺ޻෉ɺؾ͍ͮͨ͜ͱ

    View Slide

  93. ϩοΫ࣌ؒɾճ਺Λ࠷খʹ͢Δ
    σϓϩΠΛϩοΫʹՁ஋Λಧ͚ΔνϟϯεΛݮΒ͢
    .

    View Slide

  94. ୆਺

    View Slide

  95. ୆਺
    ۀ຿࣌ؒ σϓϩΠՄೳ࣌ؒଳ

    View Slide

  96. ୆਺
    ۀ຿࣌ؒ σϓϩΠՄೳ࣌ؒଳ

    αʔϏε"ϩοΫճ
    αʔϏε#ϩοΫճ

    View Slide

  97. ୆਺
    Ұ౓ʹ૿ݮ͢Δ୆਺Λௐ੔͢Δ͜ͱͰ
    ۀ຿࣌ؒதͷσϓϩΠϩοΫճ਺Λ࠷খݶʹ
    ۀ຿࣌ؒ σϓϩΠՄೳ࣌ؒଳ

    αʔϏε"ϩοΫճ
    αʔϏε#ϩοΫճ

    View Slide

  98. ୆਺
    Ұ౓ʹ૿ݮ͢Δ୆਺Λௐ੔͢Δ͜ͱͰ
    ۀ຿࣌ؒதͷσϓϩΠϩοΫճ਺Λ࠷খݶʹ
    ۀ຿࣌ؒ σϓϩΠՄೳ࣌ؒଳ

    αʔϏε"ϩοΫճ
    αʔϏε#ϩοΫճ
    ۀ຿࣌ؒ֎ͷͨΊɺ
    ݮΒ͢ͱ͖͸খࠁΈʹ

    View Slide

  99. ςετΛॻ͘
    Φʔτεέʔϧ͸αʔόΛམͱ͢ϓϩάϥϜ
    /

    View Slide

  100. ఆظతʹઃఆΛݟ௚͢
    ์͓͍ͬͯͯ΋εέʔϧ͢ΔΑ͏ʹݟ͑Δ͕…
    !
    ΞϓϦαʔόͷ୆਺૿ˠόοΫΤϯυ΁ͷίωΫγϣϯ૿
    ΞϓϦέʔγϣϯͷύϑΥʔϚϯε΋ৗʹมԽ
    0

    View Slide

  101. TUBUJDOPEFTEZOBNJDOPEFTߏ੒
    ৗʹཱͪଓ͚ͯΔΠϯελϯε͕͍ͨ΄͏͕
    ͳΜ͔ͩΜͩͰศར
    TUBUJD
    EZOBNJD

    View Slide

  102. View Slide

  103. TUBUJDOPEFT EZOBNJDOPEFT
    Φʔτεέʔϧର৅֎
    ୆਺͸ݻఆ
    TIVUEPXO͞Εͳ͍
    Φʔτεέʔϧର৅
    ୆਺͸Մม
    CBMBODFS

    View Slide

  104. αʔό໊Ͳ͏͠Α͏໰୊
    dynamic nodes Ͱ͋Ζ͏ͱɺ

    ݁ہ hostname ͸͋ͬͨ΄͏͕؅ཧ͠΍͍͢
    !
    10.0.123.45 ͩͬͨΒ app-10-0-123-45 ͱ͍͏

    Name λά͕ͭ͘Α͏ʹ͍ͯ͠Δ
    &

    View Slide

  105. Φʔτεέʔϧͷ෭࢈෺
    ΦʔτεέʔϧΛ૊Ήͱɺ Immutable ͳ

    Golden Image ͕࡞ΕΔ
    !
    ͍ͭͰ΋ҰൃͰ࠷৽ͷαʔό্ཱ͕ͪ͛ΒΕΔ͜ͱ͕

    อূ͞ΕΔ +

    View Slide

  106. ·ͱΊ

    View Slide

  107. EFQMPZTEBZWT
    "VUP4DBMJOH
    10 deploys/day ͳ؀ڥͰ΋Φʔτεέʔϧ͸Մೳ
    ίετͱ։ൃ଎౓ͷτϨʔυΦϑΛߟྀͨ͠νϡʔχϯά

    View Slide

  108. "VUP4DBMJOHBT
    *NNVUBCMF*OGSBTUSVDUVSF
    Immutable Infrastructure ͸ະདྷͷ࿩͡Όͳ͘ɺ

    ΫοΫύουͰ͸ΦʔτεέʔϧʹΑͬͯ͢Ͱʹ
    ݱ࣮ʹͳͬͯΔ

    View Slide

  109. 2"
    '

    View Slide

  110. DSFEJUT
    http://aws.amazon.com/jp/architecture/icons/
    http://www.entypo.com/
    http://fortawesome.github.io/Font-Awesome/
    http://www.flickr.com/photos/sora_h/8916379094/

    View Slide