Cookpad's deployment and auto scaling

クックパッドのデプロイとオートスケール
2014.03.15 JAWS DAYS 2014 Immutable Infrastructure track
http://jawsdays2014.jaws-ug.jp/speaker/i_naruta/

Transcript

1. ΫοΫύουͷ
   σϓϩΠͱ
   Φʔτεέʔϧ
   ΫοΫύουגࣜձࣾ੒ాҰੜ
   +"84%":4
   

   View Slide

2. KBXTEBZTJOGSB
   

   View Slide

3. ੒ాҰੜ
   ͳΔͨ ͍͍ͬͤ
   !NJSBLVJ
   ΫοΫύουגࣜձࣾ
   ΠϯϑϥετϥΫνϟʔ෦௕
   

   View Slide

4. ࠓ೔ͷ࿩
   ΫοΫύουʹ͍ͭͯ
   ϓϩϏδϣχϯάʢαʔϏεσΟεΧόϦʣ
   ΞϓϦέʔγϣϯͷσϓϩΠ
   Φʔτεέʔϧ
   !
   

   View Slide

5. ΫοΫύουʹ͍ͭͯ
   

   View Slide

6. 'VMM"84
   2011೥3݄ ౦ژϦʔδϣϯΦʔϓϯ
   2011೥8݄ DC͔ΒAWS΁Ҡߦ
   ✈
   

   View Slide

7. .6#NP
   .14VTFST
   .DPPLJOHSFDJQFT
   ˞೥݄ݱࡏ
   !"
   #
   #
   

   View Slide

8. &$JOTUBODFT
   &-#T
   4#VDLFUT
   3PVUF3FDPSET
   

   View Slide

9. 3FETIJGUGPS#*
   UBCFNJSVDPN
   4/4GPS.PCJMF1VTI/PUJpDBUJPO
   

   View Slide

10. 0QT
    %FWFMPQFST
    %FQMPZTEBZ
    ⚒
    #
    

    View Slide

11. ΞϓϦέʔγϣϯίʔυͷσϓϩΠ
    
    

    View Slide

12. $
    QVMMSFRVFTU
    ()&
    #
    EFWFMPQFS
    

    View Slide

13. $
    QVMMSFRVFTU
    NFOUJPO
    ()&
    #
    EFWFMPQFS
    #
    SFWJFXFS
    

    View Slide

14. $
    -(5.
    QVMMSFRVFTU
    NFOUJPO
    ()&
    #
    EFWFMPQFS
    #
    SFWJFXFS
    

    View Slide

15. $
    -(5.
    QVMMSFRVFTU
    NFOUJPO
    ()&
    #
    EFWFMPQFS
    #
    SFWJFXFS
    NFSHF
    

    View Slide

16. $
    -(5.
    QVMMSFRVFTU
    NFOUJPO
    ()&
    #
    EFWFMPQFS
    #
    SFWJFXFS
    $*
    +FOLJOT
    NFSHF
    

    View Slide

17. $
    TVDDFTT
    -(5.
    QVMMSFRVFTU
    NFOUJPO
    ()&
    #
    EFWFMPQFS
    #
    SFWJFXFS
    $*
    +FOLJOT
    NFSHF
    

    View Slide

18. $
    TVDDFTT
    -(5.
    QVMMSFRVFTU
    NFOUJPO
    EFQMPZ
    &$
    ()&
    #
    EFWFMPQFS
    #
    SFWJFXFS
    $*
    +FOLJOT
    NFSHF
    

    View Slide

19. σϓϩΠϧʔϧʢҰ෦ʣ
    • CI Λύεͨ͠ϦϏδϣϯͷΈσϓϩΠͯ͠Α͍
    • σϓϩΠ͸ίʔυΛ push ͨ͠։ൃऀࣗ਎͕ߦ͏
    • Ӧۀ࣌ؒ಺ͷΈσϓϩΠՄೳ
    • σϓϩΠޙ͸։ൃऀ͕ಈ࡞֬ೝ͠ɺ

    ෆ۩߹Λݟ͚ͭͨΒ͙͢ʹϩʔϧόοΫ͢Δ
    
    

    View Slide

20. σϓϩΠޙʹΤϥʔ΍ϨεϙϯελΠϜྼԽ͕ແ͍͜ͱΛ֬ೝ
    ϦΞϧλΠϜϞχλϦϯάπʔϧlJNPOz
    

    View Slide

21. ΫοΫύουͷ
    αʔόϓϩϏδϣχϯά
    

    View Slide

22. αʔϏεσΟεΧόϦ໰୊
    αʔό͕ಈతʹ૿͑ͨΓݮͬͨΓ͢Δঢ়گͰ͸

    ੩తͳ”αʔό؅ཧ୆ா”͸ӡ༻Ͱ͖ͳ͍
    !
    αʔό͕௥Ճ͞ΕͨΒݕ஌͢Δ࢓૊Έ͕ඞཁ
    →Serf, Zookeeper ౳
    %
    

    View Slide

23. &$λάʹΑΔσΟεΧόϦ
    Name λά
    Role λά
    Status λά
    &
    

    View Slide

24. &$ͷλάػೳʹ͍ͭͯ
    key: value ͷϖΞ
    

    View Slide

25. ΫοΫύουͰͷ&$λά
    ͦͷαʔόͷଐੑ΍ঢ়ଶΛهड़
    !
    Πϯελϯε͸ࣗ෼ࣗ਎΍ɺଞΠϯελϯεͷλάΛݟͯ
    ࣗ཯ڠௐతʹಈ͘
    

    View Slide

26. /BNFλά
    ͦͷΠϯελϯεͷϗετ໊
    DNSαʔό͕ࣗಈݕ஌͠ɺAϨίʔυΛ௥Ճ
    !
    ྫ:
    Name: app-001
    &
    

    View Slide

27. 
    
    

    View Slide

28. ' /BNFEC
    
    
    

    View Slide

29. ' /BNFEC
    ' /BNFEC
    
    
    

    View Slide

30. ' /BNFEC
    ' /BNFEC
    
    
    1PXFS%/4
    

    View Slide

31. ' /BNFEC
    ' /BNFEC
    
    
    1PXFS%/4
    ⟲
    (
    

    View Slide

32. ' /BNFEC
    ' /BNFEC
    
    
    1PXFS%/4
    ⟲
    (
    EC"
    EC"
    %/4αʔό͕ఆظతʹશ/BNFλάΛ૸ࠪ͠ɺ
    %/4ϨίʔυΛߋ৽
    

    View Slide

33. 3PMFλά
    Πϯελϯεͷ໾ׂΛهड़
    !
    ྫ:
    Role: app
    &
    

    View Slide

34. QVQQFU
    NBTUFS )
    

    View Slide

35. ' 3PMFBQQ
    QVQQFU
    NBTUFS )
    

    View Slide

36. ' 3PMFBQQ
    BQQMZlBQQz
    QVQQFU
    NBTUFS )
    

    View Slide

37. ' 3PMFBQQ
    ' 3PMFEC
    BQQMZlBQQz
    QVQQFU
    NBTUFS )
    

    View Slide

38. ' 3PMFBQQ
    ' 3PMFEC
    BQQMZlBQQz
    QVQQFU
    NBTUFS
    BQQMZlECz
    )
    

    View Slide

39. ' 3PMFBQQ
    ' 3PMFEC
    BQQMZlBQQz
    QVQQFU
    NBTUFS
    BQQMZlECz
    QVQQFUNBTUFS͸ద༻ઌͷ3PMFλάΛݟͯ

    ద༻͢ΔΫϥεΛࣗಈͰܾఆ
    )
    

    View Slide

40. 4UBUVTλά
    ͦͷΠϯελϯε͕όϥϯαʔʹ

    ௥Ճ͍͍ͯ͠΋ͷ͔Ͳ͏͔Λఆٛ
    !
    ྫ:
    Status: working
    &
    

    View Slide

41. 3PMFBQQ
    *
    IBQSPYZDGH
    BQQIBQSPYZ
    3PMFBQQ
    

    View Slide

42. 3PMFBQQ
    ' 4UBUVTXPSLJOH
    *
    IBQSPYZDGH
    BQQIBQSPYZ
    3PMFBQQ
    

    View Slide

43. 3PMFBQQ
    ' 4UBUVTXPSLJOH
    ⟲
    *
    IBQSPYZDGH
    BQQIBQSPYZ
    3PMFBQQ
    

    View Slide

44. 3PMFBQQ
    ' 4UBUVTXPSLJOH
    ⟲
    *
    IBQSPYZDGH
    BQQIBQSPYZ
    3PMFBQQ
    

    View Slide

45. 3PMFBQQ
    ' 4UBUVTXPSLJOH
    ⟲
    *
    IBQSPYZDGH
    BQQIBQSPYZ
    3PMFBQQ
    ' 4UBUVTXPSLJOH
    

    View Slide

46. 3PMFBQQ
    ' 4UBUVTXPSLJOH
    ⟲
    *
    IBQSPYZDGH
    BQQIBQSPYZ
    3PMFBQQ
    ' 4UBUVTXPSLJOH
    

    View Slide

47. 3PMFBQQ
    ' 4UBUVTXPSLJOH
    ⟲
    *
    IBQSPYZDGH
    BQQIBQSPYZ
    3PMFBQQ
    ' 4UBUVTXPSLJOH
    

    View Slide

48. 3PMFBQQ
    ' 4UBUVTXPSLJOH
    ⟲
    *
    IBQSPYZDGH
    BQQIBQSPYZ
    3PMFBQQ
    ' 4UBUVTXPSLJOH
    XPSLJOHঢ়ଶͷΠϯελϯεΛࣗಈͰαʔϏεΠϯ
    

    View Slide

49. '
    ͳͥλάͰશ෦΍Δͷ͔
    Zookeeper, Serf ౳ͷಛผͳϛυϧ΢ΣΞΛඞཁͱͤͣʹ

    αʔϏεσΟεΧόϦ͕Մೳ
    !
    ඇৗʹγϯϓϧ͔ͭϓϩάϥϚϒϧ
    

    View Slide

50. λάͷϋϚϦͲ͜Ζ
    DescribeTags ͷ API Limit
    1 Ϧιʔεʹλά͸10ݸ·Ͱ
    %
    -JNJU
    DBDIF
    (
    
    

    View Slide

51. πʔϧ
    λάϑϨϯυϦʔͳ؅ཧπʔϧ͕ඞཁ
    ElasticFox-ec2tag
    

    View Slide

52. Φʔτεέʔϧ
    

    View Slide

53. τϥϑΟοΫ
    &$Πϯελϯε਺
    ࣌ؒ
    τϥϑΟοΫͷมԽʹԠͯ͡Πϯελϯε਺Λ૿ݮ
    

    View Slide

54. ͳͥΦʔτεέʔϧͳͷ͔
    '
    

    View Slide

55. *OTUBODFIPVS
    EC2ͷ࠷খ՝ۚ୯Ґ
    ىಈதͷΠϯελϯε਺ x ىಈ࣌ؒʢ1࣌ؒ୯Ґʣ
    ΦϯσϚϯυɺ౦ژϦʔδϣϯ೥݄೔ݱࡏIUUQBXTBNB[PODPNKQFDQSJDJOH
    

    View Slide

56. &$Πϯελϯε਺
    I
    Πϯελϯε࣌
    ( JOTUBODFIPVS
    Φʔτεέʔϧ
    ͳ͠
    Φʔτεέʔϧ
    ͋Γ
    

    View Slide

57. "NB[PO"VUP4DBMJOH
    

    View Slide

58. $MPVE8BUDI
    "VUP
    4DBMJOH
    "VUP4DBMJOH(SPVQ
    ".*
    

    View Slide

59. ͨͱ͑͹
    "NB[PO"VUP4DBMJOH
    Λ΍ΊΔ
    

    View Slide

60. ͳͥ"NB[PO"VUP4DBMJOHͰ͸ͳ͍ͷ͔
    CloudWatch 1෼ͷน
    Graceful Terminate ໰୊
    ELB Ͱ͸ͳ͘ haproxy Λ࢖͍͍ͨ
    σϓϩΠͱͷഉଞ੍ޚΛεϜʔζʹߦ͏ͨΊ
    

    View Slide

61. $MPVE8BUDIͷੑೳ΁ͷෆຬ
    ෼ͷนɹϝτϦΫε͕औಘͰ͖Δͷ͸࠷଎Ͱ΋1෼ޙ
    +
    

    View Slide

62. (SBDFGVM5FSNJOBUF໰୊
    desired_capacity ͕ݮΔͱ͖ɺ

    Πϯελϯε͸໰౴ແ༻ͰTerminate ͞Εͯ͠·͏
    ҆શͳαʔϏεΞ΢τΛߦ͏ʹ͸޻෉͕ඞཁ
    

    View Slide

63. *OUFSOBM&-#Ͱ͸ͳ͘IBQSPYZΛ࢖͍͍ͨ
    ELB ͷ IP ΞυϨεͷಈతͳมԽʹ

    ϛυϧ΢ΣΞ͕ରԠ͖͠Εͳ͍
    !
    ϒϥοΫϘοΫε౓͕ߴ͘ɺ໰୊͕೺Ѳͮ͠Β͍
    ˞&-#ͳ͠Ͱ΋"NB[PO"VUP4DBMJOH͸࢖͑·͢
    
    

    View Slide

64. "NB[PO"VUP4DBMJOH
    WT
    EFQMPZTEBZ
    
    
    

    View Slide

65. W W W
    W
    #
    

    View Slide

66. W W W
    W
    W
    
    #
    

    View Slide

67. W W W
    W
    W
    
    #
    

    View Slide

68. W W W
    W
    W
    
    W W W
    #
    

    View Slide

69. W W W W W W
    W
    W
    
    W W W
    #
    

    View Slide

70. W W W W W W
    W W W
    ݹ͍όʔδϣϯͷΠϯελϯε͕࡞ΒΕͯ͠·͏
    

    View Slide

71. "NB[PO"VUP4DBMJOHͰഉଞ੍ޚ
    σϓϩΠத͸ AutoScaling ΛࢭΊɺ
    AutoScaling த͸σϓϩΠΛېࢭ͍ͨ͠
    !
    AutoScaling ͕ΠϯελϯεΛ૿΍͢ͱ͖ɾݮΒ͢ͱ͖ʹ
    ॲཧΛׂΓ͜·ͤΔඞཁ͕͋Δ
    →ࠔ೉
    
    

    View Slide

72. CloudWatch 1෼ͷน
    Graceful Terminate ໰୊
    ELB Ͱ͸ͳ͘ haproxy Λ࢖͍͍ͨ
    σϓϩΠͱͷഉଞ੍ޚΛεϜʔζʹߦ͍͍ͨ
    ࣗ෼Ͱશ෦ॻ͍ͯ΋͋·Γखؒ͸มΘΒͳͦ͏
    "NB[PO"VUP4DBMJOHͰղܾෆՄೳͰ͸ͳ͍͚Ͳɺ
    ଟ͘ͷίʔυΛॻ͖ɺΧελϚΠζ͢Δඞཁ͕͋Δ
    +
    +
    

    View Slide

73. $PPLQBE"VUP4DBMJOHͷ࣮૷
    ,
    

    View Slide

74. JNPO "4
    #
    W
    W
    EFWFMPQFS
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    

    View Slide

75. JNPO "4
    #
    
    W
    W
    EFWFMPQFS
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    
    

    View Slide

76. W
    W
    JNPO "4
    #
    
    W
    W
    EFWFMPQFS
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    
    

    View Slide

77. W
    W
    JNPO "4
    #
    W
    W
    EFWFMPQFS
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    

    View Slide

78. W
    W
    JNPO "4
    #
    W
    W
    EFWFMPQFS
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    

    View Slide

79. W
    W
    JNPO "4
    #
    W
    W
    EFWFMPQFS
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    

    View Slide

80. W
    W
    JNPO "4
    #
    W
    W
    EFWFMPQFS
    W
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    

    View Slide

81. W
    W
    JNPO "4
    #
    W
    W
    EFWFMPQFS
    W
    W
    W
    W
    W
    W
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    

    View Slide

82. W
    W
    JNPO "4
    #
    W
    W
    EFWFMPQFS
    W
    W
    W
    W
    W
    W
    SQT
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    '4UBUVTXPSLJOH
    

    View Slide

83. Φʔτεέʔϧͷ͋Δ฻Β͠
    ✈
    

    View Slide

84. ͱ͋ΔΠϯελϯεͷ
    ୆਺
    ೔
    Φʔτεέʔϧ
    

    View Slide

85. ΦʔτεέʔϧʹΑΔσϓϩΠϩοΫத
    σϓϩΠத
    

    View Slide

86. (
    (
    ΦʔτεέʔϧʹΑΔσϓϩΠϩοΫத
    σϓϩΠத
    

    View Slide

87. Φʔτεέʔϧத
    σϓϩΠத
    

    View Slide

88. View Slide

89. View Slide

90. View Slide

91. - ೥Ҏ্ӡ༻Ͱ͖ͯΔ
    
    

    View Slide

92. ೥ͷӡ༻࣮੷Λ௨ͯ͠
    ۤ࿑ͨ͠఺ɺ޻෉ɺؾ͍ͮͨ͜ͱ
    

    View Slide

93. ϩοΫ࣌ؒɾճ਺Λ࠷খʹ͢Δ
    σϓϩΠΛϩοΫʹՁ஋Λಧ͚ΔνϟϯεΛݮΒ͢
    .
    

    View Slide

94. ୆਺
    

    View Slide

95. ୆਺
    ۀ຿࣌ؒ σϓϩΠՄೳ࣌ؒଳ
    
    

    View Slide

96. ୆਺
    ۀ຿࣌ؒ σϓϩΠՄೳ࣌ؒଳ
    
    αʔϏε"ϩοΫճ
    αʔϏε#ϩοΫճ
    

    View Slide

97. ୆਺
    Ұ౓ʹ૿ݮ͢Δ୆਺Λௐ੔͢Δ͜ͱͰ
    ۀ຿࣌ؒதͷσϓϩΠϩοΫճ਺Λ࠷খݶʹ
    ۀ຿࣌ؒ σϓϩΠՄೳ࣌ؒଳ
    
    αʔϏε"ϩοΫճ
    αʔϏε#ϩοΫճ
    

    View Slide

98. ୆਺
    Ұ౓ʹ૿ݮ͢Δ୆਺Λௐ੔͢Δ͜ͱͰ
    ۀ຿࣌ؒதͷσϓϩΠϩοΫճ਺Λ࠷খݶʹ
    ۀ຿࣌ؒ σϓϩΠՄೳ࣌ؒଳ
    
    αʔϏε"ϩοΫճ
    αʔϏε#ϩοΫճ
    ۀ຿࣌ؒ֎ͷͨΊɺ
    ݮΒ͢ͱ͖͸খࠁΈʹ
    

    View Slide

99. ςετΛॻ͘
    Φʔτεέʔϧ͸αʔόΛམͱ͢ϓϩάϥϜ
    /
    

    View Slide

100. ఆظతʹઃఆΛݟ௚͢
     ์͓͍ͬͯͯ΋εέʔϧ͢ΔΑ͏ʹݟ͑Δ͕…
     !
     ΞϓϦαʔόͷ୆਺૿ˠόοΫΤϯυ΁ͷίωΫγϣϯ૿
     ΞϓϦέʔγϣϯͷύϑΥʔϚϯε΋ৗʹมԽ
     0
     

     View Slide

101. TUBUJDOPEFTEZOBNJDOPEFTߏ੒
     ৗʹཱͪଓ͚ͯΔΠϯελϯε͕͍ͨ΄͏͕
     ͳΜ͔ͩΜͩͰศར
     TUBUJD
     EZOBNJD
     

     View Slide

102. View Slide

103. TUBUJDOPEFT EZOBNJDOPEFT
     Φʔτεέʔϧର৅֎
     ୆਺͸ݻఆ
     TIVUEPXO͞Εͳ͍
     Φʔτεέʔϧର৅
     ୆਺͸Մม
     CBMBODFS
     

     View Slide

104. αʔό໊Ͳ͏͠Α͏໰୊
     dynamic nodes Ͱ͋Ζ͏ͱɺ

     ݁ہ hostname ͸͋ͬͨ΄͏͕؅ཧ͠΍͍͢
     !
     10.0.123.45 ͩͬͨΒ app-10-0-123-45 ͱ͍͏

     Name λά͕ͭ͘Α͏ʹ͍ͯ͠Δ
     &
     

     View Slide

105. Φʔτεέʔϧͷ෭࢈෺
     ΦʔτεέʔϧΛ૊Ήͱɺ Immutable ͳ

     Golden Image ͕࡞ΕΔ
     !
     ͍ͭͰ΋ҰൃͰ࠷৽ͷαʔό্ཱ͕ͪ͛ΒΕΔ͜ͱ͕

     อূ͞ΕΔ +
     

     View Slide

106. ·ͱΊ
     

     View Slide

107. EFQMPZTEBZWT
     "VUP4DBMJOH
     10 deploys/day ͳ؀ڥͰ΋Φʔτεέʔϧ͸Մೳ
     ίετͱ։ൃ଎౓ͷτϨʔυΦϑΛߟྀͨ͠νϡʔχϯά
     

     View Slide

108. "VUP4DBMJOHBT
     *NNVUBCMF*OGSBTUSVDUVSF
     Immutable Infrastructure ͸ະདྷͷ࿩͡Όͳ͘ɺ

     ΫοΫύουͰ͸ΦʔτεέʔϧʹΑͬͯ͢Ͱʹ
     ݱ࣮ʹͳͬͯΔ
     

     View Slide

109. 2"
     '
     

     View Slide

110. DSFEJUT
     http://aws.amazon.com/jp/architecture/icons/
     http://www.entypo.com/
     http://fortawesome.github.io/Font-Awesome/
     http://www.flickr.com/photos/sora_h/8916379094/
     

     View Slide