インフラエンジニアがConsulとStretcherをつかったデプロイ改善で開発効率の向上に貢献した話 / jtf2017-consul-stretcher-deploy

Transcript

1. ΠϯϑϥΤϯδχΞ͕
   ConsulͱStretcherΛ͔ͭͬͨσϓϩΠվળͰ
   ։ൃޮ཰ͷ޲্ʹߩݙͨ͠࿩
   July Tech Festa 2017
   Shuichi Ohsawa (@ohsawa0515)
   1/64
   

   View Slide

2. ࣗݾ঺հ
   • େᖒलҰ
   • Blog: http://blog.jicoman.info/
   • Sansanגࣜձࣾ
   • σʔλԽγεςϜͷΠϯϑϥΛ୲౰
   • Πϯϑϥߏஙɾӡ༻ɾվળɺ։ൃج൫ͷվળͳͲ
   • લ৬͸WebΤϯδχΞ(PHPer)ͱͯ͠ܞଳίϯςϯπαΠτͷ։ൃ΍AWS΁ͷ
   Ҡߦͱ͔΍͍ͬͯ·ͨ͠
   2/64
   

   View Slide

3. ຊεϥΠυʹ͍ͭͯ
   ։ൃऀͱΠϯϑϥ୲౰ͱͷϓϩδΣΫτͱͯ͠औΓ૊ΜͩҰࣄྫ
   ʹ͍͓ͭͯ࿩͍͖ͤͯͨͩ͞·͢ɻ։ൃج൫ͷվળࣄྫͱͯ͠গ
   ͠Ͱ΋ࢀߟʹͳΕΕ͹޾͍Ͱ͢ɻ
   ಺༰తʹ৽͘͠ͳ͍͔΋͠Εͳ͍͚Ͳ͝༰͍ࣻͩ͘͞!
   ࣮૷಺༰ͳͲ͸εϥΠυͷ࠷ޙʹ͋Γ·͢ͷͰ͝ࢀߟ·Ͱʹ
   3/64
   

   View Slide

4. ΞδΣϯμ
   • ͳͥσϓϩΠͷվળʹऔΓ૊Μͩͷ͔
   • γεςϜߏ੒
   • ैདྷͷσϓϩΠͱ՝୊఺
   • վળ΁ͷऔΓ૊Έ
   • ৽σϓϩΠΞʔΩςΫνϟ
   • ·ͱΊ
   4/64
   

   View Slide

5. ๏ਓ޲໊͚ࢗ؅ཧαʔϏεʮSansanʯ
   5/64
   

   View Slide

6. ݸਓ޲໊͚ࢗΞϓϦʮEightʯ
   6/64
   

   View Slide

7. Data Strategy & Operation Center
   • ໊ࢗͷσʔλԽ&σʔλ׆༻
   • Development Group
   • ໊ࢗͷσʔλԽͷγεςϜʮGEESʯશମͷ։ൃɾӡ༻
   • ओʹRuby on RailsͰ։ൃ
   • R&D Group
   • ໊ࢗͷσʔλԽͷࣗಈԽͷͨΊʹඞཁͳը૾ೝࣝɺػցֶशɺςΩετղੳͳͲ
   • σʔλαΠΤϯςΟετ΋ࡏ੶͠ɺσʔλ׆༻ʹ஫ྗ
   • ओͳ։ൃݴޠ: C#, Python
   • Πϯϑϥ୲౰͸Devlopment Groupʹॴଐ͍ͯ͠Δ͕ɺR&D Group΋ݟ͍ͯΔ
   • 3໊(಺Ұਓ͸։ൃͱ݉຿)
   7/64
   

   View Slide

8. ։ൃऀͱΠϯϑϥ୲౰ͱͷؔΘΓ
   • جຊతʹ։ൃऀͱҰॹʹϓϩδΣΫτʹೖΔ͜ͱ͸ͳ͍
   • ΠϯϑϥνʔϜͱͯ͠ผϓϩδΣΫτΛ૸Β͍ͤͯΔ͜ͱ͕ଟ͍
   • ґཔɾ૬ஊϕʔεͰͷؔΘΓ͕࠷΋ଟ͍
   • αʔόߏஙɺ˓˓Λ࢖͍͍ͨ etc
   • ΞϥʔτରԠ͸։ൃऀɾΠϯϑϥ୲౰
   • ։ൃऀ͸Πϯϑϥͷίʔυ(Chef, TerraformͳͲ)Λ΄ͱΜͲ͍͡Βͳ͍
   • Πϯϑϥ୲౰͸αʔϏεͷιʔείʔυΛ͍͡Βͳ͍
   8/64
   

   View Slide

9. ͳͥσϓϩΠͷվળʹ
   औΓ૊Μͩͷ͔
   9/64
   

   View Slide

10. ݩʑ͸ҰϓϩδΣΫτͷҰ؀
    • εςʔδϯά؀ڥͷࣗಈߏஙϓϩδΣΫτ
    • ༻్ʹ߹Θͤͨෳ਺ͷεςʔδϯά؀ڥ͕ඞཁͩͬͨ
    • TerraformΛ׆༻ͯࣗ͠ಈతʹߏஙͱഁյ͕Ͱ͖ΔΑ͏ʹ͢Δ
    • ࣗಈߏங͢Δʹ͸σϓϩΠͷࣗಈԽ΋ඞཁ
    • ैདྷͷσϓϩΠͰ͸ࣗಈԽ͕೉͔ͬͨ͠
    • ຊ൪؀ڥʹ͓͚ΔσϓϩΠʹ՝୊Λײ͍ͯͨͨ͡ΊվળʹऔΓ૊Ή͜ͱʹͨ͠
    • ݩʑ͸ΠϯϑϥνʔϜ಺ͷϓϩδΣΫτ͕ͩͬͨɺσϓϩΠʹؔͯ͠෼͔Βͳ͍͜ͱ͕
    ଟ͍ͷͰ։ൃऀʹڠྗͯ͠΋Βͬͨ
    10/64
    

    View Slide

11. γεςϜߏ੒
    11/64
    

    View Slide

12. αʔόߏ੒
    • ϩʔϧ
    • WebɺBatchɺDeploy etc
    • Webαʔό
    • ELB + EC2(c3.xlarge × 20਺୆)
    • Batchαʔό
    • EC2(r3.xlarge × े਺୆)
    • Deployαʔό
    • Web/BatchαʔόʹσϓϩΠ
    12/64
    

    View Slide

13. ΞϓϦέʔγϣϯ
    • Web/Batchαʔόͱ΋ʹ20ऑͷRailsΞϓϦ͕ಉډ!
    • ίετ࡟ݮ
    • සൟʹϦΫΤετ͞Εͳ͍αʔϏε
    • Apache + Passenger
    13/64
    

    View Slide

14. ैདྷͷσϓϩΠͱ՝୊఺
    14/64
    

    View Slide

15. ैདྷͷσϓϩΠ
    • ఆظϦϦʔε(िҰ)
    • ͢΂ͯͷRailsΞϓϦΛσϓϩΠ
    • ඞཁʹԠͯ͡hotfixϦϦʔε(िʹ਺ճ)
    • ඞཁͳΞϓϦ͚ͩσϓϩΠ
    • Web/Batchαʔόશ୆Ұ੪ʹσϓϩΠ
    • σϓϩΠαʔό͔Β࣮ߦ
    • PushܕσϓϩΠ/In place/All at once
    15/64
    

    View Slide

16. ैདྷͷσϓϩΠ
    • σϓϩΠεΫϦϓτ(Rakefile)ͰσϓϩΠ࣮ߦ
    • Capistrano 2ϕʔε
    • ΞϓϦຖʹCapλεΫϑΝΠϧͷੜ੒
    • ฒྻͰෳ਺ΞϓϦΛσϓϩΠ
    • ϩʔϧ͸Chef ServerͷλάͰ؅ཧ
    • Web, BatchͰॲཧ͕ҟͳΔͨΊϩʔϧͰ۠ผ
    • knife tagίϚϯυͰ෇͚֎͢͠Δ
    16/64
    

    View Slide

17. ՝୊఺
    1. σϓϩΠʹ͕͔͔࣌ؒΔ
    2. ಛఆͷαʔό͚ͩσϓϩΠͰ͖ͳ͍
    3. GitHub/RubyGems͕མͪΔͱσϓϩΠͰ͖ͳ͍
    4. ࣗಈԽͮ͠Β͍
    17/64
    

    View Slide

18. ͕͔͔࣌ؒΔ
    • ֤αʔόͰBundleΠϯετʔϧͱΞηοτϓϦίϯύΠϧ͢Δ
    ͨΊɺ΋ͷ͕͔͔࣌ؒ͘͢͝Δ
    • ஗͍࣌ʹ͸໿30෼͔͔Δ
    • ్தͰίέΔͱσϓϩΠ΍Γ௚͠(·ͨ30෼͔͔Δ)
    • ༧ఆ௨ΓʹऴΘΔ͔ϋϥϋϥ͢Δ͜ͱ΋͠͹͠͹
    18/64
    

    View Slide

19. ಛఆͷαʔό͚ͩσϓϩΠͰ͖ͳ͍
    • ಛఆͷαʔό͚ͩσϓϩΠ͢ΔͱଞͷαʔόͱϑΝΠϧύε͕ҟͳͬͯ͠·͏
    • ϓϦίϯύΠϧ͞Εͨը૾/CSS/JavaScript
    • ͦͷαʔό΁ϦΫΤετ͕དྷΔͱ404 NotFound
    • ଞαʔό͔Βtar.gzͰݻΊͯ࣋ͬͯ͘ΔͷͰ͕͔͔࣌ؒΔ
    • ৽نαʔόͷ௥Ճʹ͙͢ʹରԠͰ͖ͳ͍
    • ෛՙ૿Ճ࣌ʹ͙͢ʹεέʔϧΞ΢τͰ͖ͣɺରԠ͕ऴΘΔࠒʹෛՙ͕མͪண
    ͍͍ͯΔ͜ͱ΋͋ͬͨ
    19/64
    

    View Slide

20. GitHub/RubyGems͕མͪΔͱσϓϩΠࣦഊ
    • GitHub͕ͨ·ʹ(?)DDOS߈ܸΛड͚ͯgit pullͰ͖ͳ͍
    • rubygems.org͕མͪΔͱBundleΠϯετʔϧͰ͖ͳ͍
    • ճ෮͢ΔͷΛ଴͔ͭ͠ͳ͘ਆཔΈ!
    20/64
    

    View Slide

21. ࣗಈԽͮ͠Β͍
    • Chef Server΁ͷϗετొ࿥ͱ࡟আ͕໘౗
    • αʔόΛ௥Ճ͢Δͱ͖͸knife-ec2Λ࢖͏͕ɺࣗಈ௥Ճ(ΦʔτεέʔϧͳͲ)ͩͱ໘౗
    • αʔό࡟আ͢Δͱ͖͸खಈͰϗετͱλάΛ࡟আ
    • ϩʔϧͷλά෇͚֎͠͸Chef Serverʹґଘ
    • knife tagίϚϯυ΋खಈ
    • ίϚϯυ࣮ߦͯ͠΋͙͢ʹ൓ө͞Εͳ͍
    • ΠϯϑϥνʔϜͱͯ͠Chef ServerΛࣺ͔ͯͨͬͨ
    21/64
    

    View Slide

22. վળ΁ͷऔΓ૊Έ
    22/64
    

    View Slide

23. σϓϩΠπʔϧબఆͷํ਑
    • PullܕσϓϩΠ
    • αʔόࣗ਎͕σϓϩΠ͢Δ
    • σϓϩΠ͕࣌ؒҰఆ(୆਺ʹൺྫ͠ͳ͍)
    • Amazon S3ʹϏϧυࡁͷϑΝΠϧΛ഑ஔ
    • μ΢ϯϩʔυ͠഑ஔɺ࠶ىಈ͢Δ͚ͩͰσϓϩΠ׬ྃ
    • S3ͷՄ༻ੑ͸99.99ˋ
    • In place/All at once
    • Blue/GreenσϓϩΠͩͱ࢓૊Έ͕ෳࡶԽ͢ΔͨΊҰ୴ݟૹΓ
    • One by one(1୆ͣͭ)ͩͱ͕͔͔࣌ؒΔ
    • ϗετͷొ࿥ɾ࡟আ͸ࣗಈతʹߦ͍͍ͨ
    23/64
    

    View Slide

24. AWS CodeDeploy
    • AWSͷσϓϩΠαʔϏε
    • ΤʔδΣϯτʹΑͬͯPullܕσϓϩΠ͕Մೳ
    • AWSଆͰόʔδϣϯ؅ཧ͞Ε͍ͯΔͨΊɺϩʔϧόοΫ΋؆୯
    • GUIͰσϓϩΠঢ়گ͕֬ೝͰ͖Δͷ͕ศར
    24/64
    

    View Slide

25. AWS CodeDeploy
    • ࠾༻Λݟૹͬͨཧ༝!
    • ෳ਺ΞϓϦΛҰ੪ʹσϓϩΠ͢Δͱ஗͍(γϦΞϧʹ࣮ߦ͞ΕΔ)
    • All at onceͷ৔߹ɺ1୆Ͱ΋ࣦഊͨ͠ΒσϓϩΠࣦഊͱͳΒͳ͍
    • σϓϩΠࣦഊʹͳΒͳ͍͔ΒϩʔϧόοΫͰ͖ͳ͍
    • CodeDeployͷ૝ఆ͍ͯ͠Δ؀ڥͱࣗ෼ୡͷͱϚον͠ͳ͔ͬͨ
    25/64
    

    View Slide

26. Stretcher
    • https://github.com/fujiwara/stretcher
    • Consulͱ૊Έ߹ΘͤΔ͜ͱͰɺΠϕϯτΛτϦΨʔͱͨ͠PullܕσϓϩΠ͕Մೳ
    • ࠾༻ͨ͠ཧ༝!
    • ෳ਺ΞϓϦΛҰ੪ʹσϓϩΠͯ͠΋ૣ͍
    • 20ऑͷRailsΞϓϦΛฒྻσϓϩΠ͢Δͱ1ʙ2෼ఔ౓ͰऴΘΔ
    • ෳ਺ͷઃఆϑΝΠϧ(ϚχϑΣετϑΝΠϧ)Λ༻ҙ͢Δ͜ͱͰɺৼΔ෣͍Λม͑Δ͜ͱ͕Ͱ͖Δ
    • σϓϩΠϑΝΠϧ഑ஔ༻
    • γϯϘϦοΫϦϯΫ੾ସ༻
    • Capistranoͷ࢓૊ΈΛྲྀ༻
    • Consulͱͷ਌࿨ੑ͕ߴ͍
    26/64
    

    View Slide

27. Consul
    • https://www.consul.io/
    • HashiCorpࣾఏڙͷΦʔέετϨʔγϣϯπʔϧ
    • αʔόͱΫϥΠΞϯτͱ͍͏̎ͭͷ໾ׂͰߏ੒͞ΕɺΫϥελΛܗ੒͢Δ
    • ো֐ݕ஌ɺ಺෦DNSɺKey/ValueετΞ(KVS)
    • Πϕϯτൃߦ
    • ֤ConsulΫϥΠΞϯτ͸ΠϕϯτΛ଴ͪड͚͍ͯΔঢ়ଶͰɺಛఆͷΠϕϯτ͕ൃߦ͞ΕͨΒ೚ҙͷΞ
    ΫγϣϯΛ࣮ߦ
    • αʔϏελάʹΑΔϩʔϧ؅ཧ
    • ىಈ࣌ʹΫϥελϦϯά΁ࣗಈδϣΠϯ
    27/64
    

    View Slide

28. طଘͷ࢓૊ΈΛ௥͏
    • σϓϩΠεΫϦϓτΛಡΉ
    • ͘Θ͍͠ਓʹ֓ཁΛฉ͍ͯɺৄࡉ͸ಡΜͰཧղ
    • Capistrano 2ͷιʔείʔυ΋ซͤͯಡΉ
    • cap deployͰͲͷΑ͏ʹ࣮ߦ͞Ε͍ͯΔͷ͔
    • ϝϯςφϯε੾ସɺϩʔϧόοΫͷ࢓૊Έ΋௥ͬͯΈΔ
    • ։ൃऀͰ΋෼͔Βͳ͍͜ͱ͕ҙ֎ͱଟ͍
    28/64
    

    View Slide

29. σϓϩΠεΫϦϓτͷվम
    • capistrano-stretcher
    • https://github.com/pepabo/capistrano-stretcher
    • Capistrano͔ΒStretcher + ConsulΛݺͼग़ͯ͠σϓϩΠͰ͖Δ
    • ͦͷ··࢖͍͔͚ͨͬͨͲͰ͖ͳ͔ͬͨŋŋŋ
    • Capistrano3 ରԠ
    • طଘͷσϓϩΠεΫϦϓτͰ͸ಠࣗίϚϯυ͕ଟ͍
    • ࢀߟʹͭͭ͠ɺҰ͔ΒػೳΛ࣮૷
    29/64
    

    View Slide

30. େ͖͘ม͑͗͢ͳ͍
    • େ͖ͳมߋ͸ֶशίετ͕ଟ͘ͳΔɺ৺ཧతোน͕ߴ͘ͳΔ
    • ࢖ͬͯ΋Β͏ͷ͸։ൃऀ
    • গͣͭ͠ม͍͑ͯ͘
    • Ұؾʹม͑ͣʹɺͰ͖Δ͚ͩࠓ·Ͱͷํ๏Λอͪͳ͕Β
    • ௕ظͰվળ͍͚ͯ͠͹݁Ռͱͯ͠େ͖͘มΘ͍͚ͬͯΔ
    30/64
    

    View Slide

31. ϨϏϡʔ/ϑΟʔυόοΫ
    • GitHub Flowʹଇͬͯ։ൃɺϨϏϡʔͯ͠΋Β͏
    • ϦϑΝΫλϦϯά΍ςετ΋ॻ͍ͯ΋Β͑ͨ
    • σϓϩΠͯ͠΋ΒͬͯϑΟʔυόοΫ
    • ίϚϯυ໊ͷมߋɺ࢓༷ͷमਖ਼Λదٓमਖ਼
    • SlackͰͷ͞Γ͛ͳ͍ͭͿ΍͖΋ݟಀ͞ͳ͍
    31/64
    

    View Slide

32. υΩϡϝϯτ࡞੒
    • ֓ཁ
    • ৽͍͠σϓϩΠͷશମ૾Λ௫ΜͰ΋Β͏
    • ίϚϯυͱ࣮ߦํ๏
    • σϓϩΠ͢Δͱ͖ͷϚχϡΞϧ
    • Կ΋ߟ͑ͣʹίϐϖ͢Δ͚ͩͰσϓϩΠͰ͖ΔΑ͏ʹ
    • τϥϒϧγϡʔςΟϯά
    • ૝ఆ͞ΕΔΤϥʔ಺༰ͱͦͷରॲํ๏
    • ࣗ෼͕͍ͳͯ͘΋ղܾͰ͖ΔΑ͏ʹ
    • ໰୊͕ىͬͨ͜ͱ͖ʹ୭Ͱ΋ରॲͰ͖ΔΑ͏ʹͳΔͷ͕ཧ૝త
    32/64
    

    View Slide

33. આ໌͢Δ৔Λઃ͚Δ
    • ࢿྉΛಡΜͰ΋Β͏͚ͩͩͱ఻ΘΓͮΒ͍͜ͱ͕ଟ͍
    • ৽͍͠σϓϩΠͷϝϦοτ
    • ࠓ·Ͱͷํ๏͔ΒͲͷΑ͏ʹมΘΔͷ͔
    33/64
    

    View Slide

34. ৽σϓϩΠΞʔΩςΫνϟ
    34/64
    

    View Slide

35. ֓ཁ
    • ιʔείʔυΛϏϧυɺtar.gzͰݻΊͯɺS3ʹΞοϓϩʔυ
    • ϏϧυαʔόʹσϓϩΠ
    • ϩʔϧຖʹϚχϑΣετϑΝΠϧΛੜ੒ɺS3ʹΞοϓϩʔυ
    • consul eventͰΠϕϯτൃՐɺWeb/BatchαʔόͰStretcher࣮ߦ
    • σϓϩΠϑΝΠϧͷऔಘɺ഑ஔ
    • ϦϦʔεલ·ͰʹऴΘΒ͓ͤͯ͘
    • γϯϘϦοΫϦϯΫ੾ସɺRails࠶ىಈ
    • ϦϦʔε࣌ʹ࣮ߦ(ϦϦʔε࣌ؒͷ୹ॖ)
    • σΟϨΫτϦߏ଄͸Capistranoͱಉ͡
    35/64
    

    View Slide

36. ߏ੒
    36/64
    

    View Slide

37. Consulͷઃఆ
    • ChefͰઃఆϑΝΠϧ(JSON)Λੜ੒
    • ConsulΫϥΠΞϯτ
    • ϩʔϧλά
    • Πϕϯτ
    • σϓϩΠ
    • ϩʔϧλάͷ෇͚֎͠
    37/64
    

    View Slide

38. Ϗϧυαʔό΁ͷσϓϩΠ
    38/64
    

    View Slide

39. ϚχϑΣετϑΝΠϧͷੜ੒
    • ϚχϑΣετϑΝΠϧͷςϯϓϨʔτ(ERB)
    • update_code_#{application}_#{version}.yml
    • S3͔ΒσϓϩΠϑΝΠϧΛμ΢ϯϩʔυ͠ɺ഑ஔ
    • deploy_#{application}_#{role}_#{version}.yml
    • γϯϘϦοΫϦϯΫ੾ସɺRails࠶ىಈ
    39/64
    

    View Slide

40. update_code_#{application}_#{version}.yml
    src: s3://<%= s3_bucket %>/<%= application %>
    checksum: <%= checksum %>
    dest: app/<%= application %>/release/YYYYMMDDHHMMSS
    commands:
    pre:
    - mkdir -p app/<%= application %>
    success:
    - echo 'success'
    failure:
    - echo 'failed!!'
    excludes:
    - "*.pid"
    - "*.socket"
    40/64
    

    View Slide

41. deploy_#{application}_#{role}_#{version}.yml
    sync_strategy: mv
    commands:
    post:
    - ln -sfn "app/<%= application %>/release/YYYYMMDDHHMMSS" "app/<%= application %>/current"
    <%=# Rails࠶ىಈ %>
    - cd app/<%= application %>/release/YYYYMMDDHHMMSS && touch tmp/restart.txt
    <%=# ಈ࡞֬ೝ & ஆػӡస %>
    - <%= File.join(script_path, "pre-warming.sh") %> <%= application %>
    success:
    <%=# աڈͷϦϦʔεΛ࡟আ %>
    - <%= File.join(script_path, "cleanup.sh") %> <%= application %>
    - echo 'release success'
    failure:
    - echo 'release failed!!'
    41/64
    

    View Slide

42. S3΁ͷΞοϓϩʔυ
    42/64
    

    View Slide

43. ΞϓϦέʔγϣϯαʔόʹ഑෍
    43/64
    

    View Slide

44. σϓϩΠ(γϯϘϦοΫϦϯΫ੾ସ)
    44/64
    

    View Slide

45. Consul KVSͰόʔδϣϯ؅ཧ
    • σϓϩΠ੒ޭޙɺConsul KVSʹ੒ޭͨ͠࠷৽ͱ1ͭલͷόʔδϣϯΛ
    ߋ৽͢Δ
    •
    http://127.0.0.1:8500/v1/kv/#{application}/current
    •
    http://127.0.0.1:8500/v1/kv/#{application}/previous
    • όʔδϣϯ͸ϦϦʔεσΟϨΫτϦͷ೔෇(
    YYYYMMDDHHMMSS
    )
    • αʔό௥Ճ͢Δͱ͖͸࠷৽ɺϩʔϧόοΫ͢Δͱ͖͸1ͭલͷόʔδ
    ϣϯΛࢦఆ͢Δ
    45/64
    

    View Slide

46. ϩʔϧόοΫ
    • 1ͭલͷόʔδϣϯͷσϓϩΠϑΝΠϧ͕αʔόʹ͋Δͱ͸ݶΒ
    ͳ͍
    • σϓϩΠʹࣦഊͨ͠৔߹͸ɺ1ͭલͷόʔδϣϯΛσϓϩΠ͢Δ
    • σϓϩΠϑΝΠϧͷ഑෍ͱγϯϘϦοΫϦϯΫ੾ସͷ྆ํΛߦ
    ͏
    46/64
    

    View Slide

47. ·ͱΊͯσϓϩΠ͢Δ
    ͢΂ͯͷΞϓϦΛϚϧνεϨουͰฒྻσϓϩΠ
    task deploy_all: :setup do
    deploy_projects = "application_A,apllication_B,..."
    threads_count = ENV['THREADS']
    Parallel.map(deploy_projects, in_threads: threads_count.to_i) do |project|
    # Ϗϧυαʔό΁ͷσϓϩΠ
    # ϚχϑΣετϑΝΠϧͷੜ੒
    # S3΁ͷΞοϓϩʔυ
    # ΞϓϦέʔγϣϯαʔόʹ഑෍
    end
    end
    task release_all: :setup do
    deploy_projects = "application_A,apllication_B,..."
    threads_count = ENV['THREADS']
    Parallel.map(deploy_projects, in_threads: threads_count.to_i) do |project|
    # σϓϩΠ(γϯϘϦοΫϦϯΫ੾ସ)
    end
    end
    47/64
    

    View Slide

48. Slack
    SlackͰσϓϩΠঢ়گ͕֬ೝͰ͖Δ
    ໰୊͕ੜͨ͡৔߹͸όοΫτϨʔε͕ද
    ࣔ
    48/64
    

    View Slide

49. ݁Ռ
    1. σϓϩΠʹ͕͔͔࣌ؒΔ
    • => ϦϦʔε࣌ؒͷ୹ॖ(30෼->5෼)
    2. ಛఆͷαʔό͚ͩσϓϩΠͰ͖ͳ͍
    • => ࠷৽ͷσϓϩΠϑΝΠϧΛ࣋ͬͯ͘Δ͚ͩ
    3. GitHub/RubyGems͕མͪΔͱσϓϩΠͰ͖ͳ͍
    • => S3ͳͷͰ΄΅໰୊ͳ͠
    4. ࣗಈԽͮ͠Β͍
    • => ConsulʹΑͬͯΫϥελϦϯά΁ͷࣗಈδϣΠϯ/୤ୀ͕Մೳ
    49/64
    

    View Slide

50. تͼͷ੠ !
    50/64
    

    View Slide

51. ·ͱΊ
    • Πϯϑϥ୲౰͕σϓϩΠվળΛͨ͜͠ͱͰ։ൃج൫ʹߩݙͰ͖ͨ
    • ։ൃͱΠϯϑϥͷ྆ํΛ஌Βͳ͍ͱվળͰ͖ͳ͍ྖҬ͕͋Δ
    • Πϯϑϥ͚ͩͰͰ͖Δ෼໺͸গͳ͍
    • ։ൃ΋Ͱ͖ΔΠϯϑϥ(ࣗ෼΋ΩϟϦΞతଆ໘΋ؚΊͯ)
    • օʹͱͬͯ޾ͤͳಓΛબͿ
    • ։ൃଆͱΠϯϑϥଆͷཁٻ͕Ϛον͢Ε͹ϕετ
    • গͣͭ͠ม͍͑ͯ͘
    • Ұؾʹ΍ΓํΛม͑ͣʹɺͰ͖Δ͚ͩࠓ·Ͱͷํ๏Λอͪͳ͕Β
    • ௕ظؒվળ͍͚ͯ͠͹݁Ռͱͯ͠େ͖͘มΘ͍͚ͬͯΔ
    51/64
    

    View Slide

52. ࠓޙͷ༧ఆ
    • Φʔτεέʔϧ(ઈࢍରԠத)
    • όοναʔόΛAmazon ECS΁Ҡߦ
    • Webαʔόʹಉډ͍ͯ͠ΔRailsΞϓϦͷ෼ׂ
    • SlackίϚϯυʹΑΔσϓϩΠ
    52/64
    

    View Slide

53. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
    53/64
    

    View Slide

54. ͓·͚
    

    View Slide

55. Consul - ΫϥΠΞϯτ
    # /etc/consul.d/settings.json
    {
    "data_dir": "/opt/consul",
    "log_level": "err",
    "start_join": [
    "consul-server001",
    "consul-server002",
    "consul-server003"
    ]
    }
    55/64
    

    View Slide

56. Consul - ϩʔϧλά
    # /etc/consul.d/role.json
    {
    "service": {
    "name": "role",
    "tags": [ # ෳ਺ઃఆՄ
    "web",
    "newrelic_monitoring"
    ]
    }
    }
    56/64
    

    View Slide

57. Consul - Πϕϯτ(σϓϩΠ)
    # /etc/consul.d/deploy.json
    {
    "watches": [
    {
    "type": "event",
    "name": "deploy_application_A",
    "handler": "export TMPDIR=/ephemeral0; stretcher -retry 3 >> $TMPDIR/deploy_application_A.log 2>&1"
    },
    {
    "type": "event",
    "name": "deploy_application_B",
    "handler": "export TMPDIR=/ephemeral0; stretcher -retry 3 >> $TMPDIR/deploy_application_B.log 2>&1"
    },
    ...
    }
    • consul eventίϚϯυͰΠϕϯτൃՐ
    • consul event -service=role -tag=#{role} -name=deploy_#{application} #{manifest_file}
    • σϓϩΠϑΝΠϧͷμ΢ϯϩʔυઌ(TMPDIR)ΛEC2ͷΠϯελϯεετΞʹมߋ
    57/64
    

    View Slide

58. ϩʔϧ
    # ϩʔϧͷࢦఆ
    role :web, *consul_role('web')
    role :batch, *consul_role('batch')
    # ϩʔϧࢦఆ͞Εͨϗετͷऔಘ
    def consul_role(roles)
    response = `curl -fsSL http://127.0.0.1:8500/v1/catalog/service/role`
    nodes = JSON.parse(response)
    lists = []
    nodes.each do |node|
    if (node['ServiceTags'] & roles.split(',')) == roles.split(',')
    lists << node['Address']
    end
    end
    lists
    end
    58/64
    

    View Slide

59. Consul - Πϕϯτ(ϩʔϧλάͷ෇͚֎͠)
    # /etc/consul.d/change_role.json
    {
    "watches": [
    {
    "type": "event",
    "name": "change_role",
    "handler": "change_consul_role.rb >> /tmp/change_log.log 2>&1"
    }
    ]
    }
    #!/bin/env ruby
    require "json"
    consul_url = "http://127.0.0.1:8500"
    response = %x(curl -fsSL #{consul_url}/v1/catalog/node/$(curl -fsSL #{consul_url}/v1/agent/self | jq -r .Member.Name))
    node = JSON.parse(response)
    new_role = node["Services"]["role"]["Tags"]
    consul_role_file_path = "/etc/consul.d/role.json"
    if ! File.exist?(consul_role_file_path) then
    exit 0
    end
    consul_role = open(consul_role_file_path) do |io|
    JSON.load(io)
    end
    if consul_role["service"]["tags"] != new_role then
    consul_role["service"]["tags"] = new_role
    open(consul_role_file_path, 'w') do |io|
    io.puts(JSON.pretty_generate(consul_role))
    end
    %x(consul reload)
    end
    59/64
    

    View Slide

60. S3΁ͷΞοϓϩʔυ
    def upload2s3(local_src_path, bucket, path)
    run "aws s3api put-object --region #{aws_region} --bucket #{bucket} --key #{path} --body #{local_src_path}"
    end
    def create_tarball
    run "cd #{latest_release} && rm -rf #{tarball_name} && shopt -s dotglob && tar zcf #{tarball_name} *"
    end
    def upload_tarball
    upload2s3("#{latest_release}/#{tarball_name}", app_s3_bucket, "#{app_s3_path}/#{tarball_name}")
    end
    

    View Slide

61. σϓϩΠϑΝΠϧͷ഑෍
    namespace :stretcher do
    task :update_code, roles: :build do
    roles.split(',').each do |role|
    manifest_file = "update_code_#{application}_#{version}.yml"
    capture "consul event -service=role -tag=#{role} -name=deploy_#{application} #{manifest_file}"
    end
    puts 'Finished(Not released yet!)'
    end
    end
    

    View Slide

62. ϦϦʔε(γϯϘϦοΫϦϯΫ੾ସ)
    namespace :stretcher do
    task :release, roles: :build do
    transaction do
    on_rollback do
    rollback
    end
    roles.split(',').each do |role|
    manifest_file = "deploy_#{application}_#{role}_#{version}.yml"
    capture "consul event -service=role -tag=#{role} -name=deploy_#{application} #{manifest_file}"
    end
    unless check_event_success('release_failure')
    puts 'Release Failed!! & Starting rolling back.'
    exit 1
    end
    puts 'Release finished.'
    end
    end
    end
    

    View Slide

63. όʔδϣϯߋ৽
    ConsulͷKVSʹ֨ೲ͍ͯ͠Δόʔδϣϯ஋Λߋ৽
    after 'stretcher:release', 'stretcher:update_release_version'
    namespace :stretcher do
    task :update_release_version, roles: :build do
    # લճ੒ޭͨ͠σϓϩΠͷόʔδϣϯΛpreviousʹ͢Δ
    capture "curl -sS -X PUT -d '#{current_version}'http://127.0.0.1:8500/v1/kv/#{application}/previous"
    # ੒ޭͨ͠σϓϩΠͷόʔδϣϯΛߋ৽͢Δ
    capture "curl -sS -X PUT -d '#{latest_version)}' http://127.0.0.1:8500/v1/kv/#{application}/current"
    end
    end
    

    View Slide

64. ϩʔϧόοΫ
    namespace :stretcher do
    task :rollback, roles: :build do
    fetch(:deploy_roles).split(',').each do |target_role|
    manifest_file = "update_code_#{application}_#{previous_version}.yml"
    capture "consul event -service=role -tag=#{target_role} -name=deploy_#{application} #{manifest_file}"
    end
    reset_release_status
    fetch(:deploy_roles).split(',').each do |target_role|
    manifest_file = "deploy_#{application}_#{role}_#{previous_version}.yml"
    capture "/usr/local/bin/consul event -service=role -tag=#{target_role} -name=deploy_#{application} #{manifest_file}"
    end
    end
    

    View Slide