インフラエンジニアがConsulとStretcherをつかったデプロイ改善で開発効率の向上に貢献した話 / jtf2017-consul-stretcher-deploy

Transcript

1. ΠϯϑϥΤϯδχΞ͕ ConsulͱStretcherΛ͔ͭͬͨσϓϩΠվળͰ ։ൃޮ཰ͷ޲্ʹߩݙͨ͠࿩ July Tech Festa 2017 Shuichi Ohsawa (@ohsawa0515) 1/64
2. ࣗݾ঺հ • େᖒलҰ • Blog: http://blog.jicoman.info/ • Sansanגࣜձࣾ • σʔλԽγεςϜͷΠϯϑϥΛ୲౰ • Πϯϑϥߏஙɾӡ༻ɾվળɺ։ൃج൫ͷվળͳͲ • લ৬͸WebΤϯδχΞ(PHPer)ͱͯ͠ܞଳίϯςϯπαΠτͷ։ൃ΍AWS΁ͷ Ҡߦͱ͔΍͍ͬͯ·ͨ͠ 2/64
3. ຊεϥΠυʹ͍ͭͯ ։ൃऀͱΠϯϑϥ୲౰ͱͷϓϩδΣΫτͱͯ͠औΓ૊ΜͩҰࣄྫ ʹ͍͓ͭͯ࿩͍͖ͤͯͨͩ͞·͢ɻ։ൃج൫ͷվળࣄྫͱͯ͠গ ͠Ͱ΋ࢀߟʹͳΕΕ͹޾͍Ͱ͢ɻ ಺༰తʹ৽͘͠ͳ͍͔΋͠Εͳ͍͚Ͳ͝༰͍ࣻͩ͘͞! ࣮૷಺༰ͳͲ͸εϥΠυͷ࠷ޙʹ͋Γ·͢ͷͰ͝ࢀߟ·Ͱʹ 3/64
4. ΞδΣϯμ • ͳͥσϓϩΠͷվળʹऔΓ૊Μͩͷ͔ • γεςϜߏ੒ • ैདྷͷσϓϩΠͱ՝୊఺ • վળ΁ͷऔΓ૊Έ • ৽σϓϩΠΞʔΩςΫνϟ • ·ͱΊ 4/64
5. ๏ਓ޲໊͚ࢗ؅ཧαʔϏεʮSansanʯ 5/64
6. ݸਓ޲໊͚ࢗΞϓϦʮEightʯ 6/64
7. Data Strategy & Operation Center • ໊ࢗͷσʔλԽ&σʔλ׆༻ • Development Group • ໊ࢗͷσʔλԽͷγεςϜʮGEESʯશମͷ։ൃɾӡ༻ • ओʹRuby on RailsͰ։ൃ • R&D Group • ໊ࢗͷσʔλԽͷࣗಈԽͷͨΊʹඞཁͳը૾ೝࣝɺػցֶशɺςΩετղੳͳͲ • σʔλαΠΤϯςΟετ΋ࡏ੶͠ɺσʔλ׆༻ʹ஫ྗ • ओͳ։ൃݴޠ: C#, Python • Πϯϑϥ୲౰͸Devlopment Groupʹॴଐ͍ͯ͠Δ͕ɺR&D Group΋ݟ͍ͯΔ • 3໊(಺Ұਓ͸։ൃͱ݉຿) 7/64
8. ։ൃऀͱΠϯϑϥ୲౰ͱͷؔΘΓ • جຊతʹ։ൃऀͱҰॹʹϓϩδΣΫτʹೖΔ͜ͱ͸ͳ͍ • ΠϯϑϥνʔϜͱͯ͠ผϓϩδΣΫτΛ૸Β͍ͤͯΔ͜ͱ͕ଟ͍ • ґཔɾ૬ஊϕʔεͰͷؔΘΓ͕࠷΋ଟ͍ • αʔόߏஙɺ˓˓Λ࢖͍͍ͨ etc • ΞϥʔτରԠ͸։ൃऀɾΠϯϑϥ୲౰ • ։ൃऀ͸Πϯϑϥͷίʔυ(Chef, TerraformͳͲ)Λ΄ͱΜͲ͍͡Βͳ͍ • Πϯϑϥ୲౰͸αʔϏεͷιʔείʔυΛ͍͡Βͳ͍ 8/64
9. ͳͥσϓϩΠͷվળʹ औΓ૊Μͩͷ͔ 9/64
10. ݩʑ͸ҰϓϩδΣΫτͷҰ؀ • εςʔδϯά؀ڥͷࣗಈߏஙϓϩδΣΫτ • ༻్ʹ߹Θͤͨෳ਺ͷεςʔδϯά؀ڥ͕ඞཁͩͬͨ • TerraformΛ׆༻ͯࣗ͠ಈతʹߏஙͱഁյ͕Ͱ͖ΔΑ͏ʹ͢Δ • ࣗಈߏங͢Δʹ͸σϓϩΠͷࣗಈԽ΋ඞཁ • ैདྷͷσϓϩΠͰ͸ࣗಈԽ͕೉͔ͬͨ͠ • ຊ൪؀ڥʹ͓͚ΔσϓϩΠʹ՝୊Λײ͍ͯͨͨ͡ΊվળʹऔΓ૊Ή͜ͱʹͨ͠ • ݩʑ͸ΠϯϑϥνʔϜ಺ͷϓϩδΣΫτ͕ͩͬͨɺσϓϩΠʹؔͯ͠෼͔Βͳ͍͜ͱ͕ ଟ͍ͷͰ։ൃऀʹڠྗͯ͠΋Βͬͨ 10/64
11. γεςϜߏ੒ 11/64
12. αʔόߏ੒ • ϩʔϧ • WebɺBatchɺDeploy etc • Webαʔό • ELB + EC2(c3.xlarge × 20਺୆) • Batchαʔό • EC2(r3.xlarge × े਺୆) • Deployαʔό • Web/BatchαʔόʹσϓϩΠ 12/64
13. ΞϓϦέʔγϣϯ • Web/Batchαʔόͱ΋ʹ20ऑͷRailsΞϓϦ͕ಉډ! • ίετ࡟ݮ • සൟʹϦΫΤετ͞Εͳ͍αʔϏε • Apache + Passenger 13/64
14. ैདྷͷσϓϩΠͱ՝୊఺ 14/64
15. ैདྷͷσϓϩΠ • ఆظϦϦʔε(िҰ) • ͢΂ͯͷRailsΞϓϦΛσϓϩΠ • ඞཁʹԠͯ͡hotfixϦϦʔε(िʹ਺ճ) • ඞཁͳΞϓϦ͚ͩσϓϩΠ • Web/Batchαʔόશ୆Ұ੪ʹσϓϩΠ • σϓϩΠαʔό͔Β࣮ߦ • PushܕσϓϩΠ/In place/All at once 15/64
16. ैདྷͷσϓϩΠ • σϓϩΠεΫϦϓτ(Rakefile)ͰσϓϩΠ࣮ߦ • Capistrano 2ϕʔε • ΞϓϦຖʹCapλεΫϑΝΠϧͷੜ੒ • ฒྻͰෳ਺ΞϓϦΛσϓϩΠ • ϩʔϧ͸Chef ServerͷλάͰ؅ཧ • Web, BatchͰॲཧ͕ҟͳΔͨΊϩʔϧͰ۠ผ • knife tagίϚϯυͰ෇͚֎͢͠Δ 16/64
17. ՝୊఺ 1. σϓϩΠʹ͕͔͔࣌ؒΔ 2. ಛఆͷαʔό͚ͩσϓϩΠͰ͖ͳ͍ 3. GitHub/RubyGems͕མͪΔͱσϓϩΠͰ͖ͳ͍ 4. ࣗಈԽͮ͠Β͍ 17/64
18. ͕͔͔࣌ؒΔ • ֤αʔόͰBundleΠϯετʔϧͱΞηοτϓϦίϯύΠϧ͢Δ ͨΊɺ΋ͷ͕͔͔࣌ؒ͘͢͝Δ • ஗͍࣌ʹ͸໿30෼͔͔Δ • ్தͰίέΔͱσϓϩΠ΍Γ௚͠(·ͨ30෼͔͔Δ) • ༧ఆ௨ΓʹऴΘΔ͔ϋϥϋϥ͢Δ͜ͱ΋͠͹͠͹ 18/64
19. ಛఆͷαʔό͚ͩσϓϩΠͰ͖ͳ͍ • ಛఆͷαʔό͚ͩσϓϩΠ͢ΔͱଞͷαʔόͱϑΝΠϧύε͕ҟͳͬͯ͠·͏ • ϓϦίϯύΠϧ͞Εͨը૾/CSS/JavaScript • ͦͷαʔό΁ϦΫΤετ͕དྷΔͱ404 NotFound • ଞαʔό͔Βtar.gzͰݻΊͯ࣋ͬͯ͘ΔͷͰ͕͔͔࣌ؒΔ • ৽نαʔόͷ௥Ճʹ͙͢ʹରԠͰ͖ͳ͍ • ෛՙ૿Ճ࣌ʹ͙͢ʹεέʔϧΞ΢τͰ͖ͣɺରԠ͕ऴΘΔࠒʹෛՙ͕མͪண ͍͍ͯΔ͜ͱ΋͋ͬͨ 19/64
20. GitHub/RubyGems͕མͪΔͱσϓϩΠࣦഊ • GitHub͕ͨ·ʹ(?)DDOS߈ܸΛड͚ͯgit pullͰ͖ͳ͍ • rubygems.org͕མͪΔͱBundleΠϯετʔϧͰ͖ͳ͍ • ճ෮͢ΔͷΛ଴͔ͭ͠ͳ͘ਆཔΈ! 20/64
21. ࣗಈԽͮ͠Β͍ • Chef Server΁ͷϗετొ࿥ͱ࡟আ͕໘౗ • αʔόΛ௥Ճ͢Δͱ͖͸knife-ec2Λ࢖͏͕ɺࣗಈ௥Ճ(ΦʔτεέʔϧͳͲ)ͩͱ໘౗ • αʔό࡟আ͢Δͱ͖͸खಈͰϗετͱλάΛ࡟আ • ϩʔϧͷλά෇͚֎͠͸Chef Serverʹґଘ • knife tagίϚϯυ΋खಈ • ίϚϯυ࣮ߦͯ͠΋͙͢ʹ൓ө͞Εͳ͍ • ΠϯϑϥνʔϜͱͯ͠Chef ServerΛࣺ͔ͯͨͬͨ 21/64
22. վળ΁ͷऔΓ૊Έ 22/64
23. σϓϩΠπʔϧબఆͷํ਑ • PullܕσϓϩΠ • αʔόࣗ਎͕σϓϩΠ͢Δ • σϓϩΠ͕࣌ؒҰఆ(୆਺ʹൺྫ͠ͳ͍) • Amazon S3ʹϏϧυࡁͷϑΝΠϧΛ഑ஔ • μ΢ϯϩʔυ͠഑ஔɺ࠶ىಈ͢Δ͚ͩͰσϓϩΠ׬ྃ • S3ͷՄ༻ੑ͸99.99ˋ • In place/All at once • Blue/GreenσϓϩΠͩͱ࢓૊Έ͕ෳࡶԽ͢ΔͨΊҰ୴ݟૹΓ • One by one(1୆ͣͭ)ͩͱ͕͔͔࣌ؒΔ • ϗετͷొ࿥ɾ࡟আ͸ࣗಈతʹߦ͍͍ͨ 23/64
24. AWS CodeDeploy • AWSͷσϓϩΠαʔϏε • ΤʔδΣϯτʹΑͬͯPullܕσϓϩΠ͕Մೳ • AWSଆͰόʔδϣϯ؅ཧ͞Ε͍ͯΔͨΊɺϩʔϧόοΫ΋؆୯ • GUIͰσϓϩΠঢ়گ͕֬ೝͰ͖Δͷ͕ศར 24/64
25. AWS CodeDeploy • ࠾༻Λݟૹͬͨཧ༝! • ෳ਺ΞϓϦΛҰ੪ʹσϓϩΠ͢Δͱ஗͍(γϦΞϧʹ࣮ߦ͞ΕΔ) • All at onceͷ৔߹ɺ1୆Ͱ΋ࣦഊͨ͠ΒσϓϩΠࣦഊͱͳΒͳ͍ • σϓϩΠࣦഊʹͳΒͳ͍͔ΒϩʔϧόοΫͰ͖ͳ͍ • CodeDeployͷ૝ఆ͍ͯ͠Δ؀ڥͱࣗ෼ୡͷͱϚον͠ͳ͔ͬͨ 25/64
26. Stretcher • https://github.com/fujiwara/stretcher • Consulͱ૊Έ߹ΘͤΔ͜ͱͰɺΠϕϯτΛτϦΨʔͱͨ͠PullܕσϓϩΠ͕Մೳ • ࠾༻ͨ͠ཧ༝! • ෳ਺ΞϓϦΛҰ੪ʹσϓϩΠͯ͠΋ૣ͍ • 20ऑͷRailsΞϓϦΛฒྻσϓϩΠ͢Δͱ1ʙ2෼ఔ౓ͰऴΘΔ • ෳ਺ͷઃఆϑΝΠϧ(ϚχϑΣετϑΝΠϧ)Λ༻ҙ͢Δ͜ͱͰɺৼΔ෣͍Λม͑Δ͜ͱ͕Ͱ͖Δ • σϓϩΠϑΝΠϧ഑ஔ༻ • γϯϘϦοΫϦϯΫ੾ସ༻ • Capistranoͷ࢓૊ΈΛྲྀ༻ • Consulͱͷ਌࿨ੑ͕ߴ͍ 26/64
27. Consul • https://www.consul.io/ • HashiCorpࣾఏڙͷΦʔέετϨʔγϣϯπʔϧ • αʔόͱΫϥΠΞϯτͱ͍͏̎ͭͷ໾ׂͰߏ੒͞ΕɺΫϥελΛܗ੒͢Δ • ো֐ݕ஌ɺ಺෦DNSɺKey/ValueετΞ(KVS) • Πϕϯτൃߦ • ֤ConsulΫϥΠΞϯτ͸ΠϕϯτΛ଴ͪड͚͍ͯΔঢ়ଶͰɺಛఆͷΠϕϯτ͕ൃߦ͞ΕͨΒ೚ҙͷΞ ΫγϣϯΛ࣮ߦ • αʔϏελάʹΑΔϩʔϧ؅ཧ • ىಈ࣌ʹΫϥελϦϯά΁ࣗಈδϣΠϯ 27/64
28. طଘͷ࢓૊ΈΛ௥͏ • σϓϩΠεΫϦϓτΛಡΉ • ͘Θ͍͠ਓʹ֓ཁΛฉ͍ͯɺৄࡉ͸ಡΜͰཧղ • Capistrano 2ͷιʔείʔυ΋ซͤͯಡΉ • cap deployͰͲͷΑ͏ʹ࣮ߦ͞Ε͍ͯΔͷ͔ • ϝϯςφϯε੾ସɺϩʔϧόοΫͷ࢓૊Έ΋௥ͬͯΈΔ • ։ൃऀͰ΋෼͔Βͳ͍͜ͱ͕ҙ֎ͱଟ͍ 28/64
29. σϓϩΠεΫϦϓτͷվम • capistrano-stretcher • https://github.com/pepabo/capistrano-stretcher • Capistrano͔ΒStretcher + ConsulΛݺͼग़ͯ͠σϓϩΠͰ͖Δ • ͦͷ··࢖͍͔͚ͨͬͨͲͰ͖ͳ͔ͬͨŋŋŋ • Capistrano3 ରԠ • طଘͷσϓϩΠεΫϦϓτͰ͸ಠࣗίϚϯυ͕ଟ͍ • ࢀߟʹͭͭ͠ɺҰ͔ΒػೳΛ࣮૷ 29/64
30. େ͖͘ม͑͗͢ͳ͍ • େ͖ͳมߋ͸ֶशίετ͕ଟ͘ͳΔɺ৺ཧతোน͕ߴ͘ͳΔ • ࢖ͬͯ΋Β͏ͷ͸։ൃऀ • গͣͭ͠ม͍͑ͯ͘ • Ұؾʹม͑ͣʹɺͰ͖Δ͚ͩࠓ·Ͱͷํ๏Λอͪͳ͕Β • ௕ظͰվળ͍͚ͯ͠͹݁Ռͱͯ͠େ͖͘มΘ͍͚ͬͯΔ 30/64
31. ϨϏϡʔ/ϑΟʔυόοΫ • GitHub Flowʹଇͬͯ։ൃɺϨϏϡʔͯ͠΋Β͏ • ϦϑΝΫλϦϯά΍ςετ΋ॻ͍ͯ΋Β͑ͨ • σϓϩΠͯ͠΋ΒͬͯϑΟʔυόοΫ • ίϚϯυ໊ͷมߋɺ࢓༷ͷमਖ਼Λదٓमਖ਼ • SlackͰͷ͞Γ͛ͳ͍ͭͿ΍͖΋ݟಀ͞ͳ͍ 31/64
32. υΩϡϝϯτ࡞੒ • ֓ཁ • ৽͍͠σϓϩΠͷશମ૾Λ௫ΜͰ΋Β͏ • ίϚϯυͱ࣮ߦํ๏ • σϓϩΠ͢Δͱ͖ͷϚχϡΞϧ • Կ΋ߟ͑ͣʹίϐϖ͢Δ͚ͩͰσϓϩΠͰ͖ΔΑ͏ʹ • τϥϒϧγϡʔςΟϯά • ૝ఆ͞ΕΔΤϥʔ಺༰ͱͦͷରॲํ๏ • ࣗ෼͕͍ͳͯ͘΋ղܾͰ͖ΔΑ͏ʹ • ໰୊͕ىͬͨ͜ͱ͖ʹ୭Ͱ΋ରॲͰ͖ΔΑ͏ʹͳΔͷ͕ཧ૝త 32/64
33. આ໌͢Δ৔Λઃ͚Δ • ࢿྉΛಡΜͰ΋Β͏͚ͩͩͱ఻ΘΓͮΒ͍͜ͱ͕ଟ͍ • ৽͍͠σϓϩΠͷϝϦοτ • ࠓ·Ͱͷํ๏͔ΒͲͷΑ͏ʹมΘΔͷ͔ 33/64
34. ৽σϓϩΠΞʔΩςΫνϟ 34/64
35. ֓ཁ • ιʔείʔυΛϏϧυɺtar.gzͰݻΊͯɺS3ʹΞοϓϩʔυ • ϏϧυαʔόʹσϓϩΠ • ϩʔϧຖʹϚχϑΣετϑΝΠϧΛੜ੒ɺS3ʹΞοϓϩʔυ • consul eventͰΠϕϯτൃՐɺWeb/BatchαʔόͰStretcher࣮ߦ • σϓϩΠϑΝΠϧͷऔಘɺ഑ஔ • ϦϦʔεલ·ͰʹऴΘΒ͓ͤͯ͘ • γϯϘϦοΫϦϯΫ੾ସɺRails࠶ىಈ • ϦϦʔε࣌ʹ࣮ߦ(ϦϦʔε࣌ؒͷ୹ॖ) • σΟϨΫτϦߏ଄͸Capistranoͱಉ͡ 35/64
36. ߏ੒ 36/64
37. Consulͷઃఆ • ChefͰઃఆϑΝΠϧ(JSON)Λੜ੒ • ConsulΫϥΠΞϯτ • ϩʔϧλά • Πϕϯτ • σϓϩΠ • ϩʔϧλάͷ෇͚֎͠ 37/64
38. Ϗϧυαʔό΁ͷσϓϩΠ 38/64
39. ϚχϑΣετϑΝΠϧͷੜ੒ • ϚχϑΣετϑΝΠϧͷςϯϓϨʔτ(ERB) • update_code_#{application}_#{version}.yml • S3͔ΒσϓϩΠϑΝΠϧΛμ΢ϯϩʔυ͠ɺ഑ஔ • deploy_#{application}_#{role}_#{version}.yml • γϯϘϦοΫϦϯΫ੾ସɺRails࠶ىಈ 39/64
40. update_code_#{application}_#{version}.yml src: s3://<%= s3_bucket %>/<%= application %> checksum: <%= checksum %> dest: app/<%= application %>/release/YYYYMMDDHHMMSS commands: pre: - mkdir -p app/<%= application %> success: - echo 'success' failure: - echo 'failed!!' excludes: - "*.pid" - "*.socket" 40/64
41. deploy_#{application}_#{role}_#{version}.yml sync_strategy: mv commands: post: - ln -sfn "app/<%= application %>/release/YYYYMMDDHHMMSS" "app/<%= application %>/current" <%=# Rails࠶ىಈ %> - cd app/<%= application %>/release/YYYYMMDDHHMMSS && touch tmp/restart.txt <%=# ಈ࡞֬ೝ & ஆػӡస %> - <%= File.join(script_path, "pre-warming.sh") %> <%= application %> success: <%=# աڈͷϦϦʔεΛ࡟আ %> - <%= File.join(script_path, "cleanup.sh") %> <%= application %> - echo 'release success' failure: - echo 'release failed!!' 41/64
42. S3΁ͷΞοϓϩʔυ 42/64
43. ΞϓϦέʔγϣϯαʔόʹ഑෍ 43/64
44. σϓϩΠ(γϯϘϦοΫϦϯΫ੾ସ) 44/64
45. Consul KVSͰόʔδϣϯ؅ཧ • σϓϩΠ੒ޭޙɺConsul KVSʹ੒ޭͨ͠࠷৽ͱ1ͭલͷόʔδϣϯΛ ߋ৽͢Δ • http://127.0.0.1:8500/v1/kv/#{application}/current • http://127.0.0.1:8500/v1/kv/#{application}/previous • όʔδϣϯ͸ϦϦʔεσΟϨΫτϦͷ೔෇( YYYYMMDDHHMMSS ) • αʔό௥Ճ͢Δͱ͖͸࠷৽ɺϩʔϧόοΫ͢Δͱ͖͸1ͭલͷόʔδ ϣϯΛࢦఆ͢Δ 45/64
46. ϩʔϧόοΫ • 1ͭલͷόʔδϣϯͷσϓϩΠϑΝΠϧ͕αʔόʹ͋Δͱ͸ݶΒ ͳ͍ • σϓϩΠʹࣦഊͨ͠৔߹͸ɺ1ͭલͷόʔδϣϯΛσϓϩΠ͢Δ • σϓϩΠϑΝΠϧͷ഑෍ͱγϯϘϦοΫϦϯΫ੾ସͷ྆ํΛߦ ͏ 46/64
47. ·ͱΊͯσϓϩΠ͢Δ ͢΂ͯͷΞϓϦΛϚϧνεϨουͰฒྻσϓϩΠ task deploy_all: :setup do deploy_projects = "application_A,apllication_B,..." threads_count = ENV['THREADS'] Parallel.map(deploy_projects, in_threads: threads_count.to_i) do |project| # Ϗϧυαʔό΁ͷσϓϩΠ # ϚχϑΣετϑΝΠϧͷੜ੒ # S3΁ͷΞοϓϩʔυ # ΞϓϦέʔγϣϯαʔόʹ഑෍ end end task release_all: :setup do deploy_projects = "application_A,apllication_B,..." threads_count = ENV['THREADS'] Parallel.map(deploy_projects, in_threads: threads_count.to_i) do |project| # σϓϩΠ(γϯϘϦοΫϦϯΫ੾ସ) end end 47/64
48. Slack SlackͰσϓϩΠঢ়گ͕֬ೝͰ͖Δ ໰୊͕ੜͨ͡৔߹͸όοΫτϨʔε͕ද ࣔ 48/64
49. ݁Ռ 1. σϓϩΠʹ͕͔͔࣌ؒΔ • => ϦϦʔε࣌ؒͷ୹ॖ(30෼->5෼) 2. ಛఆͷαʔό͚ͩσϓϩΠͰ͖ͳ͍ • => ࠷৽ͷσϓϩΠϑΝΠϧΛ࣋ͬͯ͘Δ͚ͩ 3. GitHub/RubyGems͕མͪΔͱσϓϩΠͰ͖ͳ͍ • => S3ͳͷͰ΄΅໰୊ͳ͠ 4. ࣗಈԽͮ͠Β͍ • => ConsulʹΑͬͯΫϥελϦϯά΁ͷࣗಈδϣΠϯ/୤ୀ͕Մೳ 49/64
50. تͼͷ੠ ! 50/64
51. ·ͱΊ • Πϯϑϥ୲౰͕σϓϩΠվળΛͨ͜͠ͱͰ։ൃج൫ʹߩݙͰ͖ͨ • ։ൃͱΠϯϑϥͷ྆ํΛ஌Βͳ͍ͱվળͰ͖ͳ͍ྖҬ͕͋Δ • Πϯϑϥ͚ͩͰͰ͖Δ෼໺͸গͳ͍ • ։ൃ΋Ͱ͖ΔΠϯϑϥ(ࣗ෼΋ΩϟϦΞతଆ໘΋ؚΊͯ) • օʹͱͬͯ޾ͤͳಓΛબͿ • ։ൃଆͱΠϯϑϥଆͷཁٻ͕Ϛον͢Ε͹ϕετ • গͣͭ͠ม͍͑ͯ͘ • Ұؾʹ΍ΓํΛม͑ͣʹɺͰ͖Δ͚ͩࠓ·Ͱͷํ๏Λอͪͳ͕Β • ௕ظؒվળ͍͚ͯ͠͹݁Ռͱͯ͠େ͖͘มΘ͍͚ͬͯΔ 51/64
52. ࠓޙͷ༧ఆ • Φʔτεέʔϧ(ઈࢍରԠத) • όοναʔόΛAmazon ECS΁Ҡߦ • Webαʔόʹಉډ͍ͯ͠ΔRailsΞϓϦͷ෼ׂ • SlackίϚϯυʹΑΔσϓϩΠ 52/64
53. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ 53/64
54. ͓·͚
55. Consul - ΫϥΠΞϯτ # /etc/consul.d/settings.json { "data_dir": "/opt/consul", "log_level": "err", "start_join": [ "consul-server001", "consul-server002", "consul-server003" ] } 55/64
56. Consul - ϩʔϧλά # /etc/consul.d/role.json { "service": { "name": "role", "tags": [ # ෳ਺ઃఆՄ "web", "newrelic_monitoring" ] } } 56/64
57. Consul - Πϕϯτ(σϓϩΠ) # /etc/consul.d/deploy.json { "watches": [ { "type": "event", "name": "deploy_application_A", "handler": "export TMPDIR=/ephemeral0; stretcher -retry 3 >> $TMPDIR/deploy_application_A.log 2>&1" }, { "type": "event", "name": "deploy_application_B", "handler": "export TMPDIR=/ephemeral0; stretcher -retry 3 >> $TMPDIR/deploy_application_B.log 2>&1" }, ... } • consul eventίϚϯυͰΠϕϯτൃՐ • consul event -service=role -tag=#{role} -name=deploy_#{application} #{manifest_file} • σϓϩΠϑΝΠϧͷμ΢ϯϩʔυઌ(TMPDIR)ΛEC2ͷΠϯελϯεετΞʹมߋ 57/64
58. ϩʔϧ # ϩʔϧͷࢦఆ role :web, *consul_role('web') role :batch, *consul_role('batch') # ϩʔϧࢦఆ͞Εͨϗετͷऔಘ def consul_role(roles) response = `curl -fsSL http://127.0.0.1:8500/v1/catalog/service/role` nodes = JSON.parse(response) lists = [] nodes.each do |node| if (node['ServiceTags'] & roles.split(',')) == roles.split(',') lists << node['Address'] end end lists end 58/64
59. Consul - Πϕϯτ(ϩʔϧλάͷ෇͚֎͠) # /etc/consul.d/change_role.json { "watches": [ { "type": "event", "name": "change_role", "handler": "change_consul_role.rb >> /tmp/change_log.log 2>&1" } ] } #!/bin/env ruby require "json" consul_url = "http://127.0.0.1:8500" response = %x(curl -fsSL #{consul_url}/v1/catalog/node/$(curl -fsSL #{consul_url}/v1/agent/self | jq -r .Member.Name)) node = JSON.parse(response) new_role = node["Services"]["role"]["Tags"] consul_role_file_path = "/etc/consul.d/role.json" if ! File.exist?(consul_role_file_path) then exit 0 end consul_role = open(consul_role_file_path) do |io| JSON.load(io) end if consul_role["service"]["tags"] != new_role then consul_role["service"]["tags"] = new_role open(consul_role_file_path, 'w') do |io| io.puts(JSON.pretty_generate(consul_role)) end %x(consul reload) end 59/64
60. S3΁ͷΞοϓϩʔυ def upload2s3(local_src_path, bucket, path) run "aws s3api put-object --region #{aws_region} --bucket #{bucket} --key #{path} --body #{local_src_path}" end def create_tarball run "cd #{latest_release} && rm -rf #{tarball_name} && shopt -s dotglob && tar zcf #{tarball_name} *" end def upload_tarball upload2s3("#{latest_release}/#{tarball_name}", app_s3_bucket, "#{app_s3_path}/#{tarball_name}") end
61. σϓϩΠϑΝΠϧͷ഑෍ namespace :stretcher do task :update_code, roles: :build do roles.split(',').each do |role| manifest_file = "update_code_#{application}_#{version}.yml" capture "consul event -service=role -tag=#{role} -name=deploy_#{application} #{manifest_file}" end puts 'Finished(Not released yet!)' end end
62. ϦϦʔε(γϯϘϦοΫϦϯΫ੾ସ) namespace :stretcher do task :release, roles: :build do transaction do on_rollback do rollback end roles.split(',').each do |role| manifest_file = "deploy_#{application}_#{role}_#{version}.yml" capture "consul event -service=role -tag=#{role} -name=deploy_#{application} #{manifest_file}" end unless check_event_success('release_failure') puts 'Release Failed!! & Starting rolling back.' exit 1 end puts 'Release finished.' end end end
63. όʔδϣϯߋ৽ ConsulͷKVSʹ֨ೲ͍ͯ͠Δόʔδϣϯ஋Λߋ৽ after 'stretcher:release', 'stretcher:update_release_version' namespace :stretcher do task :update_release_version, roles: :build do # લճ੒ޭͨ͠σϓϩΠͷόʔδϣϯΛpreviousʹ͢Δ capture "curl -sS -X PUT -d '#{current_version}'http://127.0.0.1:8500/v1/kv/#{application}/previous" # ੒ޭͨ͠σϓϩΠͷόʔδϣϯΛߋ৽͢Δ capture "curl -sS -X PUT -d '#{latest_version)}' http://127.0.0.1:8500/v1/kv/#{application}/current" end end
64. ϩʔϧόοΫ namespace :stretcher do task :rollback, roles: :build do fetch(:deploy_roles).split(',').each do |target_role| manifest_file = "update_code_#{application}_#{previous_version}.yml" capture "consul event -service=role -tag=#{target_role} -name=deploy_#{application} #{manifest_file}" end reset_release_status fetch(:deploy_roles).split(',').each do |target_role| manifest_file = "deploy_#{application}_#{role}_#{previous_version}.yml" capture "/usr/local/bin/consul event -service=role -tag=#{target_role} -name=deploy_#{application} #{manifest_file}" end end