インフラエンジニアがConsulとStretcherをつかったデプロイ改善で開発効率の向上に貢献した話 / jtf2017-consul-stretcher-deploy

ΠϯϑϥΤϯδχΞ͕
ConsulͱStretcherΛ͔ͭͬͨσϓϩΠվળͰ
։ൃޮ཰ͷ޲্ʹߩݙͨ͠࿩
July Tech Festa 2017
Shuichi Ohsawa (@ohsawa0515)
1/64

View Slide

ࣗݾ঺հ
• େᖒलҰ
• Blog: http://blog.jicoman.info/
• Sansanגࣜձࣾ
• σʔλԽγεςϜͷΠϯϑϥΛ୲౰
• Πϯϑϥߏஙɾӡ༻ɾվળɺ։ൃج൫ͷվળͳͲ
• લ৬͸WebΤϯδχΞ(PHPer)ͱͯ͠ܞଳίϯςϯπαΠτͷ։ൃ΍AWS΁ͷ
Ҡߦͱ͔΍͍ͬͯ·ͨ͠
2/64

View Slide

ຊεϥΠυʹ͍ͭͯ
։ൃऀͱΠϯϑϥ୲౰ͱͷϓϩδΣΫτͱͯ͠औΓ૊ΜͩҰࣄྫ
ʹ͍͓ͭͯ࿩͍͖ͤͯͨͩ͞·͢ɻ։ൃج൫ͷվળࣄྫͱͯ͠গ
͠Ͱ΋ࢀߟʹͳΕΕ͹޾͍Ͱ͢ɻ
಺༰తʹ৽͘͠ͳ͍͔΋͠Εͳ͍͚Ͳ͝༰͍ࣻͩ͘͞!
࣮૷಺༰ͳͲ͸εϥΠυͷ࠷ޙʹ͋Γ·͢ͷͰ͝ࢀߟ·Ͱʹ
3/64

View Slide

ΞδΣϯμ
• ͳͥσϓϩΠͷվળʹऔΓ૊Μͩͷ͔
• γεςϜߏ੒
• ैདྷͷσϓϩΠͱ՝୊఺
• վળ΁ͷऔΓ૊Έ
• ৽σϓϩΠΞʔΩςΫνϟ
• ·ͱΊ
4/64

View Slide

๏ਓ޲໊͚ࢗ؅ཧαʔϏεʮSansanʯ
5/64

View Slide

ݸਓ޲໊͚ࢗΞϓϦʮEightʯ
6/64

View Slide

Data Strategy & Operation Center
• ໊ࢗͷσʔλԽ&σʔλ׆༻
• Development Group
• ໊ࢗͷσʔλԽͷγεςϜʮGEESʯશମͷ։ൃɾӡ༻
• ओʹRuby on RailsͰ։ൃ
• R&D Group
• ໊ࢗͷσʔλԽͷࣗಈԽͷͨΊʹඞཁͳը૾ೝࣝɺػցֶशɺςΩετղੳͳͲ
• σʔλαΠΤϯςΟετ΋ࡏ੶͠ɺσʔλ׆༻ʹ஫ྗ
• ओͳ։ൃݴޠ: C#, Python
• Πϯϑϥ୲౰͸Devlopment Groupʹॴଐ͍ͯ͠Δ͕ɺR&D Group΋ݟ͍ͯΔ
• 3໊(಺Ұਓ͸։ൃͱ݉຿)
7/64

View Slide

։ൃऀͱΠϯϑϥ୲౰ͱͷؔΘΓ
• جຊతʹ։ൃऀͱҰॹʹϓϩδΣΫτʹೖΔ͜ͱ͸ͳ͍
• ΠϯϑϥνʔϜͱͯ͠ผϓϩδΣΫτΛ૸Β͍ͤͯΔ͜ͱ͕ଟ͍
• ґཔɾ૬ஊϕʔεͰͷؔΘΓ͕࠷΋ଟ͍
• αʔόߏஙɺ˓˓Λ࢖͍͍ͨ etc
• ΞϥʔτରԠ͸։ൃऀɾΠϯϑϥ୲౰
• ։ൃऀ͸Πϯϑϥͷίʔυ(Chef, TerraformͳͲ)Λ΄ͱΜͲ͍͡Βͳ͍
• Πϯϑϥ୲౰͸αʔϏεͷιʔείʔυΛ͍͡Βͳ͍
8/64

View Slide

ͳͥσϓϩΠͷվળʹ
औΓ૊Μͩͷ͔
9/64

View Slide

ݩʑ͸ҰϓϩδΣΫτͷҰ؀
• εςʔδϯά؀ڥͷࣗಈߏஙϓϩδΣΫτ
• ༻్ʹ߹Θͤͨෳ਺ͷεςʔδϯά؀ڥ͕ඞཁͩͬͨ
• TerraformΛ׆༻ͯࣗ͠ಈతʹߏஙͱഁյ͕Ͱ͖ΔΑ͏ʹ͢Δ
• ࣗಈߏங͢Δʹ͸σϓϩΠͷࣗಈԽ΋ඞཁ
• ैདྷͷσϓϩΠͰ͸ࣗಈԽ͕೉͔ͬͨ͠
• ຊ൪؀ڥʹ͓͚ΔσϓϩΠʹ՝୊Λײ͍ͯͨͨ͡ΊվળʹऔΓ૊Ή͜ͱʹͨ͠
• ݩʑ͸ΠϯϑϥνʔϜ಺ͷϓϩδΣΫτ͕ͩͬͨɺσϓϩΠʹؔͯ͠෼͔Βͳ͍͜ͱ͕
ଟ͍ͷͰ։ൃऀʹڠྗͯ͠΋Βͬͨ
10/64

View Slide

γεςϜߏ੒
11/64

View Slide

αʔόߏ੒
• ϩʔϧ
• WebɺBatchɺDeploy etc
• Webαʔό
• ELB + EC2(c3.xlarge × 20਺୆)
• Batchαʔό
• EC2(r3.xlarge × े਺୆)
• Deployαʔό
• Web/BatchαʔόʹσϓϩΠ
12/64

View Slide

ΞϓϦέʔγϣϯ
• Web/Batchαʔόͱ΋ʹ20ऑͷRailsΞϓϦ͕ಉډ!
• ίετ࡟ݮ
• සൟʹϦΫΤετ͞Εͳ͍αʔϏε
• Apache + Passenger
13/64

View Slide

ैདྷͷσϓϩΠͱ՝୊఺
14/64

View Slide

ैདྷͷσϓϩΠ
• ఆظϦϦʔε(िҰ)
• ͢΂ͯͷRailsΞϓϦΛσϓϩΠ
• ඞཁʹԠͯ͡hotﬁxϦϦʔε(िʹ਺ճ)
• ඞཁͳΞϓϦ͚ͩσϓϩΠ
• Web/Batchαʔόશ୆Ұ੪ʹσϓϩΠ
• σϓϩΠαʔό͔Β࣮ߦ
• PushܕσϓϩΠ/In place/All at once
15/64

View Slide

ैདྷͷσϓϩΠ
• σϓϩΠεΫϦϓτ(Rakeﬁle)ͰσϓϩΠ࣮ߦ
• Capistrano 2ϕʔε
• ΞϓϦຖʹCapλεΫϑΝΠϧͷੜ੒
• ฒྻͰෳ਺ΞϓϦΛσϓϩΠ
• ϩʔϧ͸Chef ServerͷλάͰ؅ཧ
• Web, BatchͰॲཧ͕ҟͳΔͨΊϩʔϧͰ۠ผ
• knife tagίϚϯυͰ෇͚֎͢͠Δ
16/64

View Slide

՝୊఺
1. σϓϩΠʹ͕͔͔࣌ؒΔ
2. ಛఆͷαʔό͚ͩσϓϩΠͰ͖ͳ͍
3. GitHub/RubyGems͕མͪΔͱσϓϩΠͰ͖ͳ͍
4. ࣗಈԽͮ͠Β͍
17/64

View Slide

͕͔͔࣌ؒΔ
• ֤αʔόͰBundleΠϯετʔϧͱΞηοτϓϦίϯύΠϧ͢Δ
ͨΊɺ΋ͷ͕͔͔࣌ؒ͘͢͝Δ
• ஗͍࣌ʹ͸໿30෼͔͔Δ
• ్தͰίέΔͱσϓϩΠ΍Γ௚͠(·ͨ30෼͔͔Δ)
• ༧ఆ௨ΓʹऴΘΔ͔ϋϥϋϥ͢Δ͜ͱ΋͠͹͠͹
18/64

View Slide

ಛఆͷαʔό͚ͩσϓϩΠͰ͖ͳ͍
• ಛఆͷαʔό͚ͩσϓϩΠ͢ΔͱଞͷαʔόͱϑΝΠϧύε͕ҟͳͬͯ͠·͏
• ϓϦίϯύΠϧ͞Εͨը૾/CSS/JavaScript
• ͦͷαʔό΁ϦΫΤετ͕དྷΔͱ404 NotFound
• ଞαʔό͔Βtar.gzͰݻΊͯ࣋ͬͯ͘ΔͷͰ͕͔͔࣌ؒΔ
• ৽نαʔόͷ௥Ճʹ͙͢ʹରԠͰ͖ͳ͍
• ෛՙ૿Ճ࣌ʹ͙͢ʹεέʔϧΞ΢τͰ͖ͣɺରԠ͕ऴΘΔࠒʹෛՙ͕མͪண
͍͍ͯΔ͜ͱ΋͋ͬͨ
19/64

View Slide

GitHub/RubyGems͕མͪΔͱσϓϩΠࣦഊ
• GitHub͕ͨ·ʹ(?)DDOS߈ܸΛड͚ͯgit pullͰ͖ͳ͍
• rubygems.org͕མͪΔͱBundleΠϯετʔϧͰ͖ͳ͍
• ճ෮͢ΔͷΛ଴͔ͭ͠ͳ͘ਆཔΈ!
20/64

View Slide

ࣗಈԽͮ͠Β͍
• Chef Server΁ͷϗετొ࿥ͱ࡟আ͕໘౗
• αʔόΛ௥Ճ͢Δͱ͖͸knife-ec2Λ࢖͏͕ɺࣗಈ௥Ճ(ΦʔτεέʔϧͳͲ)ͩͱ໘౗
• αʔό࡟আ͢Δͱ͖͸खಈͰϗετͱλάΛ࡟আ
• ϩʔϧͷλά෇͚֎͠͸Chef Serverʹґଘ
• knife tagίϚϯυ΋खಈ
• ίϚϯυ࣮ߦͯ͠΋͙͢ʹ൓ө͞Εͳ͍
• ΠϯϑϥνʔϜͱͯ͠Chef ServerΛࣺ͔ͯͨͬͨ
21/64

View Slide

վળ΁ͷऔΓ૊Έ
22/64

View Slide

σϓϩΠπʔϧબఆͷํ਑
• PullܕσϓϩΠ
• αʔόࣗ਎͕σϓϩΠ͢Δ
• σϓϩΠ͕࣌ؒҰఆ(୆਺ʹൺྫ͠ͳ͍)
• Amazon S3ʹϏϧυࡁͷϑΝΠϧΛ഑ஔ
• μ΢ϯϩʔυ͠഑ஔɺ࠶ىಈ͢Δ͚ͩͰσϓϩΠ׬ྃ
• S3ͷՄ༻ੑ͸99.99ˋ
• In place/All at once
• Blue/GreenσϓϩΠͩͱ࢓૊Έ͕ෳࡶԽ͢ΔͨΊҰ୴ݟૹΓ
• One by one(1୆ͣͭ)ͩͱ͕͔͔࣌ؒΔ
• ϗετͷొ࿥ɾ࡟আ͸ࣗಈతʹߦ͍͍ͨ
23/64

View Slide

AWS CodeDeploy
• AWSͷσϓϩΠαʔϏε
• ΤʔδΣϯτʹΑͬͯPullܕσϓϩΠ͕Մೳ
• AWSଆͰόʔδϣϯ؅ཧ͞Ε͍ͯΔͨΊɺϩʔϧόοΫ΋؆୯
• GUIͰσϓϩΠঢ়گ͕֬ೝͰ͖Δͷ͕ศར
24/64

View Slide

AWS CodeDeploy
• ࠾༻Λݟૹͬͨཧ༝!
• ෳ਺ΞϓϦΛҰ੪ʹσϓϩΠ͢Δͱ஗͍(γϦΞϧʹ࣮ߦ͞ΕΔ)
• All at onceͷ৔߹ɺ1୆Ͱ΋ࣦഊͨ͠ΒσϓϩΠࣦഊͱͳΒͳ͍
• σϓϩΠࣦഊʹͳΒͳ͍͔ΒϩʔϧόοΫͰ͖ͳ͍
• CodeDeployͷ૝ఆ͍ͯ͠Δ؀ڥͱࣗ෼ୡͷͱϚον͠ͳ͔ͬͨ
25/64

View Slide

Stretcher
• https://github.com/fujiwara/stretcher
• Consulͱ૊Έ߹ΘͤΔ͜ͱͰɺΠϕϯτΛτϦΨʔͱͨ͠PullܕσϓϩΠ͕Մೳ
• ࠾༻ͨ͠ཧ༝!
• ෳ਺ΞϓϦΛҰ੪ʹσϓϩΠͯ͠΋ૣ͍
• 20ऑͷRailsΞϓϦΛฒྻσϓϩΠ͢Δͱ1ʙ2෼ఔ౓ͰऴΘΔ
• ෳ਺ͷઃఆϑΝΠϧ(ϚχϑΣετϑΝΠϧ)Λ༻ҙ͢Δ͜ͱͰɺৼΔ෣͍Λม͑Δ͜ͱ͕Ͱ͖Δ
• σϓϩΠϑΝΠϧ഑ஔ༻
• γϯϘϦοΫϦϯΫ੾ସ༻
• Capistranoͷ࢓૊ΈΛྲྀ༻
• Consulͱͷ਌࿨ੑ͕ߴ͍
26/64

View Slide

Consul
• https://www.consul.io/
• HashiCorpࣾఏڙͷΦʔέετϨʔγϣϯπʔϧ
• αʔόͱΫϥΠΞϯτͱ͍͏̎ͭͷ໾ׂͰߏ੒͞ΕɺΫϥελΛܗ੒͢Δ
• ো֐ݕ஌ɺ಺෦DNSɺKey/ValueετΞ(KVS)
• Πϕϯτൃߦ
• ֤ConsulΫϥΠΞϯτ͸ΠϕϯτΛ଴ͪड͚͍ͯΔঢ়ଶͰɺಛఆͷΠϕϯτ͕ൃߦ͞ΕͨΒ೚ҙͷΞ
ΫγϣϯΛ࣮ߦ
• αʔϏελάʹΑΔϩʔϧ؅ཧ
• ىಈ࣌ʹΫϥελϦϯά΁ࣗಈδϣΠϯ
27/64

View Slide

طଘͷ࢓૊ΈΛ௥͏
• σϓϩΠεΫϦϓτΛಡΉ
• ͘Θ͍͠ਓʹ֓ཁΛฉ͍ͯɺৄࡉ͸ಡΜͰཧղ
• Capistrano 2ͷιʔείʔυ΋ซͤͯಡΉ
• cap deployͰͲͷΑ͏ʹ࣮ߦ͞Ε͍ͯΔͷ͔
• ϝϯςφϯε੾ସɺϩʔϧόοΫͷ࢓૊Έ΋௥ͬͯΈΔ
• ։ൃऀͰ΋෼͔Βͳ͍͜ͱ͕ҙ֎ͱଟ͍
28/64

View Slide

σϓϩΠεΫϦϓτͷվम
• capistrano-stretcher
• https://github.com/pepabo/capistrano-stretcher
• Capistrano͔ΒStretcher + ConsulΛݺͼग़ͯ͠σϓϩΠͰ͖Δ
• ͦͷ··࢖͍͔͚ͨͬͨͲͰ͖ͳ͔ͬͨŋŋŋ
• Capistrano3 ରԠ
• طଘͷσϓϩΠεΫϦϓτͰ͸ಠࣗίϚϯυ͕ଟ͍
• ࢀߟʹͭͭ͠ɺҰ͔ΒػೳΛ࣮૷
29/64

View Slide

େ͖͘ม͑͗͢ͳ͍
• େ͖ͳมߋ͸ֶशίετ͕ଟ͘ͳΔɺ৺ཧతোน͕ߴ͘ͳΔ
• ࢖ͬͯ΋Β͏ͷ͸։ൃऀ
• গͣͭ͠ม͍͑ͯ͘
• Ұؾʹม͑ͣʹɺͰ͖Δ͚ͩࠓ·Ͱͷํ๏Λอͪͳ͕Β
• ௕ظͰվળ͍͚ͯ͠͹݁Ռͱͯ͠େ͖͘มΘ͍͚ͬͯΔ
30/64

View Slide

ϨϏϡʔ/ϑΟʔυόοΫ
• GitHub Flowʹଇͬͯ։ൃɺϨϏϡʔͯ͠΋Β͏
• ϦϑΝΫλϦϯά΍ςετ΋ॻ͍ͯ΋Β͑ͨ
• σϓϩΠͯ͠΋ΒͬͯϑΟʔυόοΫ
• ίϚϯυ໊ͷมߋɺ࢓༷ͷमਖ਼Λదٓमਖ਼
• SlackͰͷ͞Γ͛ͳ͍ͭͿ΍͖΋ݟಀ͞ͳ͍
31/64

View Slide

υΩϡϝϯτ࡞੒
• ֓ཁ
• ৽͍͠σϓϩΠͷશମ૾Λ௫ΜͰ΋Β͏
• ίϚϯυͱ࣮ߦํ๏
• σϓϩΠ͢Δͱ͖ͷϚχϡΞϧ
• Կ΋ߟ͑ͣʹίϐϖ͢Δ͚ͩͰσϓϩΠͰ͖ΔΑ͏ʹ
• τϥϒϧγϡʔςΟϯά
• ૝ఆ͞ΕΔΤϥʔ಺༰ͱͦͷରॲํ๏
• ࣗ෼͕͍ͳͯ͘΋ղܾͰ͖ΔΑ͏ʹ
• ໰୊͕ىͬͨ͜ͱ͖ʹ୭Ͱ΋ରॲͰ͖ΔΑ͏ʹͳΔͷ͕ཧ૝త
32/64

View Slide

આ໌͢Δ৔Λઃ͚Δ
• ࢿྉΛಡΜͰ΋Β͏͚ͩͩͱ఻ΘΓͮΒ͍͜ͱ͕ଟ͍
• ৽͍͠σϓϩΠͷϝϦοτ
• ࠓ·Ͱͷํ๏͔ΒͲͷΑ͏ʹมΘΔͷ͔
33/64

View Slide

৽σϓϩΠΞʔΩςΫνϟ
34/64

View Slide

֓ཁ
• ιʔείʔυΛϏϧυɺtar.gzͰݻΊͯɺS3ʹΞοϓϩʔυ
• ϏϧυαʔόʹσϓϩΠ
• ϩʔϧຖʹϚχϑΣετϑΝΠϧΛੜ੒ɺS3ʹΞοϓϩʔυ
• consul eventͰΠϕϯτൃՐɺWeb/BatchαʔόͰStretcher࣮ߦ
• σϓϩΠϑΝΠϧͷऔಘɺ഑ஔ
• ϦϦʔεલ·ͰʹऴΘΒ͓ͤͯ͘
• γϯϘϦοΫϦϯΫ੾ସɺRails࠶ىಈ
• ϦϦʔε࣌ʹ࣮ߦ(ϦϦʔε࣌ؒͷ୹ॖ)
• σΟϨΫτϦߏ଄͸Capistranoͱಉ͡
35/64

View Slide

ߏ੒
36/64

View Slide

Consulͷઃఆ
• ChefͰઃఆϑΝΠϧ(JSON)Λੜ੒
• ConsulΫϥΠΞϯτ
• ϩʔϧλά
• Πϕϯτ
• σϓϩΠ
• ϩʔϧλάͷ෇͚֎͠
37/64

View Slide

Ϗϧυαʔό΁ͷσϓϩΠ
38/64

View Slide

ϚχϑΣετϑΝΠϧͷੜ੒
• ϚχϑΣετϑΝΠϧͷςϯϓϨʔτ(ERB)
• update_code_#{application}_#{version}.yml
• S3͔ΒσϓϩΠϑΝΠϧΛμ΢ϯϩʔυ͠ɺ഑ஔ
• deploy_#{application}_#{role}_#{version}.yml
• γϯϘϦοΫϦϯΫ੾ସɺRails࠶ىಈ
39/64

View Slide

update_code_#{application}_#{version}.yml
src: s3://<%= s3_bucket %>/<%= application %>
checksum: <%= checksum %>
dest: app/<%= application %>/release/YYYYMMDDHHMMSS
commands:
pre:
- mkdir -p app/<%= application %>
success:
- echo 'success'
failure:
- echo 'failed!!'
excludes:
- "*.pid"
- "*.socket"
40/64

View Slide

deploy_#{application}_#{role}_#{version}.yml
sync_strategy: mv
commands:
post:
- ln -sfn "app/<%= application %>/release/YYYYMMDDHHMMSS" "app/<%= application %>/current"
<%=# Rails࠶ىಈ %>
- cd app/<%= application %>/release/YYYYMMDDHHMMSS && touch tmp/restart.txt
<%=# ಈ࡞֬ೝ & ஆػӡస %>
- <%= File.join(script_path, "pre-warming.sh") %> <%= application %>
success:
<%=# աڈͷϦϦʔεΛ࡟আ %>
- <%= File.join(script_path, "cleanup.sh") %> <%= application %>
- echo 'release success'
failure:
- echo 'release failed!!'
41/64

View Slide

S3΁ͷΞοϓϩʔυ
42/64

View Slide

ΞϓϦέʔγϣϯαʔόʹ഑෍
43/64

View Slide

σϓϩΠ(γϯϘϦοΫϦϯΫ੾ସ)
44/64

View Slide

Consul KVSͰόʔδϣϯ؅ཧ
• σϓϩΠ੒ޭޙɺConsul KVSʹ੒ޭͨ͠࠷৽ͱ1ͭલͷόʔδϣϯΛ
ߋ৽͢Δ
•
http://127.0.0.1:8500/v1/kv/#{application}/current
•
http://127.0.0.1:8500/v1/kv/#{application}/previous
• όʔδϣϯ͸ϦϦʔεσΟϨΫτϦͷ೔෇(
YYYYMMDDHHMMSS
)
• αʔό௥Ճ͢Δͱ͖͸࠷৽ɺϩʔϧόοΫ͢Δͱ͖͸1ͭલͷόʔδ
ϣϯΛࢦఆ͢Δ
45/64

View Slide

ϩʔϧόοΫ
• 1ͭલͷόʔδϣϯͷσϓϩΠϑΝΠϧ͕αʔόʹ͋Δͱ͸ݶΒ
ͳ͍
• σϓϩΠʹࣦഊͨ͠৔߹͸ɺ1ͭલͷόʔδϣϯΛσϓϩΠ͢Δ
• σϓϩΠϑΝΠϧͷ഑෍ͱγϯϘϦοΫϦϯΫ੾ସͷ྆ํΛߦ
͏
46/64

View Slide

·ͱΊͯσϓϩΠ͢Δ
͢΂ͯͷΞϓϦΛϚϧνεϨουͰฒྻσϓϩΠ
task deploy_all: :setup do
deploy_projects = "application_A,apllication_B,..."
threads_count = ENV['THREADS']
Parallel.map(deploy_projects, in_threads: threads_count.to_i) do |project|
# Ϗϧυαʔό΁ͷσϓϩΠ
# ϚχϑΣετϑΝΠϧͷੜ੒
# S3΁ͷΞοϓϩʔυ
# ΞϓϦέʔγϣϯαʔόʹ഑෍
end
end
task release_all: :setup do
deploy_projects = "application_A,apllication_B,..."
threads_count = ENV['THREADS']
Parallel.map(deploy_projects, in_threads: threads_count.to_i) do |project|
# σϓϩΠ(γϯϘϦοΫϦϯΫ੾ସ)
end
end
47/64

View Slide

Slack
SlackͰσϓϩΠঢ়گ͕֬ೝͰ͖Δ
໰୊͕ੜͨ͡৔߹͸όοΫτϨʔε͕ද
ࣔ
48/64

View Slide

݁Ռ
1. σϓϩΠʹ͕͔͔࣌ؒΔ
• => ϦϦʔε࣌ؒͷ୹ॖ(30෼->5෼)
2. ಛఆͷαʔό͚ͩσϓϩΠͰ͖ͳ͍
• => ࠷৽ͷσϓϩΠϑΝΠϧΛ࣋ͬͯ͘Δ͚ͩ
3. GitHub/RubyGems͕མͪΔͱσϓϩΠͰ͖ͳ͍
• => S3ͳͷͰ΄΅໰୊ͳ͠
4. ࣗಈԽͮ͠Β͍
• => ConsulʹΑͬͯΫϥελϦϯά΁ͷࣗಈδϣΠϯ/୤ୀ͕Մೳ
49/64

View Slide

تͼͷ੠ !
50/64

View Slide

·ͱΊ
• Πϯϑϥ୲౰͕σϓϩΠվળΛͨ͜͠ͱͰ։ൃج൫ʹߩݙͰ͖ͨ
• ։ൃͱΠϯϑϥͷ྆ํΛ஌Βͳ͍ͱվળͰ͖ͳ͍ྖҬ͕͋Δ
• Πϯϑϥ͚ͩͰͰ͖Δ෼໺͸গͳ͍
• ։ൃ΋Ͱ͖ΔΠϯϑϥ(ࣗ෼΋ΩϟϦΞతଆ໘΋ؚΊͯ)
• օʹͱͬͯ޾ͤͳಓΛબͿ
• ։ൃଆͱΠϯϑϥଆͷཁٻ͕Ϛον͢Ε͹ϕετ
• গͣͭ͠ม͍͑ͯ͘
• Ұؾʹ΍ΓํΛม͑ͣʹɺͰ͖Δ͚ͩࠓ·Ͱͷํ๏Λอͪͳ͕Β
• ௕ظؒվળ͍͚ͯ͠͹݁Ռͱͯ͠େ͖͘มΘ͍͚ͬͯΔ
51/64

View Slide

ࠓޙͷ༧ఆ
• Φʔτεέʔϧ(ઈࢍରԠத)
• όοναʔόΛAmazon ECS΁Ҡߦ
• Webαʔόʹಉډ͍ͯ͠ΔRailsΞϓϦͷ෼ׂ
• SlackίϚϯυʹΑΔσϓϩΠ
52/64

View Slide

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
53/64

View Slide

͓·͚

View Slide

Consul - ΫϥΠΞϯτ
# /etc/consul.d/settings.json
{
"data_dir": "/opt/consul",
"log_level": "err",
"start_join": [
"consul-server001",
"consul-server002",
"consul-server003"
]
}
55/64

View Slide

Consul - ϩʔϧλά
# /etc/consul.d/role.json
{
"service": {
"name": "role",
"tags": [ # ෳ਺ઃఆՄ
"web",
"newrelic_monitoring"
]
}
}
56/64

View Slide

Consul - Πϕϯτ(σϓϩΠ)
# /etc/consul.d/deploy.json
{
"watches": [
{
"type": "event",
"name": "deploy_application_A",
"handler": "export TMPDIR=/ephemeral0; stretcher -retry 3 >> $TMPDIR/deploy_application_A.log 2>&1"
},
{
"type": "event",
"name": "deploy_application_B",
"handler": "export TMPDIR=/ephemeral0; stretcher -retry 3 >> $TMPDIR/deploy_application_B.log 2>&1"
},
...
}
• consul eventίϚϯυͰΠϕϯτൃՐ
• consul event -service=role -tag=#{role} -name=deploy_#{application} #{manifest_file}
• σϓϩΠϑΝΠϧͷμ΢ϯϩʔυઌ(TMPDIR)ΛEC2ͷΠϯελϯεετΞʹมߋ
57/64

View Slide

ϩʔϧ
# ϩʔϧͷࢦఆ
role :web, *consul_role('web')
role :batch, *consul_role('batch')
# ϩʔϧࢦఆ͞Εͨϗετͷऔಘ
def consul_role(roles)
response = `curl -fsSL http://127.0.0.1:8500/v1/catalog/service/role`
nodes = JSON.parse(response)
lists = []
nodes.each do |node|
if (node['ServiceTags'] & roles.split(',')) == roles.split(',')
lists << node['Address']
end
end
lists
end
58/64

View Slide

Consul - Πϕϯτ(ϩʔϧλάͷ෇͚֎͠)
# /etc/consul.d/change_role.json
{
"watches": [
{
"type": "event",
"name": "change_role",
"handler": "change_consul_role.rb >> /tmp/change_log.log 2>&1"
}
]
}
#!/bin/env ruby
require "json"
consul_url = "http://127.0.0.1:8500"
response = %x(curl -fsSL #{consul_url}/v1/catalog/node/$(curl -fsSL #{consul_url}/v1/agent/self | jq -r .Member.Name))
node = JSON.parse(response)
new_role = node["Services"]["role"]["Tags"]
consul_role_file_path = "/etc/consul.d/role.json"
if ! File.exist?(consul_role_file_path) then
exit 0
end
consul_role = open(consul_role_file_path) do |io|
JSON.load(io)
end
if consul_role["service"]["tags"] != new_role then
consul_role["service"]["tags"] = new_role
open(consul_role_file_path, 'w') do |io|
io.puts(JSON.pretty_generate(consul_role))
end
%x(consul reload)
end
59/64

View Slide

S3΁ͷΞοϓϩʔυ
def upload2s3(local_src_path, bucket, path)
run "aws s3api put-object --region #{aws_region} --bucket #{bucket} --key #{path} --body #{local_src_path}"
end
def create_tarball
run "cd #{latest_release} && rm -rf #{tarball_name} && shopt -s dotglob && tar zcf #{tarball_name} *"
end
def upload_tarball
upload2s3("#{latest_release}/#{tarball_name}", app_s3_bucket, "#{app_s3_path}/#{tarball_name}")
end

View Slide

σϓϩΠϑΝΠϧͷ഑෍
namespace :stretcher do
task :update_code, roles: :build do
roles.split(',').each do |role|
manifest_file = "update_code_#{application}_#{version}.yml"
capture "consul event -service=role -tag=#{role} -name=deploy_#{application} #{manifest_file}"
end
puts 'Finished(Not released yet!)'
end
end

View Slide

ϦϦʔε(γϯϘϦοΫϦϯΫ੾ସ)
task :release, roles: :build do
transaction do
on_rollback do
rollback
end
roles.split(',').each do |role|
manifest_file = "deploy_#{application}_#{role}_#{version}.yml"
capture "consul event -service=role -tag=#{role} -name=deploy_#{application} #{manifest_file}"
end
unless check_event_success('release_failure')
puts 'Release Failed!! & Starting rolling back.'
exit 1
end
puts 'Release finished.'
end
end
end

View Slide

όʔδϣϯߋ৽
ConsulͷKVSʹ֨ೲ͍ͯ͠Δόʔδϣϯ஋Λߋ৽
after 'stretcher:release', 'stretcher:update_release_version'
task :update_release_version, roles: :build do
# લճ੒ޭͨ͠σϓϩΠͷόʔδϣϯΛpreviousʹ͢Δ
capture "curl -sS -X PUT -d '#{current_version}'http://127.0.0.1:8500/v1/kv/#{application}/previous"
# ੒ޭͨ͠σϓϩΠͷόʔδϣϯΛߋ৽͢Δ
capture "curl -sS -X PUT -d '#{latest_version)}' http://127.0.0.1:8500/v1/kv/#{application}/current"
end
end

View Slide

ϩʔϧόοΫ
task :rollback, roles: :build do
fetch(:deploy_roles).split(',').each do |target_role|
manifest_file = "update_code_#{application}_#{previous_version}.yml"
capture "consul event -service=role -tag=#{target_role} -name=deploy_#{application} #{manifest_file}"
end
reset_release_status
fetch(:deploy_roles).split(',').each do |target_role|
manifest_file = "deploy_#{application}_#{role}_#{previous_version}.yml"
capture "/usr/local/bin/consul event -service=role -tag=#{target_role} -name=deploy_#{application} #{manifest_file}"
end
end

View Slide

インフラエンジニアがConsulとStretcherをつかったデプロイ改善で開発効率の向上に貢献した話 / jtf2017-consul-stretcher-deploy

インフラエンジニアがConsulとStretcherをつかったデプロイ改善で開発効率の向上に貢献した話 / jtf2017-consul-stretcher-deploy

Shuichi Ohsawa

More Decks by Shuichi Ohsawa

Other Decks in Technology

Featured

Transcript