Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Webサービスを1日10回デプロイするための取り組み / SRE NEXT 2020

Webサービスを1日10回デプロイするための取り組み / SRE NEXT 2020

FUJIWARA Shunichiro

January 25, 2020
Tweet

More Decks by FUJIWARA Shunichiro

Other Decks in Technology

Transcript

  1. "HFOEB • σϓϩΠͷߏ੒ཁૉͱߴ଎Խͷྺ࢙ 1. CI 2. ഑෍ 3. Τϥʔݕ஌ 4.

    ϩʔϧόοΫ • ͩΕͰ΋σϓϩΠͰ͖Δ࢓૊Έͱ؅ཧ • ίϯςφҠߦޙͷσϓϩΠ
  2. -PCJךءأذيך娖〷 2010೥ ʮφΧϚοϓʯͱͯ͠ϦϦʔε (AWS us-east-1) 2011೥ ΦϯϓϨ؀ڥ΁Ҡߦ 2013೥ ʮLobiʯʹվ໊ɺAWS (ap-northeast-1)

    ʹҠߦ 2019೥ EC2 → ECS Ҡߦ։࢝ (ਐߦத) ΋͏͙͢10೥બख 30ʙ40ਓͷνʔϜɺΤϯδχΞ͕൒਺ఔ౓ SRE 2໊(݉೚)
  3. -PCJךءأذي圓䧭 • APIαʔό Perl on EC2 • ϚΠΫϩαʔϏεతͳ΋ͷ Go (͔ͭͯ͸Nodeɺ࠷ۙશഇʹ੒ޭ)

    • σʔλετΞ MySQL (on EC2... ΋͏͙͢ Aurora ʹͳΔ༧ఆ) Redis (ElastiCache) • ϒϥ΢β޲͚ϑϩϯτΤϯυ Nuxt.js (Vue.js) (͔ͭͯ͸ Angular.JS 1.x ࡢ೥ Nuxt ʹҠߦ) • iOS, Android ޲͚ωΠςΟϒΞϓϦ
  4. 傈ח㔐رفٗ؎׃׋ְ ͜͜Ͱ͍͏σϓϩΠ = ҎԼͷ͢΂ͯ • αʔόΞϓϦέʔγϣϯ (ຊମAPIɺϚΠΫϩαʔϏε) • ϑϩϯτΤϯυ (Nuxt.JS)

    • EC2 ͷϓϩϏδϣχϯά (Chef) • AWSͷߏ੒มߋ (Terraform) मਖ਼΍ػೳ௥Ճɺ࡟আ͸͙͢ʹσϓϩΠ͍ͨ͠ Ұ౓ͷࠩ෼Λখࣦͯ͘͞͠ഊΛڐͤΔΑ͏ʹɾӨڹൣғΛݶఆ ৗʹσϓϩΠ͍ͯ͠Ε͹ɺرفٗ؎ָ䙳ֻזֻז׷
  5. 傈ח⡦儗꟦⢪ִ׷ַ Ӧۀ࣌ؒ 9:29ʙ18:29 (9࣌ؒ) Ӧۀ࣌ؒ֎ɺٳલ೔(ޕޙ)͸جຊతʹσϓϩΠ͠ͳ͍ ٳલ೔ʹσϓϩΠ͢Δͱͦͷ࣌఺Ͱ͸໰୊͕ͳͯ͘΋… → ͦͷ໷ͷϐʔΫλΠϜʹ໰୊͕ൃ֮ → ࣍ͷ໷ؒɾૣேͷόονॲཧʹ໰୊͕ൃੜ

    → Ϣʔβʔͷ໰͍߹Θ͕ͤٳ೔தʹ૿͑Δ ͲΕ΋ٳ೔ରԠ͕ඞཁʹͳΔ սٕ٦ٕվͲ͏ͯ͠΋ٳલ೔ʹσϓϩΠ͍ͨ͠৔߹͸ ରԠͰ͖ΔਓһΛࣄલ֬อ͢Δ(ٳ೔ग़ۈͯ͠΋Β͏)
  6. չرفٗ؎պך銲稆 1. CI → ςετ͕௨͍ͬͯͳ͍΋ͷ͸ϦϦʔεͰ͖ͳ͍ 2. Ϗϧυ/഑෍ (ڱٛͷσϓϩΠ) 3. Τϥʔݕ஌

    → ໰୊͸͙͢ʹݕ஌͍ͨ͠ 4. ϩʔϧόοΫ → ଈ࠲ʹ໭ͯ͠μϝʔδΛ࠷খݶʹ ͜ΕΒׅץגך銲稆׾넝鸞⻉͍ͯ͘͠
  7. $*넝鸞⻉ך娖〷 ຊମ API = Perl ͷΘΓͱେ͖͍ϞϊϦε ΞϓϦέʔγϣϯ 26ສߦ ʴ ςετ

    26ສߦ (Cookpad ຊମͷRuby͕27ສߦ1 ϥΠϒυΞϒϩάͷίʔυ͕41ສߦ2) ࣮ࡍʹ MySQL ΍ Redis Λىಈͯ͠ςετ͍ͯ͠Δ ςετΛ۪௚ʹ΍Δͱ΋ͷֻ͍͕͔࣌ؒ͢͝Δ ۩ମతʹ͸Կ΋ߟ͑ͣ௚ྻʹ΍Δͱ೔͕฻ΕΔϨϕϧ 2 https://logmi.jp/tech/articles/322235 1 https://employment.en-japan.com/engineerhub/entry/2019/09/17/103000
  8. $*넝鸞⻉ך娖〷 Jenkins master + slave (ڧ͍EC2Πϯελϯε) Perl ͷςετ͸جຊతʹγϯάϧϓϩηε ίΞ͕͍ͭ͋ͬͯ͘΋ͦͷ··Ͱ͸଎͘ͳΒͳ͍ͨΊࣗલͰฒྻԽ MySQL

    / Redis / etc ΋ฒྻͰىಈ͠ͳ͍ͱεέʔϧ͠ͳ͍ (͕ɺίΞ਺෼ىಈ͢Δͷ͸ແཧ) 1ϓϩηεͷ MySQL Λ8ϓϩηεͷ Perl Ͱར༻͢Δ = 1ηοτ ͦΕΛ͞Βʹ8ฒྻͰಈ͔͢ = ߹ܭ64ฒྻ 64ίΞϚγϯͰ10෼͙Β͍ͰऴΘΔঢ়ଶͰ͸͋ͬͨ
  9. $*ך넝鸞⻉٥أ؛٦ٓؽٔذ؍ך栻䖤 1. ίʔυͱςετΛॻ͘ 2. commit & push 3. GitHub Ͱ

    PR Λ࡞ͬͯઆ໌Λॻ͍͍ͯΔ͏ͪʹ… 4. CI ͕ऴΘ͍ͬͯΔ ετϨε͕ͳ͍ ࠓޙςετ͕૿͑ͯ΋ฒྻ਺Λ্͛ͯεέʔϧͰ͖Δ
  10. UJQT$JSDMF$*ךذأز؎ً٦آꂁ縧㜥䨽 Lobi ͷαʔό͸͢΂ͯ AWS ౦ژϦʔδϣϯ (ap-northeast-1) ʹ͋Δ Կ΋ߟ͑ͣʹ ECR (ίϯςφϨδετϦ)

    ΋౦ژϦʔδϣϯΛ࢖͍͕ͬͯͨ CircleCI ͷ࣮ߦ؀ڥ͸ us-east-1 ʹ͋Δ (ݱ࣌఺) → CI ࣮ߦ͝ͱʹόʔδχΞ͔Β౦ژͷίϯςφΠϝʔδΛ Pull (×ฒྻ਺)
  11. ؿ؋؎ٕꂁ䋒넝鸞⻉ך娖〷 2010೥ʙ σϓϩΠαʔό(EC2) ͔Β֤ϗετ͕ git pull • ୆਺͕૿Ճ͢Δͱ Git ͷෛՙ͕όΧʹͳΒͳ͍

    • όΠφϦͷ੒Ռ෺Λ commit ͨ͘͠ͳ͍ 2013೥ʙ σϓϩΠαʔό͔Β֤ϗετʹฒྻ rsync (push) • AWSʹҠߦͯ͠ϗετ୆਺΋૿Ճɺ਺े୆ʹσϓϩΠ͕ඞཁʹ • 10ฒྻͰ rsync ͯ͠΋ऴΘΔ·Ͱ3ʙ4෼(Ջ)
  12. ؔ٦زأ؛٦ٕפך㼎䘔 2014೥ήʔϜಈը࿥ըαʔϏεΛSDKఏڙ։࢝(ݱࡏ͸ऴྃ) ࠷ॳ͸ Elastic Transcoder Λར༻ ϞϯελʔετϥΠΫʹಋೖ → Elastic Transcoder

    Ͱ͸嫣剢㹺ָ䒉אꆃ겘ח ΠϯελϯεΛେྔʹىಈͯࣗ͠લͰಈըม׵͢Δ࡞ઓʹมߋ → ؔ٦زأ؛٦ٕ׃׋ְ (੾࣮) ͋Δϗετ͔ΒϑΝΠϧΛ഑ΔઓུͰ͸Φʔτεέʔϧ΁ͷରԠ͕ࠔ೉ AMI Λຖճ࡞͍ͬͯͨΒ1೔ʹ10ճσϓϩΠͰ͖ͳ͍
  13. 4USFUDIFS׾Ꟛ涪 HJUIVCDPNGVKJXBSBTUSFUDIFS ഑෍Λ push → pull ʹ σϓϩΠͷ഑෍෺ (tar.gz)Λ͋Β͔͡Ί S3

    ʹஔ͘ ֤ϗετͷ Stretcher(ίϚϯυ) ͕ S3 ͔Β औಘɺ ల։ɺϓϩηε࠶ىಈΛߦ͏ ΦʔτεέʔϧͰىಈͨ͠ϗετ͸࠷৽ͷtarΛS3 ͔Βऔಘͯ͠ΞϓϦέʔγϣϯΛىಈ͢Δ ࢥ૝తʹ͸ίϯςφͷσϓϩΠͱࣅͨΑ͏ͳ΋ͷ ίϯςφ࣮ӡ༻͸·ͩૣ͍(2014೥຤) ͔͠͠Φʔτεέʔϧ΁ͷରԠ͸ࠓ͙͍ͨ͢͠ ં஭Ҋ
  14. 4USFUDIFSח״׷넝鸞⻉⸬卓 • rsync ࣌୅ 3ʙ4෼ • tar࡞੒ / S3 ΁Ξοϓϩʔυ

    30ඵ • શ୆ʹΠϕϯτ௨஌ͯ͠ Stretcher ͕σϓϩΠΛ׬ྃ͢Δ·Ͱ 30ඵ ߹ܭ1෼ఔ౓ʹ୹ॖ 1෼ͳΒͦͷ··଴ͯΔ 3ʙ4෼Ջͩͱͦͷؒʹଞͷ͜ͱΛͨ͘͠ͳΔ
  15. ،فٔ؛٦ءّٝךַؚٗ׵ז׿׵ַךؒٓ٦׾嗚濼 app_exception count:1 app-i-0ad5e77d [613][667331357105553408] [WARN]error: Can't call method "as_json_for_invitation"

    on an undefined value ... SELECT 'app_exception' AS title, count(*) AS count, last(_hostname) AS _hostname, last(message) AS message FROM app.win:time_batch(1 min) WHERE (message LIKE '%Caught exception%' OR message LIKE '%error:%' OR message LIKE '%limit exceeded:%') AND (message NOT LIKE '%[INFO]%') ͜ͷΫΤϦͰϚον͢Δͱ1෼ʹ1౓௨஌͕དྷΔ
  16. رفٗ؎넝鸞⻉תה׭ 1. CI: Jenkins ࣌୅ 10෼ → $JSDMF$*ⴓ 2. ഑෍:

    rsync ࣌୅ 3ʙ4෼ → 4USFUDIFSⴓ 3. Τϥʔݕ஌: /PSJLSBⴓ 4. ϩʔϧόοΫ: ϦόʔτPR 5෼ → 4USFUDIFS猱 ຖճͷσϓϩΠΛ5෼ఔ౓Ͱ׬ྃͰ͖ΔΑ͏ʹͳͬͨ
  17. 3VOEFDL www.rundeck.com/open-source Job ఆٛɺ࣮ߦΛߦ͑Δ Automation Software ϦϞʔτϗετ΁ͷ࣮ߦ΍εέδϡʔϧ࣮ߦ΋(LobiͰ͸࢖͍ͬͯͳ͍) σϓϩΠʹඞཁͳίϚϯυΛୟͨ͘Ίͷ Web UI

    ͱͯ͠ར༻ • EC2, ECS, Lambda ͷσϓϩΠ / ϩʔϧόοΫ • Chef ʹΑΔϓϩϏδϣχϯά • Terraform ʹΑΔߏ੒มߋ • ΦʔτεέʔϦϯάͷૢ࡞(୆਺Λ૿΍͢ͳͲ)
  18. HIDIדرفٗ؎חろת ׸׷13׾《䖤׃ג鎸ꐮ ghch --from $recent_tag { "pull_requests": [ { "number":

    23013, "state": "closed", "title": "...........", Google ΧϨϯμʔʹه࿥ Mackerel ͷάϥϑΞϊςʔγϣϯʹه࿥
  19. ؝ٝذش獳遤䖓ךرفٗ؎ ݱࡏγεςϜશମΛ EC2 ͔Β Amazon ECS ʹҠߦத Կ͕มΘΔ͔ CI: มΘΒͳ͍

    Ϗϧυ: EC2 → CircleCI (image build) ꂁ䋒ծٗ٦ٕغحؙ4USFUDIFS̔&$4 ؒٓ٦嗚濼/PSJLSB̔'JSFIPTF 4 -BNCEB
  20. FDTQSFTTP github.com/kayac/ecspresso γϯϓϧ / ϛχϚϧͳ ECS σϓϩΠπʔϧ (Go੡) • λεΫͱαʔϏεఆٛΛ

    JSON ͔Βੜ੒, ొ࿥ • σϓϩΠ = αʔϏεͷλεΫఆٛΛೖΕ׵͑Δ • ϩʔϧόοΫ = ௚લͷλεΫఆٛʹࠩ͠ସ͑Δ AWS SDK Go ͷߏ଄ମΛ࢖͏͜ͱͰ৽ػೳͷΩϟονΞοϓΛ༰қʹ (ྫ: EFSϚ΢ϯτػೳ 1/18ϦϦʔε → 1/20ରԠ൛ ecspresso ϦϦʔε) Blue/Green deployment ͸ CodeDeploy ͱ࿈ܞ͢Δ͜ͱͰαϙʔτ
  21. 傀㶷&$4؟٦ؽأ׾盖椚⻉ח init αϒίϚϯυΛ࣮ߦ͢Δͱ $ ecspresso init --region ap-northeast-1 --cluster default

    --service myservice --config config.yaml 2019/10/12 01:31:48 myservice/default save service definition to ecs-service-def.json 2019/10/12 01:31:48 myservice/default save task definition to ecs-task-def.json 2019/10/12 01:31:48 myservice/default save config to config.yaml config.yaml, ecs-service-def.json, ecs-task-def.json ͕ੜ੒͞ΕΔ λεΫఆٛΛదٓॻ͖׵͑ͯ deploy ͢Δ͚ͩͰҠߦՄೳ $ ecspresso deploy --config config.yaml ଞͷπʔϧ΍ίϯιʔϧͰ࡞ͬͨ ECS λεΫ/αʔϏε΋ ؆୯ʹίʔυ؅ཧʹམͱ͠ࠐΊΔ
  22. أ؛٦ٓـٕזؒٓ٦嗚濼 Norikra ͸εέʔϧ͕೉͍͠ 1. ϩάΛ Firehose ͔Β S3 ʹ1෼͝ͱʹอଘ 2.

    S3 ͷΠϕϯττϦΨͰ Lambda Λىಈ 3. ϩάͷதʹ͋ΔΤϥʔΛݕग़ͯ͠௨஌ 1෼୯ҐͰݕग़ͯ͠௨஌Λൃੜͤ͞Δཁ݅Λ อͪͭͭϚωʔδυʹ ྲྀྔ͕૿͑ͯ΋ Firehose / S3 / Lambda ͸εέʔϧ͢Δ