Spinnakerで実現する、クラウド開発者のためのデプロイパターン入門 / Introduction to deployment patterns with Spinnaker

13d936e697fe0f4fa96f926d0a712f6c?s=47 Sansan
July 23, 2019

Spinnakerで実現する、クラウド開発者のためのデプロイパターン入門 / Introduction to deployment patterns with Spinnaker

■イベント
イベント名 CloudNative Days Tokyo 2019
URL https://cloudnativedays.jp/cndt2019/

■登壇概要
タイトル:Spinnakerで実現する、クラウド開発者のためのデプロイパターン入門
登壇者: DSOC Infrastructure Group 大澤秀一

▼Sansan Builders Box
https://buildersbox.corp-sansan.com/

13d936e697fe0f4fa96f926d0a712f6c?s=128

Sansan

July 23, 2019
Tweet

Transcript

  1. 2.

    େᖒ लҰʢShuichi Ohsawaʣ DSOCʢData Strategy & Operation Centerʣ
 ɹInfrastructure Group

    ݉ CSIRT ID : @ohsawa0515 Blog : https://blog.jicoman.info/ աڈొஃྺ July Tech Festa 2017
 インフラエンジニアが Consul と Stretcher をつかったデプロイ改善で
 開発効率の向上に貢献した話 Vuls 祭り #2
 サーバレスで Vuls スキャン on AWS Google Cloud INSIDE Digital #2
 Windows Server VM Migration & Continuous Delivery with Spinnaker
  2. 4.
  3. 5.

    Data Strategy and Operation Center ໊͕ࢗࣔ͢৘ใ ձࣾɾݸਓͷ৘ใ ͭͳ͕Γˠਓ຺ ڧΈ ۀछɺ৬छɺ໾৬ɺ஍Ҭ

    ͍ͭɺͲͷ෦ॺɾ໾৬ͷ
 ਓͱग़ձ͔ͬͨ ԣͷܨ͕Γͷ๛෋͔͞Βɺ
 ໊ࢗॴ༗ऀͷφϨοδɾܦݧ ͕͋ΔྖҬ͕ࣔࠦ͞Ε͍ͯΔ ໾৬໊ ࢯ໊ اۀ໊ ϝʔϧΞυϨεి࿩൪߸63- 1 2 3 Sansanגࣜձࣾͷࣄۀ੒௕Λࠜװ͔Βࢧ͑Δ ʮग़ձ͍ʯͷσʔλϕʔεΛߏங
  4. 6.

    Data Strategy and Operation Center ໊ࢗऔΓࠐΈ എܠ෼཭ ը૾ิਖ਼ 1 ߲໨෼ׂ

    2 ηΩϡϦςΟʔ߲໨ࡉ෼ׂɺ߲໨ೖྗ 3 νΣοΫˍิਖ਼ 5 Ϛʔδ 4 ϚΠΫϩλεΫ×ϚϧνιʔγϯάʹΑΔಠࣗͷ໊ࢗσʔλԽγεςϜ ໊ࢗσʔλԽγεςϜʮGEESʯ ηΩϡΞͳ؀ڥΛߏங
  5. 13.

    Spinnaker ͱ͸ - OSS ͷ ܧଓతσϦόϦʔ (Continuous Delivery; CD) ϓϥοτϑΥʔϜ

    - Netflix ͕։ൃɺ2015೥ʹOSSԽ - Netflix ͰഓΘΕͨܧଓతσϦόϦʔͷϓϥΫςΟε͕൓ө - GoogleɺMicrosoftɺPivotal ͱఏܞ͢Δ͜ͱͰɺϚϧνΫϥ΢υରԠͷՃ଎౓޲্ - OracleɺTargetɺVeritasɺSchibstedɺArmoryɺKenzan ͳͲͷاۀ΋։ൃʹࢀՃ
  6. 14.

    ௕ॴ - ϚϧνΫϥ΢υରԠ AWSɺGCPɺAzureɺOpenStack etc - ϚϧνϓϩόΠμରԠ Amazon EC2, Google

    Compute Engine, Kubernetes etc - ෳ਺ͷσϓϩΠઓུΛαϙʔτ - Red/BlackʢGlue/Greenʣ͕ඪ४αϙʔτɺCanary σϓϩΠ΍ΧελϚΠζ΋Մೳ - GUI ্ͰύΠϓϥΠϯʢϑϩʔʣΛ࡞ΕΔ - CI πʔϧͱͷ࿈ܞखஈ͕ଟ͍ɻCD ͱͯ͠ͷ໾ׂʹಛԽ͍ͯ͠Δ
  7. 16.

    ୹ॴ - ηϧϑϗεςΟϯά - Spinnaker ࣗ਎ͷσϓϩΠʹͦΕͳΓͷϚγϯϦιʔεΛཁ͢Δ - AWS: m5.xlarge, GCP:

    n1-highmem-4ʢCPU 4, Memory 16GBʣ - υΩϡϝϯτ͕গͳ͍ɻυΩϡϝϯτͱ࣮ࡍͷ UI ΍݁Ռ͕ҟͳΔ৔߹͕͋Δ - όά͕ଟ͍ʢύΠϓϥΠϯͷૢ࡞ܥɻσϓϩΠʹؔ͢Δόά͸͋·Γૺ۰͠ͳ͍ʣ - Spinnaker ಠࣗͷઃఆ͕ଟ͘ɺ࢖͍͜ͳ͢ͷ͕େม - Spinnaker ͓͡͞ΜΛੜΈग़͢ةݥੑ
  8. 18.

    ίϯηϓτ - Application management - - ΞϓϦέʔγϣϯͷ֓೦ΛϞσϧԽ͢Δ͜ͱͰ
 ϚϧνΫϥ΢υͰ΋ಉ͡Α͏ʹѻ͑Δ - Instance:

    Amazon EC2, Google Compute Engine(GCE), Kubernetes Pod etc - Server Group: Instance ͷάϧʔϓԽ - Auto Scaling GroupʢAWSʣɺInstance Group ʢGCPʣɺReplicaSetʢKubernetesʣ - σϓϩΠՄೳͳΞʔςΟϑΝΫτ
 ʢVMΠϝʔδɺDockerΠϝʔδɺϚχϑΣετ etcʣɺ
 Πϯελϯε਺ɺΦʔτεέʔϦϯάϙϦγʔ͔Βߏ੒ - Cluster: Server Group Λ࿦ཧతʹάϧʔϓԽ - Application: Cluster ͷू·Γ - Firewall - Load Balancer https://www.spinnaker.io/concepts/#application-management
  9. 19.

    ίϯηϓτ - Application deployment - - Pipeline: σϓϩΠϑϩʔΛߏ੒ - Stage

    - Triggers - Notifications - Stage Pipeline ͷߏ੒ཁૉɻύΠϓϥΠϯ͕࣮ߦ͢ΔΞΫγϣϯΛهड़ - Infrastructure Stagesʢج൫ͱͳΔΠϯϑϥετϥΫνϟΛߏஙʣ - External Systems Integrationsʢ֎෦γεςϜͱͷ࿈ܞʣ - Controlling FlowʢঝೝɺλΠϛϯάɺ෼ذϩδοΫͳͲɺύΠϓϥΠϯͷྲྀΕΛ੍ޚ͢Δʣ - Deployment strategies - Red/Black (Blue/Green) - Canary
  10. 20.

    Infrastructure Stages - Ϧιʔε࡞੒ɺߋ৽ɺ࡟আͳͲج൫ͱͳΔΠϯϑϥετϥΫνϟΛߏங͢Δεςʔδ - ϚϧνΫϥ΢υΛར༻͍ͯ͠Δ৔߹ɺҰ؏ͨ͠ํ๏ͰͦΕͧΕͷΫϥ΢υʹల։Ͱ͖ΔͨΊɺ
 ೝ஌తͳෛՙ͕ܰݮ͞ΕΔ - Bake (VMΠϝʔδ,

    DockerΠϝʔδͷ࡞੒) - Find Image From {Cluster, Tag} ʢΠϝʔδΛಛఆ͢Δʣ - DeployʢRed/Black, ϚχϑΣετద༻ʣ - {Disable, Shrink, Scale Down, Rollback} ClusterʢClusterͷૢ࡞ʣ - {Clone, Enable, Disable, Resize, Destroy} Server GroupʢServer Groupͷૢ࡞ʣ - Run Job (ίϯςφ࣮ߦ) - Script (Jenkins্Ͱ೚ҙͷεΫϦϓτ࣮ߦ) ओͳεςʔδ
  11. 21.

    External Systems Integrations - ֎෦γεςϜͱͷ࿈ܞ - Continuous IntegrationʢCIʣ - Jenkins

    / TravisCI / Wercker / Concourse CI - Webhook - CircleCIͳͲαϙʔτ͞Ε͍ͯͳ͍αʔϏεͷ৔߹ʹར༻
  12. 22.

    Controlling Flow - ύΠϓϥΠϯͷྲྀΕΛ੍ޚ͢Δεςʔδ - Check Preconditions (৚݅෼ذ) - Manual

    Judgment (खಈঝೝ) - Waitʢࢦఆ͞Εͨඵ਺Λ଴ػ) - Pipeline (ผͷύΠϓϥΠϯΛ࣮ߦ)
  13. 23.

    Triggers - ύΠϓϥΠϯͷ։࢝Λ੍ޚ - Time-based triggers - Manual - Cron

    - Event-based triggers - Git (GitHub, GitLab etc) - Continuous Integration (Jenkins etc) - Docker Registry (Docker Hub, GitLab Container Registry, Amazon ECR, Google Container Registry etc) - Pipeline - Pub/Sub (Cloud Pub/Sub, Amazon SNS etc)
  14. 24.

    Notifications - Email - Slack - SMS via Twilio -

    Pub/Sub - Spinnaker ͔Β֎෦γεςϜ΁ͷ࿈ܞ΋Մೳ - ύΠϓϥΠϯͷ։࢝ɺ੒ޭɺࣦഊ࣌ʹ௨஌ - ֤εςʔδͰ΋௨஌Մೳ ରԠ͍ͯ͠Δ௨஌ઌ
  15. 25.

    Deployment strategies - Red/Black (Blue/Green) https://www.spinnaker.io/concepts/#deployment-strategies - ՔಇதʢBlueʣ͔Β৽όʔδϣϯʢGreenʣʹগͣͭ͠ೖΕସ͍͑ͯ͘ - Blue

    γεςϜʹมߋΛՃ͑ͳ͍ʢImmutable Infrastructureʣ - ϩʔϧόοΫʢ੾Γ໭͠ʣ͕༰қ - ޓ׵ੑ͕ͳ͍มߋʢDBϚΠάϨʔγϣϯͳͲʣ͕͋Δ৔߹͸ผ్ରԠ͕ඞཁ
  16. 26.

    Deployment strategies - Canary - 3ͭͷΫϥελΛߏ੒͠ɺͦΕͧΕʹτϥϑΟοΫΛྲྀ͢ - Production: ຊ൪Ϋϥελʢෳ਺Πϯελϯε) -

    Baseline: ຊ൪Ϋϥελͱಉ͡όʔδϣϯͷίʔυɻCanary ͱൺֱ͢Δʢ௨ৗ 3Πϯελϯεʣ - Canary: ৽όʔδϣϯͷίʔυ΍৽ߏ੒ʢBaseline ͱಉ͘͡ 3Πϯελϯεʣ Load Balancer Production (v1) Traffic Many Instances Baseline (v1) 3+ Instances Canary (v2) 3+ Instances Metrics Canary Analysis Refer: https://medium.com/netflix-techblog/automated-canary-analysis-at-netflix-with-kayenta-3260bc7acc69
  17. 27.

    Deployment strategies - Canary (Evaluation) - Baseline ͱ Canary ͸ಉ͡৚݅Ͱἧ͑ͯධՁ͢Δඞཁ͕͋Δ

    - ಉ࣌ؒ͡ - ಉ͡αΠζ - ಉ͡τϥϑΟοΫྔ - े෼ͳ͕࣌ؒඞཁ - ධՁ࣌ؒ: 3࣌ؒ - ΧφϦΞ෼ੳִؒ: 1࣌ؒ - ࣌ܥྻσʔλ: 50ݸ+ Load Balancer Baseline (v1) 3+ Instances Canary (v2) 3+ Instances Traffic
  18. 28.

    Deployment strategies - Canary (Judgement) - ϝτϦΫεΛऩूʢlatency, errors, saturation etcʣ

    - ෳ਺ϝτϦΫεΛάϧʔϓԽͰ͖Δ - ϝτϦΫεͷऩूઌͱͯ͠ɺStackdriver, Prometheus, Datadog ͕αϙʔτ͞Ε͍ͯΔ - Τϥʔ਺ɺࣦഊ਺Λ௚઀ධՁͤͣɺϝτϦΫεͷภࠩΛධՁ͢Δ - Canary ͕ Baseline ͱಉ౳Ͱ͋Δ͔ or ѱ͘ͳ͍ͬͯͳ͍͔ - ͖͍͠஋ʹରͯ͠ɺͦͷϝτϦΫε͕ʮ߹֨ʯ͍ͯ͠Δ͔Λ൑அ - ࣗಈΧφϦΞ෼ੳʢKayenta) ͰείΞϦϯάͯ͠ϦϦʔε͢΂͖͔Λࣗಈ൑அ - ϝτϦΫεͷ૯਺ʹରͯ͠ʮ߹֨ʯ͍ͯ͠ΔϝτϦΫεͷׂ߹Λܭࢉ - ྫʣ10 ϝτϦΫεதɺ߹֨ϝτϦΫε͕ 8 ͷ৔߹ɺ80 ͱͯ͠ࢉग़ - ϝτϦΫεάϧʔϓຖʹॏΈ෇͚͢Δ͜ͱ΋Ͱ͖Δ
  19. 29.

    Pipeline ྫ https://www.spinnaker.io/concepts/#pipeline - TEST ؀ڥͰطʹσϓϩΠ͞Ε͍ͯΔΠϝʔδΛར༻͠ɺCanary ͰগྔͷτϥϑΟοΫΛςετͯ͠ ͔ΒɺPROD ʹ Red/Black

    σϓϩΠ͕࣮ߦ͞ΕΔ - Canary ͷϝτϦοΫऩूʢWait 30 mins) ͱ खಈঝೝͷ྆ํ͕׬ྃ͢ΔͱσϓϩΠ͕࣮ߦ͞ΕΔ - σϓϩΠ׬ྃޙɺCanary ༻ Server Group ͷఀࢭͱ 2࣌ؒ଴͔ͬͯΒݹ͍ Server Group Λഁغ͢Δ
  20. 31.

    σϓϩΠύλʔϯ 1. Triggering pipeline using GitHub 2. Triggering pipeline using

    Pub/Sub 3. Triggering pipeline using Docker Registry 1. VM machine deployment pipeline - AWS, GCP 2. Kubernetes deployment pipeline 3. Multi-Cloud, Multi-Region deployment pipeline 4. Canary deployment pipeline Integration ͲͷΑ͏ʹ֎෦γεςϜͱ࿈ܞ͢Δ͔ Pipeline ͲͷΑ͏ʹύΠϓϥΠϯΛߏங͢Δ͔
  21. 32.

    1. Triggering pipeline using GitHub 2. Triggering pipeline using Pub/Sub

    3. Triggering pipeline using Docker Registry 1. VM machine deployment pipeline - AWS, GCP 2. Kubernetes deployment pipeline 3. Multi-Cloud, Multi-Region deployment pipeline 4. Canary deployment pipeline Integration ͲͷΑ͏ʹ֎෦γεςϜͱ࿈ܞ͢Δ͔ Pipeline ͲͷΑ͏ʹύΠϓϥΠϯΛߏங͢Δ͔ σϓϩΠύλʔϯ
  22. 33.

    1. Triggering pipeline using GitHub - Git push ΛτϦΨʔʹ Pipeline

    Λ࣮ߦ - GitHub ͔Β Clone ͖ͯͯ͠ΞʔςΟϑΝΫτʢ੒Ռ෺ʣΛऔಘ - Jenkins ͱ࿈ܞͯ͠ςετΛ࣮ߦ - ςετ௨աͨ͠ΒखಈঝೝΛܦͯσϓϩΠ GitHub Trigger Manual Judgement Testing
  23. 34.

    1. Triggering pipeline using GitHub - Configuration: ΞʔςΟϑΝΫτΛड͚औΔʢGit clone) -

    Test: Jenkins ͱ࿈ܞͯ͠ςετΛ࣮ߦ - खಈঝೝ: Slack ͳͲͰ௨஌ - Bake: Πϝʔδੜ੒ʢAMIɺDockerΠϝʔδʣ - AMIͷ৔߹͸ Packerʢby HashicorpʣͰੜ੒ - Deploy to Prod: Red/Black
  24. 35.

    2. Triggering pipeline using Pub/Sub - Test & Build ͸

    CI πʔϧଆʢࠓར༻͍ͯ͠Δ CI αʔϏεͷมߋͳ͠ʣ - ετϨʔδʢS3ͳͲʣ΁ͷ PUT Πϕϯτݕ஌ɺPub/Sub ܦ༝Ͱ Pipeline ࣮ߦ - खಈঝೝΛܦͯσϓϩΠ GitHub Testing & Build Event Upload Queue Trigger Event Upload Trigger Cloud Build Cloud Storage Cloud Pub/Sub Testing & Build Manual Judgement
  25. 36.

    2. Triggering pipeline using Pub/Sub - Configuration: ΞʔςΟϑΝΫτΛड͚औΔʢS3 or GCSʣ

    - खಈঝೝ: Slack ͳͲͰ௨஌ - Bake: Πϝʔδੜ੒ʢAMIɺDockerΠϝʔδʣ - Deploy to Prod: Red/Black
  26. 37.

    3. Triggering pipeline using Docker Registry - CIʢTestingʣ͸ผϑϩʔ - Docker

    ΠϝʔδΛ Docker Registry ʹ PushɺΠϕϯτݕ஌Ͱ Pipeline ࣮ߦ - खಈঝೝΛܦͯσϓϩΠ GitHub Testing & Build Trigger Push Testing & Build Push Trigger Cloud Build Container Registry Manual Judgement
  27. 38.

    3. Triggering pipeline using Docker Registry - Configuration: ΞʔςΟϑΝΫτΛड͚औΔ -

    खಈঝೝ: Slack ͳͲͰ௨஌ - Deploy to Prod: Red/Black
  28. 39.

    1. Triggering pipeline using GitHub 2. Triggering pipeline using Pub/Sub

    3. Triggering pipeline using Docker Registry 1. VM machine deployment pipeline - AWS, GCP 2. Kubernetes deployment pipeline 3. Multi-Cloud, Multi-Region deployment pipeline 4. Canary deployment pipeline Integration ͲͷΑ͏ʹ֎෦γεςϜͱ࿈ܞ͢Δ͔ Pipeline ͲͷΑ͏ʹύΠϓϥΠϯΛߏங͢Δ͔ σϓϩΠύλʔϯ
  29. 40.

    Auto Scaling group 1. AWS deployment - Server Group: Auto

    Scaling Group - Instance: EC2 app-v001 Auto Scaling group app-v002
  30. 41.

    Serving Instance
 Compute Engine Multiple Instances 1. GCP deployment -

    Server Group: Managed Instance Group - Instance: GCE app-v001 app-v002 Cloud Load Balancing Instance Group Serving Instance
 Compute Engine Multiple Instances Instance Group
  31. 42.

    1. VM machine deployment pipeline - Cofiguration: ΞʔςΟϑΝΫτΛड͚औΔ - Bake

    εςʔδͰ AMI ࡞੒ - Packerʢby HashicorpʣͰੜ੒ - Red/Black (Blue/Green) σϓϩΠ - खಈঝೝ - σϓϩΠޙɺಈ࡞ʹ໰୊͕ͳ͍͔ΛνΣοΫ - [໰୊ͳ͠ͷ৔߹] Red ༻ Server Group ͷΠϯελϯε਺ॖୀ - [໰୊͋Γͷ৔߹] ϩʔϧόοΫ
  32. 43.

    2. Kubernetes deployment pipeline - Cofiguration: ΞʔςΟϑΝΫτΛड͚औΔ - Red/Black (Blue/Green)

    σϓϩΠ - ϚχϑΣετϑΝΠϧͷద༻ʢV2ϓϩόΠμʣ - खಈঝೝ - σϓϩΠޙɺಈ࡞ʹ໰୊͕ͳ͍͔ΛνΣοΫ - [໰୊͋Γͷ৔߹] ϩʔϧόοΫ
  33. 44.

    3. Multi-Cloud, Multi-Region deployment pipeline - AWS, GCP ͦΕͧΕ US

    ͱ౦ژϦʔδϣϯʹσϓϩΠ - खಈঝೝ - σϓϩΠޙɺಈ࡞ʹ໰୊͕ͳ͍͔ΛνΣοΫ - [໰୊͋Γͷ৔߹] ϩʔϧόοΫ
  34. 45.

    4. Canary deployment pipeline - Cofiguration: ΞʔςΟϑΝΫτΛड͚औΔ - Deploy Canary

    - Canary (v2) ͷσϓϩΠ - Find Baseline Image / Deploy Baseline - Baseline (v1) ͷσϓϩΠ - Canary Analytics - Canary ͱ Baseline ΛൺֱɺείΞʹج͍ͮͯࣗಈతʹ൑அ - είΞΛຬͨͨ͠৔߹ɺຊ൪ʹσϓϩΠ
  35. 48.
  36. 49.

    AWS - AWS ͔Β Quickstart ͕ఏڙ͞Ε͍ͯΔ͕ɺߋ৽͞Ε͓ͯΒͣݹ͍ - Ubuntu 14ͱ͔ŋŋŋ -

    Spinnaker υΩϡϝϯτͱެ͔ࣜΒఏڙ͞Ε͍ͯΔ CloudFormation ςϯϓϨʔτͰߏங͍ͯ͘͠ - Amazon EKS Ͱ͸ಈ͔ͳ͔ͬͨʢҰ෦ͷαʔϏε͕ແݶʹΫϥογϡ & ࠶ىಈʣ - Amazon EC2 ΁ͷΠϯετʔϧ͸໰୊ͳ͠
  37. 51.

    Spinnaker ࢀߟจݙ - Continuous Delivery With Spinnaker https://www.spinnaker.io/publications/ebook/ - Kubernetes࣮ફΨΠυ

    Ϋϥ΢υωΠςΟϒΞϓϦέʔγϣϯΛࢧ͑Δٕज़ʢ๺ࢁ ৾ޗ / ૣ઒ ത ஶʣ - Slack https://join.spinnaker.io/

  38. 52.

    એ఻ - 9݄22೔ ٕज़ॻయ7 Ͱ Spinnaker ຊΛग़͢༧ఆͰ͢ʂ - ίϯςϯπʢԾʣ -

    Spinnaker ͷߏஙํ๏ʢAWSɺGCPʣ - ࠓ೔঺հͨ͠Ϣʔεέʔεผઃఆํ๏ - CLI ʹΑΔύΠϓϥΠϯߏங