Spinnakerで実現する、クラウド開発者のためのデプロイパターン入門 / Introduction to deployment patterns with Spinnaker

■イベント
イベント名 CloudNative Days Tokyo 2019
URL https://cloudnativedays.jp/cndt2019/

■登壇概要
タイトル：Spinnakerで実現する、クラウド開発者のためのデプロイパターン入門
登壇者： DSOC Infrastructure Group 大澤秀一

▼Sansan Builders Box
https://buildersbox.corp-sansan.com/

Transcript

1. 4QJOOBLFSͰ࣮ݱ͢Δ
   Ϋϥ΢υ։ൃऀͷͨΊͷσϓϩΠύλʔϯೖ໳ %40$
   *OGSBTUSVDUVSF
   (SPVQ
   େᖒलҰ 4BOTBOגࣜձࣾ
2. େᖒ लҰʢShuichi Ohsawaʣ DSOCʢData Strategy & Operation Centerʣ
 ɹInfrastructure Group ݉ CSIRT ID : @ohsawa0515 Blog : https://blog.jicoman.info/ աڈొஃྺ July Tech Festa 2017
 インフラエンジニアが Consul と Stretcher をつかったデプロイ改善で
 開発効率の向上に貢献した話 Vuls 祭り #2
 サーバレスで Vuls スキャン on AWS Google Cloud INSIDE Digital #2
 Windows Server VM Migration & Continuous Delivery with Spinnaker
3. Company Profile Sansanגࣜձ͕ࣾల։͢Δ2ͭͷࣄۀ ๏ਓ޲͚Ϋϥ΢υ໊ࢗ؅ཧαʔϏε ݸਓ޲໊͚ࢗΞϓϦ
5. Data Strategy and Operation Center ໊͕ࢗࣔ͢৘ใ ձࣾɾݸਓͷ৘ใ ͭͳ͕Γˠਓ຺ ڧΈ ۀछɺ৬छɺ໾৬ɺ஍Ҭ ͍ͭɺͲͷ෦ॺɾ໾৬ͷ
 ਓͱग़ձ͔ͬͨ ԣͷܨ͕Γͷ๛෋͔͞Βɺ
 ໊ࢗॴ༗ऀͷφϨοδɾܦݧ ͕͋ΔྖҬ͕ࣔࠦ͞Ε͍ͯΔ ໾৬໊ ࢯ໊ اۀ໊ ϝʔϧΞυϨε
   ి࿩൪߸
   
   63- 1 2 3 Sansanגࣜձࣾͷࣄۀ੒௕Λࠜװ͔Βࢧ͑Δ ʮग़ձ͍ʯͷσʔλϕʔεΛߏங
6. Data Strategy and Operation Center ໊ࢗऔΓࠐΈ എܠ෼཭ ը૾ิਖ਼ 1 ߲໨෼ׂ 2 ηΩϡϦςΟʔ߲໨ࡉ෼ׂɺ߲໨ೖྗ 3 νΣοΫˍิਖ਼ 5 Ϛʔδ 4 ϚΠΫϩλεΫ×ϚϧνιʔγϯάʹΑΔಠࣗͷ໊ࢗσʔλԽγεςϜ ໊ࢗσʔλԽγεςϜʮGEESʯ ηΩϡΞͳ؀ڥΛߏங
7. ࠓ೔ͷΰʔϧ - Spinnaker ͷϝϦοτΛ஌ͬͯ΋Β͏ - Spinnaker Λ׆༻ͨ͠σϓϩΠύλʔϯΛ஌ͬͯ΋Β͏͜ͱͰɺ
 օ༷ͷγεςϜʹద༻Ͱ͖ͳ͍͔Λߟ͑Δ͖͔͚ͬʹͳͬͯ΋Β͏
8. Agenda - Spinnaker ͷ֓ཁ - Spinnaker ͷίϯηϓτ - Spinnaker Λར༻ͨ͠σϓϩΠύλʔϯ - ͙͢ʹ࢝ΊΔʹ͸ - ·ͱΊ
9. ࿩͞ͳ͍͜ͱ - ଞύϒϦοΫΫϥ΢υɺϓϥΠϕʔτΫϥ΢υʹ͓͚ΔϢʔεέʔε - Microservies ΍ Service mesh ʹ͍ͭͯ
10. ࣭໰ᶃ Spinnaker ΛͲͷ͙Β͍஌ͬͯ·͔͢ʁ Α͘஌ͬͯΔ ໊લ͸ฉ͍ͨ ͜ͱ͋Δ ͳʹͦΕʁ ඒຯ͍͠ͷʁ ͳΜͱͳ͘ ஌ͬͯΔ
11. ࣭໰ᶄ ͲͷΫϥ΢υαʔϏεΛ͔͍ͭͬͯ·͔͢ʁ Amazon Web Services ʢAWSʣ Microsoft Azure ϓϥΠϕʔτΫϥ ΢υɾͦͷଞ Google Cloud Platoform ʢGCPʣ
12. 4QJOOBLFS
    ͷ֓ཁ
13. Spinnaker ͱ͸ - OSS ͷ ܧଓతσϦόϦʔ (Continuous Delivery; CD) ϓϥοτϑΥʔϜ - Netflix ͕։ൃɺ2015೥ʹOSSԽ - Netflix ͰഓΘΕͨܧଓతσϦόϦʔͷϓϥΫςΟε͕൓ө - GoogleɺMicrosoftɺPivotal ͱఏܞ͢Δ͜ͱͰɺϚϧνΫϥ΢υରԠͷՃ଎౓޲্ - OracleɺTargetɺVeritasɺSchibstedɺArmoryɺKenzan ͳͲͷاۀ΋։ൃʹࢀՃ
14. ௕ॴ - ϚϧνΫϥ΢υରԠ AWSɺGCPɺAzureɺOpenStack etc - ϚϧνϓϩόΠμରԠ Amazon EC2, Google Compute Engine, Kubernetes etc - ෳ਺ͷσϓϩΠઓུΛαϙʔτ - Red/BlackʢGlue/Greenʣ͕ඪ४αϙʔτɺCanary σϓϩΠ΍ΧελϚΠζ΋Մೳ - GUI ্ͰύΠϓϥΠϯʢϑϩʔʣΛ࡞ΕΔ - CI πʔϧͱͷ࿈ܞखஈ͕ଟ͍ɻCD ͱͯ͠ͷ໾ׂʹಛԽ͍ͯ͠Δ
15. https://jp.techcrunch.com/2019/07/23/2019-07-22-google-cloud-makes-it-easier-to-set-up-continuous-delivery-with-spinnaker/
16. ୹ॴ - ηϧϑϗεςΟϯά - Spinnaker ࣗ਎ͷσϓϩΠʹͦΕͳΓͷϚγϯϦιʔεΛཁ͢Δ - AWS: m5.xlarge, GCP: n1-highmem-4ʢCPU 4, Memory 16GBʣ - υΩϡϝϯτ͕গͳ͍ɻυΩϡϝϯτͱ࣮ࡍͷ UI ΍݁Ռ͕ҟͳΔ৔߹͕͋Δ - όά͕ଟ͍ʢύΠϓϥΠϯͷૢ࡞ܥɻσϓϩΠʹؔ͢Δόά͸͋·Γૺ۰͠ͳ͍ʣ - Spinnaker ಠࣗͷઃఆ͕ଟ͘ɺ࢖͍͜ͳ͢ͷ͕େม - Spinnaker ͓͡͞ΜΛੜΈग़͢ةݥੑ
17. 4QJOOBLFS
    ͷίϯηϓτ
18. ίϯηϓτ - Application management - - ΞϓϦέʔγϣϯͷ֓೦ΛϞσϧԽ͢Δ͜ͱͰ
 ϚϧνΫϥ΢υͰ΋ಉ͡Α͏ʹѻ͑Δ - Instance: Amazon EC2, Google Compute Engine(GCE), Kubernetes Pod etc - Server Group: Instance ͷάϧʔϓԽ - Auto Scaling GroupʢAWSʣɺInstance Group ʢGCPʣɺReplicaSetʢKubernetesʣ - σϓϩΠՄೳͳΞʔςΟϑΝΫτ
 ʢVMΠϝʔδɺDockerΠϝʔδɺϚχϑΣετ etcʣɺ
 Πϯελϯε਺ɺΦʔτεέʔϦϯάϙϦγʔ͔Βߏ੒ - Cluster: Server Group Λ࿦ཧతʹάϧʔϓԽ - Application: Cluster ͷू·Γ - Firewall - Load Balancer https://www.spinnaker.io/concepts/#application-management
19. ίϯηϓτ - Application deployment - - Pipeline: σϓϩΠϑϩʔΛߏ੒ - Stage - Triggers - Notifications - Stage Pipeline ͷߏ੒ཁૉɻύΠϓϥΠϯ͕࣮ߦ͢ΔΞΫγϣϯΛهड़ - Infrastructure Stagesʢج൫ͱͳΔΠϯϑϥετϥΫνϟΛߏஙʣ - External Systems Integrationsʢ֎෦γεςϜͱͷ࿈ܞʣ - Controlling FlowʢঝೝɺλΠϛϯάɺ෼ذϩδοΫͳͲɺύΠϓϥΠϯͷྲྀΕΛ੍ޚ͢Δʣ - Deployment strategies - Red/Black (Blue/Green) - Canary
20. Infrastructure Stages - Ϧιʔε࡞੒ɺߋ৽ɺ࡟আͳͲج൫ͱͳΔΠϯϑϥετϥΫνϟΛߏங͢Δεςʔδ - ϚϧνΫϥ΢υΛར༻͍ͯ͠Δ৔߹ɺҰ؏ͨ͠ํ๏ͰͦΕͧΕͷΫϥ΢υʹల։Ͱ͖ΔͨΊɺ
 ೝ஌తͳෛՙ͕ܰݮ͞ΕΔ - Bake (VMΠϝʔδ, DockerΠϝʔδͷ࡞੒) - Find Image From {Cluster, Tag} ʢΠϝʔδΛಛఆ͢Δʣ - DeployʢRed/Black, ϚχϑΣετద༻ʣ - {Disable, Shrink, Scale Down, Rollback} ClusterʢClusterͷૢ࡞ʣ - {Clone, Enable, Disable, Resize, Destroy} Server GroupʢServer Groupͷૢ࡞ʣ - Run Job (ίϯςφ࣮ߦ) - Script (Jenkins্Ͱ೚ҙͷεΫϦϓτ࣮ߦ) ओͳεςʔδ
21. External Systems Integrations - ֎෦γεςϜͱͷ࿈ܞ - Continuous IntegrationʢCIʣ - Jenkins / TravisCI / Wercker / Concourse CI - Webhook - CircleCIͳͲαϙʔτ͞Ε͍ͯͳ͍αʔϏεͷ৔߹ʹར༻
22. Controlling Flow - ύΠϓϥΠϯͷྲྀΕΛ੍ޚ͢Δεςʔδ - Check Preconditions (৚݅෼ذ) - Manual Judgment (खಈঝೝ) - Waitʢࢦఆ͞Εͨඵ਺Λ଴ػ) - Pipeline (ผͷύΠϓϥΠϯΛ࣮ߦ)
23. Triggers - ύΠϓϥΠϯͷ։࢝Λ੍ޚ - Time-based triggers - Manual - Cron - Event-based triggers - Git (GitHub, GitLab etc) - Continuous Integration (Jenkins etc) - Docker Registry (Docker Hub, GitLab Container Registry, Amazon ECR, Google Container Registry etc) - Pipeline - Pub/Sub (Cloud Pub/Sub, Amazon SNS etc)
24. Notifications - Email - Slack - SMS via Twilio - Pub/Sub - Spinnaker ͔Β֎෦γεςϜ΁ͷ࿈ܞ΋Մೳ - ύΠϓϥΠϯͷ։࢝ɺ੒ޭɺࣦഊ࣌ʹ௨஌ - ֤εςʔδͰ΋௨஌Մೳ ରԠ͍ͯ͠Δ௨஌ઌ
25. Deployment strategies - Red/Black (Blue/Green) https://www.spinnaker.io/concepts/#deployment-strategies - ՔಇதʢBlueʣ͔Β৽όʔδϣϯʢGreenʣʹগͣͭ͠ೖΕସ͍͑ͯ͘ - Blue γεςϜʹมߋΛՃ͑ͳ͍ʢImmutable Infrastructureʣ - ϩʔϧόοΫʢ੾Γ໭͠ʣ͕༰қ - ޓ׵ੑ͕ͳ͍มߋʢDBϚΠάϨʔγϣϯͳͲʣ͕͋Δ৔߹͸ผ్ରԠ͕ඞཁ
26. Deployment strategies - Canary - 3ͭͷΫϥελΛߏ੒͠ɺͦΕͧΕʹτϥϑΟοΫΛྲྀ͢ - Production: ຊ൪Ϋϥελʢෳ਺Πϯελϯε) - Baseline: ຊ൪Ϋϥελͱಉ͡όʔδϣϯͷίʔυɻCanary ͱൺֱ͢Δʢ௨ৗ 3Πϯελϯεʣ - Canary: ৽όʔδϣϯͷίʔυ΍৽ߏ੒ʢBaseline ͱಉ͘͡ 3Πϯελϯεʣ Load Balancer Production (v1) Traffic Many Instances Baseline (v1) 3+ Instances Canary (v2) 3+ Instances Metrics Canary Analysis Refer: https://medium.com/netflix-techblog/automated-canary-analysis-at-netflix-with-kayenta-3260bc7acc69
27. Deployment strategies - Canary (Evaluation) - Baseline ͱ Canary ͸ಉ͡৚݅Ͱἧ͑ͯධՁ͢Δඞཁ͕͋Δ - ಉ࣌ؒ͡ - ಉ͡αΠζ - ಉ͡τϥϑΟοΫྔ - े෼ͳ͕࣌ؒඞཁ - ධՁ࣌ؒ: 3࣌ؒ - ΧφϦΞ෼ੳִؒ: 1࣌ؒ - ࣌ܥྻσʔλ: 50ݸ+ Load Balancer Baseline (v1) 3+ Instances Canary (v2) 3+ Instances Traffic
28. Deployment strategies - Canary (Judgement) - ϝτϦΫεΛऩूʢlatency, errors, saturation etcʣ - ෳ਺ϝτϦΫεΛάϧʔϓԽͰ͖Δ - ϝτϦΫεͷऩूઌͱͯ͠ɺStackdriver, Prometheus, Datadog ͕αϙʔτ͞Ε͍ͯΔ - Τϥʔ਺ɺࣦഊ਺Λ௚઀ධՁͤͣɺϝτϦΫεͷภࠩΛධՁ͢Δ - Canary ͕ Baseline ͱಉ౳Ͱ͋Δ͔ or ѱ͘ͳ͍ͬͯͳ͍͔ - ͖͍͠஋ʹରͯ͠ɺͦͷϝτϦΫε͕ʮ߹֨ʯ͍ͯ͠Δ͔Λ൑அ - ࣗಈΧφϦΞ෼ੳʢKayenta) ͰείΞϦϯάͯ͠ϦϦʔε͢΂͖͔Λࣗಈ൑அ - ϝτϦΫεͷ૯਺ʹରͯ͠ʮ߹֨ʯ͍ͯ͠ΔϝτϦΫεͷׂ߹Λܭࢉ - ྫʣ10 ϝτϦΫεதɺ߹֨ϝτϦΫε͕ 8 ͷ৔߹ɺ80 ͱͯ͠ࢉग़ - ϝτϦΫεάϧʔϓຖʹॏΈ෇͚͢Δ͜ͱ΋Ͱ͖Δ
29. Pipeline ྫ https://www.spinnaker.io/concepts/#pipeline - TEST ؀ڥͰطʹσϓϩΠ͞Ε͍ͯΔΠϝʔδΛར༻͠ɺCanary ͰগྔͷτϥϑΟοΫΛςετͯ͠ ͔ΒɺPROD ʹ Red/Black σϓϩΠ͕࣮ߦ͞ΕΔ - Canary ͷϝτϦοΫऩूʢWait 30 mins) ͱ खಈঝೝͷ྆ํ͕׬ྃ͢ΔͱσϓϩΠ͕࣮ߦ͞ΕΔ - σϓϩΠ׬ྃޙɺCanary ༻ Server Group ͷఀࢭͱ 2࣌ؒ଴͔ͬͯΒݹ͍ Server Group Λഁغ͢Δ
30. Spinnaker Λར༻ͨ͠σϓϩΠύλʔϯ
31. σϓϩΠύλʔϯ 1. Triggering pipeline using GitHub 2. Triggering pipeline using Pub/Sub 3. Triggering pipeline using Docker Registry 1. VM machine deployment pipeline - AWS, GCP 2. Kubernetes deployment pipeline 3. Multi-Cloud, Multi-Region deployment pipeline 4. Canary deployment pipeline Integration ͲͷΑ͏ʹ֎෦γεςϜͱ࿈ܞ͢Δ͔ Pipeline ͲͷΑ͏ʹύΠϓϥΠϯΛߏங͢Δ͔
32. 1. Triggering pipeline using GitHub 2. Triggering pipeline using Pub/Sub 3. Triggering pipeline using Docker Registry 1. VM machine deployment pipeline - AWS, GCP 2. Kubernetes deployment pipeline 3. Multi-Cloud, Multi-Region deployment pipeline 4. Canary deployment pipeline Integration ͲͷΑ͏ʹ֎෦γεςϜͱ࿈ܞ͢Δ͔ Pipeline ͲͷΑ͏ʹύΠϓϥΠϯΛߏங͢Δ͔ σϓϩΠύλʔϯ
33. 1. Triggering pipeline using GitHub - Git push ΛτϦΨʔʹ Pipeline Λ࣮ߦ - GitHub ͔Β Clone ͖ͯͯ͠ΞʔςΟϑΝΫτʢ੒Ռ෺ʣΛऔಘ - Jenkins ͱ࿈ܞͯ͠ςετΛ࣮ߦ - ςετ௨աͨ͠ΒखಈঝೝΛܦͯσϓϩΠ GitHub Trigger Manual Judgement Testing
34. 1. Triggering pipeline using GitHub - Configuration: ΞʔςΟϑΝΫτΛड͚औΔʢGit clone) - Test: Jenkins ͱ࿈ܞͯ͠ςετΛ࣮ߦ - खಈঝೝ: Slack ͳͲͰ௨஌ - Bake: Πϝʔδੜ੒ʢAMIɺDockerΠϝʔδʣ - AMIͷ৔߹͸ Packerʢby HashicorpʣͰੜ੒ - Deploy to Prod: Red/Black
35. 2. Triggering pipeline using Pub/Sub - Test & Build ͸ CI πʔϧଆʢࠓར༻͍ͯ͠Δ CI αʔϏεͷมߋͳ͠ʣ - ετϨʔδʢS3ͳͲʣ΁ͷ PUT Πϕϯτݕ஌ɺPub/Sub ܦ༝Ͱ Pipeline ࣮ߦ - खಈঝೝΛܦͯσϓϩΠ GitHub Testing & Build Event Upload Queue Trigger Event Upload Trigger Cloud Build Cloud Storage Cloud Pub/Sub Testing & Build Manual Judgement
36. 2. Triggering pipeline using Pub/Sub - Configuration: ΞʔςΟϑΝΫτΛड͚औΔʢS3 or GCSʣ - खಈঝೝ: Slack ͳͲͰ௨஌ - Bake: Πϝʔδੜ੒ʢAMIɺDockerΠϝʔδʣ - Deploy to Prod: Red/Black
37. 3. Triggering pipeline using Docker Registry - CIʢTestingʣ͸ผϑϩʔ - Docker ΠϝʔδΛ Docker Registry ʹ PushɺΠϕϯτݕ஌Ͱ Pipeline ࣮ߦ - खಈঝೝΛܦͯσϓϩΠ GitHub Testing & Build Trigger Push Testing & Build Push Trigger Cloud Build Container Registry Manual Judgement
38. 3. Triggering pipeline using Docker Registry - Configuration: ΞʔςΟϑΝΫτΛड͚औΔ - खಈঝೝ: Slack ͳͲͰ௨஌ - Deploy to Prod: Red/Black
39. 1. Triggering pipeline using GitHub 2. Triggering pipeline using Pub/Sub 3. Triggering pipeline using Docker Registry 1. VM machine deployment pipeline - AWS, GCP 2. Kubernetes deployment pipeline 3. Multi-Cloud, Multi-Region deployment pipeline 4. Canary deployment pipeline Integration ͲͷΑ͏ʹ֎෦γεςϜͱ࿈ܞ͢Δ͔ Pipeline ͲͷΑ͏ʹύΠϓϥΠϯΛߏங͢Δ͔ σϓϩΠύλʔϯ
40. Auto Scaling group 1. AWS deployment - Server Group: Auto Scaling Group - Instance: EC2 app-v001 Auto Scaling group app-v002
41. Serving Instance
 Compute Engine Multiple Instances 1. GCP deployment - Server Group: Managed Instance Group - Instance: GCE app-v001 app-v002 Cloud Load Balancing Instance Group Serving Instance
 Compute Engine Multiple Instances Instance Group
42. 1. VM machine deployment pipeline - Cofiguration: ΞʔςΟϑΝΫτΛड͚औΔ - Bake εςʔδͰ AMI ࡞੒ - Packerʢby HashicorpʣͰੜ੒ - Red/Black (Blue/Green) σϓϩΠ - खಈঝೝ - σϓϩΠޙɺಈ࡞ʹ໰୊͕ͳ͍͔ΛνΣοΫ - [໰୊ͳ͠ͷ৔߹] Red ༻ Server Group ͷΠϯελϯε਺ॖୀ - [໰୊͋Γͷ৔߹] ϩʔϧόοΫ
43. 2. Kubernetes deployment pipeline - Cofiguration: ΞʔςΟϑΝΫτΛड͚औΔ - Red/Black (Blue/Green) σϓϩΠ - ϚχϑΣετϑΝΠϧͷద༻ʢV2ϓϩόΠμʣ - खಈঝೝ - σϓϩΠޙɺಈ࡞ʹ໰୊͕ͳ͍͔ΛνΣοΫ - [໰୊͋Γͷ৔߹] ϩʔϧόοΫ
44. 3. Multi-Cloud, Multi-Region deployment pipeline - AWS, GCP ͦΕͧΕ US ͱ౦ژϦʔδϣϯʹσϓϩΠ - खಈঝೝ - σϓϩΠޙɺಈ࡞ʹ໰୊͕ͳ͍͔ΛνΣοΫ - [໰୊͋Γͷ৔߹] ϩʔϧόοΫ
45. 4. Canary deployment pipeline - Cofiguration: ΞʔςΟϑΝΫτΛड͚औΔ - Deploy Canary - Canary (v2) ͷσϓϩΠ - Find Baseline Image / Deploy Baseline - Baseline (v1) ͷσϓϩΠ - Canary Analytics - Canary ͱ Baseline ΛൺֱɺείΞʹج͍ͮͯࣗಈతʹ൑அ - είΞΛຬͨͨ͠৔߹ɺຊ൪ʹσϓϩΠ
46. ͙͢ʹ࢝ΊΔʹ͸
47. GCP - GKE ͰηοτΞοϓͰ͖Δ؀ڥΛఏڙ͞Ε͍ͯΔ - https://github.com/GoogleCloudPlatform/spinnaker-for-gcp - Cloud Shell ্ͰίϚϯυΛίϐϖ͢Δ͚ͩͰ Production-ready ͳ؀ڥΛߏங - গʑ͕͔͔࣌ؒΔ͕ҰൃͰΠϯετʔϧͯ͘͠ΕΔͷͰΊͬͪΌָ
49. AWS - AWS ͔Β Quickstart ͕ఏڙ͞Ε͍ͯΔ͕ɺߋ৽͞Ε͓ͯΒͣݹ͍ - Ubuntu 14ͱ͔ŋŋŋ - Spinnaker υΩϡϝϯτͱެ͔ࣜΒఏڙ͞Ε͍ͯΔ CloudFormation ςϯϓϨʔτͰߏங͍ͯ͘͠ - Amazon EKS Ͱ͸ಈ͔ͳ͔ͬͨʢҰ෦ͷαʔϏε͕ແݶʹΫϥογϡ & ࠶ىಈʣ - Amazon EC2 ΁ͷΠϯετʔϧ͸໰୊ͳ͠
50. ·ͱΊ - Spinnaker ʹ͍ͭͯ঺հ - Spinnaker ͷϝϦοτɾσϝϦοτɺίϯηϓτ - Spinnaker Λར༻ͨ͠σϓϩΠύλʔϯΛ঺հ - ֎෦αʔϏεͱͷ࿈ܞύλʔϯ - AWSɺGCP ʹ͓͚ΔσϓϩΠύλʔϯ
51. Spinnaker ࢀߟจݙ - Continuous Delivery With Spinnaker https://www.spinnaker.io/publications/ebook/ - Kubernetes࣮ફΨΠυ Ϋϥ΢υωΠςΟϒΞϓϦέʔγϣϯΛࢧ͑Δٕज़ʢ๺ࢁ ৾ޗ / ૣ઒ ത ஶʣ - Slack https://join.spinnaker.io/

52. એ఻ - 9݄22೔ ٕज़ॻయ7 Ͱ Spinnaker ຊΛग़͢༧ఆͰ͢ʂ - ίϯςϯπʢԾʣ - Spinnaker ͷߏஙํ๏ʢAWSɺGCPʣ - ࠓ೔঺հͨ͠Ϣʔεέʔεผઃఆํ๏ - CLI ʹΑΔύΠϓϥΠϯߏங