Why and how we build a unified CD system

Transcript

1. nghialv Developer Productivity Team, CyberAgent Oct 08, 2021 ౷ҰͳCDγεςϜΛߏஙͨ͠࿩ ๻ͨͪࢲͨͪͷCI/CD͸͜Εͩʂ- cndjp #17
2. ࣗݾ঺հ @nghialv @nghialv2607 @nghialv Le Van Nghia - ΪΞ
3. ࣗݾ঺հ - ৬ྺ @CyberAgent • PipeCDΛ։ൃɾӡ༻ - DPࣨ • Work fl ow Automation SystemΛ։ൃɾӡ༻ - OSSS • Feature Flags SystemΛ։ൃɾӡ༻ - AbemaTV • Prometheus & GrafanaͰMonitoring SystemΛߏஙɾӡ༻ - AbemaTV • DeploymentπʔϧΛ։ൃɾӡ༻ - AbemaTV • Microservicesɾج൫पΓ - AbemaTV
4. ࠓ೔ͷ࿩͢಺༰ • CyberAgentͰCDʹؔ͢Δ՝୊ • ౷ҰͳσϦόϦʔج൫Λݕ౼ • PipeCDͷ஀ੜ • ݱࡏͷPipeCD @ CyberAgent • PipeCDͷࠓޙ
5. CyberAgentͰCDʹؔ͢Δ՝୊
6. ϓϩμΫτͷߏ੒ ABEMA AWA WinTicket CyberZ AI Studio ... • ଟ͘ͷϓϩμΫτ͕ଘࡏ͠ɺ૿Ճத • ֤ϓϩμΫτͰ͸ࣗ෼ʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબ΂Δ Group • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾ಺Ҡಈ͸Մೳ ʢ਺ेݸ͕͋Δʣ
7. ϓϩμΫτͷߏ੒ ABEMA AWA WinTicket CyberZ AI Studio ... • ֤ϓϩμΫτͰ͸ࣗ෼ʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબ΂Δ Group • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾ಺Ҡಈ͸Մೳ ʢ਺ेݸ͕͋Δʣ CD΋ϓϩμΫτΤϯδχΞʹࣗ༝ʹ೚ͤΔͷͰνʔϜ͝ͱʹঢ়گ͕ҟͳΔ • ଟ͘ͷϓϩμΫτ͕ଘࡏ͠ɺ૿Ճத
8. ݩʑͷCDͷঢ়گ શମత͸ͳΜͰ΋͋Δ͜ͱʹͳͬͯ͠·ͬͨ खಈ Jenkins CircleCI TravisCI DroneCI Harness Concource ArgoCD GH Actions Spinnaker Flux Code Deploy Cloud Build Cloud Deploy ୲౰ऀ͕खಈͰ΍Δ CIͰCDΛ΍Δ (CI Ops) ઐ༻CDΛ࢖͏ ࣗ࡞ ChatOps
9. CDʹؔ͢Δ՝୊ Multi Cloud Product • σϦόϦͷ੒ख़౓͕ߴ͘ͳ͍ͱ͜Ζ͕ଘࡏ • खಈϦϦʔεͷνΣοΫϦετ͕େม • Canary, BlueGreenͳͲͷ҆શରࡦ͕ͳ͍ • ೚ͤΔͨΊɺࣗ෼ͰCDͷߏஙͱӡ༻͕େม • ౷ҰੑʢҰ؏ੑʣ͕௿͘ͳ͍ͬͯΔ • ࣾ಺ελϯμʔυΛ੒ཱ͢Δ͜ͱ͕೉͘͠ͳΔ • ࣾ಺ͷϕετϓϥΫςΟεͷීٴ͕େม • શମతͳӡ༻ίετ͕ߴ͍ • Developer ExperienceʹӨڹͯ͠͠·͏ • ࣾ಺ҠಈͰ΋Onboardingίετ͕ൃੜ ٕज़ελοΫ͕ಉ͡Ͱ΋
 ϓϩμΫτຖʹCDγεςϜɾϓϩηε͕ҟͳΔ ಉ͡ϓϩμΫτͰ΋
 ෳ਺CDγεςϜɾϓϩηε͕ଘࡏ͢Δ Product A Product B CD System X CD System Y CD System X for GCP services CD System Y for AWS Services CD System Z for Infra
10. ౷ҰͳσϦόϦʔج൫Λݕ౼
11. CIͱCDΛ׬શతʹ෼཭ TestͰૣظతʹ։ൃऀ΁ϑΟʔυόοΫΛఏڙ ArtifactΛHost Environment΁σϓϩΠɾϩʔϧόοΫ BuildͰImmutable ArtifactΛੜ੒͢Δ Host Environment (ClusterͳͲʣ΁ܨ͕Βͳ͍ Host Environment (ClusterͳͲʣ΁௚઀తʹܨ͕Δ ੜ੒͞ΕͨArtifactʹؔ͢ΔϑΟʔυόοΫΛૣΊʹఏڙ
12. ౷ҰͳCDγεςϜ Delivery Infrastructure Product Team͸࠷େͳσϦόϦύϑΥʔϚϯεΛ ग़ͤͳ͕Βɺࣗ෼ʹ߹͏ٕज़ελοΫͱσϦόϦ ઓུΛબͿϑϦʔμϜ͕͋Δ Platform Team͸ॊೈͳDelivery InfrastructureΛ
 ఏڙͱCDͷϕετϓϥΫςΟεΛීٴ Platform Team & Product Teamͷ྆ํʹϝϦοτΛग़ͤΔ Standard/Consistency vs Freedomͷྑ͍όϥϯεΛऔΕΔ Ͳ͏͢Ε͹ɺ
13. Platform Team & Product Team Management Security γεςϜͷӡ༻͸ίετ͕ߴ͍͔Ͳ͏͔
 ϓϩμΫτಋೖͷεέʔϥϏϦςΟ ϓϩμΫτͷΫϨσϯγϟϧΛͲ͏؅ཧ͞ΕΔ͔
 
 CDηΩϡϦςΟʔपΓͷϕετϓϥΫςΟεΛϓϩμΫτνʔϜ΁ීٴͰ͖Δ͔Ͳ͏͔ Automation Visibility / Accessibility Control / 
 Flexibility νʔϜʹϑΟοτٕज़ελοΫɾσϦόϦख๏Λࣗ༝ʹબ͹ΕΔͷ͔
 
 νʔϜͷݖݶΛద੾ʹ؅ཧͰ͖Δͷ͔ ϦϦʔεϓϩηεͷதʹɺਓؒͷखಈλεΫ͕Ͳ͜·Ͱ࡟ݮͰ͖Δͷ͔
 
 ϦϦʔεத΋ޙ΋ܧଓతʹ໰୊Λݕ஌Ͱ͖Δͷ͔ ։ൃऀ΁े෼ͳϑΟʔυόοΫΛ଎ΊʹఏڙͰ͖Δ͔
 ໰୊͕͋Δ࣌ʹݪҼ΋ؚΉঢ়ଶΛͪΌΜͱݟ͑Δͷ͔
 ؅ཧऀ΁νʔϜͷσϦόϦʔύϑΥʔϚϯεΛͪΌΜͱݟͤΔͷ͔ Platform
 Team Product
 Team
14. PipeCDͷ஀ੜ
15. PipeCD
16. PipeCD 🤍 OSS Thanks to the contributors of PipeCD! https://github.com/pipe-cd/pipe https://pipecd.dev https://pipecd.dev/docs 27 Contributors 4 Full-time contributors 1 Part-time contributor
 1800 Pull requests ʢ෭ۀʣ
17. PipeCD 🤍 GitOps • GitOpsΛ࠾༻ • GitʹશͯͷCon fi gurationΛ؅ཧ • શͯͷΦϖϨʔγϣϯ͕Git Pull Requestܦ༝Λߦ͑Δ • Πϯϑϥ͔ΒΞϓϦέʔγϣϯ·Ͱ౷ҰͳGitOps • Kubernetes • Terraform • CloudRun • AWS Lambda • ECS, Fargate • ...
18. PipeCD͸ͲͷΑ͏ʹզʑͷ՝୊Λղܾ͢Δ͔ Automation Visibility / Accessibility Control / 
 Flexibility Management Security Platform Team Product Team
19. Platform Team - Management • ӡ༻ίετ͕௿͍ • શͯͷσϓϩΠϝϯτ͸ҰͭͷγεςϜͷΈ͕ඞཁ • PipeCDͷશͯͷίϯϙʔωϯτ͕εςʔτϨε • ετϨʔδ͸ϚωʔδυαʔϏεΛར༻͢Δ͜ͱ͕Մೳ • ϓϩμΫτଆ͸γϯάϧόΠφϦͷPiped agentΛΠϯετʔϧͷΈ • ϓϩμΫτͷεέʔϥϏϦςΟ • ৽نͷϓϩμΫτͷ௥Ճ͕୯౬ • Ͳͷ؀ڥͰ΋ಈ͚Δઃܭ • ωοτϫʔΫ੍ݶͷPrivate Cloud͔ΒPublic Cloud·Ͱ • খن໛νʔϜ͔Βେن໛νʔϜ·Ͱ Product B Piped Control Plane Firewall
 Friendly Outbout
 Requests
 Only Product A Piped Piped Piped
20. Platform Team - Security • ϓϩμΫτνʔϜͷΫϨσϯγϟϧ͸֎ʹҰ੾ग़ͳ͍ઃܭ • Control-planeʹ΋อଘɾ؅ཧ͠ͳ͍ • GitOpsͰͷSecret؅ཧͷϕετϓϥΫςΟεͱͯ͠ͷbuilt-inػೳΛؚΉ • Piped agent͕Ξ΢τό΢ϯυϦΫΤετͷΈΛߦ͏ͷͰɺެ։ϙʔτͳͲ͸ඞཁ͕ͳ͍
21. Product Team - Visibility / Accessibility • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε https://pipecd.dev/docs/user-guide/triggering-a-deployment
22. Product Team - Visibility / Accessibility https://pipecd.dev/docs/user-guide/application-live-state • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε
23. Product Team - Visibility / Accessibility https://pipecd.dev/docs/user-guide/plan-preview • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε Git di ff is enought? No. e.g. using remote Helm chart, Kustomize package, Terraform module... Reviewer needs more early feedback to merge PR with con fi dence Better to see dry-run result, terraform plan, deployment policy... on the PR then Git di ff is just showing the change of version number
24. Product Team - Visibility / Accessibility Terraform Deployment CloudRun Deployment • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε Scale In Scale Out Rollout New Image Update 
 Con fi g Deploy Infra Deploy Serverless Rollback Deploy Kubernetes Constant process for all operations
25. Product Team - Automation https://pipecd.dev/docs/user-guide/rolling-back-a-deployment • σϓϩΠͷΠϯύΫτΛࣗಈ෼ੳ • ϝτϦΫεɾϩάɾhttp • աڈͷσʔλɾcanary-baselineͷൺֱ • σϓϩΠதʹ໰୊͕ൃੜ͢ΔͱࣗಈϩʔϧόοΫ • Gitͱ࣮ࡍͷঢ়ଶͷCon fi guration DriftΛࣗಈݕ஌ • ৽ίϯςφΠϝʔδɾHelm Chart͕ग़Δͱ
 ࣗಈσϓϩΠͷEventWatcher https://pipecd.dev/docs/user-guide/con fi guration-drift-detection
26. Product Team - Control / Flexibility • Piped AgentΛ޷͖ͳελΠϧͰ૊Έ߹ΘͤΔ͜ͱ͕Մೳ • 1 Piped AgentͰશͯ؅ཧύλʔϯ • ؀ڥຖʹઐ༻ͷPiped Agentύλʔϯ • ޷͖ͳ৔ॴʹΠϯετʔϧʢKubernetes podɺVMʹதɺFargateαʔϏε...ʣ • σϓϩΠઓུ͸ࣗ༝ʹ૊Έ߹Θͤɾఆ͕ٛՄೳ • Quick Sync • Progressive Sync (Canary, BlueGreen...) • νʔϜʹ߹͏ϓϩόΠμʔΛબ΂ΒΕΔ • Cloud Provider (GCP, AWS, Azure, Private Cloud) • Analysis Provider (Prometheus, Datadog, Stackdriver...) https://github.com/pipe-cd/examples/
 blob/master/kubernetes/canary/.pipe.yaml
27. PipeCD Platform Team͕Delivery InfrastructureΛఏڙͰ͖ɺ
 ϕετϓϥΫςΟεΛܧଓతʹ࠾༻ɾීٴͰ͖Δ Product Team͕CDͷߏஙɾӡ༻͕ෆཁʹͳΓɺҰճಋೖ͢Δ͚ͩͰɺ
 ࣗ෼ʹ߹͏σϦόϦख๏Λ࠾༻Ͱ͖ɺܧଓతʹվળͰ͖Δ ͜ΕͰɺ
28. ݱࡏͷPipeCD @ CyberAgent
29. νʔϜͱγεςϜͷߏ੒ • Platform Team • PipeCDΛ։ൃ • ࣾ಺༻PipeCD Control-planeΛӡ༻ • ֤Product Team • Piped agentΛΠϯετʔϧ
 • ࣗ෼ʹ߹͏σϓϩΠϝϯτछྨΛ࠾༻ Control-Plane https://pipecd.dev/docs/operator-manual/piped/installation https://github.com/pipe-cd/examples
30. ΞϓϦέʔγϣϯɾαʔϏε͕૿Ճத 0 225 450 675 900 2020/10 2020/12 2021/02 2021/04 2021/06 2021/8 806 Applications/Servicesʹୡ੒ CyberAgentʹPipeCDͰӡ༻͍ͯ͠Δ Kubernetes, Terraform, Lambda, CloudRun, Fargate... 
 ͷΞϓϦέʔγϣϯɾαʔϏε਺
31. PipeCDͷࠓޙ
32. ֤࣠Λ͞ΒʹڧԽ Automation Visibility / Accessibility Control / 
 Flexibility Management Security
33. • ࠓ·ͰͷPiped agentͷӡ༻ • Product Team͕Piped agentΛΠϯετʔϧ • ৽͍͠όʔδϣϯ͕͋Δͱ࠶Πϯετʔϧ • ͜Ε͔Β • ҰճͷΈΠϯετʔϧ͢Δ • Web consoleͰόʔδϣϯΛΞοϓͰ͖Δ • ͦΕͰӡ༻͕͞ΒʹָʹͳΔ Management - Remote Upgrade ͜ͷػೳ͸དྷिʹϦϦʔε༧ఆ Product B Piped Control Plane Firewall
 Friendly Outbout
 Requests
 Only Product A Piped Piped Piped
34. Visibility / Accessiblity • InsightsͰσϦόϦʔύϑΥʔϚϯεΛՄࢹԽ • Deployස౓ɺLead Time, ࣦഊ཰, MTTR... • ApplicationͷϦιʔεͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • Terraform, CloudRun, ECS... • Deployment StageͷϩάΛվળ • Plan-Previewʹ΋ͬͱϑΟʔυόοΫΛՃ͑Δ • Kubernetes validating webhook, dry-run݁Ռ, Terraform sentinelͳͲͷDeployment Policy ֎ͷϢʔβʔ޲͚ͷPlayground؀ڥΛ४උதʂ
 https://play.pipecd.dev
35. Automation • LogͷσʔλͰσϓϩΠϝϯτΛࣗಈ෼ੳ • աڈͷMetricsσʔλͱͷൺֱͰࣗಈ෼ੳ • Primary/Baseline & CanaryͷMetricsσʔλͷൺֱͰࣗಈ෼ੳ • Terraform, CloudRun, LambdaͳͲͷࣗಈDrift Detection
36. Control / Flexibility • ਂ͍Ϩϕϧͷݖݶ؅ཧͰ͖ΔACL • Piped AgentʹPlug-in ArchitectureΛ࠾༻Ͱɺ
 ಠࣗͷσϓϩΠϝϯτϩδοΫΛຒΊࠐΊΔΑ͏ʹ • Deployment ChainͰmulti-clusterͷΞϓϦέʔγϣϯ΍
 σϓϩΠͷॱ൪੍ޚ͕Ͱ͖ΔΑ͏ʹ
37. Deployment Chain Application X Region A Region B Region C ᶃ ᶄ ᶅ Application X Cluster A Cluster B Cluster C ᶃ ᶄ ᶅ Application X - Dev Env Application X - Stg Env Application X - Prod Env Application Infra Application X Application Y ੒ޭͳΒ͹࣍ʹਐΉ ੒ޭͳΒ͹࣍ʹਐΉ
38. Feature Status ۩ମతͳػೳͷঢ়ଶ͸ҎԼͷϖʔδͰ֬ೝͰ͖·͢ https://pipecd.dev/docs/feature-status
39. Thank you! PipeCD OSSͷBackendͱFrontendͷϑϧλΠϜɾ෭ۀʢ࣌ؒͷ੍ݶ͸ͳ͠ʣΛืू͍ͯ͠·͢
 ͝ڵຯ͕͋ΔํɺTwitterͷDMͳͲ͝࿈བྷ͍ͩ͘͞