$30 off During Our Annual Pro Sale. View Details »

Why and how we build a unified CD system

nghialv
October 08, 2021

Why and how we build a unified CD system

nghialv

October 08, 2021
Tweet

More Decks by nghialv

Other Decks in Technology

Transcript

  1. ࣗݾ঺հ - ৬ྺ @CyberAgent • PipeCDΛ։ൃɾӡ༻ - DPࣨ • Work

    fl ow Automation SystemΛ։ൃɾӡ༻ - OSSS • Feature Flags SystemΛ։ൃɾӡ༻ - AbemaTV • Prometheus & GrafanaͰMonitoring SystemΛߏஙɾӡ༻ - AbemaTV • DeploymentπʔϧΛ։ൃɾӡ༻ - AbemaTV • Microservicesɾج൫पΓ - AbemaTV
  2. ϓϩμΫτͷߏ੒ ABEMA AWA WinTicket CyberZ AI Studio ... • ଟ͘ͷϓϩμΫτ͕ଘࡏ͠ɺ૿Ճத

    • ֤ϓϩμΫτͰ͸ࣗ෼ʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબ΂Δ Group • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾ಺Ҡಈ͸Մೳ ʢ਺ेݸ͕͋Δʣ
  3. ϓϩμΫτͷߏ੒ ABEMA AWA WinTicket CyberZ AI Studio ... • ֤ϓϩμΫτͰ͸ࣗ෼ʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબ΂Δ

    Group • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾ಺Ҡಈ͸Մೳ ʢ਺ेݸ͕͋Δʣ CD΋ϓϩμΫτΤϯδχΞʹࣗ༝ʹ೚ͤΔͷͰνʔϜ͝ͱʹঢ়گ͕ҟͳΔ • ଟ͘ͷϓϩμΫτ͕ଘࡏ͠ɺ૿Ճத
  4. ݩʑͷCDͷঢ়گ શମత͸ͳΜͰ΋͋Δ͜ͱʹͳͬͯ͠·ͬͨ खಈ Jenkins CircleCI TravisCI DroneCI Harness Concource ArgoCD

    GH Actions Spinnaker Flux Code Deploy Cloud Build Cloud Deploy ୲౰ऀ͕खಈͰ΍Δ CIͰCDΛ΍Δ (CI Ops) ઐ༻CDΛ࢖͏ ࣗ࡞ ChatOps
  5. CDʹؔ͢Δ՝୊ Multi Cloud Product • σϦόϦͷ੒ख़౓͕ߴ͘ͳ͍ͱ͜Ζ͕ଘࡏ • खಈϦϦʔεͷνΣοΫϦετ͕େม • Canary,

    BlueGreenͳͲͷ҆શରࡦ͕ͳ͍ • ೚ͤΔͨΊɺࣗ෼ͰCDͷߏஙͱӡ༻͕େม • ౷ҰੑʢҰ؏ੑʣ͕௿͘ͳ͍ͬͯΔ • ࣾ಺ελϯμʔυΛ੒ཱ͢Δ͜ͱ͕೉͘͠ͳΔ • ࣾ಺ͷϕετϓϥΫςΟεͷීٴ͕େม • શମతͳӡ༻ίετ͕ߴ͍ • Developer ExperienceʹӨڹͯ͠͠·͏ • ࣾ಺ҠಈͰ΋Onboardingίετ͕ൃੜ ٕज़ελοΫ͕ಉ͡Ͱ΋
 ϓϩμΫτຖʹCDγεςϜɾϓϩηε͕ҟͳΔ ಉ͡ϓϩμΫτͰ΋
 ෳ਺CDγεςϜɾϓϩηε͕ଘࡏ͢Δ Product A Product B CD System X CD System Y CD System X for GCP services CD System Y for AWS Services CD System Z for Infra
  6. ౷ҰͳCDγεςϜ Delivery Infrastructure Product Team͸࠷େͳσϦόϦύϑΥʔϚϯεΛ ग़ͤͳ͕Βɺࣗ෼ʹ߹͏ٕज़ελοΫͱσϦόϦ ઓུΛબͿϑϦʔμϜ͕͋Δ Platform Team͸ॊೈͳDelivery InfrastructureΛ


    ఏڙͱCDͷϕετϓϥΫςΟεΛීٴ Platform Team & Product Teamͷ྆ํʹϝϦοτΛग़ͤΔ Standard/Consistency vs Freedomͷྑ͍όϥϯεΛऔΕΔ Ͳ͏͢Ε͹ɺ
  7. Platform Team & Product Team Management Security γεςϜͷӡ༻͸ίετ͕ߴ͍͔Ͳ͏͔
 ϓϩμΫτಋೖͷεέʔϥϏϦςΟ ϓϩμΫτͷΫϨσϯγϟϧΛͲ͏؅ཧ͞ΕΔ͔


    
 CDηΩϡϦςΟʔपΓͷϕετϓϥΫςΟεΛϓϩμΫτνʔϜ΁ීٴͰ͖Δ͔Ͳ͏͔ Automation Visibility / Accessibility Control / 
 Flexibility νʔϜʹϑΟοτٕज़ελοΫɾσϦόϦख๏Λࣗ༝ʹબ͹ΕΔͷ͔
 
 νʔϜͷݖݶΛద੾ʹ؅ཧͰ͖Δͷ͔ ϦϦʔεϓϩηεͷதʹɺਓؒͷखಈλεΫ͕Ͳ͜·Ͱ࡟ݮͰ͖Δͷ͔
 
 ϦϦʔεத΋ޙ΋ܧଓతʹ໰୊Λݕ஌Ͱ͖Δͷ͔ ։ൃऀ΁े෼ͳϑΟʔυόοΫΛ଎ΊʹఏڙͰ͖Δ͔
 ໰୊͕͋Δ࣌ʹݪҼ΋ؚΉঢ়ଶΛͪΌΜͱݟ͑Δͷ͔
 ؅ཧऀ΁νʔϜͷσϦόϦʔύϑΥʔϚϯεΛͪΌΜͱݟͤΔͷ͔ Platform
 Team Product
 Team
  8. PipeCD 🤍 OSS Thanks to the contributors of PipeCD! https://github.com/pipe-cd/pipe

    https://pipecd.dev https://pipecd.dev/docs 27 Contributors 4 Full-time contributors 1 Part-time contributor
 1800 Pull requests ʢ෭ۀʣ
  9. PipeCD 🤍 GitOps • GitOpsΛ࠾༻ • GitʹશͯͷCon fi gurationΛ؅ཧ •

    શͯͷΦϖϨʔγϣϯ͕Git Pull Requestܦ༝Λߦ͑Δ • Πϯϑϥ͔ΒΞϓϦέʔγϣϯ·Ͱ౷ҰͳGitOps • Kubernetes • Terraform • CloudRun • AWS Lambda • ECS, Fargate • ...
  10. Platform Team - Management • ӡ༻ίετ͕௿͍ • શͯͷσϓϩΠϝϯτ͸ҰͭͷγεςϜͷΈ͕ඞཁ • PipeCDͷશͯͷίϯϙʔωϯτ͕εςʔτϨε

    • ετϨʔδ͸ϚωʔδυαʔϏεΛར༻͢Δ͜ͱ͕Մೳ • ϓϩμΫτଆ͸γϯάϧόΠφϦͷPiped agentΛΠϯετʔϧͷΈ • ϓϩμΫτͷεέʔϥϏϦςΟ • ৽نͷϓϩμΫτͷ௥Ճ͕୯౬ • Ͳͷ؀ڥͰ΋ಈ͚Δઃܭ • ωοτϫʔΫ੍ݶͷPrivate Cloud͔ΒPublic Cloud·Ͱ • খن໛νʔϜ͔Βେن໛νʔϜ·Ͱ Product B Piped Control Plane Firewall
 Friendly Outbout
 Requests
 Only Product A Piped Piped Piped
  11. Product Team - Visibility / Accessibility • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ

    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε https://pipecd.dev/docs/user-guide/triggering-a-deployment
  12. Product Team - Visibility / Accessibility https://pipecd.dev/docs/user-guide/application-live-state • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ •

    ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε
  13. Product Team - Visibility / Accessibility https://pipecd.dev/docs/user-guide/plan-preview • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ •

    ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε Git di ff is enought? No. e.g. using remote Helm chart, Kustomize package, Terraform module... Reviewer needs more early feedback to merge PR with con fi dence Better to see dry-run result, terraform plan, deployment policy... on the PR then Git di ff is just showing the change of version number
  14. Product Team - Visibility / Accessibility Terraform Deployment CloudRun Deployment

    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε Scale In Scale Out Rollout New Image Update 
 Con fi g Deploy Infra Deploy Serverless Rollback Deploy Kubernetes Constant process for all operations
  15. Product Team - Automation https://pipecd.dev/docs/user-guide/rolling-back-a-deployment • σϓϩΠͷΠϯύΫτΛࣗಈ෼ੳ • ϝτϦΫεɾϩάɾhttp •

    աڈͷσʔλɾcanary-baselineͷൺֱ • σϓϩΠதʹ໰୊͕ൃੜ͢ΔͱࣗಈϩʔϧόοΫ • Gitͱ࣮ࡍͷঢ়ଶͷCon fi guration DriftΛࣗಈݕ஌ • ৽ίϯςφΠϝʔδɾHelm Chart͕ग़Δͱ
 ࣗಈσϓϩΠͷEventWatcher https://pipecd.dev/docs/user-guide/con fi guration-drift-detection
  16. Product Team - Control / Flexibility • Piped AgentΛ޷͖ͳελΠϧͰ૊Έ߹ΘͤΔ͜ͱ͕Մೳ •

    1 Piped AgentͰશͯ؅ཧύλʔϯ • ؀ڥຖʹઐ༻ͷPiped Agentύλʔϯ • ޷͖ͳ৔ॴʹΠϯετʔϧʢKubernetes podɺVMʹதɺFargateαʔϏε...ʣ • σϓϩΠઓུ͸ࣗ༝ʹ૊Έ߹Θͤɾఆ͕ٛՄೳ • Quick Sync • Progressive Sync (Canary, BlueGreen...) • νʔϜʹ߹͏ϓϩόΠμʔΛબ΂ΒΕΔ • Cloud Provider (GCP, AWS, Azure, Private Cloud) • Analysis Provider (Prometheus, Datadog, Stackdriver...) https://github.com/pipe-cd/examples/
 blob/master/kubernetes/canary/.pipe.yaml
  17. νʔϜͱγεςϜͷߏ੒ • Platform Team • PipeCDΛ։ൃ • ࣾ಺༻PipeCD Control-planeΛӡ༻ •

    ֤Product Team • Piped agentΛΠϯετʔϧ
 • ࣗ෼ʹ߹͏σϓϩΠϝϯτछྨΛ࠾༻ Control-Plane https://pipecd.dev/docs/operator-manual/piped/installation https://github.com/pipe-cd/examples
  18. ΞϓϦέʔγϣϯɾαʔϏε͕૿Ճத 0 225 450 675 900 2020/10 2020/12 2021/02 2021/04

    2021/06 2021/8 806 Applications/Servicesʹୡ੒ CyberAgentʹPipeCDͰӡ༻͍ͯ͠Δ Kubernetes, Terraform, Lambda, CloudRun, Fargate... 
 ͷΞϓϦέʔγϣϯɾαʔϏε਺
  19. • ࠓ·ͰͷPiped agentͷӡ༻ • Product Team͕Piped agentΛΠϯετʔϧ • ৽͍͠όʔδϣϯ͕͋Δͱ࠶Πϯετʔϧ •

    ͜Ε͔Β • ҰճͷΈΠϯετʔϧ͢Δ • Web consoleͰόʔδϣϯΛΞοϓͰ͖Δ • ͦΕͰӡ༻͕͞ΒʹָʹͳΔ Management - Remote Upgrade ͜ͷػೳ͸དྷिʹϦϦʔε༧ఆ Product B Piped Control Plane Firewall
 Friendly Outbout
 Requests
 Only Product A Piped Piped Piped
  20. Visibility / Accessiblity • InsightsͰσϦόϦʔύϑΥʔϚϯεΛՄࢹԽ • Deployස౓ɺLead Time, ࣦഊ཰, MTTR...

    • ApplicationͷϦιʔεͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • Terraform, CloudRun, ECS... • Deployment StageͷϩάΛվળ • Plan-Previewʹ΋ͬͱϑΟʔυόοΫΛՃ͑Δ • Kubernetes validating webhook, dry-run݁Ռ, Terraform sentinelͳͲͷDeployment Policy ֎ͷϢʔβʔ޲͚ͷPlayground؀ڥΛ४උதʂ
 https://play.pipecd.dev
  21. Deployment Chain Application X Region A Region B Region C

    ᶃ ᶄ ᶅ Application X Cluster A Cluster B Cluster C ᶃ ᶄ ᶅ Application X - Dev Env Application X - Stg Env Application X - Prod Env Application Infra Application X Application Y ੒ޭͳΒ͹࣍ʹਐΉ ੒ޭͳΒ͹࣍ʹਐΉ