Save 37% off PRO during our Black Friday Sale! »

Why and how we build a unified CD system

439d941a6e41920af9bf4d5bd5f97e3a?s=47 nghialv
October 08, 2021

Why and how we build a unified CD system

439d941a6e41920af9bf4d5bd5f97e3a?s=128

nghialv

October 08, 2021
Tweet

Transcript

  1. nghialv Developer Productivity Team, CyberAgent Oct 08, 2021 ౷ҰͳCDγεςϜΛߏஙͨ͠࿩ ๻ͨͪࢲͨͪͷCI/CD͸͜Εͩʂ-

    cndjp #17
  2. ࣗݾ঺հ @nghialv @nghialv2607 @nghialv Le Van Nghia - ΪΞ

  3. ࣗݾ঺հ - ৬ྺ @CyberAgent • PipeCDΛ։ൃɾӡ༻ - DPࣨ • Work

    fl ow Automation SystemΛ։ൃɾӡ༻ - OSSS • Feature Flags SystemΛ։ൃɾӡ༻ - AbemaTV • Prometheus & GrafanaͰMonitoring SystemΛߏஙɾӡ༻ - AbemaTV • DeploymentπʔϧΛ։ൃɾӡ༻ - AbemaTV • Microservicesɾج൫पΓ - AbemaTV
  4. ࠓ೔ͷ࿩͢಺༰ • CyberAgentͰCDʹؔ͢Δ՝୊ • ౷ҰͳσϦόϦʔج൫Λݕ౼ • PipeCDͷ஀ੜ • ݱࡏͷPipeCD @

    CyberAgent • PipeCDͷࠓޙ
  5. CyberAgentͰCDʹؔ͢Δ՝୊

  6. ϓϩμΫτͷߏ੒ ABEMA AWA WinTicket CyberZ AI Studio ... • ଟ͘ͷϓϩμΫτ͕ଘࡏ͠ɺ૿Ճத

    • ֤ϓϩμΫτͰ͸ࣗ෼ʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબ΂Δ Group • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾ಺Ҡಈ͸Մೳ ʢ਺ेݸ͕͋Δʣ
  7. ϓϩμΫτͷߏ੒ ABEMA AWA WinTicket CyberZ AI Studio ... • ֤ϓϩμΫτͰ͸ࣗ෼ʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબ΂Δ

    Group • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾ಺Ҡಈ͸Մೳ ʢ਺ेݸ͕͋Δʣ CD΋ϓϩμΫτΤϯδχΞʹࣗ༝ʹ೚ͤΔͷͰνʔϜ͝ͱʹঢ়گ͕ҟͳΔ • ଟ͘ͷϓϩμΫτ͕ଘࡏ͠ɺ૿Ճத
  8. ݩʑͷCDͷঢ়گ શମత͸ͳΜͰ΋͋Δ͜ͱʹͳͬͯ͠·ͬͨ खಈ Jenkins CircleCI TravisCI DroneCI Harness Concource ArgoCD

    GH Actions Spinnaker Flux Code Deploy Cloud Build Cloud Deploy ୲౰ऀ͕खಈͰ΍Δ CIͰCDΛ΍Δ (CI Ops) ઐ༻CDΛ࢖͏ ࣗ࡞ ChatOps
  9. CDʹؔ͢Δ՝୊ Multi Cloud Product • σϦόϦͷ੒ख़౓͕ߴ͘ͳ͍ͱ͜Ζ͕ଘࡏ • खಈϦϦʔεͷνΣοΫϦετ͕େม • Canary,

    BlueGreenͳͲͷ҆શରࡦ͕ͳ͍ • ೚ͤΔͨΊɺࣗ෼ͰCDͷߏஙͱӡ༻͕େม • ౷ҰੑʢҰ؏ੑʣ͕௿͘ͳ͍ͬͯΔ • ࣾ಺ελϯμʔυΛ੒ཱ͢Δ͜ͱ͕೉͘͠ͳΔ • ࣾ಺ͷϕετϓϥΫςΟεͷීٴ͕େม • શମతͳӡ༻ίετ͕ߴ͍ • Developer ExperienceʹӨڹͯ͠͠·͏ • ࣾ಺ҠಈͰ΋Onboardingίετ͕ൃੜ ٕज़ελοΫ͕ಉ͡Ͱ΋
 ϓϩμΫτຖʹCDγεςϜɾϓϩηε͕ҟͳΔ ಉ͡ϓϩμΫτͰ΋
 ෳ਺CDγεςϜɾϓϩηε͕ଘࡏ͢Δ Product A Product B CD System X CD System Y CD System X for GCP services CD System Y for AWS Services CD System Z for Infra
  10. ౷ҰͳσϦόϦʔج൫Λݕ౼

  11. CIͱCDΛ׬શతʹ෼཭ TestͰૣظతʹ։ൃऀ΁ϑΟʔυόοΫΛఏڙ ArtifactΛHost Environment΁σϓϩΠɾϩʔϧόοΫ BuildͰImmutable ArtifactΛੜ੒͢Δ Host Environment (ClusterͳͲʣ΁ܨ͕Βͳ͍ Host

    Environment (ClusterͳͲʣ΁௚઀తʹܨ͕Δ ੜ੒͞ΕͨArtifactʹؔ͢ΔϑΟʔυόοΫΛૣΊʹఏڙ
  12. ౷ҰͳCDγεςϜ Delivery Infrastructure Product Team͸࠷େͳσϦόϦύϑΥʔϚϯεΛ ग़ͤͳ͕Βɺࣗ෼ʹ߹͏ٕज़ελοΫͱσϦόϦ ઓུΛબͿϑϦʔμϜ͕͋Δ Platform Team͸ॊೈͳDelivery InfrastructureΛ


    ఏڙͱCDͷϕετϓϥΫςΟεΛීٴ Platform Team & Product Teamͷ྆ํʹϝϦοτΛग़ͤΔ Standard/Consistency vs Freedomͷྑ͍όϥϯεΛऔΕΔ Ͳ͏͢Ε͹ɺ
  13. Platform Team & Product Team Management Security γεςϜͷӡ༻͸ίετ͕ߴ͍͔Ͳ͏͔
 ϓϩμΫτಋೖͷεέʔϥϏϦςΟ ϓϩμΫτͷΫϨσϯγϟϧΛͲ͏؅ཧ͞ΕΔ͔


    
 CDηΩϡϦςΟʔपΓͷϕετϓϥΫςΟεΛϓϩμΫτνʔϜ΁ීٴͰ͖Δ͔Ͳ͏͔ Automation Visibility / Accessibility Control / 
 Flexibility νʔϜʹϑΟοτٕज़ελοΫɾσϦόϦख๏Λࣗ༝ʹબ͹ΕΔͷ͔
 
 νʔϜͷݖݶΛద੾ʹ؅ཧͰ͖Δͷ͔ ϦϦʔεϓϩηεͷதʹɺਓؒͷखಈλεΫ͕Ͳ͜·Ͱ࡟ݮͰ͖Δͷ͔
 
 ϦϦʔεத΋ޙ΋ܧଓతʹ໰୊Λݕ஌Ͱ͖Δͷ͔ ։ൃऀ΁े෼ͳϑΟʔυόοΫΛ଎ΊʹఏڙͰ͖Δ͔
 ໰୊͕͋Δ࣌ʹݪҼ΋ؚΉঢ়ଶΛͪΌΜͱݟ͑Δͷ͔
 ؅ཧऀ΁νʔϜͷσϦόϦʔύϑΥʔϚϯεΛͪΌΜͱݟͤΔͷ͔ Platform
 Team Product
 Team
  14. PipeCDͷ஀ੜ

  15. PipeCD

  16. PipeCD 🤍 OSS Thanks to the contributors of PipeCD! https://github.com/pipe-cd/pipe

    https://pipecd.dev https://pipecd.dev/docs 27 Contributors 4 Full-time contributors 1 Part-time contributor
 1800 Pull requests ʢ෭ۀʣ
  17. PipeCD 🤍 GitOps • GitOpsΛ࠾༻ • GitʹશͯͷCon fi gurationΛ؅ཧ •

    શͯͷΦϖϨʔγϣϯ͕Git Pull Requestܦ༝Λߦ͑Δ • Πϯϑϥ͔ΒΞϓϦέʔγϣϯ·Ͱ౷ҰͳGitOps • Kubernetes • Terraform • CloudRun • AWS Lambda • ECS, Fargate • ...
  18. PipeCD͸ͲͷΑ͏ʹզʑͷ՝୊Λղܾ͢Δ͔ Automation Visibility / Accessibility Control / 
 Flexibility Management

    Security Platform Team Product Team
  19. Platform Team - Management • ӡ༻ίετ͕௿͍ • શͯͷσϓϩΠϝϯτ͸ҰͭͷγεςϜͷΈ͕ඞཁ • PipeCDͷશͯͷίϯϙʔωϯτ͕εςʔτϨε

    • ετϨʔδ͸ϚωʔδυαʔϏεΛར༻͢Δ͜ͱ͕Մೳ • ϓϩμΫτଆ͸γϯάϧόΠφϦͷPiped agentΛΠϯετʔϧͷΈ • ϓϩμΫτͷεέʔϥϏϦςΟ • ৽نͷϓϩμΫτͷ௥Ճ͕୯౬ • Ͳͷ؀ڥͰ΋ಈ͚Δઃܭ • ωοτϫʔΫ੍ݶͷPrivate Cloud͔ΒPublic Cloud·Ͱ • খن໛νʔϜ͔Βେن໛νʔϜ·Ͱ Product B Piped Control Plane Firewall
 Friendly Outbout
 Requests
 Only Product A Piped Piped Piped
  20. Platform Team - Security • ϓϩμΫτνʔϜͷΫϨσϯγϟϧ͸֎ʹҰ੾ग़ͳ͍ઃܭ • Control-planeʹ΋อଘɾ؅ཧ͠ͳ͍ • GitOpsͰͷSecret؅ཧͷϕετϓϥΫςΟεͱͯ͠ͷbuilt-inػೳΛؚΉ

    • Piped agent͕Ξ΢τό΢ϯυϦΫΤετͷΈΛߦ͏ͷͰɺެ։ϙʔτͳͲ͸ඞཁ͕ͳ͍
  21. Product Team - Visibility / Accessibility • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ

    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε https://pipecd.dev/docs/user-guide/triggering-a-deployment
  22. Product Team - Visibility / Accessibility https://pipecd.dev/docs/user-guide/application-live-state • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ •

    ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε
  23. Product Team - Visibility / Accessibility https://pipecd.dev/docs/user-guide/plan-preview • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ •

    ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε Git di ff is enought? No. e.g. using remote Helm chart, Kustomize package, Terraform module... Reviewer needs more early feedback to merge PR with con fi dence Better to see dry-run result, terraform plan, deployment policy... on the PR then Git di ff is just showing the change of version number
  24. Product Team - Visibility / Accessibility Terraform Deployment CloudRun Deployment

    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε Scale In Scale Out Rollout New Image Update 
 Con fi g Deploy Infra Deploy Serverless Rollback Deploy Kubernetes Constant process for all operations
  25. Product Team - Automation https://pipecd.dev/docs/user-guide/rolling-back-a-deployment • σϓϩΠͷΠϯύΫτΛࣗಈ෼ੳ • ϝτϦΫεɾϩάɾhttp •

    աڈͷσʔλɾcanary-baselineͷൺֱ • σϓϩΠதʹ໰୊͕ൃੜ͢ΔͱࣗಈϩʔϧόοΫ • Gitͱ࣮ࡍͷঢ়ଶͷCon fi guration DriftΛࣗಈݕ஌ • ৽ίϯςφΠϝʔδɾHelm Chart͕ग़Δͱ
 ࣗಈσϓϩΠͷEventWatcher https://pipecd.dev/docs/user-guide/con fi guration-drift-detection
  26. Product Team - Control / Flexibility • Piped AgentΛ޷͖ͳελΠϧͰ૊Έ߹ΘͤΔ͜ͱ͕Մೳ •

    1 Piped AgentͰશͯ؅ཧύλʔϯ • ؀ڥຖʹઐ༻ͷPiped Agentύλʔϯ • ޷͖ͳ৔ॴʹΠϯετʔϧʢKubernetes podɺVMʹதɺFargateαʔϏε...ʣ • σϓϩΠઓུ͸ࣗ༝ʹ૊Έ߹Θͤɾఆ͕ٛՄೳ • Quick Sync • Progressive Sync (Canary, BlueGreen...) • νʔϜʹ߹͏ϓϩόΠμʔΛબ΂ΒΕΔ • Cloud Provider (GCP, AWS, Azure, Private Cloud) • Analysis Provider (Prometheus, Datadog, Stackdriver...) https://github.com/pipe-cd/examples/
 blob/master/kubernetes/canary/.pipe.yaml
  27. PipeCD Platform Team͕Delivery InfrastructureΛఏڙͰ͖ɺ
 ϕετϓϥΫςΟεΛܧଓతʹ࠾༻ɾීٴͰ͖Δ Product Team͕CDͷߏஙɾӡ༻͕ෆཁʹͳΓɺҰճಋೖ͢Δ͚ͩͰɺ
 ࣗ෼ʹ߹͏σϦόϦख๏Λ࠾༻Ͱ͖ɺܧଓతʹվળͰ͖Δ ͜ΕͰɺ

  28. ݱࡏͷPipeCD @ CyberAgent

  29. νʔϜͱγεςϜͷߏ੒ • Platform Team • PipeCDΛ։ൃ • ࣾ಺༻PipeCD Control-planeΛӡ༻ •

    ֤Product Team • Piped agentΛΠϯετʔϧ
 • ࣗ෼ʹ߹͏σϓϩΠϝϯτछྨΛ࠾༻ Control-Plane https://pipecd.dev/docs/operator-manual/piped/installation https://github.com/pipe-cd/examples
  30. ΞϓϦέʔγϣϯɾαʔϏε͕૿Ճத 0 225 450 675 900 2020/10 2020/12 2021/02 2021/04

    2021/06 2021/8 806 Applications/Servicesʹୡ੒ CyberAgentʹPipeCDͰӡ༻͍ͯ͠Δ Kubernetes, Terraform, Lambda, CloudRun, Fargate... 
 ͷΞϓϦέʔγϣϯɾαʔϏε਺
  31. PipeCDͷࠓޙ

  32. ֤࣠Λ͞ΒʹڧԽ Automation Visibility / Accessibility Control / 
 Flexibility Management

    Security
  33. • ࠓ·ͰͷPiped agentͷӡ༻ • Product Team͕Piped agentΛΠϯετʔϧ • ৽͍͠όʔδϣϯ͕͋Δͱ࠶Πϯετʔϧ •

    ͜Ε͔Β • ҰճͷΈΠϯετʔϧ͢Δ • Web consoleͰόʔδϣϯΛΞοϓͰ͖Δ • ͦΕͰӡ༻͕͞ΒʹָʹͳΔ Management - Remote Upgrade ͜ͷػೳ͸དྷिʹϦϦʔε༧ఆ Product B Piped Control Plane Firewall
 Friendly Outbout
 Requests
 Only Product A Piped Piped Piped
  34. Visibility / Accessiblity • InsightsͰσϦόϦʔύϑΥʔϚϯεΛՄࢹԽ • Deployස౓ɺLead Time, ࣦഊ཰, MTTR...

    • ApplicationͷϦιʔεͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • Terraform, CloudRun, ECS... • Deployment StageͷϩάΛվળ • Plan-Previewʹ΋ͬͱϑΟʔυόοΫΛՃ͑Δ • Kubernetes validating webhook, dry-run݁Ռ, Terraform sentinelͳͲͷDeployment Policy ֎ͷϢʔβʔ޲͚ͷPlayground؀ڥΛ४උதʂ
 https://play.pipecd.dev
  35. Automation • LogͷσʔλͰσϓϩΠϝϯτΛࣗಈ෼ੳ • աڈͷMetricsσʔλͱͷൺֱͰࣗಈ෼ੳ • Primary/Baseline & CanaryͷMetricsσʔλͷൺֱͰࣗಈ෼ੳ •

    Terraform, CloudRun, LambdaͳͲͷࣗಈDrift Detection
  36. Control / Flexibility • ਂ͍Ϩϕϧͷݖݶ؅ཧͰ͖ΔACL • Piped AgentʹPlug-in ArchitectureΛ࠾༻Ͱɺ
 ಠࣗͷσϓϩΠϝϯτϩδοΫΛຒΊࠐΊΔΑ͏ʹ

    • Deployment ChainͰmulti-clusterͷΞϓϦέʔγϣϯ΍
 σϓϩΠͷॱ൪੍ޚ͕Ͱ͖ΔΑ͏ʹ
  37. Deployment Chain Application X Region A Region B Region C

    ᶃ ᶄ ᶅ Application X Cluster A Cluster B Cluster C ᶃ ᶄ ᶅ Application X - Dev Env Application X - Stg Env Application X - Prod Env Application Infra Application X Application Y ੒ޭͳΒ͹࣍ʹਐΉ ੒ޭͳΒ͹࣍ʹਐΉ
  38. Feature Status ۩ମతͳػೳͷঢ়ଶ͸ҎԼͷϖʔδͰ֬ೝͰ͖·͢ https://pipecd.dev/docs/feature-status

  39. Thank you! PipeCD OSSͷBackendͱFrontendͷϑϧλΠϜɾ෭ۀʢ࣌ؒͷ੍ݶ͸ͳ͠ʣΛืू͍ͯ͠·͢
 ͝ڵຯ͕͋ΔํɺTwitterͷDMͳͲ͝࿈བྷ͍ͩ͘͞