Why and how we build a unified CD system

Slide 1

Slide 1 text

nghialv Developer Productivity Team, CyberAgent Oct 08, 2021 ౷ҰͳCDγεςϜΛߏஙͨ͠࿩ ๻ͨͪࢲͨͪͷCI/CD͸͜Εͩʂ- cndjp #17

Slide 2

Slide 2 text

ࣗݾ঺հ @nghialv @nghialv2607 @nghialv Le Van Nghia - ΪΞ

Slide 3

Slide 3 text

ࣗݾ঺հ - ৬ྺ @CyberAgent • PipeCDΛ։ൃɾӡ༻ - DPࣨ • Work fl ow Automation SystemΛ։ൃɾӡ༻ - OSSS • Feature Flags SystemΛ։ൃɾӡ༻ - AbemaTV • Prometheus & GrafanaͰMonitoring SystemΛߏஙɾӡ༻ - AbemaTV • DeploymentπʔϧΛ։ൃɾӡ༻ - AbemaTV • Microservicesɾج൫पΓ - AbemaTV

Slide 4

Slide 4 text

ࠓ೔ͷ࿩͢಺༰ • CyberAgentͰCDʹؔ͢Δ՝୊ • ౷ҰͳσϦόϦʔج൫Λݕ౼ • PipeCDͷ஀ੜ • ݱࡏͷPipeCD @ CyberAgent • PipeCDͷࠓޙ

Slide 5

Slide 5 text

CyberAgentͰCDʹؔ͢Δ՝୊

Slide 6

Slide 6 text

ϓϩμΫτͷߏ੒ ABEMA AWA WinTicket CyberZ AI Studio ... • ଟ͘ͷϓϩμΫτ͕ଘࡏ͠ɺ૿Ճத • ֤ϓϩμΫτͰ͸ࣗ෼ʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબ΂Δ Group • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾ಺Ҡಈ͸Մೳ ʢ਺ेݸ͕͋Δʣ

Slide 7

Slide 7 text

ϓϩμΫτͷߏ੒ ABEMA AWA WinTicket CyberZ AI Studio ... • ֤ϓϩμΫτͰ͸ࣗ෼ʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબ΂Δ Group • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾ಺Ҡಈ͸Մೳ ʢ਺ेݸ͕͋Δʣ CD΋ϓϩμΫτΤϯδχΞʹࣗ༝ʹ೚ͤΔͷͰνʔϜ͝ͱʹঢ়گ͕ҟͳΔ • ଟ͘ͷϓϩμΫτ͕ଘࡏ͠ɺ૿Ճத

Slide 8

Slide 8 text

ݩʑͷCDͷঢ়گ શମత͸ͳΜͰ΋͋Δ͜ͱʹͳͬͯ͠·ͬͨ खಈ Jenkins CircleCI TravisCI DroneCI Harness Concource ArgoCD GH Actions Spinnaker Flux Code Deploy Cloud Build Cloud Deploy ୲౰ऀ͕खಈͰ΍Δ CIͰCDΛ΍Δ (CI Ops) ઐ༻CDΛ࢖͏ ࣗ࡞ ChatOps

Slide 9

Slide 9 text

CDʹؔ͢Δ՝୊ Multi Cloud Product • σϦόϦͷ੒ख़౓͕ߴ͘ͳ͍ͱ͜Ζ͕ଘࡏ • खಈϦϦʔεͷνΣοΫϦετ͕େม • Canary, BlueGreenͳͲͷ҆શରࡦ͕ͳ͍ • ೚ͤΔͨΊɺࣗ෼ͰCDͷߏஙͱӡ༻͕େม • ౷ҰੑʢҰ؏ੑʣ͕௿͘ͳ͍ͬͯΔ • ࣾ಺ελϯμʔυΛ੒ཱ͢Δ͜ͱ͕೉͘͠ͳΔ • ࣾ಺ͷϕετϓϥΫςΟεͷීٴ͕େม • શମతͳӡ༻ίετ͕ߴ͍ • Developer ExperienceʹӨڹͯ͠͠·͏ • ࣾ಺ҠಈͰ΋Onboardingίετ͕ൃੜ ٕज़ελοΫ͕ಉ͡Ͱ΋  ϓϩμΫτຖʹCDγεςϜɾϓϩηε͕ҟͳΔ ಉ͡ϓϩμΫτͰ΋  ෳ਺CDγεςϜɾϓϩηε͕ଘࡏ͢Δ Product A Product B CD System X CD System Y CD System X for GCP services CD System Y for AWS Services CD System Z for Infra

Slide 10

Slide 10 text

౷ҰͳσϦόϦʔج൫Λݕ౼

Slide 11

Slide 11 text

CIͱCDΛ׬શతʹ෼཭ TestͰૣظతʹ։ൃऀ΁ϑΟʔυόοΫΛఏڙ ArtifactΛHost Environment΁σϓϩΠɾϩʔϧόοΫ BuildͰImmutable ArtifactΛੜ੒͢Δ Host Environment (ClusterͳͲʣ΁ܨ͕Βͳ͍ Host Environment (ClusterͳͲʣ΁௚઀తʹܨ͕Δ ੜ੒͞ΕͨArtifactʹؔ͢ΔϑΟʔυόοΫΛૣΊʹఏڙ

Slide 12

Slide 12 text

౷ҰͳCDγεςϜ Delivery Infrastructure Product Team͸࠷େͳσϦόϦύϑΥʔϚϯεΛ ग़ͤͳ͕Βɺࣗ෼ʹ߹͏ٕज़ελοΫͱσϦόϦ ઓུΛબͿϑϦʔμϜ͕͋Δ Platform Team͸ॊೈͳDelivery InfrastructureΛ  ఏڙͱCDͷϕετϓϥΫςΟεΛීٴ Platform Team & Product Teamͷ྆ํʹϝϦοτΛग़ͤΔ Standard/Consistency vs Freedomͷྑ͍όϥϯεΛऔΕΔ Ͳ͏͢Ε͹ɺ

Slide 13

Slide 13 text

Platform Team & Product Team Management Security γεςϜͷӡ༻͸ίετ͕ߴ͍͔Ͳ͏͔  ϓϩμΫτಋೖͷεέʔϥϏϦςΟ ϓϩμΫτͷΫϨσϯγϟϧΛͲ͏؅ཧ͞ΕΔ͔    CDηΩϡϦςΟʔपΓͷϕετϓϥΫςΟεΛϓϩμΫτνʔϜ΁ීٴͰ͖Δ͔Ͳ͏͔ Automation Visibility / Accessibility Control /   Flexibility νʔϜʹϑΟοτٕज़ελοΫɾσϦόϦख๏Λࣗ༝ʹબ͹ΕΔͷ͔    νʔϜͷݖݶΛద੾ʹ؅ཧͰ͖Δͷ͔ ϦϦʔεϓϩηεͷதʹɺਓؒͷखಈλεΫ͕Ͳ͜·Ͱ࡟ݮͰ͖Δͷ͔    ϦϦʔεத΋ޙ΋ܧଓతʹ໰୊Λݕ஌Ͱ͖Δͷ͔ ։ൃऀ΁े෼ͳϑΟʔυόοΫΛ଎ΊʹఏڙͰ͖Δ͔  ໰୊͕͋Δ࣌ʹݪҼ΋ؚΉঢ়ଶΛͪΌΜͱݟ͑Δͷ͔  ؅ཧऀ΁νʔϜͷσϦόϦʔύϑΥʔϚϯεΛͪΌΜͱݟͤΔͷ͔ Platform  Team Product  Team

Slide 14

Slide 14 text

PipeCDͷ஀ੜ

Slide 15

Slide 15 text

PipeCD

Slide 16

Slide 16 text

PipeCD 🤍 OSS Thanks to the contributors of PipeCD! https://github.com/pipe-cd/pipe https://pipecd.dev https://pipecd.dev/docs 27 Contributors 4 Full-time contributors 1 Part-time contributor  1800 Pull requests ʢ෭ۀʣ

Slide 17

Slide 17 text

PipeCD 🤍 GitOps • GitOpsΛ࠾༻ • GitʹશͯͷCon fi gurationΛ؅ཧ • શͯͷΦϖϨʔγϣϯ͕Git Pull Requestܦ༝Λߦ͑Δ • Πϯϑϥ͔ΒΞϓϦέʔγϣϯ·Ͱ౷ҰͳGitOps • Kubernetes • Terraform • CloudRun • AWS Lambda • ECS, Fargate • ...

Slide 18

Slide 18 text

PipeCD͸ͲͷΑ͏ʹզʑͷ՝୊Λղܾ͢Δ͔ Automation Visibility / Accessibility Control /   Flexibility Management Security Platform Team Product Team

Slide 19

Slide 19 text

Platform Team - Management • ӡ༻ίετ͕௿͍ • શͯͷσϓϩΠϝϯτ͸ҰͭͷγεςϜͷΈ͕ඞཁ • PipeCDͷશͯͷίϯϙʔωϯτ͕εςʔτϨε • ετϨʔδ͸ϚωʔδυαʔϏεΛར༻͢Δ͜ͱ͕Մೳ • ϓϩμΫτଆ͸γϯάϧόΠφϦͷPiped agentΛΠϯετʔϧͷΈ • ϓϩμΫτͷεέʔϥϏϦςΟ • ৽نͷϓϩμΫτͷ௥Ճ͕୯౬ • Ͳͷ؀ڥͰ΋ಈ͚Δઃܭ • ωοτϫʔΫ੍ݶͷPrivate Cloud͔ΒPublic Cloud·Ͱ • খن໛νʔϜ͔Βେن໛νʔϜ·Ͱ Product B Piped Control Plane Firewall  Friendly Outbout  Requests  Only Product A Piped Piped Piped

Slide 20

Slide 20 text

Platform Team - Security • ϓϩμΫτνʔϜͷΫϨσϯγϟϧ͸֎ʹҰ੾ग़ͳ͍ઃܭ • Control-planeʹ΋อଘɾ؅ཧ͠ͳ͍ • GitOpsͰͷSecret؅ཧͷϕετϓϥΫςΟεͱͯ͠ͷbuilt-inػೳΛؚΉ • Piped agent͕Ξ΢τό΢ϯυϦΫΤετͷΈΛߦ͏ͷͰɺެ։ϙʔτͳͲ͸ඞཁ͕ͳ͍

Slide 21

Slide 21 text

Product Team - Visibility / Accessibility • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε https://pipecd.dev/docs/user-guide/triggering-a-deployment

Slide 22

Slide 22 text

Product Team - Visibility / Accessibility https://pipecd.dev/docs/user-guide/application-live-state • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε

Slide 23

Slide 23 text

Product Team - Visibility / Accessibility https://pipecd.dev/docs/user-guide/plan-preview • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε Git di ff is enought? No. e.g. using remote Helm chart, Kustomize package, Terraform module... Reviewer needs more early feedback to merge PR with con fi dence Better to see dry-run result, terraform plan, deployment policy... on the PR then Git di ff is just showing the change of version number

Slide 24

Slide 24 text

Product Team - Visibility / Accessibility Terraform Deployment CloudRun Deployment • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε Scale In Scale Out Rollout New Image Update   Con fi g Deploy Infra Deploy Serverless Rollback Deploy Kubernetes Constant process for all operations

Slide 25

Slide 25 text

Product Team - Automation https://pipecd.dev/docs/user-guide/rolling-back-a-deployment • σϓϩΠͷΠϯύΫτΛࣗಈ෼ੳ • ϝτϦΫεɾϩάɾhttp • աڈͷσʔλɾcanary-baselineͷൺֱ • σϓϩΠதʹ໰୊͕ൃੜ͢ΔͱࣗಈϩʔϧόοΫ • Gitͱ࣮ࡍͷঢ়ଶͷCon fi guration DriftΛࣗಈݕ஌ • ৽ίϯςφΠϝʔδɾHelm Chart͕ग़Δͱ  ࣗಈσϓϩΠͷEventWatcher https://pipecd.dev/docs/user-guide/con fi guration-drift-detection

Slide 26

Slide 26 text

Product Team - Control / Flexibility • Piped AgentΛ޷͖ͳελΠϧͰ૊Έ߹ΘͤΔ͜ͱ͕Մೳ • 1 Piped AgentͰશͯ؅ཧύλʔϯ • ؀ڥຖʹઐ༻ͷPiped Agentύλʔϯ • ޷͖ͳ৔ॴʹΠϯετʔϧʢKubernetes podɺVMʹதɺFargateαʔϏε...ʣ • σϓϩΠઓུ͸ࣗ༝ʹ૊Έ߹Θͤɾఆ͕ٛՄೳ • Quick Sync • Progressive Sync (Canary, BlueGreen...) • νʔϜʹ߹͏ϓϩόΠμʔΛબ΂ΒΕΔ • Cloud Provider (GCP, AWS, Azure, Private Cloud) • Analysis Provider (Prometheus, Datadog, Stackdriver...) https://github.com/pipe-cd/examples/  blob/master/kubernetes/canary/.pipe.yaml

Slide 27

Slide 27 text

PipeCD Platform Team͕Delivery InfrastructureΛఏڙͰ͖ɺ  ϕετϓϥΫςΟεΛܧଓతʹ࠾༻ɾීٴͰ͖Δ Product Team͕CDͷߏஙɾӡ༻͕ෆཁʹͳΓɺҰճಋೖ͢Δ͚ͩͰɺ  ࣗ෼ʹ߹͏σϦόϦख๏Λ࠾༻Ͱ͖ɺܧଓతʹվળͰ͖Δ ͜ΕͰɺ

Slide 28

Slide 28 text

ݱࡏͷPipeCD @ CyberAgent

Slide 29

Slide 29 text

νʔϜͱγεςϜͷߏ੒ • Platform Team • PipeCDΛ։ൃ • ࣾ಺༻PipeCD Control-planeΛӡ༻ • ֤Product Team • Piped agentΛΠϯετʔϧ  • ࣗ෼ʹ߹͏σϓϩΠϝϯτछྨΛ࠾༻ Control-Plane https://pipecd.dev/docs/operator-manual/piped/installation https://github.com/pipe-cd/examples

Slide 30

Slide 30 text

ΞϓϦέʔγϣϯɾαʔϏε͕૿Ճத 0 225 450 675 900 2020/10 2020/12 2021/02 2021/04 2021/06 2021/8 806 Applications/Servicesʹୡ੒ CyberAgentʹPipeCDͰӡ༻͍ͯ͠Δ Kubernetes, Terraform, Lambda, CloudRun, Fargate...   ͷΞϓϦέʔγϣϯɾαʔϏε਺

Slide 31

Slide 31 text

PipeCDͷࠓޙ

Slide 32

Slide 32 text

֤࣠Λ͞ΒʹڧԽ Automation Visibility / Accessibility Control /   Flexibility Management Security

Slide 33

Slide 33 text

• ࠓ·ͰͷPiped agentͷӡ༻ • Product Team͕Piped agentΛΠϯετʔϧ • ৽͍͠όʔδϣϯ͕͋Δͱ࠶Πϯετʔϧ • ͜Ε͔Β • ҰճͷΈΠϯετʔϧ͢Δ • Web consoleͰόʔδϣϯΛΞοϓͰ͖Δ • ͦΕͰӡ༻͕͞ΒʹָʹͳΔ Management - Remote Upgrade ͜ͷػೳ͸དྷिʹϦϦʔε༧ఆ Product B Piped Control Plane Firewall  Friendly Outbout  Requests  Only Product A Piped Piped Piped

Slide 34

Slide 34 text

Visibility / Accessiblity • InsightsͰσϦόϦʔύϑΥʔϚϯεΛՄࢹԽ • Deployස౓ɺLead Time, ࣦഊ཰, MTTR... • ApplicationͷϦιʔεͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • Terraform, CloudRun, ECS... • Deployment StageͷϩάΛվળ • Plan-Previewʹ΋ͬͱϑΟʔυόοΫΛՃ͑Δ • Kubernetes validating webhook, dry-run݁Ռ, Terraform sentinelͳͲͷDeployment Policy ֎ͷϢʔβʔ޲͚ͷPlayground؀ڥΛ४උதʂ  https://play.pipecd.dev

Slide 35

Slide 35 text

Automation • LogͷσʔλͰσϓϩΠϝϯτΛࣗಈ෼ੳ • աڈͷMetricsσʔλͱͷൺֱͰࣗಈ෼ੳ • Primary/Baseline & CanaryͷMetricsσʔλͷൺֱͰࣗಈ෼ੳ • Terraform, CloudRun, LambdaͳͲͷࣗಈDrift Detection

Slide 36

Slide 36 text

Control / Flexibility • ਂ͍Ϩϕϧͷݖݶ؅ཧͰ͖ΔACL • Piped AgentʹPlug-in ArchitectureΛ࠾༻Ͱɺ  ಠࣗͷσϓϩΠϝϯτϩδοΫΛຒΊࠐΊΔΑ͏ʹ • Deployment ChainͰmulti-clusterͷΞϓϦέʔγϣϯ΍  σϓϩΠͷॱ൪੍ޚ͕Ͱ͖ΔΑ͏ʹ

Slide 37

Slide 37 text

Deployment Chain Application X Region A Region B Region C ᶃ ᶄ ᶅ Application X Cluster A Cluster B Cluster C ᶃ ᶄ ᶅ Application X - Dev Env Application X - Stg Env Application X - Prod Env Application Infra Application X Application Y ੒ޭͳΒ͹࣍ʹਐΉ ੒ޭͳΒ͹࣍ʹਐΉ

Slide 38

Slide 38 text

Feature Status ۩ମతͳػೳͷঢ়ଶ͸ҎԼͷϖʔδͰ֬ೝͰ͖·͢ https://pipecd.dev/docs/feature-status

Slide 39

Slide 39 text

Thank you! PipeCD OSSͷBackendͱFrontendͷϑϧλΠϜɾ෭ۀʢ࣌ؒͷ੍ݶ͸ͳ͠ʣΛืू͍ͯ͠·͢  ͝ڵຯ͕͋ΔํɺTwitterͷDMͳͲ͝࿈བྷ͍ͩ͘͞