Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Why and how we build a unified CD system

nghialv
October 08, 2021

Why and how we build a unified CD system

nghialv

October 08, 2021
Tweet

More Decks by nghialv

Other Decks in Technology

Transcript

  1. nghialv
    Developer Productivity Team, CyberAgent
    Oct 08, 2021
    ౷ҰͳCDγεςϜΛߏஙͨ͠࿩
    ๻ͨͪࢲͨͪͷCI/CD͸͜Εͩʂ- cndjp #17

    View full-size slide

  2. ࣗݾ঺հ
    @nghialv
    @nghialv2607
    @nghialv
    Le Van Nghia - ΪΞ

    View full-size slide

  3. ࣗݾ঺հ - ৬ྺ @CyberAgent
    • PipeCDΛ։ൃɾӡ༻ - DPࣨ

    • Work
    fl
    ow Automation SystemΛ։ൃɾӡ༻ - OSSS

    • Feature Flags SystemΛ։ൃɾӡ༻ - AbemaTV

    • Prometheus & GrafanaͰMonitoring SystemΛߏஙɾӡ༻ - AbemaTV

    • DeploymentπʔϧΛ։ൃɾӡ༻ - AbemaTV

    • Microservicesɾج൫पΓ - AbemaTV

    View full-size slide

  4. ࠓ೔ͷ࿩͢಺༰
    • CyberAgentͰCDʹؔ͢Δ՝୊

    • ౷ҰͳσϦόϦʔج൫Λݕ౼

    • PipeCDͷ஀ੜ

    • ݱࡏͷPipeCD @ CyberAgent

    • PipeCDͷࠓޙ

    View full-size slide

  5. CyberAgentͰCDʹؔ͢Δ՝୊

    View full-size slide

  6. ϓϩμΫτͷߏ੒
    ABEMA AWA WinTicket CyberZ AI Studio
    ...
    • ଟ͘ͷϓϩμΫτ͕ଘࡏ͠ɺ૿Ճத
    • ֤ϓϩμΫτͰ͸ࣗ෼ʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબ΂Δ
    Group
    • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾ಺Ҡಈ͸Մೳ
    ʢ਺ेݸ͕͋Δʣ

    View full-size slide

  7. ϓϩμΫτͷߏ੒
    ABEMA AWA WinTicket CyberZ AI Studio
    ...
    • ֤ϓϩμΫτͰ͸ࣗ෼ʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબ΂Δ
    Group
    • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾ಺Ҡಈ͸Մೳ
    ʢ਺ेݸ͕͋Δʣ
    CD΋ϓϩμΫτΤϯδχΞʹࣗ༝ʹ೚ͤΔͷͰνʔϜ͝ͱʹঢ়گ͕ҟͳΔ
    • ଟ͘ͷϓϩμΫτ͕ଘࡏ͠ɺ૿Ճத

    View full-size slide

  8. ݩʑͷCDͷঢ়گ
    શମత͸ͳΜͰ΋͋Δ͜ͱʹͳͬͯ͠·ͬͨ
    खಈ
    Jenkins CircleCI
    TravisCI
    DroneCI
    Harness
    Concource
    ArgoCD
    GH Actions
    Spinnaker
    Flux
    Code
    Deploy
    Cloud
    Build Cloud
    Deploy
    ୲౰ऀ͕खಈͰ΍Δ CIͰCDΛ΍Δ (CI Ops) ઐ༻CDΛ࢖͏
    ࣗ࡞
    ChatOps

    View full-size slide

  9. CDʹؔ͢Δ՝୊
    Multi Cloud Product
    • σϦόϦͷ੒ख़౓͕ߴ͘ͳ͍ͱ͜Ζ͕ଘࡏ

    • खಈϦϦʔεͷνΣοΫϦετ͕େม

    • Canary, BlueGreenͳͲͷ҆શରࡦ͕ͳ͍

    • ೚ͤΔͨΊɺࣗ෼ͰCDͷߏஙͱӡ༻͕େม
    • ౷ҰੑʢҰ؏ੑʣ͕௿͘ͳ͍ͬͯΔ

    • ࣾ಺ελϯμʔυΛ੒ཱ͢Δ͜ͱ͕೉͘͠ͳΔ

    • ࣾ಺ͷϕετϓϥΫςΟεͷීٴ͕େม

    • શମతͳӡ༻ίετ͕ߴ͍

    • Developer ExperienceʹӨڹͯ͠͠·͏

    • ࣾ಺ҠಈͰ΋Onboardingίετ͕ൃੜ
    ٕज़ελοΫ͕ಉ͡Ͱ΋

    ϓϩμΫτຖʹCDγεςϜɾϓϩηε͕ҟͳΔ
    ಉ͡ϓϩμΫτͰ΋

    ෳ਺CDγεςϜɾϓϩηε͕ଘࡏ͢Δ
    Product A Product B
    CD System X CD System Y
    CD System X for GCP services
    CD System Y for AWS Services
    CD System Z for Infra

    View full-size slide

  10. ౷ҰͳσϦόϦʔج൫Λݕ౼

    View full-size slide

  11. CIͱCDΛ׬શతʹ෼཭
    TestͰૣظతʹ։ൃऀ΁ϑΟʔυόοΫΛఏڙ
    ArtifactΛHost Environment΁σϓϩΠɾϩʔϧόοΫ
    BuildͰImmutable ArtifactΛੜ੒͢Δ
    Host Environment (ClusterͳͲʣ΁ܨ͕Βͳ͍ Host Environment (ClusterͳͲʣ΁௚઀తʹܨ͕Δ
    ੜ੒͞ΕͨArtifactʹؔ͢ΔϑΟʔυόοΫΛૣΊʹఏڙ

    View full-size slide

  12. ౷ҰͳCDγεςϜ
    Delivery Infrastructure
    Product Team͸࠷େͳσϦόϦύϑΥʔϚϯεΛ
    ग़ͤͳ͕Βɺࣗ෼ʹ߹͏ٕज़ελοΫͱσϦόϦ
    ઓུΛબͿϑϦʔμϜ͕͋Δ
    Platform Team͸ॊೈͳDelivery InfrastructureΛ

    ఏڙͱCDͷϕετϓϥΫςΟεΛීٴ
    Platform Team & Product Teamͷ྆ํʹϝϦοτΛग़ͤΔ
    Standard/Consistency vs Freedomͷྑ͍όϥϯεΛऔΕΔ
    Ͳ͏͢Ε͹ɺ

    View full-size slide

  13. Platform Team & Product Team
    Management
    Security
    γεςϜͷӡ༻͸ίετ͕ߴ͍͔Ͳ͏͔

    ϓϩμΫτಋೖͷεέʔϥϏϦςΟ
    ϓϩμΫτͷΫϨσϯγϟϧΛͲ͏؅ཧ͞ΕΔ͔


    CDηΩϡϦςΟʔपΓͷϕετϓϥΫςΟεΛϓϩμΫτνʔϜ΁ීٴͰ͖Δ͔Ͳ͏͔
    Automation
    Visibility /
    Accessibility
    Control /

    Flexibility
    νʔϜʹϑΟοτٕज़ελοΫɾσϦόϦख๏Λࣗ༝ʹબ͹ΕΔͷ͔


    νʔϜͷݖݶΛద੾ʹ؅ཧͰ͖Δͷ͔
    ϦϦʔεϓϩηεͷதʹɺਓؒͷखಈλεΫ͕Ͳ͜·Ͱ࡟ݮͰ͖Δͷ͔


    ϦϦʔεத΋ޙ΋ܧଓతʹ໰୊Λݕ஌Ͱ͖Δͷ͔
    ։ൃऀ΁े෼ͳϑΟʔυόοΫΛ଎ΊʹఏڙͰ͖Δ͔

    ໰୊͕͋Δ࣌ʹݪҼ΋ؚΉঢ়ଶΛͪΌΜͱݟ͑Δͷ͔

    ؅ཧऀ΁νʔϜͷσϦόϦʔύϑΥʔϚϯεΛͪΌΜͱݟͤΔͷ͔
    Platform

    Team
    Product

    Team

    View full-size slide

  14. PipeCDͷ஀ੜ

    View full-size slide

  15. PipeCD 🤍 OSS
    Thanks to the contributors of PipeCD!
    https://github.com/pipe-cd/pipe
    https://pipecd.dev
    https://pipecd.dev/docs
    27 Contributors
    4 Full-time contributors

    1 Part-time contributor

    1800 Pull requests
    ʢ෭ۀʣ

    View full-size slide

  16. PipeCD 🤍 GitOps
    • GitOpsΛ࠾༻

    • GitʹશͯͷCon
    fi
    gurationΛ؅ཧ

    • શͯͷΦϖϨʔγϣϯ͕Git Pull Requestܦ༝Λߦ͑Δ

    • Πϯϑϥ͔ΒΞϓϦέʔγϣϯ·Ͱ౷ҰͳGitOps

    • Kubernetes

    • Terraform

    • CloudRun

    • AWS Lambda

    • ECS, Fargate

    • ...

    View full-size slide

  17. PipeCD͸ͲͷΑ͏ʹզʑͷ՝୊Λղܾ͢Δ͔
    Automation
    Visibility /
    Accessibility
    Control /

    Flexibility
    Management Security
    Platform Team
    Product Team

    View full-size slide

  18. Platform Team - Management
    • ӡ༻ίετ͕௿͍

    • શͯͷσϓϩΠϝϯτ͸ҰͭͷγεςϜͷΈ͕ඞཁ

    • PipeCDͷશͯͷίϯϙʔωϯτ͕εςʔτϨε

    • ετϨʔδ͸ϚωʔδυαʔϏεΛར༻͢Δ͜ͱ͕Մೳ

    • ϓϩμΫτଆ͸γϯάϧόΠφϦͷPiped agentΛΠϯετʔϧͷΈ

    • ϓϩμΫτͷεέʔϥϏϦςΟ

    • ৽نͷϓϩμΫτͷ௥Ճ͕୯౬

    • Ͳͷ؀ڥͰ΋ಈ͚Δઃܭ

    • ωοτϫʔΫ੍ݶͷPrivate Cloud͔ΒPublic Cloud·Ͱ

    • খن໛νʔϜ͔Βେن໛νʔϜ·Ͱ
    Product B
    Piped
    Control Plane
    Firewall

    Friendly
    Outbout

    Requests

    Only
    Product A
    Piped
    Piped
    Piped

    View full-size slide

  19. Platform Team - Security
    • ϓϩμΫτνʔϜͷΫϨσϯγϟϧ͸֎ʹҰ੾ग़ͳ͍ઃܭ

    • Control-planeʹ΋อଘɾ؅ཧ͠ͳ͍

    • GitOpsͰͷSecret؅ཧͷϕετϓϥΫςΟεͱͯ͠ͷbuilt-inػೳΛؚΉ

    • Piped agent͕Ξ΢τό΢ϯυϦΫΤετͷΈΛߦ͏ͷͰɺެ։ϙʔτͳͲ͸ඞཁ͕ͳ͍

    View full-size slide

  20. Product Team - Visibility / Accessibility
    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ

    • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ

    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌

    • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ

    • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ

    • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε https://pipecd.dev/docs/user-guide/triggering-a-deployment

    View full-size slide

  21. Product Team - Visibility / Accessibility
    https://pipecd.dev/docs/user-guide/application-live-state
    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ

    • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ

    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌

    • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ

    • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ

    • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε

    View full-size slide

  22. Product Team - Visibility / Accessibility
    https://pipecd.dev/docs/user-guide/plan-preview
    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ

    • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ

    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌

    • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ

    • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ

    • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε
    Git di
    ff
    is enought? No.
    e.g. using remote Helm chart, Kustomize package, Terraform module...
    Reviewer needs more early feedback to merge PR with con
    fi
    dence
    Better to see dry-run result, terraform plan, deployment policy... on the PR
    then Git di
    ff
    is just showing the change of version number

    View full-size slide

  23. Product Team - Visibility / Accessibility
    Terraform Deployment
    CloudRun Deployment
    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ

    • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ

    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌

    • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ

    • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ

    • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε
    Scale In Scale Out
    Rollout
    New Image
    Update

    Con
    fi
    g
    Deploy
    Infra
    Deploy
    Serverless
    Rollback
    Deploy


    Kubernetes
    Constant process for all operations

    View full-size slide

  24. Product Team - Automation
    https://pipecd.dev/docs/user-guide/rolling-back-a-deployment
    • σϓϩΠͷΠϯύΫτΛࣗಈ෼ੳ

    • ϝτϦΫεɾϩάɾhttp

    • աڈͷσʔλɾcanary-baselineͷൺֱ

    • σϓϩΠதʹ໰୊͕ൃੜ͢ΔͱࣗಈϩʔϧόοΫ

    • Gitͱ࣮ࡍͷঢ়ଶͷCon
    fi
    guration DriftΛࣗಈݕ஌

    • ৽ίϯςφΠϝʔδɾHelm Chart͕ग़Δͱ

    ࣗಈσϓϩΠͷEventWatcher
    https://pipecd.dev/docs/user-guide/con
    fi
    guration-drift-detection

    View full-size slide

  25. Product Team - Control / Flexibility
    • Piped AgentΛ޷͖ͳελΠϧͰ૊Έ߹ΘͤΔ͜ͱ͕Մೳ

    • 1 Piped AgentͰશͯ؅ཧύλʔϯ

    • ؀ڥຖʹઐ༻ͷPiped Agentύλʔϯ

    • ޷͖ͳ৔ॴʹΠϯετʔϧʢKubernetes podɺVMʹதɺFargateαʔϏε...ʣ

    • σϓϩΠઓུ͸ࣗ༝ʹ૊Έ߹Θͤɾఆ͕ٛՄೳ

    • Quick Sync

    • Progressive Sync (Canary, BlueGreen...)

    • νʔϜʹ߹͏ϓϩόΠμʔΛબ΂ΒΕΔ

    • Cloud Provider (GCP, AWS, Azure, Private Cloud)

    • Analysis Provider (Prometheus, Datadog, Stackdriver...)
    https://github.com/pipe-cd/examples/

    blob/master/kubernetes/canary/.pipe.yaml

    View full-size slide

  26. PipeCD
    Platform Team͕Delivery InfrastructureΛఏڙͰ͖ɺ

    ϕετϓϥΫςΟεΛܧଓతʹ࠾༻ɾීٴͰ͖Δ
    Product Team͕CDͷߏஙɾӡ༻͕ෆཁʹͳΓɺҰճಋೖ͢Δ͚ͩͰɺ

    ࣗ෼ʹ߹͏σϦόϦख๏Λ࠾༻Ͱ͖ɺܧଓతʹվળͰ͖Δ
    ͜ΕͰɺ

    View full-size slide

  27. ݱࡏͷPipeCD @ CyberAgent

    View full-size slide

  28. νʔϜͱγεςϜͷߏ੒
    • Platform Team

    • PipeCDΛ։ൃ

    • ࣾ಺༻PipeCD Control-planeΛӡ༻

    • ֤Product Team

    • Piped agentΛΠϯετʔϧ

    • ࣗ෼ʹ߹͏σϓϩΠϝϯτछྨΛ࠾༻
    Control-Plane
    https://pipecd.dev/docs/operator-manual/piped/installation
    https://github.com/pipe-cd/examples

    View full-size slide

  29. ΞϓϦέʔγϣϯɾαʔϏε͕૿Ճத
    0
    225
    450
    675
    900
    2020/10 2020/12 2021/02 2021/04 2021/06 2021/8
    806
    Applications/Servicesʹୡ੒
    CyberAgentʹPipeCDͰӡ༻͍ͯ͠Δ Kubernetes, Terraform, Lambda, CloudRun, Fargate... 

    ͷΞϓϦέʔγϣϯɾαʔϏε਺

    View full-size slide

  30. PipeCDͷࠓޙ

    View full-size slide

  31. ֤࣠Λ͞ΒʹڧԽ
    Automation
    Visibility /
    Accessibility
    Control /

    Flexibility
    Management Security

    View full-size slide

  32. • ࠓ·ͰͷPiped agentͷӡ༻

    • Product Team͕Piped agentΛΠϯετʔϧ

    • ৽͍͠όʔδϣϯ͕͋Δͱ࠶Πϯετʔϧ

    • ͜Ε͔Β

    • ҰճͷΈΠϯετʔϧ͢Δ

    • Web consoleͰόʔδϣϯΛΞοϓͰ͖Δ

    • ͦΕͰӡ༻͕͞ΒʹָʹͳΔ
    Management - Remote Upgrade
    ͜ͷػೳ͸དྷिʹϦϦʔε༧ఆ
    Product B
    Piped
    Control Plane
    Firewall

    Friendly
    Outbout

    Requests

    Only
    Product A
    Piped
    Piped
    Piped

    View full-size slide

  33. Visibility / Accessiblity
    • InsightsͰσϦόϦʔύϑΥʔϚϯεΛՄࢹԽ

    • Deployස౓ɺLead Time, ࣦഊ཰, MTTR...

    • ApplicationͷϦιʔεͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ

    • Terraform, CloudRun, ECS...

    • Deployment StageͷϩάΛվળ

    • Plan-Previewʹ΋ͬͱϑΟʔυόοΫΛՃ͑Δ

    • Kubernetes validating webhook, dry-run݁Ռ, Terraform sentinelͳͲͷDeployment Policy
    ֎ͷϢʔβʔ޲͚ͷPlayground؀ڥΛ४උதʂ

    https://play.pipecd.dev

    View full-size slide

  34. Automation
    • LogͷσʔλͰσϓϩΠϝϯτΛࣗಈ෼ੳ

    • աڈͷMetricsσʔλͱͷൺֱͰࣗಈ෼ੳ

    • Primary/Baseline & CanaryͷMetricsσʔλͷൺֱͰࣗಈ෼ੳ

    • Terraform, CloudRun, LambdaͳͲͷࣗಈDrift Detection

    View full-size slide

  35. Control / Flexibility
    • ਂ͍Ϩϕϧͷݖݶ؅ཧͰ͖ΔACL

    • Piped AgentʹPlug-in ArchitectureΛ࠾༻Ͱɺ

    ಠࣗͷσϓϩΠϝϯτϩδοΫΛຒΊࠐΊΔΑ͏ʹ

    • Deployment ChainͰmulti-clusterͷΞϓϦέʔγϣϯ΍

    σϓϩΠͷॱ൪੍ޚ͕Ͱ͖ΔΑ͏ʹ

    View full-size slide

  36. Deployment Chain
    Application X
    Region A
    Region B
    Region C



    Application X
    Cluster A
    Cluster B
    Cluster C



    Application X - Dev Env
    Application X - Stg Env
    Application X - Prod Env
    Application Infra
    Application X
    Application Y
    ੒ޭͳΒ͹࣍ʹਐΉ ੒ޭͳΒ͹࣍ʹਐΉ

    View full-size slide

  37. Feature Status
    ۩ମతͳػೳͷঢ়ଶ͸ҎԼͷϖʔδͰ֬ೝͰ͖·͢
    https://pipecd.dev/docs/feature-status

    View full-size slide

  38. Thank you!
    PipeCD OSSͷBackendͱFrontendͷϑϧλΠϜɾ෭ۀʢ࣌ؒͷ੍ݶ͸ͳ͠ʣΛืू͍ͯ͠·͢

    ͝ڵຯ͕͋ΔํɺTwitterͷDMͳͲ͝࿈བྷ͍ͩ͘͞

    View full-size slide