Why and how we build a unified CD system

Transcript

1. nghialv
   Developer Productivity Team, CyberAgent
   Oct 08, 2021
   ౷ҰͳCDγεςϜΛߏஙͨ͠࿩
   ๻ͨͪࢲͨͪͷCI/CD͸͜Εͩʂ- cndjp #17
   

   View Slide

2. ࣗݾ঺հ
   @nghialv
   @nghialv2607
   @nghialv
   Le Van Nghia - ΪΞ
   

   View Slide

3. ࣗݾ঺հ - ৬ྺ @CyberAgent
   • PipeCDΛ։ൃɾӡ༻ - DPࣨ
   
   • Work
   fl
   ow Automation SystemΛ։ൃɾӡ༻ - OSSS
   
   • Feature Flags SystemΛ։ൃɾӡ༻ - AbemaTV
   
   • Prometheus & GrafanaͰMonitoring SystemΛߏஙɾӡ༻ - AbemaTV
   
   • DeploymentπʔϧΛ։ൃɾӡ༻ - AbemaTV
   
   • Microservicesɾج൫पΓ - AbemaTV
   

   View Slide

4. ࠓ೔ͷ࿩͢಺༰
   • CyberAgentͰCDʹؔ͢Δ՝୊
   
   • ౷ҰͳσϦόϦʔج൫Λݕ౼
   
   • PipeCDͷ஀ੜ
   
   • ݱࡏͷPipeCD @ CyberAgent
   
   • PipeCDͷࠓޙ
   

   View Slide

5. CyberAgentͰCDʹؔ͢Δ՝୊
   

   View Slide

6. ϓϩμΫτͷߏ੒
   ABEMA AWA WinTicket CyberZ AI Studio
   ...
   • ଟ͘ͷϓϩμΫτ͕ଘࡏ͠ɺ૿Ճத
   • ֤ϓϩμΫτͰ͸ࣗ෼ʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબ΂Δ
   Group
   • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾ಺Ҡಈ͸Մೳ
   ʢ਺ेݸ͕͋Δʣ
   

   View Slide

7. ϓϩμΫτͷߏ੒
   ABEMA AWA WinTicket CyberZ AI Studio
   ...
   • ֤ϓϩμΫτͰ͸ࣗ෼ʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબ΂Δ
   Group
   • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾ಺Ҡಈ͸Մೳ
   ʢ਺ेݸ͕͋Δʣ
   CD΋ϓϩμΫτΤϯδχΞʹࣗ༝ʹ೚ͤΔͷͰνʔϜ͝ͱʹঢ়گ͕ҟͳΔ
   • ଟ͘ͷϓϩμΫτ͕ଘࡏ͠ɺ૿Ճத
   

   View Slide

8. ݩʑͷCDͷঢ়گ
   શମత͸ͳΜͰ΋͋Δ͜ͱʹͳͬͯ͠·ͬͨ
   खಈ
   Jenkins CircleCI
   TravisCI
   DroneCI
   Harness
   Concource
   ArgoCD
   GH Actions
   Spinnaker
   Flux
   Code
   Deploy
   Cloud
   Build Cloud
   Deploy
   ୲౰ऀ͕खಈͰ΍Δ CIͰCDΛ΍Δ (CI Ops) ઐ༻CDΛ࢖͏
   ࣗ࡞
   ChatOps
   

   View Slide

9. CDʹؔ͢Δ՝୊
   Multi Cloud Product
   • σϦόϦͷ੒ख़౓͕ߴ͘ͳ͍ͱ͜Ζ͕ଘࡏ
   
   • खಈϦϦʔεͷνΣοΫϦετ͕େม
   
   • Canary, BlueGreenͳͲͷ҆શରࡦ͕ͳ͍
   
   • ೚ͤΔͨΊɺࣗ෼ͰCDͷߏஙͱӡ༻͕େม
   • ౷ҰੑʢҰ؏ੑʣ͕௿͘ͳ͍ͬͯΔ
   
   • ࣾ಺ελϯμʔυΛ੒ཱ͢Δ͜ͱ͕೉͘͠ͳΔ
   
   • ࣾ಺ͷϕετϓϥΫςΟεͷීٴ͕େม
   
   • શମతͳӡ༻ίετ͕ߴ͍
   
   • Developer ExperienceʹӨڹͯ͠͠·͏
   
   • ࣾ಺ҠಈͰ΋Onboardingίετ͕ൃੜ
   ٕज़ελοΫ͕ಉ͡Ͱ΋

   ϓϩμΫτຖʹCDγεςϜɾϓϩηε͕ҟͳΔ
   ಉ͡ϓϩμΫτͰ΋

   ෳ਺CDγεςϜɾϓϩηε͕ଘࡏ͢Δ
   Product A Product B
   CD System X CD System Y
   CD System X for GCP services
   CD System Y for AWS Services
   CD System Z for Infra
   

   View Slide

10. ౷ҰͳσϦόϦʔج൫Λݕ౼
    

    View Slide

11. CIͱCDΛ׬શతʹ෼཭
    TestͰૣظతʹ։ൃऀ΁ϑΟʔυόοΫΛఏڙ
    ArtifactΛHost Environment΁σϓϩΠɾϩʔϧόοΫ
    BuildͰImmutable ArtifactΛੜ੒͢Δ
    Host Environment (ClusterͳͲʣ΁ܨ͕Βͳ͍ Host Environment (ClusterͳͲʣ΁௚઀తʹܨ͕Δ
    ੜ੒͞ΕͨArtifactʹؔ͢ΔϑΟʔυόοΫΛૣΊʹఏڙ
    

    View Slide

12. ౷ҰͳCDγεςϜ
    Delivery Infrastructure
    Product Team͸࠷େͳσϦόϦύϑΥʔϚϯεΛ
    ग़ͤͳ͕Βɺࣗ෼ʹ߹͏ٕज़ελοΫͱσϦόϦ
    ઓུΛબͿϑϦʔμϜ͕͋Δ
    Platform Team͸ॊೈͳDelivery InfrastructureΛ

    ఏڙͱCDͷϕετϓϥΫςΟεΛීٴ
    Platform Team & Product Teamͷ྆ํʹϝϦοτΛग़ͤΔ
    Standard/Consistency vs Freedomͷྑ͍όϥϯεΛऔΕΔ
    Ͳ͏͢Ε͹ɺ
    

    View Slide

13. Platform Team & Product Team
    Management
    Security
    γεςϜͷӡ༻͸ίετ͕ߴ͍͔Ͳ͏͔

    ϓϩμΫτಋೖͷεέʔϥϏϦςΟ
    ϓϩμΫτͷΫϨσϯγϟϧΛͲ͏؅ཧ͞ΕΔ͔

    

    CDηΩϡϦςΟʔपΓͷϕετϓϥΫςΟεΛϓϩμΫτνʔϜ΁ීٴͰ͖Δ͔Ͳ͏͔
    Automation
    Visibility /
    Accessibility
    Control /
    

    Flexibility
    νʔϜʹϑΟοτٕज़ελοΫɾσϦόϦख๏Λࣗ༝ʹબ͹ΕΔͷ͔

    

    νʔϜͷݖݶΛద੾ʹ؅ཧͰ͖Δͷ͔
    ϦϦʔεϓϩηεͷதʹɺਓؒͷखಈλεΫ͕Ͳ͜·Ͱ࡟ݮͰ͖Δͷ͔

    

    ϦϦʔεத΋ޙ΋ܧଓతʹ໰୊Λݕ஌Ͱ͖Δͷ͔
    ։ൃऀ΁े෼ͳϑΟʔυόοΫΛ଎ΊʹఏڙͰ͖Δ͔

    ໰୊͕͋Δ࣌ʹݪҼ΋ؚΉঢ়ଶΛͪΌΜͱݟ͑Δͷ͔

    ؅ཧऀ΁νʔϜͷσϦόϦʔύϑΥʔϚϯεΛͪΌΜͱݟͤΔͷ͔
    Platform

    Team
    Product

    Team
    

    View Slide

14. PipeCDͷ஀ੜ
    

    View Slide

15. PipeCD
    

    View Slide

16. PipeCD 🤍 OSS
    Thanks to the contributors of PipeCD!
    https://github.com/pipe-cd/pipe
    https://pipecd.dev
    https://pipecd.dev/docs
    27 Contributors
    4 Full-time contributors
    
    1 Part-time contributor

    1800 Pull requests
    ʢ෭ۀʣ
    

    View Slide

17. PipeCD 🤍 GitOps
    • GitOpsΛ࠾༻
    
    • GitʹશͯͷCon
    fi
    gurationΛ؅ཧ
    
    • શͯͷΦϖϨʔγϣϯ͕Git Pull Requestܦ༝Λߦ͑Δ
    
    • Πϯϑϥ͔ΒΞϓϦέʔγϣϯ·Ͱ౷ҰͳGitOps
    
    • Kubernetes
    
    • Terraform
    
    • CloudRun
    
    • AWS Lambda
    
    • ECS, Fargate
    
    • ...
    

    View Slide

18. PipeCD͸ͲͷΑ͏ʹզʑͷ՝୊Λղܾ͢Δ͔
    Automation
    Visibility /
    Accessibility
    Control /
    

    Flexibility
    Management Security
    Platform Team
    Product Team
    

    View Slide

19. Platform Team - Management
    • ӡ༻ίετ͕௿͍
    
    • શͯͷσϓϩΠϝϯτ͸ҰͭͷγεςϜͷΈ͕ඞཁ
    
    • PipeCDͷશͯͷίϯϙʔωϯτ͕εςʔτϨε
    
    • ετϨʔδ͸ϚωʔδυαʔϏεΛར༻͢Δ͜ͱ͕Մೳ
    
    • ϓϩμΫτଆ͸γϯάϧόΠφϦͷPiped agentΛΠϯετʔϧͷΈ
    
    • ϓϩμΫτͷεέʔϥϏϦςΟ
    
    • ৽نͷϓϩμΫτͷ௥Ճ͕୯౬
    
    • Ͳͷ؀ڥͰ΋ಈ͚Δઃܭ
    
    • ωοτϫʔΫ੍ݶͷPrivate Cloud͔ΒPublic Cloud·Ͱ
    
    • খن໛νʔϜ͔Βେن໛νʔϜ·Ͱ
    Product B
    Piped
    Control Plane
    Firewall

    Friendly
    Outbout

    Requests

    Only
    Product A
    Piped
    Piped
    Piped
    

    View Slide

20. Platform Team - Security
    • ϓϩμΫτνʔϜͷΫϨσϯγϟϧ͸֎ʹҰ੾ग़ͳ͍ઃܭ
    
    • Control-planeʹ΋อଘɾ؅ཧ͠ͳ͍
    
    • GitOpsͰͷSecret؅ཧͷϕετϓϥΫςΟεͱͯ͠ͷbuilt-inػೳΛؚΉ
    
    • Piped agent͕Ξ΢τό΢ϯυϦΫΤετͷΈΛߦ͏ͷͰɺެ։ϙʔτͳͲ͸ඞཁ͕ͳ͍
    

    View Slide

21. Product Team - Visibility / Accessibility
    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ
    
    • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ
    
    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌
    
    • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ
    
    • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ
    
    • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε https://pipecd.dev/docs/user-guide/triggering-a-deployment
    

    View Slide

22. Product Team - Visibility / Accessibility
    https://pipecd.dev/docs/user-guide/application-live-state
    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ
    
    • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ
    
    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌
    
    • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ
    
    • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ
    
    • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε
    

    View Slide

23. Product Team - Visibility / Accessibility
    https://pipecd.dev/docs/user-guide/plan-preview
    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ
    
    • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ
    
    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌
    
    • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ
    
    • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ
    
    • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε
    Git di
    ff
    is enought? No.
    e.g. using remote Helm chart, Kustomize package, Terraform module...
    Reviewer needs more early feedback to merge PR with con
    fi
    dence
    Better to see dry-run result, terraform plan, deployment policy... on the PR
    then Git di
    ff
    is just showing the change of version number
    

    View Slide

24. Product Team - Visibility / Accessibility
    Terraform Deployment
    CloudRun Deployment
    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ
    
    • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ
    
    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌
    
    • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ
    
    • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ
    
    • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε
    Scale In Scale Out
    Rollout
    New Image
    Update
    

    Con
    fi
    g
    Deploy
    Infra
    Deploy
    Serverless
    Rollback
    Deploy
    
    
    Kubernetes
    Constant process for all operations
    

    View Slide

25. Product Team - Automation
    https://pipecd.dev/docs/user-guide/rolling-back-a-deployment
    • σϓϩΠͷΠϯύΫτΛࣗಈ෼ੳ
    
    • ϝτϦΫεɾϩάɾhttp
    
    • աڈͷσʔλɾcanary-baselineͷൺֱ
    
    • σϓϩΠதʹ໰୊͕ൃੜ͢ΔͱࣗಈϩʔϧόοΫ
    
    • Gitͱ࣮ࡍͷঢ়ଶͷCon
    fi
    guration DriftΛࣗಈݕ஌
    
    • ৽ίϯςφΠϝʔδɾHelm Chart͕ग़Δͱ

    ࣗಈσϓϩΠͷEventWatcher
    https://pipecd.dev/docs/user-guide/con
    fi
    guration-drift-detection
    

    View Slide

26. Product Team - Control / Flexibility
    • Piped AgentΛ޷͖ͳελΠϧͰ૊Έ߹ΘͤΔ͜ͱ͕Մೳ
    
    • 1 Piped AgentͰશͯ؅ཧύλʔϯ
    
    • ؀ڥຖʹઐ༻ͷPiped Agentύλʔϯ
    
    • ޷͖ͳ৔ॴʹΠϯετʔϧʢKubernetes podɺVMʹதɺFargateαʔϏε...ʣ
    
    • σϓϩΠઓུ͸ࣗ༝ʹ૊Έ߹Θͤɾఆ͕ٛՄೳ
    
    • Quick Sync
    
    • Progressive Sync (Canary, BlueGreen...)
    
    • νʔϜʹ߹͏ϓϩόΠμʔΛબ΂ΒΕΔ
    
    • Cloud Provider (GCP, AWS, Azure, Private Cloud)
    
    • Analysis Provider (Prometheus, Datadog, Stackdriver...)
    https://github.com/pipe-cd/examples/

    blob/master/kubernetes/canary/.pipe.yaml
    

    View Slide

27. PipeCD
    Platform Team͕Delivery InfrastructureΛఏڙͰ͖ɺ

    ϕετϓϥΫςΟεΛܧଓతʹ࠾༻ɾීٴͰ͖Δ
    Product Team͕CDͷߏஙɾӡ༻͕ෆཁʹͳΓɺҰճಋೖ͢Δ͚ͩͰɺ

    ࣗ෼ʹ߹͏σϦόϦख๏Λ࠾༻Ͱ͖ɺܧଓతʹվળͰ͖Δ
    ͜ΕͰɺ
    

    View Slide

28. ݱࡏͷPipeCD @ CyberAgent
    

    View Slide

29. νʔϜͱγεςϜͷߏ੒
    • Platform Team
    
    • PipeCDΛ։ൃ
    
    • ࣾ಺༻PipeCD Control-planeΛӡ༻
    
    • ֤Product Team
    
    • Piped agentΛΠϯετʔϧ

    • ࣗ෼ʹ߹͏σϓϩΠϝϯτछྨΛ࠾༻
    Control-Plane
    https://pipecd.dev/docs/operator-manual/piped/installation
    https://github.com/pipe-cd/examples
    

    View Slide

30. ΞϓϦέʔγϣϯɾαʔϏε͕૿Ճத
    0
    225
    450
    675
    900
    2020/10 2020/12 2021/02 2021/04 2021/06 2021/8
    806
    Applications/Servicesʹୡ੒
    CyberAgentʹPipeCDͰӡ༻͍ͯ͠Δ Kubernetes, Terraform, Lambda, CloudRun, Fargate... 

    ͷΞϓϦέʔγϣϯɾαʔϏε਺
    

    View Slide

31. PipeCDͷࠓޙ
    

    View Slide

32. ֤࣠Λ͞ΒʹڧԽ
    Automation
    Visibility /
    Accessibility
    Control /
    

    Flexibility
    Management Security
    

    View Slide

33. • ࠓ·ͰͷPiped agentͷӡ༻
    
    • Product Team͕Piped agentΛΠϯετʔϧ
    
    • ৽͍͠όʔδϣϯ͕͋Δͱ࠶Πϯετʔϧ
    
    • ͜Ε͔Β
    
    • ҰճͷΈΠϯετʔϧ͢Δ
    
    • Web consoleͰόʔδϣϯΛΞοϓͰ͖Δ
    
    • ͦΕͰӡ༻͕͞ΒʹָʹͳΔ
    Management - Remote Upgrade
    ͜ͷػೳ͸དྷिʹϦϦʔε༧ఆ
    Product B
    Piped
    Control Plane
    Firewall

    Friendly
    Outbout

    Requests

    Only
    Product A
    Piped
    Piped
    Piped
    

    View Slide

34. Visibility / Accessiblity
    • InsightsͰσϦόϦʔύϑΥʔϚϯεΛՄࢹԽ
    
    • Deployස౓ɺLead Time, ࣦഊ཰, MTTR...
    
    • ApplicationͷϦιʔεͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ
    
    • Terraform, CloudRun, ECS...
    
    • Deployment StageͷϩάΛվળ
    
    • Plan-Previewʹ΋ͬͱϑΟʔυόοΫΛՃ͑Δ
    
    • Kubernetes validating webhook, dry-run݁Ռ, Terraform sentinelͳͲͷDeployment Policy
    ֎ͷϢʔβʔ޲͚ͷPlayground؀ڥΛ४උதʂ

    https://play.pipecd.dev
    

    View Slide

35. Automation
    • LogͷσʔλͰσϓϩΠϝϯτΛࣗಈ෼ੳ
    
    • աڈͷMetricsσʔλͱͷൺֱͰࣗಈ෼ੳ
    
    • Primary/Baseline & CanaryͷMetricsσʔλͷൺֱͰࣗಈ෼ੳ
    
    • Terraform, CloudRun, LambdaͳͲͷࣗಈDrift Detection
    

    View Slide

36. Control / Flexibility
    • ਂ͍Ϩϕϧͷݖݶ؅ཧͰ͖ΔACL
    
    • Piped AgentʹPlug-in ArchitectureΛ࠾༻Ͱɺ

    ಠࣗͷσϓϩΠϝϯτϩδοΫΛຒΊࠐΊΔΑ͏ʹ
    
    • Deployment ChainͰmulti-clusterͷΞϓϦέʔγϣϯ΍

    σϓϩΠͷॱ൪੍ޚ͕Ͱ͖ΔΑ͏ʹ
    

    View Slide

37. Deployment Chain
    Application X
    Region A
    Region B
    Region C
    ᶃ
    ᶄ
    ᶅ
    Application X
    Cluster A
    Cluster B
    Cluster C
    ᶃ
    ᶄ
    ᶅ
    Application X - Dev Env
    Application X - Stg Env
    Application X - Prod Env
    Application Infra
    Application X
    Application Y
    ੒ޭͳΒ͹࣍ʹਐΉ ੒ޭͳΒ͹࣍ʹਐΉ
    

    View Slide

38. Feature Status
    ۩ମతͳػೳͷঢ়ଶ͸ҎԼͷϖʔδͰ֬ೝͰ͖·͢
    https://pipecd.dev/docs/feature-status
    

    View Slide

39. Thank you!
    PipeCD OSSͷBackendͱFrontendͷϑϧλΠϜɾ෭ۀʢ࣌ؒͷ੍ݶ͸ͳ͠ʣΛืू͍ͯ͠·͢

    ͝ڵຯ͕͋ΔํɺTwitterͷDMͳͲ͝࿈བྷ͍ͩ͘͞
    

    View Slide