PipeCD at CyberAgent

Transcript

1. nghialv
   Developer Productivity Team, CyberAgent
   Sep 03, 2021
   PipeCD @ CyberAgent
   CI/CD Conference 2021 by CloudNative Days
   

   View Slide

2. ࣗݾ঺հ
   @nghialv
   @nghialv2607
   @nghialv
   Le Van Nghia - ΪΞ
   

   View Slide

3. ࣗݾ঺հ - ৬ྺ @CyberAgent
   • PipeCDΛ։ൃɾӡ༻ - DPࣨ
   
   • Work
   fl
   ow Automation SystemΛ։ൃɾӡ༻ - OSSS
   
   • Feature Flags SystemΛ։ൃɾӡ༻ - AbemaTV
   
   • Prometheus & GrafanaͰMonitoring SystemΛߏஙɾӡ༻ - AbemaTV
   
   • DeploymentπʔϧΛ։ൃɾӡ༻ - AbemaTV
   
   • Microservicesɾج൫पΓ - AbemaTV
   

   View Slide

4. ࠓ೔ͷ࿩͢಺༰
   • CyberAgentͰCDͷ՝୊
   
   • PipeCDͷ஀ੜ
   
   • ݱࡏͷPipeCD @ CyberAgent
   
   • PipeCDͷࠓޙ
   

   View Slide

5. ຊ୊΁ೖΔલʹ
   

   View Slide

6. CI/CD
   

   View Slide

7. CI != CD
   

   View Slide

8. CyberAgentͰCDͷ՝୊
   

   View Slide

9. ϓϩμΫτͷߏ੒
   ABEMA AWA WinTicket Game AI Studio
   ...
   • ଟ͘ͷϓϩμΫτ͕ଘࡏ͍ͯ͠Δ
   • ֤ϓϩμΫτͰ͸ࣗ෼ʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબ΂Δ
   Group
   • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾ಺Ҡಈͷ৔߹͸Մೳ
   

   View Slide

10. CDͷ՝୊
    Multi Cloud Product
    • σϦόϦͷ੒ख़౓͕ߴ͘ͳ͍νʔϜ͕ଘࡏ
    
    • खಈϦϦʔεͷνΣοΫϦετ͕େม
    
    • Canary, BlueGreenͳͲͷ҆શରࡦ͕ͳ͍
    
    • ཧ༝͸ࣗ෼ͰCDͷߏஙͱӡ༻͕େมͱϦιʔε͕ͳ͍
    • ౷ҰੑʢҰ؏ੑʣ͕௿͘ͳ͍ͬͯΔ
    
    • ࣾ಺ελϯμʔυΛ੒ཱ͢Δ͜ͱ͕೉͘͠ͳΔ
    
    • ࣾ಺ͷϕετϓϥΫςΟεͷීٴ͕େม
    
    • ؅ཧίετ͕ߴ͍
    
    • Developer ExperienceʹӨڹͯ͠͠·͏
    
    • Onboardingίετ͕͔͔Δ
    ٕज़ελοΫ͕ಉ͡Ͱ΋

    ϓϩμΫτຖʹCDγεςϜɾϓϩηε͕ҟͳΔ
    ಉ͡ϓϩμΫτͰ΋

    ෳ਺CDγεςϜɾϓϩηε͕ଘࡏ͢Δ
    Product A Product B
    CD System X CD System Y
    CD System X for GCP services
    CD System Y for AWS Services
    CD System Z for Infra
    

    View Slide

11. ࣾ಺౷ҰͳCDγεςϜΛݕ౼
    Delivery Infrastructure
    Product Team͸࠷େͳσϦόϦύϑΥʔϚ
    ϯεΛग़ͤͳ͕Βɺࣗ෼ʹ߹͏ٕज़ελοΫ
    ͱσϦόϦઓུΛબͿϑϦʔμϜ͕͋Δ
    Platform Team͸ॊೈͳDelivery InfrastructureΛఏڙͱ

    CDͷϕετϓϥΫςΟεΛීٴ
    Platform Team & Product Teamͷ྆ํʹϝϦοτΛग़ͤΔ
    Standards/Consistency vs Freedomͷྑ͍όϥϯεΛऔΕΔ
    Ͳ͏͢Ε͹ɺ
    

    View Slide

12. Platform Team
    Management
    Security
    γεςϜͷӡ༻͸ίετ͕ߴ͍ͷ͔

    ϓϩμΫτಋೖͷεέʔϥϏϦςΟ
    ϓϩμΫτͷΫϨσϯγϟϧΛͲ͏؅ཧ͢Δ͔

    

    CDηΩϡϦςΟʔपΓͷϕετϓϥΫςΟεΛ

    ϓϩμΫτνʔϜ΁ීٴͰ͖Δ͔Ͳ͏͔
    

    View Slide

13. Product Team
    Automation
    Visibility /
    Accessibility
    Control /
    

    Flexibility
    νʔϜʹϑΟοτٕज़ελοΫɾσϦόϦख๏Λࣗ༝ʹબ͹ΕΔͷ͔

    

    νʔϜͷݖݶΛద੾ʹ؅ཧͰ͖Δͷ͔
    ϦϦʔεϓϩηεͷதʹɺਓؒͷखಈλεΫ͕Ͳ͜·Ͱ࡟ݮͰ͖Δͷ͔

    

    ϦϦʔεத΋ޙ΋ܧଓతʹ໰୊Λݕ஌Ͱ͖Δͷ͔
    ։ൃऀ΁े෼ͳϑΟʔυόοΫΛ଎ΊʹఏڙͰ͖Δͷ͔

    ໰୊͕͋Δ࣌ʹݪҼ΋ؚΉঢ়ଶΛͪΌΜͱݟ͑Δͷ͔

    ؅ཧऀ΁νʔϜͷσϦόϦʔύϑΥʔϚϯεΛͪΌΜͱݟͤΔͷ͔
    

    View Slide

14. PipeCDͷ஀ੜ
    

    View Slide

15. PipeCD
    

    View Slide

16. PipeCD 🤍 OSS
    Thanks to the contributors of PipeCD!
    https://github.com/pipe-cd/pipe
    https://pipecd.dev
    https://pipecd.dev/docs
    ~1700 pull requests
    

    View Slide

17. PipeCD 🤍 GitOps
    • GitOpsΛ࠾༻
    
    • GitʹશͯͷCon
    fi
    gurationΛอଘ
    
    • શͯͷΦϖϨʔγϣϯ͕Git Pull Requestܦ༝Λߦ͑Δ
    
    • Πϯϑϥ͔ΒΞϓϦέʔγϣϯ·Ͱ౷ҰͳGitOps
    
    • Kubernetes
    
    • Terraform
    
    • CloudRun
    
    • AWS Lambda
    
    • ECS Fargate
    
    • ECS
    

    View Slide

18. PipeCD͸ͲͷΑ͏ʹզʑͷ՝୊Λղܾ͢Δ͔
    Automation
    Visibility /
    Accessibility
    Control /
    

    Flexibility
    Management Security
    Platform Team
    Product Team
    

    View Slide

19. Product B
    Platform Team - Management
    • ӡ༻ίετ͕௿͍
    
    • શͯ͸γϯάϧͳγεςϜͷΈ͕ඞཁ
    
    • Control-planeͷΞʔΩςΫνϟ͕؆୯
    
    • શͯͷίϯϙʔωϯτ͕εςʔτϨε
    
    • ετϨʔδ͸ϚωʔδυαʔϏεΛར༻͢Δ͜ͱ͕Մೳ
    
    • ϓϩμΫτଆ͸γϯάϧόΠφϦͷPiped AgentΛ

    ΠϯετʔϧͷΈͰɺશͯͷCDͷػೳΛར༻Ͱ͖Δ
    
    • ϓϩμΫτͷεέʔϥϏϦςΟ
    
    • ৽نͷϓϩμΫτͷ௥Ճ͕୯౬
    
    • Ͳͷ؀ڥͰ΋ಈ͚Δઃܭ
    
    • ωοτϫʔΫ੍ݶͷPrivate Cloud͔ΒPublic Cloud·Ͱ
    
    • খن໛νʔϜ͔Βେن໛νʔϜ·Ͱ
    Piped
    Control Plane
    Firewall

    Friendly
    Outbout

    Requests

    Only
    Product A
    Piped
    Piped
    Piped
    

    View Slide

20. Platform Team - Security
    • ϓϩμΫτνʔϜͷΫϨσϯγϟϧ͸֎ʹҰ੾ग़ͳ͍ઃܭ
    
    • Control-planeʹ΋อଘɾ؅ཧ͠ͳ͍
    
    • GitOpsͰͷSecret؅ཧͷϕετϓϥΫςΟεͱͯ͠ͷbuilt-inػೳΛؚΉ
    
    • Piped Agent͕Ξ΢τό΢ϯυϦΫΤετͷΈΛߦ͏ͷͰɺެ։ϙʔτͳͲ͸ඞཁ͕ͳ͍
    

    View Slide

21. Product Team - Visibility / Accessibility
    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ
    
    • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ
    
    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌
    
    • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ
    
    • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ
    
    • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε https://pipecd.dev/docs/user-guide/triggering-a-deployment
    

    View Slide

22. Product Team - Visibility / Accessibility
    https://pipecd.dev/docs/user-guide/application-live-state
    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ
    
    • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ
    
    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌
    
    • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ
    
    • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ
    
    • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε
    

    View Slide

23. Product Team - Visibility / Accessibility
    https://pipecd.dev/docs/user-guide/plan-preview
    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ
    
    • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ
    
    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌
    
    • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ
    
    • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ
    
    • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε
    Git di
    ff
    is enought? No.
    e.g. using remote Helm chart, Kustomize package, Terraform module...
    Reviewer needs more early feedback to merge PR with con
    fi
    dence
    Better to see dry-run result, terraform plan, deployment policy... on the PR
    then Git di
    ff
    is just showing the change of version number
    

    View Slide

24. Product Team - Visibility / Accessibility
    Terraform Deployment
    CloudRun Deployment
    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ
    
    • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ
    
    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌
    
    • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ
    
    • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ
    
    • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε
    Scale In Scale Out
    Rollout
    New Image
    Update
    

    Con
    fi
    g
    Deploy
    Infra
    Deploy
    Serverless
    Rollback
    Deploy
    
    
    Kubernetes
    Constant process for all operations
    

    View Slide

25. Product Team - Automation
    https://pipecd.dev/docs/user-guide/rolling-back-a-deployment
    • σϓϩΠͷΠϯύΫτΛࣗಈ෼ੳ
    
    • ϝτϦΫεɾϩάɾhttp
    
    • աڈͷσʔλɾcanary-baselineͷൺֱ
    
    • σϓϩΠதʹ໰୊͕ൃੜ͢ΔͱࣗಈϩʔϧόοΫ
    
    • Gitͱ࣮ࡍͷঢ়ଶͷCon
    fi
    guration DriftΛࣗಈݕ஌
    
    • ৽ίϯςφΠϝʔδɾHelm Chart͕ग़Δͱ

    ࣗಈσϓϩΠͷEventWatcher
    https://pipecd.dev/docs/user-guide/con
    fi
    guration-drift-detection
    

    View Slide

26. Product Team - Control / Flexibility
    • Piped AgentΛ޷͖ͳελΠϧͰ૊Έ߹ΘͤΔ͜ͱ͕Մೳ
    
    • 1 Piped AgentͰશͯ؅ཧύλʔϯ
    
    • ؀ڥຖʹઐ༻ͷPiped Agentύλʔϯ
    
    • ޷͖ͳ৔ॴʹΠϯετʔϧʢKubernetes podɺVMʹதɺFargateαʔϏε...ʣ
    
    • σϓϩΠઓུ͸ࣗ༝ʹ૊Έ߹Θͤɾఆ͕ٛՄೳ
    
    • Quick Sync
    
    • Progressive Sync (Canary, BlueGreen...)
    
    • νʔϜʹ߹͏ϓϩόΠμʔΛબ΂ΒΕΔ
    
    • Cloud Provider (GCP, AWS, Azure, Private Cloud)
    
    • Analysis Provider (Prometheus, Datadog, Stackdriver...)
    https://github.com/pipe-cd/examples/

    blob/master/kubernetes/canary/.pipe.yaml
    

    View Slide

27. PipeCD
    Platform Team͕Delivery InfrastructureΛఏڙͰ͖ɺ

    ϕετϓϥΫςΟεΛܧଓతʹ࠾༻ɾීٴͰ͖Δ
    Product Team͕CDͷߏஙɾӡ༻͕ෆཁʹͳΓɺҰճಋೖ͢Δ͚ͩͰɺ

    ࣗ෼ʹ߹͏σϦόϦख๏Λ࠾༻Ͱ͖ɺܧଓతʹվળͰ͖Δ
    ͜ΕͰɺ
    

    View Slide

28. ݱࡏͷPipeCD @ CyberAgent
    

    View Slide

29. νʔϜͱγεςϜͷߏ੒
    • Platform Team
    
    • PipeCDΛ։ൃ
    
    • ࣾ಺༻PipeCD Control-planeΛӡ༻
    
    • ֤Product Team
    
    • Piped AgentΛΠϯετʔϧ

    • ࣗ෼ʹ߹͏σϓϩΠϝϯτछྨΛ࠾༻
    Control-Plane
    https://pipecd.dev/docs/operator-manual/piped/installation
    https://github.com/pipe-cd/examples
    

    View Slide

30. ΞϓϦέʔγϣϯɾαʔϏε͕૿Ճத
    0
    175
    350
    525
    700
    2020/10 2020/12 2021/02 2021/04 2021/06 2021/8
    665
    Applications/Servicesʹୡ੒
    CyberAgent಺ʹPipeCDͰӡ༻͍ͯ͠Δ Kubernetes, Terraform, Lambda, CloudRun, Fargate... ͷΞϓϦέʔγϣϯɾαʔϏε਺
    

    View Slide

31. PipeCDͷࠓޙ
    

    View Slide

32. ֤࣠Λ͞ΒʹڧԽ
    Automation
    Visibility /
    Accessibility
    Control /
    

    Flexibility
    Management Security
    Platform Team
    Product Team
    

    View Slide

33. Visibility / Accessiblity
    • InsightsͰσϦόϦʔύϑΥʔϚϯεΛՄࢹԽ
    
    • Deployස౓ɺLead Time, ࣦഊ཰, MTTR...
    
    • ApplicationͷϦιʔεͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ
    
    • Terraform, CloudRun, ECS
    
    • Deployment StageͷϩάΛվળ
    
    • EventϦετը໘
    
    • Plan-Previewʹ΋ͬͱFeedbackΛ௥Ճ
    
    • Kubernetes validating webhook, dry-run݁Ռ,

    Terraform sentinelͳͲͷdeployment policy
    ֎ͷϢʔβʔ޲͚ͷPlayground؀ڥΛ४උதʂ

    https://play.pipecd.dev
    

    View Slide

34. Automation
    • LogͷσʔλͰσϓϩΠϝϯτΛࣗಈ෼ੳ
    
    • աڈͷMetricsσʔλͱͷൺֱͰࣗಈ෼ੳ
    
    • Primary/Baseline & CanaryͷMetricsσʔλͷൺֱͰࣗಈ෼ੳ
    

    View Slide

35. Control / Flexibility
    • ਂ͍Ϩϕϧͷݖݶ؅ཧͰ͖ΔACL
    
    • Piped AgentʹPlug-in ArchitectureΛ࠾༻Ͱɺ

    ಠࣗͷσϓϩΠϝϯτϩδοΫΛຒΊࠐΊΔΑ͏ʹ
    
    • Deployment ChainͰmulti-clusterͷΞϓϦέʔγϣϯ΍

    σϓϩΠͷॱ൪੍ޚ͕Ͱ͖ΔΑ͏ʹ
    

    View Slide

36. Deployment Chain
    Application X
    Region A
    Region B
    Region C
    ᶃ
    ᶄ
    ᶅ
    Application X
    Cluster A
    Cluster B
    Cluster C
    ᶃ
    ᶄ
    ᶅ
    Application X - Dev Env
    Application X - Stg Env
    Application X - Prod Env
    Application Infra
    Application X
    Application Y
    ੒ޭͳΒ͹࣍ʹਐΉ ੒ޭͳΒ͹࣍ʹਐΉ
    

    View Slide

37. Feature Status
    ۩ମతͳػೳͷঢ়ଶ͸ҎԼͷϖʔδͰ֬ೝͰ͖·͢
    https://pipecd.dev/docs/feature-status
    

    View Slide

38. Thank you
    PipeCD OSSͷBackendͱFrontendͷϑϧλΠϜɾ෭ۀʢ࣌ؒͷ੍ݶ͸ͳ͠ʣΛืू͍ͯ͠·͢

    ͝ڵຯ͕͋ΔํɺTwitterͷDMͳͲ͝࿈བྷ͍ͩ͘͞
    
    ʢืूͷ৘ใΛ֦ࢄ͓ͯ͠ئ͍͠·͢ʣ
    

    View Slide