Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PipeCD at CyberAgent

439d941a6e41920af9bf4d5bd5f97e3a?s=47 nghialv
September 03, 2021

PipeCD at CyberAgent

439d941a6e41920af9bf4d5bd5f97e3a?s=128

nghialv

September 03, 2021
Tweet

Transcript

  1. nghialv Developer Productivity Team, CyberAgent Sep 03, 2021 PipeCD @

    CyberAgent CI/CD Conference 2021 by CloudNative Days
  2. ࣗݾ঺հ @nghialv @nghialv2607 @nghialv Le Van Nghia - ΪΞ

  3. ࣗݾ঺հ - ৬ྺ @CyberAgent • PipeCDΛ։ൃɾӡ༻ - DPࣨ • Work

    fl ow Automation SystemΛ։ൃɾӡ༻ - OSSS • Feature Flags SystemΛ։ൃɾӡ༻ - AbemaTV • Prometheus & GrafanaͰMonitoring SystemΛߏஙɾӡ༻ - AbemaTV • DeploymentπʔϧΛ։ൃɾӡ༻ - AbemaTV • Microservicesɾج൫पΓ - AbemaTV
  4. ࠓ೔ͷ࿩͢಺༰ • CyberAgentͰCDͷ՝୊ • PipeCDͷ஀ੜ • ݱࡏͷPipeCD @ CyberAgent •

    PipeCDͷࠓޙ
  5. ຊ୊΁ೖΔલʹ

  6. CI/CD

  7. CI != CD

  8. CyberAgentͰCDͷ՝୊

  9. ϓϩμΫτͷߏ੒ ABEMA AWA WinTicket Game AI Studio ... • ଟ͘ͷϓϩμΫτ͕ଘࡏ͍ͯ͠Δ

    • ֤ϓϩμΫτͰ͸ࣗ෼ʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબ΂Δ Group • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾ಺Ҡಈͷ৔߹͸Մೳ
  10. CDͷ՝୊ Multi Cloud Product • σϦόϦͷ੒ख़౓͕ߴ͘ͳ͍νʔϜ͕ଘࡏ • खಈϦϦʔεͷνΣοΫϦετ͕େม • Canary,

    BlueGreenͳͲͷ҆શରࡦ͕ͳ͍ • ཧ༝͸ࣗ෼ͰCDͷߏஙͱӡ༻͕େมͱϦιʔε͕ͳ͍ • ౷ҰੑʢҰ؏ੑʣ͕௿͘ͳ͍ͬͯΔ • ࣾ಺ελϯμʔυΛ੒ཱ͢Δ͜ͱ͕೉͘͠ͳΔ • ࣾ಺ͷϕετϓϥΫςΟεͷීٴ͕େม • ؅ཧίετ͕ߴ͍ • Developer ExperienceʹӨڹͯ͠͠·͏ • Onboardingίετ͕͔͔Δ ٕज़ελοΫ͕ಉ͡Ͱ΋
 ϓϩμΫτຖʹCDγεςϜɾϓϩηε͕ҟͳΔ ಉ͡ϓϩμΫτͰ΋
 ෳ਺CDγεςϜɾϓϩηε͕ଘࡏ͢Δ Product A Product B CD System X CD System Y CD System X for GCP services CD System Y for AWS Services CD System Z for Infra
  11. ࣾ಺౷ҰͳCDγεςϜΛݕ౼ Delivery Infrastructure Product Team͸࠷େͳσϦόϦύϑΥʔϚ ϯεΛग़ͤͳ͕Βɺࣗ෼ʹ߹͏ٕज़ελοΫ ͱσϦόϦઓུΛબͿϑϦʔμϜ͕͋Δ Platform Team͸ॊೈͳDelivery InfrastructureΛఏڙͱ


    CDͷϕετϓϥΫςΟεΛීٴ Platform Team & Product Teamͷ྆ํʹϝϦοτΛग़ͤΔ Standards/Consistency vs Freedomͷྑ͍όϥϯεΛऔΕΔ Ͳ͏͢Ε͹ɺ
  12. Platform Team Management Security γεςϜͷӡ༻͸ίετ͕ߴ͍ͷ͔
 ϓϩμΫτಋೖͷεέʔϥϏϦςΟ ϓϩμΫτͷΫϨσϯγϟϧΛͲ͏؅ཧ͢Δ͔
 
 CDηΩϡϦςΟʔपΓͷϕετϓϥΫςΟεΛ
 ϓϩμΫτνʔϜ΁ීٴͰ͖Δ͔Ͳ͏͔

  13. Product Team Automation Visibility / Accessibility Control / 
 Flexibility

    νʔϜʹϑΟοτٕज़ελοΫɾσϦόϦख๏Λࣗ༝ʹબ͹ΕΔͷ͔
 
 νʔϜͷݖݶΛద੾ʹ؅ཧͰ͖Δͷ͔ ϦϦʔεϓϩηεͷதʹɺਓؒͷखಈλεΫ͕Ͳ͜·Ͱ࡟ݮͰ͖Δͷ͔
 
 ϦϦʔεத΋ޙ΋ܧଓతʹ໰୊Λݕ஌Ͱ͖Δͷ͔ ։ൃऀ΁े෼ͳϑΟʔυόοΫΛ଎ΊʹఏڙͰ͖Δͷ͔
 ໰୊͕͋Δ࣌ʹݪҼ΋ؚΉঢ়ଶΛͪΌΜͱݟ͑Δͷ͔
 ؅ཧऀ΁νʔϜͷσϦόϦʔύϑΥʔϚϯεΛͪΌΜͱݟͤΔͷ͔
  14. PipeCDͷ஀ੜ

  15. PipeCD

  16. PipeCD 🤍 OSS Thanks to the contributors of PipeCD! https://github.com/pipe-cd/pipe

    https://pipecd.dev https://pipecd.dev/docs ~1700 pull requests
  17. PipeCD 🤍 GitOps • GitOpsΛ࠾༻ • GitʹશͯͷCon fi gurationΛอଘ •

    શͯͷΦϖϨʔγϣϯ͕Git Pull Requestܦ༝Λߦ͑Δ • Πϯϑϥ͔ΒΞϓϦέʔγϣϯ·Ͱ౷ҰͳGitOps • Kubernetes • Terraform • CloudRun • AWS Lambda • ECS Fargate • ECS
  18. PipeCD͸ͲͷΑ͏ʹզʑͷ՝୊Λղܾ͢Δ͔ Automation Visibility / Accessibility Control / 
 Flexibility Management

    Security Platform Team Product Team
  19. Product B Platform Team - Management • ӡ༻ίετ͕௿͍ • શͯ͸γϯάϧͳγεςϜͷΈ͕ඞཁ

    • Control-planeͷΞʔΩςΫνϟ͕؆୯ • શͯͷίϯϙʔωϯτ͕εςʔτϨε • ετϨʔδ͸ϚωʔδυαʔϏεΛར༻͢Δ͜ͱ͕Մೳ • ϓϩμΫτଆ͸γϯάϧόΠφϦͷPiped AgentΛ
 ΠϯετʔϧͷΈͰɺશͯͷCDͷػೳΛར༻Ͱ͖Δ • ϓϩμΫτͷεέʔϥϏϦςΟ • ৽نͷϓϩμΫτͷ௥Ճ͕୯౬ • Ͳͷ؀ڥͰ΋ಈ͚Δઃܭ • ωοτϫʔΫ੍ݶͷPrivate Cloud͔ΒPublic Cloud·Ͱ • খن໛νʔϜ͔Βେن໛νʔϜ·Ͱ Piped Control Plane Firewall
 Friendly Outbout
 Requests
 Only Product A Piped Piped Piped
  20. Platform Team - Security • ϓϩμΫτνʔϜͷΫϨσϯγϟϧ͸֎ʹҰ੾ग़ͳ͍ઃܭ • Control-planeʹ΋อଘɾ؅ཧ͠ͳ͍ • GitOpsͰͷSecret؅ཧͷϕετϓϥΫςΟεͱͯ͠ͷbuilt-inػೳΛؚΉ

    • Piped Agent͕Ξ΢τό΢ϯυϦΫΤετͷΈΛߦ͏ͷͰɺެ։ϙʔτͳͲ͸ඞཁ͕ͳ͍
  21. Product Team - Visibility / Accessibility • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ

    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε https://pipecd.dev/docs/user-guide/triggering-a-deployment
  22. Product Team - Visibility / Accessibility https://pipecd.dev/docs/user-guide/application-live-state • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ •

    ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε
  23. Product Team - Visibility / Accessibility https://pipecd.dev/docs/user-guide/plan-preview • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ •

    ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε Git di ff is enought? No. e.g. using remote Helm chart, Kustomize package, Terraform module... Reviewer needs more early feedback to merge PR with con fi dence Better to see dry-run result, terraform plan, deployment policy... on the PR then Git di ff is just showing the change of version number
  24. Product Team - Visibility / Accessibility Terraform Deployment CloudRun Deployment

    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε Scale In Scale Out Rollout New Image Update 
 Con fi g Deploy Infra Deploy Serverless Rollback Deploy Kubernetes Constant process for all operations
  25. Product Team - Automation https://pipecd.dev/docs/user-guide/rolling-back-a-deployment • σϓϩΠͷΠϯύΫτΛࣗಈ෼ੳ • ϝτϦΫεɾϩάɾhttp •

    աڈͷσʔλɾcanary-baselineͷൺֱ • σϓϩΠதʹ໰୊͕ൃੜ͢ΔͱࣗಈϩʔϧόοΫ • Gitͱ࣮ࡍͷঢ়ଶͷCon fi guration DriftΛࣗಈݕ஌ • ৽ίϯςφΠϝʔδɾHelm Chart͕ग़Δͱ
 ࣗಈσϓϩΠͷEventWatcher https://pipecd.dev/docs/user-guide/con fi guration-drift-detection
  26. Product Team - Control / Flexibility • Piped AgentΛ޷͖ͳελΠϧͰ૊Έ߹ΘͤΔ͜ͱ͕Մೳ •

    1 Piped AgentͰશͯ؅ཧύλʔϯ • ؀ڥຖʹઐ༻ͷPiped Agentύλʔϯ • ޷͖ͳ৔ॴʹΠϯετʔϧʢKubernetes podɺVMʹதɺFargateαʔϏε...ʣ • σϓϩΠઓུ͸ࣗ༝ʹ૊Έ߹Θͤɾఆ͕ٛՄೳ • Quick Sync • Progressive Sync (Canary, BlueGreen...) • νʔϜʹ߹͏ϓϩόΠμʔΛબ΂ΒΕΔ • Cloud Provider (GCP, AWS, Azure, Private Cloud) • Analysis Provider (Prometheus, Datadog, Stackdriver...) https://github.com/pipe-cd/examples/
 blob/master/kubernetes/canary/.pipe.yaml
  27. PipeCD Platform Team͕Delivery InfrastructureΛఏڙͰ͖ɺ
 ϕετϓϥΫςΟεΛܧଓతʹ࠾༻ɾීٴͰ͖Δ Product Team͕CDͷߏஙɾӡ༻͕ෆཁʹͳΓɺҰճಋೖ͢Δ͚ͩͰɺ
 ࣗ෼ʹ߹͏σϦόϦख๏Λ࠾༻Ͱ͖ɺܧଓతʹվળͰ͖Δ ͜ΕͰɺ

  28. ݱࡏͷPipeCD @ CyberAgent

  29. νʔϜͱγεςϜͷߏ੒ • Platform Team • PipeCDΛ։ൃ • ࣾ಺༻PipeCD Control-planeΛӡ༻ •

    ֤Product Team • Piped AgentΛΠϯετʔϧ
 • ࣗ෼ʹ߹͏σϓϩΠϝϯτछྨΛ࠾༻ Control-Plane https://pipecd.dev/docs/operator-manual/piped/installation https://github.com/pipe-cd/examples
  30. ΞϓϦέʔγϣϯɾαʔϏε͕૿Ճத 0 175 350 525 700 2020/10 2020/12 2021/02 2021/04

    2021/06 2021/8 665 Applications/Servicesʹୡ੒ CyberAgent಺ʹPipeCDͰӡ༻͍ͯ͠Δ Kubernetes, Terraform, Lambda, CloudRun, Fargate... ͷΞϓϦέʔγϣϯɾαʔϏε਺
  31. PipeCDͷࠓޙ

  32. ֤࣠Λ͞ΒʹڧԽ Automation Visibility / Accessibility Control / 
 Flexibility Management

    Security Platform Team Product Team
  33. Visibility / Accessiblity • InsightsͰσϦόϦʔύϑΥʔϚϯεΛՄࢹԽ • Deployස౓ɺLead Time, ࣦഊ཰, MTTR...

    • ApplicationͷϦιʔεͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • Terraform, CloudRun, ECS • Deployment StageͷϩάΛվળ • EventϦετը໘ • Plan-Previewʹ΋ͬͱFeedbackΛ௥Ճ • Kubernetes validating webhook, dry-run݁Ռ,
 Terraform sentinelͳͲͷdeployment policy ֎ͷϢʔβʔ޲͚ͷPlayground؀ڥΛ४උதʂ
 https://play.pipecd.dev
  34. Automation • LogͷσʔλͰσϓϩΠϝϯτΛࣗಈ෼ੳ • աڈͷMetricsσʔλͱͷൺֱͰࣗಈ෼ੳ • Primary/Baseline & CanaryͷMetricsσʔλͷൺֱͰࣗಈ෼ੳ

  35. Control / Flexibility • ਂ͍Ϩϕϧͷݖݶ؅ཧͰ͖ΔACL • Piped AgentʹPlug-in ArchitectureΛ࠾༻Ͱɺ
 ಠࣗͷσϓϩΠϝϯτϩδοΫΛຒΊࠐΊΔΑ͏ʹ

    • Deployment ChainͰmulti-clusterͷΞϓϦέʔγϣϯ΍
 σϓϩΠͷॱ൪੍ޚ͕Ͱ͖ΔΑ͏ʹ
  36. Deployment Chain Application X Region A Region B Region C

    ᶃ ᶄ ᶅ Application X Cluster A Cluster B Cluster C ᶃ ᶄ ᶅ Application X - Dev Env Application X - Stg Env Application X - Prod Env Application Infra Application X Application Y ੒ޭͳΒ͹࣍ʹਐΉ ੒ޭͳΒ͹࣍ʹਐΉ
  37. Feature Status ۩ମతͳػೳͷঢ়ଶ͸ҎԼͷϖʔδͰ֬ೝͰ͖·͢ https://pipecd.dev/docs/feature-status

  38. Thank you PipeCD OSSͷBackendͱFrontendͷϑϧλΠϜɾ෭ۀʢ࣌ؒͷ੍ݶ͸ͳ͠ʣΛืू͍ͯ͠·͢
 ͝ڵຯ͕͋ΔํɺTwitterͷDMͳͲ͝࿈བྷ͍ͩ͘͞ ʢืूͷ৘ใΛ֦ࢄ͓ͯ͠ئ͍͠·͢ʣ