PipeCD at CyberAgent - Speaker Deck

Slide 1

Slide 1 text

nghialv Developer Productivity Team, CyberAgent Sep 03, 2021 PipeCD @ CyberAgent CI/CD Conference 2021 by CloudNative Days

Slide 2

Slide 2 text

ࣗݾ঺հ @nghialv @nghialv2607 @nghialv Le Van Nghia - ΪΞ

Slide 3

Slide 3 text

ࣗݾ঺հ - ৬ྺ @CyberAgent • PipeCDΛ։ൃɾӡ༻ - DPࣨ • Work fl ow Automation SystemΛ։ൃɾӡ༻ - OSSS • Feature Flags SystemΛ։ൃɾӡ༻ - AbemaTV • Prometheus & GrafanaͰMonitoring SystemΛߏஙɾӡ༻ - AbemaTV • DeploymentπʔϧΛ։ൃɾӡ༻ - AbemaTV • Microservicesɾج൫पΓ - AbemaTV

Slide 4

Slide 4 text

ࠓ೔ͷ࿩͢಺༰ • CyberAgentͰCDͷ՝୊ • PipeCDͷ஀ੜ • ݱࡏͷPipeCD @ CyberAgent • PipeCDͷࠓޙ

Slide 5

Slide 5 text

ຊ୊΁ೖΔલʹ

Slide 6

Slide 6 text

CI/CD

Slide 7

Slide 7 text

CI != CD

Slide 8

Slide 8 text

CyberAgentͰCDͷ՝୊

Slide 9

Slide 9 text

ϓϩμΫτͷߏ੒ ABEMA AWA WinTicket Game AI Studio ... • ଟ͘ͷϓϩμΫτ͕ଘࡏ͍ͯ͠Δ • ֤ϓϩμΫτͰ͸ࣗ෼ʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબ΂Δ Group • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾ಺Ҡಈͷ৔߹͸Մೳ

Slide 10

Slide 10 text

CDͷ՝୊ Multi Cloud Product • σϦόϦͷ੒ख़౓͕ߴ͘ͳ͍νʔϜ͕ଘࡏ • खಈϦϦʔεͷνΣοΫϦετ͕େม • Canary, BlueGreenͳͲͷ҆શରࡦ͕ͳ͍ • ཧ༝͸ࣗ෼ͰCDͷߏஙͱӡ༻͕େมͱϦιʔε͕ͳ͍ • ౷ҰੑʢҰ؏ੑʣ͕௿͘ͳ͍ͬͯΔ • ࣾ಺ελϯμʔυΛ੒ཱ͢Δ͜ͱ͕೉͘͠ͳΔ • ࣾ಺ͷϕετϓϥΫςΟεͷීٴ͕େม • ؅ཧίετ͕ߴ͍ • Developer ExperienceʹӨڹͯ͠͠·͏ • Onboardingίετ͕͔͔Δ ٕज़ελοΫ͕ಉ͡Ͱ΋  ϓϩμΫτຖʹCDγεςϜɾϓϩηε͕ҟͳΔ ಉ͡ϓϩμΫτͰ΋  ෳ਺CDγεςϜɾϓϩηε͕ଘࡏ͢Δ Product A Product B CD System X CD System Y CD System X for GCP services CD System Y for AWS Services CD System Z for Infra

Slide 11

Slide 11 text

ࣾ಺౷ҰͳCDγεςϜΛݕ౼ Delivery Infrastructure Product Team͸࠷େͳσϦόϦύϑΥʔϚ ϯεΛग़ͤͳ͕Βɺࣗ෼ʹ߹͏ٕज़ελοΫ ͱσϦόϦઓུΛબͿϑϦʔμϜ͕͋Δ Platform Team͸ॊೈͳDelivery InfrastructureΛఏڙͱ  CDͷϕετϓϥΫςΟεΛීٴ Platform Team & Product Teamͷ྆ํʹϝϦοτΛग़ͤΔ Standards/Consistency vs Freedomͷྑ͍όϥϯεΛऔΕΔ Ͳ͏͢Ε͹ɺ

Slide 12

Slide 12 text

Platform Team Management Security γεςϜͷӡ༻͸ίετ͕ߴ͍ͷ͔  ϓϩμΫτಋೖͷεέʔϥϏϦςΟ ϓϩμΫτͷΫϨσϯγϟϧΛͲ͏؅ཧ͢Δ͔    CDηΩϡϦςΟʔपΓͷϕετϓϥΫςΟεΛ  ϓϩμΫτνʔϜ΁ීٴͰ͖Δ͔Ͳ͏͔

Slide 13

Slide 13 text

Product Team Automation Visibility / Accessibility Control /   Flexibility νʔϜʹϑΟοτٕज़ελοΫɾσϦόϦख๏Λࣗ༝ʹબ͹ΕΔͷ͔    νʔϜͷݖݶΛద੾ʹ؅ཧͰ͖Δͷ͔ ϦϦʔεϓϩηεͷதʹɺਓؒͷखಈλεΫ͕Ͳ͜·Ͱ࡟ݮͰ͖Δͷ͔    ϦϦʔεத΋ޙ΋ܧଓతʹ໰୊Λݕ஌Ͱ͖Δͷ͔ ։ൃऀ΁े෼ͳϑΟʔυόοΫΛ଎ΊʹఏڙͰ͖Δͷ͔  ໰୊͕͋Δ࣌ʹݪҼ΋ؚΉঢ়ଶΛͪΌΜͱݟ͑Δͷ͔  ؅ཧऀ΁νʔϜͷσϦόϦʔύϑΥʔϚϯεΛͪΌΜͱݟͤΔͷ͔

Slide 14

Slide 14 text

PipeCDͷ஀ੜ

Slide 15

Slide 15 text

PipeCD

Slide 16

Slide 16 text

PipeCD 🤍 OSS Thanks to the contributors of PipeCD! https://github.com/pipe-cd/pipe https://pipecd.dev https://pipecd.dev/docs ~1700 pull requests

Slide 17

Slide 17 text

PipeCD 🤍 GitOps • GitOpsΛ࠾༻ • GitʹશͯͷCon fi gurationΛอଘ • શͯͷΦϖϨʔγϣϯ͕Git Pull Requestܦ༝Λߦ͑Δ • Πϯϑϥ͔ΒΞϓϦέʔγϣϯ·Ͱ౷ҰͳGitOps • Kubernetes • Terraform • CloudRun • AWS Lambda • ECS Fargate • ECS

Slide 18

Slide 18 text

PipeCD͸ͲͷΑ͏ʹզʑͷ՝୊Λղܾ͢Δ͔ Automation Visibility / Accessibility Control /   Flexibility Management Security Platform Team Product Team

Slide 19

Slide 19 text

Product B Platform Team - Management • ӡ༻ίετ͕௿͍ • શͯ͸γϯάϧͳγεςϜͷΈ͕ඞཁ • Control-planeͷΞʔΩςΫνϟ͕؆୯ • શͯͷίϯϙʔωϯτ͕εςʔτϨε • ετϨʔδ͸ϚωʔδυαʔϏεΛར༻͢Δ͜ͱ͕Մೳ • ϓϩμΫτଆ͸γϯάϧόΠφϦͷPiped AgentΛ  ΠϯετʔϧͷΈͰɺશͯͷCDͷػೳΛར༻Ͱ͖Δ • ϓϩμΫτͷεέʔϥϏϦςΟ • ৽نͷϓϩμΫτͷ௥Ճ͕୯౬ • Ͳͷ؀ڥͰ΋ಈ͚Δઃܭ • ωοτϫʔΫ੍ݶͷPrivate Cloud͔ΒPublic Cloud·Ͱ • খن໛νʔϜ͔Βେن໛νʔϜ·Ͱ Piped Control Plane Firewall  Friendly Outbout  Requests  Only Product A Piped Piped Piped

Slide 20

Slide 20 text

Platform Team - Security • ϓϩμΫτνʔϜͷΫϨσϯγϟϧ͸֎ʹҰ੾ग़ͳ͍ઃܭ • Control-planeʹ΋อଘɾ؅ཧ͠ͳ͍ • GitOpsͰͷSecret؅ཧͷϕετϓϥΫςΟεͱͯ͠ͷbuilt-inػೳΛؚΉ • Piped Agent͕Ξ΢τό΢ϯυϦΫΤετͷΈΛߦ͏ͷͰɺެ։ϙʔτͳͲ͸ඞཁ͕ͳ͍

Slide 21

Slide 21 text

Product Team - Visibility / Accessibility • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε https://pipecd.dev/docs/user-guide/triggering-a-deployment

Slide 22

Slide 22 text

Product Team - Visibility / Accessibility https://pipecd.dev/docs/user-guide/application-live-state • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε

Slide 23

Slide 23 text

Product Team - Visibility / Accessibility https://pipecd.dev/docs/user-guide/plan-preview • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε Git di ff is enought? No. e.g. using remote Helm chart, Kustomize package, Terraform module... Reviewer needs more early feedback to merge PR with con fi dence Better to see dry-run result, terraform plan, deployment policy... on the PR then Git di ff is just showing the change of version number

Slide 24

Slide 24 text

Product Team - Visibility / Accessibility Terraform Deployment CloudRun Deployment • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌ • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε Scale In Scale Out Rollout New Image Update   Con fi g Deploy Infra Deploy Serverless Rollback Deploy Kubernetes Constant process for all operations

Slide 25

Slide 25 text

Product Team - Automation https://pipecd.dev/docs/user-guide/rolling-back-a-deployment • σϓϩΠͷΠϯύΫτΛࣗಈ෼ੳ • ϝτϦΫεɾϩάɾhttp • աڈͷσʔλɾcanary-baselineͷൺֱ • σϓϩΠதʹ໰୊͕ൃੜ͢ΔͱࣗಈϩʔϧόοΫ • Gitͱ࣮ࡍͷঢ়ଶͷCon fi guration DriftΛࣗಈݕ஌ • ৽ίϯςφΠϝʔδɾHelm Chart͕ग़Δͱ  ࣗಈσϓϩΠͷEventWatcher https://pipecd.dev/docs/user-guide/con fi guration-drift-detection

Slide 26

Slide 26 text

Product Team - Control / Flexibility • Piped AgentΛ޷͖ͳελΠϧͰ૊Έ߹ΘͤΔ͜ͱ͕Մೳ • 1 Piped AgentͰશͯ؅ཧύλʔϯ • ؀ڥຖʹઐ༻ͷPiped Agentύλʔϯ • ޷͖ͳ৔ॴʹΠϯετʔϧʢKubernetes podɺVMʹதɺFargateαʔϏε...ʣ • σϓϩΠઓུ͸ࣗ༝ʹ૊Έ߹Θͤɾఆ͕ٛՄೳ • Quick Sync • Progressive Sync (Canary, BlueGreen...) • νʔϜʹ߹͏ϓϩόΠμʔΛબ΂ΒΕΔ • Cloud Provider (GCP, AWS, Azure, Private Cloud) • Analysis Provider (Prometheus, Datadog, Stackdriver...) https://github.com/pipe-cd/examples/  blob/master/kubernetes/canary/.pipe.yaml

Slide 27

Slide 27 text

PipeCD Platform Team͕Delivery InfrastructureΛఏڙͰ͖ɺ  ϕετϓϥΫςΟεΛܧଓతʹ࠾༻ɾීٴͰ͖Δ Product Team͕CDͷߏஙɾӡ༻͕ෆཁʹͳΓɺҰճಋೖ͢Δ͚ͩͰɺ  ࣗ෼ʹ߹͏σϦόϦख๏Λ࠾༻Ͱ͖ɺܧଓతʹվળͰ͖Δ ͜ΕͰɺ

Slide 28

Slide 28 text

ݱࡏͷPipeCD @ CyberAgent

Slide 29

Slide 29 text

νʔϜͱγεςϜͷߏ੒ • Platform Team • PipeCDΛ։ൃ • ࣾ಺༻PipeCD Control-planeΛӡ༻ • ֤Product Team • Piped AgentΛΠϯετʔϧ  • ࣗ෼ʹ߹͏σϓϩΠϝϯτछྨΛ࠾༻ Control-Plane https://pipecd.dev/docs/operator-manual/piped/installation https://github.com/pipe-cd/examples

Slide 30

Slide 30 text

ΞϓϦέʔγϣϯɾαʔϏε͕૿Ճத 0 175 350 525 700 2020/10 2020/12 2021/02 2021/04 2021/06 2021/8 665 Applications/Servicesʹୡ੒ CyberAgent಺ʹPipeCDͰӡ༻͍ͯ͠Δ Kubernetes, Terraform, Lambda, CloudRun, Fargate... ͷΞϓϦέʔγϣϯɾαʔϏε਺

Slide 31

Slide 31 text

PipeCDͷࠓޙ

Slide 32

Slide 32 text

֤࣠Λ͞ΒʹڧԽ Automation Visibility / Accessibility Control /   Flexibility Management Security Platform Team Product Team

Slide 33

Slide 33 text

Visibility / Accessiblity • InsightsͰσϦόϦʔύϑΥʔϚϯεΛՄࢹԽ • Deployස౓ɺLead Time, ࣦഊ཰, MTTR... • ApplicationͷϦιʔεͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • Terraform, CloudRun, ECS • Deployment StageͷϩάΛվળ • EventϦετը໘ • Plan-Previewʹ΋ͬͱFeedbackΛ௥Ճ • Kubernetes validating webhook, dry-run݁Ռ,  Terraform sentinelͳͲͷdeployment policy ֎ͷϢʔβʔ޲͚ͷPlayground؀ڥΛ४උதʂ  https://play.pipecd.dev

Slide 34

Slide 34 text

Automation • LogͷσʔλͰσϓϩΠϝϯτΛࣗಈ෼ੳ • աڈͷMetricsσʔλͱͷൺֱͰࣗಈ෼ੳ • Primary/Baseline & CanaryͷMetricsσʔλͷൺֱͰࣗಈ෼ੳ

Slide 35

Slide 35 text

Control / Flexibility • ਂ͍Ϩϕϧͷݖݶ؅ཧͰ͖ΔACL • Piped AgentʹPlug-in ArchitectureΛ࠾༻Ͱɺ  ಠࣗͷσϓϩΠϝϯτϩδοΫΛຒΊࠐΊΔΑ͏ʹ • Deployment ChainͰmulti-clusterͷΞϓϦέʔγϣϯ΍  σϓϩΠͷॱ൪੍ޚ͕Ͱ͖ΔΑ͏ʹ

Slide 36

Slide 36 text

Deployment Chain Application X Region A Region B Region C ᶃ ᶄ ᶅ Application X Cluster A Cluster B Cluster C ᶃ ᶄ ᶅ Application X - Dev Env Application X - Stg Env Application X - Prod Env Application Infra Application X Application Y ੒ޭͳΒ͹࣍ʹਐΉ ੒ޭͳΒ͹࣍ʹਐΉ

Slide 37

Slide 37 text

Feature Status ۩ମతͳػೳͷঢ়ଶ͸ҎԼͷϖʔδͰ֬ೝͰ͖·͢ https://pipecd.dev/docs/feature-status

Slide 38

Slide 38 text

Thank you PipeCD OSSͷBackendͱFrontendͷϑϧλΠϜɾ෭ۀʢ࣌ؒͷ੍ݶ͸ͳ͠ʣΛืू͍ͯ͠·͢  ͝ڵຯ͕͋ΔํɺTwitterͷDMͳͲ͝࿈བྷ͍ͩ͘͞ ʢืूͷ৘ใΛ֦ࢄ͓ͯ͠ئ͍͠·͢ʣ