$30 off During Our Annual Pro Sale. View Details »

PipeCD at CyberAgent

nghialv
September 03, 2021

PipeCD at CyberAgent

nghialv

September 03, 2021
Tweet

More Decks by nghialv

Other Decks in Technology

Transcript

  1. nghialv
    Developer Productivity Team, CyberAgent
    Sep 03, 2021
    PipeCD @ CyberAgent
    CI/CD Conference 2021 by CloudNative Days

    View Slide

  2. ࣗݾ঺հ
    @nghialv
    @nghialv2607
    @nghialv
    Le Van Nghia - ΪΞ

    View Slide

  3. ࣗݾ঺հ - ৬ྺ @CyberAgent
    • PipeCDΛ։ൃɾӡ༻ - DPࣨ

    • Work
    fl
    ow Automation SystemΛ։ൃɾӡ༻ - OSSS

    • Feature Flags SystemΛ։ൃɾӡ༻ - AbemaTV

    • Prometheus & GrafanaͰMonitoring SystemΛߏஙɾӡ༻ - AbemaTV

    • DeploymentπʔϧΛ։ൃɾӡ༻ - AbemaTV

    • Microservicesɾج൫पΓ - AbemaTV

    View Slide

  4. ࠓ೔ͷ࿩͢಺༰
    • CyberAgentͰCDͷ՝୊

    • PipeCDͷ஀ੜ

    • ݱࡏͷPipeCD @ CyberAgent

    • PipeCDͷࠓޙ

    View Slide

  5. ຊ୊΁ೖΔલʹ

    View Slide

  6. CI/CD

    View Slide

  7. CI != CD

    View Slide

  8. CyberAgentͰCDͷ՝୊

    View Slide

  9. ϓϩμΫτͷߏ੒
    ABEMA AWA WinTicket Game AI Studio
    ...
    • ଟ͘ͷϓϩμΫτ͕ଘࡏ͍ͯ͠Δ
    • ֤ϓϩμΫτͰ͸ࣗ෼ʹϑΟοτ͢Δٕज़ελοΫΛࣗ༝ʹબ΂Δ
    Group
    • ֤ϓϩμΫτͷΤϯδχΞνʔϜ͕ҟͳΔ͕ɺࣾ಺Ҡಈͷ৔߹͸Մೳ

    View Slide

  10. CDͷ՝୊
    Multi Cloud Product
    • σϦόϦͷ੒ख़౓͕ߴ͘ͳ͍νʔϜ͕ଘࡏ

    • खಈϦϦʔεͷνΣοΫϦετ͕େม

    • Canary, BlueGreenͳͲͷ҆શରࡦ͕ͳ͍

    • ཧ༝͸ࣗ෼ͰCDͷߏஙͱӡ༻͕େมͱϦιʔε͕ͳ͍
    • ౷ҰੑʢҰ؏ੑʣ͕௿͘ͳ͍ͬͯΔ

    • ࣾ಺ελϯμʔυΛ੒ཱ͢Δ͜ͱ͕೉͘͠ͳΔ

    • ࣾ಺ͷϕετϓϥΫςΟεͷීٴ͕େม

    • ؅ཧίετ͕ߴ͍

    • Developer ExperienceʹӨڹͯ͠͠·͏

    • Onboardingίετ͕͔͔Δ
    ٕज़ελοΫ͕ಉ͡Ͱ΋

    ϓϩμΫτຖʹCDγεςϜɾϓϩηε͕ҟͳΔ
    ಉ͡ϓϩμΫτͰ΋

    ෳ਺CDγεςϜɾϓϩηε͕ଘࡏ͢Δ
    Product A Product B
    CD System X CD System Y
    CD System X for GCP services
    CD System Y for AWS Services
    CD System Z for Infra

    View Slide

  11. ࣾ಺౷ҰͳCDγεςϜΛݕ౼
    Delivery Infrastructure
    Product Team͸࠷େͳσϦόϦύϑΥʔϚ
    ϯεΛग़ͤͳ͕Βɺࣗ෼ʹ߹͏ٕज़ελοΫ
    ͱσϦόϦઓུΛબͿϑϦʔμϜ͕͋Δ
    Platform Team͸ॊೈͳDelivery InfrastructureΛఏڙͱ

    CDͷϕετϓϥΫςΟεΛීٴ
    Platform Team & Product Teamͷ྆ํʹϝϦοτΛग़ͤΔ
    Standards/Consistency vs Freedomͷྑ͍όϥϯεΛऔΕΔ
    Ͳ͏͢Ε͹ɺ

    View Slide

  12. Platform Team
    Management
    Security
    γεςϜͷӡ༻͸ίετ͕ߴ͍ͷ͔

    ϓϩμΫτಋೖͷεέʔϥϏϦςΟ
    ϓϩμΫτͷΫϨσϯγϟϧΛͲ͏؅ཧ͢Δ͔


    CDηΩϡϦςΟʔपΓͷϕετϓϥΫςΟεΛ

    ϓϩμΫτνʔϜ΁ීٴͰ͖Δ͔Ͳ͏͔

    View Slide

  13. Product Team
    Automation
    Visibility /
    Accessibility
    Control /

    Flexibility
    νʔϜʹϑΟοτٕज़ελοΫɾσϦόϦख๏Λࣗ༝ʹબ͹ΕΔͷ͔


    νʔϜͷݖݶΛద੾ʹ؅ཧͰ͖Δͷ͔
    ϦϦʔεϓϩηεͷதʹɺਓؒͷखಈλεΫ͕Ͳ͜·Ͱ࡟ݮͰ͖Δͷ͔


    ϦϦʔεத΋ޙ΋ܧଓతʹ໰୊Λݕ஌Ͱ͖Δͷ͔
    ։ൃऀ΁े෼ͳϑΟʔυόοΫΛ଎ΊʹఏڙͰ͖Δͷ͔

    ໰୊͕͋Δ࣌ʹݪҼ΋ؚΉঢ়ଶΛͪΌΜͱݟ͑Δͷ͔

    ؅ཧऀ΁νʔϜͷσϦόϦʔύϑΥʔϚϯεΛͪΌΜͱݟͤΔͷ͔

    View Slide

  14. PipeCDͷ஀ੜ

    View Slide

  15. PipeCD

    View Slide

  16. PipeCD 🤍 OSS
    Thanks to the contributors of PipeCD!
    https://github.com/pipe-cd/pipe
    https://pipecd.dev
    https://pipecd.dev/docs
    ~1700 pull requests

    View Slide

  17. PipeCD 🤍 GitOps
    • GitOpsΛ࠾༻

    • GitʹશͯͷCon
    fi
    gurationΛอଘ

    • શͯͷΦϖϨʔγϣϯ͕Git Pull Requestܦ༝Λߦ͑Δ

    • Πϯϑϥ͔ΒΞϓϦέʔγϣϯ·Ͱ౷ҰͳGitOps

    • Kubernetes

    • Terraform

    • CloudRun

    • AWS Lambda

    • ECS Fargate

    • ECS

    View Slide

  18. PipeCD͸ͲͷΑ͏ʹզʑͷ՝୊Λղܾ͢Δ͔
    Automation
    Visibility /
    Accessibility
    Control /

    Flexibility
    Management Security
    Platform Team
    Product Team

    View Slide

  19. Product B
    Platform Team - Management
    • ӡ༻ίετ͕௿͍

    • શͯ͸γϯάϧͳγεςϜͷΈ͕ඞཁ

    • Control-planeͷΞʔΩςΫνϟ͕؆୯

    • શͯͷίϯϙʔωϯτ͕εςʔτϨε

    • ετϨʔδ͸ϚωʔδυαʔϏεΛར༻͢Δ͜ͱ͕Մೳ

    • ϓϩμΫτଆ͸γϯάϧόΠφϦͷPiped AgentΛ

    ΠϯετʔϧͷΈͰɺશͯͷCDͷػೳΛར༻Ͱ͖Δ

    • ϓϩμΫτͷεέʔϥϏϦςΟ

    • ৽نͷϓϩμΫτͷ௥Ճ͕୯౬

    • Ͳͷ؀ڥͰ΋ಈ͚Δઃܭ

    • ωοτϫʔΫ੍ݶͷPrivate Cloud͔ΒPublic Cloud·Ͱ

    • খن໛νʔϜ͔Βେن໛νʔϜ·Ͱ
    Piped
    Control Plane
    Firewall

    Friendly
    Outbout

    Requests

    Only
    Product A
    Piped
    Piped
    Piped

    View Slide

  20. Platform Team - Security
    • ϓϩμΫτνʔϜͷΫϨσϯγϟϧ͸֎ʹҰ੾ग़ͳ͍ઃܭ

    • Control-planeʹ΋อଘɾ؅ཧ͠ͳ͍

    • GitOpsͰͷSecret؅ཧͷϕετϓϥΫςΟεͱͯ͠ͷbuilt-inػೳΛؚΉ

    • Piped Agent͕Ξ΢τό΢ϯυϦΫΤετͷΈΛߦ͏ͷͰɺެ։ϙʔτͳͲ͸ඞཁ͕ͳ͍

    View Slide

  21. Product Team - Visibility / Accessibility
    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ

    • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ

    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌

    • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ

    • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ

    • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε https://pipecd.dev/docs/user-guide/triggering-a-deployment

    View Slide

  22. Product Team - Visibility / Accessibility
    https://pipecd.dev/docs/user-guide/application-live-state
    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ

    • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ

    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌

    • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ

    • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ

    • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε

    View Slide

  23. Product Team - Visibility / Accessibility
    https://pipecd.dev/docs/user-guide/plan-preview
    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ

    • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ

    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌

    • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ

    • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ

    • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε
    Git di
    ff
    is enought? No.
    e.g. using remote Helm chart, Kustomize package, Terraform module...
    Reviewer needs more early feedback to merge PR with con
    fi
    dence
    Better to see dry-run result, terraform plan, deployment policy... on the PR
    then Git di
    ff
    is just showing the change of version number

    View Slide

  24. Product Team - Visibility / Accessibility
    Terraform Deployment
    CloudRun Deployment
    • UIͰσϓϩΠϝϯτͷঢ়ଶɾϓϩηεͷ֬ೝ͕Մೳ

    • ΞϓϦέʔγϣϯͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ

    • SlackͳͲ΁Πϕϯτ΍ΞϥʔτͳͲΛ௨஌

    • InsightsͰσϦόϦύϑΥʔϚϯεΛՄࢹԽ

    • PlanPreviewͰPRͷ࣌఺ͰૣظϑΟʔυόοΫΛఏڙ

    • ͲΜͳ࣌ʹ΋୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε
    Scale In Scale Out
    Rollout
    New Image
    Update

    Con
    fi
    g
    Deploy
    Infra
    Deploy
    Serverless
    Rollback
    Deploy


    Kubernetes
    Constant process for all operations

    View Slide

  25. Product Team - Automation
    https://pipecd.dev/docs/user-guide/rolling-back-a-deployment
    • σϓϩΠͷΠϯύΫτΛࣗಈ෼ੳ

    • ϝτϦΫεɾϩάɾhttp

    • աڈͷσʔλɾcanary-baselineͷൺֱ

    • σϓϩΠதʹ໰୊͕ൃੜ͢ΔͱࣗಈϩʔϧόοΫ

    • Gitͱ࣮ࡍͷঢ়ଶͷCon
    fi
    guration DriftΛࣗಈݕ஌

    • ৽ίϯςφΠϝʔδɾHelm Chart͕ग़Δͱ

    ࣗಈσϓϩΠͷEventWatcher
    https://pipecd.dev/docs/user-guide/con
    fi
    guration-drift-detection

    View Slide

  26. Product Team - Control / Flexibility
    • Piped AgentΛ޷͖ͳελΠϧͰ૊Έ߹ΘͤΔ͜ͱ͕Մೳ

    • 1 Piped AgentͰશͯ؅ཧύλʔϯ

    • ؀ڥຖʹઐ༻ͷPiped Agentύλʔϯ

    • ޷͖ͳ৔ॴʹΠϯετʔϧʢKubernetes podɺVMʹதɺFargateαʔϏε...ʣ

    • σϓϩΠઓུ͸ࣗ༝ʹ૊Έ߹Θͤɾఆ͕ٛՄೳ

    • Quick Sync

    • Progressive Sync (Canary, BlueGreen...)

    • νʔϜʹ߹͏ϓϩόΠμʔΛબ΂ΒΕΔ

    • Cloud Provider (GCP, AWS, Azure, Private Cloud)

    • Analysis Provider (Prometheus, Datadog, Stackdriver...)
    https://github.com/pipe-cd/examples/

    blob/master/kubernetes/canary/.pipe.yaml

    View Slide

  27. PipeCD
    Platform Team͕Delivery InfrastructureΛఏڙͰ͖ɺ

    ϕετϓϥΫςΟεΛܧଓతʹ࠾༻ɾීٴͰ͖Δ
    Product Team͕CDͷߏஙɾӡ༻͕ෆཁʹͳΓɺҰճಋೖ͢Δ͚ͩͰɺ

    ࣗ෼ʹ߹͏σϦόϦख๏Λ࠾༻Ͱ͖ɺܧଓతʹվળͰ͖Δ
    ͜ΕͰɺ

    View Slide

  28. ݱࡏͷPipeCD @ CyberAgent

    View Slide

  29. νʔϜͱγεςϜͷߏ੒
    • Platform Team

    • PipeCDΛ։ൃ

    • ࣾ಺༻PipeCD Control-planeΛӡ༻

    • ֤Product Team

    • Piped AgentΛΠϯετʔϧ

    • ࣗ෼ʹ߹͏σϓϩΠϝϯτछྨΛ࠾༻
    Control-Plane
    https://pipecd.dev/docs/operator-manual/piped/installation
    https://github.com/pipe-cd/examples

    View Slide

  30. ΞϓϦέʔγϣϯɾαʔϏε͕૿Ճத
    0
    175
    350
    525
    700
    2020/10 2020/12 2021/02 2021/04 2021/06 2021/8
    665
    Applications/Servicesʹୡ੒
    CyberAgent಺ʹPipeCDͰӡ༻͍ͯ͠Δ Kubernetes, Terraform, Lambda, CloudRun, Fargate... ͷΞϓϦέʔγϣϯɾαʔϏε਺

    View Slide

  31. PipeCDͷࠓޙ

    View Slide

  32. ֤࣠Λ͞ΒʹڧԽ
    Automation
    Visibility /
    Accessibility
    Control /

    Flexibility
    Management Security
    Platform Team
    Product Team

    View Slide

  33. Visibility / Accessiblity
    • InsightsͰσϦόϦʔύϑΥʔϚϯεΛՄࢹԽ

    • Deployස౓ɺLead Time, ࣦഊ཰, MTTR...

    • ApplicationͷϦιʔεͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ

    • Terraform, CloudRun, ECS

    • Deployment StageͷϩάΛվળ

    • EventϦετը໘

    • Plan-Previewʹ΋ͬͱFeedbackΛ௥Ճ

    • Kubernetes validating webhook, dry-run݁Ռ,

    Terraform sentinelͳͲͷdeployment policy
    ֎ͷϢʔβʔ޲͚ͷPlayground؀ڥΛ४උதʂ

    https://play.pipecd.dev

    View Slide

  34. Automation
    • LogͷσʔλͰσϓϩΠϝϯτΛࣗಈ෼ੳ

    • աڈͷMetricsσʔλͱͷൺֱͰࣗಈ෼ੳ

    • Primary/Baseline & CanaryͷMetricsσʔλͷൺֱͰࣗಈ෼ੳ

    View Slide

  35. Control / Flexibility
    • ਂ͍Ϩϕϧͷݖݶ؅ཧͰ͖ΔACL

    • Piped AgentʹPlug-in ArchitectureΛ࠾༻Ͱɺ

    ಠࣗͷσϓϩΠϝϯτϩδοΫΛຒΊࠐΊΔΑ͏ʹ

    • Deployment ChainͰmulti-clusterͷΞϓϦέʔγϣϯ΍

    σϓϩΠͷॱ൪੍ޚ͕Ͱ͖ΔΑ͏ʹ

    View Slide

  36. Deployment Chain
    Application X
    Region A
    Region B
    Region C



    Application X
    Cluster A
    Cluster B
    Cluster C



    Application X - Dev Env
    Application X - Stg Env
    Application X - Prod Env
    Application Infra
    Application X
    Application Y
    ੒ޭͳΒ͹࣍ʹਐΉ ੒ޭͳΒ͹࣍ʹਐΉ

    View Slide

  37. Feature Status
    ۩ମతͳػೳͷঢ়ଶ͸ҎԼͷϖʔδͰ֬ೝͰ͖·͢
    https://pipecd.dev/docs/feature-status

    View Slide

  38. Thank you
    PipeCD OSSͷBackendͱFrontendͷϑϧλΠϜɾ෭ۀʢ࣌ؒͷ੍ݶ͸ͳ͠ʣΛืू͍ͯ͠·͢

    ͝ڵຯ͕͋ΔํɺTwitterͷDMͳͲ͝࿈བྷ͍ͩ͘͞

    ʢืूͷ৘ใΛ֦ࢄ͓ͯ͠ئ͍͠·͢ʣ

    View Slide