Le Van Nghia, CyberAgent, Mar 25, 2021
 @nghialv PipeCDͰKubernetesͷGitOps Kubernetes Meetup Tokyo #40

ࣗݾ঺հ @nghialv @nghialv2607 @nghialv Ֆค঱ͰർΕ͍ͯΔϕτφϜਓ Le Van Nghia - ΪΞ 2

ࣗݾ঺հ - ৬ྺ @CyberAgent • PipeCDΛ։ൃɾӡ༻ - DPࣨ • Work fl ow Automation SystemΛ։ൃɾӡ༻ - OSSS • Feature Flags/Experimentation SystemΛ։ൃɾӡ༻ - AbemaTV • PrometheusͰMonitoring SystemΛߏஙɾӡ༻ - AbemaTV • Deployment ToolΛ։ൃɾӡ༻ - AbemaTV • Microservicesɾج൫पΓͷ࢓ࣄ - AbemaTV ΄ͱΜͲ͸ج൫΍ϓϥοτϑΥʔϜͷ͜ͱ 3

ࣗݾ঺հ - DIY https://twitter.com/nghialv2607/status/1345936214407274496 4

ࠓճ͓࿩͢͠Δ಺༰ • CI/CDجຊͷೝࣝ߹Θͤ • PipeCDͱ͸ • ͳͥPipeCDΛ࡞੒͍ͯ͠Δͷ͔ • PipeCDͰͰ͖Δ͜ͱ • CyberAgentͰPipeCDͷར༻ঢ়گ • PipeCDͷࠓޙϩʔυϚοϓ 5

CI/CDجຊͷೝࣝ߹Θͤ Basic concepts Common misunderstandings 6

CI/CD CI and CD systems accelerate the delivery process Actions 7

CI != CD When people say “CI/CD,” they are only talking about Continuous Integration. 
 Nobody is talking about (or practicing) Continuous Deployment. AT ALL. 
 It’s like we have all forgotten it exists. It's time to change that. Charity Majors 8

CI != CD Artifact Storage Verifying and Analysing the Impact Application Code 
 (.go, .java, .js...) Infrastructure Code 
 (.tf ...) Con fi guration Code 
 (.yaml ...) DockerHub, GCR, ECR... GCS, S3... Git Repository Code Storage Actions Continuous Integration Test Code Git Repository Host Environment Artifact Continuous Delivery Artifact Build and Save Artfacts Cloud User Low-risk actions including release strategy, rollback Deployment Dependency Management Provisioning, Installing Artifact 9 Artifact = Docker Image, Helm Chart, Kustomization Module, Terraform Module, ...

Continous Delivery != Continuous Deployment Continuous Deployment means that every change goes through the pipeline and automatically gets put into production, resulting in many production deployments every day. In order to do Continuous Deployment we must be doing Continuous Delivery. Continuous Delivery just means that you are able to do frequent deployments but may choose not to do it, usually due to businesses preferring a slower rate of deployment. Artifact Continuous Deployment Dev, Test Env Artifact Continuous Delivery Prod Env An example 10 https://martinfowler.com/bliki/ContinuousDelivery.html

Deploy != Release Deployment is the process for installing the new version of artifact on prod environment. 
 When we say a new version of software is deployed, we mean it is running somewhere in the production environment. Releasing is the process of moving production tra ff i c to the new version. When we say a version of a software is released, we mean that it is responsible for serving production tra ff i c. Deployment need not expose customers to a new version of your service. Given this definition, deployment can be an almost zero-risk activity. Turbine Labs 11 https://blog.turbinelabs.io/deploy-not-equal-release-part-one-4724bc1e726b

PipeCDͱ͸ A uni fi ed continous delivery solution for multiple application kinds on multi-cloud A gitops tool that enables doing deployment operations by pull request on Git An open source project 12

PipeCDͱ͸ 13 - A uni fi ed continous delivery solution for multiple application kinds on multi-cloud - A gitops tool that enables doing deployment operations by pull request on Git - An open source project

PipeCD ❤ OSS 14 Thanks to the contributors of PipeCD! https://pipecd.dev/ https://github.com/pipe-cd/pipe https://pipecd.dev/docs/ - 2020/10݄ʹOSSͱͯ͠ϦϦʔε͠·ͨ͠ - 4ਓ͕ϑϧλΠϜͰPipeCD΁ίϛοτ͍ͯ͠Δ - 22 contributors͔Β1200 PRʹୡ੒͠·ͨ͠

ͳͥPipeCDΛ࡞੒ͨ͠ͷ͔ Need of a uni fi ed delivery system Easy to operate multi-tenancy for multiple projects Easy to manage a large number of applications with a good DX Existing solutions do not fi t our requirements 15

౷ҰͳσϦόϦγεςϜ͕ඞཁ Project 1 CircleCI 16 Consistency Flexibility • ౷ҰͳγεςϜʹͳΔͱPlatform Team͕😊😊ɺDevelopers͕😊😥 • ౷ҰͳγεςϜͰ͕͢ɺDevelopersͷFlexibilityͷอূ͕ඞཁ • ༷ʑͳΞϓϦέʔγϣϯछྨͷαϙʔτ͕ඞཁ • Kubernetes, Terraform, CloudRun, Lambda, ECS • GCP, AWS, Azure, Private Cloud • ࣗ෼Ͱ࣮૷͢ΔϩδοΫͰ΋ಈ͚Δ • ͲͷϓϩδΣΫτɾνʔϜͰ΋ϫʔΫ͢Δ͜ͱ͕ඞཁ • Ͳͷن໛Ͱ΋ϫʔΫ (3ਓνʔϜ͔Β100ਓνʔϜ·Ͱ) • νʔϜؒʹҠಈ࣌ͷΦϯϘʔσΟϯάίετ͕ແ͠ Project 2 Manually Project 4 FluxCD Project 25 Terraform Cloud + AWS Code Deploy + ArgoCD Project 3 Spinnaker ... Have to fi nd a good balance લͷঢ়ଶ

Multi-Tenancyͷӡ༻͠΍͍͢΋ͷ͕ඞཁ 17 • ωοτϫʔΫͷ੍ݶνʔϜ΋αϙʔτඞཁ • Private cloudͳͲɺ֎͔Βͷ௨৴੍͕ݶ • SecretσʔλΛνʔϜͷΫϥελͷ֎ʹஔ͔ͳ͍ • RBACɾACLͷίϯτϩʔϧ͠΍͍͢ • Platform TeamͱDevelopersͷ໾ׂͱ͸͖ͬΓ෼ׂ • Platform Team͸γεςϜӡ༻ɾϓϥΫςΟεΛීٴ • Developers͸ར༻ɾϑΟʔυόοΫ 25 projectsҎ্

طଘͷιϦϡʔγϣϯ͕ຬͨ͞ͳ͍ 18 ӡ༻ͷେม͞ ֶशίετ GitOpsͰ͸ͳ͍ʢඞਢͰ͸ͳ͍͕😊ʣ Visibilityͷ໰୊ (UIͳ͠ͳͲʣ Kubernetes ApplicationͷΈ
 Multi-Tenancyӡ༻Ͱ଍Γͳ͍ ඪ४ͳDeploymentͷ୅ΘΓʹɺRollout CRDʹมߋඞཁ Kubernetes ApplicationͷΈ
 Multi-Tenancyӡ༻Ͱ଍Γͳ͍ Developer͕୭Ͱ΋ࣗ෼ͷαʔϏεΛߴ଎ɾ҆શɾ ҆৺ͰσϓϩΠͰ͖Δ (σϓϩΠதʹkubectlΛશ͘࢖Θͳ͍͍ͯ͘😊ʣ

PipeCDͰͰ͖Δ͜ͱ Quick Sync and Progressive Sync Automated Rollback Automated Deployment Analysis Con fi guration Drift Detection
 Secret Management
 Event Watcher Noti fi cation 19

Quick Sync vs Progressive Sync 20 Sync GitOpsͷҙਤ Quick Sync͸Clusterͷঢ়ଶΛGitͷঢ়ଶΛಉظ͢ΔͨΊʹɺ͙͢Gitͷঢ়ଶ΁ભҠ Progressive Sync͸Clusterͷঢ়ଶΛGitͷঢ়ଶΛಉظ͢Δ్தʹɺઓུ (canary, bluegreen, analysis...)ʹΑΓ
 ͍͔ͭ͘ͷதؒঢ়ଶʹܦ༝͢Δɻ͔͠͠ɺ࠷ޙతʹ͸Gitͷঢ়ଶʹભҠ Git Cluster Sync Sync

Quick Sync vs Progressive Sync 21 Sync Quick Sync͸Clusterͷঢ়ଶΛGitͷঢ়ଶΛಉظ͢ΔͨΊʹɺ͙͢Gitͷঢ়ଶ΁ભҠ Progressive Sync͸Clusterͷঢ়ଶΛGitͷঢ়ଶΛಉظ͢Δ్தʹɺઓུ (canary, bluegreen, analysis...)ʹΑΓ
 ͍͔ͭ͘ͷதؒঢ়ଶʹܦ༝͢Δɻ͔͠͠ɺ࠷ޙʹ΋Gitͷঢ়ଶʹભҠ Git Cluster Sync Sync GitʹApplication directoryʹ.pipe.yamlͰArtifactͷύεɾσϓϩΠख๏ͳͲΛఆٛͰ͖Δ GitOpsͷҙਤ

Quick Sync 22 https://github.com/pipe-cd/examples/blob/master/kubernetes/simple/.pipe.yaml PipelineΛઃఆ͍ͯ͠ͳ͍৔߹͸Quick SyncΛ࣮ߦ શͯͷManifestsΛ௚઀Apply͢Δ

Progressive Sync 23 https://github.com/pipe-cd/examples/blob/master/kubernetes/canary/.pipe.yaml PipelineΛهࡌ͢Δ৔߹͸PipeCDͷPlanner͕มߋ಺༰ʹΑΓQuick Sync͔Progressive SyncΛ൑அɺྫ: - replicas numberͷมߋͷΈͰɺscaleͷ৔߹͸Quick Sync - pod templateͷมߋͷ৔߹͸Progressive Sync - con fi g map/secretͷมߋͷ৔߹͸Progressive Sync - deployment͝ͱʹڧ੍΋Մೳ

Automated Rollback 24 https://pipecd.dev/docs/user-guide/rolling-back-a-deployment/ git/path/.pipe.yaml ్தͰ໰୊͕ൃੜͨ͠ΓɺϦϦʔε͕ѱ͍ΠϯύΫτΛ༩͍͑ͯΔͱ൑அ͞Εͨ৔߹ʹ
 ࣗಈతʹϩʔϧόοΫ͢ΔΑ͏ʹઃఆՄೳ

Atomated Deployment Analysis 25 https://pipecd.dev/docs/user-guide/automated-deployment-analysis/ https://github.com/pipe-cd/examples/blob/master/kubernetes/analysis-by-metrics/.pipe.yaml ϦϦʔεͷΠϯύΫτ͸Metrics, Logs, Smoke TestͳͲͰ൑அΛߦ͏

Configuration Drift Detection 26 https://pipecd.dev/docs/user-guide/con fi guration-drift-detection/ • ࣮ࡍͷঢ়ଶ͕ظ଴ͷঢ়ଶͱဃ཭ • Ϣʔβʔ͕௚઀ௐ੔ • ଞͷαʔϏε͕௚઀ௐ੔ • ࣗಈతʹCon fi guration DriftΛݕ஌ • WebUIͰࠩ෼Λදࣔ • ௨஌ͰΞϥʔτͷઃఆ͕Մೳ • ݱࡏ͸Con fi guration Drift͕ൃੜ͢Δͱɺ উखʹApply͠ͳ͍

Secret Management 27 https://pipecd.dev/docs/user-guide/sealed-secrets/ • GitOps͸શͯͷ΋ͷΛGitʹอଘ • SecretΛ҆શʹอଘํ๏͕ඞཁ • PipeCD͸built-in secret؅ཧํ๏Λ࣋ͭ • Piped agent͕ར༻͢Δલʹ෮ݩΛߦ͏ 1 2 PipeCD webͰSecretͷ҉߸ԽΛߦ͏ ҉߸Խ͞ΕͨσʔλΛGitʹஔ͘ https://blog.stormcat.io/post/pipecd-sealed-secret/

Event Watcher 28 FluxCDͷImage Updateػೳͷઆ໌ https://toolkit. fl uxcd.io/guides/image-update/ Container Registry Git Repository ArgoCD 
 FluxCD Watches images Makes commit to update image tags • GitOps͸શͯͷoperation͕Git PRΛ௨ͯ͠΍Δݪଇ • ৽͍͠container image͕Ͱ͖ͨΒɺࣗಈతʹGitΛߋ৽

• GitOps͸શͯͷoperation͕Git PRΛ௨ͯ͠΍Δݪଇ • ৽͍͠container image͕Ͱ͖ͨΒɺࣗಈతʹGitΛߋ৽ • ͜ͷΞϓϩʔνͷ໰୊఺ • CI͔ΒCD΁౉͢Artifact͸Container Image͚ͩͰͳͳ͘ • Helm Chart • Kustomization Module • Terraform Module • Etc • աڈͷImage਺͕ଟ͍৔߹ʹRegistryͷWatchͷύϑΥʔϚϯε Event Watcher 29 Container Registry Git Repository ArgoCD 
 FluxCD Watches images Makes commit to update image tags

Event Watcher 30 https://pipecd.dev/docs/user-guide/event-watcher/ pipectl event register \ --name=helloworld-image-update \ --data=gcr.io/pipecd/helloworld:v0.2.0 apiVersion: pipecd.dev/v1beta1 kind: EventWatcher spec: events: - name: helloworld-image-update replacements: - file: helloworld/deployment.yaml yamlField: $.spec.template.spec.containers[0].image spec: containers: - name: helloworld - image: gcr.io/pipecd/helloworld:v0.1.0 + image: gcr.io/pipecd/helloworld:v0.2.0 • PipeCDͰ͸ Image Watcher ΑΓ Event WatcherػೳΛఏڙ • pipectlͰeventΛૹΔ͜ͱͰɺeventʹΑΓGitΛࣗಈతʹम ਖ਼ͯ͘͠ΕΔઃఆ͕Մೳ GitͷதʹeventʹΑΓमਖ਼ͷఆٛ Piped agent͕GitΛमਖ਼ͯ͘͠ΕΔ CIͰ೚ҙͷ࣌ؒͰeventΛൃੜ

Notification 31 https://pipecd.dev/docs/operator-manual/piped/con fi guring-noti fi cations/ • ௨஌ઌͷઃఆ͕Մೳ • Slack • Webhook • ௨஌Πϕϯτͷઃఆ͕Մೳ • Deploymentͷ࣮ߦঢ়ଶ • Con fi guration drift͕ൃੜ • Application Healthͷঢ়ଶ • Pipedͷঢ়ଶ • etc

୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε 32 શͯͷػೳ͕Kubernetes, Terraform, Lambda, CloudRun, ECSͰ࢖͑Δ GCP, AWS, AzureͳͲcloud providerΛαϙʔτ Prometheus, Datadog, CloudWatch, Stackdriver LoggingͳͲͷσʔλͰ෼ੳΛߦ͑Δ

CyberAgentͰPipeCDͷར༻ঢ়گ The structure of Team and System The numbers at CyberAgent What we have achieved 33

νʔϜͱγεςϜͷߏ੒ 34 • Platform Team • GCP্Ͱશࣾ༻Control-PlaneΛӡ༻ • GCPͷFirestore & GCSͷϚωδʔυαʔϏεΛར༻ • StatelessͷServer & Cache͸K8sͷ্ʹಈ͘ • ֤ProjectͷSREs • Single binaryͷPiped agentΛΠϯετʔϧ • K8s cluster or Fargate or VMͷதʹಈ͘ • ֤ProjectͷDevelopers • WebͰ࢖͏ • GitͰPRΛૹͬͯɺσϓϩΠΛߦ͏

ಋೖαʔϏε਺͕૿Ճத 0 100 200 300 400 2020/10 2020/11 2020/12 2021/01 2021/02 2021/03 332 Applications/Services ʢ࢒ΓͷϓϩδΣΫτ΋Ҡಈதʣ 35

ಋೖͰΑ͔ͬͨ͜ͱ 36 • Platform Team 😊 • શͯͷνʔϜͷσϓϩΠϝϯτΛ౷ҰͰ؅ཧ • ϓϥΫςΟεΛ࠾༻ɾීٴ͠΍͍͢ • ӡ༻ָ͕ • શࣾͷ֤νʔϜ͔ΒϑΟʔυόοΫΛ΋Β͑Δ • Developers 😊 • kubectlͳͲ͕ෆཁͰɺߴ଎ɾ҆શɾ҆৺ͰσϓϩΠ • શͯͷσϓϩΠϝϯτ͕୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε • ৽نͷϓϩδΣΫτɾαʔϏεͷಋೖ͕͸΍͍ • ΦϯϘʔσΟϯάίετ͕௿͍ Consistency Flexibility Good Balance

PipeCDͷࠓޙϩʔυϚοϓ 37 Improve the Visibility Improve the Flexibility Add more features

ࠓޙͷϩʔυϚοϓ • VisibilityΛ޲্ • Insights: Lead Time, Deployment Frequency, MTTR, Change Failure RateͳͲΛՄࢹԽ • Applicationͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ • Stage LogΛΑΓΘ͔Γ΍͘͢ɺ໰୊Λ͙͢ݟ͑ΔΑ͏ʹ • Multi-Provider, Multi-Tenancy • ECSͷαϙʔτ • ACLͰਂ͍Ϩϕϧͷݖݶ؅ཧ • Automated Deployment Analysis: CloudWatch, Stackdriver Logging... • AWS App Mesh, SMI • Secret Management • Sealed secretҎ֎ʹKMS, Vault΋αϙʔτ • ࣗ༝౓Λ্͛ΔͨΊʹɺϢʔβʔͷ࣮૷ͷpluginΛ࣮ߦͰ͖ΔΑ͏ʹ 38

࠷ޙʹ • ࠓޙ΋ੵۃతʹ։ൃΛଓ͘ • ௚ۙʹCyberAgentͷશͯͷαʔϏεͰ࢖͑ΔΑ͏ʹීٴͯ͠ߦ͘ • OSSͰެ։ͳͷͰɺશͯͷϑΟʔυόοΫΛ׻ܴ • OSSͷ࢓ࣄʹڵຯ͕͋ΔํɾΠϯλʔϯੜ͸TwitterͷDMΛ׻ܴ • If you like PipeCD or want to support Dev team, give it a star on GitHub! 39

Thank You