$30 off During Our Annual Pro Sale. View Details »

Introdution_to_PipeCD.pdf

nghialv
March 25, 2021

 Introdution_to_PipeCD.pdf

nghialv

March 25, 2021
Tweet

More Decks by nghialv

Other Decks in Technology

Transcript

  1. Le Van Nghia, CyberAgent, Mar 25, 2021

    @nghialv
    PipeCDͰKubernetesͷGitOps
    Kubernetes Meetup Tokyo #40

    View Slide

  2. ࣗݾ঺հ
    @nghialv
    @nghialv2607
    @nghialv
    Ֆค঱ͰർΕ͍ͯΔϕτφϜਓ
    Le Van Nghia - ΪΞ
    2

    View Slide

  3. ࣗݾ঺հ - ৬ྺ @CyberAgent
    • PipeCDΛ։ൃɾӡ༻ - DPࣨ

    • Work
    fl
    ow Automation SystemΛ։ൃɾӡ༻ - OSSS

    • Feature Flags/Experimentation SystemΛ։ൃɾӡ༻ - AbemaTV

    • PrometheusͰMonitoring SystemΛߏஙɾӡ༻ - AbemaTV

    • Deployment ToolΛ։ൃɾӡ༻ - AbemaTV

    • Microservicesɾج൫पΓͷ࢓ࣄ - AbemaTV
    ΄ͱΜͲ͸ج൫΍ϓϥοτϑΥʔϜͷ͜ͱ
    3

    View Slide

  4. ࣗݾ঺հ - DIY
    https://twitter.com/nghialv2607/status/1345936214407274496
    4

    View Slide

  5. ࠓճ͓࿩͢͠Δ಺༰
    • CI/CDجຊͷೝࣝ߹Θͤ

    • PipeCDͱ͸

    • ͳͥPipeCDΛ࡞੒͍ͯ͠Δͷ͔

    • PipeCDͰͰ͖Δ͜ͱ

    • CyberAgentͰPipeCDͷར༻ঢ়گ

    • PipeCDͷࠓޙϩʔυϚοϓ
    5

    View Slide

  6. CI/CDجຊͷೝࣝ߹Θͤ
    Basic concepts

    Common misunderstandings
    6

    View Slide

  7. CI/CD
    CI and CD systems accelerate the delivery process
    Actions
    7

    View Slide

  8. CI != CD
    When people say “CI/CD,” they are only talking about Continuous Integration. 

    Nobody is talking about (or practicing) Continuous Deployment. AT ALL. 

    It’s like we have all forgotten it exists. It's time to change that.

    Charity Majors
    8

    View Slide

  9. CI != CD
    Artifact Storage
    Verifying and Analysing the Impact
    Application Code 

    (.go, .java, .js...)
    Infrastructure Code 

    (.tf ...)
    Con
    fi
    guration Code 

    (.yaml ...)
    DockerHub, GCR, ECR...
    GCS, S3...
    Git Repository
    Code Storage
    Actions
    Continuous Integration
    Test Code
    Git Repository
    Host Environment
    Artifact
    Continuous Delivery
    Artifact
    Build and Save Artfacts
    Cloud
    User
    Low-risk actions including release strategy, rollback
    Deployment Dependency Management
    Provisioning, Installing Artifact
    9
    Artifact =

    Docker Image,

    Helm Chart,

    Kustomization Module,

    Terraform Module,

    ...

    View Slide

  10. Continous Delivery != Continuous Deployment
    Continuous Deployment means that every change goes through the pipeline and
    automatically gets put into production, resulting in many production deployments every day.
    In order to do Continuous Deployment we must be doing Continuous Delivery.
    Continuous Delivery just means that you are able to do frequent deployments but may
    choose not to do it, usually due to businesses preferring a slower rate of deployment.
    Artifact
    Continuous Deployment
    Dev, Test Env
    Artifact
    Continuous Delivery
    Prod Env
    An example
    10
    https://martinfowler.com/bliki/ContinuousDelivery.html

    View Slide

  11. Deploy != Release
    Deployment is the process for installing the new version of artifact on prod environment. 

    When we say a new version of software is deployed, we mean it is running somewhere in
    the production environment.
    Releasing is the process of moving production tra
    ff i
    c to the new version.

    When we say a version of a software is released, we mean that it is responsible for
    serving production tra
    ff i
    c.
    Deployment need not expose customers to a new version of your service.


    Given this definition, deployment can be an almost zero-risk activity.


    Turbine Labs
    11
    https://blog.turbinelabs.io/deploy-not-equal-release-part-one-4724bc1e726b

    View Slide

  12. PipeCDͱ͸
    A uni
    fi
    ed continous delivery solution for multiple application kinds on multi-cloud

    A gitops tool that enables doing deployment operations by pull request on Git

    An open source project
    12

    View Slide

  13. PipeCDͱ͸
    13
    - A uni
    fi
    ed continous delivery solution for multiple application kinds on multi-cloud

    - A gitops tool that enables doing deployment operations by pull request on Git

    - An open source project

    View Slide

  14. PipeCD ❤ OSS
    14
    Thanks to the contributors of PipeCD!
    https://pipecd.dev/
    https://github.com/pipe-cd/pipe
    https://pipecd.dev/docs/
    - 2020/10݄ʹOSSͱͯ͠ϦϦʔε͠·ͨ͠

    - 4ਓ͕ϑϧλΠϜͰPipeCD΁ίϛοτ͍ͯ͠Δ

    - 22 contributors͔Β1200 PRʹୡ੒͠·ͨ͠

    View Slide

  15. ͳͥPipeCDΛ࡞੒ͨ͠ͷ͔
    Need of a uni
    fi
    ed delivery system

    Easy to operate multi-tenancy for multiple projects

    Easy to manage a large number of applications with a good DX

    Existing solutions do not
    fi
    t our requirements
    15

    View Slide

  16. ౷ҰͳσϦόϦγεςϜ͕ඞཁ
    Project 1


    CircleCI
    16
    Consistency Flexibility
    • ౷ҰͳγεςϜʹͳΔͱPlatform Team͕😊😊ɺDevelopers͕😊😥

    • ౷ҰͳγεςϜͰ͕͢ɺDevelopersͷFlexibilityͷอূ͕ඞཁ

    • ༷ʑͳΞϓϦέʔγϣϯछྨͷαϙʔτ͕ඞཁ

    • Kubernetes, Terraform, CloudRun, Lambda, ECS

    • GCP, AWS, Azure, Private Cloud

    • ࣗ෼Ͱ࣮૷͢ΔϩδοΫͰ΋ಈ͚Δ

    • ͲͷϓϩδΣΫτɾνʔϜͰ΋ϫʔΫ͢Δ͜ͱ͕ඞཁ

    • Ͳͷن໛Ͱ΋ϫʔΫ (3ਓνʔϜ͔Β100ਓνʔϜ·Ͱ)

    • νʔϜؒʹҠಈ࣌ͷΦϯϘʔσΟϯάίετ͕ແ͠
    Project 2


    Manually
    Project 4


    FluxCD
    Project 25


    Terraform Cloud + AWS Code Deploy + ArgoCD
    Project 3


    Spinnaker
    ...
    Have to
    fi
    nd a good balance
    લͷঢ়ଶ

    View Slide

  17. Multi-Tenancyͷӡ༻͠΍͍͢΋ͷ͕ඞཁ
    17
    • ωοτϫʔΫͷ੍ݶνʔϜ΋αϙʔτඞཁ

    • Private cloudͳͲɺ֎͔Βͷ௨৴੍͕ݶ

    • SecretσʔλΛνʔϜͷΫϥελͷ֎ʹஔ͔ͳ͍

    • RBACɾACLͷίϯτϩʔϧ͠΍͍͢

    • Platform TeamͱDevelopersͷ໾ׂͱ͸͖ͬΓ෼ׂ

    • Platform Team͸γεςϜӡ༻ɾϓϥΫςΟεΛීٴ

    • Developers͸ར༻ɾϑΟʔυόοΫ
    25 projectsҎ্

    View Slide

  18. طଘͷιϦϡʔγϣϯ͕ຬͨ͞ͳ͍
    18
    ӡ༻ͷେม͞

    ֶशίετ

    GitOpsͰ͸ͳ͍ʢඞਢͰ͸ͳ͍͕😊ʣ
    Visibilityͷ໰୊ (UIͳ͠ͳͲʣ

    Kubernetes ApplicationͷΈ

    Multi-Tenancyӡ༻Ͱ଍Γͳ͍
    ඪ४ͳDeploymentͷ୅ΘΓʹɺRollout CRDʹมߋඞཁ

    Kubernetes ApplicationͷΈ

    Multi-Tenancyӡ༻Ͱ଍Γͳ͍
    Developer͕୭Ͱ΋ࣗ෼ͷαʔϏεΛߴ଎ɾ҆શɾ
    ҆৺ͰσϓϩΠͰ͖Δ

    (σϓϩΠதʹkubectlΛશ͘࢖Θͳ͍͍ͯ͘😊ʣ

    View Slide

  19. PipeCDͰͰ͖Δ͜ͱ
    Quick Sync and Progressive Sync

    Automated Rollback

    Automated Deployment Analysis

    Con
    fi
    guration Drift Detection

    Secret Management

    Event Watcher

    Noti
    fi
    cation
    19

    View Slide

  20. Quick Sync vs Progressive Sync
    20
    Sync
    GitOpsͷҙਤ
    Quick Sync͸Clusterͷঢ়ଶΛGitͷঢ়ଶΛಉظ͢ΔͨΊʹɺ͙͢Gitͷঢ়ଶ΁ભҠ

    Progressive Sync͸Clusterͷঢ়ଶΛGitͷঢ়ଶΛಉظ͢Δ్தʹɺઓུ (canary, bluegreen, analysis...)ʹΑΓ

    ͍͔ͭ͘ͷதؒঢ়ଶʹܦ༝͢Δɻ͔͠͠ɺ࠷ޙతʹ͸Gitͷঢ়ଶʹભҠ
    Git
    Cluster
    Sync Sync

    View Slide

  21. Quick Sync vs Progressive Sync
    21
    Sync
    Quick Sync͸Clusterͷঢ়ଶΛGitͷঢ়ଶΛಉظ͢ΔͨΊʹɺ͙͢Gitͷঢ়ଶ΁ભҠ

    Progressive Sync͸Clusterͷঢ়ଶΛGitͷঢ়ଶΛಉظ͢Δ్தʹɺઓུ (canary, bluegreen, analysis...)ʹΑΓ

    ͍͔ͭ͘ͷதؒঢ়ଶʹܦ༝͢Δɻ͔͠͠ɺ࠷ޙʹ΋Gitͷঢ়ଶʹભҠ
    Git
    Cluster
    Sync Sync
    GitʹApplication directoryʹ.pipe.yamlͰArtifactͷύεɾσϓϩΠख๏ͳͲΛఆٛͰ͖Δ
    GitOpsͷҙਤ

    View Slide

  22. Quick Sync
    22
    https://github.com/pipe-cd/examples/blob/master/kubernetes/simple/.pipe.yaml
    PipelineΛઃఆ͍ͯ͠ͳ͍৔߹͸Quick SyncΛ࣮ߦ

    શͯͷManifestsΛ௚઀Apply͢Δ

    View Slide

  23. Progressive Sync
    23
    https://github.com/pipe-cd/examples/blob/master/kubernetes/canary/.pipe.yaml
    PipelineΛهࡌ͢Δ৔߹͸PipeCDͷPlanner͕มߋ಺༰ʹΑΓQuick Sync͔Progressive SyncΛ൑அɺྫ:

    - replicas numberͷมߋͷΈͰɺscaleͷ৔߹͸Quick Sync

    - pod templateͷมߋͷ৔߹͸Progressive Sync

    - con
    fi
    g map/secretͷมߋͷ৔߹͸Progressive Sync

    - deployment͝ͱʹڧ੍΋Մೳ

    View Slide

  24. Automated Rollback
    24
    https://pipecd.dev/docs/user-guide/rolling-back-a-deployment/
    git/path/.pipe.yaml
    ్தͰ໰୊͕ൃੜͨ͠ΓɺϦϦʔε͕ѱ͍ΠϯύΫτΛ༩͍͑ͯΔͱ൑அ͞Εͨ৔߹ʹ

    ࣗಈతʹϩʔϧόοΫ͢ΔΑ͏ʹઃఆՄೳ

    View Slide

  25. Atomated Deployment Analysis
    25
    https://pipecd.dev/docs/user-guide/automated-deployment-analysis/
    https://github.com/pipe-cd/examples/blob/master/kubernetes/analysis-by-metrics/.pipe.yaml
    ϦϦʔεͷΠϯύΫτ͸Metrics, Logs, Smoke TestͳͲͰ൑அΛߦ͏

    View Slide

  26. Configuration Drift Detection
    26
    https://pipecd.dev/docs/user-guide/con
    fi
    guration-drift-detection/
    • ࣮ࡍͷঢ়ଶ͕ظ଴ͷঢ়ଶͱဃ཭

    • Ϣʔβʔ͕௚઀ௐ੔

    • ଞͷαʔϏε͕௚઀ௐ੔

    • ࣗಈతʹCon
    fi
    guration DriftΛݕ஌

    • WebUIͰࠩ෼Λදࣔ

    • ௨஌ͰΞϥʔτͷઃఆ͕Մೳ

    • ݱࡏ͸Con
    fi
    guration Drift͕ൃੜ͢Δͱɺ
    উखʹApply͠ͳ͍

    View Slide

  27. Secret Management
    27
    https://pipecd.dev/docs/user-guide/sealed-secrets/
    • GitOps͸શͯͷ΋ͷΛGitʹอଘ

    • SecretΛ҆શʹอଘํ๏͕ඞཁ

    • PipeCD͸built-in secret؅ཧํ๏Λ࣋ͭ

    • Piped agent͕ར༻͢Δલʹ෮ݩΛߦ͏
    1
    2
    PipeCD webͰSecretͷ҉߸ԽΛߦ͏
    ҉߸Խ͞ΕͨσʔλΛGitʹஔ͘
    https://blog.stormcat.io/post/pipecd-sealed-secret/

    View Slide

  28. Event Watcher
    28
    FluxCDͷImage Updateػೳͷઆ໌

    https://toolkit.
    fl
    uxcd.io/guides/image-update/
    Container Registry
    Git Repository
    ArgoCD

    FluxCD
    Watches images
    Makes commit to update image tags
    • GitOps͸શͯͷoperation͕Git PRΛ௨ͯ͠΍Δݪଇ

    • ৽͍͠container image͕Ͱ͖ͨΒɺࣗಈతʹGitΛߋ৽

    View Slide

  29. • GitOps͸શͯͷoperation͕Git PRΛ௨ͯ͠΍Δݪଇ

    • ৽͍͠container image͕Ͱ͖ͨΒɺࣗಈతʹGitΛߋ৽

    • ͜ͷΞϓϩʔνͷ໰୊఺

    • CI͔ΒCD΁౉͢Artifact͸Container Image͚ͩͰͳͳ͘

    • Helm Chart

    • Kustomization Module

    • Terraform Module

    • Etc

    • աڈͷImage਺͕ଟ͍৔߹ʹRegistryͷWatchͷύϑΥʔϚϯε
    Event Watcher
    29
    Container Registry
    Git Repository
    ArgoCD

    FluxCD
    Watches images
    Makes commit to update image tags

    View Slide

  30. Event Watcher
    30
    https://pipecd.dev/docs/user-guide/event-watcher/
    pipectl event register \


    --name=helloworld-image-update \


    --data=gcr.io/pipecd/helloworld:v0.2.0


    apiVersion: pipecd.dev/v1beta1


    kind: EventWatcher


    spec:


    events:


    - name: helloworld-image-update


    replacements:


    - file: helloworld/deployment.yaml


    yamlField: $.spec.template.spec.containers[0].image
    spec:


    containers:


    - name: helloworld


    - image: gcr.io/pipecd/helloworld:v0.1.0


    + image: gcr.io/pipecd/helloworld:v0.2.0
    • PipeCDͰ͸ Image Watcher ΑΓ Event WatcherػೳΛఏڙ

    • pipectlͰeventΛૹΔ͜ͱͰɺeventʹΑΓGitΛࣗಈతʹम
    ਖ਼ͯ͘͠ΕΔઃఆ͕Մೳ
    GitͷதʹeventʹΑΓमਖ਼ͷఆٛ
    Piped agent͕GitΛमਖ਼ͯ͘͠ΕΔ
    CIͰ೚ҙͷ࣌ؒͰeventΛൃੜ

    View Slide

  31. Notification
    31
    https://pipecd.dev/docs/operator-manual/piped/con
    fi
    guring-noti
    fi
    cations/
    • ௨஌ઌͷઃఆ͕Մೳ

    • Slack

    • Webhook

    • ௨஌Πϕϯτͷઃఆ͕Մೳ

    • Deploymentͷ࣮ߦঢ়ଶ

    • Con
    fi
    guration drift͕ൃੜ

    • Application Healthͷঢ়ଶ

    • Pipedͷঢ়ଶ

    • etc

    View Slide

  32. ୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε
    32
    શͯͷػೳ͕Kubernetes, Terraform, Lambda, CloudRun, ECSͰ࢖͑Δ

    GCP, AWS, AzureͳͲcloud providerΛαϙʔτ

    Prometheus, Datadog, CloudWatch, Stackdriver LoggingͳͲͷσʔλͰ෼ੳΛߦ͑Δ

    View Slide

  33. CyberAgentͰPipeCDͷར༻ঢ়گ
    The structure of Team and System

    The numbers at CyberAgent

    What we have achieved
    33

    View Slide

  34. νʔϜͱγεςϜͷߏ੒
    34
    • Platform Team

    • GCP্Ͱશࣾ༻Control-PlaneΛӡ༻

    • GCPͷFirestore & GCSͷϚωδʔυαʔϏεΛར༻

    • StatelessͷServer & Cache͸K8sͷ্ʹಈ͘

    • ֤ProjectͷSREs

    • Single binaryͷPiped agentΛΠϯετʔϧ

    • K8s cluster or Fargate or VMͷதʹಈ͘

    • ֤ProjectͷDevelopers

    • WebͰ࢖͏

    • GitͰPRΛૹͬͯɺσϓϩΠΛߦ͏

    View Slide

  35. ಋೖαʔϏε਺͕૿Ճத
    0
    100
    200
    300
    400
    2020/10 2020/11 2020/12 2021/01 2021/02 2021/03
    332 Applications/Services
    ʢ࢒ΓͷϓϩδΣΫτ΋Ҡಈதʣ
    35

    View Slide

  36. ಋೖͰΑ͔ͬͨ͜ͱ
    36
    • Platform Team 😊

    • શͯͷνʔϜͷσϓϩΠϝϯτΛ౷ҰͰ؅ཧ

    • ϓϥΫςΟεΛ࠾༻ɾීٴ͠΍͍͢

    • ӡ༻ָ͕

    • શࣾͷ֤νʔϜ͔ΒϑΟʔυόοΫΛ΋Β͑Δ

    • Developers 😊

    • kubectlͳͲ͕ෆཁͰɺߴ଎ɾ҆શɾ҆৺ͰσϓϩΠ

    • શͯͷσϓϩΠϝϯτ͕୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε

    • ৽نͷϓϩδΣΫτɾαʔϏεͷಋೖ͕͸΍͍

    • ΦϯϘʔσΟϯάίετ͕௿͍
    Consistency
    Flexibility
    Good Balance

    View Slide

  37. PipeCDͷࠓޙϩʔυϚοϓ
    37
    Improve the Visibility

    Improve the Flexibility

    Add more features

    View Slide

  38. ࠓޙͷϩʔυϚοϓ
    • VisibilityΛ޲্

    • Insights: Lead Time, Deployment Frequency, MTTR, Change Failure RateͳͲΛՄࢹԽ

    • Applicationͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ

    • Stage LogΛΑΓΘ͔Γ΍͘͢ɺ໰୊Λ͙͢ݟ͑ΔΑ͏ʹ

    • Multi-Provider, Multi-Tenancy

    • ECSͷαϙʔτ

    • ACLͰਂ͍Ϩϕϧͷݖݶ؅ཧ

    • Automated Deployment Analysis: CloudWatch, Stackdriver Logging...

    • AWS App Mesh, SMI

    • Secret Management

    • Sealed secretҎ֎ʹKMS, Vault΋αϙʔτ

    • ࣗ༝౓Λ্͛ΔͨΊʹɺϢʔβʔͷ࣮૷ͷpluginΛ࣮ߦͰ͖ΔΑ͏ʹ
    38

    View Slide

  39. ࠷ޙʹ
    • ࠓޙ΋ੵۃతʹ։ൃΛଓ͘

    • ௚ۙʹCyberAgentͷશͯͷαʔϏεͰ࢖͑ΔΑ͏ʹීٴͯ͠ߦ͘

    • OSSͰެ։ͳͷͰɺશͯͷϑΟʔυόοΫΛ׻ܴ

    • OSSͷ࢓ࣄʹڵຯ͕͋ΔํɾΠϯλʔϯੜ͸TwitterͷDMΛ׻ܴ

    • If you like PipeCD or want to support Dev team, give it a star on GitHub!
    39

    View Slide

  40. Thank You

    View Slide