Introdution_to_PipeCD.pdf

Transcript

1. Le Van Nghia, CyberAgent, Mar 25, 2021

   @nghialv
   PipeCDͰKubernetesͷGitOps
   Kubernetes Meetup Tokyo #40
   

   View Slide

2. ࣗݾ঺հ
   @nghialv
   @nghialv2607
   @nghialv
   Ֆค঱ͰർΕ͍ͯΔϕτφϜਓ
   Le Van Nghia - ΪΞ
   2
   

   View Slide

3. ࣗݾ঺հ - ৬ྺ @CyberAgent
   • PipeCDΛ։ൃɾӡ༻ - DPࣨ
   
   • Work
   fl
   ow Automation SystemΛ։ൃɾӡ༻ - OSSS
   
   • Feature Flags/Experimentation SystemΛ։ൃɾӡ༻ - AbemaTV
   
   • PrometheusͰMonitoring SystemΛߏஙɾӡ༻ - AbemaTV
   
   • Deployment ToolΛ։ൃɾӡ༻ - AbemaTV
   
   • Microservicesɾج൫पΓͷ࢓ࣄ - AbemaTV
   ΄ͱΜͲ͸ج൫΍ϓϥοτϑΥʔϜͷ͜ͱ
   3
   

   View Slide

4. ࣗݾ঺հ - DIY
   https://twitter.com/nghialv2607/status/1345936214407274496
   4
   

   View Slide

5. ࠓճ͓࿩͢͠Δ಺༰
   • CI/CDجຊͷೝࣝ߹Θͤ
   
   • PipeCDͱ͸
   
   • ͳͥPipeCDΛ࡞੒͍ͯ͠Δͷ͔
   
   • PipeCDͰͰ͖Δ͜ͱ
   
   • CyberAgentͰPipeCDͷར༻ঢ়گ
   
   • PipeCDͷࠓޙϩʔυϚοϓ
   5
   

   View Slide

6. CI/CDجຊͷೝࣝ߹Θͤ
   Basic concepts
   
   Common misunderstandings
   6
   

   View Slide

7. CI/CD
   CI and CD systems accelerate the delivery process
   Actions
   7
   

   View Slide

8. CI != CD
   When people say “CI/CD,” they are only talking about Continuous Integration. 

   Nobody is talking about (or practicing) Continuous Deployment. AT ALL. 

   It’s like we have all forgotten it exists. It's time to change that.
   
   Charity Majors
   8
   

   View Slide

9. CI != CD
   Artifact Storage
   Verifying and Analysing the Impact
   Application Code 

   (.go, .java, .js...)
   Infrastructure Code 

   (.tf ...)
   Con
   fi
   guration Code 

   (.yaml ...)
   DockerHub, GCR, ECR...
   GCS, S3...
   Git Repository
   Code Storage
   Actions
   Continuous Integration
   Test Code
   Git Repository
   Host Environment
   Artifact
   Continuous Delivery
   Artifact
   Build and Save Artfacts
   Cloud
   User
   Low-risk actions including release strategy, rollback
   Deployment Dependency Management
   Provisioning, Installing Artifact
   9
   Artifact =
   
   Docker Image,
   
   Helm Chart,
   
   Kustomization Module,
   
   Terraform Module,
   
   ...
   

   View Slide

10. Continous Delivery != Continuous Deployment
    Continuous Deployment means that every change goes through the pipeline and
    automatically gets put into production, resulting in many production deployments every day.
    In order to do Continuous Deployment we must be doing Continuous Delivery.
    Continuous Delivery just means that you are able to do frequent deployments but may
    choose not to do it, usually due to businesses preferring a slower rate of deployment.
    Artifact
    Continuous Deployment
    Dev, Test Env
    Artifact
    Continuous Delivery
    Prod Env
    An example
    10
    https://martinfowler.com/bliki/ContinuousDelivery.html
    

    View Slide

11. Deploy != Release
    Deployment is the process for installing the new version of artifact on prod environment. 

    When we say a new version of software is deployed, we mean it is running somewhere in
    the production environment.
    Releasing is the process of moving production tra
    ff i
    c to the new version.
    
    When we say a version of a software is released, we mean that it is responsible for
    serving production tra
    ff i
    c.
    Deployment need not expose customers to a new version of your service.
    
    
    Given this definition, deployment can be an almost zero-risk activity.
    
    
    Turbine Labs
    11
    https://blog.turbinelabs.io/deploy-not-equal-release-part-one-4724bc1e726b
    

    View Slide

12. PipeCDͱ͸
    A uni
    fi
    ed continous delivery solution for multiple application kinds on multi-cloud
    
    A gitops tool that enables doing deployment operations by pull request on Git
    
    An open source project
    12
    

    View Slide

13. PipeCDͱ͸
    13
    - A uni
    fi
    ed continous delivery solution for multiple application kinds on multi-cloud
    
    - A gitops tool that enables doing deployment operations by pull request on Git
    
    - An open source project
    

    View Slide

14. PipeCD ❤ OSS
    14
    Thanks to the contributors of PipeCD!
    https://pipecd.dev/
    https://github.com/pipe-cd/pipe
    https://pipecd.dev/docs/
    - 2020/10݄ʹOSSͱͯ͠ϦϦʔε͠·ͨ͠
    
    - 4ਓ͕ϑϧλΠϜͰPipeCD΁ίϛοτ͍ͯ͠Δ
    
    - 22 contributors͔Β1200 PRʹୡ੒͠·ͨ͠
    

    View Slide

15. ͳͥPipeCDΛ࡞੒ͨ͠ͷ͔
    Need of a uni
    fi
    ed delivery system
    
    Easy to operate multi-tenancy for multiple projects
    
    Easy to manage a large number of applications with a good DX
    
    Existing solutions do not
    fi
    t our requirements
    15
    

    View Slide

16. ౷ҰͳσϦόϦγεςϜ͕ඞཁ
    Project 1
    
    
    CircleCI
    16
    Consistency Flexibility
    • ౷ҰͳγεςϜʹͳΔͱPlatform Team͕😊😊ɺDevelopers͕😊😥
    
    • ౷ҰͳγεςϜͰ͕͢ɺDevelopersͷFlexibilityͷอূ͕ඞཁ
    
    • ༷ʑͳΞϓϦέʔγϣϯछྨͷαϙʔτ͕ඞཁ
    
    • Kubernetes, Terraform, CloudRun, Lambda, ECS
    
    • GCP, AWS, Azure, Private Cloud
    
    • ࣗ෼Ͱ࣮૷͢ΔϩδοΫͰ΋ಈ͚Δ
    
    • ͲͷϓϩδΣΫτɾνʔϜͰ΋ϫʔΫ͢Δ͜ͱ͕ඞཁ
    
    • Ͳͷن໛Ͱ΋ϫʔΫ (3ਓνʔϜ͔Β100ਓνʔϜ·Ͱ)
    
    • νʔϜؒʹҠಈ࣌ͷΦϯϘʔσΟϯάίετ͕ແ͠
    Project 2
    
    
    Manually
    Project 4
    
    
    FluxCD
    Project 25
    
    
    Terraform Cloud + AWS Code Deploy + ArgoCD
    Project 3
    
    
    Spinnaker
    ...
    Have to
    fi
    nd a good balance
    લͷঢ়ଶ
    

    View Slide

17. Multi-Tenancyͷӡ༻͠΍͍͢΋ͷ͕ඞཁ
    17
    • ωοτϫʔΫͷ੍ݶνʔϜ΋αϙʔτඞཁ
    
    • Private cloudͳͲɺ֎͔Βͷ௨৴੍͕ݶ
    
    • SecretσʔλΛνʔϜͷΫϥελͷ֎ʹஔ͔ͳ͍
    
    • RBACɾACLͷίϯτϩʔϧ͠΍͍͢
    
    • Platform TeamͱDevelopersͷ໾ׂͱ͸͖ͬΓ෼ׂ
    
    • Platform Team͸γεςϜӡ༻ɾϓϥΫςΟεΛීٴ
    
    • Developers͸ར༻ɾϑΟʔυόοΫ
    25 projectsҎ্
    

    View Slide

18. طଘͷιϦϡʔγϣϯ͕ຬͨ͞ͳ͍
    18
    ӡ༻ͷେม͞
    
    ֶशίετ
    
    GitOpsͰ͸ͳ͍ʢඞਢͰ͸ͳ͍͕😊ʣ
    Visibilityͷ໰୊ (UIͳ͠ͳͲʣ
    
    Kubernetes ApplicationͷΈ

    Multi-Tenancyӡ༻Ͱ଍Γͳ͍
    ඪ४ͳDeploymentͷ୅ΘΓʹɺRollout CRDʹมߋඞཁ
    
    Kubernetes ApplicationͷΈ

    Multi-Tenancyӡ༻Ͱ଍Γͳ͍
    Developer͕୭Ͱ΋ࣗ෼ͷαʔϏεΛߴ଎ɾ҆શɾ
    ҆৺ͰσϓϩΠͰ͖Δ
    
    (σϓϩΠதʹkubectlΛશ͘࢖Θͳ͍͍ͯ͘😊ʣ
    

    View Slide

19. PipeCDͰͰ͖Δ͜ͱ
    Quick Sync and Progressive Sync
    
    Automated Rollback
    
    Automated Deployment Analysis
    
    Con
    fi
    guration Drift Detection

    Secret Management

    Event Watcher
    
    Noti
    fi
    cation
    19
    

    View Slide

20. Quick Sync vs Progressive Sync
    20
    Sync
    GitOpsͷҙਤ
    Quick Sync͸Clusterͷঢ়ଶΛGitͷঢ়ଶΛಉظ͢ΔͨΊʹɺ͙͢Gitͷঢ়ଶ΁ભҠ
    
    Progressive Sync͸Clusterͷঢ়ଶΛGitͷঢ়ଶΛಉظ͢Δ్தʹɺઓུ (canary, bluegreen, analysis...)ʹΑΓ

    ͍͔ͭ͘ͷதؒঢ়ଶʹܦ༝͢Δɻ͔͠͠ɺ࠷ޙతʹ͸Gitͷঢ়ଶʹભҠ
    Git
    Cluster
    Sync Sync
    

    View Slide

21. Quick Sync vs Progressive Sync
    21
    Sync
    Quick Sync͸Clusterͷঢ়ଶΛGitͷঢ়ଶΛಉظ͢ΔͨΊʹɺ͙͢Gitͷঢ়ଶ΁ભҠ
    
    Progressive Sync͸Clusterͷঢ়ଶΛGitͷঢ়ଶΛಉظ͢Δ్தʹɺઓུ (canary, bluegreen, analysis...)ʹΑΓ

    ͍͔ͭ͘ͷதؒঢ়ଶʹܦ༝͢Δɻ͔͠͠ɺ࠷ޙʹ΋Gitͷঢ়ଶʹભҠ
    Git
    Cluster
    Sync Sync
    GitʹApplication directoryʹ.pipe.yamlͰArtifactͷύεɾσϓϩΠख๏ͳͲΛఆٛͰ͖Δ
    GitOpsͷҙਤ
    

    View Slide

22. Quick Sync
    22
    https://github.com/pipe-cd/examples/blob/master/kubernetes/simple/.pipe.yaml
    PipelineΛઃఆ͍ͯ͠ͳ͍৔߹͸Quick SyncΛ࣮ߦ
    
    શͯͷManifestsΛ௚઀Apply͢Δ
    

    View Slide

23. Progressive Sync
    23
    https://github.com/pipe-cd/examples/blob/master/kubernetes/canary/.pipe.yaml
    PipelineΛهࡌ͢Δ৔߹͸PipeCDͷPlanner͕มߋ಺༰ʹΑΓQuick Sync͔Progressive SyncΛ൑அɺྫ:
    
    - replicas numberͷมߋͷΈͰɺscaleͷ৔߹͸Quick Sync
    
    - pod templateͷมߋͷ৔߹͸Progressive Sync
    
    - con
    fi
    g map/secretͷมߋͷ৔߹͸Progressive Sync
    
    - deployment͝ͱʹڧ੍΋Մೳ
    

    View Slide

24. Automated Rollback
    24
    https://pipecd.dev/docs/user-guide/rolling-back-a-deployment/
    git/path/.pipe.yaml
    ్தͰ໰୊͕ൃੜͨ͠ΓɺϦϦʔε͕ѱ͍ΠϯύΫτΛ༩͍͑ͯΔͱ൑அ͞Εͨ৔߹ʹ

    ࣗಈతʹϩʔϧόοΫ͢ΔΑ͏ʹઃఆՄೳ
    

    View Slide

25. Atomated Deployment Analysis
    25
    https://pipecd.dev/docs/user-guide/automated-deployment-analysis/
    https://github.com/pipe-cd/examples/blob/master/kubernetes/analysis-by-metrics/.pipe.yaml
    ϦϦʔεͷΠϯύΫτ͸Metrics, Logs, Smoke TestͳͲͰ൑அΛߦ͏
    

    View Slide

26. Configuration Drift Detection
    26
    https://pipecd.dev/docs/user-guide/con
    fi
    guration-drift-detection/
    • ࣮ࡍͷঢ়ଶ͕ظ଴ͷঢ়ଶͱဃ཭
    
    • Ϣʔβʔ͕௚઀ௐ੔
    
    • ଞͷαʔϏε͕௚઀ௐ੔
    
    • ࣗಈతʹCon
    fi
    guration DriftΛݕ஌
    
    • WebUIͰࠩ෼Λදࣔ
    
    • ௨஌ͰΞϥʔτͷઃఆ͕Մೳ
    
    • ݱࡏ͸Con
    fi
    guration Drift͕ൃੜ͢Δͱɺ
    উखʹApply͠ͳ͍
    

    View Slide

27. Secret Management
    27
    https://pipecd.dev/docs/user-guide/sealed-secrets/
    • GitOps͸શͯͷ΋ͷΛGitʹอଘ
    
    • SecretΛ҆શʹอଘํ๏͕ඞཁ
    
    • PipeCD͸built-in secret؅ཧํ๏Λ࣋ͭ
    
    • Piped agent͕ར༻͢Δલʹ෮ݩΛߦ͏
    1
    2
    PipeCD webͰSecretͷ҉߸ԽΛߦ͏
    ҉߸Խ͞ΕͨσʔλΛGitʹஔ͘
    https://blog.stormcat.io/post/pipecd-sealed-secret/
    

    View Slide

28. Event Watcher
    28
    FluxCDͷImage Updateػೳͷઆ໌
    
    https://toolkit.
    fl
    uxcd.io/guides/image-update/
    Container Registry
    Git Repository
    ArgoCD
    

    FluxCD
    Watches images
    Makes commit to update image tags
    • GitOps͸શͯͷoperation͕Git PRΛ௨ͯ͠΍Δݪଇ
    
    • ৽͍͠container image͕Ͱ͖ͨΒɺࣗಈతʹGitΛߋ৽
    

    View Slide

29. • GitOps͸શͯͷoperation͕Git PRΛ௨ͯ͠΍Δݪଇ
    
    • ৽͍͠container image͕Ͱ͖ͨΒɺࣗಈతʹGitΛߋ৽
    
    • ͜ͷΞϓϩʔνͷ໰୊఺
    
    • CI͔ΒCD΁౉͢Artifact͸Container Image͚ͩͰͳͳ͘
    
    • Helm Chart
    
    • Kustomization Module
    
    • Terraform Module
    
    • Etc
    
    • աڈͷImage਺͕ଟ͍৔߹ʹRegistryͷWatchͷύϑΥʔϚϯε
    Event Watcher
    29
    Container Registry
    Git Repository
    ArgoCD
    

    FluxCD
    Watches images
    Makes commit to update image tags
    

    View Slide

30. Event Watcher
    30
    https://pipecd.dev/docs/user-guide/event-watcher/
    pipectl event register \
    
    
    --name=helloworld-image-update \
    
    
    --data=gcr.io/pipecd/helloworld:v0.2.0
    
    
    apiVersion: pipecd.dev/v1beta1
    
    
    kind: EventWatcher
    
    
    spec:
    
    
    events:
    
    
    - name: helloworld-image-update
    
    
    replacements:
    
    
    - file: helloworld/deployment.yaml
    
    
    yamlField: $.spec.template.spec.containers[0].image
    spec:
    
    
    containers:
    
    
    - name: helloworld
    
    
    - image: gcr.io/pipecd/helloworld:v0.1.0
    
    
    + image: gcr.io/pipecd/helloworld:v0.2.0
    • PipeCDͰ͸ Image Watcher ΑΓ Event WatcherػೳΛఏڙ
    
    • pipectlͰeventΛૹΔ͜ͱͰɺeventʹΑΓGitΛࣗಈతʹम
    ਖ਼ͯ͘͠ΕΔઃఆ͕Մೳ
    GitͷதʹeventʹΑΓमਖ਼ͷఆٛ
    Piped agent͕GitΛमਖ਼ͯ͘͠ΕΔ
    CIͰ೚ҙͷ࣌ؒͰeventΛൃੜ
    

    View Slide

31. Notification
    31
    https://pipecd.dev/docs/operator-manual/piped/con
    fi
    guring-noti
    fi
    cations/
    • ௨஌ઌͷઃఆ͕Մೳ
    
    • Slack
    
    • Webhook
    
    • ௨஌Πϕϯτͷઃఆ͕Մೳ
    
    • Deploymentͷ࣮ߦঢ়ଶ
    
    • Con
    fi
    guration drift͕ൃੜ
    
    • Application Healthͷঢ়ଶ
    
    • Pipedͷঢ়ଶ
    
    • etc
    

    View Slide

32. ୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε
    32
    શͯͷػೳ͕Kubernetes, Terraform, Lambda, CloudRun, ECSͰ࢖͑Δ
    
    GCP, AWS, AzureͳͲcloud providerΛαϙʔτ
    
    Prometheus, Datadog, CloudWatch, Stackdriver LoggingͳͲͷσʔλͰ෼ੳΛߦ͑Δ
    

    View Slide

33. CyberAgentͰPipeCDͷར༻ঢ়گ
    The structure of Team and System
    
    The numbers at CyberAgent
    
    What we have achieved
    33
    

    View Slide

34. νʔϜͱγεςϜͷߏ੒
    34
    • Platform Team
    
    • GCP্Ͱશࣾ༻Control-PlaneΛӡ༻
    
    • GCPͷFirestore & GCSͷϚωδʔυαʔϏεΛར༻
    
    • StatelessͷServer & Cache͸K8sͷ্ʹಈ͘
    
    • ֤ProjectͷSREs
    
    • Single binaryͷPiped agentΛΠϯετʔϧ
    
    • K8s cluster or Fargate or VMͷதʹಈ͘
    
    • ֤ProjectͷDevelopers
    
    • WebͰ࢖͏
    
    • GitͰPRΛૹͬͯɺσϓϩΠΛߦ͏
    

    View Slide

35. ಋೖαʔϏε਺͕૿Ճத
    0
    100
    200
    300
    400
    2020/10 2020/11 2020/12 2021/01 2021/02 2021/03
    332 Applications/Services
    ʢ࢒ΓͷϓϩδΣΫτ΋Ҡಈதʣ
    35
    

    View Slide

36. ಋೖͰΑ͔ͬͨ͜ͱ
    36
    • Platform Team 😊
    
    • શͯͷνʔϜͷσϓϩΠϝϯτΛ౷ҰͰ؅ཧ
    
    • ϓϥΫςΟεΛ࠾༻ɾීٴ͠΍͍͢
    
    • ӡ༻ָ͕
    
    • શࣾͷ֤νʔϜ͔ΒϑΟʔυόοΫΛ΋Β͑Δ
    
    • Developers 😊
    
    • kubectlͳͲ͕ෆཁͰɺߴ଎ɾ҆શɾ҆৺ͰσϓϩΠ
    
    • શͯͷσϓϩΠϝϯτ͕୯ҰͳΠϯλϑΣʔεɾ୯Ұͳϓϩηε
    
    • ৽نͷϓϩδΣΫτɾαʔϏεͷಋೖ͕͸΍͍
    
    • ΦϯϘʔσΟϯάίετ͕௿͍
    Consistency
    Flexibility
    Good Balance
    

    View Slide

37. PipeCDͷࠓޙϩʔυϚοϓ
    37
    Improve the Visibility
    
    Improve the Flexibility
    
    Add more features
    

    View Slide

38. ࠓޙͷϩʔυϚοϓ
    • VisibilityΛ޲্
    
    • Insights: Lead Time, Deployment Frequency, MTTR, Change Failure RateͳͲΛՄࢹԽ
    
    • Applicationͷঢ়ଶΛϦΞϧλΠϜతʹՄࢹԽ
    
    • Stage LogΛΑΓΘ͔Γ΍͘͢ɺ໰୊Λ͙͢ݟ͑ΔΑ͏ʹ
    
    • Multi-Provider, Multi-Tenancy
    
    • ECSͷαϙʔτ
    
    • ACLͰਂ͍Ϩϕϧͷݖݶ؅ཧ
    
    • Automated Deployment Analysis: CloudWatch, Stackdriver Logging...
    
    • AWS App Mesh, SMI
    
    • Secret Management
    
    • Sealed secretҎ֎ʹKMS, Vault΋αϙʔτ
    
    • ࣗ༝౓Λ্͛ΔͨΊʹɺϢʔβʔͷ࣮૷ͷpluginΛ࣮ߦͰ͖ΔΑ͏ʹ
    38
    

    View Slide

39. ࠷ޙʹ
    • ࠓޙ΋ੵۃతʹ։ൃΛଓ͘
    
    • ௚ۙʹCyberAgentͷશͯͷαʔϏεͰ࢖͑ΔΑ͏ʹීٴͯ͠ߦ͘
    
    • OSSͰެ։ͳͷͰɺશͯͷϑΟʔυόοΫΛ׻ܴ
    
    • OSSͷ࢓ࣄʹڵຯ͕͋ΔํɾΠϯλʔϯੜ͸TwitterͷDMΛ׻ܴ
    
    • If you like PipeCD or want to support Dev team, give it a star on GitHub!
    39
    

    View Slide

40. Thank You
    

    View Slide