The Journey of Software Delivery

Transcript

1. Le Van Nghia, CyberAgent, Jan 20, 2021

   @nghialv
   The Journey of Software Delivery
   Developer Productivity Study
   

   View Slide

2. About me
   @nghialv
   @nghialv2607
   

   View Slide

3. About me - CyberAgent
   • Developer Productivity
   
   • OpenSaasStudio
   
   • ABEMA
   
   • Ameba
   • Building PipeCD
   
   • Building Work
   f
   l
   ow Automation System
   
   • Building Feature Flags, A/B Testing System
   
   • Building Monitoring System
   
   • Building Deployment Tool
   
   • Infrastructure/Microservices Related Works
   
   • Building FRESH iOS App
   
   • Rebuilding Ameba iOS App
   

   View Slide

4. About me - DIY
   https://twitter.com/nghialv2607/status/1345936214407274496?s=20
   

   View Slide

5. Agenda
   • The Fundamentals
   
   • Principles and Practices
   
   • Progressive Delivery
   
   • GitOps
   

   View Slide

6. The Fundamentals
   Why do we need

   How do we achive the goal

   Basic concepts

   Common pitfalls
   

   View Slide

7. What is goal of software professionals
   If somebody thinks of a good idea, 

   how do we deliver it to users as quickly as possible?
   

   View Slide

8. How do we achieve that goal
   The feedback must be received as soon as possible
   Every change should trigger the feedback process
   Analyse and act on the feedback
   

   View Slide

9. Requiring a robust Delivery Process
   Reliable Rapid Low-Risk
   

   View Slide

10. CI/CD
    Apply CI and CD systems to accelerate the delivery process
    Actions
    

    View Slide

11. CI/CD
    When people say “CI/CD,” they are only talking about Continuous Integration. 

    Nobody is talking about (or practicing) Continuous Deployment. AT ALL. 

    It’s like we have all forgotten it exists. It's time to change that.
    
    Charity Majors
    

    View Slide

12. CI != CD - They should be seperated
    Artifact Storage
    Verifying and Analysing the Impact
    Application Code 

    (.go, .java, .js...)
    Infrastructure Code 

    (.tf ...)
    Con
    f
    i
    guration Code 

    (.yaml ...)
    DockerHub, GCR, ECR...
    GCS, S3...
    Git Repository
    Code Storage
    Actions
    Continuous Integration
    Test Code
    Git Repository
    Host Environment
    Artifact
    Continuous Delivery
    Artifact
    Build and Save Artfacts
    Cloud
    User
    Low-risk actions: release strategy, rollback
    Deployment Dependency Management
    Provisioning, Installing Artifact
    

    View Slide

13. Continous Delivery != Continuous Deployment
    Continuous Deployment means that every change goes through the pipeline and
    automatically gets put into production, resulting in many production deployments every day.
    In order to do Continuous Deployment you must be doing Continuous Delivery.
    Continuous Delivery just means that you are able to do frequent deployments but may
    choose not to do it, usually due to businesses preferring a slower rate of deployment.
    Artifact
    Continuous Deployment
    Dev, Test Env
    Artifact
    Continuous Delivery
    Prod Env
    An example
    

    View Slide

14. Deploy != Release
    Deployment is the process for installing the new version of artifact on prod environment. 

    When we say a new version of software is deployed, we mean it is running somewhere in
    the production environment.
    Releasing is the process of moving production tra
    f
    f
    i
    c to the new version.
    
    When we say a version of a software is released, we mean that it is responsible for
    serving production tra
    f
    f
    i
    c.
    Deployment need not expose customers to a new version of your service.
    
    
    Given this definition, deployment can be an almost zero-risk activity.
    
    
    Turbine Labs
    

    View Slide

15. Principles and Practices of Software Delivery
    

    View Slide

16. Achieving a robust Delivery Process
    Reliable Rapid Low-Risk
    

    View Slide

17. Reliable Process
    Uni
    f
    i
    ed process for all

    

    Idempotent and repeatable process

    

    Automation

    Visibility and Auditability

    

    Everyone is responsible
    
    

    Take DX seriously
    

    View Slide

18. Reliable Process
    Uni
    f
    i
    ed process for all

    

    Idempotent and repeatable process

    

    Automation

    Visibility and Auditability

    

    Everyone is responsible
    
    

    Take DX seriously
    The key to deploying any application in a realiable, consistent manner is
    constant practice.
    
    Use the same process to do all operations of all applications to every
    environments
    Deploying new version

    Scaling service

    Rolling back

    DB migration

    Emergency
    f
    i
    x

    Kubernetes app

    Serverless app

    Infrastructure app
    Development

    Testing

    Staging

    Production
    

    View Slide

19. Reliable Process
    Uni
    f
    i
    ed process for all

    

    Idempotent and repeatable process

    

    Automation

    Visibility and Auditability

    

    Everyone is responsible
    
    

    Take DX seriously
    Avoid side effect in the process.
    The version of tools/dependencies should be speci
    f
    i
    ed.
    

    Artifacts should be immutable and reused in all environment
    .
    
    
    containers:
    
    
    - image: gcr.io/awesome-docker-image:latest
    bases:
    
    
    - git@github.com:awesome/kustomization-base?ref=v0.1.2-15
    containers:
    
    
    - image: gcr.io/awesome-docker-image:v0.1.0
    bases:
    
    
    - git@github.com:awesome/kustomization-base
    Do not use latest
    Do not use branch
    

    View Slide

20. Reliable Process
    Uni
    f
    i
    ed process for all

    

    Idempotent and repeatable process

    

    Automation

    Visibility and Auditability

    

    Everyone is responsible
    
    

    Take DX seriously People introduce errors, and are not auditable
    

    View Slide

21. Reliable Process
    Uni
    f
    i
    ed process for all

    

    Idempotent and repeatable process

    

    Automation

    Visibility and Auditability

    

    Everyone is responsible
    
    

    Take DX seriously
    Clearly log the activities

    Keep everything in version control
    Make developer loves the process
    
    Less repository is better life
    Everybody is reponsible to build/use/improve the delivery process

    Even business man, manager can easy to release their features
    

    View Slide

22. Rapid Process
    Small change
    
    Do more frequently

    

    Automate almost everything
    

    View Slide

23. Low-risk Process
    Test
    
    Static analysis, unit test, integration test, e2e test
    
    Small changes
    
    Veri
    f
    i
    cation based on metrics, logs, events
    
    Rolling back
    
    Automation
    
    Progressive Delivery
    

    View Slide

24. Progressive Delivery
    

    View Slide

25. What is Progressive Delivery
    Continuous Delivery helps team move faster by making each change small and manageable.

    

    Progressive Delivery helps team move safer by reducing the risk of each change by controlling
    the audience exposed.
    Progressive Delivery is continous delivery with
    f
    i
    ne-grained control over the blast radius.
    
    James Governor, ReadMonk
    

    View Slide

26. How to enable Progressive Delivery
    Progressive Delivery comes in two complementary
    f
    l
    avors: environment and user.
    
    You can deliver progressively to environments using canary or blue-green
    deployments to limit exposure when new version is deployed (instance, region, env...).
    
    Progressive Delivery for users, on the other hand, lets you limit the exposure of which
    users see a new change.
    Decide the strategy of exposure
    Analyse the feedback to decide whether to go on or rollback
    

    View Slide

27. How to enable Progressive Delivery - Canary
    Canary Release + Canary Analysis + Automated Rollback
    Image Source
    

    View Slide

28. How to enable Progressive Delivery - Feature Flags
    Image Source
    Dev Team
    Beta Testers
    All Users
    Deploy
    f
    i
    rst and control the exposure from management page or by automation
    

    View Slide

29. GitOps
    

    View Slide

30. What is GitOps?
    Operations by Pull Request - weaveworks
    The core idea
    
    Store the desired state of application declaratively in the Git repository 

    and Sync them to host environment.
    Git Repository
    Host Environment
    Sync to match
    Cloud
    User
    What artifact should be released
    What strategy should be used
    What is the desired con
    f
    i
    guration
    Agent
    PR to change image from v1 to v2
    PR to scale app from 100 to 200 replicas
    PR to update revert commit to rollback
    PR to update LB con
    f
    i
    guration
    

    View Slide

31. What is GitOps?
    GitOps is only for CD not CI
    Artifact Storage
    Application Code 

    (.go, .java, .js...)
    Infrastructure Code 

    (.tf ...)
    Con
    f
    i
    guration Code 

    (.yaml ...)
    DockerHub, GCR, ECR...
    GCS, S3...
    Git Repository
    Code Storage
    Actions
    Continuous Integration
    Git Repository
    Host Environment
    Continuous Delivery
    Artifact
    Cloud
    User
    GitOps Agent
    Git Repository
    What artifact should be released
    What strategy should be used
    What is the desired con
    f
    i
    guration
    

    View Slide

32. Benefits of GitOps
    Unifying Deployment, Monitoring and
    Management.

    Git as the single source of truth of a
    system's desired state.
    
    

    ALL intended operations are committed

    by pull request

    

    ALL di
    f
    f
    s between inteneded and observed
    state with automatic convergence

    

    ALL changes are observable, veri
    f
    i
    able

    and auditable

    

    https://www.weave.works/blog/what-is-gitops-really
    

    View Slide

33. Conclusion
    • Same process for all operations, all applications, all environments
    
    • Small change, frequent deployment
    
    • Automate almost everything
    
    • Delivery process is the whole teams's responsibility
    
    • Take developer experience seriously
    

    View Slide

34. What is next
    Let's try PipeCD!
    https://pipecd.dev
    

    View Slide

35. Thank You
    

    View Slide