Le Van Nghia, CyberAgent, Jan 20, 2021
 @nghialv The Journey of Software Delivery Developer Productivity Study

About me @nghialv @nghialv2607

About me - CyberAgent • Developer Productivity • OpenSaasStudio • ABEMA • Ameba • Building PipeCD • Building Work f l ow Automation System • Building Feature Flags, A/B Testing System • Building Monitoring System • Building Deployment Tool • Infrastructure/Microservices Related Works • Building FRESH iOS App • Rebuilding Ameba iOS App

About me - DIY https://twitter.com/nghialv2607/status/1345936214407274496?s=20

Agenda • The Fundamentals • Principles and Practices • Progressive Delivery • GitOps

The Fundamentals Why do we need
 How do we achive the goal
 Basic concepts
 Common pitfalls

What is goal of software professionals If somebody thinks of a good idea, 
 how do we deliver it to users as quickly as possible?

How do we achieve that goal The feedback must be received as soon as possible Every change should trigger the feedback process Analyse and act on the feedback

Requiring a robust Delivery Process Reliable Rapid Low-Risk

CI/CD Apply CI and CD systems to accelerate the delivery process Actions

CI/CD When people say “CI/CD,” they are only talking about Continuous Integration. 
 Nobody is talking about (or practicing) Continuous Deployment. AT ALL. 
 It’s like we have all forgotten it exists. It's time to change that. Charity Majors

CI != CD - They should be seperated Artifact Storage Verifying and Analysing the Impact Application Code 
 (.go, .java, .js...) Infrastructure Code 
 (.tf ...) Con f i guration Code 
 (.yaml ...) DockerHub, GCR, ECR... GCS, S3... Git Repository Code Storage Actions Continuous Integration Test Code Git Repository Host Environment Artifact Continuous Delivery Artifact Build and Save Artfacts Cloud User Low-risk actions: release strategy, rollback Deployment Dependency Management Provisioning, Installing Artifact

Continous Delivery != Continuous Deployment Continuous Deployment means that every change goes through the pipeline and automatically gets put into production, resulting in many production deployments every day. In order to do Continuous Deployment you must be doing Continuous Delivery. Continuous Delivery just means that you are able to do frequent deployments but may choose not to do it, usually due to businesses preferring a slower rate of deployment. Artifact Continuous Deployment Dev, Test Env Artifact Continuous Delivery Prod Env An example

Deploy != Release Deployment is the process for installing the new version of artifact on prod environment. 
 When we say a new version of software is deployed, we mean it is running somewhere in the production environment. Releasing is the process of moving production tra f f i c to the new version. When we say a version of a software is released, we mean that it is responsible for serving production tra f f i c. Deployment need not expose customers to a new version of your service. Given this definition, deployment can be an almost zero-risk activity. Turbine Labs

Principles and Practices of Software Delivery

Achieving a robust Delivery Process Reliable Rapid Low-Risk

Reliable Process Uni f i ed process for all
 
 Idempotent and repeatable process
 
 Automation
 Visibility and Auditability
 
 Everyone is responsible 
 Take DX seriously

Reliable Process Uni f i ed process for all
 
 Idempotent and repeatable process
 
 Automation
 Visibility and Auditability
 
 Everyone is responsible 
 Take DX seriously The key to deploying any application in a realiable, consistent manner is constant practice. Use the same process to do all operations of all applications to every environments Deploying new version
 Scaling service
 Rolling back
 DB migration
 Emergency f i x
 Kubernetes app
 Serverless app
 Infrastructure app Development
 Testing
 Staging
 Production

Reliable Process Uni f i ed process for all
 
 Idempotent and repeatable process
 
 Automation
 Visibility and Auditability
 
 Everyone is responsible 
 Take DX seriously Avoid side effect in the process. The version of tools/dependencies should be speci f i ed. 
 Artifacts should be immutable and reused in all environment . containers: - image: gcr.io/awesome-docker-image:latest bases: - [email protected]:awesome/kustomization-base?ref=v0.1.2-15 containers: - image: gcr.io/awesome-docker-image:v0.1.0 bases: - [email protected]:awesome/kustomization-base Do not use latest Do not use branch

Reliable Process Uni f i ed process for all
 
 Idempotent and repeatable process
 
 Automation
 Visibility and Auditability
 
 Everyone is responsible 
 Take DX seriously People introduce errors, and are not auditable

Reliable Process Uni f i ed process for all
 
 Idempotent and repeatable process
 
 Automation
 Visibility and Auditability
 
 Everyone is responsible 
 Take DX seriously Clearly log the activities
 Keep everything in version control Make developer loves the process Less repository is better life Everybody is reponsible to build/use/improve the delivery process
 Even business man, manager can easy to release their features

Rapid Process Small change Do more frequently
 
 Automate almost everything

Low-risk Process Test Static analysis, unit test, integration test, e2e test Small changes Veri f i cation based on metrics, logs, events Rolling back Automation Progressive Delivery

Progressive Delivery

What is Progressive Delivery Continuous Delivery helps team move faster by making each change small and manageable.
 
 Progressive Delivery helps team move safer by reducing the risk of each change by controlling the audience exposed. Progressive Delivery is continous delivery with f i ne-grained control over the blast radius. James Governor, ReadMonk

How to enable Progressive Delivery Progressive Delivery comes in two complementary f l avors: environment and user. You can deliver progressively to environments using canary or blue-green deployments to limit exposure when new version is deployed (instance, region, env...). Progressive Delivery for users, on the other hand, lets you limit the exposure of which users see a new change. Decide the strategy of exposure Analyse the feedback to decide whether to go on or rollback

How to enable Progressive Delivery - Canary Canary Release + Canary Analysis + Automated Rollback Image Source

How to enable Progressive Delivery - Feature Flags Image Source Dev Team Beta Testers All Users Deploy f i rst and control the exposure from management page or by automation

GitOps

What is GitOps? Operations by Pull Request - weaveworks The core idea Store the desired state of application declaratively in the Git repository 
 and Sync them to host environment. Git Repository Host Environment Sync to match Cloud User What artifact should be released What strategy should be used What is the desired con f i guration Agent PR to change image from v1 to v2 PR to scale app from 100 to 200 replicas PR to update revert commit to rollback PR to update LB con f i guration

What is GitOps? GitOps is only for CD not CI Artifact Storage Application Code 
 (.go, .java, .js...) Infrastructure Code 
 (.tf ...) Con f i guration Code 
 (.yaml ...) DockerHub, GCR, ECR... GCS, S3... Git Repository Code Storage Actions Continuous Integration Git Repository Host Environment Continuous Delivery Artifact Cloud User GitOps Agent Git Repository What artifact should be released What strategy should be used What is the desired con f i guration

Benefits of GitOps Unifying Deployment, Monitoring and Management.
 Git as the single source of truth of a system's desired state. 
 ALL intended operations are committed
 by pull request
 
 ALL di f f s between inteneded and observed state with automatic convergence
 
 ALL changes are observable, veri f i able
 and auditable
 
 https://www.weave.works/blog/what-is-gitops-really

Conclusion • Same process for all operations, all applications, all environments • Small change, frequent deployment • Automate almost everything • Delivery process is the whole teams's responsibility • Take developer experience seriously

What is next Let's try PipeCD! https://pipecd.dev

Thank You