Next-gen CI/CD with Gitops and Progressive Delivery

Slide 1

Slide 1 text

@kevindubois Next-gen CI/CD with Gitops and Progressive Delivery Kevin Dubois Principal Developer Advocate, Red Hat @kevindubois

Slide 2

Slide 2 text

@kevindubois Kevin Dubois ★ Java Champion ★ Principal Developer Advocate at Red Hat ★ Based in Belgium 󰎐 ★ Speak English, Dutch, French, Italian ★ Open Source Contributor (Quarkus, Camel, Knative, ..) ★ Community Organizer (BeJUG, BeCNCF) @kevindubois@mastodon.social youtube.com/@thekevindubois linkedin.com/in/kevindubois github.com/kdubois @kevindubois.com

Slide 3

Slide 3 text

@kevindubois How it started…

Slide 4

Slide 4 text

@kevindubois

Slide 5

Slide 5 text

@kevindubois

Slide 6

Slide 6 text

@kevindubois CI / CD

Slide 7

Slide 7 text

@kevindubois CI / CD Build Test Security Checks Release Deploy Stage Deploy Prod Continuous Integration Continuous Delivery Manual

Slide 8

Slide 8 text

@kevindubois Every 4 months Every week/day/hour

Slide 9

Slide 9 text

@kevindubois Continuous Developer Flow Outer loop Inner loop Pull/Merge Request Production Build / Package Code Push Debug Code Review Build Deploy Security Tests Compliance Inner loop Outer loop Developer Test

Slide 10

Slide 10 text

@kevindubois CI - CD - CD Build Test Security Checks Release Deploy Stage Deploy Prod Continuous Integration Continuous Delivery Continuous Deployment Manual Auto

Slide 11

Slide 11 text

@kevindubois Let’s play a little game :)

Slide 12

Slide 12 text

@kevindubois The application Push to give energy windmill Kafka Topic 2.Sends the interaction Dashboard: Green Energy Nickname Team Push/Tap to generate energy Cars that need energy Two teams competing (top 5 players) First wins

Slide 13

Slide 13 text

@kevindubois Architecture 3: Generate power (REST) Game Dashboard 1: Assign player Name & Team (REST) 6: Update dashboard (SSE) 2: Increment player cluster counter 4: Send power event 5: Receive power events

Slide 14

Slide 14 text

@kevindubois 14 Team Köttbullar

Slide 15

Slide 15 text

@kevindubois vs … 15

Slide 16

Slide 16 text

@kevindubois 16

Slide 17

Slide 17 text

@kevindubois 17

Slide 18

Slide 18 text

@kevindubois Team Surströmming

Slide 19

Slide 19 text

@kevindubois YOU PLAY! Scan the QR Code with your phone to play

Slide 20

Slide 20 text

@kevindubois What if we added a new feature?

Slide 21

Slide 21 text

@kevindubois Dev Ops Friday | 4:45 PM Wall of confusion

Slide 22

Slide 22 text

@kevindubois Live Coding

Slide 23

Slide 23 text

@kevindubois

Slide 24

Slide 24 text

@kevindubois Developer Flow Outer loop Inner loop Pull/Merge Request Production Build / Package Code Push Debug Code Review Build Deploy Security Tests Compliance Inner loop Outer loop Developer Test

Slide 25

Slide 25 text

@kevindubois Serverless CI Containers Built for container apps and runs on Kubernetes Designed with microservices and distributed teams in mind DevOps Serverless Runs serverless with no CI/CD engine to manage and maintain

Slide 26

Slide 26 text

@kevindubois Why Cloud-Native CI/CD? Traditional CI/CD Cloud-Native CI/CD Designed for Virtual Machines Designed for Containers and Kubernetes Require IT Ops for CI engine maintenance Pipeline as a service with no Ops overhead Plugins shared across CI engine Pipelines fully isolated from each other Plugin dependencies with undefined update cycles Everything lifecycled as container images No interoperability with Kubernetes resources Native Kubernetes resources Admin manages persistence Platform manages persistence Config baked into CI engine container Configured via Kubernetes ConfigMaps Declarative !

Slide 27

Slide 27 text

@kevindubois Tekton is a Graduated Continuous Delivery Foundation project and follows the OpenSSF best practices. Contributions from Google, Red Hat, Cloudbees, IBM, Elastic, Puppet, and many more An open-source project for providing a set of shared and standard components for building Kubernetes-style CI/CD systems https://tekton.dev

Slide 28

Slide 28 text

@kevindubois Step • Runs commands within container(builder image) • Mounts volumes, uses env vars • Eg. ‘mvn test’ or ‘git clone’ Task • A list of steps that are executed in sequential order • Takes inputs, outputs parameters Task Run • Runs a individual Task Pipeline • List of tasks defined to run in a certain order • Takes inputs, outputs parameters Pipeline Run • Runs a Pipeline Typed Decoupled Cloud Native Declarative Tekton Concepts

Slide 29

Slide 29 text

@kevindubois apiVersion: tekton.dev/v1beta1 kind: Pipeline metadata: name: wind-turbine-pipeline spec: params: - name: MANIFESTS_GIT_REPO type: string tasks: - name: git-clone params: - name: url value: $(params.GIT_REPO) workspaces: - name: output workspace: source workspaces: - name: source

Slide 30

Slide 30 text

@kevindubois Tekton Hub Search, discover and install Tekton Tasks hub.tekton.dev

Slide 31

Slide 31 text

@kevindubois Tekton CLI(tkn) •List and Describe • Pipeline • Resource • Task • Task Run • Pipeline Run •View logs • Task Run • Pipeline Run •https://github.com/tektoncd/cli

Slide 32

Slide 32 text

@kevindubois Gitops

Slide 33

Slide 33 text

@kevindubois What is GitOps? Treat everything as code Git is the single source of truth Operations through Git workflows

Slide 34

Slide 34 text

@kevindubois CI/CD Engines Jenkins Spinnaker Tekton Concourse CI …... CI/CD versus GitOps Desired State Cluster State Observe State Take Action GitOps Engines ACM, ArgoCD, FluxCD Razee, Faros Desired State Cluster State

Slide 35

Slide 35 text

@kevindubois ArgoCD Sync Monitor Detect drift Take action Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. Cluster and application configuration versioned in Git Automatically syncs configuration from Git to clusters Drift detection, visualization and correction

Slide 36

Slide 36 text

@kevindubois Source Git Repository Image Registry CI GitOps Application Delivery Model

Slide 37

Slide 37 text

@kevindubois Source Git Repository Image Registry CI Config Git Repository Kubernetes CD Pull Request / Commit Push Pull GitOps Application Delivery Model

Slide 38

Slide 38 text

@kevindubois GitOps Application Delivery Model Push Pull Pull Request Source Git Repository Image Registry Config Git Repository Kubernetes Deploy Monitor Detect drift CD Take action

Slide 39

Slide 39 text

@kevindubois V2 Scan the QR Code with your phone to play

Slide 40

Slide 40 text

@kevindubois By default: a big bang / all or nothing release

Slide 41

Slide 41 text

@kevindubois Progressive Delivery

Slide 42

Slide 42 text

@kevindubois What is Progressive Delivery? ● No Big Bang ● Deploy != Release ● Metrics ● Subset of Users

Slide 43

Slide 43 text

@kevindubois Why Progressive Delivery? ● Decreases Downtime ● Limits the Tragedy ● Deploy & Release to Production faster ● Less mocking or setting up unreliable ‘fake’ services

Slide 44

Slide 44 text

@kevindubois Delivery Techniques

Slide 45

Slide 45 text

@kevindubois Blue Green Deployment ● All Or Nothing ● Quick Rollback 45

Slide 46

Slide 46 text

@kevindubois Canary Releases ● Small Percentage ● Increase depending on metrics 46

Slide 47

Slide 47 text

@kevindubois Dark Launches ● Mirroring Traffic ● Dark Canaries ● Feature Flags 47

Slide 48

Slide 48 text

@kevindubois 48

Slide 49

Slide 49 text

@kevindubois The New Pyramid? 49

Slide 50

Slide 50 text

@kevindubois How to accomplish Progressive Delivery 50

Slide 51

Slide 51 text

@kevindubois Accomplishing Progressive Delivery with 51

Slide 52

Slide 52 text

@kevindubois Blue - Green apiVersion: v1 kind: Service metadata: name: my-service labels: app: mystuff spec: ports: - name: http port: 8000 selector: inservice: mypods type: LoadBalancer apiVersion: apps/v1 kind: Deployment metadata: name: mynode-deployment spec: replicas: 1 selector: matchLabels: app: mynode template: metadata: labels: app: mynode spec: containers: - name: mynode image: quay.io/rhdevelopers/mynode:v1 ports: - containerPort : 8000 kubectl label pod -l app=mynode inservice=mypods 52

Slide 53

Slide 53 text

@kevindubois Canary Releases kubectl scale deployment myapp-v1 --replicas=3 kubectl scale deployment myapp-v2 --replicas=1 53

Slide 54

Slide 54 text

@kevindubois

Slide 55

Slide 55 text

@kevindubois Controlling Microservices with a Service Mesh Code Independent (Polyglot) • Intelligent Routing and Load-Balancing • Smarter Canary Releases • Dark Launch • Chaos: Fault Injection • Resilience: Circuit Breakers • Observability & Telemetry: Metrics and Tracing • Security: Encryption & Authorization • Fleet wide policy enforcement 55

Slide 56

Slide 56 text

@kevindubois Istio Architecture Control Plane The data plane is composed of a set of intelligent proxies (Envoy) deployed as sidecars. These proxies mediate and control all network communication between microservices. They also collect and report telemetry on all mesh traffic. The control plane manages and conﬁgures the proxies to route traffic. Data Plane

Slide 57

Slide 57 text

@kevindubois Pod Container JVM Service A Sidecar Container Pod Container JVM Service C Sidecar Container Pod Container JVM Service B Sidecar Container The sidecar intercepts all network traffic 57

Slide 58

Slide 58 text

@kevindubois Canary Release apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: recommendation spec: hosts: - recommendation http: - route: - destination: host: recommendation subset: version-v1 weight: 75 - destination: host: recommendation subset: version-v2 weight: 25 58

Slide 59

Slide 59 text

@kevindubois Shadowing Traffic apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: recommendation spec: hosts: - recommendation http: - route: - destination: host: recommendation subset: version-v1 mirror: host: recommendation subset: version-v2 59

Slide 60

Slide 60 text

@kevindubois Dark Canary apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: recommendation spec: hosts: - recommendation http: - match: - headers: end-user: exact: Alexandra route: - destination: host: recommendation subset: version-v2 - route: - destination: host: recommendation subset: version-v1 60

Slide 61

Slide 61 text

@kevindubois Automated Progressive Delivery 61

Slide 62

Slide 62 text

@kevindubois Argo Rollouts 62

Slide 63

Slide 63 text

@kevindubois Argo Rollouts ArgoCD detects change rollout Monitors Data 63

Slide 64

Slide 64 text

@kevindubois Rolling out automatically apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: rollouts-demo labels: app: rollouts-demo spec: strategy: canary: steps: - setWeight: 20 - pause: duration: "1m" - setWeight: 50 - pause: duration: "2m" canaryService: rollouts-demo-canary stableService: rollouts-demo-backend trafficRouting: istio: virtualService: name: rollout-vsvc routes: - primary … 64

Slide 65

Slide 65 text

@kevindubois “Smart” Progressive Delivery Based on Metrics 65

Slide 66

Slide 66 text

@kevindubois Metrics Based Rollouts strategy: canary: analysis: args: - name: service-name value: rollouts-demo-canary.canary.svc.cluster.local templates: - templateName: success-rate canaryService: rollouts-demo-canary stableService: rollouts-demo-stable trafficRouting: istio: virtualService: name: rollout-vsvc routes: - primary steps: - setWeight: 30 - pause: { duration: 20s } - setWeight: 40 - pause: { duration: 10s } - setWeight: 60 - pause: { duration: 10s } - setWeight: 80 - pause: { duration: 5s } - setWeight: 90 - pause: { duration: 5s } - setWeight: 100 - pause: { duration: 5s } 66

Slide 67

Slide 67 text

@kevindubois apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: success-rate spec: args: - name: service-name metrics: - name: success-rate interval: 10s successCondition: len(result) == 0 || result[0] >= 0.95 failureLimit: 2 provider: prometheus: address: https://internal:demo@prometheus.istio-system.svc.cluster.local:9090 query: | sum(irate(istio_requests_total{ reporter="source", destination_service=~"{{args.service-name}}", response_code!~"5.*"}[30s]) ) Metrics Based Rollouts 67

Slide 68

Slide 68 text

@kevindubois Progressive Delivery Demo @kevindubois 68 🚀🚀🚀

Slide 69

Slide 69 text

@kevindubois Final Notes ● State is always hard ○ start with stateless; work with features; non-destructive schema changes; event-driven architectures (use eg. Debezium to work with ‘classic’ DBs). ● Step by Step ● Embrace GitOps ● If you haven’t automatically destroyed something by mistake, you aren’t automating enough ● Demos ○ https://dn.dev/istio-tutorial ○ https://github.com/kdubois/progressive-delivery ○ https://github.com/redhat-developer-demos/bubbles-progressive-delivery ○ github.com/redhat-developer-demos/quinoa-wind-turbine 69

Slide 70

Slide 70 text

@kevindubois Free Developer e-Books! https://developers.redhat.com/eventtutorials

Slide 71

Slide 71 text

@kevindubois Start exploring in the OpenShift Sandbox. Learn containers, Kubernetes, and OpenShift in your browser. developers.redhat.com/developer-sandbox Try Red Hat's products and technologies without setup or configuration.

Slide 72

Slide 72 text

@kevindubois Thank you! @kevindubois@mastodon.social youtube.com/@thekevindubois linkedin.com/in/kevindubois github.com/kdubois @kevindubois.com

Slide 73

Slide 73 text

@kevindubois Join Red Hat Developer. Build here. Go anywhere. facebook.com/RedHatDeveloper youtube.com/RedHatDevelopers twitter.com/rhdevelopers linkedin.com/showcase/red-hat-developer Thank you!