$30 off During Our Annual Pro Sale. View Details »

Next-gen CI/CD with Gitops and Progressive Delivery | fooConf Helsinki Finland

Next-gen CI/CD with Gitops and Progressive Delivery | fooConf Helsinki Finland

Even though you might be using Continuous Integration and Delivery, chances are you are still not 100% sure things will roll out without a glitch once you go to production. There will always be differences between environments and a risk for unforeseen issues related to your production environment and/or actual load, leading to potential disruption to your customers.

Progressive delivery is the next step after Continuous Delivery to test your application in production before it becomes fully available to all your user bases.

Embrace GitOps and Progressive Delivery with techniques like blue-green, canary release, shadowing traffic, dark launches and automatic metrics-based rollouts to validate the application in production using Kubernetes and tools like Istio, Prometheus, ArgoCD, and Argo Rollouts.

Come to this session to learn about Progressive Delivery in action using Kubernetes. And to have some fun as well as we release a car racing game you can participate in :)

Kevin Dubois

November 07, 2023
Tweet

More Decks by Kevin Dubois

Other Decks in Technology

Transcript

  1. @kevindubois
    Next-gen CI/CD with Gitops
    and Progressive Delivery
    Kevin Dubois
    Principal Developer Advocate, Red Hat
    @kevindubois

    View Slide

  2. @kevindubois
    How it started…

    View Slide

  3. @kevindubois

    View Slide

  4. @kevindubois
    CI - CD - CD
    Build Test
    Security
    Checks
    Release
    Deploy
    Stage
    Deploy
    Prod
    Continuous Integration
    Continuous Delivery
    Continuous Deployment
    Manual
    Auto

    View Slide

  5. @kevindubois
    Every 4 months Every week/day/hour

    View Slide

  6. @kevindubois
    Let’s start with playing a little game :)

    View Slide

  7. @kevindubois
    The application
    Push to give energy windmill Kafka
    Topic
    2.Sends the interaction
    Dashboard: Green Energy
    Nickname Team
    Push/Tap to
    generate energy
    Cars that
    need energy
    Two teams competing
    (top 5 players)
    First wins

    View Slide

  8. @kevindubois
    Architecture
    3: Generate power
    (REST)
    Game
    Dashboard
    1: Assign player
    Name & Team (REST)
    6: Update dashboard (SSE)
    2: Increment player cluster
    counter
    4: Send power event
    5: Receive power events

    View Slide

  9. @kevindubois
    YOU PLAY!
    Scan the QR Code with your phone to play

    View Slide

  10. @kevindubois
    What if we added a new feature?

    View Slide

  11. @kevindubois
    Dev Ops
    Friday | 4:45 PM
    Wall of confusion

    View Slide

  12. @kevindubois

    View Slide

  13. @kevindubois
    Developer Flow
    Outer
    loop
    Inner
    loop
    Pull/Merge Request
    Production
    Build /
    Package
    Code
    Push
    Debug
    Code
    Review Build
    Deploy
    Security
    Tests
    Compliance
    Inner loop Outer loop
    Developer
    Test

    View Slide

  14. @kevindubois
    Serverless CI
    Containers
    Built for container apps
    and runs on Kubernetes
    Designed with
    microservices and
    distributed teams in mind
    DevOps
    Serverless
    Runs serverless with no
    CI/CD engine to manage
    and maintain

    View Slide

  15. @kevindubois
    Why Cloud-Native CI/CD?
    Traditional CI/CD Cloud-Native CI/CD
    Designed for Virtual Machines Designed for Containers and Kubernetes
    Require IT Ops for CI engine maintenance Pipeline as a service with no Ops overhead
    Plugins shared across CI engine Pipelines fully isolated from each other
    Plugin dependencies with undefined update cycles Everything lifecycled as container images
    No interoperability with Kubernetes resources Native Kubernetes resources
    Admin manages persistence Platform manages persistence
    Config baked into CI engine container Configured via Kubernetes ConfigMaps
    Declarative !

    View Slide

  16. @kevindubois
    Tekton is a Graduated Continuous Delivery Foundation project and follows the OpenSSF best practices.
    Contributions from Google, Red Hat, Cloudbees, IBM, Elastic, Puppet, and many more
    An open-source project for providing a set of shared and standard
    components for building Kubernetes-style CI/CD systems
    https://tekton.dev

    View Slide

  17. @kevindubois
    Step
    • Runs commands within
    container(builder image)
    • Mounts volumes, uses env
    vars
    • Eg. ‘mvn test’ or ‘git clone’
    Task
    • A list of steps that are
    executed in sequential order
    • Takes inputs, outputs
    parameters
    Task Run
    • Runs a individual Task
    Pipeline
    • List of tasks defined to run in
    a certain order
    • Takes inputs, outputs
    parameters
    Pipeline Run
    • Runs a Pipeline
    Typed Decoupled
    Cloud Native Declarative
    Tekton
    Concepts

    View Slide

  18. @kevindubois
    apiVersion: tekton.dev/v1beta1
    kind: Pipeline
    metadata:
    name: wind-turbine-pipeline
    spec:
    params:
    - name: MANIFESTS_GIT_REPO
    type: string
    tasks:
    - name: git-clone
    params:
    - name: url
    value: $(params.GIT_REPO)
    workspaces:
    - name: output
    workspace: source
    workspaces:
    - name: source

    View Slide

  19. @kevindubois
    Tekton Hub
    Search, discover and
    install Tekton Tasks
    hub.tekton.dev

    View Slide

  20. @kevindubois
    Tekton CLI(tkn)
    •List and Describe
    • Pipeline
    • Resource
    • Task
    • Task Run
    • Pipeline Run
    •View logs
    • Task Run
    • Pipeline Run
    •https://github.com/tektoncd/cli

    View Slide

  21. @kevindubois
    Gitops

    View Slide

  22. @kevindubois
    What is GitOps?
    Treat
    everything as
    code
    Git is the
    single source
    of truth
    Operations
    through Git
    workflows

    View Slide

  23. @kevindubois
    CI/CD Engines
    Jenkins
    Spinnaker
    Tekton
    Concourse CI
    …...
    CI/CD versus GitOps
    Desired
    State
    Cluster
    State
    Observe
    State
    Take
    Action
    GitOps
    Engines
    ACM, ArgoCD,
    FluxCD
    Razee, Faros
    Desired
    State
    Cluster
    State

    View Slide

  24. @kevindubois
    ArgoCD
    Sync
    Monitor
    Detect
    drift
    Take
    action
    Argo CD is a declarative, GitOps continuous
    delivery tool for Kubernetes.
    Cluster and application configuration
    versioned in Git
    Automatically syncs configuration from Git to
    clusters
    Drift detection, visualization and correction

    View Slide

  25. @kevindubois
    Let’s deploy our new feature in a Modern, Automated, Gitops way!

    View Slide

  26. @kevindubois
    Live Coding

    View Slide

  27. @kevindubois
    Source
    Git Repository
    Image
    Registry
    CI
    GitOps Application Delivery Model

    View Slide

  28. @kevindubois
    Source
    Git Repository
    Image
    Registry
    CI
    Config
    Git Repository
    Kubernetes
    CD
    Pull Request
    / Commit
    Push
    Pull
    GitOps Application Delivery Model

    View Slide

  29. @kevindubois
    GitOps Application Delivery Model
    Push
    Pull
    Pull
    Request
    Source
    Git
    Repository
    Image
    Registry
    Config
    Git
    Repository
    Kubernetes
    Deploy
    Monitor
    Detect
    drift
    CD
    Take
    action

    View Slide

  30. @kevindubois
    V2
    Scan the QR Code with your phone to play

    View Slide

  31. @kevindubois
    By default: a big bang / all or nothing release

    View Slide

  32. @kevindubois
    Progressive Delivery

    View Slide

  33. @kevindubois
    What is Progressive Delivery?
    Build Test
    Security
    Checks
    Release
    Deploy
    Stage
    Deploy
    Prod
    Continuous Integration
    Continuous Delivery
    Continuous Deployment
    Manual
    Auto

    View Slide

  34. @kevindubois
    What is Progressive Delivery?
    ● No Big Bang
    ● Deploy != Release
    ● Metrics
    ● Subset of Users

    View Slide

  35. @kevindubois
    Why Progressive Delivery?
    ● Decreases Downtime
    ● Limits the Tragedy
    ● Deploy & Release to Production faster
    ● Less mocking or setting up unreliable ‘fake’ services

    View Slide

  36. @kevindubois
    Delivery Techniques

    View Slide

  37. @kevindubois
    Blue Green Deployment
    ● All Or Nothing
    ● Quick Rollback
    37

    View Slide

  38. @kevindubois
    Canary Releases
    ● Small Percentage
    ● Increase depending on metrics
    38

    View Slide

  39. @kevindubois
    Dark Launches
    ● Mirroring Traffic
    ● Dark Canaries
    ● Feature Flags
    39

    View Slide

  40. @kevindubois
    40

    View Slide

  41. @kevindubois
    41

    View Slide

  42. @kevindubois
    The New Pyramid?
    42

    View Slide

  43. @kevindubois
    How to accomplish Progressive Delivery
    43

    View Slide

  44. @kevindubois
    Accomplishing
    Progressive Delivery with
    44

    View Slide

  45. @kevindubois
    Blue - Green
    apiVersion: v1
    kind: Service
    metadata:
    name: my-service
    labels:
    app: mystuff
    spec:
    ports:
    - name: http
    port: 8000
    selector:
    inservice: mypods
    type: LoadBalancer
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: mynode-deployment
    spec:
    replicas: 1
    selector:
    matchLabels:
    app: mynode
    template:
    metadata:
    labels:
    app: mynode
    spec:
    containers:
    - name: mynode
    image: quay.io/rhdevelopers/mynode:v1
    ports:
    - containerPort : 8000
    kubectl label pod -l app=mynode inservice=mypods
    45

    View Slide

  46. @kevindubois
    Canary Releases
    kubectl scale deployment myapp-v1 --replicas=3
    kubectl scale deployment myapp-v2 --replicas=1
    46

    View Slide

  47. @kevindubois

    View Slide

  48. @kevindubois
    Controlling Microservices with a Service Mesh
    Code Independent (Polyglot)
    • Intelligent Routing and Load-Balancing
    • Smarter Canary Releases
    • Dark Launch
    • Chaos: Fault Injection
    • Resilience: Circuit Breakers
    • Observability & Telemetry: Metrics and Tracing
    • Security: Encryption & Authorization
    • Fleet wide policy enforcement
    48

    View Slide

  49. @kevindubois
    Istio Architecture
    Control Plane
    The data plane is composed
    of a set of intelligent proxies
    (Envoy) deployed as
    sidecars. These proxies
    mediate and control all
    network communication
    between microservices.
    They also collect and report
    telemetry on all mesh
    traffic.
    The control plane manages
    and configures the proxies
    to route traffic.
    Data Plane

    View Slide

  50. @kevindubois
    Pod
    Container
    JVM
    Service A
    Sidecar Container
    Pod
    Container
    JVM
    Service C
    Sidecar Container
    Pod
    Container
    JVM
    Service B
    Sidecar Container
    The sidecar intercepts all network traffic
    50

    View Slide

  51. @kevindubois
    Canary Release
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    name: recommendation
    spec:
    hosts:
    - recommendation
    http:
    - route:
    - destination:
    host: recommendation
    subset: version-v1
    weight: 75
    - destination:
    host: recommendation
    subset: version-v2
    weight: 25
    51

    View Slide

  52. @kevindubois
    Shadowing Traffic
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    name: recommendation
    spec:
    hosts:
    - recommendation
    http:
    - route:
    - destination:
    host: recommendation
    subset: version-v1
    mirror:
    host: recommendation
    subset: version-v2
    52

    View Slide

  53. @kevindubois
    Dark Canary apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    name: recommendation
    spec:
    hosts:
    - recommendation
    http:
    - match:
    - headers:
    end-user:
    exact: Alexandra
    route:
    - destination:
    host: recommendation
    subset: version-v2
    - route:
    - destination:
    host: recommendation
    subset: version-v1
    53

    View Slide

  54. @kevindubois
    Automated Progressive Delivery
    54

    View Slide

  55. @kevindubois
    Argo Rollouts
    55

    View Slide

  56. @kevindubois
    Argo Rollouts
    ArgoCD detects change
    rollout
    Monitors Data
    56

    View Slide

  57. @kevindubois
    Rolling out automatically
    apiVersion: argoproj.io/v1alpha1
    kind: Rollout
    metadata:
    name: rollouts-demo
    labels:
    app: rollouts-demo
    spec:
    strategy:
    canary:
    steps:
    - setWeight: 20
    - pause:
    duration: "1m"
    - setWeight: 50
    - pause:
    duration: "2m"
    canaryService: rollouts-demo-canary
    stableService: rollouts-demo-backend
    trafficRouting:
    istio:
    virtualService:
    name: rollout-vsvc
    routes:
    - primary
    … 57

    View Slide

  58. @kevindubois
    “Smart” Progressive Delivery
    Based on Metrics
    58

    View Slide

  59. @kevindubois
    Metrics Based Rollouts
    strategy:
    canary:
    analysis:
    args:
    - name: service-name
    value:
    rollouts-demo-canary.canary.svc.cluster.local
    templates:
    - templateName: success-rate
    canaryService: rollouts-demo-canary
    stableService: rollouts-demo-stable
    trafficRouting:
    istio:
    virtualService:
    name: rollout-vsvc
    routes:
    - primary
    steps:
    - setWeight: 30
    - pause: { duration: 20s }
    - setWeight: 40
    - pause: { duration: 10s }
    - setWeight: 60
    - pause: { duration: 10s }
    - setWeight: 80
    - pause: { duration: 5s }
    - setWeight: 90
    - pause: { duration: 5s }
    - setWeight: 100
    - pause: { duration: 5s } 59

    View Slide

  60. @kevindubois
    apiVersion: argoproj.io/v1alpha1
    kind: AnalysisTemplate
    metadata:
    name: success-rate
    spec:
    args:
    - name: service-name
    metrics:
    - name: success-rate
    interval: 10s
    successCondition: len(result) == 0 || result[0] >= 0.95
    failureLimit: 2
    provider:
    prometheus:
    address: https://internal:[email protected]:9090
    query: |
    sum(irate(istio_requests_total{
    reporter="source",
    destination_service=~"{{args.service-name}}",
    response_code!~"5.*"}[30s])
    )
    Metrics Based Rollouts
    60

    View Slide

  61. @kevindubois
    Progressive Delivery Demo
    @kevindubois
    61
    🚀🚀🚀

    View Slide

  62. @kevindubois
    Final Notes
    ● State is always hard
    ○ start with stateless; work with features; non-destructive schema changes; event-driven
    architectures (use eg. Debezium to work with ‘classic’ DBs).
    ● Step by Step
    ● Embrace GitOps
    ● If you haven’t automatically destroyed something by mistake, you aren’t
    automating enough
    ● Demos
    ○ https://dn.dev/istio-tutorial
    ○ https://github.com/kdubois/progressive-delivery
    ○ https://github.com/redhat-developer-demos/bubbles-progressive-delivery
    ○ github.com/redhat-developer-demos/quinoa-wind-turbine
    62

    View Slide

  63. @kevindubois
    Free Developer e-Books!
    https://developers.redhat.com/eventtutorials

    View Slide

  64. @kevindubois
    Start exploring in the OpenShift Sandbox.
    Learn containers, Kubernetes, and OpenShift in your browser.
    developers.redhat.com/developer-sandbox
    Try Red Hat's products and technologies
    without setup or configuration.

    View Slide

  65. @kevindubois
    Thank you!
    @[email protected]
    youtube.com/@thekevindubois
    linkedin.com/in/kevindubois
    github.com/kdubois
    @kevindubois.com

    View Slide

  66. @kevindubois
    Join Red Hat Developer.
    Build here. Go anywhere.
    facebook.com/RedHatDeveloper
    youtube.com/RedHatDevelopers
    twitter.com/rhdevelopers
    linkedin.com/showcase/red-hat-developer
    Thank you!

    View Slide