Next-gen CI/CD with Gitops and Progressive Delivery | fooConf Helsinki Finland

Even though you might be using Continuous Integration and Delivery, chances are you are still not 100% sure things will roll out without a glitch once you go to production. There will always be differences between environments and a risk for unforeseen issues related to your production environment and/or actual load, leading to potential disruption to your customers.

Progressive delivery is the next step after Continuous Delivery to test your application in production before it becomes fully available to all your user bases.

Embrace GitOps and Progressive Delivery with techniques like blue-green, canary release, shadowing traffic, dark launches and automatic metrics-based rollouts to validate the application in production using Kubernetes and tools like Istio, Prometheus, ArgoCD, and Argo Rollouts.

Come to this session to learn about Progressive Delivery in action using Kubernetes. And to have some fun as well as we release a car racing game you can participate in :)

Transcript

1. @kevindubois
   Next-gen CI/CD with Gitops
   and Progressive Delivery
   Kevin Dubois
   Principal Developer Advocate, Red Hat
   @kevindubois
   

   View Slide

2. @kevindubois
   How it started…
   

   View Slide

3. @kevindubois
   

   View Slide

4. @kevindubois
   CI - CD - CD
   Build Test
   Security
   Checks
   Release
   Deploy
   Stage
   Deploy
   Prod
   Continuous Integration
   Continuous Delivery
   Continuous Deployment
   Manual
   Auto
   

   View Slide

5. @kevindubois
   Every 4 months Every week/day/hour
   

   View Slide

6. @kevindubois
   Let’s start with playing a little game :)
   

   View Slide

7. @kevindubois
   The application
   Push to give energy windmill Kafka
   Topic
   2.Sends the interaction
   Dashboard: Green Energy
   Nickname Team
   Push/Tap to
   generate energy
   Cars that
   need energy
   Two teams competing
   (top 5 players)
   First wins
   

   View Slide

8. @kevindubois
   Architecture
   3: Generate power
   (REST)
   Game
   Dashboard
   1: Assign player
   Name & Team (REST)
   6: Update dashboard (SSE)
   2: Increment player cluster
   counter
   4: Send power event
   5: Receive power events
   

   View Slide

9. @kevindubois
   YOU PLAY!
   Scan the QR Code with your phone to play
   

   View Slide

10. @kevindubois
    What if we added a new feature?
    

    View Slide

11. @kevindubois
    Dev Ops
    Friday | 4:45 PM
    Wall of confusion
    

    View Slide

12. @kevindubois
    

    View Slide

13. @kevindubois
    Developer Flow
    Outer
    loop
    Inner
    loop
    Pull/Merge Request
    Production
    Build /
    Package
    Code
    Push
    Debug
    Code
    Review Build
    Deploy
    Security
    Tests
    Compliance
    Inner loop Outer loop
    Developer
    Test
    

    View Slide

14. @kevindubois
    Serverless CI
    Containers
    Built for container apps
    and runs on Kubernetes
    Designed with
    microservices and
    distributed teams in mind
    DevOps
    Serverless
    Runs serverless with no
    CI/CD engine to manage
    and maintain
    

    View Slide

15. @kevindubois
    Why Cloud-Native CI/CD?
    Traditional CI/CD Cloud-Native CI/CD
    Designed for Virtual Machines Designed for Containers and Kubernetes
    Require IT Ops for CI engine maintenance Pipeline as a service with no Ops overhead
    Plugins shared across CI engine Pipelines fully isolated from each other
    Plugin dependencies with undefined update cycles Everything lifecycled as container images
    No interoperability with Kubernetes resources Native Kubernetes resources
    Admin manages persistence Platform manages persistence
    Config baked into CI engine container Configured via Kubernetes ConfigMaps
    Declarative !
    

    View Slide

16. @kevindubois
    Tekton is a Graduated Continuous Delivery Foundation project and follows the OpenSSF best practices.
    Contributions from Google, Red Hat, Cloudbees, IBM, Elastic, Puppet, and many more
    An open-source project for providing a set of shared and standard
    components for building Kubernetes-style CI/CD systems
    https://tekton.dev
    

    View Slide

17. @kevindubois
    Step
    • Runs commands within
    container(builder image)
    • Mounts volumes, uses env
    vars
    • Eg. ‘mvn test’ or ‘git clone’
    Task
    • A list of steps that are
    executed in sequential order
    • Takes inputs, outputs
    parameters
    Task Run
    • Runs a individual Task
    Pipeline
    • List of tasks defined to run in
    a certain order
    • Takes inputs, outputs
    parameters
    Pipeline Run
    • Runs a Pipeline
    Typed Decoupled
    Cloud Native Declarative
    Tekton
    Concepts
    

    View Slide

18. @kevindubois
    apiVersion: tekton.dev/v1beta1
    kind: Pipeline
    metadata:
    name: wind-turbine-pipeline
    spec:
    params:
    - name: MANIFESTS_GIT_REPO
    type: string
    tasks:
    - name: git-clone
    params:
    - name: url
    value: $(params.GIT_REPO)
    workspaces:
    - name: output
    workspace: source
    workspaces:
    - name: source
    

    View Slide

19. @kevindubois
    Tekton Hub
    Search, discover and
    install Tekton Tasks
    hub.tekton.dev
    

    View Slide

20. @kevindubois
    Tekton CLI(tkn)
    •List and Describe
    • Pipeline
    • Resource
    • Task
    • Task Run
    • Pipeline Run
    •View logs
    • Task Run
    • Pipeline Run
    •https://github.com/tektoncd/cli
    

    View Slide

21. @kevindubois
    Gitops
    

    View Slide

22. @kevindubois
    What is GitOps?
    Treat
    everything as
    code
    Git is the
    single source
    of truth
    Operations
    through Git
    workflows
    

    View Slide

23. @kevindubois
    CI/CD Engines
    Jenkins
    Spinnaker
    Tekton
    Concourse CI
    …...
    CI/CD versus GitOps
    Desired
    State
    Cluster
    State
    Observe
    State
    Take
    Action
    GitOps
    Engines
    ACM, ArgoCD,
    FluxCD
    Razee, Faros
    Desired
    State
    Cluster
    State
    

    View Slide

24. @kevindubois
    ArgoCD
    Sync
    Monitor
    Detect
    drift
    Take
    action
    Argo CD is a declarative, GitOps continuous
    delivery tool for Kubernetes.
    Cluster and application configuration
    versioned in Git
    Automatically syncs configuration from Git to
    clusters
    Drift detection, visualization and correction
    

    View Slide

25. @kevindubois
    Let’s deploy our new feature in a Modern, Automated, Gitops way!
    

    View Slide

26. @kevindubois
    Live Coding
    

    View Slide

27. @kevindubois
    Source
    Git Repository
    Image
    Registry
    CI
    GitOps Application Delivery Model
    

    View Slide

28. @kevindubois
    Source
    Git Repository
    Image
    Registry
    CI
    Config
    Git Repository
    Kubernetes
    CD
    Pull Request
    / Commit
    Push
    Pull
    GitOps Application Delivery Model
    

    View Slide

29. @kevindubois
    GitOps Application Delivery Model
    Push
    Pull
    Pull
    Request
    Source
    Git
    Repository
    Image
    Registry
    Config
    Git
    Repository
    Kubernetes
    Deploy
    Monitor
    Detect
    drift
    CD
    Take
    action
    

    View Slide

30. @kevindubois
    V2
    Scan the QR Code with your phone to play
    

    View Slide

31. @kevindubois
    By default: a big bang / all or nothing release
    

    View Slide

32. @kevindubois
    Progressive Delivery
    

    View Slide

33. @kevindubois
    What is Progressive Delivery?
    Build Test
    Security
    Checks
    Release
    Deploy
    Stage
    Deploy
    Prod
    Continuous Integration
    Continuous Delivery
    Continuous Deployment
    Manual
    Auto
    

    View Slide

34. @kevindubois
    What is Progressive Delivery?
    ● No Big Bang
    ● Deploy != Release
    ● Metrics
    ● Subset of Users
    

    View Slide

35. @kevindubois
    Why Progressive Delivery?
    ● Decreases Downtime
    ● Limits the Tragedy
    ● Deploy & Release to Production faster
    ● Less mocking or setting up unreliable ‘fake’ services
    

    View Slide

36. @kevindubois
    Delivery Techniques
    

    View Slide

37. @kevindubois
    Blue Green Deployment
    ● All Or Nothing
    ● Quick Rollback
    37
    

    View Slide

38. @kevindubois
    Canary Releases
    ● Small Percentage
    ● Increase depending on metrics
    38
    

    View Slide

39. @kevindubois
    Dark Launches
    ● Mirroring Traffic
    ● Dark Canaries
    ● Feature Flags
    39
    

    View Slide

40. @kevindubois
    40
    

    View Slide

41. @kevindubois
    41
    

    View Slide

42. @kevindubois
    The New Pyramid?
    42
    

    View Slide

43. @kevindubois
    How to accomplish Progressive Delivery
    43
    

    View Slide

44. @kevindubois
    Accomplishing
    Progressive Delivery with
    44
    

    View Slide

45. @kevindubois
    Blue - Green
    apiVersion: v1
    kind: Service
    metadata:
    name: my-service
    labels:
    app: mystuff
    spec:
    ports:
    - name: http
    port: 8000
    selector:
    inservice: mypods
    type: LoadBalancer
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: mynode-deployment
    spec:
    replicas: 1
    selector:
    matchLabels:
    app: mynode
    template:
    metadata:
    labels:
    app: mynode
    spec:
    containers:
    - name: mynode
    image: quay.io/rhdevelopers/mynode:v1
    ports:
    - containerPort : 8000
    kubectl label pod -l app=mynode inservice=mypods
    45
    

    View Slide

46. @kevindubois
    Canary Releases
    kubectl scale deployment myapp-v1 --replicas=3
    kubectl scale deployment myapp-v2 --replicas=1
    46
    

    View Slide

47. @kevindubois
    

    View Slide

48. @kevindubois
    Controlling Microservices with a Service Mesh
    Code Independent (Polyglot)
    • Intelligent Routing and Load-Balancing
    • Smarter Canary Releases
    • Dark Launch
    • Chaos: Fault Injection
    • Resilience: Circuit Breakers
    • Observability & Telemetry: Metrics and Tracing
    • Security: Encryption & Authorization
    • Fleet wide policy enforcement
    48
    

    View Slide

49. @kevindubois
    Istio Architecture
    Control Plane
    The data plane is composed
    of a set of intelligent proxies
    (Envoy) deployed as
    sidecars. These proxies
    mediate and control all
    network communication
    between microservices.
    They also collect and report
    telemetry on all mesh
    traffic.
    The control plane manages
    and configures the proxies
    to route traffic.
    Data Plane
    

    View Slide

50. @kevindubois
    Pod
    Container
    JVM
    Service A
    Sidecar Container
    Pod
    Container
    JVM
    Service C
    Sidecar Container
    Pod
    Container
    JVM
    Service B
    Sidecar Container
    The sidecar intercepts all network traffic
    50
    

    View Slide

51. @kevindubois
    Canary Release
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    name: recommendation
    spec:
    hosts:
    - recommendation
    http:
    - route:
    - destination:
    host: recommendation
    subset: version-v1
    weight: 75
    - destination:
    host: recommendation
    subset: version-v2
    weight: 25
    51
    

    View Slide

52. @kevindubois
    Shadowing Traffic
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    name: recommendation
    spec:
    hosts:
    - recommendation
    http:
    - route:
    - destination:
    host: recommendation
    subset: version-v1
    mirror:
    host: recommendation
    subset: version-v2
    52
    

    View Slide

53. @kevindubois
    Dark Canary apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    name: recommendation
    spec:
    hosts:
    - recommendation
    http:
    - match:
    - headers:
    end-user:
    exact: Alexandra
    route:
    - destination:
    host: recommendation
    subset: version-v2
    - route:
    - destination:
    host: recommendation
    subset: version-v1
    53
    

    View Slide

54. @kevindubois
    Automated Progressive Delivery
    54
    

    View Slide

55. @kevindubois
    Argo Rollouts
    55
    

    View Slide

56. @kevindubois
    Argo Rollouts
    ArgoCD detects change
    rollout
    Monitors Data
    56
    

    View Slide

57. @kevindubois
    Rolling out automatically
    apiVersion: argoproj.io/v1alpha1
    kind: Rollout
    metadata:
    name: rollouts-demo
    labels:
    app: rollouts-demo
    spec:
    strategy:
    canary:
    steps:
    - setWeight: 20
    - pause:
    duration: "1m"
    - setWeight: 50
    - pause:
    duration: "2m"
    canaryService: rollouts-demo-canary
    stableService: rollouts-demo-backend
    trafficRouting:
    istio:
    virtualService:
    name: rollout-vsvc
    routes:
    - primary
    … 57
    

    View Slide

58. @kevindubois
    “Smart” Progressive Delivery
    Based on Metrics
    58
    

    View Slide

59. @kevindubois
    Metrics Based Rollouts
    strategy:
    canary:
    analysis:
    args:
    - name: service-name
    value:
    rollouts-demo-canary.canary.svc.cluster.local
    templates:
    - templateName: success-rate
    canaryService: rollouts-demo-canary
    stableService: rollouts-demo-stable
    trafficRouting:
    istio:
    virtualService:
    name: rollout-vsvc
    routes:
    - primary
    steps:
    - setWeight: 30
    - pause: { duration: 20s }
    - setWeight: 40
    - pause: { duration: 10s }
    - setWeight: 60
    - pause: { duration: 10s }
    - setWeight: 80
    - pause: { duration: 5s }
    - setWeight: 90
    - pause: { duration: 5s }
    - setWeight: 100
    - pause: { duration: 5s } 59
    

    View Slide

60. @kevindubois
    apiVersion: argoproj.io/v1alpha1
    kind: AnalysisTemplate
    metadata:
    name: success-rate
    spec:
    args:
    - name: service-name
    metrics:
    - name: success-rate
    interval: 10s
    successCondition: len(result) == 0 || result[0] >= 0.95
    failureLimit: 2
    provider:
    prometheus:
    address: https://internal:demo@prometheus.istio-system.svc.cluster.local:9090
    query: |
    sum(irate(istio_requests_total{
    reporter="source",
    destination_service=~"{{args.service-name}}",
    response_code!~"5.*"}[30s])
    )
    Metrics Based Rollouts
    60
    

    View Slide

61. @kevindubois
    Progressive Delivery Demo
    @kevindubois
    61
    🚀🚀🚀
    

    View Slide

62. @kevindubois
    Final Notes
    ● State is always hard
    ○ start with stateless; work with features; non-destructive schema changes; event-driven
    architectures (use eg. Debezium to work with ‘classic’ DBs).
    ● Step by Step
    ● Embrace GitOps
    ● If you haven’t automatically destroyed something by mistake, you aren’t
    automating enough
    ● Demos
    ○ https://dn.dev/istio-tutorial
    ○ https://github.com/kdubois/progressive-delivery
    ○ https://github.com/redhat-developer-demos/bubbles-progressive-delivery
    ○ github.com/redhat-developer-demos/quinoa-wind-turbine
    62
    

    View Slide

63. @kevindubois
    Free Developer e-Books!
    https://developers.redhat.com/eventtutorials
    

    View Slide

64. @kevindubois
    Start exploring in the OpenShift Sandbox.
    Learn containers, Kubernetes, and OpenShift in your browser.
    developers.redhat.com/developer-sandbox
    Try Red Hat's products and technologies
    without setup or configuration.
    

    View Slide

65. @kevindubois
    Thank you!
    @kevindubois@mastodon.social
    youtube.com/@thekevindubois
    linkedin.com/in/kevindubois
    github.com/kdubois
    @kevindubois.com
    

    View Slide

66. @kevindubois
    Join Red Hat Developer.
    Build here. Go anywhere.
    facebook.com/RedHatDeveloper
    youtube.com/RedHatDevelopers
    twitter.com/rhdevelopers
    linkedin.com/showcase/red-hat-developer
    Thank you!
    

    View Slide