Progressive Delivery in the Kubernetes Era - Voxxed Days Bucharest

Production is the place where all applications should live. Even though you are using continuous integration and delivery, you might wonder every time you release a new version to production if it will work or there will be some breakage on the latest version, eventually making production unavailable to the customers.

Progressive delivery is the next step after Continuous Delivery to test your application in production before it becomes fully available to all your user bases.

Embrace progressive delivery with techniques like the blue-green, canary release, shadowing traffic, or dark launches to validate the application in production using Kubernetes and tools like Istio, Prometheus, ArgoCD, or Argo Rollouts.

Come to this session to learn progressive delivery and GitOps in action using Kubernetes.

Transcript

1. Progressive Delivery
   In the Kubernetes Era
   Kevin Dubois
   Principal Developer Advocate, Red Hat
   Natale Vinto,
   Head of Developer Advocacy, Red Hat
   

   View Slide

2. @kevindubois @natalevinto
   Natale Vinto
   $ whoami
   Developer Advocate for OpenShift, helping people within
   communities and customers having success with their
   Kubernetes and Cloud Native strategy.
   Love Free and Open Source software, Blues 🎸 and coffee ☕
   @natalevinto
   OpenShift.TV Coffee Break every Wed 10 AM CEST
   

   View Slide

3. @kevindubois @natalevinto
   Kevin Dubois
   ● Principal Developer Advocate at Red Hat
   ● 18+ years of experience writing and delivering software applications
   ● Speak English, Dutch, French, Italian
   ● Passionate about improving Dev Experience with Open Source
   @kevindubois@mastodon.social
   youtube.com/@thekevindubois
   linkedin.com/in/kevindubois
   github.com/kdubois
   

   View Slide

4. @kevindubois @natalevinto
   Trial and
   Onboarding
   Inner Loop
   (for Code, Build, Debug and Push)
   Outer Loop
   (for Code review, Compliance, Security, Test and Deploy)
   Developer
   Sandbox
   Fastest and least friction
   way, at zero cost, for a
   developer to try our
   OpenShift platform, key
   developer tools and
   services, and Red Hat
   cloud services.
   https://developers.redhat.
   com/developer-sandbox
   OpenShift
   Local
   Formerly known as “Red
   Hat CodeReady
   Containers”
   Pre-built development
   environment, based on
   OpenShift v4, for quick
   container-based
   application development -
   on developer laptops.
   https://developers.redhat.
   com/products/openshift-lo
   cal/overview
   ◉ Podman Desktop
   This upstream project enables developers to easily build and share
   containerized applications, facilitating their transition to Kubernetes.
   https://podman.io
   ◉ Docker Desktop extension
   Enables deployment of local images to OpenShift directly from Docker
   Desktop.
   https://github.com/redhat-developer/openshift-dd-ext
   ◉ OpenShift Dev Spaces
   Formerly Red Hat CodeReady Workspaces
   Centralized, shareable, in-browser Kubernetes-based Integrated
   Development Environment, focused on OpenShift customers.
   https://developers.redhat.com/products/openshift-dev-spaces/overview
   ◉ IDE Extensions (for Visual Studio Code,
   IntelliJ, Eclipse) for Languages (Java,
   XML, YAML, Apache Camel) and Runtimes
   (Quarkus, EAP)
   Makes it easy for developers to build cloud-native and hybrid cloud
   applications with RH supported languages, and use RH preferred
   runtimes.
   ◉ CLI tools: odo
   OpenShift Developer CLI (command-line interface) that helps developers
   to build on Kubernetes/OpenShift, hiding Kubernetes complexity and
   supporting developers to get up and running
   Maven/Grade tooling (JKube)
   Adds the ability to bring Java applications to Kubernetes by extending
   native Maven/Gradle packaging tools.
   ◉ Dependency analytics
   Formerly known as “CodeReady Dependency Analytics”
   Hosted service, reachable by IDE extension and CLI/API for Software
   Composition Analysis and vulnerability remediation of Java, Javascript,
   Python, and Go applications.
   App Studio
   Hosted, fully managed experience to build full stack applications, easily connect to leading cloud services, adopt DevSecOps practices, and deploy
   to any hybrid-cloud platform.
   https://developers.redhat.com/app-studio/overview
   Plug-ins for CI/CD tools (GitHub Actions, Microsoft Azure DevOps, GitLab Runner,
   Jenkins)
   Provide templates for automated deployment of applications to OpenShift
   OpenShift Pipelines, Tekton Hub, Tekton IDE extension
   Is a cloud-native CI/CD solution to securely build containerized applications and deploy to OpenShift. Tekton Hub provides a catalog of Tekton task
   templates. Tekton IDE extension enables developers to create and manage Tekton pipelines from their IDE
   IDE Extensions (for Visual Studio Code, IntelliJ, Eclipse) for Deployment (OpenShift
   Connector, Serverless)
   Enable developers to push their code to OpenShift without needing to build containers, and to also configure serverless deployments.
   OpenShift GitOps
   Enables developers to declare and version application and cluster configuration in Git, automate application deployments, configuration drift
   detection and remediation. kam (Kubernetes application management) enables developers to easily seed GitOps declarative application models in
   Git.
   OpenShift Builds
   Provides a Kubernetes-native build tool on OpenShift. It supports multiple build strategies, and generates container images that are portable across
   Kubernetes distributions.
   OpenShift UI tools: OpenShift Dev Console
   Provides an end-to-end developer experience in OpenShift, across all cloud platforms, for building, deploying, and managing containerized
   applications and their related services.
   Web Terminal
   Runs within the OpenShift Dev Console. It provides a command line interface, pre-loaded with developer-focused CLIs.
   Helm
   is a package manager for Kubernetes. We provide support for that packaging in OpenShift, and a certified catalog for developers to quickly create
   applications based on curated Helm charts.
   Service Binding
   Enables developers to easily connect their applications to services running on the OpenShift cluster and Red Hat cloud services.
   Developer Tools Portfolio
   

   View Slide

5. @kevindubois @natalevinto
   Agenda
   Progressive
   Delivery
   Explained
   Delivery
   Techniques
   Progressive
   Delivery in
   Practice
   Continuous
   Delivery
   Metrics based
   Progressive
   Delivery
   

   View Slide

6. @kevindubois @natalevinto
   What we’ll talk about today
   ● Inner/Outer Loop
   ● Continuous Delivery
   ● CD with Kubernetes
   ● Gitops
   ● Traffic control with Istio
   ● Progressive Delivery
   ● Argo Rollouts
   

   View Slide

7. @kevindubois @natalevinto
   Developer Flow
   Outer
   loop
   Inner
   loop
   Pull/Merge Request
   Production
   Build /
   Package
   Code
   Push
   Debug
   Code
   Review Build
   Deploy
   Security
   Tests
   Compliance
   Inner loop Outer loop
   Developer
   Test
   

   View Slide

8. @kevindubois @natalevinto
   @kevindubois @natalevinto
   Today, let’s focus on the
   “Outer Loop”
   The outer loop consists of the larger team
   processes that your code flows through on its way
   to the cluster: code reviews, integration tests,
   security and compliance. It needs be transparent
   and fast for developers from all teams.
   Outer Loop Development
   

   View Slide

9. @kevindubois @natalevinto
   Love Thy Mono
   Every 4 months Every week/day/hour
   

   View Slide

10. @kevindubois @natalevinto
    Before we go on,
    let’s play a game :)
    

    View Slide

11. @kevindubois @natalevinto
    The application
    Push to give energy windmill
    1.Sends click
    Kafka
    Topic
    2.Sends the interaction
    Dashboard: Green Energy
    Nickname Team
    Push/Tap to
    generate energy
    Cars that
    need energy
    Two teams competing
    (top 5 players)
    First wins
    

    View Slide

12. @kevindubois @natalevinto
    Architecture
    3: Generate power
    (REST)
    Game
    Dashboard
    1: Assign player
    Name & Team (REST)
    6: Update dashboard (SSE)
    2: Increment player cluster
    counter
    4: Send power event
    5: Receive power events
    12
    

    View Slide

13. @kevindubois @natalevinto
    YOU PLAY!
    red.ht/voxxed23
    13
    

    View Slide

14. @kevindubois @natalevinto
    What if we added a new feature?
    

    View Slide

15. @kevindubois @natalevinto
    

    View Slide

16. @kevindubois @natalevinto
    github.com/
    redhat-developer-demos/
    quinoa-wind-turbine
    github.com/
    redhat-developer-demos/
    quinoa-wind-turbine-manifests
    

    View Slide

17. @kevindubois @natalevinto
    Let’s deploy a new feature
    in a Gitops way!
    

    View Slide

18. @kevindubois @natalevinto
    Developer Flow
    Outer
    loop
    Inner
    loop
    Pull/Merge Request
    Production
    Build /
    Package
    Code
    Push
    Debug
    Code
    Review Build
    Deploy
    Security
    Tests
    Compliance
    Inner loop Outer loop
    Developer
    Test
    

    View Slide

19. @kevindubois @natalevinto
    19
    Gitops
    

    View Slide

20. @kevindubois @natalevinto
    What is GitOps?
    Treat
    everything as
    code
    Git is the
    single source
    of truth
    Operations
    through Git
    workflows
    

    View Slide

21. @kevindubois @natalevinto
    GitOps Application Delivery Model
    Source
    Git Repository
    Image
    Registry
    CI
    

    View Slide

22. @kevindubois @natalevinto
    GitOps Application Delivery Model
    Source
    Git Repository
    Image
    Registry
    CI
    Config
    Git Repository
    Kubernetes
    CD
    Pull Request
    / Commit
    Push
    Pull
    

    View Slide

23. @kevindubois @natalevinto
    V2
    red.ht/voxxed23
    

    View Slide

24. @kevindubois @natalevinto
    Progressive Delivery
    24
    

    View Slide

25. @kevindubois @natalevinto
    What is Progressive Delivery?
    Build Test
    Security
    Checks
    Release
    Deploy
    Stage
    Deploy
    Prod
    Continuous Integration
    Continuous Delivery
    

    View Slide

26. @kevindubois @natalevinto
    What is Progressive Delivery?
    ● No Big Bang
    ● Deploy != Release
    ● Metrics
    ● Subset of Users
    

    View Slide

27. @kevindubois @natalevinto
    Why Progressive Delivery?
    ● Decreases Downtime
    ● Limits the Tragedy
    ● Deploy & Release to Production faster
    ● Less mocking or setting up unreliable ‘fake’ services
    

    View Slide

28. @kevindubois @natalevinto
    28
    

    View Slide

29. @kevindubois @natalevinto
    Delivery Techniques
    

    View Slide

30. @kevindubois @natalevinto
    Blue Green Deployment
    ● All Or Nothing
    ● Quick Rollback
    30
    

    View Slide

31. @kevindubois @natalevinto
    Canary Releases
    ● Small Percentage
    ● Increase depending on metrics
    31
    

    View Slide

32. @kevindubois @natalevinto
    Dark Launches
    ● Mirroring Traffic
    ● Dark Canaries
    ● Feature Flags
    32
    

    View Slide

33. @kevindubois @natalevinto
    33
    

    View Slide

34. @kevindubois @natalevinto
    34
    Production is (again) not sacred anymore
    

    View Slide

35. @kevindubois @natalevinto
    The New Pyramid?
    35
    

    View Slide

36. @kevindubois @natalevinto
    How to accomplish Progressive Delivery
    36
    

    View Slide

37. @kevindubois @natalevinto
    Progressive Delivery with
    

    View Slide

38. @kevindubois @natalevinto
    Blue - Green
    38
    apiVersion: v1
    kind: Service
    metadata:
    name: my-service
    labels:
    app: mystuff
    spec:
    ports:
    - name: http
    port: 8000
    selector:
    inservice: mypods
    type: LoadBalancer
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: mynode-deployment
    spec:
    replicas: 1
    selector:
    matchLabels:
    app: mynode
    template:
    metadata:
    labels:
    app: mynode
    spec:
    containers:
    - name: mynode
    image: quay.io/rhdevelopers/mynode:v1
    ports:
    - containerPort: 8000
    kubectl label pod -l app=mynode inservice=mypods
    

    View Slide

39. @kevindubois @natalevinto
    Canary Releases
    kubectl scale deployment myapp-v1 --replicas=3
    kubectl scale deployment myapp-v2 --replicas=1
    39
    

    View Slide

40. View Slide

41. @kevindubois @natalevinto
    Controlling Microservices with a Service Mesh
    41
    Code Independent (Polyglot)
    • Intelligent Routing and Load-Balancing
    • Smarter Canary Releases
    • Dark Launch
    • Chaos: Fault Injection
    • Resilience: Circuit Breakers
    • Observability & Telemetry: Metrics and Tracing
    • Security: Encryption & Authorization
    • Fleet wide policy enforcement
    

    View Slide

42. @kevindubois @natalevinto
    Sidecar Container
    42
    

    View Slide

43. @kevindubois @natalevinto
    Sidecar Container
    Pod
    Container
    JVM
    Service A
    Sidecar Container
    Pod
    Container
    JVM
    Service C
    Sidecar Container
    Pod
    Container
    JVM
    Service B
    Sidecar Container
    With Istio
    The sidecar intercepts all network traffic
    

    View Slide

44. @kevindubois @natalevinto
    Canary Release
    44
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    name: recommendation
    spec:
    hosts:
    - recommendation
    http:
    - route:
    - destination:
    host: recommendation
    subset: version-v1
    weight: 75
    - destination:
    host: recommendation
    subset: version-v2
    weight: 25
    

    View Slide

45. @kevindubois @natalevinto
    Shadowing Traffic
    45
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    name: recommendation
    spec:
    hosts:
    - recommendation
    http:
    - route:
    - destination:
    host: recommendation
    subset: version-v1
    mirror:
    host: recommendation
    subset: version-v2
    

    View Slide

46. @kevindubois @natalevinto
    Dark Canary
    46
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
    name: recommendation
    spec:
    hosts:
    - recommendation
    http:
    - match:
    - headers:
    end-user:
    exact: Alexandra
    route:
    - destination:
    host: recommendation
    subset: version-v2
    - route:
    - destination:
    host: recommendation
    subset: version-v1
    

    View Slide

47. @kevindubois @natalevinto
    Automated Progressive Delivery
    47
    

    View Slide

48. @kevindubois @natalevinto
    48
    Argo Rollouts
    

    View Slide

49. @kevindubois @natalevinto
    Argo Rollouts
    49
    ArgoCD detects change
    rollout
    Monitors Data
    

    View Slide

50. @kevindubois @natalevinto
    Rolling out automatically
    50
    apiVersion: argoproj.io/v1alpha1
    kind: Rollout
    metadata:
    name: rollouts-demo
    labels:
    app: rollouts-demo
    spec:
    strategy:
    canary:
    steps:
    - setWeight: 20
    - pause:
    duration: "1m"
    - setWeight: 50
    - pause:
    duration: "2m"
    canaryService: rollouts-demo-canary
    stableService: rollouts-demo-backend
    trafficRouting:
    istio:
    virtualService:
    name: rollout-vsvc
    routes:
    - primary
    …
    

    View Slide

51. @kevindubois @natalevinto
    “Smart” Progressive Delivery
    Based on Metrics
    51
    

    View Slide

52. @kevindubois @natalevinto
    Metrics Based Rollouts
    strategy:
    canary:
    analysis:
    args:
    - name: service-name
    value: rollouts-demo-canary.canary.svc.cluster.local
    templates:
    - templateName: success-rate
    canaryService : rollouts-demo-canary
    stableService : rollouts-demo-stable
    trafficRouting :
    istio:
    virtualService :
    name: rollout-vsvc
    routes:
    - primary
    steps:
    - setWeight: 30
    - pause: { duration: 20s }
    - setWeight: 40
    - pause: { duration: 10s }
    - setWeight: 60
    - pause: { duration: 10s }
    - setWeight: 80
    - pause: { duration: 5s }
    - setWeight: 90
    - pause: { duration: 5s }
    - setWeight: 100
    - pause: { duration: 5s }
    52
    

    View Slide

53. @kevindubois @natalevinto
    apiVersion: argoproj.io/v1alpha1
    kind: AnalysisTemplate
    metadata:
    name: success-rate
    spec:
    args:
    - name: service-name
    metrics:
    - name: success-rate
    interval: 10s
    successCondition: len(result) == 0 || result[0] >= 0.95
    failureLimit: 2
    provider:
    prometheus:
    address: https://internal:demo@prometheus.istio-system.svc.cluster.local:9090
    query: |
    sum(irate(istio_requests_total{
    reporter="source",
    destination_service=~"{{args.service-name}}",
    response_code!~"5.*"}[30s])
    )
    53
    Metrics Based Rollouts
    

    View Slide

54. @kevindubois @natalevinto
    Live Demo
    @kevindubois @natalevinto
    @kevindubois @natalevinto
    

    View Slide

55. @kevindubois @natalevinto
    Final Notes
    ● State is always hard, start with stateless
    ● Step by Step
    ● Embrace GitOps
    ● If you haven’t automatically destroyed something by mistake,
    you aren’t automating enough
    ● Demos
    ○ https://dn.dev/istio-tutorial
    ○ https://github.com/kdubois/ocp-sm-ar
    ○ https://github.com/redhat-developer-demos/bubbles-progressive-delivery
    ○ github.com/redhat-developer-demos/quinoa-wind-turbine
    55
    

    View Slide

56. @kevindubois @natalevinto
    https://red.ht/gitops-cookbook
    

    View Slide

57. @kevindubois @natalevinto
    https://red.ht/modernize-enterprise-java
    

    View Slide

58. @kevindubois @natalevinto
    58
    Learn OpenShift and all Cloud Native for free in our web based learning portal
    Resources
    Try OpenShift GitOps on our learning portal
    Red Hat Developers Deep Dive
    Join our workshops to explore Argo CD, Tekton, Kubernetes, Helm and more!
    Sign up at developers.redhat.com
    Find out more about Red Hat’s products and what it offers
    developers
    GitOps ebooks
    Find out more on GitOps with ebooks https://developers.redhat.com/e-books
    Get more about GitOps on Red Hat Developer Portal!
    

    View Slide

59. @kevindubois @natalevinto
    Start exploring in the OpenShift Sandbox.
    Learn containers, Kubernetes, and OpenShift in your browser.
    developers.redhat.com/developer-sandbox
    Try Red Hat's products and technologies
    without setup or configuration.
    

    View Slide

60. @kevindubois @natalevinto
    Join Red Hat Developer.
    Build here. Go anywhere.
    facebook.com/RedHatDeveloper
    youtube.com/RedHatDevelopers
    twitter.com/rhdevelopers
    linkedin.com/showcase/red-hat-developer
    Thank you!
    

    View Slide