DevOpsDays Kansas City 2018: A DevOps State of Mind: Managing Microservices + Database with Kubernetes

Rapid innovation, changing business landscapes, and new IT demands force businesses to make changes
quickly.  In the eyes of many, DevOps + Microservices + Containers are at the brink of becoming pervasive in IT to accelerate business innovation.   In this presentation, you'll learn about:
* Managing Microservices + Databases at scale with Kubernetes
* Deployment strategies for Microservices with Kubernetes including Recreate, Rolling, Blue/Green, Canary, and A/B testing
* Database migrations in a CI/CD pipeline with Kubernetes

Transcript

1. A DevOps State of Mind:
   Managing Microservices and Databases
   with Kubernetes
   Chris Van Tuin
   Red Hat
   Chief Technologist, NA West / Silicon Valley
   linkedin: Chris Van Tuin

   email: cvantuin@redhat.com

   twitter: @chrisvantuin
   

   View Slide

2. “Only the paranoid survive”
   - Andy Grove, 1996
   

   View Slide

3. THE WORLD IS AUTOMATING
   Those who succeed in automation will win
   

   View Slide

4. THE CHALLENGE: 

   ENABLE INNOVATION AT SPEED, WHILE
   EXECUTING AT SCALE WITH EFFICIENCY
   Static &

   Planned
   Dynamic & 

   Policy Driven
   Execution
   Innovation
   Old New
   Execution
   Innovation
   

   View Slide

5. IT’S NOT JUST SOFTWARE,
   THE DISRUPTERS =
   Empowered
   organization
   Speed Up 

   Innovation
   Time
   Change
   Move Fast,
   Break Things
   Culture of
   experimentation
   A
   20% vs. 25%
   Shorten the
   Feedback Loop
   Real-time
   data-driven
   intelligence &
   personalization
   AI /

   ML
   Data,
   Data,
   Data
   B
   

   View Slide

6. I.T. MUST EVOLVE 

   FROM A COST CENTER TO INNOVATION CENTER
   Development
   Model
   Application
   Architecture
   Deployment
   &
   Application
   Infrastructur
   Storage
   Waterfall
   Agile
   Monolithic
   N-tier
   Bare Metal
   Virtual Servers
   Data Center
   Hosted
   Scale Up
   Scale Out
   DevOps MicroServices Containers Hybrid Cloud
   Storage as
   a Service
   

   View Slide

7. LAPTOP
   Container
   Application
   OS dependencies
   Guest VM
   LINUX
   BARE METAL
   Container
   Application
   OS dependencies
   LINUX
   VIRTUALIZATION
   Container
   Application
   OS dependencies
   Virtual Machine
   LINUX
   PRIVATE CLOUD
   Container
   Application
   OS dependencies
   Virtual Machine
   LINUX
   PUBLIC CLOUD
   Container
   Application
   OS dependencies
   Virtual Machine
   LINUX
   CONTAINERIZED MICROSERVICES

   Build Once, Deploy Anywhere
   

   View Slide

8. BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD
   Automated Software Factory

   Speed, Resiliency, Scalability, Security 

   

   View Slide

9. Databases
   Images
   Automation
   MANAGING CONTAINERIZED MICROSERVICES

   WITH KUBERNETES
   A/B Testing
   Migrations
   External

   Services
   Deployment
   Strategies
   Security
   What’s Next…
   CI/CD
   

   View Slide

10. KUBERNETES
    AUTOMATION
    

    View Slide

11. Web App
    role=web role=app role=web
    replicas=1, 

    role=app
    replicas=2, 

    role=web
    ORCHESTRATION
    Pod
    Node
    Service
    Controller
    Manager
    &
    Data Store
    (etcd)
    Ingress /
    (external

    route rule
    e.g. haproxy)
    

    View Slide

12. Web App
    replicas=1, 

    role=app
    replicas=2, 

    role=web
    HEALTH CHECK
    Pod
    Node
    Service
    role=web role=app role=web
    Controller
    Manager
    &
    Data Store
    (etcd)
    Ingress /
    Route
    role=app
    

    View Slide

13. Web App
    AUTO-SCALE
    80% CPU ->
    Pod
    Node
    Service
    role=web role=app role=web
    Controller
    Manager
    &
    Data Store
    (etcd)
    replicas=2 

    role=app
    replicas=2, 

    role=web
    horizontalpodautoscaler
    -cpu-percent=50
    --min=1 —max=3
    Ingress /
    Route
    50% CPU
    role=app
    

    View Slide

14. CONTAINER IMAGES
    

    View Slide

15. docker.io
    Registry
    Private
    Registry
    FROM fedora:1.0
    CMD echo “Hello”
    Build
    file
    Physical, Virtual, Cloud
    Container
    Image
    Container
    Instance
    Build Run
    Ship
    CONTAINERS ENABLE DEVOPS
    

    View Slide

16. CONTAINER IMAGE
    JAR CONTAINER IMAGE
    Application Application
    Language runtimes
    OS dependencies
    1.2/latest
    1.1
    

    View Slide

17. A CONVERGED SOFTWARE 

    SUPPLY CHAIN
    

    View Slide

18. Config Data
    Kubernetes
    configmaps
    secrets
    Container
    image
    Traditional 

    data services,
    Kubernetes 

    persistent volumes
    TREAT CONTAINERS AS IMMUTABLE
    To keep containerized apps portable
    Application
    Language runtimes
    OS dependencies
    

    View Slide

19. KUBERNETES CONFIGMAP
    Decouple configuration from container image
    Application
    Language runtimes
    OS dependencies
    Environment
    Variable or Volume/File
    CONTAINER INSTANCE
    key:value
    from directories, files, or values
    KUBERNETES

    CONFIGMAP
    APPLICATION
    CONFIG FILE
    Application
    Configuration
    File
    e.g. XML
    etcd Pod
    Source Code
    Repository
    EnvVar require pod restart
    Files refresh in time
    

    View Slide

20. CI/CD PIPELINE
    

    View Slide

21. CI/CD PIPELINE
    Continuous
    Integration
    Continuous
    Build
    Continuous
    Deployment
    Developer ->
    Source ->
    Git
    Git ->
    RPMS ->
    Images->
    Registry
    Images from 

    Registry ->
    Clusters
    

    View Slide

22. CI/CD PIPELINE WITH KUBERNETES
    BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD
    

    View Slide

23. CONTAINER IMAGE SECURITY
    

    View Slide

24. WHAT’S INSIDE MATTERS…
    

    View Slide

25. PRIVATE REGISTRY
    

    View Slide

26. Security
    CONTINUOUS INTEGRATION WITH
    SECURITY SCAN
    

    View Slide

27. Java Build
    Environment
    Language runtimes
    OS dependencies
    Build Image
    Java Code
    Application
    Language runtimes
    OS dependencies
    Container Image
    Image
    Registry
    Source
    Repository
    Image
    Registry
    REPRODUCIBLE BUILDS
    Source to Image with Build Images
    Source
    v3.1 v1.0.1 v3.1
    

    View Slide

28. RAPID INNOVATION &
    EXPERIMENTATION
    WITH A/B TESTING
    

    View Slide

29. ”only about 1/3 of ideas improve the metrics 

    they were designed to improve.”

    Ronny Kohavi, Microsoft (Amazon)
    MICROSERVICES
    RAPID INNNOVATION & EXPERIMENTATION
    

    View Slide

30. A/B TESTING USING CANARY DEPLOYMENTS
    

    View Slide

31. 25% Conversion Rate ?! Conversion Rate
    100%
    Version B
    Version A
    Ingress
    CANARY DEPLOYMENTS
    Tests / CI
    

    View Slide

32. 25% Conversion Rate 30% Conversion Rate
    75% 25%
    Version B
    Version A
    Ingress
    CANARY DEPLOYMENTS
    

    View Slide

33. 25% Conversion Rate 30% Conversion Rate
    100%
    Version B
    Version A
    Ingress
    CANARY DEPLOYMENTS
    

    View Slide

34. 25% Conversion Rate 20% Conversion Rate
    100%
    Version B
    Version A
    Rollback
    Ingress
    CANARY DEPLOYMENTS
    

    View Slide

35. CONTINUOUS FEEDBACK LOOP
    

    View Slide

36. MONITORING CONSIDERATIONS
    Kubernetes*
    Container*
    Host
    Cluster services, services, pods, 

    deployments metrics
    Container native metrics
    Traditional resource metrics
    - cpu, memory, network, storage
    prometheus + grafana
    kubernetes-state-metrics
    probes
    Stack Metrics Tool
    node-exporter
    kubelet:cAdvisor
    Microservices
    Distributed applications
    - traditional app metrics
    - service discovery
    - distributed tracing
    prometheus + grafana
    jaeger tracing
    istio
    

    View Slide

37. Databases
    Images
    Automation
    MANAGING CONTAINERIZED MICROSERVICES

    WITH KUBERNETES
    A/B Testing
    Migrations
    External

    Services
    Deployment
    Strategies
    Security
    What’s Next…
    CI/CD
    

    View Slide

38. DEPLOYMENT STRATEGIES
    

    View Slide

39. CONTINUOUS DELIVERY WITH CONTAINERS
    

    View Slide

40. CONTINUOUS DELIVERY DEPLOYMENT STRATEGIES
    DEPLOYMENT STRATEGIES
    • Recreate
    • Rolling updates
    • Blue / Green deployment
    

    View Slide

41. Recreate
    

    View Slide

42. Version 1 Version 1
    Version 1
    Version 1.2
    `
    Tests / CI
    RECREATE WITH DOWNTIME
    

    View Slide

43. Version 1 Version 1
    Version 1
    Version 1.2
    `
    Tests / CI
    RECREATE WITH DOWNTIME
    

    View Slide

44. Version 1.2 Version 1.2
    Version 1.2
    RECREATE WITH DOWNTIME
    Use Case
    • Non-mission critical services
    Pros
    • Simple, clean
    • No Schema incompatibilities
    • No API versioning
    Cons
    • Downtime
    

    View Slide

45. Rolling Updates
    

    View Slide

46. Version 1 Version 1
    Version 1
    Version 1.2
    `
    Tests / CI
    ROLLING UPDATES with ZERO DOWNTIME
    Rollingupdate

    maxUnavailable=0
    maxSurge=1
    

    View Slide

47. Deploy new version and wait until it’s ready…
    Health Check:
    readiness probe
    e.g. tcp, http, script
    Version 1 Version 1 Version 

    1.2
    Version 1
    Rollingupdate

    maxUnavailable=0
    maxSurge=1
    

    View Slide

48. Each container/pod is updated one by one
    Version 1.2
    50%
    Version 1 V1 V1.2
    

    View Slide

49. Each container/pod is updated one by one
    Version 1.2
    Version 1.2
    Version 1.2
    100%
    Use Case
    • Horizontally scaled
    • Backward compatible
    API/data
    • Microservices
    Pros
    • Zero downtime
    • Reduced risk, gradual
    rollout w/health checks
    • Ready for rollback
    Cons
    • Require backward
    compatible APIs/data
    • Resource overhead
    

    View Slide

50. Blue / Green Deployment
    

    View Slide

51. BLUE
    Version 1
    Ingress
    e.g haproxy
    BLUE / GREEN DEPLOYMENT
    Using Ingress
    100%
    

    View Slide

52. BLUE GREEN
    Version 1 Version 2
    Ingress
    e.g haproxy
    BLUE / GREEN DEPLOYMENT
    Using Ingress
    100% Health Check:
    readiness probe
    e.g. tcp, http, script
    

    View Slide

53. BLUE GREEN
    Version 1 Version 2
    Ingress
    e.g haproxy
    BLUE / GREEN DEPLOYMENT
    Using Ingress
    100%
    

    View Slide

54. BLUE / GREEN DEPLOYMENT
    Rollback
    BLUE GREEN
    Version 1 Version 2
    Ingress
    Use Case
    • Self-contained micro
    services (data)
    Pros
    • Low risk, never
    change production
    • No downtime
    • Production like testing
    • Rollback
    Cons
    • Resource overhead
    • Data synchronization
    

    View Slide

55. EXTERNAL SERVICES
    

    View Slide

56. EXTERNAL SERVICES
    Database outside cluster with IP address
    External
    Mongo
    Database
    Service
    External
    Mongo
    Database
    Service
    Development Production
    IP=10.200.0.2
    port=27017
    IP=10.100.0.9
    port=27017
    

    View Slide

57. EXTERNAL SERVICES
    Database outside cluster with IP address
    Pods
    Nodes
    Services
    WebApp
    role=webapp
    replicas=2, 

    role=webapp
    External
    Mongo
    Database
    Service
    IP=10.200.0.2
    port=27017
    Network
    External
    Mongo
    Database
    Service
    IP=10.100.0.9
    port=27017
    

    View Slide

58. EXTERNAL SERVICES
    Database outside cluster with IP address
    Pods
    Nodes
    Services
    WebApp
    role=webapp
    replicas=2, 

    role=webapp
    External
    Mongo
    Database
    Service
    IP=10.200.0.2
    port=27017
    Network
    External
    Mongo
    Database
    Service
    IP=10.100.0.9
    port=27017
    Database
    name=mongo
    port=27017
    targetport=27017
    Endpoint
    IP=10.200.0.2
    port=27017
    

    View Slide

59. EXTERNAL SERVICES
    Database outside cluster with IP address
    Pods
    Nodes
    Services
    WebApp
    role=webapp
    replicas=2, 

    role=webapp
    External
    Mongo
    Database
    Service
    IP=10.200.0.2
    port=27017
    Network
    External
    Mongo
    Database
    Service
    IP=10.100.0.9
    port=27017
    Database
    name=mongo
    port=27017
    targetport=27017
    Endpoint
    IP=10.100.0.9
    port=27017
    

    View Slide

60. Cloud
    Mongo
    Database
    Service
    

    mongodb://:

    @mongo48909.domain.name:48909/dev
    

    mongodb://:

    @mongo52101.domain.name:52101/dev
    Cloud
    Mongo
    Database
    Service
    Development Production
    EXTERNAL SERVICES
    Remotely hosted database with URI
    

    View Slide

61. Pods
    Nodes
    Services Database
    name: mongo
    type: ExternalName
    externalName: mongo52101.domain,.name
    EXTERNAL SERVICES
    Using CNAME redirection
    mongodb://

    :

    

    @mongo:/dev
    

    mongodb://:

    @mongo52101.domain.name:52101/dev
    Cloud
    Mongo
    Database
    Service
    WebApp
    role=webapp
    replicas=2, 

    role=webapp
    

    View Slide

62. DATABASES
    

    View Slide

63. PERSISTENT VOLUMES
    Host
    Container
    Host
    Container
    Host
    Container
    Data in
    Container
    Data lost when
    Container terminates
    Data lost when
    Host terminates
    Independent of
    Container & Host
    Data in
    a Host Volume
    Networked
    Volume
    

    View Slide

64. 1. Maintains a sticky network ID/name across restarts

    e.g. mongo-0, mongo-1, mongo-2
    2. Ordered Operations with ordinal index 

    e.g. name-0, name-1, name-2
    3. Stable, persistent storage (linked to ordinal index/name)
    4. Mandatory headless service (no single IP) for integrations
    KUBERNETES

    STATEFULSETS
    

    View Slide

65. role=mongo
    type=leader
    Nodes
    Pods
    Services
    Mongo
    StatefulSet
    replicas=2
    role=mongo
    Client
    mongo-0
    D
    A B C
    C
    DATABASE STATEFUL SETS
    StatefulSet with 2 replicas , headless service, direct access to pods
    pvc
    Read / Write
    Persistent
    Volume
    

    View Slide

66. DATABASE STATEFUL SETS
    role=mongo
    type=leader
    role=mongo
    type=follower
    Nodes
    Pods
    Services
    Client
    Mongo-0 Mongo-1
    D
    A B C
    C
    Mongo
    StatefulSet
    replicas=2
    role=mongo
    pvc pvc
    Read / Write Read / Only
    Persistent
    Volume
    

    View Slide

67. role=mongo
    type=leader
    role=mongo
    type=follower
    role=mongo
    type=follower
    Nodes
    Pods
    Services
    Mongo-0 Mongo-1 Mongo-2
    pvc pvc pvc
    Persistent
    Volume
    A B C
    C D
    Mongo
    StatefulSet
    replicas=3
    role=mongo
    Read / Write Read / Only Read / Only
    DATABASE STATEFUL SETS
    Scale to 3 replicas
    Client
    

    View Slide

68. role=mongo
    type=leader
    role=mongo
    type=follower
    role=mongo
    type=follower
    Nodes
    Pods
    Services
    Mongo-0 Mongo-1 Mongo-2
    pvc pvc pvc
    Persistent
    Volume
    A B C D
    Mongo
    StatefulSet
    replicas=3
    role=mongo
    DATABASE STATEFUL SETS
    Unresponsive Pod
    Client
    

    View Slide

69. role=mongo
    type=leader
    role=mongo
    type=follower
    Nodes
    Pods
    Services
    Mongo-0 Mongo-1
    pvc pvc
    Persistent
    Volume
    A B D
    role=mongo
    type=follower
    Mongo-2
    pvc
    C
    Mongo
    StatefulSet
    replicas=3
    role=mongo
    DATABASE STATEFUL SETS
    Auto recovery
    Client
    

    View Slide

70. DATABASE MIGRATIONS
    

    View Slide

71. DATABASE MIGRATIONS
    Version control database updates with Containers
    CONTAINER IMAGE
    CONTAINER BUILD FILE
    SQL MIGRATION SCRIPT
    Source Code
    Repository
    V2__add_table.sql
    Source Code
    Repository
    V2__add_table.sql
    /var/flyway/data
    Flyway
    flyway-mydb:v2.0.0
    Registry
    +
    Dockerfile
    

    View Slide

72. Application
    v3
    Development
    Application
    V2
    Test
    Application
    v1
    Production
    DB
    v1
    DB
    v2
    DB
    v3
    CI/CD PIPELINE
    Version control database updates, ex: flyway
    V3__add_table_scooter.sql V2__add_table_truck.sql V1__add_table_car.sql
    

    View Slide

73. Nodes
    Pods
    Services
    postgresql-0
    Persistent
    Volume
    A B D
    C
    PostgreSQL
    StatefulSet
    replicas=1
    role=postgresq
    pvcl
    DATABASE MIGRATION
    StatefulSet deployment with headless Service
    v1
    

    View Slide

74. Nodes
    Pods
    Services
    postgresql-0
    Persistent
    Volume
    A B D
    C
    PostgreSQL
    StatefulSet
    replicas=1
    role=postgresql
    Pvc
    DATABASE MIGRATIONS
    Create a Job for Flyway
    Flyway
    Job
    Secrets = Database Connection Info
    v1
    flyway-mydb:v2.0.0
    Image
    Registry
    Flyway
    

    View Slide

75. role=postgressql
    type=primary
    Nodes
    Pods
    Services
    postgresql-0
    Persistent
    Volume
    A B D
    C
    PostgreSQL
    StatefulSet
    replicas=1
    role=postgresql
    pvc
    DATABASE MIGRATIONS
    Apply schema changes to database
    Flyway
    Job
    Secrets = Database Connection Info
    V2
    flyway-mydb:v2.0.0
    Flyway
    

    View Slide

76. role=postgresql
    type=primary
    Nodes
    Pods
    Services
    postgresql-0
    Persistent
    Volume
    A B D
    C
    PostgreSQL
    StatefulSet
    replicas=1
    role=postgresql
    Pvc
    DATABASE MIGRATIONS
    Version control for database with Kubernetes
    V2
    

    View Slide

77. WHAT’S NEXT
    

    View Slide

78. Traffic
    Control
    Service
    Resiliency
    Chaos
    Testing
    Observ-
    ability
    Security
    

    View Slide

79. OPERATORS
    

    View Slide

80. THANK YOU
    linkedin: Chris Van Tuin
    email: cvantuin@redhat.com
    twitter: @chrisvantuin
    

    View Slide