Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DevOpsDays Kansas City 2018: A DevOps State of Mind: Managing Microservices + Database with Kubernetes

DevOpsDays Kansas City 2018: A DevOps State of Mind: Managing Microservices + Database with Kubernetes

Rapid innovation, changing business landscapes, and new IT demands force businesses to make changes
quickly.  In the eyes of many, DevOps + Microservices + Containers are at the brink of becoming pervasive in IT to accelerate business innovation.   In this presentation, you'll learn about:
* Managing Microservices + Databases at scale with Kubernetes
* Deployment strategies for Microservices with Kubernetes including Recreate, Rolling, Blue/Green, Canary, and A/B testing
* Database migrations in a CI/CD pipeline with Kubernetes

7c6a033dd957d547b49630f626e1a143?s=128

Chris Van Tuin

October 18, 2018
Tweet

Transcript

  1. A DevOps State of Mind: Managing Microservices and Databases with

    Kubernetes Chris Van Tuin Red Hat Chief Technologist, NA West / Silicon Valley linkedin: Chris Van Tuin
 email: cvantuin@redhat.com
 twitter: @chrisvantuin
  2. “Only the paranoid survive” - Andy Grove, 1996

  3. THE WORLD IS AUTOMATING Those who succeed in automation will

    win
  4. THE CHALLENGE: 
 ENABLE INNOVATION AT SPEED, WHILE EXECUTING AT

    SCALE WITH EFFICIENCY Static &
 Planned Dynamic & 
 Policy Driven Execution Innovation Old New Execution Innovation
  5. IT’S NOT JUST SOFTWARE, THE DISRUPTERS = Empowered organization Speed

    Up 
 Innovation Time Change Move Fast, Break Things Culture of experimentation A 20% vs. 25% Shorten the Feedback Loop Real-time data-driven intelligence & personalization AI /
 ML Data, Data, Data B
  6. I.T. MUST EVOLVE 
 FROM A COST CENTER TO INNOVATION

    CENTER Development Model Application Architecture Deployment & Application Infrastructur Storage Waterfall Agile Monolithic N-tier Bare Metal Virtual Servers Data Center Hosted Scale Up Scale Out DevOps MicroServices Containers Hybrid Cloud Storage as a Service
  7. LAPTOP Container Application OS dependencies Guest VM LINUX BARE METAL

    Container Application OS dependencies LINUX VIRTUALIZATION Container Application OS dependencies Virtual Machine LINUX PRIVATE CLOUD Container Application OS dependencies Virtual Machine LINUX PUBLIC CLOUD Container Application OS dependencies Virtual Machine LINUX CONTAINERIZED MICROSERVICES
 Build Once, Deploy Anywhere
  8. BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory


    Speed, Resiliency, Scalability, Security 

  9. Databases Images Automation MANAGING CONTAINERIZED MICROSERVICES
 WITH KUBERNETES A/B Testing

    Migrations External
 Services Deployment Strategies Security What’s Next… CI/CD
  10. KUBERNETES AUTOMATION

  11. Web App role=web role=app role=web replicas=1, 
 role=app replicas=2, 


    role=web ORCHESTRATION Pod Node Service Controller Manager & Data Store (etcd) Ingress / (external
 route rule e.g. haproxy)
  12. Web App replicas=1, 
 role=app replicas=2, 
 role=web HEALTH CHECK

    Pod Node Service role=web role=app role=web Controller Manager & Data Store (etcd) Ingress / Route role=app
  13. Web App AUTO-SCALE 80% CPU -> Pod Node Service role=web

    role=app role=web Controller Manager & Data Store (etcd) replicas=2 
 role=app replicas=2, 
 role=web horizontalpodautoscaler -cpu-percent=50 --min=1 —max=3 Ingress / Route 50% CPU role=app
  14. CONTAINER IMAGES

  15. docker.io Registry Private Registry FROM fedora:1.0 CMD echo “Hello” Build

    file Physical, Virtual, Cloud Container Image Container Instance Build Run Ship CONTAINERS ENABLE DEVOPS
  16. CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS

    dependencies 1.2/latest 1.1
  17. A CONVERGED SOFTWARE 
 SUPPLY CHAIN

  18. Config Data Kubernetes configmaps secrets Container image Traditional 
 data

    services, Kubernetes 
 persistent volumes TREAT CONTAINERS AS IMMUTABLE To keep containerized apps portable Application Language runtimes OS dependencies
  19. KUBERNETES CONFIGMAP Decouple configuration from container image Application Language runtimes

    OS dependencies Environment Variable or Volume/File CONTAINER INSTANCE key:value from directories, files, or values KUBERNETES
 CONFIGMAP APPLICATION CONFIG FILE Application Configuration File e.g. XML etcd Pod Source Code Repository EnvVar require pod restart Files refresh in time
  20. CI/CD PIPELINE

  21. CI/CD PIPELINE Continuous Integration Continuous Build Continuous Deployment Developer ->

    Source -> Git Git -> RPMS -> Images-> Registry Images from 
 Registry -> Clusters
  22. CI/CD PIPELINE WITH KUBERNETES BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC

    CLOUD
  23. CONTAINER IMAGE SECURITY

  24. WHAT’S INSIDE MATTERS…

  25. PRIVATE REGISTRY

  26. Security CONTINUOUS INTEGRATION WITH SECURITY SCAN

  27. Java Build Environment Language runtimes OS dependencies Build Image Java

    Code Application Language runtimes OS dependencies Container Image Image Registry Source Repository Image Registry REPRODUCIBLE BUILDS Source to Image with Build Images Source v3.1 v1.0.1 v3.1
  28. RAPID INNOVATION & EXPERIMENTATION WITH A/B TESTING

  29. ”only about 1/3 of ideas improve the metrics 
 they

    were designed to improve.”
 Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION
  30. A/B TESTING USING CANARY DEPLOYMENTS

  31. 25% Conversion Rate ?! Conversion Rate 100% Version B Version

    A Ingress CANARY DEPLOYMENTS Tests / CI
  32. 25% Conversion Rate 30% Conversion Rate 75% 25% Version B

    Version A Ingress CANARY DEPLOYMENTS
  33. 25% Conversion Rate 30% Conversion Rate 100% Version B Version

    A Ingress CANARY DEPLOYMENTS
  34. 25% Conversion Rate 20% Conversion Rate 100% Version B Version

    A Rollback Ingress CANARY DEPLOYMENTS
  35. CONTINUOUS FEEDBACK LOOP

  36. MONITORING CONSIDERATIONS Kubernetes* Container* Host Cluster services, services, pods, 


    deployments metrics Container native metrics Traditional resource metrics - cpu, memory, network, storage prometheus + grafana kubernetes-state-metrics probes Stack Metrics Tool node-exporter kubelet:cAdvisor Microservices Distributed applications - traditional app metrics - service discovery - distributed tracing prometheus + grafana jaeger tracing istio
  37. Databases Images Automation MANAGING CONTAINERIZED MICROSERVICES
 WITH KUBERNETES A/B Testing

    Migrations External
 Services Deployment Strategies Security What’s Next… CI/CD
  38. DEPLOYMENT STRATEGIES

  39. CONTINUOUS DELIVERY WITH CONTAINERS

  40. CONTINUOUS DELIVERY DEPLOYMENT STRATEGIES DEPLOYMENT STRATEGIES • Recreate • Rolling

    updates • Blue / Green deployment
  41. Recreate

  42. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI RECREATE WITH DOWNTIME
  43. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI RECREATE WITH DOWNTIME
  44. Version 1.2 Version 1.2 Version 1.2 RECREATE WITH DOWNTIME Use

    Case • Non-mission critical services Pros • Simple, clean • No Schema incompatibilities • No API versioning Cons • Downtime
  45. Rolling Updates

  46. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI ROLLING UPDATES with ZERO DOWNTIME Rollingupdate
 maxUnavailable=0 maxSurge=1
  47. Deploy new version and wait until it’s ready… Health Check:

    readiness probe e.g. tcp, http, script Version 1 Version 1 Version 
 1.2 Version 1 Rollingupdate
 maxUnavailable=0 maxSurge=1
  48. Each container/pod is updated one by one Version 1.2 50%

    Version 1 V1 V1.2
  49. Each container/pod is updated one by one Version 1.2 Version

    1.2 Version 1.2 100% Use Case • Horizontally scaled • Backward compatible API/data • Microservices Pros • Zero downtime • Reduced risk, gradual rollout w/health checks • Ready for rollback Cons • Require backward compatible APIs/data • Resource overhead
  50. Blue / Green Deployment

  51. BLUE Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT

    Using Ingress 100%
  52. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100% Health Check: readiness probe e.g. tcp, http, script
  53. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100%
  54. BLUE / GREEN DEPLOYMENT Rollback BLUE GREEN Version 1 Version

    2 Ingress Use Case • Self-contained micro services (data) Pros • Low risk, never change production • No downtime • Production like testing • Rollback Cons • Resource overhead • Data synchronization
  55. EXTERNAL SERVICES

  56. EXTERNAL SERVICES Database outside cluster with IP address External Mongo

    Database Service External Mongo Database Service Development Production IP=10.200.0.2 port=27017 IP=10.100.0.9 port=27017
  57. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017
  58. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.200.0.2 port=27017
  59. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.100.0.9 port=27017
  60. Cloud Mongo Database Service 
 mongodb://<dbuser>:<dbpassword>
 @mongo48909.domain.name:48909/dev 
 mongodb://<dbuser>:<dbpassword>
 @mongo52101.domain.name:52101/dev

    Cloud Mongo Database Service Development Production EXTERNAL SERVICES Remotely hosted database with URI
  61. Pods Nodes Services Database name: mongo type: ExternalName externalName: mongo52101.domain,.name

    EXTERNAL SERVICES Using CNAME redirection mongodb://
 <dbuser>:
 <dbpassword>
 @mongo:<port>/dev 
 mongodb://<dbuser>:<dbpassword>
 @mongo52101.domain.name:52101/dev Cloud Mongo Database Service WebApp role=webapp replicas=2, 
 role=webapp
  62. DATABASES

  63. PERSISTENT VOLUMES Host Container Host Container Host Container Data in

    Container Data lost when Container terminates Data lost when Host terminates Independent of Container & Host Data in a Host Volume Networked Volume
  64. 1. Maintains a sticky network ID/name across restarts
 e.g. mongo-0,

    mongo-1, mongo-2 2. Ordered Operations with ordinal index 
 e.g. name-0, name-1, name-2 3. Stable, persistent storage (linked to ordinal index/name) 4. Mandatory headless service (no single IP) for integrations KUBERNETES
 STATEFULSETS
  65. role=mongo type=leader Nodes Pods Services Mongo StatefulSet replicas=2 role=mongo Client

    mongo-0 D A B C C DATABASE STATEFUL SETS StatefulSet with 2 replicas , headless service, direct access to pods pvc Read / Write Persistent Volume
  66. DATABASE STATEFUL SETS role=mongo type=leader role=mongo type=follower Nodes Pods Services

    Client Mongo-0 Mongo-1 D A B C C Mongo StatefulSet replicas=2 role=mongo pvc pvc Read / Write Read / Only Persistent Volume
  67. role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0

    Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C C D Mongo StatefulSet replicas=3 role=mongo Read / Write Read / Only Read / Only DATABASE STATEFUL SETS Scale to 3 replicas Client
  68. role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0

    Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C D Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Unresponsive Pod Client
  69. role=mongo type=leader role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 pvc

    pvc Persistent Volume A B D role=mongo type=follower Mongo-2 pvc C Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Auto recovery Client
  70. DATABASE MIGRATIONS

  71. DATABASE MIGRATIONS Version control database updates with Containers CONTAINER IMAGE

    CONTAINER BUILD FILE SQL MIGRATION SCRIPT Source Code Repository V2__add_table.sql Source Code Repository V2__add_table.sql /var/flyway/data Flyway flyway-mydb:v2.0.0 Registry + Dockerfile
  72. Application v3 Development Application V2 Test Application v1 Production DB

    v1 DB v2 DB v3 CI/CD PIPELINE Version control database updates, ex: flyway V3__add_table_scooter.sql V2__add_table_truck.sql V1__add_table_car.sql
  73. Nodes Pods Services postgresql-0 Persistent Volume A B D C

    PostgreSQL StatefulSet replicas=1 role=postgresq pvcl DATABASE MIGRATION StatefulSet deployment with headless Service v1
  74. Nodes Pods Services postgresql-0 Persistent Volume A B D C

    PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Create a Job for Flyway Flyway Job Secrets = Database Connection Info v1 flyway-mydb:v2.0.0 Image Registry Flyway
  75. role=postgressql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B

    D C PostgreSQL StatefulSet replicas=1 role=postgresql pvc DATABASE MIGRATIONS Apply schema changes to database Flyway Job Secrets = Database Connection Info V2 flyway-mydb:v2.0.0 Flyway
  76. role=postgresql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B

    D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Version control for database with Kubernetes V2
  77. WHAT’S NEXT

  78. Traffic Control Service Resiliency Chaos Testing Observ- ability Security

  79. OPERATORS

  80. THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin