Save 37% off PRO during our Black Friday Sale! »

A DevOps State of Mind: Microservices and Databases with Kubernetes

A DevOps State of Mind: Microservices and Databases with Kubernetes

Red Hat MSA Day in Atlanta

7c6a033dd957d547b49630f626e1a143?s=128

Chris Van Tuin

January 16, 2019
Tweet

Transcript

  1. A DevOps State of Mind: Microservices and Databases With Kubernetes

    Chris Van Tuin Chief Technologist, West @chrisvantuin cvantuin@redhat.com
  2. “Only the paranoid survive” - Andy Grove, 1996

  3. THE WORLD IS AUTOMATING Those who succeed in automation will

    win
  4. THE CHALLENGE: 
 ENABLE INNOVATION AT SPEED, WHILE EXECUTING AT

    SCALE WITH EFFICIENCY Static &
 Planned Dynamic & 
 Policy Driven Execution Innovation Old New Execution Innovation
  5. THE STRATEGIC DIFFERENTIATOR The Fab
 Powered by Automation “copy exactly”

    The Software Factory Powered by Automation
  6. IT’S NOT JUST SOFTWARE, THE DISRUPTERS = Empowered organization Speed

    Up 
 Innovation Time Change Move Fast, Break Things Culture of experimentation A 20% vs. 25% Shorten the Feedback Loop Real-time data-driven intelligence & personalization AI /
 ML Data, Data, Data B
  7. DEV QA OPS THE AVERAGE ENTERPRISE 
 DOES DEPLOYMENTS EVERY

    6 TO 9 MONTHS. Walled off people, walled off processes, walled off technologies with surprisingly little to no automation
  8. I.T. MUST EVOLVE 
 FROM A COST CENTER TO INNOVATION

    CENTER Development Model Application Architecture Deployment & Packaging Application Infrastructure Storage Waterfall Agile Monolithic N-tier Bare Metal Virtual Servers Data Center Hosted Scale Up Scale Out DevOps MicroServices Containers Hybrid Cloud Storage as a Service
  9. I.T. MUST TRANSFORM FROM A COST CENTER 
 INTO AN

    INNOVATION CENTER Powered by DevOps + Automation + + DEV QA OPS Culture Process 
 Automation Technology Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Cloud Native Applications Hybrid Cloud Open Source Agile, Iterative, Continuous, Infrastructure as Code Collaborative Transparent Open THE SOFTWARE FACTORY
  10. ”only about 1/3 of ideas improve the metrics 
 they

    were designed to improve.”
 Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION
  11. CONTAINERS Software packaging concept that typically includes an application and

    all of its runtime dependencies • HIGHER quality software releases • SHORTER test cycles • EASIER application management CONTAINER CONTAINER APP LIBS HOST OS SERVER APP LIBS BENEFITS
  12. LAPTOP Container Application OS dependencies Guest VM LINUX BARE METAL

    Container Application OS dependencies LINUX VIRTUALIZATION Container Application OS dependencies Virtual Machine LINUX PRIVATE CLOUD Container Application OS dependencies Virtual Machine LINUX PUBLIC CLOUD Container Application OS dependencies Virtual Machine LINUX CONTAINERIZED MICROSERVICES
 Build Once, Deploy Anywhere
  13. Image Format Distribution Spec Runtime Spec

  14. Scheduling Monitoring Persistence Discovery Lifecycle & health Scaling Aggregation Security

    MORE THAN CONTAINERS…
  15. BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory


    Speed, Resiliency, Scalability, Security 

  16. Databases Images Automation MANAGING CONTAINERIZED MICROSERVICES
 WITH KUBERNETES A/B Testing

    Migrations External
 Services Deployment Strategies Security What’s Next… CI/CD Security CI/CD
  17. KUBERNETES AUTOMATION

  18. Web App role=web role=app role=web replicas=1, 
 role=app replicas=2, 


    role=web ORCHESTRATION Pod Node Service Controller Manager & Data Store (etcd) Ingress / (external
 route rule e.g. haproxy)
  19. Web App replicas=1, 
 role=app replicas=2, 
 role=web HEALTH CHECK

    Pod Node Service role=web role=app role=web Controller Manager & Data Store (etcd) Ingress / Route role=app
  20. Web App AUTO-SCALE 80% CPU -> Pod Node Service role=web

    role=app role=web Controller Manager & Data Store (etcd) replicas=2 
 role=app replicas=2, 
 role=web horizontalpodautoscaler -cpu-percent=50 --min=1 —max=3 Ingress / Route 50% CPU role=app
  21. CONTAINER IMAGES

  22. docker.io Registry Private Registry FROM fedora:1.0 CMD echo “Hello” Build

    file Physical, Virtual, Cloud Container Image Container Instance Build Run Ship CONTAINERS ENABLE DEVOPS
  23. CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS

    dependencies 1.2/latest 1.1
  24. Config Data Kubernetes configmaps secrets Container image Traditional 
 data

    services, Kubernetes 
 persistent volumes TREAT CONTAINERS AS IMMUTABLE To keep containerized apps portable Application Language runtimes OS dependencies
  25. KUBERNETES CONFIGMAP Decouple configuration from container image Application Language runtimes

    OS dependencies Environment Variable or Volume/File CONTAINER INSTANCE key:value from directories, files, or values KUBERNETES
 CONFIGMAP APPLICATION CONFIG FILE Application Configuration File e.g. XML etcd Pod Source Code Repository EnvVar require pod restart Files refresh in time
  26. A CONVERGED SOFTWARE 
 SUPPLY CHAIN

  27. CONTAINER IMAGE SECURITY

  28. WHAT’S INSIDE MATTERS…

  29. PRIVATE REGISTRY

  30. Security CONTINUOUS INTEGRATION WITH SECURITY SCAN

  31. Java Build Environment Language runtimes OS dependencies Build Image Java

    Code Application Language runtimes OS dependencies Container Image Image Registry Source Repository Image Registry REPRODUCIBLE BUILDS Source to Image with Build Images Source v3.1 v1.0.1 v3.1
  32. CI/CD PIPELINE

  33. CI/CD PIPELINE Continuous Integration Continuous Build Continuous Deployment Developer ->

    Source -> Git Git -> RPMS -> Images-> Registry Images from 
 Registry -> Clusters
  34. CI/CD PIPELINE WITH KUBERNETES BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC

    CLOUD
  35. RAPID INNOVATION & EXPERIMENTATION WITH A/B TESTING

  36. A/B TESTING USING CANARY DEPLOYMENTS

  37. 25% Conversion Rate ?! Conversion Rate 100% Version B Version

    A Ingress CANARY DEPLOYMENTS Tests / CI
  38. 25% Conversion Rate 30% Conversion Rate 75% 25% Version B

    Version A Ingress CANARY DEPLOYMENTS
  39. 25% Conversion Rate 30% Conversion Rate 100% Version B Version

    A Ingress CANARY DEPLOYMENTS
  40. 25% Conversion Rate 20% Conversion Rate 100% Version B Version

    A Rollback Ingress CANARY DEPLOYMENTS
  41. CONTINUOUS FEEDBACK LOOP

  42. MONITORING CONSIDERATIONS Kubernetes* Container* Host Cluster services, services, pods, 


    deployments metrics Container native metrics Traditional resource metrics - cpu, memory, network, storage prometheus + grafana kubernetes-state-metrics probes Stack Metrics Tool node-exporter kubelet:cAdvisor Microservices Distributed applications - traditional app metrics - service discovery - distributed tracing prometheus + grafana jaeger tracing istio
  43. Databases Images Automation MANAGING CONTAINERIZED MICROSERVICES
 WITH KUBERNETES A/B Testing

    Migrations External
 Services Deployment Strategies Security What’s Next… CI/CD Security CI/CD
  44. DEPLOYMENT STRATEGIES

  45. CONTINUOUS DELIVERY WITH CONTAINERS

  46. CONTINUOUS DELIVERY DEPLOYMENT STRATEGIES DEPLOYMENT STRATEGIES • Recreate • Rolling

    updates • Blue / Green deployment
  47. Recreate

  48. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI RECREATE WITH DOWNTIME
  49. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI RECREATE WITH DOWNTIME
  50. Version 1.2 Version 1.2 Version 1.2 RECREATE WITH DOWNTIME Use

    Case • Non-mission critical services Pros • Simple, clean • No Schema incompatibilities • No API versioning Cons • Downtime
  51. Rolling Updates

  52. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI ROLLING UPDATES with ZERO DOWNTIME Rollingupdate
 maxUnavailable=0 maxSurge=1
  53. Deploy new version and wait until it’s ready… Health Check:

    readiness probe e.g. tcp, http, script Version 1 Version 1 Version 
 1.2 Version 1 Rollingupdate
 maxUnavailable=0 maxSurge=1
  54. Each container/pod is updated one by one Version 1.2 50%

    Version 1 V1 V1.2
  55. Each container/pod is updated one by one Version 1.2 Version

    1.2 Version 1.2 100% Use Case • Horizontally scaled • Backward compatible API/data • Microservices Pros • Zero downtime • Reduced risk, gradual rollout w/health checks • Ready for rollback Cons • Require backward compatible APIs/data • Resource overhead
  56. Blue / Green Deployment

  57. BLUE Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT

    Using Ingress 100%
  58. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100% Health Check: readiness probe e.g. tcp, http, script
  59. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100%
  60. BLUE / GREEN DEPLOYMENT Rollback BLUE GREEN Version 1 Version

    2 Ingress Use Case • Self-contained micro services (data) Pros • Low risk, never change production • No downtime • Production like testing • Rollback Cons • Resource overhead • Data synchronization
  61. EXTERNAL SERVICES

  62. EXTERNAL SERVICES Database outside cluster with IP address External Mongo

    Database Service External Mongo Database Service Development Production IP=10.200.0.2 port=27017 IP=10.100.0.9 port=27017
  63. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017
  64. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.200.0.2 port=27017
  65. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.100.0.9 port=27017
  66. Cloud Mongo Database Service 
 mongodb://<dbuser>:<dbpassword>
 @mongo48909.domain.name:48909/dev 
 mongodb://<dbuser>:<dbpassword>
 @mongo52101.domain.name:52101/dev

    Cloud Mongo Database Service Development Production EXTERNAL SERVICES Remotely hosted database with URI mongodb://<dbuser>:<dbpassword>
 @mongo48909.domain.name:48909/dev mongodb://<dbuser>:<dbpassword>
 @mongo52101.domain.name:52101/prod
  67. Pods Nodes Services Database name: mongo type: ExternalName externalName: mongo52101.domain,.name

    EXTERNAL SERVICES Using CNAME redirection mongodb://
 <dbuser>:
 <dbpassword>
 @mongo:<port>/dev 
 mongodb://<dbuser>:<dbpassword>
 @mongo52101.domain.name:52101/dev Cloud Mongo Database Service WebApp role=webapp replicas=2, 
 role=webapp .name mongodb://<dbuser>:<dbpassword>
 @mongo52101.domain.name:52101/prod
  68. DATABASES

  69. PERSISTENT VOLUMES Host Container Host Container Host Container Data in

    Container Data lost when Container terminates Data lost when Host terminates Independent of Container & Host Data in a Host Volume Networked Volume Data lost when Cloud instance terminates Data lost when Container terminates Independent of 
 Container & 
 Cloud instance DATA PERSISTENCE
  70. 1. Maintains a sticky network ID/name across restarts
 e.g. mongo-0,

    mongo-1, mongo-2 2. Ordered Operations with ordinal index 
 e.g. name-0, name-1, name-2 3. Stable, persistent storage (linked to ordinal index/name) 4. Mandatory headless service (no single IP) for integrations KUBERNETES
 STATEFULSETS
  71. role=mongo type=leader Nodes Pods Services Mongo StatefulSet replicas=2 role=mongo Client

    mongo-0 D A B C C DATABASE STATEFUL SETS StatefulSet with 2 replicas , headless service, direct access to pods pvc Read / Write Persistent Volume
  72. DATABASE STATEFUL SETS role=mongo type=leader role=mongo type=follower Nodes Pods Services

    Client Mongo-0 Mongo-1 D A B C C Mongo StatefulSet replicas=2 role=mongo pvc pvc Read / Write Read / Only Persistent Volume
  73. role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0

    Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C C D Mongo StatefulSet replicas=3 role=mongo Read / Write Read / Only Read / Only DATABASE STATEFUL SETS Scale to 3 replicas Client
  74. role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0

    Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C D Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Unresponsive Pod Client
  75. role=mongo type=leader role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 pvc

    pvc Persistent Volume A B D role=mongo type=follower Mongo-2 pvc C Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Auto recovery Client
  76. DATABASE MIGRATIONS

  77. Application v3 Development Application V2 Test Application v1 Production DB

    v1 DB v2 DB v3 CI/CD PIPELINE Version control database updates, ex: flyway V3__add_table_scooter.sql V2__add_table_truck.sql V1__add_table_car.sql
  78. DATABASE MIGRATIONS Version control database updates with Containers CONTAINER IMAGE

    CONTAINER BUILD FILE SQL MIGRATION SCRIPT Source Code Repository V2__add_table.sql Source Code Repository V2__add_table.sql /var/flyway/data Flyway flyway-mydb:v2.0.0 Registry + Dockerfile
  79. Nodes Pods Services postgresql-0 Persistent Volume A B D C

    PostgreSQL StatefulSet replicas=1 role=postgresq pvcl DATABASE MIGRATION StatefulSet deployment with headless Service v1
  80. Nodes Pods Services postgresql-0 Persistent Volume A B D C

    PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Create a Job for Flyway Flyway Job Secrets = Database Connection Info v1 flyway-mydb:v2.0.0 Image Registry Flyway
  81. role=postgressql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B

    D C PostgreSQL StatefulSet replicas=1 role=postgresql pvc DATABASE MIGRATIONS Apply schema changes to database Flyway Job Secrets = Database Connection Info V2 flyway-mydb:v2.0.0 Flyway
  82. role=postgresql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B

    D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Version control for database with Kubernetes V2
  83. DATABASE MIGRATIONS WITH BLUE / GREEN DEPLOYMENT

  84. WHAT’S NEXT

  85. Traffic Control Service Resiliency Chaos Testing Observ- ability Security SERVICE

    MESH
  86. OPERATORS

  87. learn.openshift.com

  88. THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin