DevOpsDays Kansas City 2018: A DevOps State of Mind: Managing Microservices + Database with Kubernetes

Rapid innovation, changing business landscapes, and new IT demands force businesses to make changes
quickly.  In the eyes of many, DevOps + Microservices + Containers are at the brink of becoming pervasive in IT to accelerate business innovation.   In this presentation, you'll learn about:
* Managing Microservices + Databases at scale with Kubernetes
* Deployment strategies for Microservices with Kubernetes including Recreate, Rolling, Blue/Green, Canary, and A/B testing
* Database migrations in a CI/CD pipeline with Kubernetes

Transcript

1. A DevOps State of Mind: Managing Microservices and Databases with Kubernetes Chris Van Tuin Red Hat Chief Technologist, NA West / Silicon Valley linkedin: Chris Van Tuin
 email: cvantuin@redhat.com
 twitter: @chrisvantuin
2. “Only the paranoid survive” - Andy Grove, 1996
3. THE WORLD IS AUTOMATING Those who succeed in automation will win
4. THE CHALLENGE: 
 ENABLE INNOVATION AT SPEED, WHILE EXECUTING AT SCALE WITH EFFICIENCY Static &
 Planned Dynamic & 
 Policy Driven Execution Innovation Old New Execution Innovation
5. IT’S NOT JUST SOFTWARE, THE DISRUPTERS = Empowered organization Speed Up 
 Innovation Time Change Move Fast, Break Things Culture of experimentation A 20% vs. 25% Shorten the Feedback Loop Real-time data-driven intelligence & personalization AI /
 ML Data, Data, Data B
6. I.T. MUST EVOLVE 
 FROM A COST CENTER TO INNOVATION CENTER Development Model Application Architecture Deployment & Application Infrastructur Storage Waterfall Agile Monolithic N-tier Bare Metal Virtual Servers Data Center Hosted Scale Up Scale Out DevOps MicroServices Containers Hybrid Cloud Storage as a Service
7. LAPTOP Container Application OS dependencies Guest VM LINUX BARE METAL Container Application OS dependencies LINUX VIRTUALIZATION Container Application OS dependencies Virtual Machine LINUX PRIVATE CLOUD Container Application OS dependencies Virtual Machine LINUX PUBLIC CLOUD Container Application OS dependencies Virtual Machine LINUX CONTAINERIZED MICROSERVICES
 Build Once, Deploy Anywhere
8. BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory
 Speed, Resiliency, Scalability, Security 

9. Databases Images Automation MANAGING CONTAINERIZED MICROSERVICES
 WITH KUBERNETES A/B Testing Migrations External
 Services Deployment Strategies Security What’s Next… CI/CD
10. KUBERNETES AUTOMATION
11. Web App role=web role=app role=web replicas=1, 
 role=app replicas=2, 
 role=web ORCHESTRATION Pod Node Service Controller Manager & Data Store (etcd) Ingress / (external
 route rule e.g. haproxy)
12. Web App replicas=1, 
 role=app replicas=2, 
 role=web HEALTH CHECK Pod Node Service role=web role=app role=web Controller Manager & Data Store (etcd) Ingress / Route role=app
13. Web App AUTO-SCALE 80% CPU -> Pod Node Service role=web role=app role=web Controller Manager & Data Store (etcd) replicas=2 
 role=app replicas=2, 
 role=web horizontalpodautoscaler -cpu-percent=50 --min=1 —max=3 Ingress / Route 50% CPU role=app
14. CONTAINER IMAGES
15. docker.io Registry Private Registry FROM fedora:1.0 CMD echo “Hello” Build file Physical, Virtual, Cloud Container Image Container Instance Build Run Ship CONTAINERS ENABLE DEVOPS
16. CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS dependencies 1.2/latest 1.1
17. A CONVERGED SOFTWARE 
 SUPPLY CHAIN
18. Config Data Kubernetes configmaps secrets Container image Traditional 
 data services, Kubernetes 
 persistent volumes TREAT CONTAINERS AS IMMUTABLE To keep containerized apps portable Application Language runtimes OS dependencies
19. KUBERNETES CONFIGMAP Decouple configuration from container image Application Language runtimes OS dependencies Environment Variable or Volume/File CONTAINER INSTANCE key:value from directories, files, or values KUBERNETES
 CONFIGMAP APPLICATION CONFIG FILE Application Configuration File e.g. XML etcd Pod Source Code Repository EnvVar require pod restart Files refresh in time
20. CI/CD PIPELINE
21. CI/CD PIPELINE Continuous Integration Continuous Build Continuous Deployment Developer -> Source -> Git Git -> RPMS -> Images-> Registry Images from 
 Registry -> Clusters
22. CI/CD PIPELINE WITH KUBERNETES BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD
23. CONTAINER IMAGE SECURITY
24. WHAT’S INSIDE MATTERS…
25. PRIVATE REGISTRY
26. Security CONTINUOUS INTEGRATION WITH SECURITY SCAN
27. Java Build Environment Language runtimes OS dependencies Build Image Java Code Application Language runtimes OS dependencies Container Image Image Registry Source Repository Image Registry REPRODUCIBLE BUILDS Source to Image with Build Images Source v3.1 v1.0.1 v3.1
28. RAPID INNOVATION & EXPERIMENTATION WITH A/B TESTING
29. ”only about 1/3 of ideas improve the metrics 
 they were designed to improve.”
 Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION
30. A/B TESTING USING CANARY DEPLOYMENTS
31. 25% Conversion Rate ?! Conversion Rate 100% Version B Version A Ingress CANARY DEPLOYMENTS Tests / CI
32. 25% Conversion Rate 30% Conversion Rate 75% 25% Version B Version A Ingress CANARY DEPLOYMENTS
33. 25% Conversion Rate 30% Conversion Rate 100% Version B Version A Ingress CANARY DEPLOYMENTS
34. 25% Conversion Rate 20% Conversion Rate 100% Version B Version A Rollback Ingress CANARY DEPLOYMENTS
35. CONTINUOUS FEEDBACK LOOP
36. MONITORING CONSIDERATIONS Kubernetes* Container* Host Cluster services, services, pods, 
 deployments metrics Container native metrics Traditional resource metrics - cpu, memory, network, storage prometheus + grafana kubernetes-state-metrics probes Stack Metrics Tool node-exporter kubelet:cAdvisor Microservices Distributed applications - traditional app metrics - service discovery - distributed tracing prometheus + grafana jaeger tracing istio
37. Databases Images Automation MANAGING CONTAINERIZED MICROSERVICES
 WITH KUBERNETES A/B Testing Migrations External
 Services Deployment Strategies Security What’s Next… CI/CD
38. DEPLOYMENT STRATEGIES
39. CONTINUOUS DELIVERY WITH CONTAINERS
40. CONTINUOUS DELIVERY DEPLOYMENT STRATEGIES DEPLOYMENT STRATEGIES • Recreate • Rolling updates • Blue / Green deployment
41. Recreate
42. Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME
43. Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME
44. Version 1.2 Version 1.2 Version 1.2 RECREATE WITH DOWNTIME Use Case • Non-mission critical services Pros • Simple, clean • No Schema incompatibilities • No API versioning Cons • Downtime
45. Rolling Updates
46. Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI ROLLING UPDATES with ZERO DOWNTIME Rollingupdate
 maxUnavailable=0 maxSurge=1
47. Deploy new version and wait until it’s ready… Health Check: readiness probe e.g. tcp, http, script Version 1 Version 1 Version 
 1.2 Version 1 Rollingupdate
 maxUnavailable=0 maxSurge=1
48. Each container/pod is updated one by one Version 1.2 50% Version 1 V1 V1.2
49. Each container/pod is updated one by one Version 1.2 Version 1.2 Version 1.2 100% Use Case • Horizontally scaled • Backward compatible API/data • Microservices Pros • Zero downtime • Reduced risk, gradual rollout w/health checks • Ready for rollback Cons • Require backward compatible APIs/data • Resource overhead
50. Blue / Green Deployment
51. BLUE Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100%
52. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% Health Check: readiness probe e.g. tcp, http, script
53. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100%
54. BLUE / GREEN DEPLOYMENT Rollback BLUE GREEN Version 1 Version 2 Ingress Use Case • Self-contained micro services (data) Pros • Low risk, never change production • No downtime • Production like testing • Rollback Cons • Resource overhead • Data synchronization
55. EXTERNAL SERVICES
56. EXTERNAL SERVICES Database outside cluster with IP address External Mongo Database Service External Mongo Database Service Development Production IP=10.200.0.2 port=27017 IP=10.100.0.9 port=27017
57. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017
58. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.200.0.2 port=27017
59. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.100.0.9 port=27017
60. Cloud Mongo Database Service 
 mongodb://<dbuser>:<dbpassword>
 @mongo48909.domain.name:48909/dev 
 mongodb://<dbuser>:<dbpassword>
 @mongo52101.domain.name:52101/dev Cloud Mongo Database Service Development Production EXTERNAL SERVICES Remotely hosted database with URI
61. Pods Nodes Services Database name: mongo type: ExternalName externalName: mongo52101.domain,.name EXTERNAL SERVICES Using CNAME redirection mongodb://
 <dbuser>:
 <dbpassword>
 @mongo:<port>/dev 
 mongodb://<dbuser>:<dbpassword>
 @mongo52101.domain.name:52101/dev Cloud Mongo Database Service WebApp role=webapp replicas=2, 
 role=webapp
62. DATABASES
63. PERSISTENT VOLUMES Host Container Host Container Host Container Data in Container Data lost when Container terminates Data lost when Host terminates Independent of Container & Host Data in a Host Volume Networked Volume
64. 1. Maintains a sticky network ID/name across restarts
 e.g. mongo-0, mongo-1, mongo-2 2. Ordered Operations with ordinal index 
 e.g. name-0, name-1, name-2 3. Stable, persistent storage (linked to ordinal index/name) 4. Mandatory headless service (no single IP) for integrations KUBERNETES
 STATEFULSETS
65. role=mongo type=leader Nodes Pods Services Mongo StatefulSet replicas=2 role=mongo Client mongo-0 D A B C C DATABASE STATEFUL SETS StatefulSet with 2 replicas , headless service, direct access to pods pvc Read / Write Persistent Volume
66. DATABASE STATEFUL SETS role=mongo type=leader role=mongo type=follower Nodes Pods Services Client Mongo-0 Mongo-1 D A B C C Mongo StatefulSet replicas=2 role=mongo pvc pvc Read / Write Read / Only Persistent Volume
67. role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C C D Mongo StatefulSet replicas=3 role=mongo Read / Write Read / Only Read / Only DATABASE STATEFUL SETS Scale to 3 replicas Client
68. role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C D Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Unresponsive Pod Client
69. role=mongo type=leader role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 pvc pvc Persistent Volume A B D role=mongo type=follower Mongo-2 pvc C Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Auto recovery Client
70. DATABASE MIGRATIONS
71. DATABASE MIGRATIONS Version control database updates with Containers CONTAINER IMAGE CONTAINER BUILD FILE SQL MIGRATION SCRIPT Source Code Repository V2__add_table.sql Source Code Repository V2__add_table.sql /var/flyway/data Flyway flyway-mydb:v2.0.0 Registry + Dockerfile
72. Application v3 Development Application V2 Test Application v1 Production DB v1 DB v2 DB v3 CI/CD PIPELINE Version control database updates, ex: flyway V3__add_table_scooter.sql V2__add_table_truck.sql V1__add_table_car.sql
73. Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresq pvcl DATABASE MIGRATION StatefulSet deployment with headless Service v1
74. Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Create a Job for Flyway Flyway Job Secrets = Database Connection Info v1 flyway-mydb:v2.0.0 Image Registry Flyway
75. role=postgressql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql pvc DATABASE MIGRATIONS Apply schema changes to database Flyway Job Secrets = Database Connection Info V2 flyway-mydb:v2.0.0 Flyway
76. role=postgresql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Version control for database with Kubernetes V2
77. WHAT’S NEXT
78. Traffic Control Service Resiliency Chaos Testing Observ- ability Security
79. OPERATORS
80. THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin