A DevOps State of Mind: Microservices and Databases with Kubernetes

Red Hat MSA Day in Atlanta

1. A DevOps State of Mind: Microservices and Databases With Kubernetes Chris Van Tuin Chief Technologist, West @chrisvantuin cvantuin@redhat.com
2. “Only the paranoid survive” - Andy Grove, 1996
3. THE WORLD IS AUTOMATING Those who succeed in automation will win
4. THE CHALLENGE: 
 ENABLE INNOVATION AT SPEED, WHILE EXECUTING AT SCALE WITH EFFICIENCY Static &
 Planned Dynamic & 
 Policy Driven Execution Innovation Old New Execution Innovation
5. THE STRATEGIC DIFFERENTIATOR The Fab
 Powered by Automation “copy exactly” The Software Factory Powered by Automation
6. IT’S NOT JUST SOFTWARE, THE DISRUPTERS = Empowered organization Speed Up 
 Innovation Time Change Move Fast, Break Things Culture of experimentation A 20% vs. 25% Shorten the Feedback Loop Real-time data-driven intelligence & personalization AI /
 ML Data, Data, Data B
7. DEV QA OPS THE AVERAGE ENTERPRISE 
 DOES DEPLOYMENTS EVERY 6 TO 9 MONTHS. Walled off people, walled off processes, walled off technologies with surprisingly little to no automation
8. I.T. MUST EVOLVE 
 FROM A COST CENTER TO INNOVATION CENTER Development Model Application Architecture Deployment & Packaging Application Infrastructure Storage Waterfall Agile Monolithic N-tier Bare Metal Virtual Servers Data Center Hosted Scale Up Scale Out DevOps MicroServices Containers Hybrid Cloud Storage as a Service
9. I.T. MUST TRANSFORM FROM A COST CENTER 
 INTO AN INNOVATION CENTER Powered by DevOps + Automation + + DEV QA OPS Culture Process 
 Automation Technology Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Cloud Native Applications Hybrid Cloud Open Source Agile, Iterative, Continuous, Infrastructure as Code Collaborative Transparent Open THE SOFTWARE FACTORY
10. ”only about 1/3 of ideas improve the metrics 
 they were designed to improve.”
 Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION
11. CONTAINERS Software packaging concept that typically includes an application and all of its runtime dependencies • HIGHER quality software releases • SHORTER test cycles • EASIER application management CONTAINER CONTAINER APP LIBS HOST OS SERVER APP LIBS BENEFITS
12. LAPTOP Container Application OS dependencies Guest VM LINUX BARE METAL Container Application OS dependencies LINUX VIRTUALIZATION Container Application OS dependencies Virtual Machine LINUX PRIVATE CLOUD Container Application OS dependencies Virtual Machine LINUX PUBLIC CLOUD Container Application OS dependencies Virtual Machine LINUX CONTAINERIZED MICROSERVICES
 Build Once, Deploy Anywhere
13. Image Format Distribution Spec Runtime Spec
14. Scheduling Monitoring Persistence Discovery Lifecycle & health Scaling Aggregation Security MORE THAN CONTAINERS…
15. BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory
 Speed, Resiliency, Scalability, Security 

16. Databases Images Automation MANAGING CONTAINERIZED MICROSERVICES
 WITH KUBERNETES A/B Testing Migrations External
 Services Deployment Strategies Security What’s Next… CI/CD Security CI/CD
17. KUBERNETES AUTOMATION
18. Web App role=web role=app role=web replicas=1, 
 role=app replicas=2, 
 role=web ORCHESTRATION Pod Node Service Controller Manager & Data Store (etcd) Ingress / (external
 route rule e.g. haproxy)
19. Web App replicas=1, 
 role=app replicas=2, 
 role=web HEALTH CHECK Pod Node Service role=web role=app role=web Controller Manager & Data Store (etcd) Ingress / Route role=app
20. Web App AUTO-SCALE 80% CPU -> Pod Node Service role=web role=app role=web Controller Manager & Data Store (etcd) replicas=2 
 role=app replicas=2, 
 role=web horizontalpodautoscaler -cpu-percent=50 --min=1 —max=3 Ingress / Route 50% CPU role=app
21. CONTAINER IMAGES
22. docker.io Registry Private Registry FROM fedora:1.0 CMD echo “Hello” Build file Physical, Virtual, Cloud Container Image Container Instance Build Run Ship CONTAINERS ENABLE DEVOPS
23. CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS dependencies 1.2/latest 1.1
24. Config Data Kubernetes configmaps secrets Container image Traditional 
 data services, Kubernetes 
 persistent volumes TREAT CONTAINERS AS IMMUTABLE To keep containerized apps portable Application Language runtimes OS dependencies
25. KUBERNETES CONFIGMAP Decouple configuration from container image Application Language runtimes OS dependencies Environment Variable or Volume/File CONTAINER INSTANCE key:value from directories, files, or values KUBERNETES
 CONFIGMAP APPLICATION CONFIG FILE Application Configuration File e.g. XML etcd Pod Source Code Repository EnvVar require pod restart Files refresh in time
26. A CONVERGED SOFTWARE 
 SUPPLY CHAIN
27. CONTAINER IMAGE SECURITY
28. WHAT’S INSIDE MATTERS…
29. PRIVATE REGISTRY
30. Security CONTINUOUS INTEGRATION WITH SECURITY SCAN
31. Java Build Environment Language runtimes OS dependencies Build Image Java Code Application Language runtimes OS dependencies Container Image Image Registry Source Repository Image Registry REPRODUCIBLE BUILDS Source to Image with Build Images Source v3.1 v1.0.1 v3.1
32. CI/CD PIPELINE
33. CI/CD PIPELINE Continuous Integration Continuous Build Continuous Deployment Developer -> Source -> Git Git -> RPMS -> Images-> Registry Images from 
 Registry -> Clusters
34. CI/CD PIPELINE WITH KUBERNETES BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD
35. RAPID INNOVATION & EXPERIMENTATION WITH A/B TESTING
36. A/B TESTING USING CANARY DEPLOYMENTS
37. 25% Conversion Rate ?! Conversion Rate 100% Version B Version A Ingress CANARY DEPLOYMENTS Tests / CI
38. 25% Conversion Rate 30% Conversion Rate 75% 25% Version B Version A Ingress CANARY DEPLOYMENTS
39. 25% Conversion Rate 30% Conversion Rate 100% Version B Version A Ingress CANARY DEPLOYMENTS
40. 25% Conversion Rate 20% Conversion Rate 100% Version B Version A Rollback Ingress CANARY DEPLOYMENTS
41. CONTINUOUS FEEDBACK LOOP
42. MONITORING CONSIDERATIONS Kubernetes* Container* Host Cluster services, services, pods, 
 deployments metrics Container native metrics Traditional resource metrics - cpu, memory, network, storage prometheus + grafana kubernetes-state-metrics probes Stack Metrics Tool node-exporter kubelet:cAdvisor Microservices Distributed applications - traditional app metrics - service discovery - distributed tracing prometheus + grafana jaeger tracing istio
43. Databases Images Automation MANAGING CONTAINERIZED MICROSERVICES
 WITH KUBERNETES A/B Testing Migrations External
 Services Deployment Strategies Security What’s Next… CI/CD Security CI/CD
44. DEPLOYMENT STRATEGIES
45. CONTINUOUS DELIVERY WITH CONTAINERS
46. CONTINUOUS DELIVERY DEPLOYMENT STRATEGIES DEPLOYMENT STRATEGIES • Recreate • Rolling updates • Blue / Green deployment
47. Recreate
48. Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME
49. Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME
50. Version 1.2 Version 1.2 Version 1.2 RECREATE WITH DOWNTIME Use Case • Non-mission critical services Pros • Simple, clean • No Schema incompatibilities • No API versioning Cons • Downtime
51. Rolling Updates
52. Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI ROLLING UPDATES with ZERO DOWNTIME Rollingupdate
 maxUnavailable=0 maxSurge=1
53. Deploy new version and wait until it’s ready… Health Check: readiness probe e.g. tcp, http, script Version 1 Version 1 Version 
 1.2 Version 1 Rollingupdate
 maxUnavailable=0 maxSurge=1
54. Each container/pod is updated one by one Version 1.2 50% Version 1 V1 V1.2
55. Each container/pod is updated one by one Version 1.2 Version 1.2 Version 1.2 100% Use Case • Horizontally scaled • Backward compatible API/data • Microservices Pros • Zero downtime • Reduced risk, gradual rollout w/health checks • Ready for rollback Cons • Require backward compatible APIs/data • Resource overhead
56. Blue / Green Deployment
57. BLUE Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100%
58. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% Health Check: readiness probe e.g. tcp, http, script
59. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100%
60. BLUE / GREEN DEPLOYMENT Rollback BLUE GREEN Version 1 Version 2 Ingress Use Case • Self-contained micro services (data) Pros • Low risk, never change production • No downtime • Production like testing • Rollback Cons • Resource overhead • Data synchronization
61. EXTERNAL SERVICES
62. EXTERNAL SERVICES Database outside cluster with IP address External Mongo Database Service External Mongo Database Service Development Production IP=10.200.0.2 port=27017 IP=10.100.0.9 port=27017
63. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017
64. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.200.0.2 port=27017
65. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.100.0.9 port=27017
66. Cloud Mongo Database Service 
 mongodb://<dbuser>:<dbpassword>
 @mongo48909.domain.name:48909/dev 
 mongodb://<dbuser>:<dbpassword>
 @mongo52101.domain.name:52101/dev Cloud Mongo Database Service Development Production EXTERNAL SERVICES Remotely hosted database with URI mongodb://<dbuser>:<dbpassword>
 @mongo48909.domain.name:48909/dev mongodb://<dbuser>:<dbpassword>
 @mongo52101.domain.name:52101/prod
67. Pods Nodes Services Database name: mongo type: ExternalName externalName: mongo52101.domain,.name EXTERNAL SERVICES Using CNAME redirection mongodb://
 <dbuser>:
 <dbpassword>
 @mongo:<port>/dev 
 mongodb://<dbuser>:<dbpassword>
 @mongo52101.domain.name:52101/dev Cloud Mongo Database Service WebApp role=webapp replicas=2, 
 role=webapp .name mongodb://<dbuser>:<dbpassword>
 @mongo52101.domain.name:52101/prod
68. DATABASES
69. PERSISTENT VOLUMES Host Container Host Container Host Container Data in Container Data lost when Container terminates Data lost when Host terminates Independent of Container & Host Data in a Host Volume Networked Volume Data lost when Cloud instance terminates Data lost when Container terminates Independent of 
 Container & 
 Cloud instance DATA PERSISTENCE
70. 1. Maintains a sticky network ID/name across restarts
 e.g. mongo-0, mongo-1, mongo-2 2. Ordered Operations with ordinal index 
 e.g. name-0, name-1, name-2 3. Stable, persistent storage (linked to ordinal index/name) 4. Mandatory headless service (no single IP) for integrations KUBERNETES
 STATEFULSETS
71. role=mongo type=leader Nodes Pods Services Mongo StatefulSet replicas=2 role=mongo Client mongo-0 D A B C C DATABASE STATEFUL SETS StatefulSet with 2 replicas , headless service, direct access to pods pvc Read / Write Persistent Volume
72. DATABASE STATEFUL SETS role=mongo type=leader role=mongo type=follower Nodes Pods Services Client Mongo-0 Mongo-1 D A B C C Mongo StatefulSet replicas=2 role=mongo pvc pvc Read / Write Read / Only Persistent Volume
73. role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C C D Mongo StatefulSet replicas=3 role=mongo Read / Write Read / Only Read / Only DATABASE STATEFUL SETS Scale to 3 replicas Client
74. role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C D Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Unresponsive Pod Client
75. role=mongo type=leader role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 pvc pvc Persistent Volume A B D role=mongo type=follower Mongo-2 pvc C Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Auto recovery Client
76. DATABASE MIGRATIONS
77. Application v3 Development Application V2 Test Application v1 Production DB v1 DB v2 DB v3 CI/CD PIPELINE Version control database updates, ex: flyway V3__add_table_scooter.sql V2__add_table_truck.sql V1__add_table_car.sql
78. DATABASE MIGRATIONS Version control database updates with Containers CONTAINER IMAGE CONTAINER BUILD FILE SQL MIGRATION SCRIPT Source Code Repository V2__add_table.sql Source Code Repository V2__add_table.sql /var/flyway/data Flyway flyway-mydb:v2.0.0 Registry + Dockerfile
79. Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresq pvcl DATABASE MIGRATION StatefulSet deployment with headless Service v1
80. Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Create a Job for Flyway Flyway Job Secrets = Database Connection Info v1 flyway-mydb:v2.0.0 Image Registry Flyway
81. role=postgressql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql pvc DATABASE MIGRATIONS Apply schema changes to database Flyway Job Secrets = Database Connection Info V2 flyway-mydb:v2.0.0 Flyway
82. role=postgresql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Version control for database with Kubernetes V2
83. DATABASE MIGRATIONS WITH BLUE / GREEN DEPLOYMENT
84. WHAT’S NEXT
85. Traffic Control Service Resiliency Chaos Testing Observ- ability Security SERVICE MESH
86. OPERATORS
87. learn.openshift.com
88. THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin