Tweet
Tweet

Slide 1

Slide 1 text

A DevOps State of Mind: Managing Microservices and Databases with Kubernetes Chris Van Tuin Red Hat Chief Technologist, NA West / Silicon Valley linkedin: Chris Van Tuin
 email: [email protected]
 twitter: @chrisvantuin

Slide 2

Slide 2 text

“Only the paranoid survive” - Andy Grove, 1996

Slide 3

Slide 3 text

THE WORLD IS AUTOMATING Those who succeed in automation will win

Slide 4

Slide 4 text

THE CHALLENGE: 
 ENABLE INNOVATION AT SPEED, WHILE EXECUTING AT SCALE WITH EFFICIENCY Static &
 Planned Dynamic & 
 Policy Driven Execution Innovation Old New Execution Innovation

Slide 5

Slide 5 text

IT’S NOT JUST SOFTWARE, THE DISRUPTERS = Empowered organization Speed Up 
 Innovation Time Change Move Fast, Break Things Culture of experimentation A 20% vs. 25% Shorten the Feedback Loop Real-time data-driven intelligence & personalization AI /
 ML Data, Data, Data B

Slide 6

Slide 6 text

I.T. MUST EVOLVE 
 FROM A COST CENTER TO INNOVATION CENTER Development Model Application Architecture Deployment & Application Infrastructur Storage Waterfall Agile Monolithic N-tier Bare Metal Virtual Servers Data Center Hosted Scale Up Scale Out DevOps MicroServices Containers Hybrid Cloud Storage as a Service

Slide 7

Slide 7 text

LAPTOP Container Application OS dependencies Guest VM LINUX BARE METAL Container Application OS dependencies LINUX VIRTUALIZATION Container Application OS dependencies Virtual Machine LINUX PRIVATE CLOUD Container Application OS dependencies Virtual Machine LINUX PUBLIC CLOUD Container Application OS dependencies Virtual Machine LINUX CONTAINERIZED MICROSERVICES
 Build Once, Deploy Anywhere

Slide 8

Slide 8 text

BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory
 Speed, Resiliency, Scalability, Security 


Slide 9

Slide 9 text

Databases Images Automation MANAGING CONTAINERIZED MICROSERVICES
 WITH KUBERNETES A/B Testing Migrations External
 Services Deployment Strategies Security What’s Next… CI/CD

Slide 10

Slide 10 text

KUBERNETES AUTOMATION

Slide 11

Slide 11 text

Web App role=web role=app role=web replicas=1, 
 role=app replicas=2, 
 role=web ORCHESTRATION Pod Node Service Controller Manager & Data Store (etcd) Ingress / (external
 route rule e.g. haproxy)

Slide 12

Slide 12 text

Web App replicas=1, 
 role=app replicas=2, 
 role=web HEALTH CHECK Pod Node Service role=web role=app role=web Controller Manager & Data Store (etcd) Ingress / Route role=app

Slide 13

Slide 13 text

Web App AUTO-SCALE 80% CPU -> Pod Node Service role=web role=app role=web Controller Manager & Data Store (etcd) replicas=2 
 role=app replicas=2, 
 role=web horizontalpodautoscaler -cpu-percent=50 --min=1 —max=3 Ingress / Route 50% CPU role=app

Slide 14

Slide 14 text

CONTAINER IMAGES

Slide 15

Slide 15 text

docker.io Registry Private Registry FROM fedora:1.0 CMD echo “Hello” Build file Physical, Virtual, Cloud Container Image Container Instance Build Run Ship CONTAINERS ENABLE DEVOPS

Slide 16

Slide 16 text

CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS dependencies 1.2/latest 1.1

Slide 17

Slide 17 text

A CONVERGED SOFTWARE 
 SUPPLY CHAIN

Slide 18

Slide 18 text

Config Data Kubernetes configmaps secrets Container image Traditional 
 data services, Kubernetes 
 persistent volumes TREAT CONTAINERS AS IMMUTABLE To keep containerized apps portable Application Language runtimes OS dependencies

Slide 19

Slide 19 text

KUBERNETES CONFIGMAP Decouple configuration from container image Application Language runtimes OS dependencies Environment Variable or Volume/File CONTAINER INSTANCE key:value from directories, files, or values KUBERNETES
 CONFIGMAP APPLICATION CONFIG FILE Application Configuration File e.g. XML etcd Pod Source Code Repository EnvVar require pod restart Files refresh in time

Slide 20

Slide 20 text

CI/CD PIPELINE

Slide 21

Slide 21 text

CI/CD PIPELINE Continuous Integration Continuous Build Continuous Deployment Developer -> Source -> Git Git -> RPMS -> Images-> Registry Images from 
 Registry -> Clusters

Slide 22

Slide 22 text

CI/CD PIPELINE WITH KUBERNETES BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD

Slide 23

Slide 23 text

CONTAINER IMAGE SECURITY

Slide 24

Slide 24 text

WHAT’S INSIDE MATTERS…

Slide 25

Slide 25 text

PRIVATE REGISTRY

Slide 26

Slide 26 text

Security CONTINUOUS INTEGRATION WITH SECURITY SCAN

Slide 27

Slide 27 text

Java Build Environment Language runtimes OS dependencies Build Image Java Code Application Language runtimes OS dependencies Container Image Image Registry Source Repository Image Registry REPRODUCIBLE BUILDS Source to Image with Build Images Source v3.1 v1.0.1 v3.1

Slide 28

Slide 28 text

RAPID INNOVATION & EXPERIMENTATION WITH A/B TESTING

Slide 29

Slide 29 text

”only about 1/3 of ideas improve the metrics 
 they were designed to improve.”
 Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION

Slide 30

Slide 30 text

A/B TESTING USING CANARY DEPLOYMENTS

Slide 31

Slide 31 text

25% Conversion Rate ?! Conversion Rate 100% Version B Version A Ingress CANARY DEPLOYMENTS Tests / CI

Slide 32

Slide 32 text

25% Conversion Rate 30% Conversion Rate 75% 25% Version B Version A Ingress CANARY DEPLOYMENTS

Slide 33

Slide 33 text

25% Conversion Rate 30% Conversion Rate 100% Version B Version A Ingress CANARY DEPLOYMENTS

Slide 34

Slide 34 text

25% Conversion Rate 20% Conversion Rate 100% Version B Version A Rollback Ingress CANARY DEPLOYMENTS

Slide 35

Slide 35 text

CONTINUOUS FEEDBACK LOOP

Slide 36

Slide 36 text

MONITORING CONSIDERATIONS Kubernetes* Container* Host Cluster services, services, pods, 
 deployments metrics Container native metrics Traditional resource metrics - cpu, memory, network, storage prometheus + grafana kubernetes-state-metrics probes Stack Metrics Tool node-exporter kubelet:cAdvisor Microservices Distributed applications - traditional app metrics - service discovery - distributed tracing prometheus + grafana jaeger tracing istio

Slide 37

Slide 37 text

Databases Images Automation MANAGING CONTAINERIZED MICROSERVICES
 WITH KUBERNETES A/B Testing Migrations External
 Services Deployment Strategies Security What’s Next… CI/CD

Slide 38

Slide 38 text

DEPLOYMENT STRATEGIES

Slide 39

Slide 39 text

CONTINUOUS DELIVERY WITH CONTAINERS

Slide 40

Slide 40 text

CONTINUOUS DELIVERY DEPLOYMENT STRATEGIES DEPLOYMENT STRATEGIES • Recreate • Rolling updates • Blue / Green deployment

Slide 41

Slide 41 text

Recreate

Slide 42

Slide 42 text

Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME

Slide 43

Slide 43 text

Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME

Slide 44

Slide 44 text

Version 1.2 Version 1.2 Version 1.2 RECREATE WITH DOWNTIME Use Case • Non-mission critical services Pros • Simple, clean • No Schema incompatibilities • No API versioning Cons • Downtime

Slide 45

Slide 45 text

Rolling Updates

Slide 46

Slide 46 text

Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI ROLLING UPDATES with ZERO DOWNTIME Rollingupdate
 maxUnavailable=0 maxSurge=1

Slide 47

Slide 47 text

Deploy new version and wait until it’s ready… Health Check: readiness probe e.g. tcp, http, script Version 1 Version 1 Version 
 1.2 Version 1 Rollingupdate
 maxUnavailable=0 maxSurge=1

Slide 48

Slide 48 text

Each container/pod is updated one by one Version 1.2 50% Version 1 V1 V1.2

Slide 49

Slide 49 text

Each container/pod is updated one by one Version 1.2 Version 1.2 Version 1.2 100% Use Case • Horizontally scaled • Backward compatible API/data • Microservices Pros • Zero downtime • Reduced risk, gradual rollout w/health checks • Ready for rollback Cons • Require backward compatible APIs/data • Resource overhead

Slide 50

Slide 50 text

Blue / Green Deployment

Slide 51

Slide 51 text

BLUE Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100%

Slide 52

Slide 52 text

BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% Health Check: readiness probe e.g. tcp, http, script

Slide 53

Slide 53 text

BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100%

Slide 54

Slide 54 text

BLUE / GREEN DEPLOYMENT Rollback BLUE GREEN Version 1 Version 2 Ingress Use Case • Self-contained micro services (data) Pros • Low risk, never change production • No downtime • Production like testing • Rollback Cons • Resource overhead • Data synchronization

Slide 55

Slide 55 text

EXTERNAL SERVICES

Slide 56

Slide 56 text

EXTERNAL SERVICES Database outside cluster with IP address External Mongo Database Service External Mongo Database Service Development Production IP=10.200.0.2 port=27017 IP=10.100.0.9 port=27017

Slide 57

Slide 57 text

EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017

Slide 58

Slide 58 text

EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.200.0.2 port=27017

Slide 59

Slide 59 text

EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.100.0.9 port=27017

Slide 60

Slide 60 text

Cloud Mongo Database Service 
 mongodb://:
 @mongo48909.domain.name:48909/dev 
 mongodb://:
 @mongo52101.domain.name:52101/dev Cloud Mongo Database Service Development Production EXTERNAL SERVICES Remotely hosted database with URI

Slide 61

Slide 61 text

Pods Nodes Services Database name: mongo type: ExternalName externalName: mongo52101.domain,.name EXTERNAL SERVICES Using CNAME redirection mongodb://
 :
 
 @mongo:/dev 
 mongodb://:
 @mongo52101.domain.name:52101/dev Cloud Mongo Database Service WebApp role=webapp replicas=2, 
 role=webapp

Slide 62

Slide 62 text

DATABASES

Slide 63

Slide 63 text

PERSISTENT VOLUMES Host Container Host Container Host Container Data in Container Data lost when Container terminates Data lost when Host terminates Independent of Container & Host Data in a Host Volume Networked Volume

Slide 64

Slide 64 text

1. Maintains a sticky network ID/name across restarts
 e.g. mongo-0, mongo-1, mongo-2 2. Ordered Operations with ordinal index 
 e.g. name-0, name-1, name-2 3. Stable, persistent storage (linked to ordinal index/name) 4. Mandatory headless service (no single IP) for integrations KUBERNETES
 STATEFULSETS

Slide 65

Slide 65 text

role=mongo type=leader Nodes Pods Services Mongo StatefulSet replicas=2 role=mongo Client mongo-0 D A B C C DATABASE STATEFUL SETS StatefulSet with 2 replicas , headless service, direct access to pods pvc Read / Write Persistent Volume

Slide 66

Slide 66 text

DATABASE STATEFUL SETS role=mongo type=leader role=mongo type=follower Nodes Pods Services Client Mongo-0 Mongo-1 D A B C C Mongo StatefulSet replicas=2 role=mongo pvc pvc Read / Write Read / Only Persistent Volume

Slide 67

Slide 67 text

role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C C D Mongo StatefulSet replicas=3 role=mongo Read / Write Read / Only Read / Only DATABASE STATEFUL SETS Scale to 3 replicas Client

Slide 68

Slide 68 text

role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C D Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Unresponsive Pod Client

Slide 69

Slide 69 text

role=mongo type=leader role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 pvc pvc Persistent Volume A B D role=mongo type=follower Mongo-2 pvc C Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Auto recovery Client

Slide 70

Slide 70 text

DATABASE MIGRATIONS

Slide 71

Slide 71 text

DATABASE MIGRATIONS Version control database updates with Containers CONTAINER IMAGE CONTAINER BUILD FILE SQL MIGRATION SCRIPT Source Code Repository V2__add_table.sql Source Code Repository V2__add_table.sql /var/flyway/data Flyway flyway-mydb:v2.0.0 Registry + Dockerfile

Slide 72

Slide 72 text

Application v3 Development Application V2 Test Application v1 Production DB v1 DB v2 DB v3 CI/CD PIPELINE Version control database updates, ex: flyway V3__add_table_scooter.sql V2__add_table_truck.sql V1__add_table_car.sql

Slide 73

Slide 73 text

Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresq pvcl DATABASE MIGRATION StatefulSet deployment with headless Service v1

Slide 74

Slide 74 text

Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Create a Job for Flyway Flyway Job Secrets = Database Connection Info v1 flyway-mydb:v2.0.0 Image Registry Flyway

Slide 75

Slide 75 text

role=postgressql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql pvc DATABASE MIGRATIONS Apply schema changes to database Flyway Job Secrets = Database Connection Info V2 flyway-mydb:v2.0.0 Flyway

Slide 76

Slide 76 text

role=postgresql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Version control for database with Kubernetes V2

Slide 77

Slide 77 text

WHAT’S NEXT

Slide 78

Slide 78 text

Traffic Control Service Resiliency Chaos Testing Observ- ability Security

Slide 79

Slide 79 text

OPERATORS

Slide 80

Slide 80 text

THANK YOU linkedin: Chris Van Tuin email: [email protected] twitter: @chrisvantuin