DevOpsDays Victoria 2019: Microservices and Databases with Kubernetes

Slide 1

Slide 1 text

A DEVOPS STATE OF MIND: MICROSERVICES AND DATABASES WITH KUBERNETES Chris Van Tuin  Chief Technologist, NA West @chrisvantuin cvantuin@redhat.com

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

THE WORLD IS AUTOMATING Those who succeed in automation will win

Slide 4

Slide 4 text

DEV QA OPS THE AVERAGE ENTERPRISE   DOES DEPLOYMENTS EVERY 6 TO 9 MONTHS. Walled off people, walled off processes, walled off technologies with surprisingly little to no automation

Slide 5

Slide 5 text

HOW DOES I.T. ACCELERATE   BUSINESS INNOVATION? 6 to 9 months Innovation Hours to Weeks

Slide 6

Slide 6 text

ENABLING INNOVATION, WHILE EXECUTING AT SCALE Static &  Planned Dynamic &   Policy Driven Execution Innovation Innovation Execution Old New

Slide 7

Slide 7 text

+ ≠ TAXI TRANSPORTATION DISRUPTER https://goo.gl/MP7QQH Ack: Andrew Ng APP IT’S NOT JUST A… +

Slide 8

Slide 8 text

THE STRATEGIC DIFFERENTIATOR The Fab  Powered by Automation “copy exactly” The Software Factory Powered by Automation

Slide 9

Slide 9 text

MOST THAN JUST SOFTWARE,   THE DISRUPTERS… Empowered organization Speed Up   Innovation Time Change Move Fast, Break Things Culture of experimentation A 20% vs. 25% Shorten the Feedback Loop Real-time data-driven intelligence & personalization AI /  ML Data, Data, Data B

Slide 10

Slide 10 text

I.T. ENABLING   BUSINESS DIFFERENTIATION Taxi Ridesharing $2.50/mile, $0.50/idle Fare Static  Pricing Dynamic   Pricing vs. Ack: William Benton Fare   based on   real-time and historical data: supply/demand distance, time  route, traffic etc.

Slide 11

Slide 11 text

I.T. MUST TRANSFORM FROM A COST CENTER   INTO AN INNOVATION CENTER Powered by DevOps + Automation + + DEV QA OPS Culture Process   Automation Technology Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Cloud Native Applications Hybrid Cloud Open Source Agile, Iterative, Continuous, Infrastructure as Code Collaborative Transparent Open THE SOFTWARE FACTORY

Slide 12

Slide 12 text

IT MUST EVOLVE & KEEP UP

Slide 13

Slide 13 text

CONTAINERS ENABLE DEVOPS

Slide 14

Slide 14 text

CONTAINERS Software packaging concept that typically includes an application and all of its runtime dependencies • HIGHER quality software releases • SHORTER test cycles • EASIER application management CONTAINER CONTAINER APP LIBS HOST OS SERVER APP LIBS BENEFITS

Slide 15

Slide 15 text

LAPTOP Container Application OS dependencies Guest VM LINUX BARE METAL Container Application OS dependencies LINUX VIRTUALIZATION Container Application OS dependencies Virtual Machine LINUX PRIVATE CLOUD Container Application OS dependencies Virtual Machine LINUX PUBLIC CLOUD Container Application OS dependencies Virtual Machine LINUX CONTAINERIZED MICROSERVICES  Build Once, Deploy Anywhere

Slide 16

Slide 16 text

Image Format Distribution Spec Runtime Spec

Slide 17

Slide 17 text

CLOUD NATIVE CONTAINER DESIGN

Slide 18

Slide 18 text

https://medium.com/@bibryam/cloud-native-container-design-principles-144ceaa98dba CLOUD NATIVE CONTAINER DESIGN PRINCIPLES: Build Time Does one thing well Built with depenedencies, depends on Linux kernel Build once, deploy anywhere Single Concern Principle Self Containment Principle Image Imutability Principle

Slide 19

Slide 19 text

Process Dispensability Principle CLOUD NATIVE CONTAINER DESIGN PRINCIPLES: Runtime Ephemeral - short lived, replaceable Read & react to events APIs to obeserver & manage Resource requirements defined and restricted High Observability Principle Lifecycle Conformance Principle Runtime Confinement Principle

Slide 20

Slide 20 text

CONTAINER IMAGES

Slide 21

Slide 21 text

docker.io Registry Private Registry FROM fedora:1.0 CMD echo “Hello” Build file Physical, Virtual, Cloud Container Image Container Instance Build Run Ship CONTAINERS ENABLE DEVOPS CONTAINERS ENABLE DEVSECOPS FROM registry.redhat.com/rhel7 RUN groupadd -g 999 appuser && \ useradd -r -u 999 -g appuser appuser USER appuser CMD echo “Hello”

Slide 22

Slide 22 text

CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS dependencies 1.2/latest 1.1

Slide 23

Slide 23 text

ERGED SOFTWARE   UPPLY CHAIN TAINER IMAGE CONTAINER IMAGE Application Language runtimes OS dependencies 1.2/latest 1.1 TAINER IMAGE CONTAINER IMAGE Application Language runtimes OS dependencies 1.2/latest 1.1 TAINER IMAGE CONTAINER IMAGE Application Language runtimes OS dependencies 1.2/latest 1.1 SUPPLY CHAIN CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS dependencies 1.2/latest 1.1 CONVERGED SOFTWARE SUPPLY CHAIN Build file Container Image CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS dependencies Container Instance BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory  Speed, Resiliency, Scalability, Security   Configs / Storage Developer Apps / DB Operations

Slide 24

Slide 24 text

CONTAINERS AT SCALE

Slide 25

Slide 25 text

Scheduling Monitoring Persistence Discovery Lifecycle & health Scaling Aggregation Security MORE THAN CONTAINERS…

Slide 26

Slide 26 text

BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory  Speed, Resiliency, Scalability, Security   BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory  Speed, Resiliency, Scalability, Security   BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory  Speed, Resiliency, Scalability, Security   Speed, Agility, Resiliency, Scalability, Efficiency, Security

Slide 27

Slide 27 text

KUBERNETES AUTOMATION

Slide 28

Slide 28 text

DEMO APPLICATION Web App DEV QA OPS THE AVERAGE ENTERPRISE   DOES DEPLOYMENTS EVERY 6 TO 9 MONTHS. Walled off people, walled off processes, walled off technologies with surprisingly little to no automation DEV QA OPS THE AVERAGE ENTERPRISE   DOES DEPLOYMENTS EVERY 6 TO 9 MONTHS. Walled off people, walled off processes, walled off technologies with surprisingly little to no automation

Slide 29

Slide 29 text

Web Application replicas: 1,   role: app image: myapp:1.0 replicas: 2,   role: web image: httpd:1.7.9 ORCHESTRATION Speed & Agility: Declarative, Deployment Controller Manager & Data Store (etcd)

Slide 30

Slide 30 text

Web Application ORCHESTRATION Declarative, Deployment Nodes Controller Manager & Data Store (etcd) Physical, VM,   Cloud Instances replicas: 2,   role: web image: httpd:1.7.9 replicas: 1,   role: app image: myapp:1.0

Slide 31

Slide 31 text

role: app role: web role: web Pods Nodes Image Registry ORCHESTRATION Schedule + Provision Pods (Compute/Storage/Network) Web Application replicas: 2,   role: web image: httpd:1.7.9 replicas: 1,   role: app image: myapp:1.0

Slide 32

Slide 32 text

Web Application role: web role: app role: web replicas: 1,   role: app replicas: 2,   role: web ORCHESTRATION Services (Load Balancer), Service discovery with selectors and pod labels Pods Nodes Services Controller Manager & Data Store (etcd)

Slide 33

Slide 33 text

Web Application ORCHESTRATION Service (Load Balancer) Pods Nodes Controller Manager & Data Store (etcd) Ingress / Routes role: web role: app role: web replicas: 1,   role: app replicas: 2,   role: web Services

Slide 34

Slide 34 text

HEALTH CHECK Resiliency Monitoring & Logging Pods Nodes Services Web Application role: web role: app role: web Ingress / Routes Health Check replicas: 1,   role: app replicas: 2,   role: web

Slide 35

Slide 35 text

Pods Nodes Services Web Application role: web role: app role: web replicas: 1,   role: app replicas: 2,   role: web role: web Controller Manager & Data Store (etcd) HEALTH CHECK Readiness Probe e.g. tcp, http, script Ingress / Routes

Slide 36

Slide 36 text

Web Application replicas: 1,   role: app replicas: 2,   role: web Pods Nodes Services role: web role: app role: web Controller Manager & Data Store (etcd) HEALTH CHECK Ingress / Routes

Slide 37

Slide 37 text

Web Application Monitoring & Logging 80% CPU Pods Nodes Services role: web role: app role: web Ingress / Routes replicas: 1,   role: app replicas: 2,   role: web AUTO-SCALE Scalability & Efficiency

Slide 38

Slide 38 text

Web Application 80% CPU Pods Nodes Services role: web role: app role: web Controller Manager & Data Store (etcd) role: app AUTO-SCALE Ingress / Routes replicas: 2   role: app replicas: 2,   role: web

Slide 39

Slide 39 text

Pods Nodes Services Web Application 50% CPU role: web role: app role: app role: web Controller Manager & Data Store (etcd) AUTO-SCALE Ingress / Routes replicas: 2,   role: web replicas: 2,   role: app

Slide 40

Slide 40 text

CI/CD

Slide 41

Slide 41 text

CONTINUOUS DELIVERY WITH CONTAINERS CI/CD PIPLINE

Slide 42

Slide 42 text

A/B TESTING WITH CANARIES

Slide 43

Slide 43 text

”only about 1/3 of ideas improve the metrics   they were designed to improve.”  Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION

Slide 44

Slide 44 text

A/B TESTING USING CANARY DEPLOYMENTS

Slide 45

Slide 45 text

25% Conversion Rate ?! Conversion Rate 100% Version B Version A Ingress CANARY DEPLOYMENTS Tests / CI CANARY DEPLOYMENTS  Build confidence in new version Service  selector:  app=demo version=A label:  app=demo  version=A 25% Conversion Rate ??% Conversion Rate

Slide 46

Slide 46 text

25% Conversion Rate 30% Conversion Rate 75% 25% Version B Version A Ingress CANARY DEPLOYMENTS CANARY DEPLOYMENTS  Requires app to support side-by-side version Service Service  selector:  app=demo label:  app=demo  version=A 25% Conversion Rate % Conversion Rate label:  app=demo  version=B

Slide 47

Slide 47 text

25% Conversion Rate 30% Conversion Rate 100% Version B Version A Ingress CANARY DEPLOYMENTS Service  selector:  app=demo version=B label:  app=demo  version=A 25% Conversion Rate 30% Conversion Rate label:  app=demo  version=B

Slide 48

Slide 48 text

EXTERNAL SERVICES

Slide 49

Slide 49 text

EXTERNAL SERVICES Database outside cluster with IP address External Mongo Database Service External Mongo Database Service Development Production IP=10.200.0.2 port=27017 IP=10.100.0.9 port=27017

Slide 50

Slide 50 text

Slide 51

Slide 51 text

EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes Services WebApp role=webapp replicas=2,   role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.200.0.2 port=27017 Database kind=Service type=ClusterIP name=mongo port=27017 targetport=27017

Slide 52

Slide 52 text

EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes Services WebApp role=webapp replicas=2,   role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.200.0.2 port=27017 Connect with mongodb://mongo Database kind=Service type=ClusterIP name=mongo port=27017 targetport=27017 kind=Endpoints name=mongo ip=10.200.0.2 port=27017

Slide 53

Slide 53 text

EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes Services WebApp role=webapp replicas=2,   role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.100.0.9 port=27017 kind=Service type=ClusterIP name=mongo port=27017 targetport=27017 kind=Endpoints name=mongo ip=10.200.0.9 port=27017 Connect with mongodb://mongo Database

Slide 54

Slide 54 text

Pods Nodes Services Database name: mongo type: ExternalName externalName: mongo52101.domain,.name EXTERNAL SERVICES Using CNAME redirection mongodb://  :    @mongo:/dev   mongodb://:  @mongo52101.domain.name:52101/dev Cloud Mongo Database Service WebApp role=webapp replicas=2,   role=webapp .name EXTERNAL SERVICE Connecting to Service with dynamic URI with a static ExternalName Kubernetes service

Slide 55

Slide 55 text

DATABASES

Slide 56

Slide 56 text

PERSISTENT VOLUMES Host Container Host Container Host Container Data in Container Data lost when Container terminates Data lost when Host terminates Independent of Container & Host Data in a Host Volume Networked Volume Data lost when Cloud instance    (local) Data lost when Container terminates      (image) Independent of   Container &   Cloud instance (EBS/network) DATA PERSISTENCE

Slide 57

Slide 57 text

1. Maintains a sticky network ID/name across restarts  e.g. mongo-0, mongo-1, mongo-2 2. Ordered Operations with ordinal index   e.g. name-0, name-1, name-2 3. Stable, persistent storage (linked to ordinal index/name) 4. Mandatory headless service (no single IP) for integrations KUBERNETES  STATEFULSETS

Slide 58

Slide 58 text

role=mongo type=leader Nodes Pods Services Mongo StatefulSet replicas=2 role=mongo Client mongo-0 D A B C C DATABASE STATEFUL SETS StatefulSet with 2 replicas , headless service, direct access to pods pvc Read / Write Persistent Volume

Slide 59

Slide 59 text

DATABASE STATEFUL SETS role=mongo type=leader role=mongo type=follower Nodes Pods Services Client Mongo-0 Mongo-1 D A B C C Mongo StatefulSet replicas=2 role=mongo pvc pvc Read / Write Read / Only Persistent Volume

Slide 60

Slide 60 text

role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C C D Mongo StatefulSet replicas=3 role=mongo Read / Write Read / Only Read / Only DATABASE STATEFUL SETS Scale to 3 replicas Client

Slide 61

Slide 61 text

role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C D Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Unresponsive Pod Client

Slide 62

Slide 62 text

role=mongo type=leader role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 pvc pvc Persistent Volume A B D role=mongo type=follower Mongo-2 pvc C Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Auto recovery Client

Slide 63

Slide 63 text

DATABASE MIGRATIONS

Slide 64

Slide 64 text

Application v3 Development Application V2 Test Application v1 Production DB v1 DB v2 DB v3 CI/CD PIPELINE Version control database updates, ex: flyway V3__add_table_scooter.sql V2__add_table_truck.sql V1__add_table_car.sql

Slide 65

Slide 65 text

DATABASE MIGRATIONS Version control database updates with Containers CONTAINER IMAGE CONTAINER BUILD FILE SQL MIGRATION SCRIPT Source Code Repository V2__add_table.sql Source Code Repository V2__add_table.sql /var/flyway/data Flyway flyway-mydb:v2.0.0 Registry + Dockerfile

Slide 66

Slide 66 text

Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresq pvcl DATABASE MIGRATION StatefulSet deployment with headless Service v1

Slide 67

Slide 67 text

Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Create a Job for Flyway Flyway Job Secrets = Database Connection Info v1 flyway-mydb:v2.0.0 Image Registry Flyway

Slide 68

Slide 68 text

role=postgressql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql pvc DATABASE MIGRATIONS Apply schema changes to database Flyway Job Secrets = Database Connection Info V2 flyway-mydb:v2.0.0 Flyway

Slide 69

Slide 69 text

role=postgresql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Version control for database with Kubernetes V2

Slide 70

Slide 70 text

MONITORING CONSIDERATIONS Kubernetes* Container* Host Cluster services, services, pods,   deployments metrics Container native metrics Traditional resource metrics - cpu, memory, network, storage prometheus + grafana kubernetes-state-metrics probes Stack Metrics Tool node-exporter Kubernetes metrics server: kubelet:cAdvisor Microservices Distributed applications - traditional app metrics - service discovery - distributed tracing prometheus + grafana jaeger tracing istio

Slide 71

Slide 71 text

WHAT’S NEXT?

Slide 72

Slide 72 text

KUBERNETES NATIVE ADD-ONS knative github.com/knative istio istio.io Server-  less Service Mesh CI/CD tekton tekton.dev

Slide 73

Slide 73 text

KNATIVE FOR SERVERLESS

Slide 74

Slide 74 text

SERVICE MESH

Slide 75

Slide 75 text

GENERAL DISTRIBUTION ISTIO BENEFITS Istio makes it easy to create a network of deployed services with load balancing, service-to-service authentication, monitoring, and more, helping to avoid operational nightmares. POLICY ROUTING TELEMETRY Provides the observability needed to manage microservices, such as how services are invoked, communication flows, and points of latency Allows for the control of routing flows Grants the ability to write policy that applies to all applications and is not language specific

Slide 76

Slide 76 text

GENERAL DISTRIBUTION INCREASING THE VALUE OF ISTIO VISUALIZATION TRACING Kiali includes Jaeger Tracing, which provides distributed tracing out of the box. Kiali works with Istio to visualize the service mesh topology, features like circuit breakers or request rates.

Slide 77

Slide 77 text

Traffic Control Service Resiliency Chaos Testing Observ- ability Security SERVICE MESH WITH ISTIO Dedicated infrastructure layer for making service-to-service communication   safe, fast, and reliable Deploy as a lightweight side-car network proxy

Slide 78

Slide 78 text

CONFIDENTIAL - FOR INTERNAL USE ONLY MICROSERVICES WITHOUT ISTIO Container JVM service A discovery load-balancer resiliency metrics tracing app logic JVM service B discovery load-balancer resiliency metrics tracing app logic Container JVM service C discovery load-balancer resiliency metrics tracing app logic

Slide 79

Slide 79 text

CONFIDENTIAL - FOR INTERNAL USE ONLY MICROSERVICES WITH ISTIO Container JVM service C app logic Pod Sidecar Container Envoy Container JVM service A app logic Pod Sidecar Container Envoy Container JVM service B app logic Pod Sidecar Container Envoy

Slide 80

Slide 80 text

ISTIO SERVICE MESH Envoy istio-ingress Envoy App A Envoy App B Envoy App C istio-pilot istio-mixer istio-auth HTTP Req/Resp Kubernetes Pods Istio Components Config to Envoy Access Control and Telemetry

Slide 81

Slide 81 text

Istio (& Kiali)

Slide 82

Slide 82 text

24% 76% v1 v2 apiVersion: config.istio.io/v1alpha2 kind: RouteRule metadata: name: recommendation-v1-v2 spec: destination: namespace: tutorial name: recommendation precedence: 5 route: - labels: version: v1 weight: 76 - labels: version: v2 weight: 24 CANARY RELEASE BY WEIGHT RouteRule #2:   Route 94% to v1 and 6% to v2

Slide 83

Slide 83 text

“.*Safari.*” Default v1 v2 apiVersion: config.istio.io/v1alpha2 kind: RouteRule metadata: name: recommendation-safari spec: destination: namespace: tutorial name: recommendation precedence: 2 match: request: headers: user-agent: regex: ".*Safari.*" route: - labels: version: v2 ROUTING BY HEADER By Geography, Mobile Device, Browser, Customer, … RouteRule #3:   Route “Safari” to v2

Slide 84

Slide 84 text

v1 v2 apiVersion: config.istio.io/v1alpha2 kind: RouteRule metadata: name: recommendation-mirror spec: destination: namespace: tutorial name: recommendation precedence: 2 route: - labels: version: v1 weight: 100 - labels: version: v2 weight: 0 mirror: namespace: tutorial name: recommendation labels: version: v2 DARK LAUNCH Mirror production traffic for pre-release testing RouteRule #3:   Route “Safari” to v2 Mirror Production Traffic To v2 100% 100% Test Production

Slide 85

Slide 85 text

apiVersion: config.istio.io/v1alpha2 kind: EgressRule metadata: name: httpbin-egress-rule spec: destination: service: httpbin.org ports: - port: 80 protocol: http SECURE BY DEFAULT Egress blocks all traffic unless unless whitelisted with EgressRule EgressRule: Allow httpbin.org:80   (http) role=web Pods Nodes http://httpbin.org Istio EgressRule

Slide 86

Slide 86 text

THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin