Save 37% off PRO during our Black Friday Sale! »

DevOpsDays Victoria 2019: Microservices and Databases with Kubernetes

DevOpsDays Victoria 2019: Microservices and Databases with Kubernetes

DevOpsDays Victoria 2019

7c6a033dd957d547b49630f626e1a143?s=128

Chris Van Tuin

May 31, 2019
Tweet

Transcript

  1. A DEVOPS STATE OF MIND: MICROSERVICES AND DATABASES WITH KUBERNETES

    Chris Van Tuin
 Chief Technologist, NA West @chrisvantuin cvantuin@redhat.com
  2. None
  3. THE WORLD IS AUTOMATING Those who succeed in automation will

    win
  4. DEV QA OPS THE AVERAGE ENTERPRISE 
 DOES DEPLOYMENTS EVERY

    6 TO 9 MONTHS. Walled off people, walled off processes, walled off technologies with surprisingly little to no automation
  5. HOW DOES I.T. ACCELERATE 
 BUSINESS INNOVATION? 6 to 9

    months Innovation Hours to Weeks
  6. ENABLING INNOVATION, WHILE EXECUTING AT SCALE Static &
 Planned Dynamic

    & 
 Policy Driven Execution Innovation Innovation Execution Old New
  7. + ≠ TAXI TRANSPORTATION DISRUPTER https://goo.gl/MP7QQH Ack: Andrew Ng APP

    IT’S NOT JUST A… +
  8. THE STRATEGIC DIFFERENTIATOR The Fab
 Powered by Automation “copy exactly”

    The Software Factory Powered by Automation
  9. MOST THAN JUST SOFTWARE, 
 THE DISRUPTERS… Empowered organization Speed

    Up 
 Innovation Time Change Move Fast, Break Things Culture of experimentation A 20% vs. 25% Shorten the Feedback Loop Real-time data-driven intelligence & personalization AI /
 ML Data, Data, Data B
  10. I.T. ENABLING 
 BUSINESS DIFFERENTIATION Taxi Ridesharing $2.50/mile, $0.50/idle Fare

    Static
 Pricing Dynamic 
 Pricing vs. Ack: William Benton Fare 
 based on 
 real-time and historical data: supply/demand distance, time
 route, traffic etc.
  11. I.T. MUST TRANSFORM FROM A COST CENTER 
 INTO AN

    INNOVATION CENTER Powered by DevOps + Automation + + DEV QA OPS Culture Process 
 Automation Technology Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Cloud Native Applications Hybrid Cloud Open Source Agile, Iterative, Continuous, Infrastructure as Code Collaborative Transparent Open THE SOFTWARE FACTORY
  12. IT MUST EVOLVE & KEEP UP

  13. CONTAINERS ENABLE DEVOPS

  14. CONTAINERS Software packaging concept that typically includes an application and

    all of its runtime dependencies • HIGHER quality software releases • SHORTER test cycles • EASIER application management CONTAINER CONTAINER APP LIBS HOST OS SERVER APP LIBS BENEFITS
  15. LAPTOP Container Application OS dependencies Guest VM LINUX BARE METAL

    Container Application OS dependencies LINUX VIRTUALIZATION Container Application OS dependencies Virtual Machine LINUX PRIVATE CLOUD Container Application OS dependencies Virtual Machine LINUX PUBLIC CLOUD Container Application OS dependencies Virtual Machine LINUX CONTAINERIZED MICROSERVICES
 Build Once, Deploy Anywhere
  16. Image Format Distribution Spec Runtime Spec

  17. CLOUD NATIVE CONTAINER DESIGN

  18. https://medium.com/@bibryam/cloud-native-container-design-principles-144ceaa98dba CLOUD NATIVE CONTAINER DESIGN PRINCIPLES: Build Time Does one

    thing well Built with depenedencies, depends on Linux kernel Build once, deploy anywhere Single Concern Principle Self Containment Principle Image Imutability Principle
  19. Process Dispensability Principle CLOUD NATIVE CONTAINER DESIGN PRINCIPLES: Runtime Ephemeral

    - short lived, replaceable Read & react to events APIs to obeserver & manage Resource requirements defined and restricted High Observability Principle Lifecycle Conformance Principle Runtime Confinement Principle
  20. CONTAINER IMAGES

  21. docker.io Registry Private Registry FROM fedora:1.0 CMD echo “Hello” Build

    file Physical, Virtual, Cloud Container Image Container Instance Build Run Ship CONTAINERS ENABLE DEVOPS CONTAINERS ENABLE DEVSECOPS FROM registry.redhat.com/rhel7 RUN groupadd -g 999 appuser && \ useradd -r -u 999 -g appuser appuser USER appuser CMD echo “Hello”
  22. CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS

    dependencies 1.2/latest 1.1
  23. ERGED SOFTWARE 
 UPPLY CHAIN TAINER IMAGE CONTAINER IMAGE Application

    Language runtimes OS dependencies 1.2/latest 1.1 TAINER IMAGE CONTAINER IMAGE Application Language runtimes OS dependencies 1.2/latest 1.1 TAINER IMAGE CONTAINER IMAGE Application Language runtimes OS dependencies 1.2/latest 1.1 SUPPLY CHAIN CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS dependencies 1.2/latest 1.1 CONVERGED SOFTWARE SUPPLY CHAIN Build file Container Image CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS dependencies Container Instance BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory
 Speed, Resiliency, Scalability, Security 
 Configs / Storage Developer Apps / DB Operations
  24. CONTAINERS AT SCALE

  25. Scheduling Monitoring Persistence Discovery Lifecycle & health Scaling Aggregation Security

    MORE THAN CONTAINERS…
  26. BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory


    Speed, Resiliency, Scalability, Security 
 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory
 Speed, Resiliency, Scalability, Security 
 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory
 Speed, Resiliency, Scalability, Security 
 Speed, Agility, Resiliency, Scalability, Efficiency, Security
  27. KUBERNETES AUTOMATION

  28. DEMO APPLICATION Web App DEV QA OPS THE AVERAGE ENTERPRISE

    
 DOES DEPLOYMENTS EVERY 6 TO 9 MONTHS. Walled off people, walled off processes, walled off technologies with surprisingly little to no automation DEV QA OPS THE AVERAGE ENTERPRISE 
 DOES DEPLOYMENTS EVERY 6 TO 9 MONTHS. Walled off people, walled off processes, walled off technologies with surprisingly little to no automation
  29. Web Application replicas: 1, 
 role: app image: myapp:1.0 replicas:

    2, 
 role: web image: httpd:1.7.9 ORCHESTRATION Speed & Agility: Declarative, Deployment Controller Manager & Data Store (etcd)
  30. Web Application ORCHESTRATION Declarative, Deployment Nodes Controller Manager & Data

    Store (etcd) Physical, VM, 
 Cloud Instances replicas: 2, 
 role: web image: httpd:1.7.9 replicas: 1, 
 role: app image: myapp:1.0
  31. role: app role: web role: web Pods Nodes Image Registry

    ORCHESTRATION Schedule + Provision Pods (Compute/Storage/Network) Web Application replicas: 2, 
 role: web image: httpd:1.7.9 replicas: 1, 
 role: app image: myapp:1.0
  32. Web Application role: web role: app role: web replicas: 1,

    
 role: app replicas: 2, 
 role: web ORCHESTRATION Services (Load Balancer), Service discovery with selectors and pod labels Pods Nodes Services Controller Manager & Data Store (etcd)
  33. Web Application ORCHESTRATION Service (Load Balancer) Pods Nodes Controller Manager

    & Data Store (etcd) Ingress / Routes role: web role: app role: web replicas: 1, 
 role: app replicas: 2, 
 role: web Services
  34. HEALTH CHECK Resiliency Monitoring & Logging Pods Nodes Services Web

    Application role: web role: app role: web Ingress / Routes Health Check replicas: 1, 
 role: app replicas: 2, 
 role: web
  35. Pods Nodes Services Web Application role: web role: app role:

    web replicas: 1, 
 role: app replicas: 2, 
 role: web role: web Controller Manager & Data Store (etcd) HEALTH CHECK Readiness Probe e.g. tcp, http, script Ingress / Routes
  36. Web Application replicas: 1, 
 role: app replicas: 2, 


    role: web Pods Nodes Services role: web role: app role: web Controller Manager & Data Store (etcd) HEALTH CHECK Ingress / Routes
  37. Web Application Monitoring & Logging 80% CPU Pods Nodes Services

    role: web role: app role: web Ingress / Routes replicas: 1, 
 role: app replicas: 2, 
 role: web AUTO-SCALE Scalability & Efficiency
  38. Web Application 80% CPU Pods Nodes Services role: web role:

    app role: web Controller Manager & Data Store (etcd) role: app AUTO-SCALE Ingress / Routes replicas: 2 
 role: app replicas: 2, 
 role: web
  39. Pods Nodes Services Web Application 50% CPU role: web role:

    app role: app role: web Controller Manager & Data Store (etcd) AUTO-SCALE Ingress / Routes replicas: 2, 
 role: web replicas: 2, 
 role: app
  40. CI/CD

  41. CONTINUOUS DELIVERY WITH CONTAINERS CI/CD PIPLINE

  42. A/B TESTING WITH CANARIES

  43. ”only about 1/3 of ideas improve the metrics 
 they

    were designed to improve.”
 Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION
  44. A/B TESTING USING CANARY DEPLOYMENTS

  45. 25% Conversion Rate ?! Conversion Rate 100% Version B Version

    A Ingress CANARY DEPLOYMENTS Tests / CI CANARY DEPLOYMENTS
 Build confidence in new version Service
 selector:
 app=demo version=A label:
 app=demo
 version=A 25% Conversion Rate ??% Conversion Rate
  46. 25% Conversion Rate 30% Conversion Rate 75% 25% Version B

    Version A Ingress CANARY DEPLOYMENTS CANARY DEPLOYMENTS
 Requires app to support side-by-side version Service Service
 selector:
 app=demo label:
 app=demo
 version=A 25% Conversion Rate % Conversion Rate label:
 app=demo
 version=B
  47. 25% Conversion Rate 30% Conversion Rate 100% Version B Version

    A Ingress CANARY DEPLOYMENTS Service
 selector:
 app=demo version=B label:
 app=demo
 version=A 25% Conversion Rate 30% Conversion Rate label:
 app=demo
 version=B
  48. EXTERNAL SERVICES

  49. EXTERNAL SERVICES Database outside cluster with IP address External Mongo

    Database Service External Mongo Database Service Development Production IP=10.200.0.2 port=27017 IP=10.100.0.9 port=27017
  50. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017
  51. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.200.0.2 port=27017 Database kind=Service type=ClusterIP name=mongo port=27017 targetport=27017
  52. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.200.0.2 port=27017 Connect with mongodb://mongo Database kind=Service type=ClusterIP name=mongo port=27017 targetport=27017 kind=Endpoints name=mongo ip=10.200.0.2 port=27017
  53. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.100.0.9 port=27017 kind=Service type=ClusterIP name=mongo port=27017 targetport=27017 kind=Endpoints name=mongo ip=10.200.0.9 port=27017 Connect with mongodb://mongo Database
  54. Pods Nodes Services Database name: mongo type: ExternalName externalName: mongo52101.domain,.name

    EXTERNAL SERVICES Using CNAME redirection mongodb://
 <dbuser>:
 <dbpassword>
 @mongo:<port>/dev 
 mongodb://<dbuser>:<dbpassword>
 @mongo52101.domain.name:52101/dev Cloud Mongo Database Service WebApp role=webapp replicas=2, 
 role=webapp .name EXTERNAL SERVICE Connecting to Service with dynamic URI with a static ExternalName Kubernetes service
  55. DATABASES

  56. PERSISTENT VOLUMES Host Container Host Container Host Container Data in

    Container Data lost when Container terminates Data lost when Host terminates Independent of Container & Host Data in a Host Volume Networked Volume Data lost when Cloud instance
 
 (local) Data lost when Container terminates
 
 
 (image) Independent of 
 Container & 
 Cloud instance (EBS/network) DATA PERSISTENCE
  57. 1. Maintains a sticky network ID/name across restarts
 e.g. mongo-0,

    mongo-1, mongo-2 2. Ordered Operations with ordinal index 
 e.g. name-0, name-1, name-2 3. Stable, persistent storage (linked to ordinal index/name) 4. Mandatory headless service (no single IP) for integrations KUBERNETES
 STATEFULSETS
  58. role=mongo type=leader Nodes Pods Services Mongo StatefulSet replicas=2 role=mongo Client

    mongo-0 D A B C C DATABASE STATEFUL SETS StatefulSet with 2 replicas , headless service, direct access to pods pvc Read / Write Persistent Volume
  59. DATABASE STATEFUL SETS role=mongo type=leader role=mongo type=follower Nodes Pods Services

    Client Mongo-0 Mongo-1 D A B C C Mongo StatefulSet replicas=2 role=mongo pvc pvc Read / Write Read / Only Persistent Volume
  60. role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0

    Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C C D Mongo StatefulSet replicas=3 role=mongo Read / Write Read / Only Read / Only DATABASE STATEFUL SETS Scale to 3 replicas Client
  61. role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0

    Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C D Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Unresponsive Pod Client
  62. role=mongo type=leader role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 pvc

    pvc Persistent Volume A B D role=mongo type=follower Mongo-2 pvc C Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Auto recovery Client
  63. DATABASE MIGRATIONS

  64. Application v3 Development Application V2 Test Application v1 Production DB

    v1 DB v2 DB v3 CI/CD PIPELINE Version control database updates, ex: flyway V3__add_table_scooter.sql V2__add_table_truck.sql V1__add_table_car.sql
  65. DATABASE MIGRATIONS Version control database updates with Containers CONTAINER IMAGE

    CONTAINER BUILD FILE SQL MIGRATION SCRIPT Source Code Repository V2__add_table.sql Source Code Repository V2__add_table.sql /var/flyway/data Flyway flyway-mydb:v2.0.0 Registry + Dockerfile
  66. Nodes Pods Services postgresql-0 Persistent Volume A B D C

    PostgreSQL StatefulSet replicas=1 role=postgresq pvcl DATABASE MIGRATION StatefulSet deployment with headless Service v1
  67. Nodes Pods Services postgresql-0 Persistent Volume A B D C

    PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Create a Job for Flyway Flyway Job Secrets = Database Connection Info v1 flyway-mydb:v2.0.0 Image Registry Flyway
  68. role=postgressql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B

    D C PostgreSQL StatefulSet replicas=1 role=postgresql pvc DATABASE MIGRATIONS Apply schema changes to database Flyway Job Secrets = Database Connection Info V2 flyway-mydb:v2.0.0 Flyway
  69. role=postgresql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B

    D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Version control for database with Kubernetes V2
  70. MONITORING CONSIDERATIONS Kubernetes* Container* Host Cluster services, services, pods, 


    deployments metrics Container native metrics Traditional resource metrics - cpu, memory, network, storage prometheus + grafana kubernetes-state-metrics probes Stack Metrics Tool node-exporter Kubernetes metrics server: kubelet:cAdvisor Microservices Distributed applications - traditional app metrics - service discovery - distributed tracing prometheus + grafana jaeger tracing istio
  71. WHAT’S NEXT?

  72. KUBERNETES NATIVE ADD-ONS knative github.com/knative istio istio.io Server-
 less Service

    Mesh CI/CD tekton tekton.dev
  73. KNATIVE FOR SERVERLESS

  74. SERVICE MESH

  75. GENERAL DISTRIBUTION ISTIO BENEFITS Istio makes it easy to create

    a network of deployed services with load balancing, service-to-service authentication, monitoring, and more, helping to avoid operational nightmares. POLICY ROUTING TELEMETRY Provides the observability needed to manage microservices, such as how services are invoked, communication flows, and points of latency Allows for the control of routing flows Grants the ability to write policy that applies to all applications and is not language specific
  76. GENERAL DISTRIBUTION INCREASING THE VALUE OF ISTIO VISUALIZATION TRACING Kiali

    includes Jaeger Tracing, which provides distributed tracing out of the box. Kiali works with Istio to visualize the service mesh topology, features like circuit breakers or request rates.
  77. Traffic Control Service Resiliency Chaos Testing Observ- ability Security SERVICE

    MESH WITH ISTIO Dedicated infrastructure layer for making service-to-service communication 
 safe, fast, and reliable Deploy as a lightweight side-car network proxy
  78. CONFIDENTIAL - FOR INTERNAL USE ONLY MICROSERVICES WITHOUT ISTIO Container

    JVM service A discovery load-balancer resiliency metrics tracing app logic JVM service B discovery load-balancer resiliency metrics tracing app logic Container JVM service C discovery load-balancer resiliency metrics tracing app logic
  79. CONFIDENTIAL - FOR INTERNAL USE ONLY MICROSERVICES WITH ISTIO Container

    JVM service C app logic Pod Sidecar Container Envoy Container JVM service A app logic Pod Sidecar Container Envoy Container JVM service B app logic Pod Sidecar Container Envoy
  80. ISTIO SERVICE MESH Envoy istio-ingress Envoy App A Envoy App

    B Envoy App C istio-pilot istio-mixer istio-auth HTTP Req/Resp Kubernetes Pods Istio Components Config to Envoy Access Control and Telemetry
  81. Istio (& Kiali)

  82. 24% 76% v1 v2 apiVersion: config.istio.io/v1alpha2 kind: RouteRule metadata: name:

    recommendation-v1-v2 spec: destination: namespace: tutorial name: recommendation precedence: 5 route: - labels: version: v1 weight: 76 - labels: version: v2 weight: 24 CANARY RELEASE BY WEIGHT RouteRule #2: 
 Route 94% to v1 and 6% to v2
  83. “.*Safari.*” Default v1 v2 apiVersion: config.istio.io/v1alpha2 kind: RouteRule metadata: name:

    recommendation-safari spec: destination: namespace: tutorial name: recommendation precedence: 2 match: request: headers: user-agent: regex: ".*Safari.*" route: - labels: version: v2 ROUTING BY HEADER By Geography, Mobile Device, Browser, Customer, … RouteRule #3: 
 Route “Safari” to v2
  84. v1 v2 apiVersion: config.istio.io/v1alpha2 kind: RouteRule metadata: name: recommendation-mirror spec:

    destination: namespace: tutorial name: recommendation precedence: 2 route: - labels: version: v1 weight: 100 - labels: version: v2 weight: 0 mirror: namespace: tutorial name: recommendation labels: version: v2 DARK LAUNCH Mirror production traffic for pre-release testing RouteRule #3: 
 Route “Safari” to v2 Mirror Production Traffic To v2 100% 100% Test Production
  85. apiVersion: config.istio.io/v1alpha2 kind: EgressRule metadata: name: httpbin-egress-rule spec: destination: service:

    httpbin.org ports: - port: 80 protocol: http SECURE BY DEFAULT Egress blocks all traffic unless unless whitelisted with EgressRule EgressRule: Allow httpbin.org:80 
 (http) role=web Pods Nodes http://httpbin.org Istio EgressRule
  86. THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin