Save 37% off PRO during our Black Friday Sale! »

A DevOps State of Mind: Continuous Security with Kubernetes

A DevOps State of Mind: Continuous Security with Kubernetes

7c6a033dd957d547b49630f626e1a143?s=128

Chris Van Tuin

February 22, 2019
Tweet

Transcript

  1. A DEVOP STATE OF MIND: CONTINUOUS SECURITY 
 WITH KUBERNETES

    Chris Van Tuin Chief Technologist, NA West 
 linkedin: Chris Van Tuin
 email: cvantuin@redhat.com
 twitter: @chrisvantuin
  2. “Only the paranoid survive” - Andy Grove, 1996

  3. I.T. MUST EVOLVE FROM A COST CENTER 
 TO INNOVATION

    CENTER Development Model Application Architecture Deployment & Packaging Application Infrastructur e Storage Waterfall Agile Monolithic N-tier Bare Metal Virtual Servers Data Center Hosted Scale Up Scale Out DevOps MicroServices Containers Hybrid Cloud Storage as a Service
  4. BARE METAL VIRTUAL PRIVATE CLOUD OFF-PREMISE ON-PREMISE PUBLIC CLOUD DATA

    DATA DISTRIBUTED APPLICATIONS
  5. ANY COMBINATION, WHETHER TRADITIONAL OR CONTAINERIZED LEGACY APPS (1,000+) BARE

    METAL PRIVATE CLOUD PUBLIC CLOUD VIRTUAL PRODUCTION DEV/TEST HYBRID CLOUD ENVIRONMENTS
  6. MULTI-TENANCY

  7. DEV QA OPS SECURITY IS AN AFTERTHOUGHT | SECURITY |

    “Patch? The servers are behind the firewall.” - Anonymous (far too many to name), 2005 - …
  8. DEVSECOPS + + End to End Security DEV QA OPS

    Culture Process Technology Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Open Source
  9. DEVSECOPS Continuous Security Improvement Process Optimization Security Automation Dev QA

    Prod Reduce Risks, Lower Costs, Speed Delivery, Speed Reaction
  10. CONTAINERS AT SCALE

  11. LAPTOP Container Application OS dependencies Guest VM LINUX BARE METAL

    Container Application OS dependencies LINUX VIRTUALIZATION Container Application OS dependencies Virtual Machine LINUX PRIVATE CLOUD Container Application OS dependencies Virtual Machine LINUX PUBLIC CLOUD Container Application OS dependencies Virtual Machine LINUX CONTAINERIZED MICROSERVICES
 Build Once, Deploy Anywhere
  12. Image Format Distribution Spec Runtime Spec

  13. BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory


    Speed, Resiliency, Scalability, Security 

  14. Databases Images Automation MANAGING CONTAINERIZED MICROSERVICES
 WITH KUBERNETES A/B Testing

    Migrations External
 Services Deployment Strategies Security What’s Next… CI/CD Scanning ENABLING DEVSECOPS WITH KUBERNETES External 
 Services Databases Migrations Infrastructure Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100%
  15. KUBERNETES AUTOMATION

  16. Web Application ORCHESTRATION Pods Nodes Controller Manager & Data Store

    (etcd) Ingress / Routes role: web role: app role: web replicas: 1, 
 role: app replicas: 2, 
 role: web Services
  17. Pods Nodes Services Web Application role: web role: app role:

    web replicas: 1, 
 role: app replicas: 2, 
 role: web role: web Controller Manager & Data Store (etcd) HEALTH CHECK Ingress / Routes Health Check
  18. Web Application 80% CPU Pods Nodes Services role: web role:

    app role: web Controller Manager & Data Store (etcd) role: app AUTO-SCALE Ingress / Routes replicas: 2 
 role: app replicas: 2, 
 role: web Readiness Probe e.g. tcp, http, script
  19. CONTAINER IMAGES

  20. docker.io Registry Private Registry FROM fedora:1.0 CMD echo “Hello” Build

    file Physical, Virtual, Cloud Container Image Container Instance Build Run Ship CONTAINERS ENABLE DEVOPS
  21. CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS

    dependencies 1.2/latest 1.1
  22. Config Data Kubernetes configmaps secrets Container image Traditional 
 data

    services, Kubernetes 
 persistent volumes TREAT CONTAINERS AS IMMUTABLE To keep containerized apps portable Application Language runtimes OS dependencies
  23. KUBERNETES CONFIGMAP Decouple configuration from container image Application Language runtimes

    OS dependencies Environment Variable or Volume/File CONTAINER INSTANCE key:value from directories, files, or values KUBERNETES
 CONFIGMAP APPLICATION CONFIG FILE Application Configuration File e.g. XML etcd Pod Source Code Repository EnvVar require pod restart Files refresh in time
  24. CONTINUOUS BUILDS

  25. A CONVERGED SOFTWARE 
 SUPPLY CHAIN

  26. CUSTOM SUPPLY CHAIN CASCADING REBUILDS

  27. Java Build Environment Language runtimes OS dependencies Build Image Java

    Code Application Language runtimes OS dependencies Container Image Image Registry Source Repository Image Registry REPRODUCIBLE BUILDS Source to Image with Build Images Source v3.1 v1.0.1 v3.1
  28. CI/CD PIPELINE WITH KUBERNETES BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC

    CLOUD
  29. CONTAINER SCANNING

  30. WHAT’S INSIDE MATTERS…

  31. PRIVATE REGISTRY

  32. Security CONTINUOUS INTEGRATION WITH SECURITY SCAN

  33. AUTOMATED SECURITY SCANNING with OpenSCAP Reports & Remediation Scan SCAP

    Security Guide for RHEL CCE-27002-5 Set Password Minimum Length Content Scan physical servers, virtual machines, docker images and containers
 for Security Policy Compliance (CCEs) and known Security Vulnerabilities (CVEs)
  34. DEPLOYMENT STRATEGIES

  35. CONTINUOUS DELIVERY WITH CONTAINERS CI/CD - CONTAINER UPDATES

  36. CI/CD DEPLOYMENT STRATEGIES
 Automate and reduce deployment risk DEPLOYMENT STRATEGIES

    • Recreate • Rolling updates • Blue / Green deployment • Canary with A/B testing
  37. Recreate

  38. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI RECREATE WITH DOWNTIME RECREATE WITH DOWNTIME
 Using Recreate deployment strategy Kubernetes
 Service
  39. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI RECREATE WITH DOWNTIME RECREATE WITH DOWNTIME
 Shutdown existing deployment Kubernetes
 Service
  40. Version 1.2 Version 1.2 Version 1.2 RECREATE WITH DOWNTIME Use

    Case • Non-mission critical services Pros • Simple, clean • No Schema incompatibilities • No API versioning Cons • Downtime RECREATE WITH DOWNTIME
 Shutdown existing deployment Kubernetes
 Service
  41. Rolling Updates

  42. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI ROLLING UPDATES with ZERO DOWNTIME Rollingupdate
 maxUnavailable=0 maxSurge=1 ROLLING UPDATES
 Replace each pod using RollingUpdate deployment strategy Kubernetes
 Service
  43. Deploy new version and wait until it’s ready… Health Check:

    readiness probe e.g. tcp, http, script Version 1 Version 1 Version 
 1.2 Version 1 Rollingupdate
 maxUnavailable=0 maxSurge=1 ROLLING UPDATES
 Deploy new version, wait until it’s ready Kubernetes
 Service
  44. Each container/pod is updated one by one Version 1.2 50%

    Version 1 V1 V1.2 ROLLING UPDATES
 Requires backward compatibility, as two versions run side-by-side Kubernetes
 Service
  45. Each container/pod is updated one by one Version 1.2 Version

    1.2 Version 1.2 100% Use Case • Horizontally scaled • Backward compatible API/data • Microservices Pros • Zero downtime • Reduced risk, gradual rollout w/health checks • Ready for rollback Cons • Require backward compatible APIs/data • Resource overhead ROLLING UPDATES Kubernetes
 Service
  46. Blue / Green Deployment

  47. BLUE Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT

    Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Single service, run two complete Deployments BLUE Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% Service
 selector:
 production=BLUE Kubernetes
 Deployment
  48. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100% Health Check: readiness probe e.g. tcp, http, script BLUE / GREEN DEPLOYMENT
 Using Deployments, Ingress Service
 selector:
 production=BLUE Kubernetes
 Deployment Kubernetes
 Deployment
  49. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Route all new request to Green, Blue sessions Service
 selector:
 version=GREEN
  50. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Using Deployments, Ingress Service
 selector:
 production=GREEN
  51. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Scale-down, reduce resources Service
 selector:
 production=GREEN
  52. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Hot Backup Service
 selector:
 production=GREEN Version 2
  53. BLUE / GREEN DEPLOYMENT Rollback BLUE GREEN Version 1 Version

    2 Ingress Use Case • Self-contained micro services (data) Pros • Low risk, never change production • No downtime • Production like testing • Rollback Cons • Resource overhead • Data synchronization BLUE / GREEN DEPLOYMENT
 Rollback Service
 selector:
 production=BLUE
  54. RAPID INNOVATION & EXPERIMENTATION WITH A/B TESTING

  55. ”only about 1/3 of ideas improve the metrics 
 they

    were designed to improve.”
 Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION
  56. A/B TESTING USING CANARY DEPLOYMENTS

  57. 25% Conversion Rate ?! Conversion Rate 100% Version B Version

    A Ingress CANARY DEPLOYMENTS Tests / CI CANARY DEPLOYMENTS
 Build confidence in new version Service
 selector:
 app=demo version=A label:
 app=demo
 version=A 25% Conversion Rate ??% Conversion Rate
  58. 25% Conversion Rate 30% Conversion Rate 75% 25% Version B

    Version A Ingress CANARY DEPLOYMENTS CANARY DEPLOYMENTS
 Requires app to support side-by-side version Service Service
 selector:
 app=demo label:
 app=demo
 version=A 25% Conversion Rate % Conversion Rate label:
 app=demo
 version=B
  59. 25% Conversion Rate 30% Conversion Rate 100% Version B Version

    A Ingress CANARY DEPLOYMENTS Service
 selector:
 app=demo version=B label:
 app=demo
 version=A 25% Conversion Rate 30% Conversion Rate label:
 app=demo
 version=B
  60. Databases Images Automation MANAGING CONTAINERIZED MICROSERVICES
 WITH KUBERNETES A/B Testing

    Migrations External
 Services Deployment Strategies Security What’s Next… CI/CD Scanning ENABLING DEVSECOPS WITH KUBERNETES External 
 Services Databases Migrations Infrastructure Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100%
  61. EXTERNAL SERVICES

  62. EXTERNAL SERVICES Database outside cluster with IP address External Mongo

    Database Service External Mongo Database Service Development Production IP=10.200.0.2 port=27017 IP=10.100.0.9 port=27017
  63. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017
  64. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.200.0.2 port=27017 Database kind=Service type=ClusterIP name=mongo port=27017 targetport=27017
  65. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.200.0.2 port=27017 Connect with mongodb://mongo Database kind=Service type=ClusterIP name=mongo port=27017 targetport=27017 kind=Endpoints name=mongo ip=10.200.0.2 port=27017
  66. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.100.0.9 port=27017 kind=Service type=ClusterIP name=mongo port=27017 targetport=27017 kind=Endpoints name=mongo ip=10.200.0.9 port=27017 Connect with mongodb://mongo Database
  67. Pods Nodes Services Database name: mongo type: ExternalName externalName: mongo52101.domain,.name

    EXTERNAL SERVICES Using CNAME redirection mongodb://
 <dbuser>:
 <dbpassword>
 @mongo:<port>/dev 
 mongodb://<dbuser>:<dbpassword>
 @mongo52101.domain.name:52101/dev Cloud Mongo Database Service WebApp role=webapp replicas=2, 
 role=webapp .name EXTERNAL SERVICE Connecting to Service with dynamic URI with a static ExternalName Kubernetes service
  68. DATABASES

  69. PERSISTENT VOLUMES Host Container Host Container Host Container Data in

    Container Data lost when Container terminates Data lost when Host terminates Independent of Container & Host Data in a Host Volume Networked Volume Data lost when Cloud instance terminates Data lost when Container terminates Independent of 
 Container & 
 Cloud instance DATA PERSISTENCE
  70. 1. Maintains a sticky network ID/name across restarts
 e.g. mongo-0,

    mongo-1, mongo-2 2. Ordered Operations with ordinal index 
 e.g. name-0, name-1, name-2 3. Stable, persistent storage (linked to ordinal index/name) 4. Mandatory headless service (no single IP) for integrations KUBERNETES
 STATEFULSETS
  71. role=mongo type=leader Nodes Pods Services Mongo StatefulSet replicas=2 role=mongo Client

    mongo-0 D A B C C DATABASE STATEFUL SETS StatefulSet with 2 replicas , headless service, direct access to pods pvc Read / Write Persistent Volume
  72. DATABASE STATEFUL SETS role=mongo type=leader role=mongo type=follower Nodes Pods Services

    Client Mongo-0 Mongo-1 D A B C C Mongo StatefulSet replicas=2 role=mongo pvc pvc Read / Write Read / Only Persistent Volume
  73. role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0

    Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C C D Mongo StatefulSet replicas=3 role=mongo Read / Write Read / Only Read / Only DATABASE STATEFUL SETS Scale to 3 replicas Client
  74. role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0

    Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C D Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Unresponsive Pod Client
  75. role=mongo type=leader role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 pvc

    pvc Persistent Volume A B D role=mongo type=follower Mongo-2 pvc C Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Auto recovery Client
  76. DATABASE MIGRATIONS

  77. Application v3 Development Application V2 Test Application v1 Production DB

    v1 DB v2 DB v3 CI/CD PIPELINE Version control database updates, ex: flyway V3__add_table_scooter.sql V2__add_table_truck.sql V1__add_table_car.sql
  78. DATABASE MIGRATIONS Version control database updates with Containers CONTAINER IMAGE

    CONTAINER BUILD FILE SQL MIGRATION SCRIPT Source Code Repository V2__add_table.sql Source Code Repository V2__add_table.sql /var/flyway/data Flyway flyway-mydb:v2.0.0 Registry + Dockerfile
  79. Nodes Pods Services postgresql-0 Persistent Volume A B D C

    PostgreSQL StatefulSet replicas=1 role=postgresq pvcl DATABASE MIGRATION StatefulSet deployment with headless Service v1
  80. Nodes Pods Services postgresql-0 Persistent Volume A B D C

    PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Create a Job for Flyway Flyway Job Secrets = Database Connection Info v1 flyway-mydb:v2.0.0 Image Registry Flyway
  81. role=postgressql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B

    D C PostgreSQL StatefulSet replicas=1 role=postgresql pvc DATABASE MIGRATIONS Apply schema changes to database Flyway Job Secrets = Database Connection Info V2 flyway-mydb:v2.0.0 Flyway
  82. role=postgresql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B

    D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Version control for database with Kubernetes V2
  83. INFRASTRUCTURE

  84. MONITORING CONSIDERATIONS Kubernetes* Container* Host Cluster services, services, pods, 


    deployments metrics Container native metrics Traditional resource metrics - cpu, memory, network, storage prometheus + grafana kubernetes-state-metrics probes Stack Metrics Tool node-exporter kubelet:cAdvisor Microservices Distributed applications - traditional app metrics - service discovery - distributed tracing prometheus + grafana jaeger tracing istio
  85. Network Namespace 
 provides resource isolation NETWORK ISOLATION Multi-Environment Multi-Tenant

  86. NETWORK POLICY example: 
 all pods in namespace ‘project-a’ allow

    traffic 
 from any other pods in the same namespace.”
  87. Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers

    Unit File Docker Image Container CLI SYSTEMD Cgroups Namespaces SELinux Drivers CONTAINERS ARE LINUX seccomp Read Only mounts
  88. CGROUPS - RESOURCE ISOLATION

  89. NAMESPACES - PROCESS ISOLATION

  90. SELINUX - MANDATORY ACCESS CONTROLS Password Files Web Server Attacker

    Discretionary Access Controls 
 (file permissions) Mandatory Access Controls 
 (selinux) Internal Network Firewall Rules Password Files Firewall Rules Internal Network Web Server selinux policy
  91. SECCOMP - DROPPING PRIVILEGES Tooling: strace, go2seccomp, … 300+ system

    calls, create whitelist…
  92. READ ONLY MOUNTS

  93. Best Practices • Don’t run as root • Limit SSH

    Access • Use namespaces • Define resource quotas • Enable logging • Apply Security Errata • Apply Security Context and seccomp filters • Most namespace 
 sysctls are not safe http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html CONTAINER HOST SECURITY Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container CLI SYSTEMD Cgroups Namespaces SELinux Drivers seccomp Read Only mounts
  94. WHAT’S NEXT

  95. EXTENDING KUBERNETES kubevirt github.com/kubevirt operators coreos.com/operators knative github.com/knative istio istio.io

    Virtual Machines Day 2 Operations Server-
 less Service Mesh
  96. Deployment Frequency Lead Time Deployment
 Failure Rate Mean Time to

    Recover 99.999 Service Availability DEVSECOPS METRICS Compliance Score
  97. THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin