Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DeveloperWeek Austin 2018: A DevOps State of Mind: Continuous Security with Kubernetes

DeveloperWeek Austin 2018: A DevOps State of Mind: Continuous Security with Kubernetes

7c6a033dd957d547b49630f626e1a143?s=128

Chris Van Tuin

November 07, 2018
Tweet

Transcript

  1. A DevOps State of Mind: Continuous Security with Kubernetes Chris

    Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.com @chrisvantuin
  2. “Only the paranoid survive” - Andy Grove, 1996

  3. ENABLING INNOVATION, WHILE EXECUTING AT SCALE Static &
 Planned Dynamic

    & 
 Policy Driven Execution Innovation Innovation Execution Old New
  4. THE WORLD IS AUTOMATING Those who succeed in automation will

    win
  5. I.T. MUST EVOLVE FROM A COST CENTER 
 TO INNOVATION

    CENTER Development Model Application Architecture Deployment & Packaging Application Infrastructur e Storage Waterfall Agile Monolithic N-tier Bare Metal Virtual Servers Data Center Hosted Scale Up Scale Out DevOps MicroServices Containers Hybrid Cloud Storage as a Service
  6. Applications & devices outside of IT control Cloud computing Software-defined

    infrastructure Dissolving security perimeter Menacing threat landscape TRADITIONAL NETWORK-BASED DEFENSES ARE NO LONGER ENOUGH SECURING THE ENTERPRISE IS HARDER THAN EVER The way we develop, deploy and manage IT is changing dramatically led by DevOps, Cloud Native Applications, and Hybrid Cloud
  7. DEVSECOPS Continuous Security Improvement Process Optimization Security Automation Dev QA

    Prod Reduce Risks, Lower Costs, Speed Delivery, Speed Reaction
  8. DEVSECOPS + + End to End Security DEV QA OPS

    Culture Process Technology Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Open Source
  9. CONTAINERS ENABLE DEVSECOPS

  10. CONTAINERS Software packaging concept that typically includes an application and

    all of its runtime dependencies • HIGHER quality software releases • SHORTER test cycles • EASIER application management CONTAINER CONTAINER APP LIBS HOST OS SERVER APP LIBS BENEFITS
  11. LAPTOP Container Application OS dependencies Guest VM LINUX BARE METAL

    Container Application OS dependencies LINUX VIRTUALIZATION Container Application OS dependencies Virtual Machine LINUX PRIVATE CLOUD Container Application OS dependencies Virtual Machine LINUX PUBLIC CLOUD Container Application OS dependencies Virtual Machine LINUX CONTAINERS - Build Once, Deploy Anywhere 
 Reducing Risk and Improving Security with Improved Consistency
  12. Image Format Distribution Spec Runtime Spec

  13. Scheduling Monitoring Persistence Discovery Lifecycle & health Scaling Aggregation Security

    MORE THAN CONTAINERS…
  14. BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Speed, Resiliency, Scalability,

    Security 

  15. AUTOMATION

  16. Web Application replicas: 1, 
 role: app image: myapp:1.0 replicas:

    2, 
 role: web image: httpd:1.7.9 ORCHESTRATION Declarative, Deployment Controller Manager & Data Store (etcd)
  17. Web Application ORCHESTRATION Declarative, Deployment Nodes Controller Manager & Data

    Store (etcd) Physical, VM, 
 Cloud Instances replicas: 2, 
 role: web image: httpd:1.7.9 replicas: 1, 
 role: app image: myapp:1.0
  18. role: app role: web role: web Pods Nodes Image Registry

    ORCHESTRATION Schedule + Provision Pods (Compute/Storage/Network) Web Application replicas: 2, 
 role: web image: httpd:1.7.9 replicas: 1, 
 role: app image: myapp:1.0
  19. Web Application role: web role: app role: web replicas: 1,

    
 role: app replicas: 2, 
 role: web ORCHESTRATION Services (Load Balancer), Service discovery with selectors and pod labels Pods Nodes Services Controller Manager & Data Store (etcd)
  20. Web Application ORCHESTRATION Service (Load Balancer) Pods Nodes Controller Manager

    & Data Store (etcd) Ingress / Routes role: web role: app role: web replicas: 1, 
 role: app replicas: 2, 
 role: web Services
  21. HEALTH CHECK Monitoring & Logging Pods Nodes Services Web Application

    role: web role: app role: web Ingress / Routes Health Check replicas: 1, 
 role: app replicas: 2, 
 role: web
  22. Pods Nodes Services Web Application role: web role: app role:

    web replicas: 1, 
 role: app replicas: 2, 
 role: web role: web Controller Manager & Data Store (etcd) HEALTH CHECK Readiness Probe e.g. tcp, http, script Ingress / Routes
  23. Web Application replicas: 1, 
 role: app replicas: 2, 


    role: web Pods Nodes Services role: web role: app role: web Controller Manager & Data Store (etcd) HEALTH CHECK Ingress / Routes
  24. Web Application AUTO-SCALE Monitoring & Logging 80% CPU Pods Nodes

    Services role: web role: app role: web Ingress / Routes replicas: 1, 
 role: app replicas: 2, 
 role: web
  25. Web Application 80% CPU Pods Nodes Services role: web role:

    app role: web Controller Manager & Data Store (etcd) role: app AUTO-SCALE Ingress / Routes replicas: 2 
 role: app replicas: 2, 
 role: web
  26. Pods Nodes Services Web Application 50% CPU role: web role:

    app role: app role: web Controller Manager & Data Store (etcd) AUTO-SCALE Ingress / Routes replicas: 2, 
 role: web replicas: 2, 
 role: app
  27. Network isolation API & Platform access Federated clusters Storage {}

    CI/CD Monitoring & Logging Builds Images SECURING YOUR CONTAINER ENVIRONMENT Container host Registry
  28. CONTAINER IMAGES

  29. CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS

    dependencies 1.2/latest 1.1 CONTAINER IMAGE
  30. Config Data Kubernetes configmaps secrets Container image Traditional 
 data

    services, Kubernetes 
 persistent volumes TREAT CONTAINERS AS IMMUTABLE To keep containerized apps portable Application Language runtimes OS dependencies
  31. KUBERNETES CONFIGMAP Decouple configuration from container image Application Language runtimes

    OS dependencies Environment Variable or Volume/File CONTAINER INSTANCE key:value from directories, files, or values KUBERNETES
 CONFIGMAP APPLICATION CONFIG FILE Application Configuration File e.g. XML etcd Pod Source Code Repository EnvVar require pod restart Files refresh in time
  32. CONTAINER BUILDS

  33. docker.io Registry Private Registry Build file Physical, Virtual, Cloud Image

    Instance Build Run Ship FROM registry.redhat.com/rhel7 RUN groupadd -g 999 appuser && \ useradd -r -u 999 -g appuser appuser USER appuser CMD echo “Hello” CONTAINER BUILDS
  34. • Treat build file as a Blueprint • Version control

    build file • Don’t login to build/configure • Be explicit with versions, not latest • Always list registry pulling FROM • Specify USER, default is root • Each Run creates a new layer BUILD FILE BEST PRACTICES FROM registry.redhat.com/rhel7 RUN groupadd -g 999 appuser && \ useradd -r -u 999 -g appuser appuser USER appuser CMD echo “Hello” Build file
  35. A CONVERGED SOFTWARE 
 SUPPLY CHAIN

  36. CONTAINER REGISTRY

  37. 64% of official images in Docker Hub 
 contain high

    priority security vulnerabilities examples: ShellShock (bash) Heartbleed (OpenSSL) Poodle (OpenSSL) Source: Over 30% of Official Images in Docker Hub Contain High Priority Security Vulnerabilities, Jayanth Gummaraju, Tarun Desikan, and Yoshio Turner, BanyanOps, May 2015 (http://www.banyanops.com/pdf/BanyanOps-AnalyzingDockerHub-WhitePaper.pdf) WHAT’S INSIDE THE CONTAINER MATTERS
  38. PRIVATE REGISTRY

  39. •Authenticating authorship •Non-repudiation •Ensuring image integrity CONTAINER IMAGE SIGNING Validate

    what images and version are running
  40. CONTAINER HOST SECURITY

  41. Chris Van Tuin Chief Technologist, NA West / Silicon Valley

    cvantuin@redhat.co Best Practices • Don’t run as root • If you must, 
 limit Linux Capabilities • Limit SSH Access • Use namespaces • Define resource quotas • Enable logging • Apply Security Errata • Apply Security Context and seccomp filters • Run production 
 unprivileged containers 
 as read-only http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container CLI SYSTEMD Cgroups Namespaces SELinux Drivers seccomp Read Only mounts Capabilities CONTAINER HOST SECURITY CONTAINER HOST SECURITY
  42. CONTINUOUS INTEGRATION

  43. WHAT’S INSIDE MATTERS…

  44. CI/CD PIPELINE Continuous Integration Continuous Build Continuous Deployment Developer ->

    Source -> Git Git -> RPMS -> Images-> Registry Images from 
 Registry -> Clusters
  45. Security CONTINUOUS INTEGRATION WITH SECURITY SCAN

  46. Java Build Environment Language runtimes OS dependencies Build Image Java

    Code Application Language runtimes OS dependencies Container Image Image Registry Source Repository Image Registry REPRODUCIBLE BUILDS Source to Image with Build Images Source v3.1 v1.0.1 v3.1
  47. CONTINUOUS DELIVERY

  48. CONTINUOUS DELIVERY WITH CONTAINERS

  49. CUSTOM SUPPLY CHAIN CUSTOM SUPPLY CHAIN

  50. CONTINUOUS DELIVERY: DEPLOYMENT STRATEGIES

  51. CONTINUOUS DELIVERY 
 DEPLOYMENT STRATEGIES DEPLOYMENT STRATEGIES • Recreate •

    Rolling updates • Blue / Green deployment • Canary with A/B testing • Database migrations
  52. RECREATE

  53. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI RECREATE WITH DOWNTIME
  54. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI RECREATE WITH DOWNTIME
  55. Version 1.2 Version 1.2 Version 1.2 RECREATE WITH DOWNTIME Use

    Case • Non-mission critical services Pros • Simple, clean • No Schema incompatibilities • No API versioning Cons • Downtime
  56. ROLLING UPDATES

  57. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI ROLLING UPDATES with ZERO DOWNTIME Rollingupdate
 maxUnavailable=0 maxSurge=1
  58. Deploy new version and wait until it’s ready… Health Check:

    readiness probe e.g. tcp, http, script Version 1 Version 1 Version 
 1.2 Version 1 Rollingupdate
 maxUnavailable=0 maxSurge=1
  59. Each container/pod is updated one by one Version 1.2 50%

    Version 1 V1 V1.2
  60. Each container/pod is updated one by one Version 1.2 Version

    1.2 Version 1.2 100% Use Case • Horizontally scaled • Backward compatible API/data • Microservices Pros • Zero downtime • Reduced risk, gradual rollout w/health checks • Ready for rollback Cons • Require backward compatible APIs/data • Resource overhead
  61. BLUE / GREEN

  62. BLUE Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT

    Using Ingress 100%
  63. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100% Health Check: readiness probe e.g. tcp, http, script
  64. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100%
  65. BLUE / GREEN DEPLOYMENT Rollback BLUE GREEN Version 1 Version

    2 Ingress Use Case • Self-contained micro services (data) Pros • Low risk, never change production • No downtime • Production like testing • Rollback Cons • Resource overhead • Data synchronization
  66. CANARY WITH A/B TESTING

  67. ”only about 1/3 of ideas improve the metrics 
 they

    were designed to improve.”
 Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION
  68. A/B TESTING USING CANARY DEPLOYMENTS

  69. 25% Conversion Rate ?! Conversion Rate 100% Version B Version

    A Ingress CANARY DEPLOYMENTS Tests / CI
  70. 25% Conversion Rate 30% Conversion Rate 75% 25% Version B

    Version A Ingress CANARY DEPLOYMENTS
  71. 25% Conversion Rate 30% Conversion Rate 100% Version B Version

    A Ingress CANARY DEPLOYMENTS
  72. 25% Conversion Rate 20% Conversion Rate 100% Version B Version

    A Rollback Ingress CANARY DEPLOYMENTS
  73. DATABASE MIGRATIONS

  74. DATABASE MIGRATIONS Version control database updates with Containers CONTAINER IMAGE

    CONTAINER BUILD FILE SQL MIGRATION SCRIPT Source Code Repository V2__add_table.sql Source Code Repository V2__add_table.sql /var/flyway/data Flyway flyway-mydb:v2.0.0 Registry + Dockerfile
  75. Application v3 Development Application V2 Test Application v1 Production DB

    v1 DB v2 DB v3 CI/CD PIPELINE Version control database updates, ex: flyway V3__add_table_scooter.sql V2__add_table_truck.sql V1__add_table_car.sql
  76. Nodes Pods Services postgresql-0 Persistent Volume A B D C

    PostgreSQL StatefulSet replicas=1 role=postgresq pvcl DATABASE MIGRATION StatefulSet deployment with headless Service v1
  77. Nodes Pods Services postgresql-0 Persistent Volume A B D C

    PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Create a Job for Flyway Flyway Job Secrets = Database Connection Info v1 flyway-mydb:v2.0.0 Image Registry Flyway
  78. role=postgressql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B

    D C PostgreSQL StatefulSet replicas=1 role=postgresql pvc DATABASE MIGRATIONS Apply schema changes to database Flyway Job Secrets = Database Connection Info V2 flyway-mydb:v2.0.0 Flyway
  79. role=postgresql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B

    D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Version control for database with Kubernetes V2
  80. Network isolation API & Platform access Federated clusters Storage {}

    CI/CD Monitoring & Logging Images Builds Container host Registry SECURING YOUR CONTAINER ENVIRONMENT
  81. NETWORK ISOLATION

  82. Network Namespace 
 provides resource isolation NETWORK ISOLATION Multi-Environment Multi-Tenant

  83. NETWORK POLICY example: 
 all pods in namespace ‘project-a’ allow

    traffic 
 from any other pods in the same namespace.”
  84. NETWORK SECURITY MODELS Co-Existence Approaches One Cluster Multiple Zones Kubernete

    Cluster Physical Compute 
 isolation based on 
 Network Zones Kubernete Cluster One Cluster Per Zone Kubernete Cluster B Kubernete Cluster A Kubernetes Cluster B C D https://blog.openshift.com/openshift-and-network-security-zones-coexistence-approaches/
  85. MONITORING & LOGGING

  86. CONTINUOUS FEEDBACK LOOP

  87. KUBERNETES MONITORING CONSIDERATIONS Kubernetes* Container* Host Cluster services, services, pods,

    
 deployments metrics Container native metrics Traditional resource metrics - cpu, memory, network, storage prometheus + grafana kubernetes-state-metrics probes Stack Metrics Tool node-exporter kubelet:cAdvisor Application Distributed applications - traditional app metrics - service discovery - distributed tracing prometheus + grafana jaeger tracing istio
  88. Aggregate platform and application log access via Kibana + Elasticsearch

    LOGGING
  89. STORAGE SECURITY

  90. Local Storage Quota Security Context 
 Constraints STORAGE SECURITY Data

    Encryption 
 at DataStore layer
  91. API & PLATFORM ACCESS

  92. Authentication via OAuth tokens and SSL certificate Authorization via Policy

    Engine checks User/Group Defined Roles API & PLATFORM ACCESS
  93. FEDERATION

  94. Amazon East OpenStack FEDERATED CLUSTERS Roles & access management (in-dev)

  95. MICROSERVICES

  96. Traffic Control Service Resiliency Chaos Testing Observ- ability Security ISTIO

    SERVICE MESH
  97. Your app... automated like the cloud... but runs on... OPERATORS

    FOR KUBERNETES
  98. OPERATOR MATURITY MODEL

  99. Deployment Frequency Lead Time Deployment
 Failure Rate Mean Time to

    Recover 99.999 Service Availability DEVSECOPS METRICS Compliance Score
  100. THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin