Save 37% off PRO during our Black Friday Sale! »

DevOpsDays Boise 2019: Continuous Security with Kubernetes

7c6a033dd957d547b49630f626e1a143?s=47 Chris Van Tuin
May 30, 2019
7

DevOpsDays Boise 2019: Continuous Security with Kubernetes

7c6a033dd957d547b49630f626e1a143?s=128

Chris Van Tuin

May 30, 2019
Tweet

Transcript

  1. A DEVOPS STATE OF MIND: CONTINUOUS SECURITY WITH KUBERNETES Chris

    Van Tuin
 Chief Technologist, NA West @chrisvantuin cvantuin@redhat.com
  2. None
  3. ENABLING INNOVATION, WHILE EXECUTING AT SCALE Static &
 Planned Dynamic

    & 
 Policy Driven Execution Innovation Innovation Execution Old New
  4. I.T. MUST EVOLVE & KEEP UP

  5. DEVSECOPS

  6. DEV QA OPS SECURITY IS AN AFTERTHOUGHT | SECURITY |

    “Patch? The servers are behind the firewall.” - Anonymous (far too many to name), 2005 - … | Security | SECURITY IS AN AFTERTHOUGHT
  7. DEVSECOPS + + End to End Security DEV QA OPS

    Culture Process Technology Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Open Source
  8. DEVSECOPS Continuous Security Improvement Process Optimization Security Automation Dev QA

    Prod Reduce Risks, Lower Costs, Speed Delivery, Speed Reaction
  9. CONTAINERS ENABLE DEVSECOPS

  10. https://medium.com/@bibryam/cloud-native-container-design-principles-144ceaa98dba CLOUD NATIVE CONTAINER DESIGN PRINCIPLES: Build Time Does one

    thing well Built with depenedencies, depends on Linux kernel Build once, deploy anywhere Single Concern Principle Self Containment Principle Image Imutability Principle
  11. Process Dispensability Principle CLOUD NATIVE CONTAINER DESIGN PRINCIPLES: Runtime Ephemeral

    - short lived, replaceable Read & react to events APIs to obeserver & manage Resource requirements defined and restricted High Observability Principle Lifecycle Conformance Principle Runtime Confinement Principle
  12. SECURING CONTAINERS BEST PRACTICES Reduce Attack Surface Least Privilege Layered

    Security $ sudo Immutable
  13. CONTAINERS AT SCALE

  14. BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory


    Speed, Resiliency, Scalability, Security 
 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory
 Speed, Resiliency, Scalability, Security 
 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory
 Speed, Resiliency, Scalability, Security 
 Speed, Agility, Resiliency, Scalability, Efficiency, Security
  15. CONTAINER SECURITY AT SCALE

  16. ENABLING DEVSECOPS WITH KUBERNETES Databases Images Automation MANAGING CONTAINERIZED MICROSERVICES


    WITH KUBERNETES A/B Testing Migrations External
 Services Deployment Strategies Security What’s Next… CI/CD Scanning External 
 Services Databases Migrations Infrastructure Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100%
  17. KUBERNETES AUTOMATION

  18. Web Application ORCHESTRATION Speed, Agility Pods Nodes Controller Manager &

    Data Store (etcd) Ingress / Routes role: web role: app role: web replicas: 1, 
 role: app replicas: 2, 
 role: web Services
  19. Pods Nodes Services Web Application role: web role: app role:

    web replicas: 1, 
 role: app replicas: 2, 
 role: web role: web Controller Manager & Data Store (etcd) Ingress / Routes Health Check HEALTH CHECK Resiliency
  20. Web Application 80% CPU Pods Nodes Services role: web role:

    app role: web Controller Manager & Data Store (etcd) role: app Ingress / Routes replicas: 2 
 role: app replicas: 2, 
 role: web Readiness Probe e.g. tcp, http, script AUTO SCALE Scalability & Efficiency
  21. CONTAINER IMAGES

  22. Image Format Distribution Spec Runtime Spec

  23. docker.io Registry Private Registry FROM fedora:1.0 CMD echo “Hello” Build

    file Physical, Virtual, Cloud Container Image Container Instance Build Run Ship CONTAINERS ENABLE DEVOPS CONTAINERS ENABLE DEVSECOPS
  24. CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS

    dependencies 1.2/latest 1.1
  25. Config Data Kubernetes configmaps secrets Container image Traditional 
 data

    services, Kubernetes 
 persistent volumes TREAT CONTAINERS AS IMMUTABLE To keep containerized apps portable Application Language runtimes OS dependencies
  26. KUBERNETES CONFIGMAP Decouple configuration from container image Application Language runtimes

    OS dependencies Environment Variable or Volume/File CONTAINER INSTANCE key:value from directories, files, or values KUBERNETES
 CONFIGMAP APPLICATION CONFIG FILE Application Configuration File e.g. XML etcd Pod Source Code Repository EnvVar require pod restart Files refresh in time
  27. NODE MASTER Container Distributed Store Container KUBERNETES SECRETS " Secure

    mechanism for holding sensitive data e.g. ◦ Passwords and credentials ◦ SSH Keys ◦ Certificates " Secrets are made available as ◦ Environment variables ◦ Volume mounts ◦ Interaction with external systems " Encrypted in transit and
 support for encryption at rest " Never rest on the nodes, stored in memory (tmpfs)
  28. CONTAINER IMAGE SIGNING Validate what images and version are running

    • Authenticating authorship • Non-repudiation • Ensuring image integrity
  29. • Treat build file as a Blueprint • Don’t login

    to build/configure • Version control build file • Be explicit with versions, not latest • Always list registry pulling FROM • Each Run creates a new layer • Specify USER, default is root BUILD FILE BEST PRACTICES FROM registry.redhat.com/rhel7 RUN groupadd -g 999 appuser && \ useradd -r -u 999 -g appuser appuser USER appuser CMD echo “Hello” Build file
  30. CONTINUOUS BUILDS

  31. CI/CD PIPELINE WITH KUBERNETES BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC

    CLOUD CI/CD PIPELINE WITH KUBERNETES BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory
 Speed, Resiliency, Scalability, Security 

  32. A CONVERGED SOFTWARE 
 SUPPLY CHAIN

  33. CUSTOM SUPPLY CHAIN CASCADING REBUILDS

  34. Java Build Environment Language runtimes OS dependencies Build Image Java

    Code Application Language runtimes OS dependencies Container Image Image Registry Source Repository Image Registry REPRODUCIBLE BUILDS Source to Image with Build Images Source v3.1 v1.0.1 v3.1 Java Build Environment Language runtimes OS dependencies Build Image Java Code Application Language runtimes OS dependencies Container Image Image Registry Source Repository Image Registry REPRODUCIBLE BUILDS Source to Image with Build Images Source v3.1 v1.0.1 v3.1 + REPRODUCIBLE BUILDS with build images
  35. CONTAINER REGISTRY & SCANNING

  36. WHAT’S INSIDE MATTERS…

  37. PRIVATE REGISTRY PRIVATE REGISTRY Trusted Content

  38. Security CONTINUOUS INTEGRATION WITH SECURITY SCAN

  39. AUTOMATED SECURITY SCANNING with OpenSCAP Reports & Remediation Scan SCAP

    Security Guide for RHEL CCE-27002-5 Set Password Minimum Length Content Scan physical servers, virtual machines, docker images and containers
 for Security Policy Compliance (CCEs) and known Security Vulnerabilities (CVEs)
  40. DEPLOYMENT STRATEGIES

  41. CONTINUOUS DELIVERY WITH CONTAINERS CI/CD - CONTAINER UPDATES

  42. CI/CD DEPLOYMENT STRATEGIES
 Automate and reduce deployment risk DEPLOYMENT STRATEGIES

    • Recreate • Rolling updates • Blue / Green deployment • Canary with A/B testing
  43. RECREATE

  44. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI RECREATE WITH DOWNTIME RECREATE WITH DOWNTIME
 Using Recreate deployment strategy Kubernetes
 Service
  45. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI RECREATE WITH DOWNTIME RECREATE WITH DOWNTIME
 Shutdown existing deployment Kubernetes
 Service
  46. Version 1.2 Version 1.2 Version 1.2 RECREATE WITH DOWNTIME Use

    Case • Non-mission critical services Pros • Simple, clean • No Schema incompatibilities • No API versioning Cons • Downtime RECREATE WITH DOWNTIME
 Shutdown existing deployment Kubernetes
 Service
  47. ROLLING UPDATES

  48. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI ROLLING UPDATES with ZERO DOWNTIME Rollingupdate
 maxUnavailable=0 maxSurge=1 ROLLING UPDATES
 Replace each pod using RollingUpdate deployment strategy Kubernetes
 Service
  49. Deploy new version and wait until it’s ready… Health Check:

    readiness probe e.g. tcp, http, script Version 1 Version 1 Version 
 1.2 Version 1 Rollingupdate
 maxUnavailable=0 maxSurge=1 ROLLING UPDATES
 Deploy new version, wait until it’s ready Kubernetes
 Service
  50. Each container/pod is updated one by one Version 1.2 50%

    Version 1 V1 V1.2 ROLLING UPDATES
 Requires backward compatibility, as two versions run side-by-side Kubernetes
 Service
  51. Each container/pod is updated one by one Version 1.2 Version

    1.2 Version 1.2 100% Use Case • Horizontally scaled • Backward compatible API/data • Microservices Pros • Zero downtime • Reduced risk, gradual rollout w/health checks • Ready for rollback Cons • Require backward compatible APIs/data • Resource overhead ROLLING UPDATES Kubernetes
 Service
  52. BLUE / GREEN DEPLOYMENT

  53. BLUE Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT

    Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Single service, run two complete Deployments BLUE Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% Service
 selector:
 production=BLUE Kubernetes
 Deployment
  54. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100% Health Check: readiness probe e.g. tcp, http, script BLUE / GREEN DEPLOYMENT
 Using Deployments, Ingress Service
 selector:
 production=BLUE Kubernetes
 Deployment Kubernetes
 Deployment
  55. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100% Health Check: readiness probe e.g. tcp, http, script BLUE / GREEN DEPLOYMENT
 Using Deployments, Ingress Service
 selector:
 production=BLUE
  56. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100% Health Check: readiness probe e.g. tcp, http, script BLUE / GREEN DEPLOYMENT
 Using Deployments, Ingress Service
 selector:
 production=BLUE
  57. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Route all new request to Green, Blue sessions Service
 selector:
 version=GREEN
  58. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Using Deployments, Ingress Service
 selector:
 production=GREEN
  59. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Scale-down, reduce resources Service
 selector:
 production=GREEN
  60. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE

    / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Hot Backup Service
 selector:
 production=GREEN Version 2
  61. BLUE / GREEN DEPLOYMENT Rollback BLUE GREEN Version 1 Version

    2 Ingress Use Case • Self-contained micro services (data) Pros • Low risk, never change production • No downtime • Production like testing • Rollback Cons • Resource overhead • Data synchronization BLUE / GREEN DEPLOYMENT
 Rollback Service
 selector:
 production=BLUE
  62. ENABLING DEVSECOPS WITH KUBERNETES Databases Images Automation MANAGING CONTAINERIZED MICROSERVICES


    WITH KUBERNETES A/B Testing Migrations External
 Services Deployment Strategies Security What’s Next… CI/CD Scanning External 
 Services Databases Migrations Infrastructure Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100%
  63. EXTERNAL SERVICES

  64. EXTERNAL SERVICES Database outside cluster with IP address External Mongo

    Database Service External Mongo Database Service Development Production IP=10.200.0.2 port=27017 IP=10.100.0.9 port=27017
  65. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017
  66. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.200.0.2 port=27017 Database kind=Service type=ClusterIP name=mongo port=27017 targetport=27017
  67. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.200.0.2 port=27017 Connect with mongodb://mongo Database kind=Service type=ClusterIP name=mongo port=27017 targetport=27017 kind=Endpoints name=mongo ip=10.200.0.2 port=27017
  68. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes

    Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.100.0.9 port=27017 kind=Service type=ClusterIP name=mongo port=27017 targetport=27017 kind=Endpoints name=mongo ip=10.200.0.9 port=27017 Connect with mongodb://mongo Database
  69. DATABASE MIGRATIONS

  70. Application v3 Development Application V2 Test Application v1 Production DB

    v1 DB v2 DB v3 CI/CD PIPELINE Version control database updates, ex: flyway V3__add_table_scooter.sql V2__add_table_truck.sql V1__add_table_car.sql
  71. DATABASE MIGRATIONS Version control database updates with Containers CONTAINER IMAGE

    CONTAINER BUILD FILE SQL MIGRATION SCRIPT Source Code Repository V2__add_table.sql Source Code Repository V2__add_table.sql /var/flyway/data Flyway flyway-mydb:v2.0.0 Registry + Dockerfile
  72. Nodes Pods Services postgresql-0 Persistent Volume A B D C

    PostgreSQL StatefulSet replicas=1 role=postgresq pvcl DATABASE MIGRATION StatefulSet deployment with headless Service v1
  73. Nodes Pods Services postgresql-0 Persistent Volume A B D C

    PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Create a Job for Flyway Flyway Job Secrets = Database Connection Info v1 flyway-mydb:v2.0.0 Image Registry Flyway
  74. role=postgressql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B

    D C PostgreSQL StatefulSet replicas=1 role=postgresql pvc DATABASE MIGRATIONS Apply schema changes to database Flyway Job Secrets = Database Connection Info V2 flyway-mydb:v2.0.0 Flyway
  75. role=postgresql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B

    D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Version control for database with Kubernetes V2
  76. INFRASTRUCTURE

  77. • No security on K8s dashboard • IT infrastructure credentials

    exposed • Enabled access to a large part of Weight Watchers' network • K8s dashboard exposed • AWS environment with telemetry data compromised • Tesla’s infrastructure was used for crypto mining THE CONTAINERS NEWS YOU DON’T WANT • 17 tainted crypto-mining containers on dockerhub • Remained for ~1 year
 with 5 million pulls and • Harvested ~90k in crypto currency.
  78. KUBERNETES ARCHITECTURE Authorization API Server Controller Scheduler etcd etcd etcd

    Kubernetes Master API UI CLI Node 3 Node 1 Node 2 Node 4 Cluster User
  79. CONTROLLING ACCESS TO KUBERNETES API Authorization User BLUE GREEN Version

    1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% Pod (Service Account) etcd 1 Kubernetes API Server Authentication Authorization Admission Control 2 3 4 5
  80. KUBERNETES: POD SECURITY POLICIES Cluster level, Implemented as an Admission

    Controller apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: privileged annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*' spec: privileged: true allowPrivilegeEscalation: true allowedCapabilities: - '*' volumes: - '*' hostNetwork: true hostPorts: - min: 0 max: 65535 hostIPC: true hostPID: true runAsUser: rule: 'RunAsAny' seLinux: rule: 'RunAsAny' supplementalGroups: rule: 'RunAsAny' fsGroup: rule: 'RunAsAny' apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: restricted annotations: seccomp.security.alpha.kubernetes.io/defaultProfileName: 'runtime/default' spec: privileged: false # Required to prevent escalations to root. allowPrivilegeEscalation: false # This is redundant with non-root + disallow privilege escalation, # but we can provide it for defense in depth. requiredDropCapabilities: - ALL # Allow core volume types. volumes: - 'configMap' - 'secret'' # Assume that persistentVolumes set up by the cluster admin are safe to use. - 'persistentVolumeClaim' hostNetwork: false hostPID: false ......
  81. KUBERNETES ARCHITECTURE Authorization API Server Controller Scheduler etcd etcd etcd

    Kubernetes Master API UI CLI Node 3 Node 1 Node 2 Node 4 Cluster User Network, DNS Linux, Container Runtime Management, Monitoring, Logs, Security, Registry Storage
  82. MONITORING CONSIDERATIONS Kubernetes* Container* Host Cluster services, services, pods, 


    deployments metrics Container native metrics Traditional resource metrics - cpu, memory, network, storage prometheus + grafana kubernetes-state-metrics probes Stack Metrics Tool node-exporter Kubernetes metrics server: kubelet:cAdvisor Microservices Distributed applications - traditional app metrics - service discovery - distributed tracing prometheus + grafana jaeger tracing istio
  83. Network Namespace 
 provides resource isolation NETWORK ISOLATION Multi-Environment Multi-Tenant

  84. NETWORK POLICY example: 
 all pods in namespace ‘project-a’ allow

    traffic 
 from any other pods in the same namespace.”
  85. NETWORK SECURITY MODELS Co-Existence Approaches One Cluster Multiple Zones Kubernete

    Cluster Physical Compute 
 isolation based on 
 Network Zones Kubernete Cluster One Cluster Per Zone Kubernete Cluster B Kubernete Cluster A Kubernetes Cluster B C D https://blog.openshift.com/openshift-and-network-security-zones-coexistence-approaches/
  86. Chris Van Tuin Chief Technologist, NA West / Silicon Valley

    cvantuin@redhat.co Be • Don’t ru • If you m limit Lin • Limit SS • Use nam • Define r • Enable • Apply S • Apply S and se • Run pro unprivile http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container CLI SYSTEMD Cgroups Namespaces SELinux Drivers seccomp Read Only mounts Capabilities CONTAINER HOST SECURITY CONTAINERS ARE LINUX
  87. KUBERNETES NODE Network, DNS kube- proxy kubelet Kubernetes Master Linux,

    Container Runtime Management, Monitoring, Logs, Security, Registry Storage Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Best Practices • Don’t run as root • If you must, 
 limit Linux Capabilities • Limit SSH Access • Use namespaces • Define resource quotas • Enable logging • Apply Security Errata • Apply Security Context and seccomp filters • Run production 
 unprivileged containers 
 as read-only http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container CLI SYSTEMD Cgroups Namespaces SELinux Drivers seccomp Read Only mounts Capabilities CONTAINER HOST SECURITY
  88. WHAT’S NEXT?

  89. KUBERNETES NATIVE ADD-ONS kubevirt github.com/kubevirt operators coreos.com/operators knative github.com/knative istio

    istio.io Virtual Machines Day 2 Operations Server-
 less Service Mesh CI/CD tekton tekton.dev
  90. THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin