DeveloperWeek Austin 2018: A DevOps State of Mind: Continuous Security with Kubernetes

Slide 1

Slide 1 text

A DevOps State of Mind: Continuous Security with Kubernetes Chris Van Tuin Chief Technologist, NA West / Silicon Valley [email protected] @chrisvantuin

Slide 2

Slide 2 text

“Only the paranoid survive” - Andy Grove, 1996

Slide 3

Slide 3 text

ENABLING INNOVATION, WHILE EXECUTING AT SCALE Static &  Planned Dynamic &   Policy Driven Execution Innovation Innovation Execution Old New

Slide 4

Slide 4 text

THE WORLD IS AUTOMATING Those who succeed in automation will win

Slide 5

Slide 5 text

I.T. MUST EVOLVE FROM A COST CENTER   TO INNOVATION CENTER Development Model Application Architecture Deployment & Packaging Application Infrastructur e Storage Waterfall Agile Monolithic N-tier Bare Metal Virtual Servers Data Center Hosted Scale Up Scale Out DevOps MicroServices Containers Hybrid Cloud Storage as a Service

Slide 6

Slide 6 text

Applications & devices outside of IT control Cloud computing Software-defined infrastructure Dissolving security perimeter Menacing threat landscape TRADITIONAL NETWORK-BASED DEFENSES ARE NO LONGER ENOUGH SECURING THE ENTERPRISE IS HARDER THAN EVER The way we develop, deploy and manage IT is changing dramatically led by DevOps, Cloud Native Applications, and Hybrid Cloud

Slide 7

Slide 7 text

DEVSECOPS Continuous Security Improvement Process Optimization Security Automation Dev QA Prod Reduce Risks, Lower Costs, Speed Delivery, Speed Reaction

Slide 8

Slide 8 text

DEVSECOPS + + End to End Security DEV QA OPS Culture Process Technology Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Open Source

Slide 9

Slide 9 text

CONTAINERS ENABLE DEVSECOPS

Slide 10

Slide 10 text

CONTAINERS Software packaging concept that typically includes an application and all of its runtime dependencies • HIGHER quality software releases • SHORTER test cycles • EASIER application management CONTAINER CONTAINER APP LIBS HOST OS SERVER APP LIBS BENEFITS

Slide 11

Slide 11 text

LAPTOP Container Application OS dependencies Guest VM LINUX BARE METAL Container Application OS dependencies LINUX VIRTUALIZATION Container Application OS dependencies Virtual Machine LINUX PRIVATE CLOUD Container Application OS dependencies Virtual Machine LINUX PUBLIC CLOUD Container Application OS dependencies Virtual Machine LINUX CONTAINERS - Build Once, Deploy Anywhere   Reducing Risk and Improving Security with Improved Consistency

Slide 12

Slide 12 text

Image Format Distribution Spec Runtime Spec

Slide 13

Slide 13 text

Scheduling Monitoring Persistence Discovery Lifecycle & health Scaling Aggregation Security MORE THAN CONTAINERS…

Slide 14

Slide 14 text

BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Speed, Resiliency, Scalability, Security  

Slide 15

Slide 15 text

AUTOMATION

Slide 16

Slide 16 text

Web Application replicas: 1,   role: app image: myapp:1.0 replicas: 2,   role: web image: httpd:1.7.9 ORCHESTRATION Declarative, Deployment Controller Manager & Data Store (etcd)

Slide 17

Slide 17 text

Web Application ORCHESTRATION Declarative, Deployment Nodes Controller Manager & Data Store (etcd) Physical, VM,   Cloud Instances replicas: 2,   role: web image: httpd:1.7.9 replicas: 1,   role: app image: myapp:1.0

Slide 18

Slide 18 text

role: app role: web role: web Pods Nodes Image Registry ORCHESTRATION Schedule + Provision Pods (Compute/Storage/Network) Web Application replicas: 2,   role: web image: httpd:1.7.9 replicas: 1,   role: app image: myapp:1.0

Slide 19

Slide 19 text

Web Application role: web role: app role: web replicas: 1,   role: app replicas: 2,   role: web ORCHESTRATION Services (Load Balancer), Service discovery with selectors and pod labels Pods Nodes Services Controller Manager & Data Store (etcd)

Slide 20

Slide 20 text

Web Application ORCHESTRATION Service (Load Balancer) Pods Nodes Controller Manager & Data Store (etcd) Ingress / Routes role: web role: app role: web replicas: 1,   role: app replicas: 2,   role: web Services

Slide 21

Slide 21 text

HEALTH CHECK Monitoring & Logging Pods Nodes Services Web Application role: web role: app role: web Ingress / Routes Health Check replicas: 1,   role: app replicas: 2,   role: web

Slide 22

Slide 22 text

Pods Nodes Services Web Application role: web role: app role: web replicas: 1,   role: app replicas: 2,   role: web role: web Controller Manager & Data Store (etcd) HEALTH CHECK Readiness Probe e.g. tcp, http, script Ingress / Routes

Slide 23

Slide 23 text

Web Application replicas: 1,   role: app replicas: 2,   role: web Pods Nodes Services role: web role: app role: web Controller Manager & Data Store (etcd) HEALTH CHECK Ingress / Routes

Slide 24

Slide 24 text

Web Application AUTO-SCALE Monitoring & Logging 80% CPU Pods Nodes Services role: web role: app role: web Ingress / Routes replicas: 1,   role: app replicas: 2,   role: web

Slide 25

Slide 25 text

Web Application 80% CPU Pods Nodes Services role: web role: app role: web Controller Manager & Data Store (etcd) role: app AUTO-SCALE Ingress / Routes replicas: 2   role: app replicas: 2,   role: web

Slide 26

Slide 26 text

Pods Nodes Services Web Application 50% CPU role: web role: app role: app role: web Controller Manager & Data Store (etcd) AUTO-SCALE Ingress / Routes replicas: 2,   role: web replicas: 2,   role: app

Slide 27

Slide 27 text

Network isolation API & Platform access Federated clusters Storage {} CI/CD Monitoring & Logging Builds Images SECURING YOUR CONTAINER ENVIRONMENT Container host Registry

Slide 28

Slide 28 text

CONTAINER IMAGES

Slide 29

Slide 29 text

CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS dependencies 1.2/latest 1.1 CONTAINER IMAGE

Slide 30

Slide 30 text

Config Data Kubernetes configmaps secrets Container image Traditional   data services, Kubernetes   persistent volumes TREAT CONTAINERS AS IMMUTABLE To keep containerized apps portable Application Language runtimes OS dependencies

Slide 31

Slide 31 text

KUBERNETES CONFIGMAP Decouple configuration from container image Application Language runtimes OS dependencies Environment Variable or Volume/File CONTAINER INSTANCE key:value from directories, files, or values KUBERNETES  CONFIGMAP APPLICATION CONFIG FILE Application Configuration File e.g. XML etcd Pod Source Code Repository EnvVar require pod restart Files refresh in time

Slide 32

Slide 32 text

CONTAINER BUILDS

Slide 33

Slide 33 text

docker.io Registry Private Registry Build file Physical, Virtual, Cloud Image Instance Build Run Ship FROM registry.redhat.com/rhel7 RUN groupadd -g 999 appuser && \ useradd -r -u 999 -g appuser appuser USER appuser CMD echo “Hello” CONTAINER BUILDS

Slide 34

Slide 34 text

• Treat build file as a Blueprint • Version control build file • Don’t login to build/configure • Be explicit with versions, not latest • Always list registry pulling FROM • Specify USER, default is root • Each Run creates a new layer BUILD FILE BEST PRACTICES FROM registry.redhat.com/rhel7 RUN groupadd -g 999 appuser && \ useradd -r -u 999 -g appuser appuser USER appuser CMD echo “Hello” Build file

Slide 35

Slide 35 text

A CONVERGED SOFTWARE   SUPPLY CHAIN

Slide 36

Slide 36 text

CONTAINER REGISTRY

Slide 37

Slide 37 text

64% of official images in Docker Hub   contain high priority security vulnerabilities examples: ShellShock (bash) Heartbleed (OpenSSL) Poodle (OpenSSL) Source: Over 30% of Official Images in Docker Hub Contain High Priority Security Vulnerabilities, Jayanth Gummaraju, Tarun Desikan, and Yoshio Turner, BanyanOps, May 2015 (http://www.banyanops.com/pdf/BanyanOps-AnalyzingDockerHub-WhitePaper.pdf) WHAT’S INSIDE THE CONTAINER MATTERS

Slide 38

Slide 38 text

PRIVATE REGISTRY

Slide 39

Slide 39 text

•Authenticating authorship •Non-repudiation •Ensuring image integrity CONTAINER IMAGE SIGNING Validate what images and version are running

Slide 40

Slide 40 text

CONTAINER HOST SECURITY

Slide 41

Slide 41 text

Chris Van Tuin Chief Technologist, NA West / Silicon Valley [email protected] Best Practices • Don’t run as root • If you must,   limit Linux Capabilities • Limit SSH Access • Use namespaces • Define resource quotas • Enable logging • Apply Security Errata • Apply Security Context and seccomp filters • Run production   unprivileged containers   as read-only http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container CLI SYSTEMD Cgroups Namespaces SELinux Drivers seccomp Read Only mounts Capabilities CONTAINER HOST SECURITY CONTAINER HOST SECURITY

Slide 42

Slide 42 text

CONTINUOUS INTEGRATION

Slide 43

Slide 43 text

WHAT’S INSIDE MATTERS…

Slide 44

Slide 44 text

CI/CD PIPELINE Continuous Integration Continuous Build Continuous Deployment Developer -> Source -> Git Git -> RPMS -> Images-> Registry Images from   Registry -> Clusters

Slide 45

Slide 45 text

Security CONTINUOUS INTEGRATION WITH SECURITY SCAN

Slide 46

Slide 46 text

Java Build Environment Language runtimes OS dependencies Build Image Java Code Application Language runtimes OS dependencies Container Image Image Registry Source Repository Image Registry REPRODUCIBLE BUILDS Source to Image with Build Images Source v3.1 v1.0.1 v3.1

Slide 47

Slide 47 text

CONTINUOUS DELIVERY

Slide 48

Slide 48 text

CONTINUOUS DELIVERY WITH CONTAINERS

Slide 49

Slide 49 text

CUSTOM SUPPLY CHAIN CUSTOM SUPPLY CHAIN

Slide 50

Slide 50 text

CONTINUOUS DELIVERY: DEPLOYMENT STRATEGIES

Slide 51

Slide 51 text

CONTINUOUS DELIVERY   DEPLOYMENT STRATEGIES DEPLOYMENT STRATEGIES • Recreate • Rolling updates • Blue / Green deployment • Canary with A/B testing • Database migrations

Slide 52

Slide 52 text

RECREATE

Slide 53

Slide 53 text

Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME

Slide 54

Slide 54 text

Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME

Slide 55

Slide 55 text

Version 1.2 Version 1.2 Version 1.2 RECREATE WITH DOWNTIME Use Case • Non-mission critical services Pros • Simple, clean • No Schema incompatibilities • No API versioning Cons • Downtime

Slide 56

Slide 56 text

ROLLING UPDATES

Slide 57

Slide 57 text

Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI ROLLING UPDATES with ZERO DOWNTIME Rollingupdate  maxUnavailable=0 maxSurge=1

Slide 58

Slide 58 text

Deploy new version and wait until it’s ready… Health Check: readiness probe e.g. tcp, http, script Version 1 Version 1 Version   1.2 Version 1 Rollingupdate  maxUnavailable=0 maxSurge=1

Slide 59

Slide 59 text

Each container/pod is updated one by one Version 1.2 50% Version 1 V1 V1.2

Slide 60

Slide 60 text

Each container/pod is updated one by one Version 1.2 Version 1.2 Version 1.2 100% Use Case • Horizontally scaled • Backward compatible API/data • Microservices Pros • Zero downtime • Reduced risk, gradual rollout w/health checks • Ready for rollback Cons • Require backward compatible APIs/data • Resource overhead

Slide 61

Slide 61 text

BLUE / GREEN

Slide 62

Slide 62 text

BLUE Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100%

Slide 63

Slide 63 text

BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% Health Check: readiness probe e.g. tcp, http, script

Slide 64

Slide 64 text

BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100%

Slide 65

Slide 65 text

BLUE / GREEN DEPLOYMENT Rollback BLUE GREEN Version 1 Version 2 Ingress Use Case • Self-contained micro services (data) Pros • Low risk, never change production • No downtime • Production like testing • Rollback Cons • Resource overhead • Data synchronization

Slide 66

Slide 66 text

CANARY WITH A/B TESTING

Slide 67

Slide 67 text

”only about 1/3 of ideas improve the metrics   they were designed to improve.”  Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION

Slide 68

Slide 68 text

A/B TESTING USING CANARY DEPLOYMENTS

Slide 69

Slide 69 text

25% Conversion Rate ?! Conversion Rate 100% Version B Version A Ingress CANARY DEPLOYMENTS Tests / CI

Slide 70

Slide 70 text

25% Conversion Rate 30% Conversion Rate 75% 25% Version B Version A Ingress CANARY DEPLOYMENTS

Slide 71

Slide 71 text

25% Conversion Rate 30% Conversion Rate 100% Version B Version A Ingress CANARY DEPLOYMENTS

Slide 72

Slide 72 text

25% Conversion Rate 20% Conversion Rate 100% Version B Version A Rollback Ingress CANARY DEPLOYMENTS

Slide 73

Slide 73 text

DATABASE MIGRATIONS

Slide 74

Slide 74 text

DATABASE MIGRATIONS Version control database updates with Containers CONTAINER IMAGE CONTAINER BUILD FILE SQL MIGRATION SCRIPT Source Code Repository V2__add_table.sql Source Code Repository V2__add_table.sql /var/flyway/data Flyway flyway-mydb:v2.0.0 Registry + Dockerfile

Slide 75

Slide 75 text

Application v3 Development Application V2 Test Application v1 Production DB v1 DB v2 DB v3 CI/CD PIPELINE Version control database updates, ex: flyway V3__add_table_scooter.sql V2__add_table_truck.sql V1__add_table_car.sql

Slide 76

Slide 76 text

Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresq pvcl DATABASE MIGRATION StatefulSet deployment with headless Service v1

Slide 77

Slide 77 text

Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Create a Job for Flyway Flyway Job Secrets = Database Connection Info v1 flyway-mydb:v2.0.0 Image Registry Flyway

Slide 78

Slide 78 text

role=postgressql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql pvc DATABASE MIGRATIONS Apply schema changes to database Flyway Job Secrets = Database Connection Info V2 flyway-mydb:v2.0.0 Flyway

Slide 79

Slide 79 text

role=postgresql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Version control for database with Kubernetes V2

Slide 80

Slide 80 text

Network isolation API & Platform access Federated clusters Storage {} CI/CD Monitoring & Logging Images Builds Container host Registry SECURING YOUR CONTAINER ENVIRONMENT

Slide 81

Slide 81 text

NETWORK ISOLATION

Slide 82

Slide 82 text

Network Namespace   provides resource isolation NETWORK ISOLATION Multi-Environment Multi-Tenant

Slide 83

Slide 83 text

NETWORK POLICY example:   all pods in namespace ‘project-a’ allow traffic   from any other pods in the same namespace.”

Slide 84

Slide 84 text

NETWORK SECURITY MODELS Co-Existence Approaches One Cluster Multiple Zones Kubernete Cluster Physical Compute   isolation based on   Network Zones Kubernete Cluster One Cluster Per Zone Kubernete Cluster B Kubernete Cluster A Kubernetes Cluster B C D https://blog.openshift.com/openshift-and-network-security-zones-coexistence-approaches/

Slide 85

Slide 85 text

MONITORING & LOGGING

Slide 86

Slide 86 text

CONTINUOUS FEEDBACK LOOP

Slide 87

Slide 87 text

KUBERNETES MONITORING CONSIDERATIONS Kubernetes* Container* Host Cluster services, services, pods,   deployments metrics Container native metrics Traditional resource metrics - cpu, memory, network, storage prometheus + grafana kubernetes-state-metrics probes Stack Metrics Tool node-exporter kubelet:cAdvisor Application Distributed applications - traditional app metrics - service discovery - distributed tracing prometheus + grafana jaeger tracing istio

Slide 88

Slide 88 text

Aggregate platform and application log access via Kibana + Elasticsearch LOGGING

Slide 89

Slide 89 text

STORAGE SECURITY

Slide 90

Slide 90 text

Local Storage Quota Security Context   Constraints STORAGE SECURITY Data Encryption   at DataStore layer

Slide 91

Slide 91 text

API & PLATFORM ACCESS

Slide 92

Slide 92 text

Authentication via OAuth tokens and SSL certificate Authorization via Policy Engine checks User/Group Defined Roles API & PLATFORM ACCESS

Slide 93

Slide 93 text

FEDERATION

Slide 94

Slide 94 text

Amazon East OpenStack FEDERATED CLUSTERS Roles & access management (in-dev)

Slide 95

Slide 95 text

MICROSERVICES

Slide 96

Slide 96 text

Traffic Control Service Resiliency Chaos Testing Observ- ability Security ISTIO SERVICE MESH

Slide 97

Slide 97 text

Your app... automated like the cloud... but runs on... OPERATORS FOR KUBERNETES

Slide 98

Slide 98 text

OPERATOR MATURITY MODEL

Slide 99

Slide 99 text

Deployment Frequency Lead Time Deployment  Failure Rate Mean Time to Recover 99.999 Service Availability DEVSECOPS METRICS Compliance Score

Slide 100

Slide 100 text

THANK YOU linkedin: Chris Van Tuin email: [email protected] twitter: @chrisvantuin