A DevOps State of Mind: Continuous Security with Kubernetes

Slide 1

Slide 1 text

A DevOps State of Mind: Continuous Security with Kubernetes Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.com

Slide 2

Slide 2 text

DEV QA OPS SECURITY IS AN AFTERTHOUGHT | SECURITY | “Patch? The servers are behind the firewall.” - Anonymous (far too many to name), 2005 - …

Slide 3

Slide 3 text

“Only the paranoid survive” - Andy Grove, 1996

Slide 4

Slide 4 text

THE DISRUPTERS EMBRACING DEVOPS Empowered organization Speed Up   Innovation Time Change Move Fast, Break Things Culture of experimentation A 20% vs. 25% Shorten the Feedback Loop Real-time data-driven intelligence & personalization AI /  ML Data, Data, Data B

Slide 5

Slide 5 text

BARE METAL VIRTUAL PRIVATE CLOUD OFF-PREMISE ON-PREMISE PUBLIC CLOUD DATA DATA DISTRIBUTED SERVICES

Slide 6

Slide 6 text

ANY COMBINATION, WHETHER TRADITIONAL OR CONTAINERIZED LEGACY APPS (1,000+) BARE METAL PRIVATE CLOUD PUBLIC CLOUD VIRTUAL PRODUCTION DEV/TEST HYBRID CLOUD ENVIRONMENTS

Slide 7

Slide 7 text

DEVSECOPS + + Security DEV QA OPS Culture Process Technology Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Cloud Native Applications Hybrid Cloud Open Source

Slide 8

Slide 8 text

DEVSECOPS Continuous Security Improvement Process Optimization Security Automation Dev QA Prod Reduce Risks, Lower Costs, Speed Delivery, Speed Reaction

Slide 9

Slide 9 text

LAPTOP Container Application OS dependencies Guest VM LINUX BARE METAL Container Application OS dependencies LINUX VIRTUALIZATION Container Application OS dependencies Virtual Machine LINUX PRIVATE CLOUD Container Application OS dependencies Virtual Machine LINUX PUBLIC CLOUD Container Application OS dependencies Virtual Machine LINUX CONTAINERS - Build Once, Deploy Anywhere Reducing Risk and Improving Security with Improved Consistency

Slide 10

Slide 10 text

BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Security Platform

Slide 11

Slide 11 text

● No security on K8s dashboard ● IT infrastructure credentials exposed ● Enabled access to a large part of Weight Watchers' network ● K8s dashboard exposed ● AWS environment with telemetry data compromised ● Tesla’s infrastructure was used for crypto mining THE CONTAINERS NEWS YOU DON’T WANT ● 17 tainted crypto-mining containers on dockerhub ● Remained for ~1 year  with 5 million pulls and ● Harvested ~90k in crypto currency.

Slide 12

Slide 12 text

AUTOMATION

Slide 13

Slide 13 text

Web Database role=web role=db role=web replicas=1,   role=db replicas=2,   role=web ORCHESTRATION Deployment, Declarative Pods Nodes Services Controller Manager & Data Store (etcd)

Slide 14

Slide 14 text

Web Database replicas=1,   role=db replicas=2,   role=web HEALTH CHECK Pods Nodes Services role=web role=db role=web Controller Manager & Data Store (etcd)

Slide 15

Slide 15 text

Pods Nodes Services Web Database replicas=1,   role=db replicas=3   role=web AUTO-SCALE 50% CPU role=web role=db role=web role=web Controller Manager & Data Store (etcd)

Slide 16

Slide 16 text

Network isolation API & Platform access Federated clusters Storage {} CI/CD Monitoring & Logging Images Builds SECURING YOUR CONTAINER ENVIRONMENT Container host Registry

Slide 17

Slide 17 text

CONTAINER BUILDS

Slide 18

Slide 18 text

docker.io Registry Private Registry FROM fedora:1.0 CMD echo “Hello” Build file Physical, Virtual, Cloud Image Container Build Run Ship CONTAINER BUILDS

Slide 19

Slide 19 text

4 ● Are there known vulnerabilities in the application layer? ● Are the runtime and OS layers up to date? ● How frequently will the container be updated and how will I know when it’s updated? CONTENT: EACH LAYER MATTERS CONTAINER OS RUNTIME APPLICATION CONTENT: EACH LAYER MATTERS AYER MATTERS CONTAINER OS RUNTIME APPLICATION JAR CONTAINER

Slide 20

Slide 20 text

A CONVERGED SOFTWARE   SUPPLY CHAIN

Slide 21

Slide 21 text

Best Practices • Treat as a Blueprint • Specify a user, defaults to root • Don’t login to build/configure • Version control build file • Be explicit with versions, not latest • Each Run creates a new layer CONTAINER BUILDS FROM fedora:1.0 CMD echo “Hello” Build file Build

Slide 22

Slide 22 text

CONTAINER IMAGE SECURITY

Slide 23

Slide 23 text

code config data Kubernetes configmaps secrets Container image Traditional   data services, Kubernetes   persistent volumes TREAT CONTAINERS AS IMMUTABLE

Slide 24

Slide 24 text

IMAGE SIGNING Validate what images and version are running

Slide 25

Slide 25 text

CONTAINER REGISTRY SECURITY

Slide 26

Slide 26 text

64% of official images in Docker Hub   contain high priority security vulnerabilities examples: ShellShock (bash) Heartbleed (OpenSSL) Poodle (OpenSSL) Source: Over 30% of Official Images in Docker Hub Contain High Priority Security Vulnerabilities, Jayanth Gummaraju, Tarun Desikan, and Yoshio Turner, BanyanOps, May 2015 (http://www.banyanops.com/pdf/BanyanOps-AnalyzingDockerHub-WhitePaper.pdf) WHAT’S INSIDE THE CONTAINER MATTERS

Slide 27

Slide 27 text

PRIVATE REGISTRY

Slide 28

Slide 28 text

CONTAINER HOST SECURITY

Slide 29

Slide 29 text

RUNNING CONTAINER RUNTIME IN READ-ONLY MODE Improve Security, Avoid data loss, Enforce quota Read/Write (default) /volumes tmpfs (memory) rootfs (copy-on-write) Development Container CRI-O Read Only Mode /volume tmpfs (/tmp,/var/tmp,/dev/shm,/run ) /volumes rootfs (/) Production Container

Slide 30

Slide 30 text

Best Practices • Don’t run as root • Limit SSH Access • Use namespaces • Define resource quotas • Enable logging • Apply Security Errata • Apply Security Context and seccomp filters • Run production   unprivileged containers   as read-only http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html CONTAINER HOST SECURITY Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container CLI SYSTEMD Cgroups Namespaces SELinux Drivers seccomp Read Only mounts

Slide 31

Slide 31 text

CGROUPS - RESOURCE ISOLATION

Slide 32

Slide 32 text

NAMESPACES - PROCESS ISOLATION

Slide 33

Slide 33 text

SELINUX - MANDATORY ACCESS CONTROLS Password Files Web Server Attacker Discretionary Access Controls   (file permissions) Mandatory Access Controls   (selinux) Internal Network Firewall Rules Password Files Firewall Rules Internal Network Web Server selinux policy

Slide 34

Slide 34 text

SECCOMP - DROPPING PRIVILEGES

Slide 35

Slide 35 text

READ ONLY MOUNTS

Slide 36

Slide 36 text

CONTINUOUS INTEGRATION WITH CONTAINERS

Slide 37

Slide 37 text

SECURITY IMPLICATIONS What’s inside matters…

Slide 38

Slide 38 text

CONTINUOUS INTEGRATION + SECURITY

Slide 39

Slide 39 text

Security CONTINUOUS INTEGRATION WITH SECURITY SCAN

Slide 40

Slide 40 text

CONTINUOUS DELIVERY WITH CONTAINERS

Slide 41

Slide 41 text

CONTINUOUS DELIVERY WITH CONTAINERS

Slide 42

Slide 42 text

CONTINUOUS DELIVERY + SECURITY

Slide 43

Slide 43 text

CONTINUOUS DELIVERY: DEPLOYMENT STRATEGIES

Slide 44

Slide 44 text

CONTINUOUS DELIVERY DEPLOYMENT STRATEGIES DEPLOYMENT STRATEGIES • Recreate • Rolling updates • Blue / Green deployment

Slide 45

Slide 45 text

Recreate

Slide 46

Slide 46 text

Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME

Slide 47

Slide 47 text

Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME

Slide 48

Slide 48 text

Version 1.2 Version 1.2 Version 1.2 RECREATE WITH DOWNTIME Use Case • Non-mission critical services Cons • Downtime Pros • Simple, clean • No Schema incompatibilities • No API versioning

Slide 49

Slide 49 text

Rolling Updates

Slide 50

Slide 50 text

Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI ROLLING UPDATES with ZERO DOWNTIME

Slide 51

Slide 51 text

Deploy new version and wait until it’s ready… Version 1 Version 1 V1.2 Health Check: readiness probe e.g. tcp, http, script V1

Slide 52

Slide 52 text

Each container/pod is updated one by one Version 1.2 50% Version 1 V1 V1.2

Slide 53

Slide 53 text

Each container/pod is updated one by one Version 1.2 Version 1.2 Version 1.2 100% Use Case • Horizontally scaled • Backward compatible API/data • Microservices Cons • Require backward compatible APIs/data • Resource overhead Pros • Zero downtime • Reduced risk, gradual rollout w/health checks • Ready for rollback

Slide 54

Slide 54 text

Blue / Green Deployment

Slide 55

Slide 55 text

Version 1 BLUE / GREEN DEPLOYMENT Route BLUE

Slide 56

Slide 56 text

Version 1 BLUE / GREEN DEPLOYMENT Version 1.2 BLUE GREEN

Slide 57

Slide 57 text

Version 1 Tests / CI BLUE / GREEN DEPLOYMENT Version 1.2 BLUE GREEN

Slide 58

Slide 58 text

Version 1 Version 1.2 BLUE / GREEN DEPLOYMENT Route Version 1.2 BLUE GREEN

Slide 59

Slide 59 text

Version 1 BLUE / GREEN DEPLOYMENT Rollback Route Version 1.2 BLUE GREEN Use Case • Self-contained micro services (data) Cons • Resource overhead • Data synchronization Pros • Low risk, never change production • No downtime • Production like testing • Rollback

Slide 60

Slide 60 text

RAPID INNOVATION & EXPERIMENTATION

Slide 61

Slide 61 text

”only about 1/3 of ideas improve the metrics   they were designed to improve.”  Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION

Slide 62

Slide 62 text

CONTINUOUS FEEDBACK LOOP

Slide 63

Slide 63 text

A/B TESTING USING CANARY DEPLOYMENTS

Slide 64

Slide 64 text

Version 1.2 Version 1 100% Tests / CI Version 1.2 Route 25% Conversion Rate ?! Conversion Rate CANARY DEPLOYMENTS

Slide 65

Slide 65 text

50% 50% Version 1.2 Version 1 Route Version 1.2 25% Conversion Rate 30% Conversion Rate CANARY DEPLOYMENTS

Slide 66

Slide 66 text

25% Conversion Rate 100% Version 1 Version 1.2 Route Version 1.2 30% Conversion Rate CANARY DEPLOYMENTS

Slide 67

Slide 67 text

Version 1.2 Version 1 100% Route Rollback 25% Conversion Rate 20% Conversion Rate CANARY DEPLOYMENTS

Slide 68

Slide 68 text

Network isolation API & Platform access Federated clusters Storage {} CI/CD Monitoring & Logging Images Builds Container host Registry SECURING YOUR CONTAINER ENVIRONMENT

Slide 69

Slide 69 text

NETWORK SECURITY

Slide 70

Slide 70 text

Kubernetes   Logical Network Model NETWORK SECURITY • Kubernetes uses a flat SDN model • All pods get IP from same CIDR • And live on same logical network • Assumes all nodes communicate  Traditional   Physical Network Model • Each layer represents a Zone with  increased trust - DMZ > App > DB,  interzone flow generally one direction • Intrazone traffic generally unrestricted

Slide 71

Slide 71 text

Network Namespace   provides resource isolation NETWORK ISOLATION Multi-Environment Multi-Tenant

Slide 72

Slide 72 text

NETWORK POLICY example:   all pods in namespace ‘project-a’ allow traffic   from any other pods in the same namespace.”

Slide 73

Slide 73 text

NETWORK SECURITY MODELS Co-Existence Approaches One Cluster Multiple Zones Kubernete Cluster Physical Compute   isolation based on   Network Zones Kubernete Cluster One Cluster Per Zone Kubernete Cluster B Kubernete Cluster A Kubernetes Cluster B C D https://blog.openshift.com/openshift-and-network-security-zones-coexistence-approaches/

Slide 74

Slide 74 text

MONITORING & LOGGING

Slide 75

Slide 75 text

KUBERNETES MONITORING CONSIDERATIONS Kubernetes Application Container Host Cluster services, services, pods,   deployments metrics Distributed applications - traditional app metrics - service discovery - distributed tracing Container native metrics Traditional resource metrics - cpu, memory, network, storage kubernetes-state-metrics probes Stack Metrics Tool prometheus + grafana jaeger tracing istio node-exporter kuberlet:cAdvisor

Slide 76

Slide 76 text

Aggregate platform and application monitoring access  via prometheus + Grafana MONITORING Host

Slide 77

Slide 77 text

Aggregate platform and application log access via Elasticsearch+ Fluentd +Kabana (EFK) LOGGING https://www.slideshare.net/JosefKarsek/logsmetrics-gathering-with-openshift-efk-stack

Slide 78

Slide 78 text

STORAGE SECURITY

Slide 79

Slide 79 text

Local Storage Quota Security Context Constraints STORAGE SECURITY Sometimes we can also have storage isolation requirements:   pods in a network zone must use different storage endpoints   than pods in other network zones. We can create one storage class per storage endpoint and   then control which storage class(es) a project can use

Slide 80

Slide 80 text

API & PLATFORM ACCESS

Slide 81

Slide 81 text

Authentication via OAuth tokens and SSL certificate Authorization via Policy Engine checks User/Group Defined Roles API & PLATFORM ACCESS

Slide 82

Slide 82 text

FEDERATION

Slide 83

Slide 83 text

Amazon East OpenStack FEDERATED CLUSTERS Roles & access management (in-dev)

Slide 84

Slide 84 text

MICROSERVICES

Slide 85

Slide 85 text

Monitoring & Metrics -prometheus (logs) -grafana (visual) Access Control & usage policies -mixr (policy decisions) Encryption & Auth -citadel -service 2 service -user auth Traffic routing - pilot - circuit breaker - a/b testing - traffic mirroring Fault injections -envoy corner cases: abort & delays SERVICE MESH

Slide 86

Slide 86 text

OPERATORS

Slide 87

Slide 87 text

CRI-O v1.10 Feature(s): CRI-O v1.10 Description: CRI-O is an OCI compliant implementation of the Kubernetes Container Runtime Interface. By design it provides only the runtime capabilities needed by the kubelet. CRI-O is designed to be part of Kubernetes and evolve in lock-step with the platform. CRI-O brings: ● A minimal and secure architecture ● Excellent scale and performance ● Ability to run any OCI / Docker image ● Familiar operational tooling and commands Improvements include: ● crictl CLI for debugging and troubleshooting ● Podman for image tagging & management ● Installer integration & fresh install time decision: openshift_use_crio=True ○ Not available for existing cluster upgrades Kubelet Storage Image RunC CNI Networking

Slide 88

Slide 88 text

Deployment Frequency Lead Time Deployment  Failure Rate Mean Time to Recover 99.999 Service Availability DEVSECOPS METRICS Compliance Score

Slide 89

Slide 89 text

THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin