Slide 1

Slide 1 text

A DEVOPS STATE OF MIND: CONTINUOUS SECURITY WITH KUBERNETES Chris Van Tuin
 Chief Technologist, NA West @chrisvantuin [email protected]

Slide 2

Slide 2 text

“Only the paranoid survive” - Andy Grove, 1996

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

“Only the paranoid survive” - Andy Grove, 1996

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

DEVSECOPS

Slide 7

Slide 7 text

DEV QA OPS SECURITY IS AN AFTERTHOUGHT | SECURITY | “Patch? The servers are behind the firewall.” - Anonymous (far too many to name), 2005 - … | Security | SECURITY IS AN AFTERTHOUGHT

Slide 8

Slide 8 text

DEVSECOPS + + End to End Security DEV QA OPS Culture Process Technology Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Open Source

Slide 9

Slide 9 text

DEVSECOPS Continuous Security Improvement Process Optimization Security Automation Dev QA Prod Reduce Risks, Lower Costs, Speed Delivery, Speed Reaction

Slide 10

Slide 10 text

DEVSECOPS AT SCALE

Slide 11

Slide 11 text

BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory
 Speed, Resiliency, Scalability, Security 
 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory
 Speed, Resiliency, Scalability, Security 
 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory
 Speed, Resiliency, Scalability, Security 
 Speed, Agility, Resiliency, Scalability, Efficiency, Security

Slide 12

Slide 12 text

● No security on K8s dashboard ● IT infrastructure credentials exposed ● Enabled access to a large part of Weight Watchers' network ● K8s dashboard exposed ● AWS environment with telemetry data compromised ● Tesla’s infrastructure was used for crypto mining THE CONTAINERS NEWS YOU DON’T WANT ● 17 tainted crypto-mining containers on dockerhub ● Remained for ~1 year
 with 5 million pulls and ● Harvested ~90k in crypto currency.

Slide 13

Slide 13 text

SECURING CONTAINERS BEST PRACTICES Reduce Attack Surface Least Privilege Layered Security $ sudo Immutable

Slide 14

Slide 14 text

CONTAINER IMAGES

Slide 15

Slide 15 text

docker.io Registry Private Registry FROM fedora:1.0 CMD echo “Hello” Build file Physical, Virtual, Cloud Container Image Container Instance Build Run Ship CONTAINERS ENABLE DEVOPS CONTAINERS ENABLE DEVSECOPS FROM registry.redhat.com/rhel7 RUN groupadd -g 999 appuser && \ useradd -r -u 999 -g appuser appuser USER appuser CMD echo “Hello”

Slide 16

Slide 16 text

CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS dependencies 1.2/latest 1.1

Slide 17

Slide 17 text

Config Data Kubernetes configmaps secrets Container image Traditional 
 data services, Kubernetes 
 persistent volumes TREAT CONTAINERS AS IMMUTABLE To keep containerized apps portable Application Language runtimes OS dependencies

Slide 18

Slide 18 text

KUBERNETES CONFIGMAP Decouple configuration from container image Application Language runtimes OS dependencies Environment Variable or Volume/File CONTAINER INSTANCE key:value from directories, files, or values KUBERNETES
 CONFIGMAP APPLICATION CONFIG FILE Application Configuration File e.g. XML etcd Pod Source Code Repository EnvVar require pod restart Files refresh in time

Slide 19

Slide 19 text

NODE MASTER Container Distributed Store Container KUBERNETES SECRETS ! Secure mechanism for holding sensitive data e.g. ○ Passwords and credentials ○ SSH Keys ○ Certificates ! Secrets are made available as ○ Environment variables ○ Volume mounts ○ Interaction with external systems ! Encrypted in transit and
 support for encryption at rest ! Never rest on the nodes, stored in memory (tmpfs)

Slide 20

Slide 20 text

• Treat build file as a Blueprint • Don’t login to build/configure • Version control build file • Be explicit with versions, not latest • Always list registry pulling FROM • Each Run creates a new layer • Specify USER, default is root BUILD FILE BEST PRACTICES FROM registry.redhat.com/rhel7 RUN groupadd -g 999 appuser && \ useradd -r -u 999 -g appuser appuser USER appuser CMD echo “Hello” Build file

Slide 21

Slide 21 text

CONTAINER IMAGE SIGNING Validate what images and version are running • Authenticating authorship • Non-repudiation • Ensuring image integrity

Slide 22

Slide 22 text

CONTINUOUS BUILDS

Slide 23

Slide 23 text

Java Build Environment Language runtimes OS dependencies Build Image Java Code Application Language runtimes OS dependencies Container Image Image Registry Source Repository Image Registry REPRODUCIBLE BUILDS Source to Image with Build Images Source v3.1 v1.0.1 v3.1 Java Build Environment Language runtimes OS dependencies Build Image Java Code Application Language runtimes OS dependencies Container Image Image Registry Source Repository Image Registry REPRODUCIBLE BUILDS Source to Image with Build Images Source v3.1 v1.0.1 v3.1 + REPRODUCIBLE BUILDS with build images

Slide 24

Slide 24 text

CUSTOM SUPPLY CHAIN CASCADING REBUILDS

Slide 25

Slide 25 text

CONTAINER REGISTRY & SCANNING

Slide 26

Slide 26 text

WHAT’S INSIDE MATTERS…

Slide 27

Slide 27 text

PRIVATE REGISTRY PRIVATE REGISTRY Trusted Content

Slide 28

Slide 28 text

Security CONTINUOUS INTEGRATION WITH SECURITY SCAN

Slide 29

Slide 29 text

AUTOMATED SECURITY SCANNING with OpenSCAP Reports & Remediation Scan SCAP Security Guide for RHEL CCE-27002-5 Set Password Minimum Length Content Scan physical servers, virtual machines, docker images and containers
 for Security Policy Compliance (CCEs) and known Security Vulnerabilities (CVEs)

Slide 30

Slide 30 text

DEPLOYMENT STRATEGIES

Slide 31

Slide 31 text

CI/CD PIPELINE WITH KUBERNETES BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD CI/CD PIPELINE WITH KUBERNETES BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory
 Speed, Resiliency, Scalability, Security 


Slide 32

Slide 32 text

CI/CD DEPLOYMENT STRATEGIES
 Automate and reduce deployment risk DEPLOYMENT STRATEGIES • Recreate • Rolling updates • Blue / Green deployment

Slide 33

Slide 33 text

RECREATE

Slide 34

Slide 34 text

Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME RECREATE WITH DOWNTIME
 Using Recreate deployment strategy Kubernetes
 Service

Slide 35

Slide 35 text

Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME RECREATE WITH DOWNTIME
 Shutdown existing deployment Kubernetes
 Service

Slide 36

Slide 36 text

Version 1.2 Version 1.2 Version 1.2 RECREATE WITH DOWNTIME Use Case • Non-mission critical services Pros • Simple, clean • No Schema incompatibilities • No API versioning Cons • Downtime RECREATE WITH DOWNTIME
 Shutdown existing deployment Kubernetes
 Service

Slide 37

Slide 37 text

ROLLING UPDATES

Slide 38

Slide 38 text

Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI ROLLING UPDATES with ZERO DOWNTIME Rollingupdate
 maxUnavailable=0 maxSurge=1 ROLLING UPDATES
 Replace each pod using RollingUpdate deployment strategy Kubernetes
 Service

Slide 39

Slide 39 text

Deploy new version and wait until it’s ready… Health Check: readiness probe e.g. tcp, http, script Version 1 Version 1 Version 
 1.2 Version 1 Rollingupdate
 maxUnavailable=0 maxSurge=1 ROLLING UPDATES
 Deploy new version, wait until it’s ready Kubernetes
 Service

Slide 40

Slide 40 text

Each container/pod is updated one by one Version 1.2 50% Version 1 V1 V1.2 ROLLING UPDATES
 Requires backward compatibility, as two versions run side-by-side Kubernetes
 Service

Slide 41

Slide 41 text

Each container/pod is updated one by one Version 1.2 Version 1.2 Version 1.2 100% Use Case • Horizontally scaled • Backward compatible API/data • Microservices Pros • Zero downtime • Reduced risk, gradual rollout w/health checks • Ready for rollback Cons • Require backward compatible APIs/data • Resource overhead ROLLING UPDATES Kubernetes
 Service

Slide 42

Slide 42 text

BLUE / GREEN DEPLOYMENT

Slide 43

Slide 43 text

BLUE Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Single service, run two complete Deployments BLUE Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% Service
 selector:
 production=BLUE Kubernetes
 Deployment

Slide 44

Slide 44 text

BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% Health Check: readiness probe e.g. tcp, http, script BLUE / GREEN DEPLOYMENT
 Using Deployments, Ingress Service
 selector:
 production=BLUE Kubernetes
 Deployment Kubernetes
 Deployment

Slide 45

Slide 45 text

BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% Health Check: readiness probe e.g. tcp, http, script BLUE / GREEN DEPLOYMENT
 Using Deployments, Ingress Service
 selector:
 production=BLUE

Slide 46

Slide 46 text

BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% Health Check: readiness probe e.g. tcp, http, script BLUE / GREEN DEPLOYMENT
 Using Deployments, Ingress Service
 selector:
 production=BLUE

Slide 47

Slide 47 text

BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Route all new request to Green, Blue sessions Service
 selector:
 version=GREEN

Slide 48

Slide 48 text

BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Using Deployments, Ingress Service
 selector:
 production=GREEN

Slide 49

Slide 49 text

BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Scale-down, reduce resources Service
 selector:
 production=GREEN

Slide 50

Slide 50 text

BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Hot Backup Service
 selector:
 production=GREEN Version 2

Slide 51

Slide 51 text

BLUE / GREEN DEPLOYMENT Rollback BLUE GREEN Version 1 Version 2 Ingress Use Case • Self-contained micro services (data) Pros • Low risk, never change production • No downtime • Production like testing • Rollback Cons • Resource overhead • Data synchronization BLUE / GREEN DEPLOYMENT
 Rollback Service
 selector:
 production=BLUE

Slide 52

Slide 52 text

Deployment Frequency Lead Time Deployment
 Failure Rate Mean Time to Recover 99.999 Service Availability DEVSECOPS METRICS Compliance Score

Slide 53

Slide 53 text

INFRASTRUCTURE

Slide 54

Slide 54 text

KUBERNETES ARCHITECTURE Authorization API Server Controller Scheduler etcd etcd etcd Kubernetes Master API UI CLI Node 3 Node 1 Node 2 Node 4 Cluster User

Slide 55

Slide 55 text

KUBERNETES ARCHITECTURE Authorization API Server Controller Scheduler etcd etcd etcd Kubernetes Master API UI CLI Node 3 Node 1 Node 2 Node 4 Cluster User Network, DNS Linux, Container Runtime Management, Monitoring, Logs, Security, Registry Storage

Slide 56

Slide 56 text

ARCHITECTURE CONSIDERATIONS Optimize for… Cluster 
 per app / data / location, Short lived Data Sensitive, e.g. Finance Multi-AZ, Multi/
 Hybrid
 cloud Production, Mission 
 critical Bare metal HPC, AI/ML Security Scale Availability Latency Portability Performance Large cluster, multi/
 hybrid cloud Internet, SaaS Efficiency Large cluster, Bare Metal, Recreate Many apps, Large scale Consistent
 OS & Kubernetes version 1 app anywhere, e.g. ISVs Local, Small Cluster IoT, Retail

Slide 57

Slide 57 text

Kubernetes 
 Logical Network Model NETWORK SECURITY • Kubernetes uses a flat SDN model • All pods get IP from same CIDR • And live on same logical network • Assumes all nodes communicate
 Traditional 
 Physical Network Model • Each layer represents a Zone with
 increased trust - DMZ > App > DB,
 interzone flow generally one direction • Intrazone traffic generally unrestricted

Slide 58

Slide 58 text

NETWORK SECURITY MODELS Co-Existence Approaches One Cluster Multiple Zones Kubernete Cluster Physical Compute 
 isolation based on 
 Network Zones Kubernete Cluster One Cluster Per Zone Kubernete Cluster B Kubernete Cluster A Kubernetes Cluster B C D https://blog.openshift.com/openshift-and-network-security-zones-coexistence-approaches/

Slide 59

Slide 59 text

Network Namespace 
 provides resource isolation NETWORK ISOLATION Multi-Environment Multi-Tenant

Slide 60

Slide 60 text

NETWORK POLICY example: 
 all pods in namespace ‘project-a’ allow traffic 
 from any other pods in the same namespace.”

Slide 61

Slide 61 text

KUBERNETES: POD SECURITY POLICIES Cluster level, Implemented as an Admission Controller apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: privileged annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*' spec: privileged: true allowPrivilegeEscalation: true allowedCapabilities: - '*' volumes: - '*' hostNetwork: true hostPorts: - min: 0 max: 65535 hostIPC: true hostPID: true runAsUser: rule: 'RunAsAny' seLinux: rule: 'RunAsAny' supplementalGroups: rule: 'RunAsAny' fsGroup: rule: 'RunAsAny' apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: restricted annotations: seccomp.security.alpha.kubernetes.io/defaultProfileName: 'runtime/default' spec: privileged: false # Required to prevent escalations to root. allowPrivilegeEscalation: false # This is redundant with non-root + disallow privilege escalation, # but we can provide it for defense in depth. requiredDropCapabilities: - ALL # Allow core volume types. volumes: - 'configMap' - 'secret'' # Assume that persistentVolumes set up by the cluster admin are safe to use. - 'persistentVolumeClaim' hostNetwork: false hostPID: false ......

Slide 62

Slide 62 text

CONTROLLING ACCESS TO KUBERNETES API Authorization User BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% Pod (Service Account) etcd 1 Kubernetes API Server Authentication Authorization Admission Control 2 3 4 5

Slide 63

Slide 63 text

MONITORING CONSIDERATIONS Kubernetes* Container* Host Cluster services, services, pods, 
 deployments metrics Container native metrics Traditional resource metrics - cpu, memory, network, storage prometheus + grafana kubernetes-state-metrics probes Stack Metrics Tool node-exporter Kubernetes metrics server: kubelet:cAdvisor Microservices Distributed applications - traditional app metrics - service discovery - distributed tracing prometheus + grafana jaeger tracing istio

Slide 64

Slide 64 text

Chris Van Tuin Chief Technologist, NA West / Silicon Valley [email protected] Be • Don’t ru • If you m limit Lin • Limit SS • Use nam • Define r • Enable • Apply S • Apply S and se • Run pro unprivile http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container CLI SYSTEMD Cgroups Namespaces SELinux Drivers seccomp Read Only mounts Capabilities CONTAINER HOST SECURITY CONTAINERS ARE LINUX

Slide 65

Slide 65 text

CGROUPS - RESOURCE ISOLATION

Slide 66

Slide 66 text

NAMESPACES - PROCESS ISOLATION

Slide 67

Slide 67 text

SELINUX - MANDATORY ACCESS CONTROLS Password Files Web Server Attacker Discretionary Access Controls 
 (file permissions) Mandatory Access Controls 
 (selinux) Internal Network Firewall Rules Password Files Firewall Rules Internal Network Web Server selinux policy

Slide 68

Slide 68 text

SECCOMP AND LINUX CAPABILITIES
 FILTERING SYSTEM CALLS and DROPPING PRIVILEGES SECCOMP AND LINUX CAPABILITIES Filtering 300+ system calls & limiting root privileges

Slide 69

Slide 69 text

READ ONLY MOUNTS

Slide 70

Slide 70 text

KUBERNETES NODE Network, DNS kube- proxy kubelet Kubernetes Master Linux, Container Runtime Management, Monitoring, Logs, Security, Registry Storage Chris Van Tuin Chief Technologist, NA West / Silicon Valley [email protected] Best Practices • Don’t run as root • If you must, 
 limit Linux Capabilities • Limit SSH Access • Use namespaces • Define resource quotas • Enable logging • Apply Security Errata • Apply Security Context and seccomp filters • Run production 
 unprivileged containers 
 as read-only http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container CLI SYSTEMD Cgroups Namespaces SELinux Drivers seccomp Read Only mounts Capabilities CONTAINER HOST SECURITY

Slide 71

Slide 71 text

WHAT’S NEXT?

Slide 72

Slide 72 text

KUBERNETES NATIVE ADD-ONS kubevirt github.com/kubevirt operators coreos.com/operators knative github.com/knative istio istio.io Virtual Machines Day 2 Operations Server-
 less Service Mesh CI/CD tekton tekton.dev

Slide 73

Slide 73 text

OPERATORS AUTOMATED OPERATIONS WITH OPERATORS Automate operational lifecycle at Scale
 including packaging, deploying, managing
 of Kubernetes applications Takes human operational
 knowledge and encodes 
 it into software Watches over your Kubernetes environment
 and uses its current state to make decisions in milliseconds.

Slide 74

Slide 74 text

THANK YOU linkedin: Chris Van Tuin email: [email protected] twitter: @chrisvantuin