A DevOps State of Mind: Continuous Security with Kubernetes

Slide 1

Slide 1 text

A DEVOPS STATE OF MIND: Continuous Security with Kubernetes Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.com

Slide 2

Slide 2 text

THE DISRUPTORS

Slide 3

Slide 3 text

“Only the Paranoid Survive” - Andy Grove, 1998

Slide 4

Slide 4 text

Retail Finance Media Transportation ? ? SOFTWARE DISRUPTS BUSINESS

Slide 5

Slide 5 text

+ ≠ TAXI TRANSPORTATION DISRUPTER https://goo.gl/MP7QQH Ack: Andrew Ng APP IT’S NOT JUST A… +

Slide 6

Slide 6 text

Culture of experimentation A B 20% vs. 25% Empowered organization Time Change Rapid Innovation THE DISRUPTORS = AI /  ML Data-driven intelligence Data, Data, Data

Slide 7

Slide 7 text

WHAT CAN I.T. DO?

Slide 8

Slide 8 text

DEV QA OPS Walled off people, walled off processes, walled off technologies “THROW IT OVER THE WALL”

Slide 9

Slide 9 text

I.T. MUST EVOLVE TO ENABLE THE BUSINESS Development Model Application Architecture Deployment & Packaging Application Infrastructure Storage Waterfall Agile Monolithic N-tier Bare Metal Virtual Servers Data Center Hosted Scale Up Scale Out DevOps MicroServices Containers Hybrid Cloud Storage as a Service

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

DEV QA OPS Open organization +   cross-functional teams Software factory automation Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Open Source CI/CD pipelines with feedback Culture Process Technology + + BREAKING DOWN THE WALLS WITH DEVOPS

Slide 13

Slide 13 text

WHAT ABOUT SECURITY?

Slide 14

Slide 14 text

DEV QA OPS SECURITY IS AN AFTERTHOUGHT | SECURITY | “Patch? The servers are behind the firewall.” - Anonymous (far too many to name), 2005 - …

Slide 15

Slide 15 text

http://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/

Slide 16

Slide 16 text

DEVSECOPS End to End Security + + SECURITY DEV QA OPS Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Open Source Culture Process Technology

Slide 17

Slide 17 text

CONTAINERS

Slide 18

Slide 18 text

APPLICATION DELIVERY VIA CONTAINERS

Slide 19

Slide 19 text

docker.io Registry Private Registry FROM fedora:latest CMD echo “Hello” Build file Physical, Virtual, Cloud Image Container Build Run Ship CONTAINERS: BUILD, SHIP, RUN

Slide 20

Slide 20 text

CONTAINER SECURITY AT SCALE

Slide 21

Slide 21 text

Scheduling Monitoring Persistence Discovery Lifecycle & health Scaling Aggregation Security MORE THAN CONTAINERS…

Slide 22

Slide 22 text

DEVSECOPS End to End Security + + DEV QA OPS SECURITY

Slide 23

Slide 23 text

4 ● Are there known vulnerabilities in the application layer? ● Are the runtime and OS layers up to date? ● How frequently will the container be updated and how will I know when it’s updated? CONTENT: EACH LAYER MATTERS CONTAINER OS RUNTIME APPLICATION Are there known vulnerabilities   in each application layer? Are the runtime and OS layers   up to date? How frequently will the container be updated and how will I know when its updated? IS THE CONTAINER ENVIRONMENT SECURE? Is the image from a trusted source? Can I quickly deploy a security update at scale? Is my multi-tenant host secure?

Slide 24

Slide 24 text

Network isolation Storage API & Platform access Monitoring & Logging Federated clusters Registry Container host {} Builds CI/CD Images SECURING CONTAINERS

Slide 25

Slide 25 text

CONTAINER IMAGE SECURITY

Slide 26

Slide 26 text

4 ● Are there known vulnerabilities in the application layer? ● Are the runtime and OS layers up to date? ● How frequently will the container be updated and how will I know when it’s updated? CONTENT: EACH LAYER MATTERS CONTAINER OS RUNTIME APPLICATION CONTENT: EACH LAYER MATTERS AYER MATTERS CONTAINER OS RUNTIME APPLICATION JAR CONTAINER

Slide 27

Slide 27 text

SECURITY IMPLICATIONS What’s inside matters…

Slide 28

Slide 28 text

A CONVERGED SOFTWARE SUPPLY CHAIN

Slide 29

Slide 29 text

code config data Kubernetes configmaps secrets Container image Traditional   data services, Kubernetes   persistent volumes TREAT CONTAINERS AS IMMUTABLE

Slide 30

Slide 30 text

NODE MASTER Container Distributed Store Container ! Secure mechanism for holding sensitive data e.g. ○ Passwords and credentials ○ SSH Keys ○ Certificates ! Secrets are made available as ○ Environment variables ○ Volume mounts ○ Interaction with external systems ! Encrypted in transit ! Never rest on the nodes SECRETS

Slide 31

Slide 31 text

IMAGE SIGNING Validate what images and version are running

Slide 32

Slide 32 text

CONTAINER REGISTRY SECURITY

Slide 33

Slide 33 text

64% of official images in Docker Hub   contain high priority security vulnerabilities examples: ShellShock (bash) Heartbleed (OpenSSL) Poodle (OpenSSL) Source: Over 30% of Official Images in Docker Hub Contain High Priority Security Vulnerabilities, Jayanth Gummaraju, Tarun Desikan, and Yoshio Turner, BanyanOps, May 2015 (http://www.banyanops.com/pdf/BanyanOps-AnalyzingDockerHub-WhitePaper.pdf) WHAT’S INSIDE THE CONTAINER MATTERS

Slide 34

Slide 34 text

PRIVATE REGISTRY

Slide 35

Slide 35 text

CI WITH CONTAINERS

Slide 36

Slide 36 text

CI/CD PIPELINE Continuous Integration Continuous Build Continuous Deployment Developer -> Source -> Git Git -> RPMS -> Images-> Registry Images from   Registry -> Clusters

Slide 37

Slide 37 text

CI/CD with Security

Slide 38

Slide 38 text

Security CONTINUOUS INTEGRATION WITH SECURITY SCAN

Slide 39

Slide 39 text

CUSTOM SUPPLY CHAIN

Slide 40

Slide 40 text

FROM fedora:latest CMD echo “Hello” Build file Build Best Practices • Treat as a Blueprint • Don’t login to build/configure • Version control build file • Be explicit with versions, not latest • Each Run creates a new layer BUILDS

Slide 41

Slide 41 text

Compliance and Vulnerability Audits with OpenSCAP

Slide 42

Slide 42 text

AUTOMATED SECURITY SCANNING with OpenSCAP Reports Scan SCAP Security Guide for RHEL CCE-27002-5 Set Password Minimum Length Content Scan physical servers, virtual machines, docker images and containers  for Security Policy Compliance (CCEs) and known Security Vulnerabilities (CVEs)

Slide 43

Slide 43 text

Standard Docker Host Security Profile Java Runtime Environment (JRE) Upstream Firefox STIG RHEL OSP STIG Red Hat Corporate Profile for Certified Cloud Providers (RH CCP) STIG for Red Hat Enterprise Linux 6, 7 Server STIG for Red Hat Virtualization Hypervisor Common Profile for General-Purpose Debian Systems Common Profile for General-Purpose Fedora Systems Common Profile for General-Purpose Ubuntu Systems Payment Card Industry – Data Security Standard (PCI-DSS) v3 U.S. Government Commercial Cloud Services (C2S) CNSSI 1253 Low/Low/Low Control Baseline for Red Hat Enterprise Linux 7 Criminal Justice Information Services (CJIS) Security Policy Unclassified Information in Non-federal Information Systems and Organizations (NIST 800-171) U.S. Government Configuration Baseline (NIAP OSPP v4.0, USGCB, STIG) Security Policies in SCAP Security Guide (partial)

Slide 44

Slide 44 text

SECURITY POLICY REPORT

Slide 45

Slide 45 text

SECURITY POLICY REMEDIATION

Slide 46

Slide 46 text

SECURITY VULNERABILITY REPORT

Slide 47

Slide 47 text

RAPID INNOVATION & EXPERIMENTATION

Slide 48

Slide 48 text

”only about 1/3 of ideas improve the metrics   they were designed to improve.”  Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION

Slide 49

Slide 49 text

CONTINUOUS FEEDBACK LOOP

Slide 50

Slide 50 text

A/B TESTING USING CANARY DEPLOYMENTS

Slide 51

Slide 51 text

50% 50% Version 1.2 Version 1 Route Version 1.2 25% Conversion Rate 30% Conversion Rate CANARY DEPLOYMENTS

Slide 52

Slide 52 text

CD WITH CONTAINERS

Slide 53

Slide 53 text

CONTINUOUS DEPLOYMENT WITH CONTAINERS

Slide 54

Slide 54 text

CD with Security

Slide 55

Slide 55 text

CONTINUOUS DELIVERY DEPLOYMENT STRATEGIES DEPLOYMENT STRATEGIES • Blue / Green deployment • Rolling updates • Canary deployments • A / B testing

Slide 56

Slide 56 text

Version 1 BLUE / GREEN DEPLOYMENT Route

Slide 57

Slide 57 text

Version 1 BLUE / GREEN DEPLOYMENT Version 1.2

Slide 58

Slide 58 text

Version 1 Tests / CI BLUE / GREEN DEPLOYMENT Version 1.2

Slide 59

Slide 59 text

Version 1 Version 1.2 BLUE / GREEN DEPLOYMENT Route Version 1.2

Slide 60

Slide 60 text

Version 1 BLUE / GREEN DEPLOYMENT Rollback Route Version 1.2

Slide 61

Slide 61 text

Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI ROLLING UPDATES with ZERO DOWNTIME

Slide 62

Slide 62 text

Deploy new version and wait until it’s ready… Version 1 Version 1 V1.2 Health Check: Readiness   Probe e.g. tcp, http, script V1

Slide 63

Slide 63 text

Each container/pod is updated one by one Version 1.2 50% Version 1 V1 V1.2

Slide 64

Slide 64 text

Each container/pod is updated one by one Version 1.2 Version 1.2 Version 1.2 100%

Slide 65

Slide 65 text

CONTAINER HOST SECURITY

Slide 66

Slide 66 text

Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container CLI SYSTEMD Cgroups Namespaces SELinux Drivers CONTAINERS ARE LINUX seccomp Read Only mounts

Slide 67

Slide 67 text

CGROUPS - RESOURCE ISOLATION

Slide 68

Slide 68 text

NAMESPACES - PROCESS ISOLATION

Slide 69

Slide 69 text

SELINUX - MANDATORY ACCESS CONTROLS Password Files Web Server Attacker Discretionary Access Controls   (file permissions) Mandatory Access Controls   (selinux) Internal Network Firewall Rules Password Files Firewall Rules Internal Network Web Server selinux policy

Slide 70

Slide 70 text

SECCOMP - DROPPING PRIVILEGES

Slide 71

Slide 71 text

READ ONLY MOUNTS

Slide 72

Slide 72 text

Best Practices • Don’t run as root • Limit SSH Access • Use namespaces • Define resource quotas • Enable logging • Apply Security Errata • Apply Security Context and seccomp filters http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html CONTAINER HOST SECURITY Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container SYSTEM Cgroup Namespace SELinu Driver seccom Read Only

Slide 73

Slide 73 text

Network isolation Storage API & Platform access Monitoring & Logging Federated clusters Registry Container host {} Builds CI/CD Images SECURING CONTAINERS

Slide 74

Slide 74 text

NETWORK ISOLATION

Slide 75

Slide 75 text

Network Namespace   provides resource isolation NETWORK ISOLATION Multi-Environment Multi-Tenant

Slide 76

Slide 76 text

NETWORK POLICY example:   all pods in namespace ‘project-a’ allow traffic   from any other pods in the same namespace.”

Slide 77

Slide 77 text

STORAGE SECURITY

Slide 78

Slide 78 text

Local Storage Quota Security Context Constraints STORAGE SECURITY

Slide 79

Slide 79 text

API & PLATFORM ACCESS

Slide 80

Slide 80 text

Authentication via OAuth tokens and SSL certificate Authorization via Policy Engine checks User/Group Defined Roles API & PLATFORM ACCESS

Slide 81

Slide 81 text

MONITORING & LOGGING

Slide 82

Slide 82 text

Aggregate platform and application log access via Kibana + Elasticsearch LOGGING

Slide 83

Slide 83 text

Historical CPU and Memory usage provided by Heapster, Hawkular, Cassandra MONITORING

Slide 84

Slide 84 text

FEDERATION

Slide 85

Slide 85 text

Amazon East OpenStack FEDERATED CLUSTERS

Slide 86

Slide 86 text

Google Global Routing Amazon/Google Global API https://commons.wikimedia.org/wiki/File:Iscosahedral_water_cluster_100.png Java App1 Amazon East Java App1 Amazon West Java App1 FEDERATED SERVICES

Slide 87

Slide 87 text

Deployment Frequency Lead Time Deployment  Failure Rate Mean Time to Recover 99.999 Service Availability DEVSECOPS METRICS Compliance Score

Slide 88

Slide 88 text

THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin