CloudExpo West: A DevOps State of Mind: Continuous Security for Containers

Transcript

1. A DEVOPS STATE OF MIND: Continuous Security for Containers Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.com
2. “Only the Paranoid Survive” - Andy Grove, 1998
3. http://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/
4. DEV QA OPS SECURITY IS AN AFTERTHOUGHT | SECURITY | “Patch? The servers are behind the firewall.” - Anonymous (far too many to name), 2005 - …
7. DEVSECOPS End to End Security + + SECURITY DEV QA OPS Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Open Source Culture Process Technology
8. CONTAINERS
9. WHAT ARE CONTAINERS? It depends who you ask…
10. APPLICATION DELIVERY VIA CONTAINERS
11. docker.io Registry Private Registry FROM fedora:latest CMD echo “Hello” Build file Physical, Virtual, Cloud Image Container Build Run Ship CONTAINERS: BUILD, SHIP, RUN
12. CONTAINER SECURITY AT SCALE
13. Scheduling Monitoring Persistence Discovery Lifecycle & health Scaling Aggregation Security MORE THAN CONTAINERS…
14. DEVSECOPS End to End Security + + DEV QA OPS SECURITY
15. 4 • Are there known vulnerabilities in the application layer? • Are the runtime and OS layers up to date? • How frequently will the container be updated and how will I know when it’s updated? CONTENT: EACH LAYER MATTERS CONTAINER OS RUNTIME APPLICATION Are there known vulnerabilities 
 in each application layer? Are the runtime and OS layers 
 up to date? How frequently will the container be updated and how will I know when its updated? IS THE CONTAINER ENVIRONMENT SECURE? Is the image from a trusted source? Can I quickly deploy a security update at scale? Is my multi-tenant host secure?
16. Network isolation Storage API & Platform access Monitoring & Logging Federated clusters Registry Container host {} Builds CI/CD Images SECURING CONTAINERS
17. CONTAINER IMAGE SECURITY
18. 4 • Are there known vulnerabilities in the application layer? • Are the runtime and OS layers up to date? • How frequently will the container be updated and how will I know when it’s updated? CONTENT: EACH LAYER MATTERS CONTAINER OS RUNTIME APPLICATION CONTENT: EACH LAYER MATTERS AYER MATTERS CONTAINER OS RUNTIME APPLICATION JAR CONTAINER
19. SECURITY IMPLICATIONS What’s inside matters…
20. A CONVERGED SOFTWARE SUPPLY CHAIN
21. code config data Kubernetes configmaps secrets Container image Traditional 
 data services, Kubernetes 
 persistent volumes TREAT CONTAINERS AS IMMUTABLE
22. NODE MASTER Container Distributed Store Container ! Secure mechanism for holding sensitive data e.g. ◦ Passwords and credentials ◦ SSH Keys ◦ Certificates ! Secrets are made available as ◦ Environment variables ◦ Volume mounts ◦ Interaction with external systems ! Encrypted in transit ! Never rest on the nodes SECRETS
23. IMAGE SIGNING Validate what images and version are running
24. CONTAINER REGISTRY SECURITY
25. 64% of official images in Docker Hub 
 contain high priority security vulnerabilities examples: ShellShock (bash) Heartbleed (OpenSSL) Poodle (OpenSSL) Source: Over 30% of Official Images in Docker Hub Contain High Priority Security Vulnerabilities, Jayanth Gummaraju, Tarun Desikan, and Yoshio Turner, BanyanOps, May 2015 (http://www.banyanops.com/pdf/BanyanOps-AnalyzingDockerHub-WhitePaper.pdf) WHAT’S INSIDE THE CONTAINER MATTERS
26. PRIVATE REGISTRY
27. CI WITH CONTAINERS
28. CI/CD PIPELINE Continuous Integration Continuous Build Continuous Deployment Developer -> Source -> Git Git -> RPMS -> Images-> Registry Images from 
 Registry -> Clusters
29. CI/CD with Security
30. Security CONTINUOUS INTEGRATION WITH SECURITY SCAN
31. CUSTOM SUPPLY CHAIN
32. FROM fedora:latest CMD echo “Hello” Build file Build Best Practices • Treat as a Blueprint • Don’t login to build/configure • Version control build file • Be explicit with versions, not latest • Each Run creates a new layer BUILDS
33. Compliance and Vulnerability Audits with OpenSCAP
34. AUTOMATED SECURITY SCANNING with OpenSCAP Reports Scan SCAP Security Guide for RHEL CCE-27002-5 Set Password Minimum Length Content Scan physical servers, virtual machines, docker images and containers
 for Security Policy Compliance (CCEs) and known Security Vulnerabilities (CVEs)
35. Standard Docker Host Security Profile Java Runtime Environment (JRE) Upstream Firefox STIG RHEL OSP STIG Red Hat Corporate Profile for Certified Cloud Providers (RH CCP) STIG for Red Hat Enterprise Linux 6, 7 Server STIG for Red Hat Virtualization Hypervisor Common Profile for General-Purpose Debian Systems Common Profile for General-Purpose Fedora Systems Common Profile for General-Purpose Ubuntu Systems Payment Card Industry – Data Security Standard (PCI-DSS) v3 U.S. Government Commercial Cloud Services (C2S) CNSSI 1253 Low/Low/Low Control Baseline for Red Hat Enterprise Linux 7 Criminal Justice Information Services (CJIS) Security Policy Unclassified Information in Non-federal Information Systems and Organizations (NIST 800-171) U.S. Government Configuration Baseline (NIAP OSPP v4.0, USGCB, STIG) Security Policies in SCAP Security Guide (partial)
36. SECURITY POLICY REPORT
37. SECURITY POLICY REMEDIATION
38. SECURITY VULNERABILITY REPORT
39. RAPID INNOVATION & EXPERIMENTATION
40. ”only about 1/3 of ideas improve the metrics 
 they were designed to improve.”
 Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION
41. CONTINUOUS FEEDBACK LOOP
42. A/B TESTING USING CANARY DEPLOYMENTS
43. 50% 50% Version 1.2 Version 1 Route Version 1.2 25% Conversion Rate 30% Conversion Rate CANARY DEPLOYMENTS
44. CD WITH CONTAINERS
45. CONTINUOUS DEPLOYMENT WITH CONTAINERS
46. CD with Security
47. CONTINUOUS DELIVERY DEPLOYMENT STRATEGIES DEPLOYMENT STRATEGIES • Blue / Green deployment • Rolling updates • Canary deployments • A / B testing
48. Version 1 BLUE / GREEN DEPLOYMENT Route
49. Version 1 BLUE / GREEN DEPLOYMENT Version 1.2
50. Version 1 Tests / CI BLUE / GREEN DEPLOYMENT Version 1.2
51. Version 1 Version 1.2 BLUE / GREEN DEPLOYMENT Route Version 1.2
52. Version 1 BLUE / GREEN DEPLOYMENT Rollback Route Version 1.2
53. Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI ROLLING UPDATES with ZERO DOWNTIME
54. Deploy new version and wait until it’s ready… Version 1 Version 1 V1.2 Health Check: Readiness 
 Probe e.g. tcp, http, script V1
55. Each container/pod is updated one by one Version 1.2 50% Version 1 V1 V1.2
56. Each container/pod is updated one by one Version 1.2 Version 1.2 Version 1.2 100%
57. CONTAINER HOST SECURITY
58. Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container CLI SYSTEMD Cgroups Namespaces SELinux Drivers CONTAINERS ARE LINUX seccomp Read Only mounts
59. CGROUPS - RESOURCE ISOLATION
60. NAMESPACES - PROCESS ISOLATION
61. SELINUX - MANDATORY ACCESS CONTROLS Password Files Web Server Attacker Discretionary Access Controls 
 (file permissions) Mandatory Access Controls 
 (selinux) Internal Network Firewall Rules Password Files Firewall Rules Internal Network Web Server selinux policy
62. SECCOMP - DROPPING PRIVILEGES
63. READ ONLY MOUNTS
64. Best Practices • Don’t run as root • Limit SSH Access • Use namespaces • Define resource quotas • Enable logging • Apply Security Errata • Apply Security Context and seccomp filters http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html CONTAINER HOST SECURITY Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container SYSTEM Cgroup Namespace SELinu Driver seccom Read Only
65. Network isolation Storage API & Platform access Monitoring & Logging Federated clusters Registry Container host {} Builds CI/CD Images SECURING CONTAINERS
66. NETWORK ISOLATION
67. Network Namespace 
 provides resource isolation NETWORK ISOLATION Multi-Environment Multi-Tenant
68. NETWORK POLICY example: 
 all pods in namespace ‘project-a’ allow traffic 
 from any other pods in the same namespace.”
69. STORAGE SECURITY
70. Local Storage Quota Security Context Constraints STORAGE SECURITY
71. API & PLATFORM ACCESS
72. Authentication via OAuth tokens and SSL certificate Authorization via Policy Engine checks User/Group Defined Roles API & PLATFORM ACCESS
73. MONITORING & LOGGING
74. Aggregate platform and application log access via Kibana + Elasticsearch LOGGING
75. Historical CPU and Memory usage provided by Heapster, Hawkular, Cassandra MONITORING
76. FEDERATION
77. Amazon East OpenStack FEDERATED CLUSTERS Roles & access management (in-dev)
78. Deployment Frequency Lead Time Deployment
 Failure Rate Mean Time to Recover 99.999 Service Availability DEVSECOPS METRICS Compliance Score
79. THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin