Scale16x: A DevOps State of Mind: Continuous Security with Kubernetes

With the rise of DevOps, containers are at the brink of becoming a pervasive technology in Enterprise IT to accelerate application delivery for the business. When it comes to adopting containers in the enterprise, Security is the highest adoption barrier. Is your organization ready to address the security risks with containers for your DevOps environment?

In this presentation, you'll learn about:

- The top security risks with containers and how to manage theses risks at scale including Images, Builds, Registry, Deployment, Hosts, Network, Storage, APIs, Monitoring/Logging, Federation.

- How to make your Container enabled DevOps workflow more secure without slowing down your CI/CD pipeline

- Automating security vulnerability management and compliance checking for container images

Also, a demo of Kubernetes managing the container image lifecycle, automating container security scans, and deployment strategies for security updates at scale including Canary, Blue/Green deployments and A/B testing.

Transcript

1. A DEVOPS STATE OF MIND: Continuous Security with Kubernetes Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.com
2. “Only the Paranoid Survive” - Andy Grove, 1998
3. Retail Finance Media Transportation ? ? SOFTWARE DISRUPTS BUSINESS
4. INNOVATION & EXECUTION Shift to Executing at Scale in Dynamic Execution Innovation Innovation Execution
5. Innovation Months THE AVERAGE ENTERPRISE DOES 
 DEPLOYMENTS EVERY 6 TO 9 MONTHS.
6. DEV QA OPS Walled off people, walled off processes, walled off technologies “THROW IT OVER THE WALL”
7. HOW DOES I.T. TRANSFORM FROM A COST CENTER INTO AN INNOVATION CENTER? Months Innovation
8. Culture of experimentation A B 20% vs. 25% Empowered organization Time Change Rapid Innovation THE DISRUPTORS = AI /
 ML Data-driven intelligence Data, Data, Data
9. DEV QA OPS Open organization + 
 cross-functional teams Software factory automation Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Open Source CI/CD pipelines with feedback + + Culture Process Technology BREAKING DOWN THE WALLS WITH DEVOPS
10. THE VALUE OF DEVOPS Faster Time to Market Security More time To Innovate Faster Resolution Of Problems More Stable Operating Environment Improved Communication & Collaboration Less Complex Less Risk
11. DEV QA OPS SECURITY IS AN AFTERTHOUGHT | SECURITY | “Patch? The servers are behind the firewall.” - Anonymous (far too many to name), 2005 - …
12. http://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/
13. DEVSECOPS + + SECURITY DEV QA OPS Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Open Source Culture Process Technology
14. DEVSECOPS Continuous Security Improvement Process Optimization Security Automation Dev QA Prod Reduce Risks, Lower Costs, Speed Delivery, Speed Reaction
15. CONTAINERS
16. KEY TECHNOLOGY TRENDS
17. APPLICATION DELIVERY VIA CONTAINERS
18. docker.io Registry Private Registry FROM fedora:1.0 CMD echo “Hello” Build file Physical, Virtual, Cloud Image Container Build Run Ship CONTAINERS
19. CONTAINER SECURITY AT SCALE
20. Scheduling Monitoring Persistence Discovery Lifecycle & health Scaling Aggregation Security MORE THAN CONTAINERS…
21. DEVSECOPS End to End Security + + DEV QA OPS SECURITY
22. Web Database replicas=1, 
 role=db replicas=2, 
 role=web ORCHESTRATION Deployment, Declarative Nodes Controller Manager & Data Store (etcd)
23. role=web role=web ORCHESTRATION Schedule + Provision Pods (Compute/Storage/Network) Image Registry Pods Nodes Web replicas=2, 
 role=web ReplicaSet
24. role=web role=db role=web Pods Nodes Image Registry ORCHESTRATION Schedule + Provision Pods (Compute/Storage/Network) Web replicas=2, 
 role=web ReplicaSet Database replicas=1, 
 role=db StatefulSet
25. Web Database role=web role=db role=web replicas=1, 
 role=db replicas=2, 
 role=web ORCHESTRATION Service (Load Balancer) Pods Nodes Services Controller Manager & Data Store (etcd)
26. HEALTH CHECK Monitoring & Logging Pods Nodes Services Web Database role=web role=db role=web replicas=1, 
 role=db replicas=2, 
 role=web
27. HEALTH CHECK Pods Nodes Services Web Database role=web role=db role=web replicas=1, 
 role=db replicas=2, 
 role=web role=web Controller Manager & Data Store (etcd)
28. Web Database replicas=1, 
 role=db replicas=2, 
 role=web HEALTH CHECK Pods Nodes Services role=web role=db role=web Controller Manager & Data Store (etcd)
29. Web Database replicas=1, 
 role=db replicas=2, 
 role=web AUTO-SCALE Monitoring & Logging 80% CPU Pods Nodes Services role=web role=db role=web
30. Web Database replicas=1, 
 role=db replicas=3 
 role=web AUTO-SCALE 80% CPU Pods Nodes Services role=web role=db role=web role=web Controller Manager & Data Store (etcd)
31. Pods Nodes Services Web Database replicas=1, 
 role=db replicas=3 
 role=web AUTO-SCALE 50% CPU role=web role=db role=web role=web Controller Manager & Data Store (etcd)
32. Network isolation Storage API & Platform access Monitoring & Logging Federated clusters Registry Container host {} Builds CI/CD Images SECURING CONTAINERS
33. CONTAINER IMAGE SECURITY
34. 4 • Are there known vulnerabilities in the application layer? • Are the runtime and OS layers up to date? • How frequently will the container be updated and how will I know when it’s updated? CONTENT: EACH LAYER MATTERS CONTAINER OS RUNTIME APPLICATION CONTENT: EACH LAYER MATTERS AYER MATTERS CONTAINER OS RUNTIME APPLICATION JAR CONTAINER
35. SECURITY IMPLICATIONS What’s inside matters…
36. A CONVERGED SOFTWARE SUPPLY CHAIN
37. code config data Kubernetes configmaps secrets Container image Traditional 
 data services, Kubernetes 
 persistent volumes TREAT CONTAINERS AS IMMUTABLE
38. IMAGE SIGNING Validate what images and version are running
39. CONTAINER REGISTRY SECURITY
40. 64% of official images in Docker Hub 
 contain high priority security vulnerabilities examples: ShellShock (bash) Heartbleed (OpenSSL) Poodle (OpenSSL) Source: Over 30% of Official Images in Docker Hub Contain High Priority Security Vulnerabilities, Jayanth Gummaraju, Tarun Desikan, and Yoshio Turner, BanyanOps, May 2015 (http://www.banyanops.com/pdf/BanyanOps-AnalyzingDockerHub-WhitePaper.pdf) WHAT’S INSIDE THE CONTAINER MATTERS
41. PRIVATE REGISTRY
42. CI WITH CONTAINERS
43. CI/CD PIPELINE Continuous Integration Continuous Build Continuous Deployment Developer -> Source -> Git Git -> RPMS -> Images-> Registry Images from 
 Registry -> Clusters
44. CI/CD with Security
45. Security CONTINUOUS INTEGRATION WITH SECURITY SCAN
46. CUSTOM SUPPLY CHAIN
47. FROM fedora:latest CMD echo “Hello” Build file Build Best Practices • Treat as a Blueprint • Don’t login to build/configure • Version control build file • Be explicit with versions, not latest • Each Run creates a new layer BUILDS
48. Compliance and Vulnerability Audits with OpenSCAP
49. AUTOMATED SECURITY SCANNING with OpenSCAP Reports Scan SCAP Security Guide for RHEL CCE-27002-5 Set Password Minimum Length Content Scan physical servers, virtual machines, docker images and containers
 for Security Policy Compliance (CCEs) and known Security Vulnerabilities (CVEs)
50. Standard Docker Host Security Profile Java Runtime Environment (JRE) Upstream Firefox STIG RHEL OSP STIG Red Hat Corporate Profile for Certified Cloud Providers (RH CCP) STIG for Red Hat Enterprise Linux 6, 7 Server STIG for Red Hat Virtualization Hypervisor Common Profile for General-Purpose Debian Systems Common Profile for General-Purpose Fedora Systems Common Profile for General-Purpose Ubuntu Systems Payment Card Industry – Data Security Standard (PCI-DSS) v3 U.S. Government Commercial Cloud Services (C2S) CNSSI 1253 Low/Low/Low Control Baseline for Red Hat Enterprise Linux 7 Criminal Justice Information Services (CJIS) Security Policy Unclassified Information in Non-federal Information Systems and Organizations (NIST 800-171) U.S. Government Configuration Baseline (NIAP OSPP v4.0, USGCB, STIG) Security Policies in SCAP Security Guide (partial)
51. SECURITY POLICY REPORT
52. SECURITY POLICY REMEDIATION
53. SECURITY VULNERABILITY REPORT
54. CD WITH CONTAINERS
55. CONTINUOUS DEPLOYMENT WITH CONTAINERS BUILD ONCE, DEPLOY ANYWHERE
56. CD with Security
57. CONTINUOUS DELIVERY: DEPLOYMENT STRATEGIES
58. CONTINUOUS DELIVERY DEPLOYMENT STRATEGIES DEPLOYMENT STRATEGIES • Recreate • Rolling updates • Blue / Green deployment • Canary deployments • A / B testing
59. Recreate
60. Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME
61. Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME
62. Version 1.2 Version 1.2 Version 1.2 RECREATE WITH DOWNTIME Use Case • Non-mission critical services Cons • Downtime Pros • Simple, clean • No Schema incompatibilities • No API versioning
63. Rolling Updates
64. Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI ROLLING UPDATES with ZERO DOWNTIME
65. Deploy new version and wait until it’s ready… Version 1 Version 1 V1.2 Health Check: readiness probe e.g. tcp, http, script V1
66. Each container/pod is updated one by one Version 1.2 50% Version 1 V1 V1.2
67. Each container/pod is updated one by one Version 1.2 Version 1.2 Version 1.2 100% Use Case • Horizontally scaled • Backward compatible API/data • Microservices Cons • Require backward compatible APIs/data • Resource overhead Pros • Zero downtime • Reduced risk, gradual rollout w/health checks • Ready for rollback
68. Blue / Green Deployment
69. Version 1 BLUE / GREEN DEPLOYMENT Route BLUE
70. Version 1 BLUE / GREEN DEPLOYMENT Version 1.2 BLUE GREEN
71. Version 1 Tests / CI BLUE / GREEN DEPLOYMENT Version 1.2 BLUE GREEN
72. Version 1 Version 1.2 BLUE / GREEN DEPLOYMENT Route Version 1.2 BLUE GREEN
73. Version 1 BLUE / GREEN DEPLOYMENT Rollback Route Version 1.2 BLUE GREEN Use Case • Self-contained micro services (data) Cons • Resource overhead • Data synchronization Pros • Low risk, never change production • No downtime • Production like testing • Rollback
74. RAPID INNOVATION & EXPERIMENTATION
75. ”only about 1/3 of ideas improve the metrics 
 they were designed to improve.”
 Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION
76. CONTINUOUS FEEDBACK LOOP
77. A/B TESTING USING CANARY DEPLOYMENTS
78. Version 1.2 Version 1 100% Tests / CI Version 1.2 Route 25% Conversion Rate ?! Conversion Rate CANARY DEPLOYMENTS
79. 50% 50% Version 1.2 Version 1 Route Version 1.2 25% Conversion Rate 30% Conversion Rate CANARY DEPLOYMENTS
80. 25% Conversion Rate 100% Version 1 Version 1.2 Route Version 1.2 30% Conversion Rate CANARY DEPLOYMENTS
81. Version 1.2 Version 1 100% Route Rollback 25% Conversion Rate 20% Conversion Rate CANARY DEPLOYMENTS
82. CONTAINER HOST SECURITY
83. Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container CLI SYSTEMD Cgroups Namespaces SELinux Drivers CONTAINERS ARE LINUX seccomp Read Only mounts
84. CGROUPS - RESOURCE ISOLATION
85. NAMESPACES - PROCESS ISOLATION
86. SELINUX - MANDATORY ACCESS CONTROLS Password Files Web Server Attacker Discretionary Access Controls 
 (file permissions) Mandatory Access Controls 
 (selinux) Internal Network Firewall Rules Password Files Firewall Rules Internal Network Web Server selinux policy
87. SECCOMP - DROPPING PRIVILEGES
88. READ ONLY MOUNTS
89. Best Practices • Don’t run as root • Limit SSH Access • Use namespaces • Define resource quotas • Enable logging • Apply Security Errata • Apply Security Context and seccomp filters http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html CONTAINER HOST SECURITY Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container SYSTEM Cgroup Namespace SELinu Driver seccom Read Only
90. Network isolation Storage API & Platform access Monitoring & Logging Federated clusters Registry Container host {} Builds CI/CD Images SECURING CONTAINERS
91. NETWORK ISOLATION
92. Network Namespace 
 provides resource isolation NETWORK ISOLATION Multi-Environment Multi-Tenant
93. NETWORK POLICY example: 
 all pods in namespace ‘project-a’ allow traffic 
 from any other pods in the same namespace.”
94. STORAGE SECURITY
95. Local Storage Quota Security Context Constraints STORAGE SECURITY
96. API & PLATFORM ACCESS
97. Authentication via OAuth tokens and SSL certificate Authorization via Policy Engine checks User/Group Defined Roles API & PLATFORM ACCESS
98. MONITORING & LOGGING
99. Aggregate platform and application log access via Kibana + Elasticsearch LOGGING
100. Historical CPU and Memory usage provided by Heapster, Hawkular, Cassandra MONITORING
101. FEDERATION
102. Amazon East OpenStack FEDERATED CLUSTERS Roles & access management (in-dev)
103. Deployment Frequency Lead Time Deployment
 Failure Rate Mean Time to Recover 99.999 Service Availability DEVSECOPS METRICS Compliance Score
104. THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin