A DevOps State of Mind: Continuous Security with Kubernetes

Transcript

1. A DevOps State of Mind: Continuous Security with Kubernetes Chris Van Tuin Red Hat Chief Technologist, NA West / Silicon Valley linkedin: Chris Van Tuin
 email: cvantuin@redhat.com
 twitter: @chrisvantuin
2. “Only the paranoid survive” - Andy Grove, 1996
3. BUSINESSES MUST KEEP PACE The Disrupters are Disrupting the Incumbents Static &
 Planned Dynamic & 
 Policy Driven Execution Innovation Old New Execution Innovation
4. THE DISRUPTERS = Empowered organization Speed Up 
 Innovation Time Change Move Fast, Break Things Culture of experimentation A 20% vs. 25% Shorten the Feedback Loop Real-time data-driven intelligence & personalization AI /
 ML Data, Data, Data B
5. THE WORLD IS AUTOMATING Those who succeed in automation will win
6. I.T. MUST EVOLVE 
 FROM A COST CENTER TO INNOVATION CENTER Development Model Application Architecture Deployment & Application Infrastructur Storage Waterfall Agile Monolithic N-tier Bare Metal Virtual Servers Data Center Hosted Scale Up Scale Out DevOps MicroServices Containers Hybrid Cloud Storage as a Service
7. Applications & devices outside of IT control Cloud computing Software-defined infrastructure Dissolving security perimeter Menacing threat landscape TRADITIONAL NETWORK-BASED DEFENSES ARE NO LONGER ENOUGH SECURING THE ENTERPRISE IS HARDER THAN EVER The way we develop, deploy and manage IT is changing dramatically led by DevOps, Cloud Native Applications, and Hybrid Cloud
8. DEVSECOPS Continuous Security Improvement Process Optimization Security Automation Dev QA Prod Reduce Risks, Lower Costs, Speed Delivery, Speed Reaction
9. DEVSECOPS + + Security DEV QA OPS Culture Process Technology Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Cloud Native Applications Hybrid Cloud Open Source
10. docker.io Registry Private Registry FROM fedora:1.0 CMD echo “Hello” Build file Physical, Virtual, Cloud Container Image Container Instance Build Run Ship CONTAINERS ENABLE DEVSECOPS
11. Scheduling Monitoring Persistence Discovery Lifecycle & health Scaling Aggregation Security CONTAINERS AT SCALE BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD
12. BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory
 Speed, Resiliency, Scalability, Security 

13. AUTOMATION
14. Web Database role=web role=db role=web replicas=1, 
 role=db replicas=2, 
 role=web ORCHESTRATION Deployment, Declarative Pods Nodes Services Controller Manager & Data Store (etcd)
15. Web Database replicas=1, 
 role=db replicas=2, 
 role=web HEALTH CHECK Pods Nodes Services role=web role=db role=web Controller Manager & Data Store (etcd)
16. Pods Nodes Services Web Database replicas=1, 
 role=db replicas=3 
 role=web AUTO-SCALE 50% CPU role=web role=db role=web role=web Controller Manager & Data Store (etcd)
17. Network isolation API & Platform access Federated clusters Storage {} CI/CD Monitoring & Logging Builds Images SECURING YOUR CONTAINER ENVIRONMENT Container host Registry
18. CONTAINER IMAGES
19. LAPTOP Container Application OS dependencies Guest VM LINUX BARE METAL Container Application OS dependencies LINUX VIRTUALIZATION Container Application OS dependencies Virtual Machine LINUX PRIVATE CLOUD Container Application OS dependencies Virtual Machine LINUX PUBLIC CLOUD Container Application OS dependencies Virtual Machine LINUX CONTAINERS - Build Once, Deploy Anywhere Reducing Risk and Improving Security with Improved Consistency
20. CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS dependencies 1.2/latest 1.1
21. Config Data Kubernetes configmaps secrets Container image Traditional 
 data services, Kubernetes 
 persistent volumes TREAT CONTAINERS AS IMMUTABLE To keep containerized apps portable Application Language runtimes OS dependencies
22. KUBERNETES CONFIGMAP Decouple configuration from container image Application Language runtimes OS dependencies Environment Variable or Volume/File CONTAINER INSTANCE key:value from directories, files, or values KUBERNETES
 CONFIGMAP APPLICATION CONFIG FILE Application Configuration File e.g. XML etcd Pod Source Code Repository EnvVar require pod restart Files refresh in time
23. •Authenticating authorship •Non-repudiation •Ensuring image integrity CONTAINER IMAGE SIGNING Validate what images and version are running
24. CONTAINER BUILDS
25. A CONVERGED SOFTWARE 
 SUPPLY CHAIN
26. CUSTOM SUPPLY CHAIN
27. • Treat build file as a Blueprint • Version control build file • Don’t login to build/configure • Be explicit with versions, not latest • Always list registry pulling FROM • Specify USER, default is root • Each Run creates a new layer BUILD FILE BEST PRACTICES FROM registry.redhat.com/rhel7 RUN groupadd -g 999 appuser && \ useradd -r -u 999 -g appuser appuser USER appuser CMD echo “Hello” Build file
28. CONTAINER REGISTRY SECURITY
29. 64% of official images in Docker Hub 
 contain high priority security vulnerabilities examples: ShellShock (bash) Heartbleed (OpenSSL) Poodle (OpenSSL) Source: Over 30% of Official Images in Docker Hub Contain High Priority Security Vulnerabilities, Jayanth Gummaraju, Tarun Desikan, and Yoshio Turner, BanyanOps, May 2015 (http://www.banyanops.com/pdf/BanyanOps-AnalyzingDockerHub-WhitePaper.pdf) WHAT’S INSIDE THE CONTAINER MATTERS
30. PRIVATE REGISTRY
31. CONTAINER HOST SECURITY
32. Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container CLI SYSTEMD Cgroups Namespaces SELinux Drivers CONTAINERS ARE LINUX seccomp Read Only mounts
33. CGROUPS - RESOURCE ISOLATION
34. NAMESPACES - PROCESS ISOLATION
35. SELINUX - MANDATORY ACCESS CONTROLS Password Files Web Server Attacker Discretionary Access Controls 
 (file permissions) Mandatory Access Controls 
 (selinux) Internal Network Firewall Rules Password Files Firewall Rules Internal Network Web Server selinux policy
36. SECCOMP AND LINUX CAPABILITIES
 FILTERING SYSTEM CALLS and DROPPING PRIVILEGES
37. READ ONLY MOUNTS
38. Chris Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.co Best Practices • Don’t run as root • If you must, 
 limit Linux Capabilities • Limit SSH Access • Use namespaces • Define resource quotas • Enable logging • Apply Security Errata • Apply Security Context and seccomp filters • Run production 
 unprivileged containers 
 as read-only http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container CLI SYSTEMD Cgroups Namespaces SELinux Drivers seccomp Read Only mounts Capabilities CONTAINER HOST SECURITY
39. CONTINUOUS INTEGRATION WITH CONTAINERS
40. CI/CD PIPELINE Continuous Integration Continuous Build Continuous Deployment Developer -> Source -> Git Git -> RPMS -> Images-> Registry Images from 
 Registry -> Clusters
41. WHAT’S INSIDE MATTERS…
42. Security CONTINUOUS INTEGRATION WITH SECURITY SCAN
43. Java Build Environment Language runtimes OS dependencies Build Image Java Code Application Language runtimes OS dependencies Container Image Image Registry Source Repository Image Registry REPRODUCIBLE BUILDS Source to Image with Build Images Source v3.1 v1.0.1 v3.1
44. DEPLOYMENT STRATEGIES
45. CONTINUOUS DELIVERY WITH CONTAINERS
46. CONTINUOUS DELIVERY DEPLOYMENT STRATEGIES DEPLOYMENT STRATEGIES • Recreate • Rolling updates • Blue / Green deployment • Canary with A/B testing • Database migrations
47. Recreate
48. Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME
49. Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME
50. Version 1.2 Version 1.2 Version 1.2 RECREATE WITH DOWNTIME Use Case • Non-mission critical services Pros • Simple, clean • No Schema incompatibilities • No API versioning Cons • Downtime
51. Rolling Updates
52. Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI ROLLING UPDATES with ZERO DOWNTIME Rollingupdate
 maxUnavailable=0 maxSurge=1
53. Deploy new version and wait until it’s ready… Health Check: readiness probe e.g. tcp, http, script Version 1 Version 1 Version 
 1.2 Version 1 Rollingupdate
 maxUnavailable=0 maxSurge=1
54. Each container/pod is updated one by one Version 1.2 50% Version 1 V1 V1.2
55. Each container/pod is updated one by one Version 1.2 Version 1.2 Version 1.2 100% Use Case • Horizontally scaled • Backward compatible API/data • Microservices Pros • Zero downtime • Reduced risk, gradual rollout w/health checks • Ready for rollback Cons • Require backward compatible APIs/data • Resource overhead
56. Blue / Green Deployment
57. BLUE Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100%
58. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% Health Check: readiness probe e.g. tcp, http, script
59. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100%
60. BLUE / GREEN DEPLOYMENT Rollback BLUE GREEN Version 1 Version 2 Ingress Use Case • Self-contained micro services (data) Pros • Low risk, never change production • No downtime • Production like testing • Rollback Cons • Resource overhead • Data synchronization
61. RAPID INNOVATION & EXPERIMENTATION WITH A/B TESTING
62. ”only about 1/3 of ideas improve the metrics 
 they were designed to improve.”
 Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION
63. A/B TESTING USING CANARY DEPLOYMENTS
64. 25% Conversion Rate ?! Conversion Rate 100% Version B Version A Ingress CANARY DEPLOYMENTS Tests / CI
65. 25% Conversion Rate 30% Conversion Rate 75% 25% Version B Version A Ingress CANARY DEPLOYMENTS
66. 25% Conversion Rate 30% Conversion Rate 100% Version B Version A Ingress CANARY DEPLOYMENTS
67. 25% Conversion Rate 20% Conversion Rate 100% Version B Version A Rollback Ingress CANARY DEPLOYMENTS
68. DATABASE MIGRATIONS
69. Application v3 Development Application V2 Test Application v1 Production DB v1 DB v2 DB v3 CI/CD PIPELINE Version control database updates, ex: flyway V3__add_table_scooter.sql V2__add_table_truck.sql V1__add_table_car.sql
70. DATABASE MIGRATIONS Version control database updates with Containers CONTAINER IMAGE CONTAINER BUILD FILE SQL MIGRATION SCRIPT Source Code Repository V2__add_table.sql Source Code Repository V2__add_table.sql /var/flyway/data Flyway flyway-mydb:v2.0.0 Registry + Dockerfile
71. Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresq pvcl DATABASE MIGRATION StatefulSet deployment with headless Service v1
72. Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Create a Job for Flyway Flyway Job Secrets = Database Connection Info v1 flyway-mydb:v2.0.0 Image Registry Flyway
73. role=postgressql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql pvc DATABASE MIGRATIONS Apply schema changes to database Flyway Job Secrets = Database Connection Info V2 flyway-mydb:v2.0.0 Flyway
74. role=postgresql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Version control for database with Kubernetes V2
75. Network isolation API & Platform access Federated clusters Storage {} CI/CD Monitoring & Logging Images Builds Container host Registry SECURING YOUR CONTAINER ENVIRONMENT
76. NETWORK SECURITY
77. Network Namespace 
 provides resource isolation NETWORK ISOLATION Multi-Environment Multi-Tenant
78. NETWORK POLICY example: 
 all pods in namespace ‘project-a’ allow traffic 
 from any other pods in the same namespace.”
79. Kubernetes 
 Logical Network Model NETWORK SECURITY • Kubernetes uses a flat SDN model • All pods get IP from same CIDR • And live on same logical network • Assumes all nodes communicate
 Traditional 
 Physical Network Model • Each layer represents a Zone with
 increased trust - DMZ > App > DB,
 interzone flow generally one direction • Intrazone traffic generally unrestricted
80. NETWORK SECURITY MODELS Co-Existence Approaches One Cluster Multiple Zones Kubernete Cluster Physical Compute 
 isolation based on 
 Network Zones Kubernete Cluster One Cluster Per Zone Kubernete Cluster B Kubernete Cluster A Kubernetes Cluster B C D https://blog.openshift.com/openshift-and-network-security-zones-coexistence-approaches/
81. MONITORING & LOGGING
82. CONTINUOUS FEEDBACK LOOP
83. KUBERNETES MONITORING CONSIDERATIONS Kubernetes* Container* Host Cluster services, services, pods, 
 deployments metrics Container native metrics Traditional resource metrics - cpu, memory, network, storage prometheus + grafana kubernetes-state-metrics probes Stack Metrics Tool node-exporter kubelet:cAdvisor Application Distributed applications - traditional app metrics - service discovery - distributed tracing prometheus + grafana jaeger tracing istio
84. Aggregate platform and application log access via Kibana + Elasticsearch LOGGING
85. STORAGE SECURITY
86. Local Storage Quota Security Context Constraints STORAGE SECURITY Sometimes we can also have storage isolation requirements: 
 pods in a network zone must use different storage endpoints 
 than pods in other network zones. We can create one storage class per storage endpoint and 
 then control which storage class(es) a project can use
87. API & PLATFORM ACCESS
88. Authentication via OAuth tokens and SSL certificate Authorization via Policy Engine checks User/Group Defined Roles API & PLATFORM ACCESS
89. FEDERATION
90. Amazon East OpenStack FEDERATED CLUSTERS Roles & access management (in-dev)
91. WHAT’S NEXT
92. Traffic Control Service Resiliency Chaos Testing Observ- ability Security
93. OPERATORS
94. Deployment Frequency Lead Time Deployment
 Failure Rate Mean Time to Recover 99.999 Service Availability DEVSECOPS METRICS Compliance Score
95. THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin