Draft Red Hat Security Symposium @ San Jose: A DevSecOps State of Mind: Continuous Security with Kuberenetes Edit deck

Transcript

1. A DevSecOps State of Mind: Continuous Security with Kubernetes Do not use: Make a copy Chris Van Tuin Chief Technologist, West @chrisvantuin cvantuin@redhat.com
2. ENABLING INNOVATION
3. “Only the paranoid survive” - Andy Grove, 1996
4. THE WORLD IS AUTOMATING Those who succeed in automation will win
5. NOT JUST A WEBSITE, THE DISRUPTERS = Empowered organization Speed Up 
 Innovation Time Change Move Fast, Break Things Culture of experimentation A 20% vs. 25% Shorten the Feedback Loop Real-time data-driven intelligence & personalization AI /
 ML Data, Data, Data B
6. ENABLING INNOVATION, WHILE EXECUTING AT SCALE Static &
 Planned Dynamic & 
 Policy Driven Execution Innovation Innovation Execution Old New
7. I.T. MUST EVOLVE
8. HOW DOES I.T. ACCELERATE 
 BUSINESS INNOVATION? 6 to 9 months Innovation Hours to Weeks
9. THE STRATEGIC DIFFERENTIATOR The Fab
 Powered by Automation “copy exactly” The Software Factory Powered by Automation
10. I.T. MUST EVOLVE FROM A COST CENTER 
 TO INNOVATION CENTER Development Model Application Architecture Deployment & Packaging Application Infrastructur e Storage Waterfall Agile Monolithic N-tier Bare Metal Virtual Servers Data Center Hosted Scale Up Scale Out DevOps MicroServices Containers Hybrid Cloud Storage as a Service
11. I.T. MUST TRANSFORM FROM A COST CENTER 
 INTO AN INNOVATION CENTER Powered by DevOps + Automation + + DEV QA OPS Culture Process 
 Automation Technology Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Cloud Native Applications Hybrid Cloud Open Source Agile, Iterative, Continuous, Infrastructure as Code Collaborative Transparent Open THE SOFTWARE FACTORY
12. CONTINUOUS SECURITY
13. DEV QA OPS SECURITY IS AN AFTERTHOUGHT | SECURITY | “Patch? The servers are behind the firewall.” - Anonymous (far too many to name), 2005 - …
14. BARE METAL VIRTUAL PRIVATE CLOUD OFF-PREMISE ON-PREMISE PUBLIC CLOUD DATA DATA DISTRIBUTED APPLICATIONS
15. ANY COMBINATION, WHETHER TRADITIONAL OR CONTAINERIZED LEGACY APPS (1,000+) BARE METAL PRIVATE CLOUD PUBLIC CLOUD VIRTUAL PRODUCTION DEV/TEST HYBRID CLOUD ENVIRONMENTS
16. MULTI-TENANCY
17. DEVSECOPS + + End to End Security DEV QA OPS Culture Process Technology Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Open Source
18. DEVSECOPS Continuous Security Improvement Process Optimization Security Automation Dev QA Prod Reduce Risks, Lower Costs, Speed Delivery, Speed Reaction
19. CONTAINERS AT SCALE
20. LAPTOP Container Application OS dependencies Guest VM LINUX BARE METAL Container Application OS dependencies LINUX VIRTUALIZATION Container Application OS dependencies Virtual Machine LINUX PRIVATE CLOUD Container Application OS dependencies Virtual Machine LINUX PUBLIC CLOUD Container Application OS dependencies Virtual Machine LINUX CONTAINERIZED MICROSERVICES
 Build Once, Deploy Anywhere
21. docker.io Registry Private Registry FROM fedora:1.0 CMD echo “Hello” Build file Physical, Virtual, Cloud Container Image Container Instance Build Run Ship CONTAINERS ENABLE DEVOPS
22. Image Format Distribution Spec Runtime Spec
23. Scheduling Monitoring Persistence Discovery Lifecycle & health Scaling Aggregation Security MORE THAN CONTAINERS…
24. BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD Automated Software Factory
 Speed, Resiliency, Scalability, Security 

25. Databases Images Automation MANAGING CONTAINERIZED MICROSERVICES
 WITH KUBERNETES A/B Testing Migrations External
 Services Deployment Strategies Security What’s Next… CI/CD Scanning ENABLING DEVSECOPS WITH KUBERNETES External 
 Services Databases Migrations Infrastructure Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100%
26. KUBERNETES AUTOMATION
27. Web Application replicas: 1, 
 role: app image: myapp:1.0 replicas: 2, 
 role: web image: httpd:1.7.9 ORCHESTRATION Declarative, Deployment Controller Manager & Data Store (etcd)
28. Web Application ORCHESTRATION Declarative, Deployment Nodes Controller Manager & Data Store (etcd) Physical, VM, 
 Cloud Instances replicas: 2, 
 role: web image: httpd:1.7.9 replicas: 1, 
 role: app image: myapp:1.0
29. role: app role: web role: web Pods Nodes Image Registry ORCHESTRATION Schedule + Provision Pods (Compute/Storage/Network) Web Application replicas: 2, 
 role: web image: httpd:1.7.9 replicas: 1, 
 role: app image: myapp:1.0
30. Web Application role: web role: app role: web replicas: 1, 
 selector role: app replicas: 2, 
 selector role: web ORCHESTRATION Services (Load Balancer), Service discovery with selectors and pod labels Pods Nodes Services Controller Manager & Data Store (etcd)
31. Web Application ORCHESTRATION Service (Load Balancer) Pods Nodes Controller Manager & Data Store (etcd) Ingress / Routes role: web role: app role: web replicas: 1, 
 role: app replicas: 2, 
 role: web Services
32. HEALTH CHECK Monitoring & Logging Pods Nodes Services Web Application role: web role: app role: web Ingress / Routes Health Check replicas: 1, 
 role: app replicas: 2, 
 role: web
33. Pods Nodes Services Web Application role: web role: app role: web replicas: 1, 
 role: app replicas: 2, 
 role: web role: web Controller Manager & Data Store (etcd) HEALTH CHECK Readiness Probe e.g. tcp, http, script Ingress / Routes
34. Web Application replicas: 1, 
 role: app replicas: 2, 
 role: web Pods Nodes Services role: web role: app role: web Controller Manager & Data Store (etcd) HEALTH CHECK Ingress / Routes
35. Web Application AUTO-SCALE Monitoring & Logging 80% CPU Pods Nodes Services role: web role: app role: web Ingress / Routes replicas: 1, 
 role: app replicas: 2, 
 role: web
36. Web Application 80% CPU Pods Nodes Services role: web role: app role: web Controller Manager & Data Store (etcd) role: app AUTO-SCALE Ingress / Routes replicas: 2 
 role: app replicas: 2, 
 role: web
37. Pods Nodes Services Web Application 50% CPU role: web role: app role: app role: web Controller Manager & Data Store (etcd) AUTO-SCALE Ingress / Routes replicas: 2, 
 role: web replicas: 2, 
 role: app
38. CONTAINER IMAGES
39. CONTAINER IMAGE JAR CONTAINER IMAGE Application Application Language runtimes OS dependencies 1.2/latest 1.1
40. Config Data Kubernetes configmaps secrets Container image Traditional 
 data services, Kubernetes 
 persistent volumes TREAT CONTAINERS AS IMMUTABLE To keep containerized apps portable Application Language runtimes OS dependencies
41. KUBERNETES CONFIGMAP Decouple configuration from container image Application Language runtimes OS dependencies Environment Variable or Volume/File CONTAINER INSTANCE key:value from directories, files, or values KUBERNETES
 CONFIGMAP APPLICATION CONFIG FILE Application Configuration File e.g. XML etcd Pod Source Code Repository EnvVar require pod restart Files refresh in time
42. CONTINUOUS BUILDS
43. A CONVERGED SOFTWARE 
 SUPPLY CHAIN
44. CI/CD PIPELINE WITH KUBERNETES BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD
45. CUSTOM SUPPLY CHAIN CASCADING REBUILDS
46. Java Build Environment Language runtimes OS dependencies Build Image Java Code Application Language runtimes OS dependencies Container Image Image Registry Source Repository Image Registry REPRODUCIBLE BUILDS Source to Image with Build Images Source v3.1 v1.0.1 v3.1
47. CONTAINER SCANNING
48. WHAT’S INSIDE MATTERS…
49. PRIVATE REGISTRY
50. Security CONTINUOUS INTEGRATION WITH SECURITY SCAN
51. AUTOMATED SECURITY SCANNING with OpenSCAP Reports & Remediation Scan SCAP Security Guide for RHEL CCE-27002-5 Set Password Minimum Length Content Scan physical servers, virtual machines, docker images and containers
 for Security Policy Compliance (CCEs) and known Security Vulnerabilities (CVEs)
52. DEPLOYMENT STRATEGIES
53. CONTINUOUS DELIVERY WITH CONTAINERS CI/CD - CONTAINER UPDATES
54. CI/CD DEPLOYMENT STRATEGIES
 Automate and reduce deployment risk DEPLOYMENT STRATEGIES • Recreate • Rolling updates • Blue / Green deployment • Canary with A/B testing
55. Recreate
56. Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME RECREATE WITH DOWNTIME
 Using Recreate deployment strategy Kubernetes
 Service
57. Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI RECREATE WITH DOWNTIME RECREATE WITH DOWNTIME
 Shutdown existing deployment Kubernetes
 Service
58. Version 1.2 Version 1.2 Version 1.2 RECREATE WITH DOWNTIME Use Case • Non-mission critical services Pros • Simple, clean • No Schema incompatibilities • No API versioning Cons • Downtime RECREATE WITH DOWNTIME
 Shutdown existing deployment Kubernetes
 Service
59. Rolling Updates
60. Version 1 Version 1 Version 1 Version 1.2 ` Tests / CI ROLLING UPDATES with ZERO DOWNTIME Rollingupdate
 maxUnavailable=0 maxSurge=1 ROLLING UPDATES
 Replace each pod using RollingUpdate deployment strategy Kubernetes
 Service
61. Deploy new version and wait until it’s ready… Health Check: readiness probe e.g. tcp, http, script Version 1 Version 1 Version 
 1.2 Version 1 Rollingupdate
 maxUnavailable=0 maxSurge=1 ROLLING UPDATES
 Deploy new version, wait until it’s ready Kubernetes
 Service
62. Each container/pod is updated one by one Version 1.2 50% Version 1 V1 V1.2 ROLLING UPDATES
 Requires backward compatibility, as two versions run side-by-side Kubernetes
 Service
63. Each container/pod is updated one by one Version 1.2 Version 1.2 Version 1.2 100% Use Case • Horizontally scaled • Backward compatible API/data • Microservices Pros • Zero downtime • Reduced risk, gradual rollout w/health checks • Ready for rollback Cons • Require backward compatible APIs/data • Resource overhead ROLLING UPDATES Kubernetes
 Service
64. Blue / Green Deployment
65. BLUE Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Single service, run two complete Deployments BLUE Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% Service
 selector:
 production=BLUE Kubernetes
 Deployment
66. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% Health Check: readiness probe e.g. tcp, http, script BLUE / GREEN DEPLOYMENT
 Using Deployments, Ingress Service
 selector:
 production=BLUE Kubernetes
 Deployment Kubernetes
 Deployment
67. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Route all new request to Green, Blue sessions Service
 selector:
 version=GREEN
68. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Using Deployments, Ingress Service
 selector:
 production=GREEN
69. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Scale-down, reduce resources Service
 selector:
 production=GREEN
70. BLUE GREEN Version 1 Version 2 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100% BLUE / GREEN DEPLOYMENT
 Hot Backup Service
 selector:
 production=GREEN Version 2
71. BLUE / GREEN DEPLOYMENT Rollback BLUE GREEN Version 1 Version 2 Ingress Use Case • Self-contained micro services (data) Pros • Low risk, never change production • No downtime • Production like testing • Rollback Cons • Resource overhead • Data synchronization BLUE / GREEN DEPLOYMENT
 Rollback Service
 selector:
 production=BLUE
72. RAPID INNOVATION & EXPERIMENTATION WITH A/B TESTING
73. ”only about 1/3 of ideas improve the metrics 
 they were designed to improve.”
 Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION
74. A/B TESTING USING CANARY DEPLOYMENTS
75. 25% Conversion Rate ?! Conversion Rate 100% Version B Version A Ingress CANARY DEPLOYMENTS Tests / CI CANARY DEPLOYMENTS
 Build confidence in new version Service
 selector:
 app=demo version=A label:
 app=demo
 version=A 25% Conversion Rate ??% Conversion Rate
76. 25% Conversion Rate 30% Conversion Rate 75% 25% Version B Version A Ingress CANARY DEPLOYMENTS CANARY DEPLOYMENTS
 Requires app to support side-by-side version Service Service
 selector:
 app=demo label:
 app=demo
 version=A 25% Conversion Rate % Conversion Rate label:
 app=demo
 version=B
77. 25% Conversion Rate 30% Conversion Rate 100% Version B Version A Ingress CANARY DEPLOYMENTS Service
 selector:
 app=demo version=B label:
 app=demo
 version=A 25% Conversion Rate 30% Conversion Rate label:
 app=demo
 version=B
78. Databases Images Automation MANAGING CONTAINERIZED MICROSERVICES
 WITH KUBERNETES A/B Testing Migrations External
 Services Deployment Strategies Security What’s Next… CI/CD Scanning ENABLING DEVSECOPS WITH KUBERNETES External 
 Services Databases Migrations Infrastructure Version 1 Ingress e.g haproxy BLUE / GREEN DEPLOYMENT Using Ingress 100%
79. EXTERNAL SERVICES
80. EXTERNAL SERVICES Database outside cluster with IP address External Mongo Database Service External Mongo Database Service Development Production IP=10.200.0.2 port=27017 IP=10.100.0.9 port=27017
81. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017
82. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.200.0.2 port=27017 Database kind=Service type=ClusterIP name=mongo port=27017 targetport=27017
83. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.200.0.2 port=27017 Connect with mongodb://mongo Database kind=Service type=ClusterIP name=mongo port=27017 targetport=27017 kind=Endpoints name=mongo ip=10.200.0.2 port=27017
84. EXTERNAL SERVICES Database outside cluster with IP address Pods Nodes Services WebApp role=webapp replicas=2, 
 role=webapp External Mongo Database Service IP=10.200.0.2 port=27017 Network External Mongo Database Service IP=10.100.0.9 port=27017 Database name=mongo port=27017 targetport=27017 Endpoint IP=10.100.0.9 port=27017 kind=Service type=ClusterIP name=mongo port=27017 targetport=27017 kind=Endpoints name=mongo ip=10.200.0.9 port=27017 Connect with mongodb://mongo Database
85. Pods Nodes Services Database name: mongo type: ExternalName externalName: mongo52101.domain,.name EXTERNAL SERVICES Using CNAME redirection mongodb://
 <dbuser>:
 <dbpassword>
 @mongo:<port>/dev 
 mongodb://<dbuser>:<dbpassword>
 @mongo52101.domain.name:52101/dev Cloud Mongo Database Service WebApp role=webapp replicas=2, 
 role=webapp .name EXTERNAL SERVICE Connecting to Service with dynamic URI with a static ExternalName Kubernetes service
86. DATABASES
87. PERSISTENT VOLUMES Host Container Host Container Host Container Data in Container Data lost when Container terminates Data lost when Host terminates Independent of Container & Host Data in a Host Volume Networked Volume Data lost when Cloud instance terminates Data lost when Container terminates Independent of 
 Container & 
 Cloud instance DATA PERSISTENCE
88. 1. Maintains a sticky network ID/name across restarts
 e.g. mongo-0, mongo-1, mongo-2 2. Ordered Operations with ordinal index 
 e.g. name-0, name-1, name-2 3. Stable, persistent storage (linked to ordinal index/name) 4. Mandatory headless service (no single IP) for integrations KUBERNETES
 STATEFULSETS
89. role=mongo type=leader Nodes Pods Services Mongo StatefulSet replicas=2 role=mongo Client mongo-0 D A B C C DATABASE STATEFUL SETS StatefulSet with 2 replicas , headless service, direct access to pods pvc Read / Write Persistent Volume
90. DATABASE STATEFUL SETS role=mongo type=leader role=mongo type=follower Nodes Pods Services Client Mongo-0 Mongo-1 D A B C C Mongo StatefulSet replicas=2 role=mongo pvc pvc Read / Write Read / Only Persistent Volume
91. role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C C D Mongo StatefulSet replicas=3 role=mongo Read / Write Read / Only Read / Only DATABASE STATEFUL SETS Scale to 3 replicas Client
92. role=mongo type=leader role=mongo type=follower role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 Mongo-2 pvc pvc pvc Persistent Volume A B C D Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Unresponsive Pod Client
93. role=mongo type=leader role=mongo type=follower Nodes Pods Services Mongo-0 Mongo-1 pvc pvc Persistent Volume A B D role=mongo type=follower Mongo-2 pvc C Mongo StatefulSet replicas=3 role=mongo DATABASE STATEFUL SETS Auto recovery Client
94. DATABASE MIGRATIONS
95. Application v3 Development Application V2 Test Application v1 Production DB v1 DB v2 DB v3 CI/CD PIPELINE Version control database updates, ex: flyway V3__add_table_scooter.sql V2__add_table_truck.sql V1__add_table_car.sql
96. DATABASE MIGRATIONS Version control database updates with Containers CONTAINER IMAGE CONTAINER BUILD FILE SQL MIGRATION SCRIPT Source Code Repository V2__add_table.sql Source Code Repository V2__add_table.sql /var/flyway/data Flyway flyway-mydb:v2.0.0 Registry + Dockerfile
97. Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresq pvcl DATABASE MIGRATION StatefulSet deployment with headless Service v1
98. Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Create a Job for Flyway Flyway Job Secrets = Database Connection Info v1 flyway-mydb:v2.0.0 Image Registry Flyway
99. role=postgressql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql pvc DATABASE MIGRATIONS Apply schema changes to database Flyway Job Secrets = Database Connection Info V2 flyway-mydb:v2.0.0 Flyway
100. role=postgresql type=primary Nodes Pods Services postgresql-0 Persistent Volume A B D C PostgreSQL StatefulSet replicas=1 role=postgresql Pvc DATABASE MIGRATIONS Version control for database with Kubernetes V2
101. INFRASTRUCTURE
102. MONITORING CONSIDERATIONS Kubernetes* Container* Host Cluster services, services, pods, 
 deployments metrics Container native metrics Traditional resource metrics - cpu, memory, network, storage prometheus + grafana kubernetes-state-metrics probes Stack Metrics Tool node-exporter kubelet:cAdvisor Microservices Distributed applications - traditional app metrics - service discovery - distributed tracing prometheus + grafana jaeger tracing istio
103. ARCHITECTURE CONSIDERATIONS Optimize for… Cluster 
 per app / data / location, Short lived Data Sensitive, e.g. Finance Multi-AZ, Multi/
 Hybrid
 cloud Production, Mission 
 critical Security Scale Availability Latency Portability Performance Large cluster, multi/
 hybrid cloud Internet, SaaS Efficiency Large cluster, Bare Metal, Recreate Many apps, Dev/ Test Consistent
 OS & Kubernetes version 1 app anywhere, e.g. ISVs Local, Small Cluster IoT, Retail Bare metal (Multus, SR-IOV, NFD, Scheduler, CPU pin) HPC, AI/ML, NFV
104. WHAT’S NEXT
105. Traffic Control Service Resiliency Chaos Testing Observ- ability Security SERVICE MESH WITH ISTIO Dedicated infrastructure layer for making service-to-service communication 
 safe, fast, and reliable Deploy as a lightweight side-car network proxy
106. OPERATORS AUTOMATED OPERATIONS WITH OPERATORS Automate operational lifecycle at Scale
 including packaging, deploying, managing
 of Kubernetes applications Takes human operational
 knowledge and encodes 
 it into software Watches over your Kubernetes environment
 and uses its current state to make decisions in milliseconds.
107. KNATIVE FOR SERVERLESS
108. Deployment Frequency Lead Time Deployment
 Failure Rate Mean Time to Recover 99.999 Service Availability DEVSECOPS METRICS Compliance Score
109. THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin