Upgrade to Pro — share decks privately, control downloads, hide ads and more …

OpenStack Summit Vancouver: A DevOps State of Mind: Continuous Security with Kubernetes

OpenStack Summit Vancouver: A DevOps State of Mind: Continuous Security with Kubernetes

With the rise of DevOps, containers are at the brink of becoming a pervasive technology in Enterprise IT to accelerate application delivery for the business. When it comes to adopting containers in the enterprise, Security is the highest adoption barrier. Is your organization ready to address the security risks with containers for your DevOps environment? In this presentation, you'll learn about:

- Best practices for addressing the top container security risks in a container environment including images, builds, registry, deployment, hosts, network, storage, APIs, monitoring & logging, and federation.

- Automating and integrating security vulnerability management & compliance checking for container images in a DevOps CI/CD pipeline

- Deployment strategies for deploying container security updates including recreate, rolling, blue/green, canary and a/b testing.

7c6a033dd957d547b49630f626e1a143?s=128

Chris Van Tuin

May 22, 2018
Tweet

Transcript

  1. A DevOps State of Mind: Continuous Security with Kubernetes Chris

    Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.com
  2. “Only the paranoid survive” - Andy Grove, 1996

  3. Retail Finance Media Transportation ? ? SOFTWARE DISRUPTS BUSINESS

  4. HOW DOES I.T. ACCELERATE 
 BUSINESS INNOVATION? 6 to 9

    months Innovation Hours to Weeks
  5. DEV QA OPS “THROW IT OVER THE WALL” Challenges 1.

    Limited resources 2. Security bottlenecks 3. Large change set = 
 more bugs, more testing 4. Extended feedback loop 5. Environment drift Walled off people, walled off processes, walled off technologies
  6. DEVOPS Speeding Up Time to Market Leverage tooling & automation

    Reduce organizational silos Accept failure
 as normal Implement gradual change Measure everything
  7. DEV QA OPS SECURITY IS AN AFTERTHOUGHT | SECURITY |

    “Patch? The servers are behind the firewall.” - Anonymous (far too many to name), 2005 - …
  8. http://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/ SECURITY BREACH: BILLION DATA RECORDS

  9. 36% - Employees not taking proper security measures 32% -

    Outside breach 14% - Unpatched or unpatchable 11% - Internal attack by an employee 4% - Shadow IT 3% - Bring your own device/mobile Source: Techvalidate/Red Hat % of Respondants WHAT IS THE GREATEST SECURITY RISK?
  10. SECURITY MUST EVOLVE & KEEP UP

  11. ANY COMBINATION, WHETHER TRADITIONAL OR CONTAINERIZED LEGACY APPS (1,000+) BARE

    METAL PRIVATE CLOUD PUBLIC CLOUD VIRTUAL PRODUCTION DEV/TEST HYBRID CLOUD ENVIRONMENTS
  12. BARE METAL VIRTUAL PRIVATE CLOUD OFF-PREMISE ON-PREMISE PUBLIC CLOUD DATA

    DATA DISTRIBUTED APPLICATIONS
  13. DEV QA OPS Open organization + 
 cross-functional teams Software

    factory automation Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Open Source CI/CD pipelines with feedback Culture Process Technology + + BREAKING DOWN THE WALLS WITH DEVOPS
  14. DEVSECOPS + + End to End Security DEV QA OPS

    Culture Automated Process Technology Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Open Source
  15. None
  16. None
  17. DEVSECOPS Continuous Security Improvement Process Optimization Security Automation Dev QA

    Prod Reduce Risks, Lower Costs, Speed Delivery, Speed Reaction
  18. CONTAINERS

  19. CONTAINERS

  20. 4 • Are there known vulnerabilities in the application layer?

    • Are the runtime and OS layers up to date? • How frequently will the container be updated and how will I know when it’s updated? CONTENT: EACH LAYER MATTERS CONTAINER OS RUNTIME APPLICATION CONTENT: EACH LAYER MATTERS AYER MATTERS CONTAINER OS RUNTIME APPLICATION JAR CONTAINER
  21. LAPTOP Container Application OS dependencies Guest VM LINUX BARE METAL

    Container Application OS dependencies LINUX VIRTUALIZATION Container Application OS dependencies Virtual Machine LINUX PRIVATE CLOUD Container Application OS dependencies Virtual Machine LINUX PUBLIC CLOUD Container Application OS dependencies Virtual Machine LINUX APPLICATION PORTABILITY WITH CONTAINERS
  22. CONTAINERS AT SCALE

  23. Scheduling Monitoring Persistence Discovery Lifecycle & health Scaling Aggregation Security

    MORE THAN CONTAINERS…
  24. BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD

  25. DEVSECOPS End to End Security + + DEV QA OPS

    SECURITY
  26. Web Database replicas=1, 
 role=db replicas=2, 
 role=web ORCHESTRATION Deployment,

    Declarative Nodes Controller Manager & Data Store (etcd)
  27. role=web role=web ORCHESTRATION Schedule + Provision Pods (Compute/Storage/Network) Image Registry

    Pods Nodes Web replicas=2, 
 role=web ReplicaSet
  28. role=web role=db role=web Pods Nodes Image Registry ORCHESTRATION Schedule +

    Provision Pods (Compute/Storage/Network) Web replicas=2, 
 role=web ReplicaSet Database replicas=1, 
 role=db StatefulSet
  29. Web Database role=web role=db role=web replicas=1, 
 role=db replicas=2, 


    role=web ORCHESTRATION Service (Load Balancer) Pods Nodes Services Controller Manager & Data Store (etcd)
  30. HEALTH CHECK Monitoring & Logging Pods Nodes Services Web Database

    role=web role=db role=web replicas=1, 
 role=db replicas=2, 
 role=web
  31. HEALTH CHECK Pods Nodes Services Web Database role=web role=db role=web

    replicas=1, 
 role=db replicas=2, 
 role=web role=web Controller Manager & Data Store (etcd)
  32. Web Database replicas=1, 
 role=db replicas=2, 
 role=web HEALTH CHECK

    Pods Nodes Services role=web role=db role=web Controller Manager & Data Store (etcd)
  33. Web Database replicas=1, 
 role=db replicas=2, 
 role=web AUTO-SCALE Monitoring

    & Logging 80% CPU Pods Nodes Services role=web role=db role=web
  34. Web Database replicas=1, 
 role=db replicas=3 
 role=web AUTO-SCALE 80%

    CPU Pods Nodes Services role=web role=db role=web role=web Controller Manager & Data Store (etcd)
  35. Pods Nodes Services Web Database replicas=1, 
 role=db replicas=3 


    role=web AUTO-SCALE 50% CPU role=web role=db role=web role=web Controller Manager & Data Store (etcd)
  36. CONTAINER SECURITY

  37. Network isolation Storage API & Platform access Monitoring & Logging

    Federated clusters Registry Container host {} CI/CD Images SECURING CONTAINERS Builds
  38. CONTAINER BUILDS

  39. docker.io Registry Private Registry FROM fedora:1.0 CMD echo “Hello” Build

    file Physical, Virtual, Cloud Image Container Build Run Ship CONTAINER BUILDS
  40. Best Practices • Treat as a Blueprint • Don’t login

    to build/configure • Version control build file • Be explicit with versions, not latest • Each Run creates a new layer CONTAINER BUILDS FROM fedora:1.0 CMD echo “Hello” Build file Build
  41. A CONVERGED SOFTWARE 
 SUPPLY CHAIN

  42. CONTAINER IMAGE SECURITY

  43. 64% of official images in Docker Hub 
 contain high

    priority security vulnerabilities examples: ShellShock (bash) Heartbleed (OpenSSL) Poodle (OpenSSL) Source: Over 30% of Official Images in Docker Hub Contain High Priority Security Vulnerabilities, Jayanth Gummaraju, Tarun Desikan, and Yoshio Turner, BanyanOps, May 2015 (http://www.banyanops.com/pdf/BanyanOps-AnalyzingDockerHub-WhitePaper.pdf) WHAT’S INSIDE THE CONTAINER MATTERS
  44. SECURITY IMPLICATIONS What’s inside matters…

  45. code config data Kubernetes configmaps secrets Container image Traditional 


    data services, Kubernetes 
 persistent volumes TREAT CONTAINERS AS IMMUTABLE
  46. CONTAINER REGISTRY SECURITY

  47. PRIVATE REGISTRY

  48. IMAGE SIGNING Validate what images and version are running

  49. CONTINUOUS INTEGRATION WITH CONTAINERS

  50. CONTINUOUS INTEGRATION + SECURITY

  51. Security CONTINUOUS INTEGRATION WITH SECURITY SCAN

  52. CONTINUOUS DELIVERY WITH CONTAINERS

  53. CONTINUOUS DELIVERY WITH CONTAINERS

  54. CONTINUOUS DELIVERY + SECURITY

  55. CONTINUOUS DELIVERY: DEPLOYMENT STRATEGIES

  56. CONTINUOUS DELIVERY DEPLOYMENT STRATEGIES DEPLOYMENT STRATEGIES • Recreate • Rolling

    updates • Blue / Green deployment
  57. Recreate

  58. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI RECREATE WITH DOWNTIME
  59. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI RECREATE WITH DOWNTIME
  60. Version 1.2 Version 1.2 Version 1.2 RECREATE WITH DOWNTIME Use

    Case • Non-mission critical services Cons • Downtime Pros • Simple, clean • No Schema incompatibilities • No API versioning
  61. Rolling Updates

  62. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI ROLLING UPDATES with ZERO DOWNTIME
  63. Deploy new version and wait until it’s ready… Version 1

    Version 1 V1.2 Health Check: readiness probe e.g. tcp, http, script V1
  64. Each container/pod is updated one by one Version 1.2 50%

    Version 1 V1 V1.2
  65. Each container/pod is updated one by one Version 1.2 Version

    1.2 Version 1.2 100% Use Case • Horizontally scaled • Backward compatible API/data • Microservices Cons • Require backward compatible APIs/data • Resource overhead Pros • Zero downtime • Reduced risk, gradual rollout w/health checks • Ready for rollback
  66. Blue / Green Deployment

  67. Version 1 BLUE / GREEN DEPLOYMENT Route BLUE

  68. Version 1 BLUE / GREEN DEPLOYMENT Version 1.2 BLUE GREEN

  69. Version 1 Tests / CI BLUE / GREEN DEPLOYMENT Version

    1.2 BLUE GREEN
  70. Version 1 Version 1.2 BLUE / GREEN DEPLOYMENT Route Version

    1.2 BLUE GREEN
  71. Version 1 BLUE / GREEN DEPLOYMENT Rollback Route Version 1.2

    BLUE GREEN Use Case • Self-contained micro services (data) Cons • Resource overhead • Data synchronization Pros • Low risk, never change production • No downtime • Production like testing • Rollback
  72. RAPID INNOVATION & EXPERIMENTATION

  73. ”only about 1/3 of ideas improve the metrics 
 they

    were designed to improve.”
 Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION
  74. CONTINUOUS FEEDBACK LOOP

  75. A/B TESTING USING CANARY DEPLOYMENTS

  76. Version 1.2 Version 1 100% Tests / CI Version 1.2

    Route 25% Conversion Rate ?! Conversion Rate CANARY DEPLOYMENTS
  77. 50% 50% Version 1.2 Version 1 Route Version 1.2 25%

    Conversion Rate 30% Conversion Rate CANARY DEPLOYMENTS
  78. 25% Conversion Rate 100% Version 1 Version 1.2 Route Version

    1.2 30% Conversion Rate CANARY DEPLOYMENTS
  79. Version 1.2 Version 1 100% Route Rollback 25% Conversion Rate

    20% Conversion Rate CANARY DEPLOYMENTS
  80. CONTAINER HOST SECURITY

  81. Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers

    Unit File Docker Image Container CLI SYSTEMD Cgroups Namespaces SELinux Drivers CONTAINERS ARE LINUX seccomp Read Only mounts
  82. CGROUPS - RESOURCE ISOLATION

  83. NAMESPACES - PROCESS ISOLATION

  84. SELINUX - MANDATORY ACCESS CONTROLS Password Files Web Server Attacker

    Discretionary Access Controls 
 (file permissions) Mandatory Access Controls 
 (selinux) Internal Network Firewall Rules Password Files Firewall Rules Internal Network Web Server selinux policy
  85. SECCOMP - DROPPING PRIVILEGES

  86. READ ONLY MOUNTS

  87. Best Practices • Don’t run as root • Limit SSH

    Access • Use namespaces • Define resource quotas • Enable logging • Apply Security Errata • Apply Security Context and seccomp filters http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html CONTAINER HOST SECURITY Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container SYSTEM Cgroup Namespace SELinu Driver seccom Read Only
  88. Network isolation Storage API & Platform access Monitoring & Logging

    Federated clusters Registry Container host {} CI/CD Images Builds SECURING CONTAINERS
  89. NETWORK ISOLATION

  90. Network Namespace 
 provides resource isolation NETWORK ISOLATION Multi-Environment Multi-Tenant

  91. NETWORK POLICY example: 
 all pods in namespace ‘project-a’ allow

    traffic 
 from any other pods in the same namespace.”
  92. STORAGE SECURITY

  93. Local Storage Quota Security Context Constraints STORAGE SECURITY

  94. API & PLATFORM ACCESS

  95. Authentication via OAuth tokens and SSL certificate Authorization via Policy

    Engine checks User/Group Defined Roles API & PLATFORM ACCESS
  96. MONITORING & LOGGING

  97. Aggregate platform and application log access via Kibana + Elasticsearch

    LOGGING
  98. Historical CPU and Memory usage MONITORING

  99. FEDERATION

  100. Amazon East OpenStack FEDERATED CLUSTERS Roles & access management (in-dev)

  101. Deployment Frequency Lead Time Deployment
 Failure Rate Mean Time to

    Recover 99.999 Service Availability DEVSECOPS METRICS Compliance Score
  102. THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin