Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A DevOps State of Mind: Continuous Security with Kubernetes

A DevOps State of Mind: Continuous Security with Kubernetes

With the rise of DevOps, containers are at the brink of becoming a pervasive technology in Enterprise IT to accelerate application delivery for the business. When it comes to adopting containers in the enterprise, Security is the highest adoption barrier. In this talk you’ll learn about:

The top container security risks and how to manage theses risks at scale
How to make your Container enabled DevOps workflow more secure without slowing down your CI/CD pipeline
Automating security vulnerability management and compliance checking for container images

7c6a033dd957d547b49630f626e1a143?s=128

Chris Van Tuin

October 24, 2017
Tweet

Transcript

  1. A DEVOPS STATE OF MIND: Continuous Security with Kubernetes Chris

    Van Tuin Chief Technologist, NA West / Silicon Valley cvantuin@redhat.com
  2. THE DISRUPTORS

  3. “Only the Paranoid Survive” - Andy Grove, 1998

  4. Retail Finance Media Transportation ? ? SOFTWARE DISRUPTS BUSINESS

  5. + ≠ TAXI TRANSPORTATION DISRUPTER https://goo.gl/MP7QQH Ack: Andrew Ng APP

    IT’S NOT JUST A… +
  6. Culture of experimentation A B 20% vs. 25% Empowered organization

    Time Change Rapid Innovation THE DISRUPTORS = AI /
 ML Data-driven intelligence Data, Data, Data
  7. WHAT CAN I.T. DO?

  8. DEV QA OPS Walled off people, walled off processes, walled

    off technologies “THROW IT OVER THE WALL”
  9. I.T. MUST EVOLVE TO ENABLE THE BUSINESS Development Model Application

    Architecture Deployment & Packaging Application Infrastructure Storage Waterfall Agile Monolithic N-tier Bare Metal Virtual Servers Data Center Hosted Scale Up Scale Out DevOps MicroServices Containers Hybrid Cloud Storage as a Service
  10. None
  11. None
  12. DEV QA OPS Open organization + 
 cross-functional teams Software

    factory automation Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Open Source CI/CD pipelines with feedback Culture Process Technology + + BREAKING DOWN THE WALLS WITH DEVOPS
  13. WHAT ABOUT SECURITY?

  14. DEV QA OPS SECURITY IS AN AFTERTHOUGHT | SECURITY |

    “Patch? The servers are behind the firewall.” - Anonymous (far too many to name), 2005 - …
  15. http://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/

  16. DEVSECOPS End to End Security + + SECURITY DEV QA

    OPS Linux + Containers IaaS Orchestration CI/CD Source Control Management Collaboration Build and Artifact Management Testing Frameworks Open Source Culture Process Technology
  17. CONTAINERS

  18. APPLICATION DELIVERY VIA CONTAINERS

  19. docker.io Registry Private Registry FROM fedora:latest CMD echo “Hello” Build

    file Physical, Virtual, Cloud Image Container Build Run Ship CONTAINERS: BUILD, SHIP, RUN
  20. CONTAINER SECURITY AT SCALE

  21. Scheduling Monitoring Persistence Discovery Lifecycle & health Scaling Aggregation Security

    MORE THAN CONTAINERS…
  22. DEVSECOPS End to End Security + + DEV QA OPS

    SECURITY
  23. 4 • Are there known vulnerabilities in the application layer?

    • Are the runtime and OS layers up to date? • How frequently will the container be updated and how will I know when it’s updated? CONTENT: EACH LAYER MATTERS CONTAINER OS RUNTIME APPLICATION Are there known vulnerabilities 
 in each application layer? Are the runtime and OS layers 
 up to date? How frequently will the container be updated and how will I know when its updated? IS THE CONTAINER ENVIRONMENT SECURE? Is the image from a trusted source? Can I quickly deploy a security update at scale? Is my multi-tenant host secure?
  24. Network isolation Storage API & Platform access Monitoring & Logging

    Federated clusters Registry Container host {} Builds CI/CD Images SECURING CONTAINERS
  25. CONTAINER IMAGE SECURITY

  26. 4 • Are there known vulnerabilities in the application layer?

    • Are the runtime and OS layers up to date? • How frequently will the container be updated and how will I know when it’s updated? CONTENT: EACH LAYER MATTERS CONTAINER OS RUNTIME APPLICATION CONTENT: EACH LAYER MATTERS AYER MATTERS CONTAINER OS RUNTIME APPLICATION JAR CONTAINER
  27. SECURITY IMPLICATIONS What’s inside matters…

  28. A CONVERGED SOFTWARE SUPPLY CHAIN

  29. code config data Kubernetes configmaps secrets Container image Traditional 


    data services, Kubernetes 
 persistent volumes TREAT CONTAINERS AS IMMUTABLE
  30. NODE MASTER Container Distributed Store Container ! Secure mechanism for

    holding sensitive data e.g. ◦ Passwords and credentials ◦ SSH Keys ◦ Certificates ! Secrets are made available as ◦ Environment variables ◦ Volume mounts ◦ Interaction with external systems ! Encrypted in transit ! Never rest on the nodes SECRETS
  31. IMAGE SIGNING Validate what images and version are running

  32. CONTAINER REGISTRY SECURITY

  33. 64% of official images in Docker Hub 
 contain high

    priority security vulnerabilities examples: ShellShock (bash) Heartbleed (OpenSSL) Poodle (OpenSSL) Source: Over 30% of Official Images in Docker Hub Contain High Priority Security Vulnerabilities, Jayanth Gummaraju, Tarun Desikan, and Yoshio Turner, BanyanOps, May 2015 (http://www.banyanops.com/pdf/BanyanOps-AnalyzingDockerHub-WhitePaper.pdf) WHAT’S INSIDE THE CONTAINER MATTERS
  34. PRIVATE REGISTRY

  35. CI WITH CONTAINERS

  36. CI/CD PIPELINE Continuous Integration Continuous Build Continuous Deployment Developer ->

    Source -> Git Git -> RPMS -> Images-> Registry Images from 
 Registry -> Clusters
  37. CI/CD with Security

  38. Security CONTINUOUS INTEGRATION WITH SECURITY SCAN

  39. CUSTOM SUPPLY CHAIN

  40. FROM fedora:latest CMD echo “Hello” Build file Build Best Practices

    • Treat as a Blueprint • Don’t login to build/configure • Version control build file • Be explicit with versions, not latest • Each Run creates a new layer BUILDS
  41. Compliance and Vulnerability Audits with OpenSCAP

  42. AUTOMATED SECURITY SCANNING with OpenSCAP Reports Scan SCAP Security Guide

    for RHEL CCE-27002-5 Set Password Minimum Length Content Scan physical servers, virtual machines, docker images and containers
 for Security Policy Compliance (CCEs) and known Security Vulnerabilities (CVEs)
  43. Standard Docker Host Security Profile Java Runtime Environment (JRE) Upstream

    Firefox STIG RHEL OSP STIG Red Hat Corporate Profile for Certified Cloud Providers (RH CCP) STIG for Red Hat Enterprise Linux 6, 7 Server STIG for Red Hat Virtualization Hypervisor Common Profile for General-Purpose Debian Systems Common Profile for General-Purpose Fedora Systems Common Profile for General-Purpose Ubuntu Systems Payment Card Industry – Data Security Standard (PCI-DSS) v3 U.S. Government Commercial Cloud Services (C2S) CNSSI 1253 Low/Low/Low Control Baseline for Red Hat Enterprise Linux 7 Criminal Justice Information Services (CJIS) Security Policy Unclassified Information in Non-federal Information Systems and Organizations (NIST 800-171) U.S. Government Configuration Baseline (NIAP OSPP v4.0, USGCB, STIG) Security Policies in SCAP Security Guide (partial)
  44. SECURITY POLICY REPORT

  45. SECURITY POLICY REMEDIATION

  46. SECURITY VULNERABILITY REPORT

  47. RAPID INNOVATION & EXPERIMENTATION

  48. ”only about 1/3 of ideas improve the metrics 
 they

    were designed to improve.”
 Ronny Kohavi, Microsoft (Amazon) MICROSERVICES RAPID INNNOVATION & EXPERIMENTATION
  49. CONTINUOUS FEEDBACK LOOP

  50. A/B TESTING USING CANARY DEPLOYMENTS

  51. 50% 50% Version 1.2 Version 1 Route Version 1.2 25%

    Conversion Rate 30% Conversion Rate CANARY DEPLOYMENTS
  52. CD WITH CONTAINERS

  53. CONTINUOUS DEPLOYMENT WITH CONTAINERS

  54. CD with Security

  55. CONTINUOUS DELIVERY DEPLOYMENT STRATEGIES DEPLOYMENT STRATEGIES • Blue / Green

    deployment • Rolling updates • Canary deployments • A / B testing
  56. Version 1 BLUE / GREEN DEPLOYMENT Route

  57. Version 1 BLUE / GREEN DEPLOYMENT Version 1.2

  58. Version 1 Tests / CI BLUE / GREEN DEPLOYMENT Version

    1.2
  59. Version 1 Version 1.2 BLUE / GREEN DEPLOYMENT Route Version

    1.2
  60. Version 1 BLUE / GREEN DEPLOYMENT Rollback Route Version 1.2

  61. Version 1 Version 1 Version 1 Version 1.2 ` Tests

    / CI ROLLING UPDATES with ZERO DOWNTIME
  62. Deploy new version and wait until it’s ready… Version 1

    Version 1 V1.2 Health Check: Readiness 
 Probe e.g. tcp, http, script V1
  63. Each container/pod is updated one by one Version 1.2 50%

    Version 1 V1 V1.2
  64. Each container/pod is updated one by one Version 1.2 Version

    1.2 Version 1.2 100%
  65. CONTAINER HOST SECURITY

  66. Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers

    Unit File Docker Image Container CLI SYSTEMD Cgroups Namespaces SELinux Drivers CONTAINERS ARE LINUX seccomp Read Only mounts
  67. CGROUPS - RESOURCE ISOLATION

  68. NAMESPACES - PROCESS ISOLATION

  69. SELINUX - MANDATORY ACCESS CONTROLS Password Files Web Server Attacker

    Discretionary Access Controls 
 (file permissions) Mandatory Access Controls 
 (selinux) Internal Network Firewall Rules Password Files Firewall Rules Internal Network Web Server selinux policy
  70. SECCOMP - DROPPING PRIVILEGES

  71. READ ONLY MOUNTS

  72. Best Practices • Don’t run as root • Limit SSH

    Access • Use namespaces • Define resource quotas • Enable logging • Apply Security Errata • Apply Security Context and seccomp filters http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html CONTAINER HOST SECURITY Kernel Hardware (Intel, AMD) or Virtual Machine Containers Containers Containers Unit File Docker Image Container SYSTEM Cgroup Namespace SELinu Driver seccom Read Only
  73. Network isolation Storage API & Platform access Monitoring & Logging

    Federated clusters Registry Container host {} Builds CI/CD Images SECURING CONTAINERS
  74. NETWORK ISOLATION

  75. Network Namespace 
 provides resource isolation NETWORK ISOLATION Multi-Environment Multi-Tenant

  76. NETWORK POLICY example: 
 all pods in namespace ‘project-a’ allow

    traffic 
 from any other pods in the same namespace.”
  77. STORAGE SECURITY

  78. Local Storage Quota Security Context Constraints STORAGE SECURITY

  79. API & PLATFORM ACCESS

  80. Authentication via OAuth tokens and SSL certificate Authorization via Policy

    Engine checks User/Group Defined Roles API & PLATFORM ACCESS
  81. MONITORING & LOGGING

  82. Aggregate platform and application log access via Kibana + Elasticsearch

    LOGGING
  83. Historical CPU and Memory usage provided by Heapster, Hawkular, Cassandra

    MONITORING
  84. FEDERATION

  85. Amazon East OpenStack FEDERATED CLUSTERS

  86. Google Global Routing Amazon/Google Global API https://commons.wikimedia.org/wiki/File:Iscosahedral_water_cluster_100.png Java App1 Amazon

    East Java App1 Amazon West Java App1 FEDERATED SERVICES
  87. Deployment Frequency Lead Time Deployment
 Failure Rate Mean Time to

    Recover 99.999 Service Availability DEVSECOPS METRICS Compliance Score
  88. THANK YOU linkedin: Chris Van Tuin email: cvantuin@redhat.com twitter: @chrisvantuin