OpenFinTech: A DevOps State of Mind: Continuous Security with Kubernetes

Transcript

1. A DevOps State of Mind:
   Continuous Security with Kubernetes
   Chris Van Tuin
   Red Hat
   Chief Technologist, NA West / Silicon Valley
   cvantuin@redhat.com
   

   View Slide

2. “Only the paranoid survive”
   - Andy Grove, 1996
   

   View Slide

3. THE WORLD IS AUTOMATING
   Those who succeed in automation will win
   

   View Slide

4. View Slide

5. THE CHALLENGE: 

   ENABLE INNOVATION AT SPEED, WHILE
   EXECUTING AT SCALE WITH EFFICIENCY
   Static &

   Planned
   Dynamic & 

   Policy Driven
   Execution
   Innovation
   Old New
   Execution
   Innovation
   

   View Slide

6. IT’S NOT JUST SOFTWARE,
   THE DIGITAL LEADERS =
   Empowered
   organization
   Speed Up 

   Innovation
   Time
   Change
   Move Fast,
   Break Things
   Culture of
   experimentation
   A
   20% vs. 25%
   Shorten the
   Feedback Loop
   Real-time
   data-driven
   intelligence &
   personalization
   AI /

   ML
   Data,
   Data,
   Data
   B
   

   View Slide

7. IT MUST EVOLVE & KEEP UP
   

   View Slide

8. Applications &
   devices outside of
   IT control
   Cloud
   computing
   Software-defined
   infrastructure
   Dissolving
   security
   perimeter
   Menacing threat
   landscape
   TRADITIONAL NETWORK-BASED DEFENSES ARE NO LONGER ENOUGH
   SECURING THE ENTERPRISE IS HARDER THAN EVER
   The way we develop, deploy and manage IT is changing dramatically
   led by DevOps, Cloud Native Applications, and Hybrid Cloud
   

   View Slide

9. DEVSECOPS
   Continuous
   Security
   Improvement
   Process
   Optimization
   Security
   Automation
   Dev QA Prod
   Reduce Risks, Lower Costs, Speed Delivery, Speed Reaction
   

   View Slide

10. DEVSECOPS
    + +
    Security
    DEV QA OPS
    Culture Process Technology
    Linux + Containers
    IaaS
    Orchestration
    CI/CD
    Source Control Management
    Collaboration
    Build and Artifact Management
    Testing
    Frameworks
    Cloud Native Applications
    Hybrid Cloud
    Open Source
    

    View Slide

11. Chris Van Tuin
    Chief Technologist, NA West / Silicon Valley
    cvantuin@redhat.co docker.io
    Registry
    Private
    Registry
    FROM fedora:1.0
    CMD echo “Hello”
    Build file
    Physical, Virtual, Cloud
    Container
    Image
    Container
    Instance
    Build Run
    Ship
    CONTAINERS ENABLE DEVSECOPS
    

    View Slide

12. Chris Van Tuin
    Chief Technologist, NA West / Silicon Valley
    cvantuin@redhat.co
    Scheduling Monitoring
    Persistence
    Discovery
    Lifecycle & health
    Scaling Aggregation Security
    CONTAINERS AT SCALE
    BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD
    

    View Slide

13. BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD
    Security Platform
    

    View Slide

14. AUTOMATION
    

    View Slide

15. Web Database
    role=web role=db role=web
    replicas=1, 

    role=db
    replicas=2, 

    role=web
    ORCHESTRATION
    Deployment, Declarative
    Pods
    Nodes
    Services
    Controller
    Manager
    &
    Data Store
    (etcd)
    

    View Slide

16. Web Database
    replicas=1, 

    role=db
    replicas=2, 

    role=web
    HEALTH CHECK
    Pods
    Nodes
    Services
    role=web role=db role=web
    Controller
    Manager
    &
    Data Store
    (etcd)
    

    View Slide

17. Pods
    Nodes
    Services
    Web Database
    replicas=1, 

    role=db
    replicas=3 

    role=web
    AUTO-SCALE
    50% CPU
    role=web role=db role=web role=web
    Controller
    Manager
    &
    Data Store
    (etcd)
    

    View Slide

18. Network
    isolation
    API & Platform
    access
    Federated
    clusters
    Storage
    {}
    CI/CD
    Monitoring &
    Logging
    Builds
    Images
    SECURING YOUR CONTAINER ENVIRONMENT
    Container
    host
    Registry
    

    View Slide

19. CONTAINER IMAGES
    

    View Slide

20. LAPTOP
    Container
    Application
    OS dependencies
    Guest VM
    LINUX
    BARE METAL
    Container
    Application
    OS dependencies
    LINUX
    VIRTUALIZATION
    Container
    Application
    OS dependencies
    Virtual Machine
    LINUX
    PRIVATE CLOUD
    Container
    Application
    OS dependencies
    Virtual Machine
    LINUX
    PUBLIC CLOUD
    Container
    Application
    OS dependencies
    Virtual Machine
    LINUX
    CONTAINERS - Build Once, Deploy Anywhere
    Reducing Risk and Improving Security with Improved Consistency
    

    View Slide

21. CONTAINER IMAGE
    JAR CONTAINER IMAGE
    Application Application
    Language runtimes
    OS dependencies
    1.2/latest
    1.1
    

    View Slide

22. Config Data
    Kubernetes
    configmaps
    secrets
    Container
    image
    Traditional 

    data services,
    Kubernetes 

    persistent volumes
    TREAT CONTAINERS AS IMMUTABLE
    Application
    Language runtimes
    OS dependencies
    

    View Slide

23. •Authenticating authorship
    •Non-repudiation
    •Ensuring image integrity
    CONTAINER IMAGE SIGNING
    Validate what images and version are running
    

    View Slide

24. CONTAINER BUILDS
    

    View Slide

25. A CONVERGED SOFTWARE 

    SUPPLY CHAIN
    

    View Slide

26. CUSTOM SUPPLY CHAIN
    

    View Slide

27. • Treat build file as a Blueprint
    • Version control build file
    • Don’t login to build/configure
    • Be explicit with versions, not latest
    • Always list registry pulling FROM
    • Specify USER, default is root
    • Each Run creates a new layer
    BUILD FILE BEST PRACTICES
    FROM registry.redhat.com/rhel7
    RUN groupadd -g 999 appuser && \
    useradd -r -u 999 -g appuser appuser
    USER appuser
    CMD echo “Hello”
    Build file
    

    View Slide

28. CONTAINER REGISTRY SECURITY
    

    View Slide

29. 64% of official images in Docker Hub 

    contain high priority security vulnerabilities
    examples:
    ShellShock (bash)
    Heartbleed (OpenSSL)
    Poodle (OpenSSL)
    Source: Over 30% of Official Images in Docker Hub Contain High Priority Security Vulnerabilities, Jayanth Gummaraju, Tarun Desikan, and Yoshio Turner, BanyanOps,
    May 2015 (http://www.banyanops.com/pdf/BanyanOps-AnalyzingDockerHub-WhitePaper.pdf)
    WHAT’S INSIDE THE CONTAINER MATTERS
    

    View Slide

30. PRIVATE REGISTRY
    

    View Slide

31. CONTAINER HOST SECURITY
    

    View Slide

32. Kernel
    Hardware (Intel, AMD) or Virtual Machine
    Containers Containers
    Containers
    Unit File
    Docker
    Image
    Container CLI
    SYSTEMD
    Cgroups Namespaces SELinux
    Drivers
    CONTAINERS ARE LINUX
    seccomp Read Only mounts
    

    View Slide

33. CGROUPS - RESOURCE ISOLATION
    

    View Slide

34. NAMESPACES - PROCESS ISOLATION
    

    View Slide

35. SELINUX - MANDATORY ACCESS CONTROLS
    Password
    Files
    Web
    Server Attacker
    Discretionary Access Controls 

    (file permissions)
    Mandatory Access Controls 

    (selinux)
    Internal
    Network
    Firewall
    Rules
    Password
    Files
    Firewall
    Rules
    Internal
    Network
    Web
    Server
    selinux
    policy
    

    View Slide

36. SECCOMP AND LINUX CAPABILITIES

    FILTERING SYSTEM CALLS and DROPPING PRIVILEGES
    

    View Slide

37. READ ONLY MOUNTS
    

    View Slide

38. Chris Van Tuin
    Chief Technologist, NA West / Silicon Valley
    cvantuin@redhat.co
    Best Practices
    • Don’t run as root
    • If you must, 

    limit Linux Capabilities
    • Limit SSH Access
    • Use namespaces
    • Define resource quotas
    • Enable logging
    • Apply Security Errata
    • Apply Security Context
    and seccomp filters
    • Run production 

    unprivileged containers 

    as read-only
    http://blog.kubernetes.io/2016/08/security-best-practices-kubernetes-deployment.html
    Kernel
    Hardware (Intel, AMD) or Virtual Machine
    Containers Containers
    Containers
    Unit File
    Docker
    Image
    Container CLI
    SYSTEMD
    Cgroups Namespaces SELinux
    Drivers seccomp Read Only mounts
    Capabilities
    CONTAINER HOST SECURITY
    

    View Slide

39. CONTINUOUS INTEGRATION
    WITH CONTAINERS
    

    View Slide

40. CONTINUOUS INTEGRATION + BUILDS
    

    View Slide

41. WHAT’S INSIDE MATTERS…
    

    View Slide

42. Security
    CONTINUOUS INTEGRATION WITH
    SECURITY SCAN
    

    View Slide

43. AUTOMATED SECURITY SCANNING with OpenSCAP
    Reports
    Scan
    SCAP Security
    Guide
    for RHEL
    CCE-27002-5
    Set Password Minimum
    Length
    Content
    Scan physical servers, virtual machines, docker images and containers

    for Security Policy Compliance (CCEs) and known Security Vulnerabilities (CVEs)
    

    View Slide

44. Standard Docker Host Security Profile
    Java Runtime Environment (JRE)
    Upstream Firefox STIG
    RHEL OSP STIG
    Red Hat Corporate Profile for Certified Cloud Providers (RH CCP)
    STIG for Red Hat Enterprise Linux 6, 7 Server
    STIG for Red Hat Virtualization Hypervisor
    Common Profile for General-Purpose Debian Systems
    Common Profile for General-Purpose Fedora Systems
    Common Profile for General-Purpose Ubuntu Systems
    Payment Card Industry – Data Security Standard (PCI-DSS) v3
    U.S. Government Commercial Cloud Services (C2S)
    CNSSI 1253 Low/Low/Low Control Baseline for Red Hat Enterprise Linux 7
    Criminal Justice Information Services (CJIS) Security Policy
    Unclassified Information in Non-federal Information Systems and Organizations (NIST 800-171)
    U.S. Government Configuration Baseline (NIAP OSPP v4.0, USGCB, STIG)
    Security Policies in SCAP Security Guide (partial)
    

    View Slide

45. SECURITY POLICY REPORT
    

    View Slide

46. SECURITY POLICY REMEDIATION
    

    View Slide

47. CONTINUOUS DELIVERY
    WITH CONTAINERS
    

    View Slide

48. Chris Van Tuin
    Chief Technologist, NA West / Silicon Valley
    cvantuin@redhat.co
    CONTINUOUS DELIVERY WITH CONTAINERS
    

    View Slide

49. CONTINUOUS DELIVERY DEPLOYMENT STRATEGIES
    DEPLOYMENT STRATEGIES
    • Recreate
    • Rolling updates
    • Blue / Green deployment
    • Canary with A/B testing
    

    View Slide

50. Recreate
    

    View Slide

51. Version 1 Version 1
    Version 1
    Version 1.2
    `
    Tests / CI
    RECREATE WITH DOWNTIME
    

    View Slide

52. Version 1 Version 1
    Version 1
    Version 1.2
    `
    Tests / CI
    RECREATE WITH DOWNTIME
    

    View Slide

53. Version 1.2 Version 1.2
    Version 1.2
    RECREATE WITH DOWNTIME
    Use Case
    • Non-mission critical services
    Cons
    • Downtime
    Pros
    • Simple, clean
    • No Schema incompatibilities
    • No API versioning
    

    View Slide

54. Rolling Updates
    

    View Slide

55. Version 1 Version 1
    Version 1
    Version 1.2
    `
    Tests / CI
    ROLLING UPDATES with ZERO DOWNTIME
    

    View Slide

56. Deploy new version and wait until it’s ready…
    Version 1 Version 1 V1.2
    Health Check:
    readiness probe
    e.g. tcp, http, script
    V1
    

    View Slide

57. Each container/pod is updated one by one
    Version 1.2
    50%
    Version 1 V1 V1.2
    

    View Slide

58. Each container/pod is updated one by one
    Version 1.2
    Version 1.2
    Version 1.2
    100%
    Use Case
    • Horizontally scaled
    • Backward compatible
    API/data
    • Microservices
    Cons
    • Require backward
    compatible APIs/data
    • Resource overhead
    Pros
    • Zero downtime
    • Reduced risk, gradual
    rollout w/health checks
    • Ready for rollback
    

    View Slide

59. Blue / Green Deployment
    

    View Slide

60. Version 1
    BLUE / GREEN DEPLOYMENT
    Route
    BLUE
    

    View Slide

61. Version 1
    BLUE / GREEN DEPLOYMENT
    Version 1.2
    BLUE GREEN
    

    View Slide

62. Version 1 Tests / CI
    BLUE / GREEN DEPLOYMENT
    Version 1.2
    BLUE GREEN
    

    View Slide

63. Version 1 Version 1.2
    BLUE / GREEN DEPLOYMENT
    Route
    Version 1.2
    BLUE GREEN
    

    View Slide

64. Version 1
    BLUE / GREEN DEPLOYMENT
    Rollback
    Route
    Version 1.2
    BLUE GREEN
    Use Case
    • Self-contained micro
    services (data)
    Cons
    • Resource overhead
    • Data synchronization
    Pros
    • Low risk, never
    change production
    • No downtime
    • Production like testing
    • Rollback
    

    View Slide

65. RAPID INNOVATION &
    EXPERIMENTATION
    

    View Slide

66. ”only about 1/3 of ideas improve the metrics 

    they were designed to improve.”

    Ronny Kohavi, Microsoft (Amazon)
    MICROSERVICES
    RAPID INNNOVATION & EXPERIMENTATION
    

    View Slide

67. CONTINUOUS FEEDBACK LOOP
    

    View Slide

68. A/B TESTING USING CANARY DEPLOYMENTS
    

    View Slide

69. Version B
    Version A
    100%
    Tests / CI
    Route
    25% Conversion Rate ?! Conversion Rate
    CANARY DEPLOYMENTS
    

    View Slide

70. 50% 50%
    Version B
    Version A
    Route
    25% Conversion Rate 30% Conversion Rate
    CANARY DEPLOYMENTS
    

    View Slide

71. 25% Conversion Rate
    100%
    Version A Version B
    Route
    30% Conversion Rate
    CANARY DEPLOYMENTS
    

    View Slide

72. 100%
    Route
    Rollback
    25% Conversion Rate 20% Conversion Rate
    CANARY DEPLOYMENTS
    Version B
    Version A
    

    View Slide

73. Network
    isolation
    API & Platform
    access
    Federated
    clusters
    Storage
    {}
    CI/CD
    Monitoring &
    Logging
    Images
    Builds
    Container
    host
    Registry
    SECURING YOUR CONTAINER ENVIRONMENT
    

    View Slide

74. NETWORK SECURITY
    

    View Slide

75. Network Namespace 

    provides resource isolation
    NETWORK ISOLATION
    Multi-Environment Multi-Tenant
    

    View Slide

76. NETWORK POLICY
    example: 

    all pods in namespace ‘project-a’ allow traffic 

    from any other pods in the same namespace.”
    

    View Slide

77. Kubernetes 

    Logical Network Model
    NETWORK SECURITY
    • Kubernetes uses a flat SDN model
    • All pods get IP from same CIDR
    • And live on same logical network
    • Assumes all nodes communicate

    Traditional 

    Physical Network Model
    • Each layer represents a Zone with

    increased trust - DMZ > App > DB,

    interzone flow generally one direction
    • Intrazone traffic generally unrestricted
    

    View Slide

78. NETWORK SECURITY MODELS
    Co-Existence Approaches
    One Cluster
    Multiple Zones
    Kubernete Cluster
    Physical Compute 

    isolation based on 

    Network Zones
    Kubernete Cluster
    One Cluster
    Per Zone
    Kubernete Cluster B
    Kubernete Cluster A
    Kubernetes Cluster B
    C
    D
    https://blog.openshift.com/openshift-and-network-security-zones-coexistence-approaches/
    

    View Slide

79. MONITORING & LOGGING
    

    View Slide

80. KUBERNETES MONITORING CONSIDERATIONS
    Kubernetes*
    Container*
    Host
    Cluster services, services, pods, 

    deployments metrics
    Container native metrics
    Traditional resource metrics
    - cpu, memory, network, storage
    prometheus + grafana
    kubernetes-state-metrics
    probes
    Stack Metrics Tool
    node-exporter
    kubelet:cAdvisor
    Application
    Distributed applications
    - traditional app metrics
    - service discovery
    - distributed tracing
    prometheus + grafana
    jaeger tracing
    istio
    

    View Slide

81. Aggregate platform and application log access via Kibana + Elasticsearch
    LOGGING
    

    View Slide

82. STORAGE SECURITY
    

    View Slide

83. Local Storage Quota Security Context Constraints
    STORAGE SECURITY
    Sometimes we can also have
    storage isolation requirements: 

    pods in a network zone must use
    different storage endpoints 

    than pods in other network
    zones.
    We can create one storage class
    per storage endpoint and 

    then control which storage
    class(es) a project can use
    

    View Slide

84. API & PLATFORM ACCESS
    

    View Slide

85. Authentication
    via
    OAuth tokens and
    SSL certificate
    Authorization
    via
    Policy Engine
    checks
    User/Group
    Defined Roles
    API & PLATFORM ACCESS
    

    View Slide

86. FEDERATION
    

    View Slide

87. Amazon East OpenStack
    FEDERATED CLUSTERS
    Roles & access management (in-dev)
    

    View Slide

88. WHAT’S NEXT
    

    View Slide

89. Traffic
    Control
    Service
    Resiliency
    Chaos
    Testing
    Observ-
    ability
    Security
    

    View Slide

90. OPERATORS
    

    View Slide

91. Deployment
    Frequency
    Lead
    Time
    Deployment

    Failure Rate
    Mean Time
    to Recover
    99.999
    Service
    Availability
    DEVSECOPS METRICS
    Compliance
    Score
    

    View Slide

92. THANK YOU
    linkedin: Chris Van Tuin
    email: cvantuin@redhat.com
    twitter: @chrisvantuin
    

    View Slide