Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Securing Containers the Netflix Way

Bryan Payne
January 25, 2017

Securing Containers the Netflix Way

This talk was presented at the Container Security Summit 2017.

Security has been a first principle at Netflix as we created our internal container service, known as Titus. Along the way we learned many places where our security needs aligned with others in the community. But, perhaps most interestingly, we also learned about where we differed. This talk will take you on a tour of how we have handled container security at Netflix from threat modeling to container isolation and identity. You’ll see how we built containers to integrate seamlessly into our development and production ecosystem. Finally, we discuss what we see as the open security challenges for large-scale container deployments.

Bryan Payne

January 25, 2017
Tweet

More Decks by Bryan Payne

Other Decks in Programming

Transcript

  1. Securing Containers the Netflix Way
    Bryan D. Payne Engineering Manager, Product & Application Security

    View Slide

  2. View Slide

  3. Netflix by the Numbers
    • 93M+ members
    • 1000+ developers
    • 190+ countries
    • > 1/3 NA internet download traffic
    • 500+ microservices
    • 100,000+ virtual machines
    • 3+ AWS regions

    View Slide

  4. Containers at Netflix

    View Slide

  5. 2008 2016
    2012
    AWS EC2

    View Slide

  6. Why Containers? Velocity!
    • Improved developer experience
    • Improved system resource management
    • Improved deployment time
    Speeding Up Deployments
    Measuring CPU usage once a minute makes no sense for containers…
    Coping with rate of change is a big challenge for monitoring tools.
    Datacenter Snowflakes
    • Deploy in months
    • Live for years
    Virtualized and Cloud
    • Deploy in minutes
    • Live for weeks
    Container Deployments
    • Deploy in seconds
    • Live for minutes/hours
    AWS Lambda Events
    • Respond in milliseconds
    • Live for seconds
    Monitoring Microservice and Containers: A Challenge
    by Adrian Cockcroft @ Gluecon 2015

    View Slide

  7. Container Use Cases @ Netflix
    • Media Encoding - R&D Cycle
    • Using VMs: 1 month
    • Using Containers: 1 week
    • Continuous Integration
    • Build all Netflix codebase in hours
    • Saves countless hours of debugging

    View Slide

  8. Batch
    Jobs

    View Slide

  9. Batch
    Jobs
    Services

    View Slide

  10. Services Add Complexity
    • Resizing & Lifecycle Duration
    • Autoscaling
    • Harder to upgrade underlying host
    • Have More State
    • Want to use existing tool chains

    View Slide

  11. View Slide

  12. View Slide

  13. Titus / Spinnaker Integration

    View Slide

  14. View Slide

  15. View Slide

  16. AWS ECS

    View Slide

  17. Security at Netflix

    View Slide

  18. Netflix Culture —> Netflix Security
    Values are what we Value
    High Performance
    Freedom & Responsibility
    Context, not Control
    Highly Aligned, Loosely Coupled
    http://www.slideshare.net/reed2001/culture-1798664

    View Slide

  19. Scumblr
    https://github.com/Netflix/Scumblr
    • Sync from data sources
    • Github
    • Route53 DNS
    • Manual upload
    • Analysis on these results
    • Search
    • Curl
    • Static analysis

    View Slide

  20. View Slide

  21. Stethoscope
    • Help employees stay safe with BYOD
    • Education
    • Self service
    • Personalized
    • Actionable
    • User Focused Security
    • No forced updates
    • No company wide emails
    • No information overload

    View Slide

  22. View Slide

  23. Titus Security

    View Slide

  24. Titus Security =
    Titus Security =Container Security

    View Slide

  25. Titus Security =
    Titus Security =Container Security
    Understanding and Hardening Linux Containers
    by Aaron Grattafiori (NCC Group)
    https://www.nccgroup.trust/us/our-research/understanding-and-hardening-linux-containers/

    View Slide

  26. Titus Security =
    Titus Security =Container Security ++

    View Slide

  27. Titus Security =
    Titus Security =
    Container App Security
    API Security
    Identity
    AWS Security
    Monitoring
    Registry Security
    Control Plane Security

    View Slide

  28. View Slide

  29. Container App Security
    • Fresh base containers
    • Reference container app
    • Apps run as non-root

    View Slide

  30. AWS Security
    • Metadata proxy service
    • IAM roles per container
    • Security groups per
    container
    • Dedicated AWS account
    • Limit IAM perms for host

    View Slide

  31. Identity

    View Slide

  32. API Security
    • mTLS for all API access
    • Security assessment for all
    components
    • Threat modeling
    • AppSec assessment

    View Slide

  33. Monitoring
    • Insight for security patches
    • Check human access using
    Stethoscope
    • Forensic audit trail

    View Slide

  34. Registry Security
    • Container signing
    • Container revocation
    • AuthN to publish
    • Registry scanning

    View Slide

  35. Control Plane Security
    • TLS + authN for all UI access
    • Container isolation
    • Namespaces
    • Capabilities
    • MAC
    • Etc
    • General host OS hardening

    View Slide

  36. Questions?
    bryanp@netflix.com
    https://bryanpayne.org

    View Slide