Securing Containers the Netflix Way

This talk was presented at the Container Security Summit 2017.

Security has been a first principle at Netflix as we created our internal container service, known as Titus. Along the way we learned many places where our security needs aligned with others in the community. But, perhaps most interestingly, we also learned about where we differed. This talk will take you on a tour of how we have handled container security at Netflix from threat modeling to container isolation and identity. You’ll see how we built containers to integrate seamlessly into our development and production ecosystem. Finally, we discuss what we see as the open security challenges for large-scale container deployments.

Transcript

1. Securing Containers the Netflix Way
   Bryan D. Payne Engineering Manager, Product & Application Security
   

   View Slide

2. View Slide

3. Netflix by the Numbers
   • 93M+ members
   • 1000+ developers
   • 190+ countries
   • > 1/3 NA internet download traffic
   • 500+ microservices
   • 100,000+ virtual machines
   • 3+ AWS regions
   

   View Slide

4. Containers at Netflix
   

   View Slide

5. 2008 2016
   2012
   AWS EC2
   

   View Slide

6. Why Containers? Velocity!
   • Improved developer experience
   • Improved system resource management
   • Improved deployment time
   Speeding Up Deployments
   Measuring CPU usage once a minute makes no sense for containers…
   Coping with rate of change is a big challenge for monitoring tools.
   Datacenter Snowflakes
   • Deploy in months
   • Live for years
   Virtualized and Cloud
   • Deploy in minutes
   • Live for weeks
   Container Deployments
   • Deploy in seconds
   • Live for minutes/hours
   AWS Lambda Events
   • Respond in milliseconds
   • Live for seconds
   Monitoring Microservice and Containers: A Challenge
   by Adrian Cockcroft @ Gluecon 2015
   

   View Slide

7. Container Use Cases @ Netflix
   • Media Encoding - R&D Cycle
   • Using VMs: 1 month
   • Using Containers: 1 week
   • Continuous Integration
   • Build all Netflix codebase in hours
   • Saves countless hours of debugging
   

   View Slide

8. Batch
   Jobs
   

   View Slide

9. Batch
   Jobs
   Services
   

   View Slide

10. Services Add Complexity
    • Resizing & Lifecycle Duration
    • Autoscaling
    • Harder to upgrade underlying host
    • Have More State
    • Want to use existing tool chains
    

    View Slide

11. View Slide

12. View Slide

13. Titus / Spinnaker Integration
    

    View Slide

14. View Slide

15. View Slide

16. AWS ECS
    

    View Slide

17. Security at Netflix
    

    View Slide

18. Netflix Culture —> Netflix Security
    Values are what we Value
    High Performance
    Freedom & Responsibility
    Context, not Control
    Highly Aligned, Loosely Coupled
    http://www.slideshare.net/reed2001/culture-1798664
    

    View Slide

19. Scumblr
    https://github.com/Netflix/Scumblr
    • Sync from data sources
    • Github
    • Route53 DNS
    • Manual upload
    • Analysis on these results
    • Search
    • Curl
    • Static analysis
    

    View Slide

20. View Slide

21. Stethoscope
    • Help employees stay safe with BYOD
    • Education
    • Self service
    • Personalized
    • Actionable
    • User Focused Security
    • No forced updates
    • No company wide emails
    • No information overload
    

    View Slide

22. View Slide

23. Titus Security
    

    View Slide

24. Titus Security =
    Titus Security =Container Security
    

    View Slide

25. Titus Security =
    Titus Security =Container Security
    Understanding and Hardening Linux Containers
    by Aaron Grattafiori (NCC Group)
    https://www.nccgroup.trust/us/our-research/understanding-and-hardening-linux-containers/
    

    View Slide

26. Titus Security =
    Titus Security =Container Security ++
    

    View Slide

27. Titus Security =
    Titus Security =
    Container App Security
    API Security
    Identity
    AWS Security
    Monitoring
    Registry Security
    Control Plane Security
    

    View Slide

28. View Slide

29. Container App Security
    • Fresh base containers
    • Reference container app
    • Apps run as non-root
    

    View Slide

30. AWS Security
    • Metadata proxy service
    • IAM roles per container
    • Security groups per
    container
    • Dedicated AWS account
    • Limit IAM perms for host
    

    View Slide

31. Identity
    

    View Slide

32. API Security
    • mTLS for all API access
    • Security assessment for all
    components
    • Threat modeling
    • AppSec assessment
    

    View Slide

33. Monitoring
    • Insight for security patches
    • Check human access using
    Stethoscope
    • Forensic audit trail
    

    View Slide

34. Registry Security
    • Container signing
    • Container revocation
    • AuthN to publish
    • Registry scanning
    

    View Slide

35. Control Plane Security
    • TLS + authN for all UI access
    • Container isolation
    • Namespaces
    • Capabilities
    • MAC
    • Etc
    • General host OS hardening
    

    View Slide

36. Questions?
    bryanp@netflix.com
    https://bryanpayne.org
    

    View Slide