Key Management in AWS: How Netflix Secures Sensitive Data Without Its Own Data Center

Netflix has embraced the cloud paradigm by moving its entire product into AWS. Fulfilling this vision meant finding a solution for every piece of our ecosystem in the cloud, including storage of our most sensitive cryptographic keys. We accomplished this using AWS CloudHSM, combined with a custom solution that handles cross-region key replication, failover, and more. This talk will discuss Netflix’s Cryptex service and how it balances security and agility in ways that bring HSM-level security into the toolkit of modern cloud application deployments.

This talk was given at JPL in October 2015.

Thanks to Jay Zarfoss for creating many of the slides!

1. Key Management in AWS How Netflix Secures Sensitive Data Without Its Own Data Center Bryan D. Payne, Ph.D. Engineering Manager, Platform Security @ Netflix
2. Platform Security @ Netflix Case Study: AWS Authentication Cryptex Motivation Cryptex Architecture Looking Ahead
3. Platform Security Overview Microservices in the Cloud Device or Browser Netflix Open Connect Appliance 1 2 - AWS Mgmt - Security Tools - Code Review - Forensics / IR - IT Security - Content Protection - Device Security Platform Security - Foundational Security Services - Security in Common Platform - Security by Default in base AMI
4. Classic Security via AWS CloudHSM Instance Metadata Signature Identity & Access Management Trusted Services (AWS) Great Unknown Hypervisor Hardware Platform Physical Security Malicious Insider Key Management Supply Chain Firmware Side Channel Leaks Trusted Services (Netflix) Secret Deployment Service Self-Service CA Crypto / Key Management Service
5. Ubiquitous Security • Partner with other teams • Make security transparent (or easy) • Focus on common components • Also focus on strategic risks Platform Security Review Implement Im plem ent D eploy Report Service Creation Service Maintenance Security Audit IR / Forensics Plan Security Improvements Security Services Security Defaults
6. Any large scale AWS deployment needs key-leveraging cryptography • Authentication to AWS Services themselves • Business Logic use cases (if nothing else PCI/HIPAA/whatever requires encryption) Microservices in the Cloud Device or Browser Netflix Open Connect Appliance ISP or Netflix Hosted at IXs Amazon Web Services (AWS) Everyone Needs Crypto
7. Case Study: AWS Authentication
8. Background: Authenticating yourself to AWS http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html DELETE /johnsmith/photos/puppy.jpg HTTP/1.1 User-Agent: dotnet Host: s3.amazonaws.com Date: Tue, 27 Mar 2007 21:20:27 +0000 x-amz-date: Tue, 27 Mar 2007 21:20:26 +0000 Authorization: AWS AKIAIOSFODNN7EXAMPLE:lx3byBScXR6KzyMaifNkardMwNk=
9. AWS HMAC Generation Not real secret keys, sorry. Lifecycle of AccessKeyID and SecretKey is of utmost interest here. AKIAIOSFODNN7EXAMPLE:iX KQe8qXbhnN0jUe7JGVqFNXM mTxP5pI6example DELETE\n \n \n Tue, 27 Mar 2007 21:20:26 +0000\n /johnsmith/photos/puppy.jpg AccessKeyID and SecretKey HMAC- SHA-1 Customer Request lx3byBScXR6KzyMaifNkardMwNk Digest Verified by AWS
10. SDKs make this easier //fortunately, AWS provides helper objects that do most of the work BasicAWSCredentials cred = new BasicAWSCredentials(accessKeyID, secretKeyID); AmazonSimpleDBClient client = new AmazonSimpleDBClient(cred); //ugly HMAC generating code safely tucked away in here somewhere client.listDomains(); Still does not answer the question as to where accessKeyID and secretKeyID are stored in our ecosystem.
11. Where we put AccessKeyID and SecretKey
 First Attempt - Just stick them in a System Property in the AMI // if it makes you feel better, let’s pretend the properties
 // are obfuscated and don’t show up in an obvious grep BasicAWSCredentials cred = new BasicAWSCredentials(
 System.getProperty(”AccessKeyID”), System.getProperty(”SecretKeyID”)
 ); Obvious deficiencies: • Key Exposed? Rebake thousands of AMIs and Redeploy • Is a relatively big effort to rotate in a managed fashion
12. Where to Put AccessKeyID and SecretKey
 Second Attempt - Deliver via our own simple RESTful Key Server GET server/getAWSKey <AWSKEY> <accessKeyID>AKIAIOSFODNN7EXAMPLE</accessKeyID> <secretKey>iXKQe8qXbhnN0jUe7JGVqFNXMmTxP5pI6example</secretKey> </AWSKEY> Improvement! • In theory gives direct control over key delivered to machines • Add some access control and TLS because security • In practice still has problems and you’ll need to reboot VMs
13. Unfortunate Key Stickiness RESTfulObj AWSKey = RESTService.get(“server/getAWSKey”); BasicAWSCredentials cred = new BasicAWSCredentials(AWSKey.getAccessID(), AWSKey.getSecretKey()); AmazonSimpleDBClient client = new AmazonSimpleDBClient(cred); client.listDomains(); In practice in the standard Java (and other) SDKs, keys stay lodged in the client object in the above code example.
14. AWS SDKs Introduce the Provider Paradigm
 2012 seems like a different lifetime // provider paradigm dynamically asks for keys every time
    AWSCredentialsProvider prov = new AWSCredentialsProvider(){ public AWSCredentials getCredentials(){ RESTfulObj AWSKey = RESTService.get(“server/getAWSKey”); return new BasicAWSCredentials(
    AWSKey.getAccessID(), AWSKey.getSecretKey()); } }; AmazonSimpleDBClient client = new AmazonSimpleDBClient(prov); client.listDomains(); The client object in the above code example no longer caches keys.
15. Systematically Enforce Refresh GET server/getAWSKey <AWSKEY> <accessKeyID>AKIAIOSFODNN7EXAMPLE</accessKeyID> <secretKey>iXKQe8qXbhnN0jUe7JGVqFNXMmTxP5pI6example</secretKey>
 <expirationEpoch>1352083995</expirationEpoch> </AWSKEY> Huge wins to this approach as temporary mishandling by clients (writing keys to log files) no longer invokes a mass key rotation, and rotation now happens automatically.
16. On Instance Credentials 
 
 …Or Finally just let AWS Manage The Key Server $curl http://169.254.169.254/latest/meta-data/iam/security-credentials/role { "Code" : "Success", "LastUpdated" : "2015-09-17T01:29:49Z", "Type" : "AWS-HMAC", "AccessKeyId" : "ASIAIL6IJJCXLEXAMPLE", "SecretAccessKey" : "iXKQe8qXbhnN0jUe7JGVqFNXMmTxP5pI6example", "Token" : "...", "Expiration" : "2015-09-17T07:47:45Z" } The preferred approach of today — still not perfect. • Key good everywhere (not just sourced machine) • Developers still out of luck — so we maintain legacy server too* *Arguably better approaches for mimicking this like Hologram: https://github.com/AdRoll/hologram
17. Cryptex Motivation
18. Business Use Cases For Cryptographic Keys Now that we’ve (mostly) outsourced AWS Keys to AWS, we’re left with everything else we need to run our business. • Certificate Authority Private Keys • Keys to Encrypt Sensitive Data • HMAC API Keys to Sign REST requests to Partners • Long term, Short term • Symmetric, Asymmetric
19. Can you really trust a VM? Plenty of research exists to imply that keys are not safe in a VM that shares resources with an untrusted party. Some light reading as an example: Cross-VM Side Channels and Their Use to Extract Private Keys
 Available at: http://www.cs.unc.edu/~reiter/papers/2012/CCS.pdf A seemingly limitless pool of virtual hardware offers unprecedented benefits of scale, throughput, and availability. But….
20. Use Case of a Key Implies Handling Requirements TLS Session Key - Fast, Handled in Dynamic Environment
 • But easy to have a reasonable policy if we lose it Certificate Authority Private Key - Maybe not used so much • Probably way more important that you just don’t lose it
21. Simple Framework for Key Handling Throughput Protection It’s Exposed! It lives… Low Sensitivity High Low No biggie In lots of VMs Medium Sensitivity Medium Medium It’ll be a long week. In very few VMs High Sensitivity Low High No. Just. No. In Special Hardware
22. Cryptex Architecture
23. Cryptex - Our Framework for Key Handling Eureka Server(s) Eureka Server(s) Cryptex Server(s) Web Server Logic Netflix Business Application Cryptex Client Library Netflix IPC Components (Ribbon/Hystrix/etc) Many of these Not Many of these Cloud HSMs - Dedicated Hardware
24. “Low” Key Handling Cryptex Client Library Netflix Business Application Cryptex Server GetKey(ID=123) Resp(Value=iXKQ…) Client Auth TLS Encrypt/Decrypt Key Exported Out to Every Client • Extremely High Throughput • Client Library Attempts to be Mindful of Key Handling
25. “Medium” Key Handling Every Operation is a REST Call • Luckily we don’t have many bulk encrypt use cases for these • Cryptex servers not publicly facing; ostensibly harder to get onto Cryptex Client Library Netflix Business Application Cryptex Server GetKey(ID=456) Resp(Value=null) Client Auth TLS Encrypt(ID=456,PT=…) Resp(CT=5pI6…)
26. “High” Key Handling Cryptex Server Cryptex Client Library Netflix Business Application GetKey(ID=789) Resp(Value=null) Client Auth TLS Encrypt(ID=789,PT=…) Resp(CT=JGVqF…) HSM API Encrypt(ID=789,PT=…) Resp(CT=JGVqF…) Every Operation is a call to specialized hardware • HSM API challenging relative to REST calls (only Cryptex does it) • Very constrained throughput;VM side channel attacks negated
27. “Asymmetric” Key Handling Cryptex Client Library Netflix Business Application Cryptex Server GetKey(ID=111) Resp(PubValue=iXKQ…) Client Auth TLS Verify We support the basics: AES, HMAC-SHA, RSA • Optimize RSA verify/encrypt by pushing public key to edge • At scale computational intensity of RSA quite apparent
28. The ongoing struggle with integration // legacy software assumes long running and managed machines // with long lasting hand deployed filesystems, not dynamic key loading -Djavax.net.ssl.keyStore=<file pointer to key> Many OSS products provide only this mechanism for key loading • Filesystem-like integration with these is current/future work • When making your OSS API, consider a stream instead of a file
29. Upcoming and Future We are eagerly awaiting better new features to leverage • AWS KMS* — great potential; not ready for us yet • AES Symmetric Only • Not Cross Regional • Extremely constrained throughput (hundreds of ops/sec per zone) • We’re hopeful someday it will make our own products obsolete *https://aws.amazon.com/kms/ We’d love to OSS more (wanna help?)
30. Questions? bryanp@netflix.com https://www.linkedin.com/in/bdpayne