PKI at Scale Using Short-Lived Certificates

This talk was originally given at Engima 2016.
Video available at https://www.youtube.com/watch?v=7YPIsbz8Pig

While TLS is considered a “best practice” for security, deploying the underlying PKI at scale for cloud applications presents many challenges. This starts with the need to securely bootstrap secrets into each instance. The challenges continue at runtime with the need for insight into the continued trustworthiness of each instance. Unfortunately, in practice, it can be difficult to deploy and maintain such a PKI. In an effort to solve both scale and management challenges, some advocate for the use of short-­lived certificates in lieu of revocation lists. The idea is that a compromised private key is less valuable because it will only work for a limited timespan. But what is really required to deploy such a system?

This talk will take a deep dive into the world of PKI deployments at scale. We will start with a brief overview of PKIs in general before drilling into the specific use case of protecting internally facing microservices using TLS with mutual authentication. From here we will explore the pros and cons of using short-­lived certificates. Then we will look at the operational challenges around such deployments, including scaling certificate authority services, handling reloading of certificates into services at run­time, and determining if an instance is trustworthy enough to receive renewed credentials. We will close with some parting thoughts about the remaining challenges in this space.

Transcript

1. PKI at Scale Using Short-Lived Certificates Bryan D. Payne Engineering Manager, Platform Security
2. weeks <6 months 6-12 months >1 year *Notified via extortion attempt 2 weeks 3 weeks 1 month 3 months 4 months 8 months 17 months 13 months
3. Elastic Load Balancers Web Service Web Service Web Service Web Service . . . Internet Cloud / Data Center / Etc
4. Securely Deploy Certificate / Key Communicate Securely API & UI for Certificate Creation Lemur Get Certificate & Key Public CA Private CA CloudCA Seal Secrets Metatron Deployment Management Spinnaker Version Control Git AMI Server with TLS Karyon Tomcat Apache Metatron Client with TLS Ribbon Metatron
5. Securely Deploy Certificate / Key Communicate Securely API & UI for Certificate Creation Lemur Get Certificate & Key Public CA Private CA CloudCA Seal Secrets Metatron Deployment Management Spinnaker Version Control Git AMI Server with TLS Karyon Tomcat Apache Metatron Client with TLS Ribbon Metatron
6. Securely Deploy Certificate / Key Communicate Securely API & UI for Certificate Creation Lemur Get Certificate & Key Public CA Private CA CloudCA Seal Secrets Metatron Deployment Management Spinnaker Version Control Git AMI Server with TLS Karyon Tomcat Apache Metatron Client with TLS Ribbon Metatron
7. Revocation Is Hard
8. CRL (rfc2459) OCSP (rfc2560) OCSP stapling (rfc6066) OCSP must staple (draft-hallambaker-muststaple-00)
9. CRL: Certificate Revocation List Browser Web Server (Content) Web Server (CRL) Certificate Authority Update Internet CRL Cache 1: TLS Handshake 2: Check CRL
10. CRL (rfc2459) OCSP (rfc2560) OCSP stapling (rfc6066) OCSP must staple (draft-hallambaker-muststaple-00)
11. OCSP: Online Certificate Status Protocol Browser Web Server (Content) OCSP Responder Certificate Authority Update Internet 1: TLS Handshake 2: Get Certificate Status
12. CRL (rfc2459) OCSP (rfc2560) OCSP stapling (rfc6066) OCSP must staple (draft-hallambaker-muststaple-00)
13. OCSP Stapling Browser Web Server (Content) Certificate Authority Update Internet 1: TLS Handshake 2: Return Certificate Status OCSP Responder
14. CRL (rfc2459) OCSP (rfc2560) OCSP stapling (rfc6066) OCSP must staple (draft-hallambaker-muststaple-00)
15. OCSP must-staple OCSP staple OCSP CRL Java C Python JavaScript M Georgiev et al., “The most dangerous code in the world: validating SSL certificates in non-browser software”, In Proceedings of ACM CCS, 2012.
17. Photo Credit: Kayamon (CC BY-SA 3.0) https://en.wikipedia.org/wiki/File:Penny_Harvest_Field_2007.jpg
18. Short-Lived Certificates • R Rivest, “Can We Eliminate Certificate Revocation Lists?”, In Proceedings of Financial Cryptography, 1998. • E Topalovic et al., “Towards Short-Lived Certificates”, In Proceedings of IEEE Oakland Web 2.0 Security and Privacy (W2SP), 2012. 6 months 3 months 1 month 1 week 4 days 4 Hours
19. Photo Credit: Bhernandez (CC BY 2.0) https://www.flickr.com/photos/kennyuhh/2917293212
21. AWS HMAC Generation Not real secret keys, sorry. Lifecycle of AccessKeyID and SecretKey is of utmost interest here. AKIAIOSFODNN7EXAMPLE:iX KQe8qXbhnN0jUe7JGVqFNXM mTxP5pI6example DELETE\n \n \n Tue, 27 Mar 2007 21:20:26 +0000\n /johnsmith/photos/puppy.jpg AccessKeyID and SecretKey HMAC- SHA-1 Customer Request lx3byBScXR6KzyMaifNkardMwNk Digest Verified by AWS
22. Circa 2012: AWS SDKs Introduce the Provider Paradigm // provider paradigm dynamically asks for keys every time
    AWSCredentialsProvider prov = new AWSCredentialsProvider(){ public AWSCredentials getCredentials(){ RESTfulObj AWSKey = RESTService.get(“server/getAWSKey”); return new BasicAWSCredentials(
    AWSKey.getAccessID(), AWSKey.getSecretKey()); } }; AmazonSimpleDBClient client = new AmazonSimpleDBClient(prov); client.listDomains(); The client object in the above code example no longer caches keys.
23. On Instance Credentials $curl http://169.254.169.254/latest/meta-data/iam/security-credentials/role { "Code" : "Success", "LastUpdated" : "2015-09-17T01:29:49Z", "Type" : "AWS-HMAC", "AccessKeyId" : "ASIAIL6IJJCXLEXAMPLE", "SecretAccessKey" : "iXKQe8qXbhnN0jUe7JGVqFNXMmTxP5pI6example", "Token" : "...", "Expiration" : "2015-09-17T07:47:45Z" }
24. Alice Alice Bob Bob ID Proof + Credential Request New Short-Lived Credential Validate ID Generate Credential
25. Don’t use short-lived cred to get updated cred! Alice Alice Bob Bob ID Proof + Credential Request New Short-Lived Credential Validate ID Generate Credential
26. Linux Kernel (with AppArmor or SELinux) Credential Management Process Service with TLS Short-Lived Certificate and Key Files (write) (read) TLS Session System Identity Credentials (TPM or SGX) Credential Renewal Protocol
27. nel (with AppArmor or SELinux) Service with TLS Short-Lived Certificate and Key Files (write) (read) TLS Session Loading new certificates into service… • Send signal to service • Restart service • Design service to reload certificates periodically
28. How to load a new certificate and private key? Zero downtime? Apache graceful restart Maybe Nginx reload Yes Tomcat restart No HAProxy reload No Stunnel HUP No Ghostunnel SIGUSR1 Yes
29. Develop & Deploy Code Communicate Securely Provision Credentials at Startup API & UI for Certificate Creation Lemur Public CA Private CA CloudCA Initialize Secrets Metatron Deployment Management Spinnaker Version Control Git AMI Server with TLS Karyon Tomcat Nginx Metatron Client with TLS Ribbon Metatron
30. Develop & Deploy Code Communicate Securely Provision Credentials at Startup API & UI for Certificate Creation Lemur Public CA Private CA CloudCA Initialize Secrets Metatron Deployment Management Spinnaker Version Control Git AMI Server with TLS Karyon Tomcat Nginx Metatron Client with TLS Ribbon Metatron
31. Develop & Deploy Code Communicate Securely Provision Credentials at Startup API & UI for Certificate Creation Lemur Public CA Private CA CloudCA Initialize Secrets Metatron Deployment Management Spinnaker Version Control Git AMI Server with TLS Karyon Tomcat Nginx Metatron Client with TLS Ribbon Metatron
32. Develop & Deploy Code Communicate Securely Provision Credentials at Startup API & UI for Certificate Creation Lemur Public CA Private CA CloudCA Initialize Secrets Metatron Deployment Management Spinnaker Version Control Git AMI Server with TLS Karyon Tomcat Nginx Metatron Client with TLS Ribbon Metatron
33. Long-Lived Certificates Short-Lived Certificates • Improve attack detection, in practice • Retrofit your applications to support revocation • Refresh certificates • Update server / client to support graceful reloading of certificates
34. From Vision to Reality…
35. Questions? bryanp@netflix.com http://bryanpayne.org [PS… I’m hiring!]