BLESS: Better Security and Ops for SSH Access

Presented at QConNY 2017
Video available at https://www.infoq.com/presentations/bless-security-ops-ssh

How can using SSH certificates improve security and simplify operations for instance access at Netflix-scale? How can you smoothly transition existing infrastructure to use SSH Certificates? Netflix created and uses BLESS, an SSH Certificate Authority that runs as an AWS Lambda function and is used to sign SSH public keys. In this talk, you will start by learning about BLESS in general: what it is, how it works, and how you can start using it. Next, we will explore the Netflix BLESS production architecture and how other companies have used BLESS in different ways.

From there, we will dig deeper together to discuss Netflix’s deployment and operational details, leveraging BLESS for security insight, and future plans for authorization improvements. The entire talk will be interactive with demos along the way.

Transcript

1. Bryan D. Payne, Director of Product Security June 2017 BLESS: Better Security and Ops for SSH Access
6. Post by Ryan McGeehan
10. 1 2 3 4 5 Phishing & Zero Day Attack Backdoor Lateral Movement Data Gathering Exfiltrate Several users are targeted by phishing attacks. At least one succeeds. Victim machine is accessed remotely by adversary. Attack elevates access and propagates throughout the network. It exploits any privileges and information discovered along the way. Data is collected, prepared, and staged for exfiltration. Encrypted data is exfiltrated, typically to another compromised system that is external to the organization. Adapted from https:/ /blogs.rsa.com/anatomy-of-an-attack/.
11. What’s the Problem?
13. LDAP
14. LDAP
15. Operator 2 App A Instances App B Instances App C Instances Operator 3 Operator 1
16. Operator 2 Bastion App A Instances App B Instances App C Instances Operator 3 Operator 1
17. What about single use SSH keys?
18. What if they left great clues behind?
19. And offered strong protections?
20. Netflix’s Solution
21. SSH Authentication
26. Bastion’s Lambda Ephemeral Ssh Service
28. def my_handler(event, context): message = 'Hello {} {}!'.format(event['first_name'], event['last_name']) return { 'message' : message } Invoke Lambda ClientContext Lambda Response Status + Payload
29. Bastion BLESS Invoke BLESS BLESS Response Certificate Certificate Request
30. Bastion BLESS Invoke BLESS Certificate Request BLESS Response Certificate AWS KMS Decrypt SSH CA private key
31. Bastion BLESS Invoke BLESS Certificate Request BLESS Response Certificate AWS KMS Decrypt SSH CA private key Instances SSH with certificate
32. SSH Certificates
33. Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc
34. Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc User or Host Certificates
35. Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc Control over what is logged by SSHd
36. Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc Short-lived certs reduce risk
37. Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc Valid for a single target (account, app, username, etc)
38. Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc Valid from a single host
39. Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc Control what the SSH session can be used for
40. Scoping Credentials
41. Bastion BLESS Instances Developer Access to Bastion == Access to Instances
42. Bastion BLESS Bar App Developer App Defines Access List Foo App
43. Bastion BLESS Bar App Developer Foo App App Defines Multiple Roles
44. Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc instance_user:aws_account:app_name
45. # Entries to enable BLESS TrustedUserCAKeys /etc/ssh/bless_user_ssh_cas.pub AuthorizedPrincipalsFile /etc/ssh/authorized_principals/%u Config File /etc/ssh/sshd_config
46. bless_demo_instances:bless_demo_instances:123456789012:i-18badf00ddeadbeef Config File /etc/ssh/authorized_principals/blessdemo
47. Operational Wins
48. Bastion BLESS Invoke BLESS Certificate Request BLESS Response Certificate AWS KMS Decrypt SSH CA private key Instances SSH with certificate
49. 5. AWS ssh tool: Use session credentials to request a certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs
50. Key Secrecy Personal Keys Expiration Shared Keys
51. Key Rotation vs Human Machine
52. Logging Context Jun 22 00:20:55 bless-demo- instances-i-0123456789abcde sshd[####]: Accepted publickey for bless_demo_instances from 192.168.1.1 port ##### ssh2: RSA-CERT ID request[##################] for[user_name] from[10.0.1.1] command[test:us- east-1:bless_demo_instances:bles s_demo_instances-v001:oq-ssh] ssh_key[RSA de:ad:be:ef: 00:00:00:00:00:de:ad:be] ca[arn:aws:lambda:region:account :function:name] valid_to[2017/06/22 00:25:53] (serial 0) CA RSA SHA256:8badf00d000000008bad Jun 22 00:20:34 bless-demo- instances-i-0123456789abcde sshd[####]: Accepted publickey for bless_demo_instances from 192.168.1.1 port ##### ssh2: RSA SHA256:de:ad:be:ef: 00:00:00:00:00:de:ad:be Traditional SSH certificates with BLESS
53. Availability Wins LDAP
54. Yes, It’s Open Source!
55. https:/ /github.com/Netflix/bless
56. https:/ /github.com/Netflix/bless
57. https:/ /github.com/Netflix/bless
58. https:/ /github.com/Netflix/bless
59. https:/ /github.com/Netflix/bless
60. Demo Time
61. User Experience
62. 5. AWS ssh tool: Use session credentials to request a certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs
63. 5. AWS ssh tool: Use session credentials to request a certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs
65. Bastion Using BLESS
66. 5. AWS ssh tool: Use session credentials to request a certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs
67. 5. AWS ssh tool: Use session credentials to request a certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs
69. Instance SSHd Setup
70. 5. AWS ssh tool: Use session credentials to request a certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs
71. 5. AWS ssh tool: Use session credentials to request a certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs
73. Related Work • Lyft ‣ Uses BLESS with client that runs on laptops ‣ https:/ /eng.lyft.com/blessing-your-ssh-at-lyft-a1b38f81629d • Facebook ‣ Leverages signed certificates with principals ‣ https:/ /code.facebook.com/posts/365787980419535/scalable-and-secure-access-with-ssh/ • Wikimedia ‣ SSH-agent proxy to protect private key on bastion ‣ https:/ /blog.wikimedia.org/2017/03/22/keyholder/
74. Questions? bryanp@netflix.com https:/ /bryanpayne.org [PS… I’m hiring!]