BLESS: Better Security and Ops for SSH Access

BLESS: Better Security and Ops for SSH Access

Presented at QConNY 2017
Video available at https://www.infoq.com/presentations/bless-security-ops-ssh

How can using SSH certificates improve security and simplify operations for instance access at Netflix-scale? How can you smoothly transition existing infrastructure to use SSH Certificates? Netflix created and uses BLESS, an SSH Certificate Authority that runs as an AWS Lambda function and is used to sign SSH public keys. In this talk, you will start by learning about BLESS in general: what it is, how it works, and how you can start using it. Next, we will explore the Netflix BLESS production architecture and how other companies have used BLESS in different ways.

From there, we will dig deeper together to discuss Netflix’s deployment and operational details, leveraging BLESS for security insight, and future plans for authorization improvements. The entire talk will be interactive with demos along the way.

938bca9547ba1cac3e69d80efd67fe6b?s=128

Bryan Payne

June 27, 2017
Tweet

Transcript

  1. Bryan D. Payne, Director of Product Security June 2017 BLESS:

    Better Security and Ops for SSH Access
  2. None
  3. None
  4. None
  5. None
  6. Post by Ryan McGeehan

  7. None
  8. None
  9. None
  10. 1 2 3 4 5 Phishing & Zero Day Attack

    Backdoor Lateral Movement Data Gathering Exfiltrate Several users are targeted by phishing attacks. At least one succeeds. Victim machine is accessed remotely by adversary. Attack elevates access and propagates throughout the network. It exploits any privileges and information discovered along the way. Data is collected, prepared, and staged for exfiltration. Encrypted data is exfiltrated, typically to another compromised system that is external to the organization. Adapted from https:/ /blogs.rsa.com/anatomy-of-an-attack/.
  11. What’s the Problem?

  12. None
  13. LDAP

  14. LDAP

  15. Operator 2 App A Instances App B Instances App C

    Instances Operator 3 Operator 1
  16. Operator 2 Bastion App A Instances App B Instances App

    C Instances Operator 3 Operator 1
  17. What about single use SSH keys?

  18. What if they left great clues behind?

  19. And offered strong protections?

  20. Netflix’s Solution

  21. SSH Authentication

  22. None
  23. None
  24. None
  25. None
  26. Bastion’s Lambda Ephemeral Ssh Service

  27. None
  28. def my_handler(event, context): message = 'Hello {} {}!'.format(event['first_name'], event['last_name']) return

    { 'message' : message } Invoke Lambda ClientContext Lambda Response Status + Payload
  29. Bastion BLESS Invoke BLESS BLESS Response Certificate Certificate Request

  30. Bastion BLESS Invoke BLESS Certificate Request BLESS Response Certificate AWS

    KMS Decrypt SSH CA private key
  31. Bastion BLESS Invoke BLESS Certificate Request BLESS Response Certificate AWS

    KMS Decrypt SSH CA private key Instances SSH with certificate
  32. SSH Certificates

  33. Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA:

    RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc
  34. Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA:

    RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc User or Host Certificates
  35. Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA:

    RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc Control over what is logged by SSHd
  36. Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA:

    RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc Short-lived certs reduce risk
  37. Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA:

    RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc Valid for a single target (account, app, username, etc)
  38. Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA:

    RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc Valid from a single host
  39. Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA:

    RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc Control what the SSH session can be used for
  40. Scoping Credentials

  41. Bastion BLESS Instances Developer Access to Bastion == Access to

    Instances
  42. Bastion BLESS Bar App Developer App Defines Access List Foo

    App
  43. Bastion BLESS Bar App Developer Foo App App Defines Multiple

    Roles
  44. Type: ssh-rsa-cert-v01@openssh.com user certificate Public key: RSA-CERT SHA256:BLAH Signing CA:

    RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc instance_user:aws_account:app_name
  45. # Entries to enable BLESS TrustedUserCAKeys /etc/ssh/bless_user_ssh_cas.pub AuthorizedPrincipalsFile /etc/ssh/authorized_principals/%u Config

    File /etc/ssh/sshd_config
  46. bless_demo_instances:bless_demo_instances:123456789012:i-18badf00ddeadbeef Config File /etc/ssh/authorized_principals/blessdemo

  47. Operational Wins

  48. Bastion BLESS Invoke BLESS Certificate Request BLESS Response Certificate AWS

    KMS Decrypt SSH CA private key Instances SSH with certificate
  49. 5. AWS ssh tool: Use session credentials to request a

    certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs
  50. Key Secrecy Personal Keys Expiration Shared Keys

  51. Key Rotation vs Human Machine

  52. Logging Context Jun 22 00:20:55 bless-demo- instances-i-0123456789abcde sshd[####]: Accepted publickey

    for bless_demo_instances from 192.168.1.1 port ##### ssh2: RSA-CERT ID request[##################] for[user_name] from[10.0.1.1] command[test:us- east-1:bless_demo_instances:bles s_demo_instances-v001:oq-ssh] ssh_key[RSA de:ad:be:ef: 00:00:00:00:00:de:ad:be] ca[arn:aws:lambda:region:account :function:name] valid_to[2017/06/22 00:25:53] (serial 0) CA RSA SHA256:8badf00d000000008bad Jun 22 00:20:34 bless-demo- instances-i-0123456789abcde sshd[####]: Accepted publickey for bless_demo_instances from 192.168.1.1 port ##### ssh2: RSA SHA256:de:ad:be:ef: 00:00:00:00:00:de:ad:be Traditional SSH certificates with BLESS
  53. Availability Wins LDAP

  54. Yes, It’s Open Source!

  55. https:/ /github.com/Netflix/bless

  56. https:/ /github.com/Netflix/bless

  57. https:/ /github.com/Netflix/bless

  58. https:/ /github.com/Netflix/bless

  59. https:/ /github.com/Netflix/bless

  60. Demo Time

  61. User Experience

  62. 5. AWS ssh tool: Use session credentials to request a

    certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs
  63. 5. AWS ssh tool: Use session credentials to request a

    certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs
  64. None
  65. Bastion Using BLESS

  66. 5. AWS ssh tool: Use session credentials to request a

    certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs
  67. 5. AWS ssh tool: Use session credentials to request a

    certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs
  68. None
  69. Instance SSHd Setup

  70. 5. AWS ssh tool: Use session credentials to request a

    certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs
  71. 5. AWS ssh tool: Use session credentials to request a

    certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs
  72. None
  73. Related Work • Lyft ‣ Uses BLESS with client that

    runs on laptops ‣ https:/ /eng.lyft.com/blessing-your-ssh-at-lyft-a1b38f81629d • Facebook ‣ Leverages signed certificates with principals ‣ https:/ /code.facebook.com/posts/365787980419535/scalable-and-secure-access-with-ssh/ • Wikimedia ‣ SSH-agent proxy to protect private key on bastion ‣ https:/ /blog.wikimedia.org/2017/03/22/keyholder/
  74. Questions? bryanp@netflix.com https:/ /bryanpayne.org [PS… I’m hiring!]