BLESS: Better Security and Ops for SSH Access

Presented at QConNY 2017
Video available at https://www.infoq.com/presentations/bless-security-ops-ssh

How can using SSH certificates improve security and simplify operations for instance access at Netflix-scale? How can you smoothly transition existing infrastructure to use SSH Certificates? Netflix created and uses BLESS, an SSH Certificate Authority that runs as an AWS Lambda function and is used to sign SSH public keys. In this talk, you will start by learning about BLESS in general: what it is, how it works, and how you can start using it. Next, we will explore the Netflix BLESS production architecture and how other companies have used BLESS in different ways.

From there, we will dig deeper together to discuss Netflix’s deployment and operational details, leveraging BLESS for security insight, and future plans for authorization improvements. The entire talk will be interactive with demos along the way.

Transcript

1. Bryan D. Payne, Director of Product Security
   June 2017
   BLESS: Better
   Security and Ops
   for SSH Access
   

   View Slide

2. View Slide

3. View Slide

4. View Slide

5. View Slide

6. Post by Ryan McGeehan
   

   View Slide

7. View Slide

8. View Slide

9. View Slide

10. 1 2 3 4 5
    Phishing &
    Zero Day
    Attack
    Backdoor
    Lateral
    Movement
    Data
    Gathering
    Exfiltrate
    Several users
    are targeted
    by phishing
    attacks. At
    least one
    succeeds.
    Victim machine
    is accessed
    remotely by
    adversary.
    Attack elevates
    access and
    propagates
    throughout the
    network.
    It exploits any
    privileges and
    information
    discovered
    along the way.
    Data is
    collected,
    prepared,
    and staged
    for exfiltration.
    Encrypted data
    is exfiltrated,
    typically to
    another
    compromised
    system that
    is external
    to the
    organization.
    Adapted from https:/
    /blogs.rsa.com/anatomy-of-an-attack/.
    

    View Slide

11. What’s the Problem?
    

    View Slide

12. View Slide

13. LDAP
    

    View Slide

14. LDAP
    

    View Slide

15. Operator 2
    App A
    Instances
    App B
    Instances
    App C
    Instances
    Operator 3
    Operator 1
    

    View Slide

16. Operator 2 Bastion
    App A
    Instances
    App B
    Instances
    App C
    Instances
    Operator 3
    Operator 1
    

    View Slide

17. What about single use SSH keys?
    

    View Slide

18. What if they left
    great clues behind?
    

    View Slide

19. And offered
    strong protections?
    

    View Slide

20. Netflix’s Solution
    

    View Slide

21. SSH Authentication
    

    View Slide

22. View Slide

23. View Slide

24. View Slide

25. View Slide

26. Bastion’s
    Lambda
    Ephemeral
    Ssh
    Service
    

    View Slide

27. View Slide

28. def my_handler(event, context):
    message = 'Hello {} {}!'.format(event['first_name'],
    event['last_name'])
    return {
    'message' : message
    }
    Invoke Lambda
    ClientContext
    Lambda Response
    Status + Payload
    

    View Slide

29. Bastion BLESS
    Invoke BLESS BLESS Response
    Certificate
    Certificate Request
    

    View Slide

30. Bastion BLESS
    Invoke BLESS
    Certificate Request
    BLESS Response
    Certificate
    AWS KMS
    Decrypt SSH CA
    private key
    

    View Slide

31. Bastion BLESS
    Invoke BLESS
    Certificate Request
    BLESS Response
    Certificate
    AWS KMS
    Decrypt SSH CA
    private key
    Instances
    SSH with
    certificate
    

    View Slide

32. SSH Certificates
    

    View Slide

33. Type: ssh-rsa-cert-v01@openssh.com user certificate
    Public key: RSA-CERT SHA256:BLAH
    Signing CA: RSA SHA256:BLAH
    Key ID: "Any ID information you want"
    Serial: 0
    Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00
    Principals:
    host_username
    Critical Options:
    source-address 192.168.1.1
    force-command /bin/date
    Extensions:
    permit-X11-forwarding
    permit-agent-forwarding
    permit-port-forwarding
    permit-pty
    permit-user-rc
    

    View Slide

34. Type: ssh-rsa-cert-v01@openssh.com user certificate
    Public key: RSA-CERT SHA256:BLAH
    Signing CA: RSA SHA256:BLAH
    Key ID: "Any ID information you want"
    Serial: 0
    Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00
    Principals:
    host_username
    Critical Options:
    source-address 192.168.1.1
    force-command /bin/date
    Extensions:
    permit-X11-forwarding
    permit-agent-forwarding
    permit-port-forwarding
    permit-pty
    permit-user-rc
    User or Host
    Certificates
    

    View Slide

35. Type: ssh-rsa-cert-v01@openssh.com user certificate
    Public key: RSA-CERT SHA256:BLAH
    Signing CA: RSA SHA256:BLAH
    Key ID: "Any ID information you want"
    Serial: 0
    Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00
    Principals:
    host_username
    Critical Options:
    source-address 192.168.1.1
    force-command /bin/date
    Extensions:
    permit-X11-forwarding
    permit-agent-forwarding
    permit-port-forwarding
    permit-pty
    permit-user-rc
    Control over what
    is logged by SSHd
    

    View Slide

36. Type: ssh-rsa-cert-v01@openssh.com user certificate
    Public key: RSA-CERT SHA256:BLAH
    Signing CA: RSA SHA256:BLAH
    Key ID: "Any ID information you want"
    Serial: 0
    Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00
    Principals:
    host_username
    Critical Options:
    source-address 192.168.1.1
    force-command /bin/date
    Extensions:
    permit-X11-forwarding
    permit-agent-forwarding
    permit-port-forwarding
    permit-pty
    permit-user-rc
    Short-lived certs
    reduce risk
    

    View Slide

37. Type: ssh-rsa-cert-v01@openssh.com user certificate
    Public key: RSA-CERT SHA256:BLAH
    Signing CA: RSA SHA256:BLAH
    Key ID: "Any ID information you want"
    Serial: 0
    Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00
    Principals:
    host_username
    Critical Options:
    source-address 192.168.1.1
    force-command /bin/date
    Extensions:
    permit-X11-forwarding
    permit-agent-forwarding
    permit-port-forwarding
    permit-pty
    permit-user-rc
    Valid for a single
    target (account, app,
    username, etc)
    

    View Slide

38. Type: ssh-rsa-cert-v01@openssh.com user certificate
    Public key: RSA-CERT SHA256:BLAH
    Signing CA: RSA SHA256:BLAH
    Key ID: "Any ID information you want"
    Serial: 0
    Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00
    Principals:
    host_username
    Critical Options:
    source-address 192.168.1.1
    force-command /bin/date
    Extensions:
    permit-X11-forwarding
    permit-agent-forwarding
    permit-port-forwarding
    permit-pty
    permit-user-rc
    Valid from a
    single host
    

    View Slide

39. Type: ssh-rsa-cert-v01@openssh.com user certificate
    Public key: RSA-CERT SHA256:BLAH
    Signing CA: RSA SHA256:BLAH
    Key ID: "Any ID information you want"
    Serial: 0
    Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00
    Principals:
    host_username
    Critical Options:
    source-address 192.168.1.1
    force-command /bin/date
    Extensions:
    permit-X11-forwarding
    permit-agent-forwarding
    permit-port-forwarding
    permit-pty
    permit-user-rc
    Control what the
    SSH session can
    be used for
    

    View Slide

40. Scoping Credentials
    

    View Slide

41. Bastion BLESS
    Instances
    Developer
    Access to Bastion == Access to Instances
    

    View Slide

42. Bastion BLESS
    Bar App
    Developer
    App Defines Access List
    Foo App
    

    View Slide

43. Bastion BLESS
    Bar App
    Developer
    Foo App
    App Defines Multiple Roles
    

    View Slide

44. Type: ssh-rsa-cert-v01@openssh.com user certificate
    Public key: RSA-CERT SHA256:BLAH
    Signing CA: RSA SHA256:BLAH
    Key ID: "Any ID information you want"
    Serial: 0
    Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00
    Principals:
    host_username
    Critical Options:
    source-address 192.168.1.1
    force-command /bin/date
    Extensions:
    permit-X11-forwarding
    permit-agent-forwarding
    permit-port-forwarding
    permit-pty
    permit-user-rc
    instance_user:aws_account:app_name
    

    View Slide

45. # Entries to enable BLESS
    TrustedUserCAKeys /etc/ssh/bless_user_ssh_cas.pub
    AuthorizedPrincipalsFile /etc/ssh/authorized_principals/%u
    Config File
    /etc/ssh/sshd_config
    

    View Slide

46. bless_demo_instances:bless_demo_instances:123456789012:i-18badf00ddeadbeef
    Config File
    /etc/ssh/authorized_principals/blessdemo
    

    View Slide

47. Operational Wins
    

    View Slide

48. Bastion BLESS
    Invoke BLESS
    Certificate Request
    BLESS Response
    Certificate
    AWS KMS
    Decrypt SSH CA
    private key
    Instances
    SSH with
    certificate
    

    View Slide

49. 5. AWS ssh tool: Use
    session credentials to
    request a certificate
    BLESS
    AWS
    KMS
    6. BLESS: Decrypt SSH CA
    private key with KMS
    7. BLESS: Generate and
    sign SSH Certificate
    BLESS: Log
    certificate request
    & results
    CloudWatch
    Logs
    8. BLESS: Return
    a short lived
    certificate
    Instances
    9. AWS ssh tool:
    ssh with
    certificate
    10. sshd: Validate
    certificate, log
    certificate info
    RELP Server
    (syslog)
    Log
    Forwarder
    Bastion
    Daemon User
    Developer Userspace
    3. Pilgrim:
    Generate Keypair
    Request SSH Cert
    2. AWS SSH tool:
    Take request,
    determine user,
    application,
    instance
    4. Sshaman
    Daemon:
    Determine
    calling user
    information.
    Use session
    credentials to
    request a
    certificate.
    Developer
    1. SSH: Auth to
    Bastion
    Pilgrim Logs
    Sshaman Logs
    sshd Logs
    

    View Slide

50. Key
    Secrecy
    Personal Keys Expiration
    Shared Keys
    

    View Slide

51. Key
    Rotation
    vs
    Human Machine
    

    View Slide

52. Logging
    Context
    Jun 22 00:20:55 bless-demo-
    instances-i-0123456789abcde
    sshd[####]: Accepted publickey
    for bless_demo_instances from
    192.168.1.1 port ##### ssh2:
    RSA-CERT ID
    request[##################]
    for[user_name] from[10.0.1.1]
    command[test:us-
    east-1:bless_demo_instances:bles
    s_demo_instances-v001:oq-ssh]
    ssh_key[RSA de:ad:be:ef:
    00:00:00:00:00:de:ad:be]
    ca[arn:aws:lambda:region:account
    :function:name]
    valid_to[2017/06/22 00:25:53]
    (serial 0) CA RSA
    SHA256:8badf00d000000008bad
    Jun 22 00:20:34 bless-demo-
    instances-i-0123456789abcde
    sshd[####]: Accepted publickey
    for bless_demo_instances from
    192.168.1.1 port ##### ssh2: RSA
    SHA256:de:ad:be:ef:
    00:00:00:00:00:de:ad:be
    Traditional SSH certificates with BLESS
    

    View Slide

53. Availability
    Wins
    LDAP
    

    View Slide

54. Yes, It’s Open Source!
    

    View Slide

55. https:/
    /github.com/Netflix/bless
    

    View Slide

56. https:/
    /github.com/Netflix/bless
    

    View Slide

57. https:/
    /github.com/Netflix/bless
    

    View Slide

58. https:/
    /github.com/Netflix/bless
    

    View Slide

59. https:/
    /github.com/Netflix/bless
    

    View Slide

60. Demo Time
    

    View Slide

61. User Experience
    

    View Slide

62. 5. AWS ssh tool: Use
    session credentials to
    request a certificate
    BLESS
    AWS
    KMS
    6. BLESS: Decrypt SSH CA
    private key with KMS
    7. BLESS: Generate and
    sign SSH Certificate
    BLESS: Log
    certificate request
    & results
    CloudWatch
    Logs
    8. BLESS: Return
    a short lived
    certificate
    Instances
    9. AWS ssh tool:
    ssh with
    certificate
    10. sshd: Validate
    certificate, log
    certificate info
    RELP Server
    (syslog)
    Log
    Forwarder
    Bastion
    Daemon User
    Developer Userspace
    3. Pilgrim:
    Generate Keypair
    Request SSH Cert
    2. AWS SSH tool:
    Take request,
    determine user,
    application,
    instance
    4. Sshaman
    Daemon:
    Determine
    calling user
    information.
    Use session
    credentials to
    request a
    certificate.
    Developer
    1. SSH: Auth to
    Bastion
    Pilgrim Logs
    Sshaman Logs
    sshd Logs
    

    View Slide

63. 5. AWS ssh tool: Use
    session credentials to
    request a certificate
    BLESS
    AWS
    KMS
    6. BLESS: Decrypt SSH CA
    private key with KMS
    7. BLESS: Generate and
    sign SSH Certificate
    BLESS: Log
    certificate request
    & results
    CloudWatch
    Logs
    8. BLESS: Return
    a short lived
    certificate
    Instances
    9. AWS ssh tool:
    ssh with
    certificate
    10. sshd: Validate
    certificate, log
    certificate info
    RELP Server
    (syslog)
    Log
    Forwarder
    Bastion
    Daemon User
    Developer Userspace
    3. Pilgrim:
    Generate Keypair
    Request SSH Cert
    2. AWS SSH tool:
    Take request,
    determine user,
    application,
    instance
    4. Sshaman
    Daemon:
    Determine
    calling user
    information.
    Use session
    credentials to
    request a
    certificate.
    Developer
    1. SSH: Auth to
    Bastion
    Pilgrim Logs
    Sshaman Logs
    sshd Logs
    

    View Slide

64. View Slide

65. Bastion Using BLESS
    

    View Slide

66. 5. AWS ssh tool: Use
    session credentials to
    request a certificate
    BLESS
    AWS
    KMS
    6. BLESS: Decrypt SSH CA
    private key with KMS
    7. BLESS: Generate and
    sign SSH Certificate
    BLESS: Log
    certificate request
    & results
    CloudWatch
    Logs
    8. BLESS: Return
    a short lived
    certificate
    Instances
    9. AWS ssh tool:
    ssh with
    certificate
    10. sshd: Validate
    certificate, log
    certificate info
    RELP Server
    (syslog)
    Log
    Forwarder
    Bastion
    Daemon User
    Developer Userspace
    3. Pilgrim:
    Generate Keypair
    Request SSH Cert
    2. AWS SSH tool:
    Take request,
    determine user,
    application,
    instance
    4. Sshaman
    Daemon:
    Determine
    calling user
    information.
    Use session
    credentials to
    request a
    certificate.
    Developer
    1. SSH: Auth to
    Bastion
    Pilgrim Logs
    Sshaman Logs
    sshd Logs
    

    View Slide

67. 5. AWS ssh tool: Use
    session credentials to
    request a certificate
    BLESS
    AWS
    KMS
    6. BLESS: Decrypt SSH CA
    private key with KMS
    7. BLESS: Generate and
    sign SSH Certificate
    BLESS: Log
    certificate request
    & results
    CloudWatch
    Logs
    8. BLESS: Return
    a short lived
    certificate
    Instances
    9. AWS ssh tool:
    ssh with
    certificate
    10. sshd: Validate
    certificate, log
    certificate info
    RELP Server
    (syslog)
    Log
    Forwarder
    Bastion
    Daemon User
    Developer Userspace
    3. Pilgrim:
    Generate Keypair
    Request SSH Cert
    2. AWS SSH tool:
    Take request,
    determine user,
    application,
    instance
    4. Sshaman
    Daemon:
    Determine
    calling user
    information.
    Use session
    credentials to
    request a
    certificate.
    Developer
    1. SSH: Auth to
    Bastion
    Pilgrim Logs
    Sshaman Logs
    sshd Logs
    

    View Slide

68. View Slide

69. Instance SSHd Setup
    

    View Slide

70. 5. AWS ssh tool: Use
    session credentials to
    request a certificate
    BLESS
    AWS
    KMS
    6. BLESS: Decrypt SSH CA
    private key with KMS
    7. BLESS: Generate and
    sign SSH Certificate
    BLESS: Log
    certificate request
    & results
    CloudWatch
    Logs
    8. BLESS: Return
    a short lived
    certificate
    Instances
    9. AWS ssh tool:
    ssh with
    certificate
    10. sshd: Validate
    certificate, log
    certificate info
    RELP Server
    (syslog)
    Log
    Forwarder
    Bastion
    Daemon User
    Developer Userspace
    3. Pilgrim:
    Generate Keypair
    Request SSH Cert
    2. AWS SSH tool:
    Take request,
    determine user,
    application,
    instance
    4. Sshaman
    Daemon:
    Determine
    calling user
    information.
    Use session
    credentials to
    request a
    certificate.
    Developer
    1. SSH: Auth to
    Bastion
    Pilgrim Logs
    Sshaman Logs
    sshd Logs
    

    View Slide

71. 5. AWS ssh tool: Use
    session credentials to
    request a certificate
    BLESS
    AWS
    KMS
    6. BLESS: Decrypt SSH CA
    private key with KMS
    7. BLESS: Generate and
    sign SSH Certificate
    BLESS: Log
    certificate request
    & results
    CloudWatch
    Logs
    8. BLESS: Return
    a short lived
    certificate
    Instances
    9. AWS ssh tool:
    ssh with
    certificate
    10. sshd: Validate
    certificate, log
    certificate info
    RELP Server
    (syslog)
    Log
    Forwarder
    Bastion
    Daemon User
    Developer Userspace
    3. Pilgrim:
    Generate Keypair
    Request SSH Cert
    2. AWS SSH tool:
    Take request,
    determine user,
    application,
    instance
    4. Sshaman
    Daemon:
    Determine
    calling user
    information.
    Use session
    credentials to
    request a
    certificate.
    Developer
    1. SSH: Auth to
    Bastion
    Pilgrim Logs
    Sshaman Logs
    sshd Logs
    

    View Slide

72. View Slide

73. Related Work
    • Lyft
    ‣ Uses BLESS with client that runs on laptops
    ‣ https:/
    /eng.lyft.com/blessing-your-ssh-at-lyft-a1b38f81629d
    • Facebook
    ‣ Leverages signed certificates with principals
    ‣ https:/
    /code.facebook.com/posts/365787980419535/scalable-and-secure-access-with-ssh/
    • Wikimedia
    ‣ SSH-agent proxy to protect private key on bastion
    ‣ https:/
    /blog.wikimedia.org/2017/03/22/keyholder/
    

    View Slide

74. Questions?
    bryanp@netflix.com
    https:/
    /bryanpayne.org
    [PS… I’m hiring!]
    

    View Slide