Slide 1

Slide 1 text

Bryan D. Payne, Director of Product Security June 2017 BLESS: Better Security and Ops for SSH Access

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

Post by Ryan McGeehan

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

1 2 3 4 5 Phishing & Zero Day Attack Backdoor Lateral Movement Data Gathering Exfiltrate Several users are targeted by phishing attacks. At least one succeeds. Victim machine is accessed remotely by adversary. Attack elevates access and propagates throughout the network. It exploits any privileges and information discovered along the way. Data is collected, prepared, and staged for exfiltration. Encrypted data is exfiltrated, typically to another compromised system that is external to the organization. Adapted from https:/ /blogs.rsa.com/anatomy-of-an-attack/.

Slide 11

Slide 11 text

What’s the Problem?

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

LDAP

Slide 14

Slide 14 text

LDAP

Slide 15

Slide 15 text

Operator 2 App A Instances App B Instances App C Instances Operator 3 Operator 1

Slide 16

Slide 16 text

Operator 2 Bastion App A Instances App B Instances App C Instances Operator 3 Operator 1

Slide 17

Slide 17 text

What about single use SSH keys?

Slide 18

Slide 18 text

What if they left great clues behind?

Slide 19

Slide 19 text

And offered strong protections?

Slide 20

Slide 20 text

Netflix’s Solution

Slide 21

Slide 21 text

SSH Authentication

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

Bastion’s Lambda Ephemeral Ssh Service

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

def my_handler(event, context): message = 'Hello {} {}!'.format(event['first_name'], event['last_name']) return { 'message' : message } Invoke Lambda ClientContext Lambda Response Status + Payload

Slide 29

Slide 29 text

Bastion BLESS Invoke BLESS BLESS Response Certificate Certificate Request

Slide 30

Slide 30 text

Bastion BLESS Invoke BLESS Certificate Request BLESS Response Certificate AWS KMS Decrypt SSH CA private key

Slide 31

Slide 31 text

Bastion BLESS Invoke BLESS Certificate Request BLESS Response Certificate AWS KMS Decrypt SSH CA private key Instances SSH with certificate

Slide 32

Slide 32 text

SSH Certificates

Slide 33

Slide 33 text

Type: [email protected] user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc

Slide 34

Slide 34 text

Type: [email protected] user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc User or Host Certificates

Slide 35

Slide 35 text

Type: [email protected] user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc Control over what is logged by SSHd

Slide 36

Slide 36 text

Type: [email protected] user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc Short-lived certs reduce risk

Slide 37

Slide 37 text

Type: [email protected] user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc Valid for a single target (account, app, username, etc)

Slide 38

Slide 38 text

Type: [email protected] user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc Valid from a single host

Slide 39

Slide 39 text

Type: [email protected] user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc Control what the SSH session can be used for

Slide 40

Slide 40 text

Scoping Credentials

Slide 41

Slide 41 text

Bastion BLESS Instances Developer Access to Bastion == Access to Instances

Slide 42

Slide 42 text

Bastion BLESS Bar App Developer App Defines Access List Foo App

Slide 43

Slide 43 text

Bastion BLESS Bar App Developer Foo App App Defines Multiple Roles

Slide 44

Slide 44 text

Type: [email protected] user certificate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc instance_user:aws_account:app_name

Slide 45

Slide 45 text

# Entries to enable BLESS TrustedUserCAKeys /etc/ssh/bless_user_ssh_cas.pub AuthorizedPrincipalsFile /etc/ssh/authorized_principals/%u Config File /etc/ssh/sshd_config

Slide 46

Slide 46 text

bless_demo_instances:bless_demo_instances:123456789012:i-18badf00ddeadbeef Config File /etc/ssh/authorized_principals/blessdemo

Slide 47

Slide 47 text

Operational Wins

Slide 48

Slide 48 text

Bastion BLESS Invoke BLESS Certificate Request BLESS Response Certificate AWS KMS Decrypt SSH CA private key Instances SSH with certificate

Slide 49

Slide 49 text

5. AWS ssh tool: Use session credentials to request a certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs

Slide 50

Slide 50 text

Key Secrecy Personal Keys Expiration Shared Keys

Slide 51

Slide 51 text

Key Rotation vs Human Machine

Slide 52

Slide 52 text

Logging Context Jun 22 00:20:55 bless-demo- instances-i-0123456789abcde sshd[####]: Accepted publickey for bless_demo_instances from 192.168.1.1 port ##### ssh2: RSA-CERT ID request[##################] for[user_name] from[10.0.1.1] command[test:us- east-1:bless_demo_instances:bles s_demo_instances-v001:oq-ssh] ssh_key[RSA de:ad:be:ef: 00:00:00:00:00:de:ad:be] ca[arn:aws:lambda:region:account :function:name] valid_to[2017/06/22 00:25:53] (serial 0) CA RSA SHA256:8badf00d000000008bad Jun 22 00:20:34 bless-demo- instances-i-0123456789abcde sshd[####]: Accepted publickey for bless_demo_instances from 192.168.1.1 port ##### ssh2: RSA SHA256:de:ad:be:ef: 00:00:00:00:00:de:ad:be Traditional SSH certificates with BLESS

Slide 53

Slide 53 text

Availability Wins LDAP

Slide 54

Slide 54 text

Yes, It’s Open Source!

Slide 55

Slide 55 text

https:/ /github.com/Netflix/bless

Slide 56

Slide 56 text

https:/ /github.com/Netflix/bless

Slide 57

Slide 57 text

https:/ /github.com/Netflix/bless

Slide 58

Slide 58 text

https:/ /github.com/Netflix/bless

Slide 59

Slide 59 text

https:/ /github.com/Netflix/bless

Slide 60

Slide 60 text

Demo Time

Slide 61

Slide 61 text

User Experience

Slide 62

Slide 62 text

5. AWS ssh tool: Use session credentials to request a certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs

Slide 63

Slide 63 text

5. AWS ssh tool: Use session credentials to request a certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs

Slide 64

Slide 64 text

No content

Slide 65

Slide 65 text

Bastion Using BLESS

Slide 66

Slide 66 text

5. AWS ssh tool: Use session credentials to request a certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs

Slide 67

Slide 67 text

5. AWS ssh tool: Use session credentials to request a certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs

Slide 68

Slide 68 text

No content

Slide 69

Slide 69 text

Instance SSHd Setup

Slide 70

Slide 70 text

5. AWS ssh tool: Use session credentials to request a certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs

Slide 71

Slide 71 text

5. AWS ssh tool: Use session credentials to request a certificate BLESS AWS KMS 6. BLESS: Decrypt SSH CA private key with KMS 7. BLESS: Generate and sign SSH Certificate BLESS: Log certificate request & results CloudWatch Logs 8. BLESS: Return a short lived certificate Instances 9. AWS ssh tool: ssh with certificate 10. sshd: Validate certificate, log certificate info RELP Server (syslog) Log Forwarder Bastion Daemon User Developer Userspace 3. Pilgrim: Generate Keypair Request SSH Cert 2. AWS SSH tool: Take request, determine user, application, instance 4. Sshaman Daemon: Determine calling user information. Use session credentials to request a certificate. Developer 1. SSH: Auth to Bastion Pilgrim Logs Sshaman Logs sshd Logs

Slide 72

Slide 72 text

No content

Slide 73

Slide 73 text

Related Work • Lyft ‣ Uses BLESS with client that runs on laptops ‣ https:/ /eng.lyft.com/blessing-your-ssh-at-lyft-a1b38f81629d • Facebook ‣ Leverages signed certificates with principals ‣ https:/ /code.facebook.com/posts/365787980419535/scalable-and-secure-access-with-ssh/ • Wikimedia ‣ SSH-agent proxy to protect private key on bastion ‣ https:/ /blog.wikimedia.org/2017/03/22/keyholder/

Slide 74

Slide 74 text

Questions? [email protected] https:/ /bryanpayne.org [PS… I’m hiring!]