How Netflix Gives All Its Engineers SSH Access To Instances Running In Production

Slide 1

Slide 1 text

How Netﬂix Gives All Its Engineers SSH Access To Instances Running In Production Russell Lewis Product and Application Security Team

Slide 2

Slide 2 text

Dangers Of A Stolen SSH Key • Insider stole funds • Changed all their keys • Rebuilt their whole infrastructure on a new cloud provider • Hacked again • Server logs had been wiped • One stolen SSH key enabled subsequent thefts, from an installed backdoor Shapeshift, a cryptocurrency exchange startup, was hacked three times in a month. http://bit.ly/1T23WJG

Slide 3

Slide 3 text

Table of Contents 1. What Do We Need? 2. Industry Practices 3. The Netﬂix Approach 4. Detecting Undesirable SSH Usage 5. How Do We Issue SSH Certiﬁcates?

Slide 4

Slide 4 text

SSH Key Management • Private keys can get lost, stolen, or shared • Replacing a key means updating all your servers • Can you easily update every server? • Do you have an up to date inventory of all enabled SSH keys in your organization? • How do you know if there isn’t a backdoor key? • Do you rely on SSH to update all of those servers? “Many organizations don't even know how many SSH keys they have conﬁgured to grant access to their information systems or who has copies of those keys” - NISTIR 7966

Slide 5

Slide 5 text

How Can You Limit That Danger?

Slide 6

Slide 6 text

What About Single Use SSH Keys?

Slide 7

Slide 7 text

What If It Left Great Clues Behind?

Slide 8

Slide 8 text

How Can We Protect Server Access?

Slide 9

Slide 9 text

Is Anybody Doing This? Will their solution work for us?

Slide 10

Slide 10 text

Freedom & Responsibility • Share information openly and proactively • Context, not control • You build it, you run it How can we work like a startup, but with all the responsibilities of a big company?

Slide 11

Slide 11 text

So How Do We Secure Things? • Feature, not friction • Deﬁne secure defaults • Automated scanning • Secret management • Alerting and reports

Slide 12

Slide 12 text

How Should We Secure SSH?

Slide 13

Slide 13 text

1.What Do We Need? 2. Industry Practices 3. The Netﬂix Approach 4. Detecting Undesirable SSH Usage 5. How Do We Issue SSH Certiﬁcates?

Slide 14

Slide 14 text

SSH Key Protection • How can you protect your private keys? • Hardware SSH Key Protection • How can you verify they are protected?

Slide 15

Slide 15 text

Traditional Access Operator 2 App A Instances App B Instances App C Instances Operator 3 Operator 1

Slide 16

Slide 16 text

Bastion Access Operator 2 Bastion App A Instances App B Instances App C Instances Operator 3 Operator 1

Slide 17

Slide 17 text

1.What Do We Need? 2. Industry Practices 3. The Netﬂix Approach 4. Detecting Undesirable SSH Usage 5. How Do We Issue SSH Certiﬁcates?

Slide 18

Slide 18 text

System Objective • Give developers freedom to access SSH • Gather context without friction • Scan for undesirable SSH usage

Slide 19

Slide 19 text

Bastions • Choke point on the network • Simplify Authentication, using SSO • Log all activity • Users self manage Public Keys • One system to manage for server access tools

Slide 20

Slide 20 text

2FA • Verify intended use • Second factor already provisioned • PAM modules for most 2FA solutions • Discourages undesirable automated workﬂows • SSH Multiplexing to reduce additional challenges

Slide 21

Slide 21 text

SSHD Conﬁg • Secured by default SSHD conﬁg in BaseAMI • Out of the box, just works • Tooling in place to track deployed BaseAMIs

Slide 22

Slide 22 text

What About Single Use SSH Keys?

Slide 23

Slide 23 text

BLESS: Single Serving SSH Certiﬁcates

Slide 24

Slide 24 text

Type: [email protected] user certiﬁcate Public key: RSA-CERT SHA256:BLAH Signing CA: RSA SHA256:BLAH Key ID: "Any ID information you want" Serial: 0 Valid: from 2016-05-19T14:30:00 to 2016-05-19T14:34:00 Principals: host_username Critical Options: source-address 192.168.1.1 force-command /bin/date Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc User or Host certiﬁcates

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

SSH Certificate Authorities • Trust CAs with SSHD Configs • Deploy multiple trusted CAs • Leave some offline • Rotate Regularly • Emergency Preparedness

Slide 31

Slide 31 text

Enable BLESS /etc/ssh/cas.pub: ssh-rsa AAAAB3NzaC1yc2EAAAADAQ… ssh-rsa AAAAB3NzaC1yc2EAAAADAQ… ssh-rsa AAAAB3NzaC1yc2EAAAADAQ… /etc/ssh/sshd_conﬁg : TrustedUserCAKeys /etc/ssh/cas.pub

Slide 32

Slide 32 text

1.What Do We Need? 2. Industry Practices 3. The Netﬂix Approach 4. Detecting Undesirable SSH Usage 5. How Do We Issue SSH Certiﬁcates?

Slide 33

Slide 33 text

What Is Undesirable SSH Usage? • Malicious • Dangerous • Avoidable

Slide 34

Slide 34 text

Detecting Undesirable SSH Usage With as many instances as Netflix runs, administered by all its engineers, finding undesirable SSH sessions could feel like finding a needle in a haystack.

Slide 35

Slide 35 text

Finding A Needle In A Haystack The Mythbusters tested the difﬁculty of this problem with two very different approaches.

Slide 36

Slide 36 text

Shapeshift’s Approach • Had to act fast • Didn’t have tooling in place to help with the compromise • Tore down and rebuilt everything Discard the haystack, buy more hay.

Slide 37

Slide 37 text

Burn It Down! • Simple, in theory • Destructive • Still have a searching problem Burn the haystack and search the remains. Does shutting off SSH access entirely work?

Slide 38

Slide 38 text

Can We Sort The Good From The Bad? • Not overly complex • Built a machine for the searching problem • Use Our Data Pipeline Built a Machine to exploit the different densities of the hay and needles.

Slide 39

Slide 39 text

What SSH Properties Can We Exploit? Start by understanding how your engineers use SSH.

Slide 40

Slide 40 text

Security Intelligence And Response Team • Process SSH Certiﬁcate requests • Process SSH authentication attempts • Process SSH session logs • Reports and Alerts

Slide 41

Slide 41 text

1.What Do We Need? 2. Industry Practices 3. The Netﬂix Approach 4. Detecting Undesirable SSH Usage 5. How Do We Issue SSH Certiﬁcates?

Slide 42

Slide 42 text

Bastion's Lambda Ephemeral Ssh Service

Slide 43

Slide 43 text

What is BLESS? • Python AWS Lambda Function • Constructs SSH Certiﬁcates • Signs SSH Certiﬁcates

Slide 44

Slide 44 text

Why Lambda? • No circular dependencies on Netﬂix ecosystem • Easy to run in special purposed AWS account with the tightest controls • Separate account, separate rate limits • Bootstrap system with AWS KMS • Lambda Secured w/IAM instead of SSH key management • Audit-able • Alias let you manage deployments of new versions easily

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

SSD Auth.log May 19 14:30:00 host sshd[#####]: Accepted publickey for host_username from 192.168.1.1 port ##### ssh2: RSA-CERT ID request[########################### #########] for[developer_username] from[10.0.1.1] command[ssh [email protected]] ssh_key:[RSA 00:00:00:00:de:ad:be:ef00:00:00:00:de:ad:be :ef] ca: [arn:aws:lambda:region:account#:function:na me] valid_to[YYY/MM/DD HH:MM:SS] (serial 0) CA RSA 8b:ad:f0:0d: 00:00:00:00:00:00:00:00:8b:ad:f0:0d May 19 14:30:00 host sshd[#####]: Accepted publickey for host_username from 192.168.1.1 port ##### ssh2: RSA de:ad:be:ef: 00:00:00:00:00:00:00:00:de:ad:be:ef

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

Would This Have Helped Shapeshift? • 2FA on the Bastion • Audit Logs locked away • Key Rotations all the way down