How Netflix Gives All Its Engineers SSH Access To Instances Running In Production

How Netflix Gives All Its Engineers SSH Access To Instances Running In Production

One of the ways Netflix enables engineering velocity is with the Freedom and Responsibility culture that empowers individuals with the freedom to do what is needed to get the job done. As a result, the security teams at Netflix focus on reducing developer friction, making it easy to do the right thing, and then rely on auditing, automated analysis, and alerting to keep things safe. This talk begins with a review of few approaches used in the industry to secure SSH bastions (aka jumpboxes), and evaluates them through the lense of our Netflix security culture.

With the industry norms as the backdrop, we’ll explain why Netflix decided it needed to build something new to enhance SSH bastion security. We needed something that was low friction for engineers, but would allow for additional security features to be added in behind the scenes.

We’ll review our SSH bastion architecture, which at its core uses SSO to authenticate engineers, and then issues per user credentials with short lived certificates for SSH authentication of the bastion to an instance. These short lived credentials reduce the risk associated them being lost. We’ll cover how this approach allows us to audit and automatically alert after the fact, instead of slowing down engineers before granting access.

Lastly, we’ll present the SSH Certificate Authority at the core of this system. It runs as an Amazon Web Services Lambda function, and protects its private key with AWS’s Key Management Service. By relying only on AWS services, the SSH Certificate Authority is easy to bring up, and can be used to bootstrap Netflix’s cloud deployments without adding circular dependencies. Additionally, Netflix announced the open sourcing of BLESS; the Bastion's Lambda Ephemeral Ssh Service.


Russell Lewis

May 19, 2016