http://AssimilationSystems.com/ 1/16 Distributing Credentials (Secrets) Distributing Credentials (Secrets) Securely to Containers Securely to Containers Alan Robertson Charter Communications NaaS Project alan.robertson@charter.com @OSSAlanR Founder: Linux-HA (Pacemaker) project Assimilation Project

http://AssimilationSystems.com/ 2/16 Agenda Agenda ● What’s a secret? ● What does it mean to protect a secret? ● So what’s the problem? ● Two solutions: – Custodia – Custom Code

http://AssimilationSystems.com/ 3/16 What’s a Secret? What’s a Secret? ● These kinds of things are often secrets: – Passwords, Application Tokens, Secret keys, AWS keys, etc. ● You’re storing them so machines (and humans) have access to them ● It might be something small and internal ● It might be something a bit more important ● It might be the “Keys To The Kingdom”

http://AssimilationSystems.com/ 4/16 What does protecting a secret What does protecting a secret mean? mean? ● Don’t store them in plain text on disk ● Don’t pass them as command line parameters, or environment variables ● Don’t store them in GitHub... ● This is harder than it sounds...

http://AssimilationSystems.com/ 5/16 So what’s the problem? So what’s the problem? ● Microservices are by definition network-facing ● If you cannot protect your Vault key, then your secrets are at risk ● If an attacker compromises your application, they probably have access to your vault key... ● If you could protect your Vault key, you could probably protect your secrets ● Vault software helps but is not enough ● We need to be able to protect our vault keys!

http://AssimilationSystems.com/ 6/16 What to do? What to do? ● Don’t store vault keys in your containers ● Identify and validate your application without using keys they need to protect ● How to do this?

http://AssimilationSystems.com/ 7/16 UNIX sockets to the rescue! UNIX sockets to the rescue! ● UNIX domain sockets allow authentication of the application that’s connecting – User ID – Group ID – Process ID – Security Context (SE Linux)

http://AssimilationSystems.com/ 8/16 UNIX Sockets downside UNIX Sockets downside ● This information comes from the process that initially connected ● There is an attack based on this “freezing” ● It involves forking lots of processes and wrapping around pids ● It will likely fail many, many times before succeeding ● It’s much harder to carry out in a container environment ● The shorter the interval between connection and authentication, the harder this is to carry out ● In the worst case, it’s still much harder to subvert than an environment variable or a file ● It cannot subvert the security context information

http://AssimilationSystems.com/ 9/16 This helps you get even more This helps you get even more information from information from ● /proc ● Docker inspect This gives you something similar to the DNA of the caller – the cyber analog of biometrics Authenticate your caller with just this information

http://AssimilationSystems.com/ 10/16 /proc interesting things /proc interesting things ● /proc//cgroup – identifies the container ● /proc//exe - full pathname client binary ● /proc//uid_map (and gid_map) ● /proc//status – uid/gid info, ppid ● /proc//cmdline - client command line – can be modified by the application

http://AssimilationSystems.com/ 11/16 Docker inspect interesting things Docker inspect interesting things ● Name - the name of the image ● Path - full pathname of the container's "init" process ● Args - the arguments given to the init process in the container ● State.Pid - the process id of the container's "init" process ● Config.Hostname - hostname of this container ● Config.Image - the image name that client is running in ● Image - the SHA256 sum of "Config.Image"

http://AssimilationSystems.com/ 12/16 Red Hat’s Custodia Red Hat’s Custodia

http://AssimilationSystems.com/ 13/16 Could it be simpler? Could it be simpler? ● Custodia: – Proxies the entire secret management infrastructure – Duplicate configuration with Vault – Larger attack surface (even though it’s good code) ● What if you just provided vault keys? – Each client just gets a vault key when authenticated – Clients use key to connect directly to vault

http://AssimilationSystems.com/ 14/16 An alternative approach An alternative approach

http://AssimilationSystems.com/ 15/16 What’s Interesting? What’s Interesting? ● AuthProxy does all its work immediately – short interval – makes the attack harder ● AuthProxy doesn’t read any data from client – This eliminates several classes of attacks ● Total AuthProxy code is small ~ 300 lines ● Much harder to subvert than environment variables or files to store keys in

http://AssimilationSystems.com/ 16/16 References References ● Custodia article: – https://peerlyst.com/posts/sharing-secrets-with- containers-using-custodia-alan-robertson ● Authproxy article: – https://peerlyst.com/posts/the-authproxy-method- of-sharing-secrets-safely-with-containers-alan- robertson