Distributing Credentials (Secrets) to Containers

Distributing Credentials (Secrets) to Containers

Talk on distributing safely distributing secrets to containers


Alan Robertson

January 25, 2018


  1. http://AssimilationSystems.com/ 1/16 Distributing Credentials (Secrets) Distributing Credentials (Secrets) Securely to

    Containers Securely to Containers Alan Robertson Charter Communications NaaS Project alan.robertson@charter.com @OSSAlanR Founder: Linux-HA (Pacemaker) project Assimilation Project
  2. http://AssimilationSystems.com/ 2/16 Agenda Agenda • What’s a secret? • What

    does it mean to protect a secret? • So what’s the problem? • Two solutions: – Custodia – Custom Code
  3. http://AssimilationSystems.com/ 3/16 What’s a Secret? What’s a Secret? • These

    kinds of things are often secrets: – Passwords, Application Tokens, Secret keys, AWS keys, etc. • You’re storing them so machines (and humans) have access to them • It might be something small and internal • It might be something a bit more important • It might be the “Keys To The Kingdom”
  4. http://AssimilationSystems.com/ 4/16 What does protecting a secret What does protecting

    a secret mean? mean? • Don’t store them in plain text on disk • Don’t pass them as command line parameters, or environment variables • Don’t store them in GitHub... • This is harder than it sounds...
  5. http://AssimilationSystems.com/ 5/16 So what’s the problem? So what’s the problem?

    • Microservices are by definition network-facing • If you cannot protect your Vault key, then your secrets are at risk • If an attacker compromises your application, they probably have access to your vault key... • If you could protect your Vault key, you could probably protect your secrets • Vault software helps but is not enough • We need to be able to protect our vault keys!
  6. http://AssimilationSystems.com/ 6/16 What to do? What to do? • Don’t

    store vault keys in your containers • Identify and validate your application without using keys they need to protect • How to do this?
  7. http://AssimilationSystems.com/ 7/16 UNIX sockets to the rescue! UNIX sockets to

    the rescue! • UNIX domain sockets allow authentication of the application that’s connecting – User ID – Group ID – Process ID – Security Context (SE Linux)
  8. http://AssimilationSystems.com/ 8/16 UNIX Sockets downside UNIX Sockets downside • This

    information comes from the process that initially connected • There is an attack based on this “freezing” • It involves forking lots of processes and wrapping around pids • It will likely fail many, many times before succeeding • It’s much harder to carry out in a container environment • The shorter the interval between connection and authentication, the harder this is to carry out • In the worst case, it’s still much harder to subvert than an environment variable or a file • It cannot subvert the security context information
  9. http://AssimilationSystems.com/ 9/16 This helps you get even more This helps

    you get even more information from information from • /proc • Docker inspect This gives you something similar to the DNA of the caller – the cyber analog of biometrics Authenticate your caller with just this information
  10. http://AssimilationSystems.com/ 10/16 /proc interesting things /proc interesting things • /proc/<pid>/cgroup

    – identifies the container • /proc/<pid>/exe - full pathname client binary • /proc/<pid>/uid_map (and gid_map) • /proc/<pid>/status – uid/gid info, ppid • /proc/<pid>/cmdline - client command line – can be modified by the application
  11. http://AssimilationSystems.com/ 11/16 Docker inspect interesting things Docker inspect interesting things

    • Name - the name of the image • Path - full pathname of the container's "init" process • Args - the arguments given to the init process in the container • State.Pid - the process id of the container's "init" process • Config.Hostname - hostname of this container • Config.Image - the image name that client is running in • Image - the SHA256 sum of "Config.Image"
  12. http://AssimilationSystems.com/ 12/16 Red Hat’s Custodia Red Hat’s Custodia

  13. http://AssimilationSystems.com/ 13/16 Could it be simpler? Could it be simpler?

    • Custodia: – Proxies the entire secret management infrastructure – Duplicate configuration with Vault – Larger attack surface (even though it’s good code) • What if you just provided vault keys? – Each client just gets a vault key when authenticated – Clients use key to connect directly to vault
  14. http://AssimilationSystems.com/ 14/16 An alternative approach An alternative approach

  15. http://AssimilationSystems.com/ 15/16 What’s Interesting? What’s Interesting? • AuthProxy does all

    its work immediately – short interval – makes the attack harder • AuthProxy doesn’t read any data from client – This eliminates several classes of attacks • Total AuthProxy code is small ~ 300 lines • Much harder to subvert than environment variables or files to store keys in
  16. http://AssimilationSystems.com/ 16/16 References References • Custodia article: – https://peerlyst.com/posts/sharing-secrets-with- containers-using-custodia-alan-robertson

    • Authproxy article: – https://peerlyst.com/posts/the-authproxy-method- of-sharing-secrets-safely-with-containers-alan- robertson