Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Distributing Credentials (Secrets) to Containers

Distributing Credentials (Secrets) to Containers

Talk on distributing safely distributing secrets to containers

Alan Robertson

January 25, 2018

More Decks by Alan Robertson

Other Decks in Technology


  1. http://AssimilationSystems.com/
    Distributing Credentials (Secrets)
    Distributing Credentials (Secrets)
    Securely to Containers
    Securely to Containers
    Alan Robertson
    Charter Communications
    NaaS Project
    [email protected]
    Founder: Linux-HA (Pacemaker) project
    Assimilation Project

    View Slide

  2. http://AssimilationSystems.com/

    What’s a secret?

    What does it mean to protect a secret?

    So what’s the problem?

    Two solutions:
    – Custodia
    – Custom Code

    View Slide

  3. http://AssimilationSystems.com/
    What’s a Secret?
    What’s a Secret?

    These kinds of things are often secrets:
    – Passwords, Application Tokens, Secret keys, AWS
    keys, etc.

    You’re storing them so machines (and humans)
    have access to them

    It might be something small and internal

    It might be something a bit more important

    It might be the “Keys To The Kingdom”

    View Slide

  4. http://AssimilationSystems.com/
    What does protecting a secret
    What does protecting a secret

    Don’t store them in plain text on disk

    Don’t pass them as command line parameters,
    or environment variables

    Don’t store them in GitHub...

    This is harder than it sounds...

    View Slide

  5. http://AssimilationSystems.com/
    So what’s the problem?
    So what’s the problem?

    Microservices are by definition network-facing

    If you cannot protect your Vault key, then your secrets are
    at risk

    If an attacker compromises your application, they probably
    have access to your vault key...

    If you could protect your Vault key, you could probably
    protect your secrets

    Vault software helps but is not enough

    We need to be able to protect our vault keys!

    View Slide

  6. http://AssimilationSystems.com/
    What to do?
    What to do?

    Don’t store vault keys in your containers

    Identify and validate your application without
    using keys they need to protect

    How to do this?

    View Slide

  7. http://AssimilationSystems.com/
    UNIX sockets to the rescue!
    UNIX sockets to the rescue!

    UNIX domain sockets allow authentication of
    the application that’s connecting
    – User ID
    – Group ID
    – Process ID
    – Security Context (SE Linux)

    View Slide

  8. http://AssimilationSystems.com/
    UNIX Sockets downside
    UNIX Sockets downside

    This information comes from the process that initially connected

    There is an attack based on this “freezing”

    It involves forking lots of processes and wrapping around pids

    It will likely fail many, many times before succeeding

    It’s much harder to carry out in a container environment

    The shorter the interval between connection and authentication, the
    harder this is to carry out

    In the worst case, it’s still much harder to subvert than an environment
    variable or a file

    It cannot subvert the security context information

    View Slide

  9. http://AssimilationSystems.com/
    This helps you get even more
    This helps you get even more
    information from
    information from


    Docker inspect
    This gives you something similar to the DNA of
    the caller – the cyber analog of biometrics
    Authenticate your caller with just this information

    View Slide

  10. http://AssimilationSystems.com/
    /proc interesting things
    /proc interesting things

    /proc//cgroup – identifies the container

    /proc//exe - full pathname client binary

    /proc//uid_map (and gid_map)

    /proc//status – uid/gid info, ppid

    /proc//cmdline - client command line –
    can be modified by the application

    View Slide

  11. http://AssimilationSystems.com/
    Docker inspect interesting things
    Docker inspect interesting things

    Name - the name of the image

    Path - full pathname of the container's "init" process

    Args - the arguments given to the init process in the

    State.Pid - the process id of the container's "init" process

    Config.Hostname - hostname of this container

    Config.Image - the image name that client is running in

    Image - the SHA256 sum of "Config.Image"

    View Slide

  12. http://AssimilationSystems.com/
    Red Hat’s Custodia
    Red Hat’s Custodia

    View Slide

  13. http://AssimilationSystems.com/
    Could it be simpler?
    Could it be simpler?

    – Proxies the entire secret management infrastructure
    – Duplicate configuration with Vault
    – Larger attack surface (even though it’s good code)

    What if you just provided vault keys?
    – Each client just gets a vault key when authenticated
    – Clients use key to connect directly to vault

    View Slide

  14. http://AssimilationSystems.com/
    An alternative approach
    An alternative approach

    View Slide

  15. http://AssimilationSystems.com/
    What’s Interesting?
    What’s Interesting?

    AuthProxy does all its work immediately –
    short interval – makes the attack harder

    AuthProxy doesn’t read any data from client
    – This eliminates several classes of attacks

    Total AuthProxy code is small ~ 300 lines

    Much harder to subvert than environment
    variables or files to store keys in

    View Slide

  16. http://AssimilationSystems.com/

    Custodia article:
    – https://peerlyst.com/posts/sharing-secrets-with-

    Authproxy article:
    – https://peerlyst.com/posts/the-authproxy-method-

    View Slide