$30 off During Our Annual Pro Sale. View Details »

Helm Summit PDX 2018 - Securing Helm

Helm Summit PDX 2018 - Securing Helm

Lachlan Evenson

February 21, 2018

More Decks by Lachlan Evenson

Other Decks in Technology


  1. None
  2. Lachlan Evenson Principal Program Manager – Azure Containers @LachlanEvenson Securing

  3. “The only truly secure system is one that is powered

    off, cast in a block of concrete and sealed in a lead-lined room with armed guards.” – Gene Spafford
  4. Helm security in a nutshell • What does Kubernetes security

    entail? • RBAC • PodSecurityPolicy • NetworkPolicy • Secrets (storage and access) • Certificate Management • User Management • Kubeconfig Management • Role/RoleBinding Management • Source to container trust • Container signing • Secured container runtime • Seccomp • Secure node bootstrapping • Admission controllers • Auditing Helm is only as secure as the cluster as it’s installed on (mostly)
  5. Helm security in a nutshell cont.. So when it comes

    to Helm security let’s not Source: https://www.phrases.org.uk/meanings/put-the-cart-before-the-horse.html
  6. Helm security in a nutshell cont.. Helm Architecture Review •

    The Helm Client • Command-line client for end users. • The Tiller Server • An in-cluster server that interacts with the Helm client, and interfaces with the Kubernetes API server.
  7. Helm Architecture

  8. Helm security in a nutshell cont.. • Tiller is just

    an app running on Kubernetes • Ideal – Tiller performing actions using the rights of the client, instead of the rights of Tiller • Contingent upon the outcome of the Pod Identity Working Group, which has taken on the task of solving the problem in a general way • Solving Helm security is a community effort • Here’s what we can do to secure Helm
  9. Kubernetes RBAC • Turn RBAC on • By default Tiller

    uses the default service account in a namespace $ helm init --service-account <service-account> • The command does not create the associated ServiceAccount/Roles/RoleBindings
  10. Kubernetes RBAC cont.. • Be very careful when doing the

    following $ kubectl -n kube-system create sa tiller $ kubectl create clusterrolebinding tiller -- clusterrole cluster-admin -- serviceaccount=kube-system:tiller $ helm init --service-account tiller • Carefully create your roles • Roles/ClusterRoles • RoleBindings/ClusterRoleBindings
  11. Kubernetes RBAC cont.. • Multi-Tiller installations • Per developer, per

    team, per functional environment
  12. Kubernetes RBAC cont.. • Helm Secure Tiller plugin • https://github.com/michelleN/helm-secure-tiller

    $ helm secure-tiller --namespace dev-team example-profiles/dev-team/ serviceaccount "dev-team-rbac-profile" created rolebinding "dev-team-tiller-binding" created role "dev-team" created
  13. Release Information • By default, Tiller stores release information in

    configMaps $ helm init --override 'spec.template.spec.containers[0].comma nd'='{/tiller,--storage=secret}'
  14. Chart Repos • Use HTTPS • Publish signed charts •

    Helm client supports repo client certs $ helm repo add --help • Chartmuseum supports basic auth • https://github.com/kubernetes-helm/chartmuseum#basic- auth • Quay app registry plugin supports auth • https://coreos.com/blog/quay-application-registry-for- kubernetes.html
  15. gRPC • Tiller supports TLS on gRPC $ helm init

    \ –tiller-tls \ –tiller-tls-verify \ –tiller-tls-ca-cert=ca.pem \ –tiller-tls-cert=cert.pem \ –tiller-tls-key=key.pem \ –service-account=accountname $ helm ls --tls --tls-ca-cert ca.cert.pem --tls-cert helm.cert.pem --tls-key helm.key.pem
  16. Secured Helm Architecture

  17. Secure By Default • What would need to happen to

    make Helm secure “by default”? • Make sure RBAC is enabled • Allow only HTTPS, signed charts with at least basic auth or mTLS • Enforce mTLS on gRPC • Default to secrets for release information • Are these really Helm’s decisions to make? • Is the a more Kubernetes “native” way to tackle this problem? • Could we make Helm more Kubernetes ”native”?
  18. Other efforts • Optional check to restrict charts to target

    namespace • https://github.com/kubernetes/helm/pull/3212 • When enabled, this will reject charts that attempt to install resources outside the target namespace • Helm CRD (Helm 3.0?) • Use RBAC to determine weather a user can perform CRUD operations on a “HelmRelease” custom type • https://engineering.bitnami.com/articles/helm-security.html • https://github.com/wpengine/lostromos
  19. Other efforts cont.. • Kube RBAC proxy • https://github.com/brancz/kube-rbac-proxy @fredbrancz

    • This can ensure that a request is truly authorized, instead of being able to access an application simply because an entity has network access to it. (In the case of Tiller) • https://github.com/brancz/kube-rbac-proxy/issues/1 @mumoshu • Adds gRPC support
  20. Kube RBAC proxy Architecture • Experimental

  21. Other efforts cont.. • Helm Local • https://github.com/adamreese/helm-local • Runs

    Tiller locally which uses the local kubeconfig to access the Kubernetes API $ helm plugin install https://github.com/adamreese/helm-local $ helm local start $ helm local COMMAND
  22. Other efforts cont.. • Tiller-less Helm (Lostromos) • https://github.com/wpengine/lostromos/pull/59 •

    Tiller functionality client-side
  23. What’s next? • This is a community problem. Come help

    solve it • Join the Helm development calls • Plugins are a great way to start vetting/solving problem before trying to get a PR into core • Helm 3.0 proposals tomorrow
  24. References • Securing Helm • https://docs.helm.sh/using_helm/#securing-your-helm-installation • App Registry for

    Quay • https://coreos.com/blog/quay-application-registry-for-kubernetes.html • Secure Deployment of Helm Repositories through Client Side Certificates • https://www.twistlock.com/2017/09/05/secure-deployment-helm- repositories-client-side-certificates/ • Exploring The Security Of Helm • https://engineering.bitnami.com/articles/helm-security.html
  25. None