Upgrade to Pro — share decks privately, control downloads, hide ads and more …

KubernetesとHashiCorp Vaultで作るCloudNativeな秘密情報管理 / Secrets Management for Kubernetes with HashiCorp Vault

5d769d109697012317c09c6a27a6a4bf?s=47 linyows
April 16, 2019

KubernetesとHashiCorp Vaultで作るCloudNativeな秘密情報管理 / Secrets Management for Kubernetes with HashiCorp Vault

"KubernetesとHashiCorp Vaultで作るCloudNativeな秘密情報管理" Apr 16, 2019 @CloudNative Days Fukuoka でお話しした資料です。

5d769d109697012317c09c6a27a6a4bf?s=128

linyows

April 16, 2019
Tweet

Transcript

  1. Apr 16th, 2019 / CloudNative Days Fukuoka Secrets Management for

    Kubernetes with HashiCorp Vault Tomohisa Oda — GMO Pepabo, inc.
  2. Apr 16th, 2019 / CloudNative Days Fukuoka Tomohisa Oda @linyows

    Principal Engineer on GMO pepabo, inc. blog: https://tomohisaoda.com/
  3. Apr 16th, 2019 / CloudNative Days Fukuoka Fukuoka.go Organizer Fukuoka.go

    is gophers community in Fukuoka. GoConference’19 Summer in Fukuoka Call for Papers and Sponsor is open now! https://fukuoka.gocon.jp/
  4. Apr 16th, 2019 / CloudNative Days Fukuoka Private Projects https://github.com/linyows

    The application server for automated deployment with polling a repository. Management linux user and authentication with team or collaborator on Github.
  5. Apr 16th, 2019 / CloudNative Days Fukuoka

  6. Apr 16th, 2019 / CloudNative Days Fukuoka Benefit of Kubernetes

    and Changes in our system
  7. Apr 16th, 2019 / CloudNative Days Fukuoka Benefit of Kubernetes

    Provides tools necessary to build and deploy reliable, scalable distributed applications ๏Velocity for development delivery with Reliability ๏Scaling Services and Team ๏The abstraction of Infrastructure for Portability ๏Efficient use of resources
  8. Apr 16th, 2019 / CloudNative Days Fukuoka Benefit of Kubernetes

    Provides tools necessary to build and deploy reliable, scalable distributed applications ๏Velocity for development delivery with Reliability ๏Declarative Configuration ๏Self Healing System ๏Immutability ๏Scaling Services and Team ๏The abstraction of Infrastructure for Portability ๏Efficient use of resources
  9. Apr 16th, 2019 / CloudNative Days Fukuoka Benefit of Kubernetes

    Provides tools necessary to build and deploy reliable, scalable distributed applications ๏Velocity for development delivery with Reliability ๏Declarative Configuration ๏Self Healing System ๏Immutability ๏Scaling Services and Team ๏The abstraction of Infrastructure for Portability ๏Efficient use of resources }Need to change our system
  10. Apr 16th, 2019 / CloudNative Days Fukuoka Changes in our

    system ๏Using container images for virtual machine ๏To record the state of machine ๏Easy change the state of machine ๏Running anywhere ๏Easy scaling with auto ๏Provisioning and Deployment changes ๏Secrets management?
  11. Apr 16th, 2019 / CloudNative Days Fukuoka Changes in Secrets

    Management
  12. Apr 16th, 2019 / CloudNative Days Fukuoka What is secrets

    in application? ๏Database Credential ๏API Token ๏Secret Key ๏IAM Application DB External API AWS GCP
  13. Apr 16th, 2019 / CloudNative Days Fukuoka No safe applications?

    ๏Fast development cycle ๏Credential sharing, Long lifetime secrets ๏Unmanaged pods ๏Leak secrets easily by logs with exception backtrace or debugging ๏Potentially taking down multiple service tiers at once ๏Members join and leave the team
  14. Apr 16th, 2019 / CloudNative Days Fukuoka So, application is

    not safe
  15. Apr 16th, 2019 / CloudNative Days Fukuoka – searchsecurity.techtarget.com “The

    principle of least privilege (POLP), an important concept in computer security, is the practice of limiting access rights for users to the bare minimum permissions they need to perform their work. Under POLP, users are granted permission to read, write or execute only the files or resources they need to do their jobs: In other words, the least amount of privilege necessary.”
  16. Apr 16th, 2019 / CloudNative Days Fukuoka Credentials should be

    given for each deployment ๏Deployment unit is replica-set in CloudNative application ๏Only enabled containers should have enabled credentials ๏By granting the credentials in detail, we can revoke the invalid container credentials Deployment ReplicaSet Pod Pod ReplicaSet Pod Pod DB User: deploy1 Password: secret1 User: deploy2 Password: secret2
  17. Apr 16th, 2019 / CloudNative Days Fukuoka What should we

    do? ——— "Using Vault"
  18. Apr 16th, 2019 / CloudNative Days Fukuoka Dynamic secrets for

    CloudNative Application
  19. Apr 16th, 2019 / CloudNative Days Fukuoka HashiCorp Vault Manage

    Secrets and Protect Sensitive Data ๏Encrypted Key-Value Store ๏Dynamic Secrets management ๏Support many secrets and authentication ๏Abstraction secrets (Database, SSH, PKI…) ☕ Namespace problem: 1Password, Google, CentOS, Ansible and more.
  20. Apr 16th, 2019 / CloudNative Days Fukuoka Architecture of Vault

    High-Level Overview
  21. Apr 16th, 2019 / CloudNative Days Fukuoka Usecase for Kubernetes

  22. Apr 16th, 2019 / CloudNative Days Fukuoka Database Secrets for

    Kuberentes ๏ The vault agent manages the token for vault after Kubernetes authentication ๏ Application gets database credentials as a file via consul template DB Vault Server Application Vault Agent Consul Template .vault_token db.yaml Get Credential Pod Volume ServiceAccount token auth & renew Create User Auth Backend Kubernetes
  23. Apr 16th, 2019 / CloudNative Days Fukuoka Transit Secrets for

    Kuberentes ๏ Using the Transit Secrets to protecting confidential in database ๏ Vault only saves encryption key for Transit Secrets DB Vault Server Application Vault Agent Consul Template .vault_token db.yaml ServiceAccount token auth & renew Get Credential Pod Volume Create User Encrypt/Decrypt Encrypted data Kubernetes Auth Backend
  24. Apr 16th, 2019 / CloudNative Days Fukuoka Authority Overview ๏

    Secrets policy can be separated by Role by the Vault policy setting ๏ A token for the Vault is issued for per- deployment authentication ๏ This is the principle of least privilege Kubernetes Vault kubectl apply Manifest Enable AuthMethod Create Policy SRE/Ops Dev --- apiVersion: apps/v1 kind: Deployment metadata: name: db-sidecar labels: app: db-sidecar spec: replicas: 1 selector: matchLabels: app: db-sidecar template: metadata: labels: app: db-sidecar path "secret/auth/foo" { capabilities = ["create", "read", "update", "delete", "list"] }
  25. Apr 16th, 2019 / CloudNative Days Fukuoka Other usecase: Vault

    for Kubernetes Admin
  26. Apr 16th, 2019 / CloudNative Days Fukuoka PKI for Kubernetes

    Cluster Kubernetes requires PKI for the following operations: ๏ Client certificates for the kubelet to authenticate to the API server ๏ Server certificate for the API server endpoint ๏ Client certificates for administrators of the cluster to authenticate to the API server ๏ Client certificates for the API server to talk to the kubelets ๏ Client certificate for the API server to talk to etcd ๏ Client certificate/kubeconfig for the controller manager to talk to the API server ๏ Client certificate/kubeconfig for the scheduler to talk to the API server. ๏ Client and server certificates for the front-proxy
  27. Apr 16th, 2019 / CloudNative Days Fukuoka ๏ Automatically update

    short lived certificates PKI for Kubernetes Cluster with Vault Certificate Vault Server Consul Template Vault Agent Root CA
  28. Apr 16th, 2019 / CloudNative Days Fukuoka Encryption for Kubernetes

    Secrets Envelope encryption ๏https://github.com/oracle/kubernetes-vault-kms-plugin
  29. Apr 16th, 2019 / CloudNative Days Fukuoka Vault on Kubernetes

    or Vault on Baremetal
  30. Apr 16th, 2019 / CloudNative Days Fukuoka Google GCP learning

  31. Apr 16th, 2019 / CloudNative Days Fukuoka “Single Tenancy. Vault

    should be the only main process running on a machine. This reduces the risk that another process running on the same machine is compromised and can interact with Vault. Similarly, running on bare metal should be preferred to a VM, and a VM preferred to a container. This reduces the surface area introduced by additional layers of abstraction and other tenants of the hardware. Both VM and container based deployments work, but should be avoided when possible to minimize risk.” –Production Hardning
  32. Apr 16th, 2019 / CloudNative Days Fukuoka So, recommend Vault

    on baremetal???
  33. Apr 16th, 2019 / CloudNative Days Fukuoka Vault on Kubernetes

    Pros ๏Simple Architecture as all containers ๏Skip overhead for operation (automation) ๏Easily grow clusters Cons ๏Other vulnerabilities can affect vaults in containers that share resources ๏Can not manage K8s certificates?
  34. Apr 16th, 2019 / CloudNative Days Fukuoka It seems better

    to think with Managed or Self hosted, Separated or Integrated
  35. Apr 16th, 2019 / CloudNative Days Fukuoka Conclusion ๏We benefit

    a lot from Kubernetes, but we also need to adapt our systems and thinking ๏In a scalable system, minimizing privileges is very important ๏Vault grants the least privileges with minimal operations ๏Where to run Vault depends on your organization's policy
  36. Apr 16th, 2019 / CloudNative Days Fukuoka

  37. Apr 16th, 2019 / CloudNative Days Fukuoka Thank you ϖύϘ

    ࠾༻ ݕࡧ We’re hiring!
  38. Apr 16th, 2019 / CloudNative Days Fukuoka References: What is

    Kubernetes? https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/ Benefits of Kubernetes https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48 Why We Need Dynamic Secrets https://www.hashicorp.com/blog/why-we-need-dynamic-secrets Production Hardening https://learn.hashicorp.com/vault/operations/production-hardening What is principle of least privilege (POLP)? https://searchsecurity.techtarget.com/definition/principle-of-least-privilege-POLP