Apr 16th, 2019 / CloudNative Days Fukuoka Secrets Management for Kubernetes with HashiCorp Vault Tomohisa Oda — GMO Pepabo, inc.

Apr 16th, 2019 / CloudNative Days Fukuoka Tomohisa Oda @linyows Principal Engineer on GMO pepabo, inc. blog: https://tomohisaoda.com/

Apr 16th, 2019 / CloudNative Days Fukuoka Fukuoka.go Organizer Fukuoka.go is gophers community in Fukuoka. GoConference’19 Summer in Fukuoka Call for Papers and Sponsor is open now! https://fukuoka.gocon.jp/

Apr 16th, 2019 / CloudNative Days Fukuoka Private Projects https://github.com/linyows The application server for automated deployment with polling a repository. Management linux user and authentication with team or collaborator on Github.

Apr 16th, 2019 / CloudNative Days Fukuoka

Apr 16th, 2019 / CloudNative Days Fukuoka Benefit of Kubernetes and Changes in our system

Apr 16th, 2019 / CloudNative Days Fukuoka Benefit of Kubernetes Provides tools necessary to build and deploy reliable, scalable distributed applications ๏Velocity for development delivery with Reliability ๏Scaling Services and Team ๏The abstraction of Infrastructure for Portability ๏Efficient use of resources

Apr 16th, 2019 / CloudNative Days Fukuoka Benefit of Kubernetes Provides tools necessary to build and deploy reliable, scalable distributed applications ๏Velocity for development delivery with Reliability ๏Declarative Configuration ๏Self Healing System ๏Immutability ๏Scaling Services and Team ๏The abstraction of Infrastructure for Portability ๏Efficient use of resources

Apr 16th, 2019 / CloudNative Days Fukuoka Benefit of Kubernetes Provides tools necessary to build and deploy reliable, scalable distributed applications ๏Velocity for development delivery with Reliability ๏Declarative Configuration ๏Self Healing System ๏Immutability ๏Scaling Services and Team ๏The abstraction of Infrastructure for Portability ๏Efficient use of resources }Need to change our system

Apr 16th, 2019 / CloudNative Days Fukuoka Changes in our system ๏Using container images for virtual machine ๏To record the state of machine ๏Easy change the state of machine ๏Running anywhere ๏Easy scaling with auto ๏Provisioning and Deployment changes ๏Secrets management?

Apr 16th, 2019 / CloudNative Days Fukuoka Changes in Secrets Management

Apr 16th, 2019 / CloudNative Days Fukuoka What is secrets in application? ๏Database Credential ๏API Token ๏Secret Key ๏IAM Application DB External API AWS GCP

Apr 16th, 2019 / CloudNative Days Fukuoka No safe applications? ๏Fast development cycle ๏Credential sharing, Long lifetime secrets ๏Unmanaged pods ๏Leak secrets easily by logs with exception backtrace or debugging ๏Potentially taking down multiple service tiers at once ๏Members join and leave the team

Apr 16th, 2019 / CloudNative Days Fukuoka So, application is not safe

Apr 16th, 2019 / CloudNative Days Fukuoka – searchsecurity.techtarget.com “The principle of least privilege (POLP), an important concept in computer security, is the practice of limiting access rights for users to the bare minimum permissions they need to perform their work. Under POLP, users are granted permission to read, write or execute only the files or resources they need to do their jobs: In other words, the least amount of privilege necessary.”

Apr 16th, 2019 / CloudNative Days Fukuoka Credentials should be given for each deployment ๏Deployment unit is replica-set in CloudNative application ๏Only enabled containers should have enabled credentials ๏By granting the credentials in detail, we can revoke the invalid container credentials Deployment ReplicaSet Pod Pod ReplicaSet Pod Pod DB User: deploy1 Password: secret1 User: deploy2 Password: secret2

Apr 16th, 2019 / CloudNative Days Fukuoka What should we do? ——— "Using Vault"

Apr 16th, 2019 / CloudNative Days Fukuoka Dynamic secrets for CloudNative Application

Apr 16th, 2019 / CloudNative Days Fukuoka HashiCorp Vault Manage Secrets and Protect Sensitive Data ๏Encrypted Key-Value Store ๏Dynamic Secrets management ๏Support many secrets and authentication ๏Abstraction secrets (Database, SSH, PKI…) ☕ Namespace problem: 1Password, Google, CentOS, Ansible and more.

Apr 16th, 2019 / CloudNative Days Fukuoka Architecture of Vault High-Level Overview

Apr 16th, 2019 / CloudNative Days Fukuoka Usecase for Kubernetes

Apr 16th, 2019 / CloudNative Days Fukuoka Database Secrets for Kuberentes ๏ The vault agent manages the token for vault after Kubernetes authentication ๏ Application gets database credentials as a file via consul template DB Vault Server Application Vault Agent Consul Template .vault_token db.yaml Get Credential Pod Volume ServiceAccount token auth & renew Create User Auth Backend Kubernetes

Apr 16th, 2019 / CloudNative Days Fukuoka Transit Secrets for Kuberentes ๏ Using the Transit Secrets to protecting confidential in database ๏ Vault only saves encryption key for Transit Secrets DB Vault Server Application Vault Agent Consul Template .vault_token db.yaml ServiceAccount token auth & renew Get Credential Pod Volume Create User Encrypt/Decrypt Encrypted data Kubernetes Auth Backend

Apr 16th, 2019 / CloudNative Days Fukuoka Authority Overview ๏ Secrets policy can be separated by Role by the Vault policy setting ๏ A token for the Vault is issued for per- deployment authentication ๏ This is the principle of least privilege Kubernetes Vault kubectl apply Manifest Enable AuthMethod Create Policy SRE/Ops Dev --- apiVersion: apps/v1 kind: Deployment metadata: name: db-sidecar labels: app: db-sidecar spec: replicas: 1 selector: matchLabels: app: db-sidecar template: metadata: labels: app: db-sidecar path "secret/auth/foo" { capabilities = ["create", "read", "update", "delete", "list"] }

Apr 16th, 2019 / CloudNative Days Fukuoka Other usecase: Vault for Kubernetes Admin

Apr 16th, 2019 / CloudNative Days Fukuoka PKI for Kubernetes Cluster Kubernetes requires PKI for the following operations: ๏ Client certificates for the kubelet to authenticate to the API server ๏ Server certificate for the API server endpoint ๏ Client certificates for administrators of the cluster to authenticate to the API server ๏ Client certificates for the API server to talk to the kubelets ๏ Client certificate for the API server to talk to etcd ๏ Client certificate/kubeconfig for the controller manager to talk to the API server ๏ Client certificate/kubeconfig for the scheduler to talk to the API server. ๏ Client and server certificates for the front-proxy

Apr 16th, 2019 / CloudNative Days Fukuoka ๏ Automatically update short lived certificates PKI for Kubernetes Cluster with Vault Certificate Vault Server Consul Template Vault Agent Root CA

Apr 16th, 2019 / CloudNative Days Fukuoka Encryption for Kubernetes Secrets Envelope encryption ๏https://github.com/oracle/kubernetes-vault-kms-plugin

Apr 16th, 2019 / CloudNative Days Fukuoka Vault on Kubernetes or Vault on Baremetal

Apr 16th, 2019 / CloudNative Days Fukuoka Google GCP learning

Apr 16th, 2019 / CloudNative Days Fukuoka “Single Tenancy. Vault should be the only main process running on a machine. This reduces the risk that another process running on the same machine is compromised and can interact with Vault. Similarly, running on bare metal should be preferred to a VM, and a VM preferred to a container. This reduces the surface area introduced by additional layers of abstraction and other tenants of the hardware. Both VM and container based deployments work, but should be avoided when possible to minimize risk.” –Production Hardning

Apr 16th, 2019 / CloudNative Days Fukuoka So, recommend Vault on baremetal???

Apr 16th, 2019 / CloudNative Days Fukuoka Vault on Kubernetes Pros ๏Simple Architecture as all containers ๏Skip overhead for operation (automation) ๏Easily grow clusters Cons ๏Other vulnerabilities can affect vaults in containers that share resources ๏Can not manage K8s certificates?

Apr 16th, 2019 / CloudNative Days Fukuoka It seems better to think with Managed or Self hosted, Separated or Integrated

Apr 16th, 2019 / CloudNative Days Fukuoka Conclusion ๏We benefit a lot from Kubernetes, but we also need to adapt our systems and thinking ๏In a scalable system, minimizing privileges is very important ๏Vault grants the least privileges with minimal operations ๏Where to run Vault depends on your organization's policy

Apr 16th, 2019 / CloudNative Days Fukuoka

Apr 16th, 2019 / CloudNative Days Fukuoka Thank you ϖύϘ ࠾༻ ݕࡧ We’re hiring!

Apr 16th, 2019 / CloudNative Days Fukuoka References: What is Kubernetes? https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/ Benefits of Kubernetes https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48 Why We Need Dynamic Secrets https://www.hashicorp.com/blog/why-we-need-dynamic-secrets Production Hardening https://learn.hashicorp.com/vault/operations/production-hardening What is principle of least privilege (POLP)? https://searchsecurity.techtarget.com/definition/principle-of-least-privilege-POLP