hashitalks2021 rakuten

Transcript

1. How easy was the Rakuten Group's software engineering team to solve the problem? 30th September, 2021 Koichi Yanagimoto & Sato Takumi Ecosystem Services Department Rakuten Group, Inc.
2. 2 佐藤 匠 SATO TAKUMI サウナ、ドライブ 2020年 新卒⼊社 Software Engineer ⾃⼰紹介 柳本浩⼀ YANAGIMOTO KOICHI 散歩、キーボード 2009年 ⼊社 Software Engineer
3. 3 Service Operations Kaizen (SOK) Group ※ Our team member's presentation. https://codezine.jp/article/detail/12021 https://event.cloudnativedays.jp/cndo2021/talks/371 https://event.cloudnativedays.jp/cndo2021/talks/311 https://event.cloudnativedays.jp/cndo2021/talks/401 https://event.cloudnativedays.jp/cndo2021/talks/621 https://confengine.com/conferences/scrum-fest-osaka-2021/proposal/15381/dirt-up https://www.elastic.co/elasticon/solution-series/asia-pacific-jp?tab=2#agenda Our Mission : Operation Zero Our Services : around 10 services. Using from all around Rakuten group’s services.
4. 4 Today's Theme How To solve the problem w/o Hard Operation.
5. 5 Our Team had two problems HOW COMPLEX TO MANAGE VM CONFIGURATIONS... WE WANT TO MANAGE SECRET DATA MORE SECURELY...
8. 8 Our internal-System, called VM Config list. Consul - Install w/ Helm Created Ohai-plugin – For collecting data Problem When move pod between nodes, It’s changed consul server’s IP address!!!
10. 10 Our internal-System, called VM Config list. Using w/ HAProxy - For Fixed IP address No Operation recent days!!!
12. 12 About Vault シークレットの管理 データの暗号化、復号化 認証・認可 https://www.silhouette- illust.com/illust/37090 https://icon-rainbow.com/
13. 13 シークレットの管理 データの暗号化、復号化 認証・認可 https://icon-rainbow.com/ https://www.silhouette- illust.com/illust/37090
14. 14 Our Vault Environment Kubernetes Cluster Cloud Strage Statefulset 暗号化データの永続化 Load balancer
15. 15 Vault Server $ helm install vault hashicorp/vault Helmで簡単にinstall可能 GUI を使⽤してシークレットを登録可能 https://learn.hashicorp.com/img/vault/vault-ui-secrets-new-kv-secret-with-username-and- password.png
16. 16 Vault agent Kubernetes Cluster App container Pod Deployment Vault agent injector inject share secret Vault agent
17. 17 Vault agent︓シークレットの取得設定 apiVersion: v1 kind: Pod metadata: name: devwebapp labels: app: devwebapp annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/role: "devweb-app" vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config" spec: serviceAccountName: internal-app containers: - name: devwebapp - image: jweissig/app:0.0.1 /app # cat /vault/secrets/credentials.txt data: map[password:salsa username:giraffe] • Annotationを追加することで、Vault agent injectorがPodにVault agentをinject • Pod内のファイルにシークレットが保存
18. 18 Vault運⽤上の課題 • サイドカーとしてinjectするVault Agentのリソースが追加 で必要(1podあたりCPU: 250m、Memory: 64Mi) Problem-1 • Vault Serverが再起動するとseal状態になり、unseal操作を ⼿動で⾏う必要がある。 Problem-2
19. 19 改善点1︓Vault Agentのリソース削減 • Pod起動時のシークレット取得フロー • Init container︓app container起動前にVault agentがシークレットを取得。 • Sidecar︓Vault agentをSidecarとしてinject。 シークレットを動的に管理 Pod File system Vault agent Init container
20. 20 改善点1︓Vault Agentのリソース削減 • Pod起動時のシークレット取得フロー • Init container︓app container起動前にVault agentがシークレットを取得。 • Sidecar︓Vault agentをSidecarとしてinject。 シークレットを動的に管理 Pod File system Vault agent Init container Pod App container File system Vault agent Sidecar
21. 21 改善点1︓Vault Agentのリソース削減 • 動的にシークレットを扱わない場合はinit containerのみvault agentをinject ➡Vault agentのリソースを削減︕ Pod File system Vault agent Init container Pod App container File system Sidecar
22. 22 改善点1︓Vault Agentのリソース削減 annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/role: "devweb-app" vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config" vault.hashicorp.com/agent-pre-populate-only: "true" vault.hashicorp.com/agent-inject-template-credentials.txt : | {{ with secret "secrets/data/devwebapp/config" -}} export ID="{{ .Data.data.ID }}" export PASSWORD="{{ .Data.data.PASSWORD}}" {{- end }} spec: serviceAccountName: internal-app containers: - args: [ 'sh', '-c', ‘source /vault/secrets/credentials.txt && <entrypoint script>' ] • vault.hashicorp.com/agent-pre-populate-only: "true ”のannotationを追加 注⽬
23. 23 改善点2︓Vault Auto Unseal • Encrypted Keys、Master Keys、Shared keysの三つの鍵が登場 • 鍵の役割はVault内部と外部で分かれている Encrypted Keys Master Keys Shared keys encrypt encrypt restore https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1
24. 24 改善点2︓Vault Auto Unseal • Shared keysはshamirの秘密分散法で管理 Encrypted Keys Master Keys Shared keys encrypt encrypt restore https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1
25. 25 改善点2︓Vault Auto Unseal • Master keyの安全性を運⽤者ではなく、信頼できるシステムが担うことで、 auto unsealの仕組みを実現 Encrypted Keys Master Keys Cloud based key encrypt encrypt https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1
26. 26 改善点2︓Vault Auto Unseal • CloudのKMS情報をKubernetesのsecretとして作成することでauto unsealを実現 seal "azurekeyvault" { tenant_id = <Key Vault's Directory ID> client_id = <Service Principal's Application ID> client_secret = <Service Principal's generated secret> vault_name = <Name of Azure Key Vault instance> key_name = <Name of generated key on Azure Key Vault> subscription_id = <ID of the Azure Subscription> } vault operator unseal -migrate <unseal key 1> vault operator unseal -migrate <unseal key 2> vault operator unseal -migrate <unseal key 3> secret
27. 27 まとめ • SoftwareEngineerがほとんど⼿をかけずに運⽤できている。 • Kunernetes上でConsul/Vaultを運⽤する事例紹介。 • HAProxyなども組み合わせながらK8S上で。 • InstallはHelmで提供されてあるものを利⽤。 • Vault agentをinit containerのみinjectすることで不必要なリソースを削減。 • Vault auto unsealによってunsealの⼿間を削減。
28. We're Hiring! Apply from HERE!!! https://bit.ly/3CFBGdH Corp site