How easy was the Rakuten Group's software engineering team to solve the problem? 30th September, 2021 Koichi Yanagimoto & Sato Takumi Ecosystem Services Department Rakuten Group, Inc. 

2 佐藤 匠 SATO TAKUMI サウナ、ドライブ 2020年 新卒⼊社 Software Engineer ⾃⼰紹介 柳本浩⼀ YANAGIMOTO KOICHI 散歩、キーボード 2009年 ⼊社 Software Engineer 

3 Service Operations Kaizen (SOK) Group ※ Our team member's presentation. https://codezine.jp/article/detail/12021 https://event.cloudnativedays.jp/cndo2021/talks/371 https://event.cloudnativedays.jp/cndo2021/talks/311 https://event.cloudnativedays.jp/cndo2021/talks/401 https://event.cloudnativedays.jp/cndo2021/talks/621 https://confengine.com/conferences/scrum-fest-osaka-2021/proposal/15381/dirt-up https://www.elastic.co/elasticon/solution-series/asia-pacific-jp?tab=2#agenda Our Mission : Operation Zero Our Services : around 10 services. Using from all around Rakuten group’s services. 

4 Today's Theme How To solve the problem w/o Hard Operation. 

5 Our Team had two problems HOW COMPLEX TO MANAGE VM CONFIGURATIONS... WE WANT TO MANAGE SECRET DATA MORE SECURELY... 

No content

No content

8 Our internal-System, called VM Config list. Consul - Install w/ Helm Created Ohai-plugin – For collecting data Problem When move pod between nodes, It’s changed consul server’s IP address!!! 

No content

10 Our internal-System, called VM Config list. Using w/ HAProxy - For Fixed IP address No Operation recent days!!! 

No content

12 About Vault シークレットの管理 データの暗号化、復号化 認証・認可 https://www.silhouette- illust.com/illust/37090 https://icon-rainbow.com/ 

13 シークレットの管理 データの暗号化、復号化 認証・認可 https://icon-rainbow.com/ https://www.silhouette- illust.com/illust/37090 

14 Our Vault Environment Kubernetes Cluster Cloud Strage Statefulset 暗号化データの永続化 Load balancer 

15 Vault Server $ helm install vault hashicorp/vault Helmで簡単にinstall可能 GUI を使⽤してシークレットを登録可能 https://learn.hashicorp.com/img/vault/vault-ui-secrets-new-kv-secret-with-username-and- password.png 

16 Vault agent Kubernetes Cluster App container Pod Deployment Vault agent injector inject share secret Vault agent 

17 Vault agent︓シークレットの取得設定 apiVersion: v1 kind: Pod metadata: name: devwebapp labels: app: devwebapp annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/role: "devweb-app" vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config" spec: serviceAccountName: internal-app containers: - name: devwebapp - image: jweissig/app:0.0.1 /app # cat /vault/secrets/credentials.txt data: map[password:salsa username:giraffe] • Annotationを追加することで、Vault agent injectorがPodにVault agentをinject • Pod内のファイルにシークレットが保存 

18 Vault運⽤上の課題 • サイドカーとしてinjectするVault Agentのリソースが追加 で必要(1podあたりCPU: 250m、Memory: 64Mi) Problem-1 • Vault Serverが再起動するとseal状態になり、unseal操作を ⼿動で⾏う必要がある。 Problem-2 

19 改善点1︓Vault Agentのリソース削減 • Pod起動時のシークレット取得フロー • Init container︓app container起動前にVault agentがシークレットを取得。 • Sidecar︓Vault agentをSidecarとしてinject。 シークレットを動的に管理 Pod File system Vault agent Init container 

20 改善点1︓Vault Agentのリソース削減 • Pod起動時のシークレット取得フロー • Init container︓app container起動前にVault agentがシークレットを取得。 • Sidecar︓Vault agentをSidecarとしてinject。 シークレットを動的に管理 Pod File system Vault agent Init container Pod App container File system Vault agent Sidecar 

21 改善点1︓Vault Agentのリソース削減 • 動的にシークレットを扱わない場合はinit containerのみvault agentをinject ➡Vault agentのリソースを削減︕ Pod File system Vault agent Init container Pod App container File system Sidecar 

22 改善点1︓Vault Agentのリソース削減 annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/role: "devweb-app" vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config" vault.hashicorp.com/agent-pre-populate-only: "true" vault.hashicorp.com/agent-inject-template-credentials.txt : | {{ with secret "secrets/data/devwebapp/config" -}} export ID="{{ .Data.data.ID }}" export PASSWORD="{{ .Data.data.PASSWORD}}" {{- end }} spec: serviceAccountName: internal-app containers: - args: [ 'sh', '-c', ‘source /vault/secrets/credentials.txt && ' ] • vault.hashicorp.com/agent-pre-populate-only: "true ”のannotationを追加 注⽬ 

23 改善点2︓Vault Auto Unseal • Encrypted Keys、Master Keys、Shared keysの三つの鍵が登場 • 鍵の役割はVault内部と外部で分かれている Encrypted Keys Master Keys Shared keys encrypt encrypt restore https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1 

24 改善点2︓Vault Auto Unseal • Shared keysはshamirの秘密分散法で管理 Encrypted Keys Master Keys Shared keys encrypt encrypt restore https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1 

25 改善点2︓Vault Auto Unseal • Master keyの安全性を運⽤者ではなく、信頼できるシステムが担うことで、 auto unsealの仕組みを実現 Encrypted Keys Master Keys Cloud based key encrypt encrypt https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1 

26 改善点2︓Vault Auto Unseal • CloudのKMS情報をKubernetesのsecretとして作成することでauto unsealを実現 seal "azurekeyvault" { tenant_id = client_id = client_secret = vault_name = key_name = subscription_id = } vault operator unseal -migrate vault operator unseal -migrate vault operator unseal -migrate secret 

27 まとめ • SoftwareEngineerがほとんど⼿をかけずに運⽤できている。 • Kunernetes上でConsul/Vaultを運⽤する事例紹介。 • HAProxyなども組み合わせながらK8S上で。 • InstallはHelmで提供されてあるものを利⽤。 • Vault agentをinit containerのみinjectすることで不必要なリソースを削減。 • Vault auto unsealによってunsealの⼿間を削減。 

We're Hiring! Apply from HERE!!! https://bit.ly/3CFBGdH Corp site 

No content