Upgrade to Pro — share decks privately, control downloads, hide ads and more …

hashitalks2021 rakuten

TakumiSato0613
September 29, 2021

hashitalks2021 rakuten

TakumiSato0613

September 29, 2021
Tweet

More Decks by TakumiSato0613

Other Decks in Technology

Transcript

  1. How easy was the Rakuten Group's software engineering team to

    solve the problem? 30th September, 2021 Koichi Yanagimoto & Sato Takumi Ecosystem Services Department Rakuten Group, Inc.
  2. 2 佐藤 匠 SATO TAKUMI サウナ、ドライブ 2020年 新卒⼊社 Software Engineer

    ⾃⼰紹介 柳本浩⼀ YANAGIMOTO KOICHI 散歩、キーボード 2009年 ⼊社 Software Engineer
  3. 3 Service Operations Kaizen (SOK) Group ※ Our team member's

    presentation. https://codezine.jp/article/detail/12021 https://event.cloudnativedays.jp/cndo2021/talks/371 https://event.cloudnativedays.jp/cndo2021/talks/311 https://event.cloudnativedays.jp/cndo2021/talks/401 https://event.cloudnativedays.jp/cndo2021/talks/621 https://confengine.com/conferences/scrum-fest-osaka-2021/proposal/15381/dirt-up https://www.elastic.co/elasticon/solution-series/asia-pacific-jp?tab=2#agenda Our Mission : Operation Zero Our Services : around 10 services. Using from all around Rakuten group’s services.
  4. 5 Our Team had two problems HOW COMPLEX TO MANAGE

    VM CONFIGURATIONS... WE WANT TO MANAGE SECRET DATA MORE SECURELY...
  5. 8 Our internal-System, called VM Config list. Consul - Install

    w/ Helm Created Ohai-plugin – For collecting data Problem When move pod between nodes, It’s changed consul server’s IP address!!!
  6. 10 Our internal-System, called VM Config list. Using w/ HAProxy

    - For Fixed IP address No Operation recent days!!!
  7. 15 Vault Server $ helm install vault hashicorp/vault Helmで簡単にinstall可能 GUI

    を使⽤してシークレットを登録可能 https://learn.hashicorp.com/img/vault/vault-ui-secrets-new-kv-secret-with-username-and- password.png
  8. 16 Vault agent Kubernetes Cluster App container Pod Deployment Vault

    agent injector inject share secret Vault agent
  9. 17 Vault agent︓シークレットの取得設定 apiVersion: v1 kind: Pod metadata: name: devwebapp

    labels: app: devwebapp annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/role: "devweb-app" vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config" spec: serviceAccountName: internal-app containers: - name: devwebapp - image: jweissig/app:0.0.1 /app # cat /vault/secrets/credentials.txt data: map[password:salsa username:giraffe] • Annotationを追加することで、Vault agent injectorがPodにVault agentをinject • Pod内のファイルにシークレットが保存
  10. 18 Vault運⽤上の課題 • サイドカーとしてinjectするVault Agentのリソースが追加 で必要(1podあたりCPU: 250m、Memory: 64Mi) Problem-1 •

    Vault Serverが再起動するとseal状態になり、unseal操作を ⼿動で⾏う必要がある。 Problem-2
  11. 19 改善点1︓Vault Agentのリソース削減 • Pod起動時のシークレット取得フロー • Init container︓app container起動前にVault agentがシークレットを取得。

    • Sidecar︓Vault agentをSidecarとしてinject。 シークレットを動的に管理 Pod File system Vault agent Init container
  12. 20 改善点1︓Vault Agentのリソース削減 • Pod起動時のシークレット取得フロー • Init container︓app container起動前にVault agentがシークレットを取得。

    • Sidecar︓Vault agentをSidecarとしてinject。 シークレットを動的に管理 Pod File system Vault agent Init container Pod App container File system Vault agent Sidecar
  13. 22 改善点1︓Vault Agentのリソース削減 annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/role: "devweb-app" vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config"

    vault.hashicorp.com/agent-pre-populate-only: "true" vault.hashicorp.com/agent-inject-template-credentials.txt : | {{ with secret "secrets/data/devwebapp/config" -}} export ID="{{ .Data.data.ID }}" export PASSWORD="{{ .Data.data.PASSWORD}}" {{- end }} spec: serviceAccountName: internal-app containers: - args: [ 'sh', '-c', ‘source /vault/secrets/credentials.txt && <entrypoint script>' ] • vault.hashicorp.com/agent-pre-populate-only: "true ”のannotationを追加 注⽬
  14. 23 改善点2︓Vault Auto Unseal • Encrypted Keys、Master Keys、Shared keysの三つの鍵が登場 •

    鍵の役割はVault内部と外部で分かれている Encrypted Keys Master Keys Shared keys encrypt encrypt restore https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1
  15. 24 改善点2︓Vault Auto Unseal • Shared keysはshamirの秘密分散法で管理 Encrypted Keys Master

    Keys Shared keys encrypt encrypt restore https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1
  16. 25 改善点2︓Vault Auto Unseal • Master keyの安全性を運⽤者ではなく、信頼できるシステムが担うことで、 auto unsealの仕組みを実現 Encrypted

    Keys Master Keys Cloud based key encrypt encrypt https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1
  17. 26 改善点2︓Vault Auto Unseal • CloudのKMS情報をKubernetesのsecretとして作成することでauto unsealを実現 seal "azurekeyvault" {

    tenant_id = <Key Vault's Directory ID> client_id = <Service Principal's Application ID> client_secret = <Service Principal's generated secret> vault_name = <Name of Azure Key Vault instance> key_name = <Name of generated key on Azure Key Vault> subscription_id = <ID of the Azure Subscription> } vault operator unseal -migrate <unseal key 1> vault operator unseal -migrate <unseal key 2> vault operator unseal -migrate <unseal key 3> secret