Upgrade to Pro — share decks privately, control downloads, hide ads and more …

hashitalks2021 rakuten

hashitalks2021 rakuten

8c250b149ef5d379d07e47386e010ca6?s=128

TakumiSato0613

September 29, 2021
Tweet

Transcript

  1. How easy was the Rakuten Group's software engineering team to

    solve the problem? 30th September, 2021 Koichi Yanagimoto & Sato Takumi Ecosystem Services Department Rakuten Group, Inc.
  2. 2 佐藤 匠 SATO TAKUMI サウナ、ドライブ 2020年 新卒⼊社 Software Engineer

    ⾃⼰紹介 柳本浩⼀ YANAGIMOTO KOICHI 散歩、キーボード 2009年 ⼊社 Software Engineer
  3. 3 Service Operations Kaizen (SOK) Group ※ Our team member's

    presentation. https://codezine.jp/article/detail/12021 https://event.cloudnativedays.jp/cndo2021/talks/371 https://event.cloudnativedays.jp/cndo2021/talks/311 https://event.cloudnativedays.jp/cndo2021/talks/401 https://event.cloudnativedays.jp/cndo2021/talks/621 https://confengine.com/conferences/scrum-fest-osaka-2021/proposal/15381/dirt-up https://www.elastic.co/elasticon/solution-series/asia-pacific-jp?tab=2#agenda Our Mission : Operation Zero Our Services : around 10 services. Using from all around Rakuten group’s services.
  4. 4 Today's Theme How To solve the problem w/o Hard

    Operation.
  5. 5 Our Team had two problems HOW COMPLEX TO MANAGE

    VM CONFIGURATIONS... WE WANT TO MANAGE SECRET DATA MORE SECURELY...
  6. None
  7. None
  8. 8 Our internal-System, called VM Config list. Consul - Install

    w/ Helm Created Ohai-plugin – For collecting data Problem When move pod between nodes, It’s changed consul server’s IP address!!!
  9. None
  10. 10 Our internal-System, called VM Config list. Using w/ HAProxy

    - For Fixed IP address No Operation recent days!!!
  11. None
  12. 12 About Vault シークレットの管理 データの暗号化、復号化 認証・認可 https://www.silhouette- illust.com/illust/37090 https://icon-rainbow.com/

  13. 13 シークレットの管理 データの暗号化、復号化 認証・認可 https://icon-rainbow.com/ https://www.silhouette- illust.com/illust/37090

  14. 14 Our Vault Environment Kubernetes Cluster Cloud Strage Statefulset 暗号化データの永続化

    Load balancer
  15. 15 Vault Server $ helm install vault hashicorp/vault Helmで簡単にinstall可能 GUI

    を使⽤してシークレットを登録可能 https://learn.hashicorp.com/img/vault/vault-ui-secrets-new-kv-secret-with-username-and- password.png
  16. 16 Vault agent Kubernetes Cluster App container Pod Deployment Vault

    agent injector inject share secret Vault agent
  17. 17 Vault agent︓シークレットの取得設定 apiVersion: v1 kind: Pod metadata: name: devwebapp

    labels: app: devwebapp annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/role: "devweb-app" vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config" spec: serviceAccountName: internal-app containers: - name: devwebapp - image: jweissig/app:0.0.1 /app # cat /vault/secrets/credentials.txt data: map[password:salsa username:giraffe] • Annotationを追加することで、Vault agent injectorがPodにVault agentをinject • Pod内のファイルにシークレットが保存
  18. 18 Vault運⽤上の課題 • サイドカーとしてinjectするVault Agentのリソースが追加 で必要(1podあたりCPU: 250m、Memory: 64Mi) Problem-1 •

    Vault Serverが再起動するとseal状態になり、unseal操作を ⼿動で⾏う必要がある。 Problem-2
  19. 19 改善点1︓Vault Agentのリソース削減 • Pod起動時のシークレット取得フロー • Init container︓app container起動前にVault agentがシークレットを取得。

    • Sidecar︓Vault agentをSidecarとしてinject。 シークレットを動的に管理 Pod File system Vault agent Init container
  20. 20 改善点1︓Vault Agentのリソース削減 • Pod起動時のシークレット取得フロー • Init container︓app container起動前にVault agentがシークレットを取得。

    • Sidecar︓Vault agentをSidecarとしてinject。 シークレットを動的に管理 Pod File system Vault agent Init container Pod App container File system Vault agent Sidecar
  21. 21 改善点1︓Vault Agentのリソース削減 • 動的にシークレットを扱わない場合はinit containerのみvault agentをinject ➡Vault agentのリソースを削減︕ Pod

    File system Vault agent Init container Pod App container File system Sidecar
  22. 22 改善点1︓Vault Agentのリソース削減 annotations: vault.hashicorp.com/agent-inject: "true" vault.hashicorp.com/role: "devweb-app" vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config"

    vault.hashicorp.com/agent-pre-populate-only: "true" vault.hashicorp.com/agent-inject-template-credentials.txt : | {{ with secret "secrets/data/devwebapp/config" -}} export ID="{{ .Data.data.ID }}" export PASSWORD="{{ .Data.data.PASSWORD}}" {{- end }} spec: serviceAccountName: internal-app containers: - args: [ 'sh', '-c', ‘source /vault/secrets/credentials.txt && <entrypoint script>' ] • vault.hashicorp.com/agent-pre-populate-only: "true ”のannotationを追加 注⽬
  23. 23 改善点2︓Vault Auto Unseal • Encrypted Keys、Master Keys、Shared keysの三つの鍵が登場 •

    鍵の役割はVault内部と外部で分かれている Encrypted Keys Master Keys Shared keys encrypt encrypt restore https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1
  24. 24 改善点2︓Vault Auto Unseal • Shared keysはshamirの秘密分散法で管理 Encrypted Keys Master

    Keys Shared keys encrypt encrypt restore https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1
  25. 25 改善点2︓Vault Auto Unseal • Master keyの安全性を運⽤者ではなく、信頼できるシステムが担うことで、 auto unsealの仕組みを実現 Encrypted

    Keys Master Keys Cloud based key encrypt encrypt https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1
  26. 26 改善点2︓Vault Auto Unseal • CloudのKMS情報をKubernetesのsecretとして作成することでauto unsealを実現 seal "azurekeyvault" {

    tenant_id = <Key Vault's Directory ID> client_id = <Service Principal's Application ID> client_secret = <Service Principal's generated secret> vault_name = <Name of Azure Key Vault instance> key_name = <Name of generated key on Azure Key Vault> subscription_id = <ID of the Azure Subscription> } vault operator unseal -migrate <unseal key 1> vault operator unseal -migrate <unseal key 2> vault operator unseal -migrate <unseal key 3> secret
  27. 27 まとめ • SoftwareEngineerがほとんど⼿をかけずに運⽤できている。 • Kunernetes上でConsul/Vaultを運⽤する事例紹介。 • HAProxyなども組み合わせながらK8S上で。 • InstallはHelmで提供されてあるものを利⽤。

    • Vault agentをinit containerのみinjectすることで不必要なリソースを削減。 • Vault auto unsealによってunsealの⼿間を削減。
  28. We're Hiring! Apply from HERE!!! https://bit.ly/3CFBGdH Corp site

  29. None