Upgrade to Pro — share decks privately, control downloads, hide ads and more …

hashitalks2021 rakuten

hashitalks2021 rakuten

TakumiSato0613

September 29, 2021
Tweet

More Decks by TakumiSato0613

Other Decks in Technology

Transcript

  1. How easy was the Rakuten Group's software
    engineering team to solve the problem?
    30th September, 2021
    Koichi Yanagimoto & Sato Takumi
    Ecosystem Services Department
    Rakuten Group, Inc.

    View full-size slide

  2. 2
    佐藤 匠
    SATO TAKUMI
    サウナ、ドライブ
    2020年 新卒⼊社
    Software Engineer
    ⾃⼰紹介
    柳本浩⼀
    YANAGIMOTO KOICHI
    散歩、キーボード
    2009年 ⼊社
    Software Engineer

    View full-size slide

  3. 3
    Service Operations Kaizen (SOK) Group
    ※ Our team member's presentation.
    https://codezine.jp/article/detail/12021
    https://event.cloudnativedays.jp/cndo2021/talks/371
    https://event.cloudnativedays.jp/cndo2021/talks/311
    https://event.cloudnativedays.jp/cndo2021/talks/401
    https://event.cloudnativedays.jp/cndo2021/talks/621
    https://confengine.com/conferences/scrum-fest-osaka-2021/proposal/15381/dirt-up
    https://www.elastic.co/elasticon/solution-series/asia-pacific-jp?tab=2#agenda
    Our Mission : Operation Zero
    Our Services : around 10 services.
    Using from all around Rakuten group’s services.

    View full-size slide

  4. 4
    Today's Theme
    How To solve
    the problem
    w/o
    Hard Operation.

    View full-size slide

  5. 5
    Our Team had two problems
    HOW COMPLEX TO MANAGE
    VM CONFIGURATIONS...
    WE WANT TO MANAGE
    SECRET DATA MORE
    SECURELY...

    View full-size slide

  6. 8
    Our internal-System, called VM Config list.
    Consul - Install w/ Helm
    Created Ohai-plugin – For collecting data
    Problem
    When move pod between nodes,
    It’s changed consul server’s IP address!!!

    View full-size slide

  7. 10
    Our internal-System, called VM Config list.
    Using w/ HAProxy - For Fixed IP address
    No Operation recent days!!!

    View full-size slide

  8. 12
    About Vault
    シークレットの管理
    データの暗号化、復号化 認証・認可
    https://www.silhouette-
    illust.com/illust/37090
    https://icon-rainbow.com/

    View full-size slide

  9. 13
    シークレットの管理
    データの暗号化、復号化 認証・認可
    https://icon-rainbow.com/
    https://www.silhouette-
    illust.com/illust/37090

    View full-size slide

  10. 14
    Our Vault Environment
    Kubernetes Cluster
    Cloud Strage
    Statefulset
    暗号化データの永続化
    Load balancer

    View full-size slide

  11. 15
    Vault Server
    $ helm install vault hashicorp/vault
    Helmで簡単にinstall可能 GUI を使⽤してシークレットを登録可能
    https://learn.hashicorp.com/img/vault/vault-ui-secrets-new-kv-secret-with-username-and-
    password.png

    View full-size slide

  12. 16
    Vault agent
    Kubernetes Cluster
    App
    container
    Pod Deployment
    Vault agent
    injector
    inject
    share secret
    Vault agent

    View full-size slide

  13. 17
    Vault agent︓シークレットの取得設定
    apiVersion: v1
    kind: Pod
    metadata:
    name: devwebapp
    labels:
    app: devwebapp
    annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "devweb-app"
    vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config"
    spec:
    serviceAccountName: internal-app
    containers:
    - name: devwebapp
    - image: jweissig/app:0.0.1
    /app # cat /vault/secrets/credentials.txt
    data: map[password:salsa username:giraffe]
    • Annotationを追加することで、Vault agent injectorがPodにVault agentをinject
    • Pod内のファイルにシークレットが保存

    View full-size slide

  14. 18
    Vault運⽤上の課題
    • サイドカーとしてinjectするVault Agentのリソースが追加
    で必要(1podあたりCPU: 250m、Memory: 64Mi)
    Problem-1
    • Vault Serverが再起動するとseal状態になり、unseal操作を
    ⼿動で⾏う必要がある。
    Problem-2

    View full-size slide

  15. 19
    改善点1︓Vault Agentのリソース削減
    • Pod起動時のシークレット取得フロー
    • Init container︓app container起動前にVault agentがシークレットを取得。
    • Sidecar︓Vault agentをSidecarとしてinject。 シークレットを動的に管理
    Pod
    File system
    Vault agent
    Init container

    View full-size slide

  16. 20
    改善点1︓Vault Agentのリソース削減
    • Pod起動時のシークレット取得フロー
    • Init container︓app container起動前にVault agentがシークレットを取得。
    • Sidecar︓Vault agentをSidecarとしてinject。 シークレットを動的に管理
    Pod
    File system
    Vault agent
    Init container
    Pod
    App container
    File system
    Vault agent
    Sidecar

    View full-size slide

  17. 21
    改善点1︓Vault Agentのリソース削減
    • 動的にシークレットを扱わない場合はinit containerのみvault agentをinject
    ➡Vault agentのリソースを削減︕
    Pod
    File system
    Vault agent
    Init container
    Pod
    App container
    File system
    Sidecar

    View full-size slide

  18. 22
    改善点1︓Vault Agentのリソース削減
    annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "devweb-app"
    vault.hashicorp.com/agent-inject-secret-credentials.txt: "secret/data/devwebapp/config"
    vault.hashicorp.com/agent-pre-populate-only: "true"
    vault.hashicorp.com/agent-inject-template-credentials.txt : |
    {{ with secret "secrets/data/devwebapp/config" -}}
    export ID="{{ .Data.data.ID }}"
    export PASSWORD="{{ .Data.data.PASSWORD}}"
    {{- end }}
    spec:
    serviceAccountName: internal-app
    containers:
    - args: [ 'sh', '-c', ‘source /vault/secrets/credentials.txt && ' ]
    • vault.hashicorp.com/agent-pre-populate-only: "true ”のannotationを追加
    注⽬

    View full-size slide

  19. 23
    改善点2︓Vault Auto Unseal
    • Encrypted Keys、Master Keys、Shared keysの三つの鍵が登場
    • 鍵の役割はVault内部と外部で分かれている
    Encrypted Keys
    Master Keys
    Shared keys
    encrypt
    encrypt
    restore
    https://nureyon.com/key-1
    https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1

    View full-size slide

  20. 24
    改善点2︓Vault Auto Unseal
    • Shared keysはshamirの秘密分散法で管理
    Encrypted Keys
    Master Keys
    Shared keys
    encrypt
    encrypt
    restore
    https://nureyon.com/key-1
    https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1

    View full-size slide

  21. 25
    改善点2︓Vault Auto Unseal
    • Master keyの安全性を運⽤者ではなく、信頼できるシステムが担うことで、
    auto unsealの仕組みを実現
    Encrypted Keys
    Master Keys
    Cloud based
    key
    encrypt
    encrypt
    https://nureyon.com/key-1 https://nureyon.com/key-1 https://nureyon.com/key-1

    View full-size slide

  22. 26
    改善点2︓Vault Auto Unseal
    • CloudのKMS情報をKubernetesのsecretとして作成することでauto unsealを実現
    seal "azurekeyvault" {
    tenant_id =
    client_id =
    client_secret =
    vault_name =
    key_name =
    subscription_id =
    }
    vault operator unseal -migrate
    vault operator unseal -migrate
    vault operator unseal -migrate
    secret

    View full-size slide

  23. 27
    まとめ
    • SoftwareEngineerがほとんど⼿をかけずに運⽤できている。
    • Kunernetes上でConsul/Vaultを運⽤する事例紹介。
    • HAProxyなども組み合わせながらK8S上で。
    • InstallはHelmで提供されてあるものを利⽤。
    • Vault agentをinit containerのみinjectすることで不必要なリソースを削減。
    • Vault auto unsealによってunsealの⼿間を削減。

    View full-size slide

  24. We're Hiring!
    Apply from HERE!!! https://bit.ly/3CFBGdH
    Corp site

    View full-size slide