Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Unsealing Vault

Avatar for Posedio Posedio PRO
September 24, 2025

Unsealing Vault

Fundamentals and GCP Deployment for a Modern Secret Management

Avatar for Posedio

Posedio PRO

September 24, 2025
Tweet

More Decks by Posedio

Other Decks in Programming

Transcript

  1. Do it RIGHT. About Me 3 • Studied Visual Computing

    @ TU • Started Working as Backend Developer 2017 • Experience as Backend/Frontend Dev • Managing on-Prem and Cloud Workloads • Cloud Engineer @ Posedio Manuel Eiweck Concepts of Vault Setup on-Prem Transition to GCP
  2. Do it RIGHT. Secret Management 4 • Use case: •

    DB Credentials, API Keys • Focus on (production) workloads • Kubernetes integration • Fetch via API • Store/Manage Secrets via Deployment
  3. Do it RIGHT. Google Secret Manager 5 • Designed for

    simplicity • Coupled with Google IAM Authentication/Authorisation • Limited Multi-Tenancy ability - via GCP Projects • Simple Key Value store • GCP only - no cross-platform integration
  4. Do it RIGHT. Vault 6 • Extending capabilities of Google

    Secret Manager • Google IAM + K8s, JWT/OIDC, Gitlab, Keycloak, … • Full Multi Tenancy ability + Fine Granular Access Control Lists • GCP + on-Prem and other cloud vendors • Key Value Store + Dynamic Secrets • Encryption as a Service highly customizable
  5. Do it RIGHT. Install Vault 8 $ installVault curl -sSL

    "https://releases.hashicorp.com/vault/1.19.5/vault_1.19.5_linux_amd64.zip" -o vault.zip && unzip vault.zip $ ./vault server -config=“my_config.yml” ————
 
 $ vault server -dev VAULT_ADDR='http://127.0.0.1:8200' Unseal Key: aimz8RIeP4Bbs9Ug/dIqes3SWWOJIVUPlIVpYBwgcJQ= Root Token: hvs.S4HMZz6WQQgo3AGLygotNH1E Download Run
  6. Do it RIGHT. Goal - Running Vault Instance ( on-Prem

    / Local) 9 Store 
 Secret Retrieve 
 Secret POD
  7. Do it RIGHT. Running Vault Instance 10 Store 
 Secret

    Retrieve 
 Secret Free Credit card
  8. Do it RIGHT. 15 Manual Unsealing - after Reboot etc

    Secure but not automated —> Auto Unsealing
  9. Do it RIGHT. Encryption - Root Key vs Root Token

    17 Root Token → Authentication Root Key → Unsealing But Root Key → new Root Token
  10. Do it RIGHT. Setup Process - manual unsealing 18 •

    1. Install Binary • 2. Setup Shards/Root Keys and Root Token • 3. Distribute Shards to People • 4. Configure rest of Vault using Root Token (OIDC, Roles, …) • 5. Invalidate/Delete Root Token
  11. Do it RIGHT. Setup Process - auto unsealing 19 •

    1. Install Binary • 2. Setup Cloud based Key, Setup Root Token • 3. Distribute Shards to People • 4. Configure rest of Vault using Root Token (OIDC, Roles, …) • 5. Invalidate/Delete Root Token
  12. Do it RIGHT. 22 High Availability more servers PERSISTENCE /

    STORAGE Central Coordination:
 
 - Leader election … • One active node / leader • Storage as a concept / "Interface"
  13. Do it RIGHT. 23 Raft Storage Backend - Raft Consensus

    Algorithm PERSISTENCE / STORAGE Central Coordination:
 
 - Leader election … Details: https://raft.github.io/ • Not unique to Vault • Decentralised Storage (Secrets, Config) • Replication via Raft Consensus Algorithm • Node to Node Communication for Leader Election, Replication, … • Inactive Node forward traffic to leader • Readonly mode if leader can’t reach min 1 other node →
  14. Do it RIGHT. 25 High Availability more servers node1.myvault.company.com 10.0.0.1

    node2.myvault.company.com 10.0.0.2 Reverse Proxy Load Balancer
  15. Do it RIGHT. High Availability - Forward Traffic to Leader

    Node Load Balancer • “Distribute” between Leader and non Leader Nodes • —> Detect Leader via Health check • TLS/SSL Termination / Offloading at load balancer for Users: https://myvault.company.com Optional: http(s)://node1.myvault.company.com/v1/sys/health 200: if initialized, unsealed, and active 473 : if performance standby https://developer.hashicorp.com/ vault/api-docs/system/health …/v1/sys/health?standbycode=200 => Overwrite Status codes
  16. Do it RIGHT. Load Balancer PERSISTENCE / STORAGE Central Coordination:


    
 - Leader election … /v1/sys/health In Transit Encryption Encryption at rest Authentication
 Provider
  17. Do it RIGHT. 30 Transition to Google Cloud - Approach

    Managed VMs • “Low Level” direct control of resources instead of using another virtualization layer like Kubernetes • Encouraged by HashiCorp due to lower dependencies on 3rd party systems • Make use of IT concepts like VMs instead of products like Compute Engine • Loosely coupled to GCP - possibility to transition to another Cloud Vendor • Better manageable costs
  18. Do it RIGHT. Transition to Google Cloud - VM Install

    Vault 31 resource "google_compute_instance" "gcp-vault" { name = "gcp-vault" machine_type = "e2-small" zone = var.zone project = var.project boot_disk { initialize_params { image = "debian-cloud/debian-12" } } network_interface { network = var.network-name subnetwork = var.subnetwork-name } metadata_startup_script = file("${path.module}/setup-vault.sh") } $ curl -sSL "https:// releases.hashicorp.com/vault/1.19.5/ vault_1.19.5_linux_amd64.zip" -o vault.zip && unzip vault.zip $ ./vault server -config=“my_config.yml” OnPrem GCP —> Any Bash Startup Script
  19. Do it RIGHT. Transition to Google Cloud - VM Install

    Vault 32 cat /tmp/hashicorp.gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg rm /tmp/hashicorp.gpg echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list echo "--> Installing pinned Vault version" sudo apt-get update sudo apt-get install -y vault=${var.vault_version}-1 sudo tee "/etc/vault.d" > /dev/null <<EOF api_addr = "https://$hostname.$${BASE_ADDRESS}:$${VAULT_API_PORT}" ... EOF sudo /usr/bin/vault server -config=/etc/vault.d VM - Startup/Install Script Install Repo Install Vault Run Vault Improvements: -> verify GPG fingerprint -> add system.d service -> add log files Set Configuration File
  20. Do it RIGHT. 33 Transition to Google Cloud - Persistence

    GCP • Raft Storage Backend • Google GCS Backend -> Bucket OnPrem resource "google_storage_bucket" "vault_backend" { name = local.gcp_bucket_name_backend project = local.gcp_project_id location = local.gcp_region storage_class = "STANDARD" uniform_bucket_level_access = true } storage "gcs" { bucket = "$${GCP_BUCKET_NAME}" ha_enabled = "true" } https://developer.hashicorp.com/vault/ docs/configuration sudo /usr/bin/vault server -config=/etc/vault.d PERSISTENCE / STORAGE Central Coordination:
 
 - Leader election …
  21. Do it RIGHT. 34 Transition to Google Cloud - Auto

    Unseal GCP • Manual Unseal • Google KMS Key Ring OnPrem resource "random_id" "kms_random" { prefix = "${local.instance_name}-keyring" byte_length = "8" } resource "google_kms_key_ring" "key_ring" { project = local.gcp_project_id name = random_id.kms_random.id location = local.gcp_region } #Create a crypto key for the key ring resource "google_kms_crypto_key" "crypto_key" { name = google_kms_key_ring.key_ring.name key_ring = google_kms_key_ring.key_ring.id rotation_period = "100000s" } seal "gcpckms" { project = "$${GCP_PROJECT}" region = "$${KMS_LOCATION}" key_ring = "$${KMS_KEY_RING}" crypto_key = "$${KMS_CRYPTO_KEY}" } • Vault Config
  22. Do it RIGHT. Transition to Google Cloud - HA GCP

    • More Servers + Raft Backend • More VMs -> Managed Instance Group • Auto Healing, OS Patches, … OnPrem resource "google_compute_region_instance_group_manager" "vault" { provider = google-beta name = "${local.instance_name}-igm" region = local.gcp_region distribution_policy_zones = local.gcp_zone_distribution_policy version { instance_template = google_compute_instance_template.vault-template.id } base_instance_name = local.instance_name target_size = 3 } resource "google_compute_instance_template" "vault-template" { … } storage "gcs" { bucket = "$${GCP_BUCKET_NAME}" ha_enabled = "true" }
  23. Do it RIGHT. Transition to Google Cloud - HA resource

    "google_compute_instance_template" "vault-template" { machine_type = "n2-standard-2" region = local.gcp_region name_prefix = "${local.instance_name}-node-" disk { source_image = “debian-cloud/debian-12“ disk_size_gb = 20 } metadata = { startup-script = templatefile("${path.module}/scripts/start-up.tftpl", { gcp_bucket_name = google_storage_bucket.vault_backend.name, … }) shutdown-script = templatefile("${path.module}/scripts/shut-down.tfpl", { … }) … } network_interface { network = data.google_compute_network.network_vpc.id subnetwork = data.google_compute_subnetwork.network_subnet.id } }
  24. Do it RIGHT. Transition to Google Cloud - HA -

    Autoheal resource "google_compute_health_check" "autohealing" { name = "${local.instance_name}-health-check" check_interval_sec = 60 timeout_sec = 5 healthy_threshold = 2 unhealthy_threshold = 10 https_health_check { port = local.network_vault_api_port # Return 200 for standby nodes. (Active nodes it returns 200 anyways) request_path = "/v1/sys/health?standbycode=200" } } resource "google_compute_region_instance_group_manager" "vault" { target_size = 3 auto_healing_policies { health_check = google_compute_health_check.autohealing.id initial_delay_sec = 120 # 2 minutes } } Similar to K8s pod autoheal Overwrite StandbyCode to 200 so inactive node is also healthy
  25. Do it RIGHT. Transition to Google Cloud - Networking resource

    "google_compute_region_backend_service" "vault-backend-service" { project = local.gcp_project_id name = "${local.instance_name}-backend-service" protocol = "HTTPS" load_balancing_scheme = "INTERNAL_MANAGED" health_checks = [google_compute_health_check.loadbalancer.id] region = local.gcp_region backend { group = google_compute_region_instance_group_manager.vault.instance_group capacity_scaler = 1.0 balancing_mode = "UTILIZATION" } } resource "google_compute_health_check" "loadbalancer" { ... https_health_check { port = local.network_vault_api_port request_path = "/v1/sys/health" # The load balancer will only direct traffic towards the ACTIVE node. } } OnPrem GCP Highly depends on what type of service you want:
 * Regional vs Global * Public or Private Network
  26. Do it RIGHT. Recap Setup Vault on GCP 40 •

    Use basic VMs “compute instance” • Build a startup.sh Script for installing and pass to compute instance • Build the myVaultConfig.yml Config File • Use Bucket as GCS Storage Backend (Persistence + HA) • Use Key Ring for Auto Unseal • Use managed Instance Group for HA Setup • Use health check for distributing traffic to the leader node
  27. Do it RIGHT. PERSISTENCE / STORAGE GCS Bucket Central Coordination:


    - Leader election - Persistence Storage (Secrets & Config) - HA via Google Storage /v1/sys/health Authentication
 Provider Managed Instance Group VM 1 VM 2 VM 3 - Autohealing - Managed Vault Installation (bash) - Managed Vault HCL Config Google KMS Key Ring - Managed rotation of key - Auto unseal - Secured via Google IAM Forward Rule HTTPS Proxy Google Load Balancer URL Map Backend Service LB Heal Check - SSL Offloading - Redirecting to Leader Node
  28. Do it RIGHT. Improvements not covered in this talk 42

    • “Golden Image” -> Instead of Bash Scripts on Bootup: use Hashicorp Packer to build an ‘VM Image’ => instead of plain debian-cloud/debian-12 • Security Hardening: SSH, Automated VM OS Patches, Network Policies, … • Integrate Google Alerting and Dashboard
  29. Do it RIGHT. Unseal your Secret management with Vault today

    43 • Have a standardised way to share secrets within your company / team. • Take advantage of fully managed service. • Integrate separated Authentication Provider • Customize to your needs • Fully compatible to GCP, other Cloud Vendors and onPrem workloads • Secure by design by using industry standards like Vault