Unsealing Vault - Fundamentals and GCP Deployment for a Modern Secret Management

Is your secrets management a mess? Struggling to keep secrets secure while sharing them across teams? Wondering how to take your secret management to the next level? HashiCorp Vault could be your new best friend. But maybe the managed service is too expensive, and self-hosting sounds too complex? Fear not! In this talk, we’ll start from the ground up, exploring the core concepts of HashiCorp Vault. We’ll focus on what’s required to deploy and run your own instance, and how to bring this setup to Google Cloud Platform. You’ll learn how to gain full control over your environment while leveraging the high availability and scalability that GCP offers.

Transcript

2. Do it RIGHT. 1
3. Do it RIGHT. About Me 3 • Studied Visual Computing @ TU • Started Working as Backend Developer 2017 • Experience as Backend/Frontend Dev • Managing on-Prem and Cloud Workloads • Cloud Engineer @ Posedio Manuel Eiweck Concepts of Vault Setup on-Prem Transition to GCP
4. Do it RIGHT. Secret Management 4 • Use case: • DB Credentials, API Keys • Focus on (production) workloads • Kubernetes integration • Fetch via API • Store/Manage Secrets via Deployment
5. Do it RIGHT. Google Secret Manager 5 • Designed for simplicity • Coupled with Google IAM Authentication/Authorisation • Limited Multi-Tenancy ability - via GCP Projects • Simple Key Value store • GCP only - no cross-platform integration
6. Do it RIGHT. Vault 6 • Extending capabilities of Google Secret Manager • Google IAM + K8s, JWT/OIDC, Gitlab, Keycloak, … • Full Multi Tenancy ability + Fine Granular Access Control Lists • GCP + on-Prem and other cloud vendors • Key Value Store + Dynamic Secrets • Encryption as a Service highly customizable
7. Do it RIGHT. Setup Vault
8. Do it RIGHT. Install Vault 8 $ installVault curl -sSL "https://releases.hashicorp.com/vault/1.19.5/vault_1.19.5_linux_amd64.zip" -o vault.zip && unzip vault.zip $ ./vault server -config=“my_config.yml” ————
 
 $ vault server -dev VAULT_ADDR='http://127.0.0.1:8200' Unseal Key: aimz8RIeP4Bbs9Ug/dIqes3SWWOJIVUPlIVpYBwgcJQ= Root Token: hvs.S4HMZz6WQQgo3AGLygotNH1E Download Run
9. Do it RIGHT. Goal - Running Vault Instance ( on-Prem / Local) 9 Store 
 Secret Retrieve 
 Secret POD
10. Do it RIGHT. Running Vault Instance 10 Store 
 Secret Retrieve 
 Secret Free Credit card
11. Do it RIGHT. Authentication 11 Store 
 Secret Retrieve 
 Secret :( Authentication
 Provider
12. Do it RIGHT. 12 Authentication
13. Do it RIGHT. 13 Encryption - (Un) Sealing Vault
14. Do it RIGHT. Encryption - Sealing - Shamir seals 14
15. Do it RIGHT. 15 Manual Unsealing - after Reboot etc Secure but not automated —> Auto Unsealing
16. Do it RIGHT. Encryption - Sealing 16 Shamir seal Cloud seal
17. Do it RIGHT. Encryption - Root Key vs Root Token 17 Root Token → Authentication Root Key → Unsealing But Root Key → new Root Token
18. Do it RIGHT. Setup Process - manual unsealing 18 • 1. Install Binary • 2. Setup Shards/Root Keys and Root Token • 3. Distribute Shards to People • 4. Configure rest of Vault using Root Token (OIDC, Roles, …) • 5. Invalidate/Delete Root Token
19. Do it RIGHT. Setup Process - auto unsealing 19 • 1. Install Binary • 2. Setup Cloud based Key, Setup Root Token • 3. Distribute Shards to People • 4. Configure rest of Vault using Root Token (OIDC, Roles, …) • 5. Invalidate/Delete Root Token
20. Do it RIGHT. 20 High Availability
21. Do it RIGHT. 21 High Availability more servers • One active node / leader
22. Do it RIGHT. 22 High Availability more servers PERSISTENCE / STORAGE Central Coordination:
 
 - Leader election … • One active node / leader • Storage as a concept / "Interface"
23. Do it RIGHT. 23 Raft Storage Backend - Raft Consensus Algorithm PERSISTENCE / STORAGE Central Coordination:
 
 - Leader election … Details: https://raft.github.io/ • Not unique to Vault • Decentralised Storage (Secrets, Config) • Replication via Raft Consensus Algorithm • Node to Node Communication for Leader Election, Replication, … • Inactive Node forward traffic to leader • Readonly mode if leader can’t reach min 1 other node →
24. Do it RIGHT. 24 High Availability more servers node1.myvault.company.com 10.0.0.1 node2.myvault.company.com 10.0.0.2
25. Do it RIGHT. 25 High Availability more servers node1.myvault.company.com 10.0.0.1 node2.myvault.company.com 10.0.0.2 Reverse Proxy Load Balancer
26. Do it RIGHT. High Availability - Forward Traffic to Leader Node Load Balancer • “Distribute” between Leader and non Leader Nodes • —> Detect Leader via Health check • TLS/SSL Termination / Offloading at load balancer for Users: https://myvault.company.com Optional: http(s)://node1.myvault.company.com/v1/sys/health 200: if initialized, unsealed, and active 473 : if performance standby https://developer.hashicorp.com/ vault/api-docs/system/health …/v1/sys/health?standbycode=200 => Overwrite Status codes
27. Do it RIGHT. 27 Final On-Prem
28. Do it RIGHT. Load Balancer PERSISTENCE / STORAGE Central Coordination:
 
 - Leader election … /v1/sys/health In Transit Encryption Encryption at rest Authentication
 Provider
29. Do it RIGHT. 29 Transition to Google Cloud
30. Do it RIGHT. 30 Transition to Google Cloud - Approach Managed VMs • “Low Level” direct control of resources instead of using another virtualization layer like Kubernetes • Encouraged by HashiCorp due to lower dependencies on 3rd party systems • Make use of IT concepts like VMs instead of products like Compute Engine • Loosely coupled to GCP - possibility to transition to another Cloud Vendor • Better manageable costs
31. Do it RIGHT. Transition to Google Cloud - VM Install Vault 31 resource "google_compute_instance" "gcp-vault" { name = "gcp-vault" machine_type = "e2-small" zone = var.zone project = var.project boot_disk { initialize_params { image = "debian-cloud/debian-12" } } network_interface { network = var.network-name subnetwork = var.subnetwork-name } metadata_startup_script = file("${path.module}/setup-vault.sh") } $ curl -sSL "https:// releases.hashicorp.com/vault/1.19.5/ vault_1.19.5_linux_amd64.zip" -o vault.zip && unzip vault.zip $ ./vault server -config=“my_config.yml” OnPrem GCP —> Any Bash Startup Script
32. Do it RIGHT. Transition to Google Cloud - VM Install Vault 32 cat /tmp/hashicorp.gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg rm /tmp/hashicorp.gpg echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list echo "--> Installing pinned Vault version" sudo apt-get update sudo apt-get install -y vault=${var.vault_version}-1 sudo tee "/etc/vault.d" > /dev/null <<EOF api_addr = "https://$hostname.$${BASE_ADDRESS}:$${VAULT_API_PORT}" ... EOF sudo /usr/bin/vault server -config=/etc/vault.d VM - Startup/Install Script Install Repo Install Vault Run Vault Improvements: -> verify GPG fingerprint -> add system.d service -> add log files Set Configuration File
33. Do it RIGHT. 33 Transition to Google Cloud - Persistence GCP • Raft Storage Backend • Google GCS Backend -> Bucket OnPrem resource "google_storage_bucket" "vault_backend" { name = local.gcp_bucket_name_backend project = local.gcp_project_id location = local.gcp_region storage_class = "STANDARD" uniform_bucket_level_access = true } storage "gcs" { bucket = "$${GCP_BUCKET_NAME}" ha_enabled = "true" } https://developer.hashicorp.com/vault/ docs/configuration sudo /usr/bin/vault server -config=/etc/vault.d PERSISTENCE / STORAGE Central Coordination:
 
 - Leader election …
34. Do it RIGHT. 34 Transition to Google Cloud - Auto Unseal GCP • Manual Unseal • Google KMS Key Ring OnPrem resource "random_id" "kms_random" { prefix = "${local.instance_name}-keyring" byte_length = "8" } resource "google_kms_key_ring" "key_ring" { project = local.gcp_project_id name = random_id.kms_random.id location = local.gcp_region } #Create a crypto key for the key ring resource "google_kms_crypto_key" "crypto_key" { name = google_kms_key_ring.key_ring.name key_ring = google_kms_key_ring.key_ring.id rotation_period = "100000s" } seal "gcpckms" { project = "$${GCP_PROJECT}" region = "$${KMS_LOCATION}" key_ring = "$${KMS_KEY_RING}" crypto_key = "$${KMS_CRYPTO_KEY}" } • Vault Config
35. Do it RIGHT. Transition to Google Cloud - HA GCP • More Servers + Raft Backend • More VMs -> Managed Instance Group • Auto Healing, OS Patches, … OnPrem resource "google_compute_region_instance_group_manager" "vault" { provider = google-beta name = "${local.instance_name}-igm" region = local.gcp_region distribution_policy_zones = local.gcp_zone_distribution_policy version { instance_template = google_compute_instance_template.vault-template.id } base_instance_name = local.instance_name target_size = 3 } resource "google_compute_instance_template" "vault-template" { … } storage "gcs" { bucket = "$${GCP_BUCKET_NAME}" ha_enabled = "true" }
36. Do it RIGHT. Transition to Google Cloud - HA resource "google_compute_instance_template" "vault-template" { machine_type = "n2-standard-2" region = local.gcp_region name_prefix = "${local.instance_name}-node-" disk { source_image = “debian-cloud/debian-12“ disk_size_gb = 20 } metadata = { startup-script = templatefile("${path.module}/scripts/start-up.tftpl", { gcp_bucket_name = google_storage_bucket.vault_backend.name, … }) shutdown-script = templatefile("${path.module}/scripts/shut-down.tfpl", { … }) … } network_interface { network = data.google_compute_network.network_vpc.id subnetwork = data.google_compute_subnetwork.network_subnet.id } }
37. Do it RIGHT. Transition to Google Cloud - HA - Autoheal resource "google_compute_health_check" "autohealing" { name = "${local.instance_name}-health-check" check_interval_sec = 60 timeout_sec = 5 healthy_threshold = 2 unhealthy_threshold = 10 https_health_check { port = local.network_vault_api_port # Return 200 for standby nodes. (Active nodes it returns 200 anyways) request_path = "/v1/sys/health?standbycode=200" } } resource "google_compute_region_instance_group_manager" "vault" { target_size = 3 auto_healing_policies { health_check = google_compute_health_check.autohealing.id initial_delay_sec = 120 # 2 minutes } } Similar to K8s pod autoheal Overwrite StandbyCode to 200 so inactive node is also healthy
38. Do it RIGHT. Transition to Google Cloud - Networking resource "google_compute_region_backend_service" "vault-backend-service" { project = local.gcp_project_id name = "${local.instance_name}-backend-service" protocol = "HTTPS" load_balancing_scheme = "INTERNAL_MANAGED" health_checks = [google_compute_health_check.loadbalancer.id] region = local.gcp_region backend { group = google_compute_region_instance_group_manager.vault.instance_group capacity_scaler = 1.0 balancing_mode = "UTILIZATION" } } resource "google_compute_health_check" "loadbalancer" { ... https_health_check { port = local.network_vault_api_port request_path = "/v1/sys/health" # The load balancer will only direct traffic towards the ACTIVE node. } } OnPrem GCP Highly depends on what type of service you want:
 * Regional vs Global * Public or Private Network
39. Do it RIGHT. Recap
40. Do it RIGHT. Recap Setup Vault on GCP 40 • Use basic VMs “compute instance” • Build a startup.sh Script for installing and pass to compute instance • Build the myVaultConfig.yml Config File • Use Bucket as GCS Storage Backend (Persistence + HA) • Use Key Ring for Auto Unseal • Use managed Instance Group for HA Setup • Use health check for distributing traffic to the leader node
41. Do it RIGHT. PERSISTENCE / STORAGE GCS Bucket Central Coordination:
 - Leader election - Persistence Storage (Secrets & Config) - HA via Google Storage /v1/sys/health Authentication
 Provider Managed Instance Group VM 1 VM 2 VM 3 - Autohealing - Managed Vault Installation (bash) - Managed Vault HCL Config Google KMS Key Ring - Managed rotation of key - Auto unseal - Secured via Google IAM Forward Rule HTTPS Proxy Google Load Balancer URL Map Backend Service LB Heal Check - SSL Offloading - Redirecting to Leader Node
42. Do it RIGHT. Improvements not covered in this talk 42 • “Golden Image” -> Instead of Bash Scripts on Bootup: use Hashicorp Packer to build an ‘VM Image’ => instead of plain debian-cloud/debian-12 • Security Hardening: SSH, Automated VM OS Patches, Network Policies, … • Integrate Google Alerting and Dashboard
43. Do it RIGHT. Unseal your Secret management with Vault today 43 • Have a standardised way to share secrets within your company / team. • Take advantage of fully managed service. • Integrate separated Authentication Provider • Customize to your needs • Fully compatible to GCP, other Cloud Vendors and onPrem workloads • Secure by design by using industry standards like Vault
44. Do it RIGHT. THANK YOU! CONTACT US: Weyringergasse 1-3/DG 1040 Wien www.posedio.com office@posedio.com 44