Self-hosting Kubernetes at Home with Terraform Because I can’t remember all Helm packages…

About me kahnwong Karnsiree Wong karnwong.me Platform Engineer @Data Cafe Company Limited Firmly believes that all configurations should be documented as code Loves automation

Table of contents 1. What is Kubernetes 2. Anatomy of a Kubernetes Deployment 3. Enter Helm 1. Folder Structure 4. Enter Terraform 1. Terraform Project Structure 2. Project Setup 3. Setup Namespaces 4. Setup Deployments 1. Terraform Data Structures 5. Setup ConfigMaps 6. Setup Secrets 5. Conclusion

What is Kubernetes Containers orchestration platform Can scale workloads Baked in service discovery, load balancing, storage orchestration, and self-healing Ensures high availability and fault tolerance by distributing workloads across multiple nodes Available on major cloud providers Planet scale

Anatomy of a Kubernetes Deployment Deployment Service In practice Create Deployment and Service manifest Run kubectl apply -f deployment.yaml & kubectl apply -f service.yaml Sometimes you also need ConfigMap or Secret Which translate to: two more kubectl apply -f $manifest.yaml That’s a total 4 manifest files per deployment And you need to apply 4 commands to get a deployment working At minimum it would require following manifests:

Enter Helm Sample Helm values: Helm is a package manager for Kubernetes kind: Deployment name: qa-api replicaCount: 1 # deployment containers: - name: qa-api repository: ghcr.io/kahnwong/qa-api tag: "95c411c" port: 3000 envFrom: - secretRef: name: qa-api # service service: port: 3000 nodePort: 30043

Folder Structure Without Helm With Helm But then we still need to run helm command for every deployment manually… . ├── appA │ ├── deployment.yaml │ └── service.yaml ├── appB ├── deployment.yaml └── service.yaml # deploy appA $ kubectl apply -f appA/deployment.yaml $ kubectl apply -f appA/service.yaml # deploy appB $ kubectl apply -f appB/deployment.yaml $ kubectl apply -f appB/service.yaml . ├── chart │ ├── Chart.yaml │ └── templates │ ├── deployment.yaml │ ├── _helpers.tpl │ └── service.yaml └── deployments ├── appA.yaml └── appB.yaml # deploy appA $ helm install appA chart --values deployments/$appA.yaml # deploy appB $ helm install appB chart --values deployments/$appB.yaml

Enter Terraform ⚠️⚠️⚠️ For production you should not use tereraform to manage app deployments In practice, you will need other Kubernetes manifests, such as ConfigMap and Secret . Terraform will help you with: Keeping track of all resources within your cluster Find configurations drift if any Update deployments / configurations with a single command ( terraform apply ) Keep you sane :) Use code to manage infrastructure and deployments.

Terraform Project Structure . ├── configmaps │ ├── bar.yaml │ └── foo.yaml ├── secrets │ ├── bar.sops.yaml │ └── foo.sops.yaml ├── helm │ └── deployments │ └── default │ ├── bar.yaml │ └── foo.yaml ├── configmaps.tf # k8s - ConfigMap ├── deployments.tf # k8s - Deployment ├── namespaces.tf # k8s - Namespace ├── providers.tf # terraform ├── secrets.tf # k8s - Secret ├── terraform.tf # terraform └── variables.tf # terraform

Project Setup provider "helm" { kubernetes { host = var.host client_certificate = base64decode(var.client_certificate) client_key = base64decode(var.client_key) cluster_ca_certificate = base64decode(var.cluster_ca_certificate) } } provider "kubernetes" { host = var.host client_certificate = base64decode(var.client_certificate) client_key = base64decode(var.client_key) cluster_ca_certificate = base64decode(var.cluster_ca_certificate) } provider "sops" {} # for secrets decryption

Setup Namespaces locals { namespaces = toset([ "default", "manchester", "quebec", ]) } resource "kubernetes_namespace" "this" { for_each = setsubtract(local.namespaces, toset(["default"])) metadata { name = each.key } }

Setup Deployments locals { deployments = tomap({ default = ["foo", "bar"] manchester = ["manchester", "oldham"] quebec = ["montreal", "terrebonne"] }) } locals { deployments_map_raw = flatten([ for namespace, deployments in local.deployments : [ for deployment in deployments : { deployment = deployment namespace = namespace } ] ]) deployments_map = { for index, v in local.deployments_map_raw : v.deployment => v.namespace } } resource "helm_release" "this" { for_each = local.deployments_map name = each.key namespace = each.value repository = "oci://ghcr.io/kahnwong/charts" version = "0.2.0" chart = "base" values = [ file("./helm/deployments/${each.value}/${each.key}.yaml") ] }

Terraform Data Structures From To locals { deployments = tomap({ default = ["foo", "bar"] manchester = ["manchester", "oldham"] quebec = ["montreal", "terrebonne"] }) } # $deployment = $namespace { "foo" = "default" "bar" = "default" "manchester" = "manchester" "oldham" = "manchester" "montreal" = "quebec" "terrebonne" = "quebec" }

Setup ConfigMaps Local Block locals { configmaps = tomap({ default = [ { deployment = "foo" filename = "conf.yml" }, { deployment = "bar" filename = "conf.yml" } ] }) }

Setup ConfigMaps Local Block (cont) locals { configmaps_map_raw = flatten([ for namespace, configmaps in local.configmaps : [ for configmap in configmaps : { namespace = namespace deployment = configmap.deployment filename = configmap.filename } ] ]) configmaps_map = { for index, v in local.configmaps_map_raw : v.deployment => v } }

Setup ConfigMaps Resource Block resource "kubernetes_config_map" "configmaps" { for_each = local.configmaps_map metadata { name = each.key namespace = each.value.namespace } data = { (each.value.filename) = file("${path.module}/configmaps/${each.key}.sops.yaml") } }

Setup Secrets Local Block locals { secrets = tomap({ default = ["foo", "bar"] }) } locals { secrets_map_raw = flatten([ for namespace, secrets in local.secrets : [ for secret in secrets : { secret = secret namespace = namespace } ] ]) secrets_map = { for index, v in local.secrets_map_raw : "${v.namespace}-${v.secret}" => tomap(v) } }

Setup Secrets Resource Block data "sops_file" "secrets" { for_each = local.secrets_map source_file = "./secrets/${each.value.secret}.sops.yaml" } resource "kubernetes_secret" "secrets" { for_each = local.secrets_map metadata { name = each.value.secret namespace = each.value.namespace } data = nonsensitive(data.sops_file.secrets["${each.value.namespace}-${each.value.secret}"].data) depends_on = [data.sops_file.secrets] }

Any questions?

Conclusion Kubernetes can help you manage multiple workloads Deploying Kubernetes workloads manually can be tedious and prone to errors Terraform can help you keep track of cluster configurations and workloads With proper Terraform local blocks usage, you can even simplify the setup further

Thank You Slides Blog GitHub LinkedIn