Kubernetes clusters with Terraform / Terragrunt and Kops

How do you make Kops and Terraform play nice? What is special about Terraform 0.12? What is Terragrunt? And why would I want to use any of these when I can just use GKE / EKS? This talk is lessons learned from managing k8s clusters with kops since 2017. We will also look at what pros the solution discussed offers versus EKS and what common cluster operations this design takes into account. It may be quite heavily AWS focused, but it should also be useful if you're on other cloud providers.

Sample code: https://github.com/compareasiagroup/terraform-provider-kops
Join us: http://www.compareasiagroup.com/powered-by/
Presented at:
- https://www.meetup.com/Singapore-Kubernetes-User-Group/events/266944233/

Transcript

1. Page 1 12 December 2019 Date / other info December, 2019 Kops clusters with TF & Terragrunt Vincent De Smet, Systems Engineer
2. Page 2 12 December 2019 Table of contents 1. Kops → Terraform 2. Kops Templates 3. Terraform → Kops Templates 4. Terraform → Kops 5. Why Terragrunt? 6. Why not EKS / GKE? 7. Blue / Green clusters Kops is a community driven tool to create and manage Kubernetes clusters. Terraform 0.12 supports structured objects for more expressive configuration and powerful templating. Terragrunt provides best practices for Terraform code layout and accelerates common operations. Together they allow us to come up with a strategy to handle frequent platform upgrades decoupled from stateful application components Vincent De Smet Systems Engineer EXEC SUMMARY
3. Page 3 12 December 2019 - Re-use Terraform expertise - Treat as module: - wrap modifications around code generated files, or - drop tf files into folder (no encapsulation…) - No auto-prune Kops → Terraform (v1 ~2017): Kops output Terraform
4. Page 4 12 December 2019 Problems with kops default networking: Routing Tables / Network Access Control Lists / ... • VPC Peering • Direct Connections • Egress control Kops Templates (v2 ~ 2018) Templated kops manifests
5. Page 5 12 December 2019 Kops Templates Templated kops manifests
6. Page 6 12 December 2019 Kops Templates Default Values (sample from previous implementations)
7. Page 7 12 December 2019 Kops Templates Cluster specific overwrites
8. Page 8 12 December 2019 Kops Templates Execution
9. Page 9 12 December 2019 - Reduce Copy Paste - Let Kops handle Kops Manifests TF → Kops Values → Kops Templates → TF AWS Resource Identifiers are in Terraform
10. Page 10 12 December 2019 - Prior work: - Nov-Nov 2018 - MongoDB - Colin Huglund - PoC - TF 0.12 - no expand / flatten - abandoned - Nov-Jan 2019 - Wandera - Jakub Coufal - TF 0.12 - Based on Colin Hugland - Added expand / flatten for kops schema - Alternative to WanderaOrg/karch local-exec - Abandoned ~ January 2019 - compareasiagroup/terraform-provider-kops - Based on Colin Huglund work - No expand / flatten - Implemented missing functions - Actively being used, PRs welcome! Terraform → Kops (v1 ~2019): Terraform provider Kops
11. Page 11 12 December 2019 - Manage Kops manifests through Terraform - Template AWS Resource Identifiers Terraform → Kops (v1 ~2019): Terraform provider Kops
12. Page 12 12 December 2019 - TF 0.12 built-in Templating Terraform → Kops (v1 ~2019): Terraform provider Kops
13. Page 13 12 December 2019 - Recommended read: Terraform Up And Running, second Edition! Why Terragrunt? Terragrunt for TF 0.12
14. Page 14 12 December 2019 - Easily split state - Modules on steroids - Dependencies between modules - Layered infra approach Why Terragrunt? Terragrunt for TF 0.12
15. Page 15 12 December 2019 - Existing Kops expertise - Self Managed clusters in foreseeable future Why no EKS / GKE ? Hosted solutions simplify Operations
16. Page 16 12 December 2019 - Isolated Environments with dedicated stateful tier - Decouple cluster from data and exposed interfaces Blue / Green Clusters Simplify Cluster upgrades by keeping them stateless
17. Page 17 12 December 2019 Demo? Less words, more action
18. Page 18 12 December 2019 Page 18 We are Hiring! Please get in touch with Thulasi for private and confidential discussion or visit our Career Site for more details