The (Semi-)Ultimate Terraform Upgrade Guide

This is a presentation for open source practitioners who want to upgrade Terraform. It compiles useful information and patterns for reference. The example is for AWS infrastructure but can be used to practice upgrade patterns.

Transcript

1. Copyright © 2019 HashiCorp The (Semi-)Ultimate Terraform Upgrade Guide August 23, 2019 !1
2. Presenter Rosemary Wang Developer Advocate at HashiCorp she/her @joatmon08 joatmon08 linkedin.com/in/rosemarywang/ !2
3. Premise: Upgrading is good. New features! Stability! Security! !3
4. Reality: Upgrading is costly. Short-term instability! Lower delivery capacity! !4
5. Agenda Upgrading Infrastructure, Generally Upgrading Terraform Migrating from v0.7 Onward !5
6. Disclaimer How you upgrade depends on how you’ve configured Terraform.* The other day… “We upgraded from 0.10.3 to 0.11 and it broke all of our pipelines. So we’re not upgrading.” !6
7. * A theoretical approach. !7
8. Upgrading Infrastructure, Generally !8
9. Combination of Approaches… Greenfield Write new configuration in higher version. Downside: Import each component. Mitigation: Only choose specific components. Big Bang Upgrade and see what happens. Downside: Likely to break production. Mitigation: Use isolated environment. Strangler Pattern Refactor to reduce complexity and upgrade. Downside: Decoupling can be difficult. Mitigation: N/A !9
10. Where to Start Assess Configuration 1. What does it depend on? 2. What depends on it? 3. What is its internal complexity? 4. How critical is it? 5. How stateful is it? !10
11. Monolith E.g., Networking. Changes to CIDR blocks might affect VPN, DNS, and more. Microlith E.g., Application + SQS. Changes to SQS reflect in application. Dependency Graph Micro-ish E.g., EKS Cluster separate from Kubernetes deployment. !11
12. Example Repository Structure !12
13. Where to Start Examples ▪ Non-critical & simple IAM policies for human users ▪ Infrastructure in development environments ▪ Repositories without: – Complex templating or functions – Cross-account dependencies !13
14. Upgrading Terraform !14
15. !15 https://github.com/hashicorp/terraform/issues/15839
16. !16 0.8 0.9 0.10 0.11 0.12 CHANGELOG Upgrade Guide Template files & string interpolation changes AWS provider attribute deprecations CHANGELOG Upgrade Guide Migrating to Backends Deprecate Remote for Backend Configuration
 
 State Locking AWS provider changes may trigger recreation Providers separated as plugins from core repository & versioned Interactive approval for apply (breaks pipelines, add -auto-approve flag) CHANGELOG Upgrade Guide Changes to module inheritance of providers Always use splat (*) operator for count references CHANGELOG Upgrade Guide CHANGELOG Upgrade Guide Adds rich type system to a previously string-typed system Includes automated upgrade tool (with caveats) AWS Provider CHANGELOG AWS v2 Upgrade Guide
17. To ease upgrade path… ▪ Avoid naming variables or identifiers that start with: – Numeric digits or symbols – Any Terraform reserved construct !17
18. To ease upgrade path… ▪ Configure providers with: – Version constraints ▪ ~> for root-level modules ▪ => for shared modules – Aliases for multi-region modules !18
19. To ease upgrade path… ▪ Configure backends for state using: – Configuration files – A state file per environment ▪ Generally, do not edit state file directly. !19
20. General Upgrade Steps 1. Back up state. 2. Read the upgrade guide and CHANGELOG. 3. Fix configuration for upgrade gotchas. 4. Apply current version to assure state. 5. Plan and apply new version.* * In pipeline, set toggle for version. !20
21. Warning: You can only roll forward. TERMINAL > terraform-0.7.13 apply Terraform doesn't allow running any operations against a state that was written by a future Terraform version. The state is reporting it is written by Terraform '0.8.8'. Please run at least that version of Terraform to continue !21
22. When upgrades fail… !22
23. Nuanced Failures 1. It may not have worked in the first place.
 count = “${length(aws_instance.vms.*.id)}” 2. Creative template rendering or function hacks.
 subnet_ids = ["${aws_subnet.eks-public.*.id}, ${aws_subnet.eks-private.*.id}”] 3. Cross-provider chaining
 e.g., Kubernetes + AWS !23
24. Nuanced Failures 4. It has provisioner scripts. 5. It has lifecycle customizations. 6. It depends_on components.
 !24
25. Exclude Resources TERMINAL # Remove from state > terraform-0.1 state rm <module.name.resource.id> 
 -backup=<date>.tfstate # Comment out from code or set count = 0 > vim <file>.tf # Run to ensure no other changes are applied > terraform-0.1 plan > terraform-0.1 apply !25
26. Re-Import Resources TERMINAL # Run to ensure no other changes are applied > terraform-0.2 plan > terraform-0.2 apply # Using backup state file, find ID of excluded resource. > jq -r -c '.modules[].resources."<resource.id>".primary.id' <date>.tfstate tf-################### # Uncomment code or remove count and re-initialize # Add back to state terraform-0.2 import <module.name.resource.id> tf- ################### !26
27. Move within State File TERMINAL # Move state object from one module to another > terraform-0.2 state mv <module.name.resource.id> <module2.name.resource.id2> # Run to ensure state doesn’t change terraform-0.2 apply !27
28. Migrating from v0.7 Onward !28
29. What We’ll Do… Repository Link: github.com/joatmon08/tf-migration ▪ Upgrade application from 0.7 to 0.10. – git checkout, backend, provider ▪ Refactor SNS to use count. – terraform state mv ▪ Upgrade networking from 0.7 to 0.10. ▪ Destroy. – terraform plan -target !29
30. Additional Resources ▪ Community Forum | discuss.hashicorp.com ▪ Upgrade Guides | terraform.io/upgrade-guides/ ▪ Recommended Practices | terraform.io/docs/cloud/guides/recommended-practices/ !30
31. Rosemary Wang Developer Advocate at HashiCorp she/her @joatmon08 joatmon08 linkedin.com/in/rosemarywang/ !31