Terraform Without The Mess

The way Terraform configuration is written today is impacted hugely by the history of the project and when features essential to minimising complexity became available

In this talk we'll look at some of the history of the project and make specific recommendations about how to structure Terraform configuration in order to make changes simple, predictable and safe.

Transcript

1. Terraform Without The Mess James Nugent @jen20
2. The fewer facilities a programming language has, the more important rigour is in everyday use to prevent emergence of catastrophic complexity
3. Bio • CTO at Event Store, a company which builds an open source stream database • Terraform Core maintainer at HashiCorp from vEarly to ~v0.8 • Member of the first SRE team at HashiCorp, developing patterns for scalable Terraform and ran for advanced Terraform training classes • Terraform AWS provider maintainer until ~September 2019 • Pulumi contributor since June 2018 • Tweet @jen20 primarily about Rust and Brexit (ffs)
5. Survey Questions…
7. Infrastructure as Code
8. Infrastructure as Code?
9. Infrastructure as Code Infrastructure as Software
10. Ancient History…
11. Terraform 0.1
12. Terraform 0.1 • No modules - everything lived in one namespace • No terraform_remote_state resource • No remote state at all! • Very “forgiving” HCL dialect (pre-HCL 1.0)
13. Terraform 0.3
14. Terraform 0.3
15. Terraform 0.3.5
16. Terraform 0.4.0
17. Terraform 0.6.0
18. Terraform 0.6.0
19. output “private_subnets” { value = “${join(aws_subnet.private.*.id, “,”)}” } resource “aws_instance” “servers” { // other fields subnet_id = “${element(split(var.private_subnets, “,”, count.index)}” }
20. By Terraform 0.6, we had the tools to avoid creating a mess…
21. … but people already had large amounts of Terraform code, and the ‘best practices’ were already in the wild.
22. Scalable Infrastructure Patterns
23. Microservices
24. Microservices • This is a tortuous analogy because I don’t actually like the name Microservices, or many of the concepts that it has come to embody. • Perhaps Service Oriented Architecture (SOA) is a better name • The gestalt is that we prefer to keep services (“applications”): • Small, so they can easily be understood (“micro”) • Independent, so that issues in one service are less likely to affect another • Similar to the UNIX philosophy of applications doing one thing (and doing it well).
25. Terraform • We should also prefer to keep Terraform “applications” as small as possible: • An “application” is the module where we type terraform {plan,apply} • One “application” represents a single state file • A single state file is: • A Security boundary (RBAC in TFE, ) • A Blast Radius boundary in terms of accidental destruction
26. Terraform • One state file for an entire infrastructure is A Bad Time waiting to happen • Since Terraform 0.4, we don’t have to do this! • Prefer to keep configuration for each aspect of a system in a separate “application” • Aspects of the system live in a hierarchy • Refer to outputs of “applications” lower in the hierarchy by using the terraform_remote_state data source
27. Bastion Host “Application” Web Site “Application” Network “Application” terraform_remote_state Private subnet IDs Public subnet IDs terraform_remote_state Private subnet IDs
28. Prefer to have more state files which are individually smaller
29. “organizations which design systems... are constrained to produce designs which are copies of the communication structures of these organizations.” — Melvin Conway, 1967
30. The “Composition Root” Pattern
32. The “Composition Root” Pattern • “Where should we compose object module graphs?” • “As close as possible to the application’s entry point” • “A Composition Root is a (preferably) unique location in an application where modules are composed together.” • “Only at the entry point of the application is the entire module graph finally composed” • “Only applications should have composition roots. Libraries and frameworks shouldn’t have them”
33. Applying this to Terraform…
34. Modules should either manage resources or compose other modules
35. There should be one composition root per application, at the root, parameterised for environments
36. Composition Root Configuration • Composition Roots should consist of very little code and minimal logic: • Variables - only things that need to change per deployment or workspace at the current time! • State Storage Configuration - to confi gure the backend. • Data Sources - to query the state of the cloud based on variables passed in, and to link to other “applications” in the same system. • Module Declarations - to instantiate the modules which create resources. • Outputs - only the things that need to be provided to other stacks at the current time!
38. Resource Modules
39. “SOLID” Principles
40. Resource Modules • The modules instantiated by composition roots should be: • Simple - no “clever” hacks unless they absolutely cannot be avoided. • Flat - no nested module dependencies. • Pure - treat them like functions. • More importantly they should be: • Cohesive - resources related to the same aspect of a system should live in the same module. • Decoupled - no dependencies on other modules, only on the variables they are passed (and potentially limited data sources).
41. Resource Module Structure
42. Module Design - An Interface
43. Module Design - Resources repo.tf dns.tf stage.tf
44. Resource Module Anti-Patterns • Modules to create an individual resource or small group of resources which expose every option as a configuration point • Modules are an abstraction - treat them that way • Modules which instantiate other modules • State file gets complex, as do resource paths (especially pre-Terraform 0.12) • Refactoring is hard • Complexity spirals out of control • Golden rule: Only composition roots instantiate modules
45. This approach scales well both with infrastructure size and number of infrastructure developers
46. Enforcing a level of rigour regarding this in your organisation helps given the fact that the language does not provide much.
47. Testing • Testing has been a thing in application development for some time now. Why don’t we apply it to infrastructure? • The “testing pyramid” defines various stages of testing for software: • Unit Tests - Fast, do not communicate without external resources. Verify results “in the small”. Lots of them. • The unit is the test NOT the test subject. • Acceptance Tests - Slow(er), communicate with external resources and verify results throughout a system. Relatively fewer of them.
48. Applying this to Terraform • HashiCorp Sentinal • GruntWork Terratest • ServerSpec • Pulumi Testing Framework and runner • Detailed analysis out of scope for this talk, there are other talks at this conference about this with more information (similarly at FOSDEM with videos)
49. For a step-change in the usability of Infrastructure of Code, we need to instead think of Infrastructure as Software
50. Resources • Any software engineering book covering functional or object oriented design. • Terraform Design Guide on the HashiCorp website • HashiDays NYC 2017 Talk - Open source code illustrating this pattern in AWS for a wide variety of infrastructure pieces: • Code: https://github.com/jen20/hashidays-nyc • For Terraform 0.11 • Pull Requests Accepted™