Autopilot, but never let go of the wheel

From TOIL to Continuous Delivery of Infrastructure, our tail of migrating our existing Infrastructure as code tools & wrappers so that they can be used in a CD system, but with all of the control grey-beards, enterprises & governments expect.

A tail of how we took our terraform tooling from being human focussed (and thus causing much more TOIL than was reasonable) and adapted it work within a sane set of pipelines, enabling drift checking, automated deployments & approved deployments that fit with in a multi-environment, sovereign-control organisation, while still retaining the ability to "run from your laptop" in an emergency or in bootstrap mode. We'll also cover the patterns that emerged for building tools that are both human & tool friendly, for progressive roll-out of changes & why CI/CD for VM based infrastructure requires better techniques than "fixing the build".

Transcript

1. Autopilot, but never let go of the wheel Evolving interactive tools for use in Continuous Integration & Delivery
2. ./tf.sh –c aws \ –e dev –a plan In the beginning.
3. ..which quickly turned into.. • git checkout –b JIRA-001 • vi aws/foo.tf • ./tf.sh –c aws –e dev –a plan | pbcopy • printf "My PR\n\n$(pbpaste)" | hub pull-request -p -F - • # pester colleagues to approve & merge the PR • git checkout master && git pull • ./tf.sh –c aws –e dev –a apply
4. …which then resulted in… Error: Terraform doesn't allow running any operations against a state that was written by a future Terraform version. The state is reporting it is written by Terraform '0.11.11' Please run at least that version of Terraform to continue.
5. Lesson 1: Pin the versions of your tools
6. Lesson 2: Understand why your wrapper exists •To make workflow easier! •But how? •Take care of auth & setup? •Always run commands (terraform init?) •Try & keep these working in both interactive & non- interactive with minimal change to existing behavior (looks for hints of non-interactive use
7. Lesson 3: Cloud Authentication Techniques • Hashicorp Packer’s Azure RM builder requires different parameters in the JSON depending on authentication type in use (Azure CLI versus MSI) • Use jq to filter out part of the packer template:
8. BONUS: Comments in Packer JSON!
9. Lesson 4: Enforce non-interactive modes • Most people terraform apply “yes” when working interactively • Prepare: So fail when config is incomplete -input=false • Non-interactive plan/apply cycles need to capture the plan in the • -out=path • Automation tool needs to store & retrieve for approve workflows
10. Sidebar: Never start in bash • It’s my default starting point, it shouldn’t be • when you’re past your 5th line of bash, you should turn back • Python & go are better options, but have their own distribution problems (git add thing.sh is hard to beat for simplicity) • Which probably highlights a different problem, we’re still living in a git pull model instead of packaging releases
11. Lesson 5: Start with low privilege API creds • All of our interactive users had root-mode everywhere • This caused pain when we: • Added team members who still had training wheels • Correctly refused to give automation services root API access • Azure’s Roles & Resource Group model made this easier to scope access (access to the few resource groups instead of the whole subscription, keep sensitive data in separate RG to reduce risk) • Azure’s many existing roles may not fit your needs
12. Sidebar: Secrets Management • Once you’ve made a start with one technique, it’s incredibly sticky • git-crypt & SOPS aren’t not really up to scratch for terraform use • Terraform: Invest early in vault, design models that fit with your security model • We have multiple unconnected deployments, they can’t share a vault server
13. THANK YOU • Simon McCartney • Twitter: @simonmcc • Work: @axon_uk & @axon_us • We’re hiring! • US – Seattle, Scottsdale & Remote • Vietnam – Ho Chi Minh City • Australia - Sydney
14. Thank you