Autopilot, but never let go of the wheel Evolving interactive tools for use in Continuous Integration & Delivery 

./tf.sh –c aws \ –e dev –a plan In the beginning. 

..which quickly turned into.. • git checkout –b JIRA-001 • vi aws/foo.tf • ./tf.sh –c aws –e dev –a plan | pbcopy • printf "My PR\n\n$(pbpaste)" | hub pull-request -p -F - • # pester colleagues to approve & merge the PR • git checkout master && git pull • ./tf.sh –c aws –e dev –a apply 

…which then resulted in… Error: Terraform doesn't allow running any operations against a state that was written by a future Terraform version. The state is reporting it is written by Terraform '0.11.11' Please run at least that version of Terraform to continue. 

Lesson 1: Pin the versions of your tools 

Lesson 2: Understand why your wrapper exists •To make workflow easier! •But how? •Take care of auth & setup? •Always run commands (terraform init?) •Try & keep these working in both interactive & non- interactive with minimal change to existing behavior (looks for hints of non-interactive use 

Lesson 3: Cloud Authentication Techniques • Hashicorp Packer’s Azure RM builder requires different parameters in the JSON depending on authentication type in use (Azure CLI versus MSI) • Use jq to filter out part of the packer template: 

BONUS: Comments in Packer JSON! 

Lesson 4: Enforce non-interactive modes • Most people terraform apply “yes” when working interactively • Prepare: So fail when config is incomplete -input=false • Non-interactive plan/apply cycles need to capture the plan in the • -out=path • Automation tool needs to store & retrieve for approve workflows 

Sidebar: Never start in bash • It’s my default starting point, it shouldn’t be • when you’re past your 5th line of bash, you should turn back • Python & go are better options, but have their own distribution problems (git add thing.sh is hard to beat for simplicity) • Which probably highlights a different problem, we’re still living in a git pull model instead of packaging releases 

Lesson 5: Start with low privilege API creds • All of our interactive users had root-mode everywhere • This caused pain when we: • Added team members who still had training wheels • Correctly refused to give automation services root API access • Azure’s Roles & Resource Group model made this easier to scope access (access to the few resource groups instead of the whole subscription, keep sensitive data in separate RG to reduce risk) • Azure’s many existing roles may not fit your needs 

Sidebar: Secrets Management • Once you’ve made a start with one technique, it’s incredibly sticky • git-crypt & SOPS aren’t not really up to scratch for terraform use • Terraform: Invest early in vault, design models that fit with your security model • We have multiple unconnected deployments, they can’t share a vault server 

THANK YOU • Simon McCartney • Twitter: @simonmcc • Work: @axon_uk & @axon_us • We’re hiring! • US – Seattle, Scottsdale & Remote • Vietnam – Ho Chi Minh City • Australia - Sydney 

Thank you 

No content