Hacking Terraform by Constantin Weisser

Transcript

1. Hacking Terraform Engineer your Migration to IaC the Smart Way 11.03.2020, DevOps Gathering 2020 1
2. Infrastructure & Automation Lead at Novatec Consultant, Trainer, Traveller, Music Addict 2 Constantin Weißer | iSibnZe
3. 3 Working on Infrastructure has changed … a lot!
4. • Productive infrastructure was provisioned without Terraform • Other IaC tool was used (e.g. proprietary) • Multiple code bases need to merge • … Now, you want to migrate to Terraform! The Problem 4
5. Terraform Internals 5
6. Terraform Basics 6 The Cloud Cloud API Terraform Code State File Terraform (Provider) Plan File
7. Terraform Reverse Engineered 7 The Cloud Cloud API Terraform Code State File Terraform (Provider) Plan File EMPTY IMPORT Custom Code Generator PICK 1. 2. 3. Schema
8. Terraform Reverse Engineered 8 The Cloud Cloud API Terraform Code State File Terraform (Provider) Plan File EMPTY IMPORT Custom Code Generator PICK 1. 2. 3. Schema
9. Hacker’s Setup 9
10. ▪ Kotlin + JUnit Run single functions without coding control flow ▪ Embedded bash snippets + JSON Communication Terraform ⇔ Kotlin ▪ JSONPath + Kotlin built-ins Examine and transform data Choose your Weapons! 10
11. bash { """ echo 'What is a type system????' exit 42 """ } Merging the Good and the Ugly 11
12. val format = "+%H:%M:%S" println(bashCaptureOutput {"date $format"}) // 17:09:25 $format is not a Bash Variable! 12
13. ▪ bash{""} Run a bash script and redirect stdout/stderr to parent process ▪ bashCaptureOutput{""} Run a bash script and capture output as a string ▪ bashCaptureJson{""} Run a bash script and parse output as JSON into Kotlin types Toolkit 13
14. Picking Resources to Import 14
15. ▪ Compose a “shopping list” ▪ Multiple ways to proceed − Compose the list manually − Query a cloud API − Use a preexisting IaC tool In the end we need resource IDs Picking Resources for Import 15
16. val importRgs = bashCaptureJson { """ az group list \ | jq '[ .[] | select (.name | test("^importtest")) ]' """ } as List<Map<String, Any?>> Identifying Resources (2) 16
17. val importResources = importRgs.flatMap { bashCaptureJson { """ az resource list -g "${it["name"]}" """ } as List<Map<String, Any?>> }.plus(importRgs) Identifying Resources (3) 17
18. Migration Recipe 18 The Cloud Cloud API Terraform Code State File Terraform (Provider) Plan File EMPTY IMPORT Custom Code Generator PICK 1. 2. 3. Schema
19. terraform import \ -allow-missing-config \ azurerm_resource_group.importest-rg \ "/subscriptions/.../resourceGroups/rg" Mapping to Terraform Types 19
20. val workingDir = createTempDir() cd(workingDir) { file(File(it, "main.tf")) { "provider azurerm {}" } bash {""" terraform init -no-color terraform validate -no-color """} } Setup a new empty Terraform project 20
21. fun azureIdToNamedResource(id: String, name: String): String = when { Regex(".*Microsoft.DBforMySQL/servers/.*$").matches(id) -> "azurerm_mysql_server.$name" // ... more mappings else -> throw IllegalArgumentException("Unknown resource type $id") } Mapping 21
22. importResources.forEach { bash { """ terraform import -allow-missing-config "${ azureIdToNamedResource( it["id"] as String, it["name"] as String ) }" "${it["id"]}" """ } } Import 22
23. 1. Produce our shopping list 2. Init terraform in an empty directory 3. Import with -allow-missing-config From this point on, we are “cloud-agnostic” (Disclaimer: Some providers still need tending) Quick recap: What has happened so far? 23 BINGO
24. Migration Recipe 24 The Cloud Cloud API Terraform Code State File Terraform (Provider) Plan File EMPTY IMPORT Custom Code Generator PICK 1. 2. 3. Schema
25. Generating Terraform Code 25
26. provider generated terraform code base Universal Code Generator 26 state file with no code plan file schema file provider schema plan Code generator All resources are marked for deletion! plan command yields empty plan
27. Plan file contains specified and computed attributes Read the schema to get the essentials! The provider schema has all the metadata we need! 27
28. Post-order tree traversal on the plan file 28 / R R A B A A / R R A B A A R R Plan Schema B Nodes are data Nodes are types B
29. Each entry is one of ▪ Attribute / Single value → Value syntax is JSON ▪ Block → Identical to a Resource (except for the name) → Recurse Post-order tree traversal on the plan file 29
30. ▪ Resource import: easy ▪ Mapping to terraform types cloud-specific (effort!) ▪ General-purpose code generator in 60 LOC ▪ Quality depends on the provider − Azure schema yields good results − AWS schema is puzzling ▪ Better results with dedicated code generators! − Specific code for each resource type − It scales! (Time savings for big migrations) Summary & Observations 30
31. ▪ Semi-automate! ▪ Split import & and code generation! ▪ Don’t strive for perfection! Learnings 31
32. Feel free to reach out: Constantin Weißer | @iSibnZe constantin.weisser@novatec-gmbh.de Thank you for being here! Questions? Discussions? 32
33. ▪ https://github.com/i7c/hackingtf ▪ https://www.terraform.io/docs/import/index.html ▪ https://www.pulumi.com/blog/testing-your-infrastructure-a s-code-with-pulumi/ Resources 33