Terraform - From beginner to advanced usage

The why, what and how to leverage Terraform to manage Cloud resources safely.

Experience feedback from adoption by Leboncoin DataEngineering team.

In these slides you will find introduction material for beginners and advanced use cases you will quickly be facing when working within a team and with enterprise constraints.

Transcript

1. Demystifying Terraform to manage AWS @lbcde 2018-10-26
2. Xavier Krantz - Site Reliability Engineer @Leboncoin Previously: • Criteo • Viadeo • Smile (OSS integrator) https://github.com/xakraz https://speakerdeck.com/xakraz https://fr.linkedin.com/in/xavierkrantz/en About Me
3. Introduction • Terraform 101 - Bases • Terraform 102 - Working together • Terraform 103 - Easier, Better, Stronger • Terraform 104 - Automation & Tooling Conclusion Agenda
4. Introduction AWS “management” today @lbcde • Web console • Python boto scripts (some)
5. Introduction Needs • A way to work as a team • A way to document our work • History
6. Introduction Existing tools • Code libraries • Config management • AWS Service / Other SaaS https://www.terraform.io/intro/vs/index.html
7. Introduction Existing tools • Code libraries • Config management • AWS Service / Other SaaS https://www.terraform.io/intro/vs/index.html
8. Introduction Existing tools • Code libraries • Config management • AWS Service / Other SaaS https://www.terraform.io/intro/vs/index.html
9. Terraform 101 Bases
10. Terraform 101 Bases Overview Concepts Basics
11. Terraform 101 Overview Terraform is a tool for building, changing and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions. terraform.io/intro
12. Terraform 101 Overview What is Terraform • Infrastructure as code • Execution plan • Resource graph • Change automation tool https://www.terraform.io/intro/index.html
13. Terraform 101 Concepts
14. Terraform 101 Concepts: • Providers
15. Terraform 101 Concepts: • Providers • Resources A TANGIBLE component of you infrastructures • Provider specific • What you want to manage resource "aws_db_instance" "timeout_example" { allocated_storage = 10 engine = "mysql" engine_version = "5.6.17" instance_class = "db.t1.micro" name = "mydb" # ... timeouts { create = "60m" delete = "2h" } }
16. Terraform 101 Concepts: • Providers • Resources • Data sources A specific “dynamic” data you want • External source • Like dynamic variables # Find the latest available AMI that is tagged with Component = web data "aws_ami" "web" { filter { name = "state" values = ["available"] } filter { name = "tag:Component" values = ["web"] } most_recent = true }
17. Terraform 101 Concepts: • Providers • Resources • Data sources • Variables Parameters of our code • Have to be declared specifically • Different types (String, boolean, maps, list, …) • Can have defaults variable "key" { type = "string" } variable "images" { type = "map" default = { us-east-1 = "image-1234" us-west-2 = "image-4567" } } variable "zones" { default = ["us-east-1a", "us-east-1b"] }
18. Terraform 101 Concepts: • Providers • Resources • Data sources • Variables • Outputs Outputs = Informations we want to get after Terraform has run • Can be queried via CLI • Will be shared across modules and resources output "address" { value = "${aws_instance.db.public_dns}" }
19. Terraform 101 Basics
20. Terraform 101 Files • *.tf • *.tfvars *.auto.tfvars terraform.tfvars • terraform.tfstate https://www.terraform.io/intro/getting-started/install.html Basics: • Files
21. Terraform 101 4 Main commands • terraform init • terraform plan • terraform apply • terraform destroy https://www.terraform.io/intro/getting-started/install.html Basics: • Files • Commands
22. Terraform 101 Other capabilities • Templates / Files • Provisioner • Built-in “functions” • Basic conditionals https://www.terraform.io/intro/getting-started/provision.html https://www.terraform.io/docs/configuration/interpolation.html Basics: • Files • Commands • Others
23. Terraform 102 Working together
24. Terraform 102 Working together TF internals Remote state State locking
25. Terraform 102 Internals
26. Terraform 102 Internals 1 - Pre-Compiles Check syntax, types … Validate resources 5 - Applies Makes the API call to apply the changes described in the plan 2 - Refresh the state Call the providers APIs to get an updated view 4 - Plan Computes the plan to match the desired state 3 - Compiles 2 Runs DataSources Instantiates the resources -> Gets desired state Terraform internals 6 - Applies Updates the final state file
27. Terraform 102 Internals 1 - Pre-Compiles Check syntax, types … Validate resources 5 - Applies Makes the API call to apply the changes described in the plan 2 - Refresh the state Call the providers APIs to get an updated view 4 - Plan Computes the plan to match the desired state 3 - Compiles 2 Runs DataSources Instantiates the resources -> Gets desired state Terraform internals 6 - Applies Updates the final state file
28. Terraform 102 - Internals
29. Terraform 102 - Internals ? ?
30. Terraform 102 Remote State
31. Terraform Remote state “Backend”: principles Terraform 102 Remote state
32. Terraform Remote state “Backend”: types Terraform 102 Remote state
33. Terraform Remote state “Backend”: example Terraform 102 Remote state backend.tf terraform { backend "s3" { bucket = "mybucket_name" key = "path/to/my/key" } }
34. Terraform 102 State locking
35. Terraform 102 - State locking
36. Terraform 102 - State locking
37. Terraform 102 - State locking ?
38. Terraform Remote state “Backend”: • S3 • + DynamoDB Terraform 102 State locking backend.tf terraform { backend "s3" { bucket = "my_bucket_name" encrypt = "true" dynamodb_table = "my_ddb_table)name" region = "eu-west-1" role_arn = "arn:aws:iam::xxxxxxxxxxxx:role/AssumeRole" } }
39. Terraform 102 - State locking
40. Terraform 102 - State locking
41. Terraform 103 Better, easier, stronger
42. Terraform 103 Better, easier, stronger Modules Remote state access Workspaces
43. Terraform 103 Modules
44. Terraform 103 Modules Terraform Modules • Reusable set of “pre” defined / packaged resources • Helps to model the architecture Features: • Versioned • Various sources: ◦ HTTP ◦ SCM (git, svn, hg, …) ◦ Local file system https://registry.terraform.io/ https://www.terraform.io/docs/modules/index.html
45. Terraform 103 Modules Terraform Modules privacy-access.tf module "privacy-access" { source = "modules/privacy-access" instance_count = "${var.access_instance_count}" instance_type = "${var.access_instance_type}" … }
46. Terraform 103 Modules Terraform Modules data-privacy/ | ├── code/ ├── modules/ ├── shared/ └── README.md
47. Terraform 103 Modules Terraform Modules data-privacy/ | ├── code/ ├── modules/ ├── shared/ └── README.md data-privacy/ └── modules/ └── privacy-access/ ├── alb.tf ├── ec2.tf ├── iam.tf ├── rds.tf ├── s3.tf ├── sg.tf | ├── outputs.tf └── input.tf
48. Terraform 103 Modules Terraform Modules data-privacy/ | ├── code/ ├── modules/ ├── shared/ └── README.md data-privacy/ └── modules/ └── privacy-access/ ├── alb.tf ├── ec2.tf ├── iam.tf ├── rds.tf ├── s3.tf ├── sg.tf | ├── outputs.tf └── input.tf data-privacy/ └── code/ ├── modules -> ../modules/ ├── vars/ │ ├── aws-account/ │ │ ├── datadev.tfvars -> │ │ └── dataprod.tfvars -> │ │ │ └── env/ │ ├── prod.tfvars │ ├── qa0.tfvars │ └── qa2.tfvars │ ├── backend.conf ├── backend.tf -> ../shared/backend.tf ├── shared-variables.tf -> │ ├── privacy-access.tf ├── privacy-request.tf │ ├── route53.tf ├── security_groups.tf │ ├── tf-config.tf ├── data-sources.tf ├── outputs.tf └── variables.tf
49. Terraform 103 Remote state access
50. Terraform Remote state “data source” Terraform 103 Remote state access data-privacy/scripts/provision/terraform/code/data-sources.tf data "terraform_remote_state" "spark" { backend = "s3" config{ bucket = "my_bucket_name" region = "${var.region}" key = "env:/${var.env_type}/spark/main.tfstate" } }
51. Terraform Remote state “data source” Terraform 103 Remote state access data-privacy/scripts/provision/terraform/code/data-sources.tf data "terraform_remote_state" "spark" { backend = "s3" config{ bucket = "data-engineering.infrastructure.leboncoin.io-tfstates" region = "${var.region}" key = "env:/${var.env_type}/spark/main.tfstate" } } privacy-access.tf module "privacy-access" { source = "modules/privacy-access" # Spark shared cluster spark_role = "${data.terraform_remote_state.spark.spark_role}" spark_security_group_id = "${data.terraform_remote_state.spark.spark_sg}" instance_count = "${var.access_instance_count}" instance_type = "${var.access_instance_type}" ... }
52. Terraform 103 Remote state access { version: 3, terraform_version: "0.11.3", serial: 43, lineage: "c188d838-a1a0-419a-b04d-31ccb92b6e2c", modules: [ { path: [ "root" ], outputs: { spark_master_dns: { sensitive: false, type: "list", value: [ "spark-master-qa-0.data.mydomain.io" ] }, spark_master_ips: { sensitive: false, type: "list", value: [ "172.17.32.207" ] }, spark_role: { sensitive: false, type: "string", value: "spark-s3rw-qa" }, spark_sg: { sensitive: false, type: "string", value: "sg-xxxxxxxx" } },
53. Terraform 103 Workspaces
54. Terraform 103 Workspaces Terraform States “workspaces” • 1st the Monolith main.tf terraform.tfvars
55. Terraform 103 Workspaces Terraform States “workspaces” • 2nd the split backend.tf main.tf ec2.tf route53.tf security-groups.tf terraform.tfvars
56. Terraform 103 Workspaces Terraform States “workspaces” • 3rd ENVs segregation
57. Terraform 103 Workspaces Terraform States “workspaces” • 3rd ENVs segregation
58. Terraform 103 Workspaces Terraform States “workspaces” • 3rd ENVs segregation
59. Terraform 103 Workspaces Terraform States “workspaces” • 3rd ENVs segregation WHY ? →Use a variable in Backend config ?
60. Terraform 103 Workspaces Terraform States “workspaces” • 3rd ENVs segregation WHY ? →Use a variable in Backend config ?
61. Terraform 103 - Workspaces Workspaces Terraform States “workspaces” • 4th the State workspace
62. Terraform 103 - Workspaces
63. Terraform 104 Tooling & Automation
64. Terraform 104 Tooling & Automation Automation Monitoring Tools
65. Terraform 104 Automate your needs To meet your workflow
66. Why automation ? • Terraform 104 Automate you needs data-privacy/ | ├── code/ ├── modules/ ├── shared/ └── README.md data-privacy/ └── code/ ├── modules -> ../modules/ ├── vars/ │ ├── aws-account/ │ │ ├── datadev.tfvars -> │ │ └── dataprod.tfvars -> │ │ │ └── env/ │ ├── prod.tfvars │ ├── qa0.tfvars │ └── qa2.tfvars │ ├── backend.conf ├── backend.tf -> ../shared/backend.tf ├── shared-variables.tf -> │ ├── privacy-access.tf ├── privacy-request.tf │ ├── route53.tf ├── security_groups.tf │ ├── tf-config.tf ├── data-sources.tf ├── outputs.tf └── variables.tf data-privacy/ └── modules/ └── privacy-access/ ├── alb.tf ├── ec2.tf ├── iam.tf ├── rds.tf ├── s3.tf ├── sg.tf | ├── outputs.tf └── input.tf
67. Why automation ? Terraform 104 Automate you needs $ cd YOUR_PROJECT_PATH $ terraform init -backend-config=./backend.conf $ terraform apply -var-file=./vars/env/{env}.tfvars -var-file=./vars/aws-account/{aws_account}.tfvars
68. Automated actions via “invoke” Terraform 104 Automate you needs $ invoke -l Available tasks: ... provision.apply Update the whole stack (More with '--help') provision.destroy Destroy the aws resources (More with '--help') provision.init Initialize Terraform (More with '--help') provision.list-stack-envs provision.list-stacks provision.status Display IDs of current resources (More with '--help') ...
69. Terraform 104 Monitoring
70. Terraform 104 State Drift Detection • TF is imperative by usage (No daemon) • For better readability -> Split your code in “Stacks” • Shared with data-sources among teams • Manual actions in the AWS Console or other projects https://www.hashicorp.com/blog/detecting-and-managing-drift-with-terraform https://medium.com/build-acl/state-drift-detection-using-terraform-d0383628d2ea Monitoring
71. Terraform 104 State Drift Detection https://github.com/gibbster/terraform-plan-drift-checker Monitoring
72. Terraform 104 Misc tooling
73. Pre-commit - Terraform Terraform 104 Tools https://pre-commit.com/ https://github.com/antonbabenko/pre-commit-terraform
74. Terraform-landscape Terraform 104 Tools https://github.com/coinbase/terraform-landscape
75. Terraform-docs Terraform 104 Tools https://github.com/segmentio/terraform-docs
76. Blast Radius Terraform 104 Tools https://github.com/28mm/blast-radius
77. TerraBoard Terraform 104 Tools https://github.com/camptocamp/terraboard
78. Conclusion
79. Terraform 101 Bases Overview Concepts Basics
80. Terraform 102 Working together TF internals Remote state State locking
81. Terraform 103 Better, easier, stronger Modules Remote state access Workspaces
82. Terraform 104 Tooling & Automation Automation Monitoring Tools
83. Conclusion Terraform: • 1 binary, for every OS • Wide range of providers • Simple concepts Answers our needs: • Infra as Code • Operations safety • Share and reuse with ease
84. questions / réponses
85. Links References Official doc: • Terraform.io Modules registry: • registry.terraform.io Some inspiring presentations • https://speakerdeck.com/jmickey/introduction-to-terraform • https://speakerdeck.com/so0k/terraform-at-honestbee Good tools: • https://github.com/camptocamp/terraboard • https://github.com/28mm/blast-radius • https://github.com/segmentio/terraform-docs • https://github.com/coinbase/terraform-landscape • https://github.com/shuaibiyy/awesome-terraform