Share
Embed
Terraform - From beginner to advanced usage
The why, what and how to leverage Terraform to manage Cloud resources safely.
Experience feedback from adoption by Leboncoin DataEngineering team.
In these slides you will find introduction material for beginners and advanced use cases you will quickly be facing when working within a team and with enterprise constraints.
Xavier Krantz
October 26, 2018
Transcript
85 slides
-
Xavier Krantz - Site Reliability Engineer @Leboncoin Previously: • Criteo
• Viadeo • Smile (OSS integrator) https://github.com/xakraz https://speakerdeck.com/xakraz https://fr.linkedin.com/in/xavierkrantz/en About Me -
Introduction • Terraform 101 - Bases • Terraform 102 -
Working together • Terraform 103 - Easier, Better, Stronger • Terraform 104 - Automation & Tooling Conclusion Agenda -
Introduction AWS “management” today @lbcde • Web console • Python
boto scripts (some) -
Introduction Needs • A way to work as a team
• A way to document our work • History -
Introduction Existing tools • Code libraries • Config management •
AWS Service / Other SaaS https://www.terraform.io/intro/vs/index.html -
Introduction Existing tools • Code libraries • Config management •
AWS Service / Other SaaS https://www.terraform.io/intro/vs/index.html -
Introduction Existing tools • Code libraries • Config management •
AWS Service / Other SaaS https://www.terraform.io/intro/vs/index.html -
Terraform 101 Overview Terraform is a tool for building, changing
and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions. terraform.io/intro -
Terraform 101 Overview What is Terraform • Infrastructure as code
• Execution plan • Resource graph • Change automation tool https://www.terraform.io/intro/index.html -
Terraform 101 Concepts: • Providers • Resources A TANGIBLE component
of you infrastructures • Provider specific • What you want to manage resource "aws_db_instance" "timeout_example" { allocated_storage = 10 engine = "mysql" engine_version = "5.6.17" instance_class = "db.t1.micro" name = "mydb" # ... timeouts { create = "60m" delete = "2h" } } -
Terraform 101 Concepts: • Providers • Resources • Data sources
A specific “dynamic” data you want • External source • Like dynamic variables # Find the latest available AMI that is tagged with Component = web data "aws_ami" "web" { filter { name = "state" values = ["available"] } filter { name = "tag:Component" values = ["web"] } most_recent = true } -
Terraform 101 Concepts: • Providers • Resources • Data sources
• Variables Parameters of our code • Have to be declared specifically • Different types (String, boolean, maps, list, …) • Can have defaults variable "key" { type = "string" } variable "images" { type = "map" default = { us-east-1 = "image-1234" us-west-2 = "image-4567" } } variable "zones" { default = ["us-east-1a", "us-east-1b"] } -
Terraform 101 Concepts: • Providers • Resources • Data sources
• Variables • Outputs Outputs = Informations we want to get after Terraform has run • Can be queried via CLI • Will be shared across modules and resources output "address" { value = "${aws_instance.db.public_dns}" } -
Terraform 101 Files • *.tf • *.tfvars *.auto.tfvars terraform.tfvars •
terraform.tfstate https://www.terraform.io/intro/getting-started/install.html Basics: • Files -
Terraform 101 4 Main commands • terraform init • terraform
plan • terraform apply • terraform destroy https://www.terraform.io/intro/getting-started/install.html Basics: • Files • Commands -
Terraform 101 Other capabilities • Templates / Files • Provisioner
• Built-in “functions” • Basic conditionals https://www.terraform.io/intro/getting-started/provision.html https://www.terraform.io/docs/configuration/interpolation.html Basics: • Files • Commands • Others -
Terraform 102 Internals 1 - Pre-Compiles Check syntax, types …
Validate resources 5 - Applies Makes the API call to apply the changes described in the plan 2 - Refresh the state Call the providers APIs to get an updated view 4 - Plan Computes the plan to match the desired state 3 - Compiles 2 Runs DataSources Instantiates the resources -> Gets desired state Terraform internals 6 - Applies Updates the final state file -
Terraform 102 Internals 1 - Pre-Compiles Check syntax, types …
Validate resources 5 - Applies Makes the API call to apply the changes described in the plan 2 - Refresh the state Call the providers APIs to get an updated view 4 - Plan Computes the plan to match the desired state 3 - Compiles 2 Runs DataSources Instantiates the resources -> Gets desired state Terraform internals 6 - Applies Updates the final state file -
Terraform Remote state “Backend”: example Terraform 102 Remote state backend.tf
terraform { backend "s3" { bucket = "mybucket_name" key = "path/to/my/key" } } -
Terraform Remote state “Backend”: • S3 • + DynamoDB Terraform
102 State locking backend.tf terraform { backend "s3" { bucket = "my_bucket_name" encrypt = "true" dynamodb_table = "my_ddb_table)name" region = "eu-west-1" role_arn = "arn:aws:iam::xxxxxxxxxxxx:role/AssumeRole" } } -
Terraform 103 Modules Terraform Modules • Reusable set of “pre”
defined / packaged resources • Helps to model the architecture Features: • Versioned • Various sources: ◦ HTTP ◦ SCM (git, svn, hg, …) ◦ Local file system https://registry.terraform.io/ https://www.terraform.io/docs/modules/index.html -
Terraform 103 Modules Terraform Modules privacy-access.tf module "privacy-access" { source
= "modules/privacy-access" instance_count = "${var.access_instance_count}" instance_type = "${var.access_instance_type}" … } -
Terraform 103 Modules Terraform Modules data-privacy/ | ├── code/ ├──
modules/ ├── shared/ └── README.md -
Terraform 103 Modules Terraform Modules data-privacy/ | ├── code/ ├──
modules/ ├── shared/ └── README.md data-privacy/ └── modules/ └── privacy-access/ ├── alb.tf ├── ec2.tf ├── iam.tf ├── rds.tf ├── s3.tf ├── sg.tf | ├── outputs.tf └── input.tf -
Terraform 103 Modules Terraform Modules data-privacy/ | ├── code/ ├──
modules/ ├── shared/ └── README.md data-privacy/ └── modules/ └── privacy-access/ ├── alb.tf ├── ec2.tf ├── iam.tf ├── rds.tf ├── s3.tf ├── sg.tf | ├── outputs.tf └── input.tf data-privacy/ └── code/ ├── modules -> ../modules/ ├── vars/ │ ├── aws-account/ │ │ ├── datadev.tfvars -> │ │ └── dataprod.tfvars -> │ │ │ └── env/ │ ├── prod.tfvars │ ├── qa0.tfvars │ └── qa2.tfvars │ ├── backend.conf ├── backend.tf -> ../shared/backend.tf ├── shared-variables.tf -> │ ├── privacy-access.tf ├── privacy-request.tf │ ├── route53.tf ├── security_groups.tf │ ├── tf-config.tf ├── data-sources.tf ├── outputs.tf └── variables.tf -
Terraform Remote state “data source” Terraform 103 Remote state access
data-privacy/scripts/provision/terraform/code/data-sources.tf data "terraform_remote_state" "spark" { backend = "s3" config{ bucket = "my_bucket_name" region = "${var.region}" key = "env:/${var.env_type}/spark/main.tfstate" } } -
Terraform Remote state “data source” Terraform 103 Remote state access
data-privacy/scripts/provision/terraform/code/data-sources.tf data "terraform_remote_state" "spark" { backend = "s3" config{ bucket = "data-engineering.infrastructure.leboncoin.io-tfstates" region = "${var.region}" key = "env:/${var.env_type}/spark/main.tfstate" } } privacy-access.tf module "privacy-access" { source = "modules/privacy-access" # Spark shared cluster spark_role = "${data.terraform_remote_state.spark.spark_role}" spark_security_group_id = "${data.terraform_remote_state.spark.spark_sg}" instance_count = "${var.access_instance_count}" instance_type = "${var.access_instance_type}" ... } -
Terraform 103 Remote state access { version: 3, terraform_version: "0.11.3",
serial: 43, lineage: "c188d838-a1a0-419a-b04d-31ccb92b6e2c", modules: [ { path: [ "root" ], outputs: { spark_master_dns: { sensitive: false, type: "list", value: [ "spark-master-qa-0.data.mydomain.io" ] }, spark_master_ips: { sensitive: false, type: "list", value: [ "172.17.32.207" ] }, spark_role: { sensitive: false, type: "string", value: "spark-s3rw-qa" }, spark_sg: { sensitive: false, type: "string", value: "sg-xxxxxxxx" } }, -
Terraform 103 Workspaces Terraform States “workspaces” • 1st the Monolith
main.tf terraform.tfvars -
Terraform 103 Workspaces Terraform States “workspaces” • 2nd the split
backend.tf main.tf ec2.tf route53.tf security-groups.tf terraform.tfvars -
Terraform 103 Workspaces Terraform States “workspaces” • 3rd ENVs segregation
WHY ? →Use a variable in Backend config ? -
Terraform 103 Workspaces Terraform States “workspaces” • 3rd ENVs segregation
WHY ? →Use a variable in Backend config ? -
Terraform 103 - Workspaces Workspaces Terraform States “workspaces” • 4th
the State workspace -
Why automation ? • Terraform 104 Automate you needs data-privacy/
| ├── code/ ├── modules/ ├── shared/ └── README.md data-privacy/ └── code/ ├── modules -> ../modules/ ├── vars/ │ ├── aws-account/ │ │ ├── datadev.tfvars -> │ │ └── dataprod.tfvars -> │ │ │ └── env/ │ ├── prod.tfvars │ ├── qa0.tfvars │ └── qa2.tfvars │ ├── backend.conf ├── backend.tf -> ../shared/backend.tf ├── shared-variables.tf -> │ ├── privacy-access.tf ├── privacy-request.tf │ ├── route53.tf ├── security_groups.tf │ ├── tf-config.tf ├── data-sources.tf ├── outputs.tf └── variables.tf data-privacy/ └── modules/ └── privacy-access/ ├── alb.tf ├── ec2.tf ├── iam.tf ├── rds.tf ├── s3.tf ├── sg.tf | ├── outputs.tf └── input.tf -
Why automation ? Terraform 104 Automate you needs $ cd
YOUR_PROJECT_PATH $ terraform init -backend-config=./backend.conf $ terraform apply -var-file=./vars/env/{env}.tfvars -var-file=./vars/aws-account/{aws_account}.tfvars -
Automated actions via “invoke” Terraform 104 Automate you needs $
invoke -l Available tasks: ... provision.apply Update the whole stack (More with '--help') provision.destroy Destroy the aws resources (More with '--help') provision.init Initialize Terraform (More with '--help') provision.list-stack-envs provision.list-stacks provision.status Display IDs of current resources (More with '--help') ... -
Terraform 104 State Drift Detection • TF is imperative by
usage (No daemon) • For better readability -> Split your code in “Stacks” • Shared with data-sources among teams • Manual actions in the AWS Console or other projects https://www.hashicorp.com/blog/detecting-and-managing-drift-with-terraform https://medium.com/build-acl/state-drift-detection-using-terraform-d0383628d2ea Monitoring -
Conclusion Terraform: • 1 binary, for every OS • Wide
range of providers • Simple concepts Answers our needs: • Infra as Code • Operations safety • Share and reuse with ease -
Links References Official doc: • Terraform.io Modules registry: • registry.terraform.io
Some inspiring presentations • https://speakerdeck.com/jmickey/introduction-to-terraform • https://speakerdeck.com/so0k/terraform-at-honestbee Good tools: • https://github.com/camptocamp/terraboard • https://github.com/28mm/blast-radius • https://github.com/segmentio/terraform-docs • https://github.com/coinbase/terraform-landscape • https://github.com/shuaibiyy/awesome-terraform