Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Shaping Clouds with Terraform

Shaping Clouds with Terraform

Terraform is an open source tool that helps you control your infrastructure configuration through code. This talk will serve as a primer showing how to build a basic infrastructure in the Google Cloud and how we can re-use our code to construct multiple, identical environments.

F1e0e0c3c3196a63c9b17a2344fb6a61?s=128

Mike Fowler

April 26, 2018
Tweet

Transcript

  1. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler Shaping Clouds With Terraform Mike Fowler, Senior Site Reliability Engineer
  2. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler About Me Associate: SA, Developer Specialist: Big Data Data Engineer Cloud Architect
  3. None
  4. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • What is Terraform? • The Terraform lifecycle • A simple infrastructure in Google Cloud • Improving from the basics Overview
  5. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • Infrastructure as code software from HashiCorp • Open sourced under Mozilla Public Licence v2.0 • First released July 2014 • Latest version 0.11.7, released April 10 2018 • Written in Go • Get it: www.terraform.io Terraform
  6. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • Define your infrastructure • Initialise your execution environment • Plan your execution • Apply your changes • Destroy your infrastructure Terraform Lifecycle
  7. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • Collection of configuration .tf files in a single directory • Expressed in either HCL Terraform syntax or JSON • Declarations of – Providers – Resources – Data Sources Infrastructure Definition
  8. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • Exposes collections of programmable resources & data sources • Essentially a plugin adding the capability to manage an offering • Currently 73 different providers – Major Cloud (AWS, GCP, Azure) – Cloud (Heroku) – SaaS (PagerDuty, DNSimple, CloudFlare) – Software (PostgreSQL, MySQL, Icinga2) – Miscellaneous (Local, External, Null) Providers
  9. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • Community Providers – Not maintained by HashiCorp – Highlights include Kafka, Helm & Google Calendar • Custom/fork provider – Very easy to fork providers – Useful for quick development & future submissions – https://github.com/BashtonLtd/terraform- gcebashtonsql Additional Providers
  10. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • All fields are optional and can be set by environment variables • Credentials can be inferred from – Local configuration with gcloud auth – Service accounts on GCE instances Google Cloud Provider provider "google" { credentials = "${file("account.json")}" project = "my-gce-project-id" region = "europe-west2" }
  11. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • Resources declare what should exist • Available resources are determined by the provider used • AWS resources are completely different to GCP resources • General naming pattern is provider_resource • Each resource will require a number of arguments • Once created, each resource can provide attributes Resources
  12. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • Resource type is “google_compute_network” • Resource name in Terraform is “skynet” • name is required, everything else is optional Anatomy of a Resource resource "google_compute_network" "skynet" { name = "skynet" auto_create_subnetworks = "false" }
  13. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • Our subnetwork needs to know the network it exists in • Interpolation performed with “${}” • Naming pattern to access resource attributes is of the form: TYPE.NAME.ATTR Extending our Network resource "google_compute_subnetwork" "skynet-subnet-10" { name = "skynet-eu-west2" ip_cidr_range = "192.168.10.0/24" network = "${google_compute_network.skynet.self_link}" region = "europe-west2" }
  14. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • Many resources require other resources to exist first – subnetwork required network • Start with the resource you need and work backward • Creating a google_compute_instance will require: – Subnetwork – Zone – Image Adding More Resources
  15. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • Allows data to be fetched or computed for use in configuration • Essentially a read-only view to existing data • Avoids hard coding giving us dynamic values for things like: – Zones available in a region – Images to use for a Compute Engine Instance Data Sources
  16. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • We’re looking for zones that are UP in eu-west2 • All arguments are optional for this data source • Naming pattern to access data source attributes is of the form: data.TYPE.NAME.ATTR Finding zones in our region data "google_compute_zones" "zones" { region = "europe-west2" status = "UP" }
  17. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • family will retrieve the latest version in a family • project is the project in which the image lives • Google provides a range of Linux distros and Windows • Returns a lot of attributes but we only need self_link Choosing a Compute Image data "google_compute_image" "ubuntu" { family = "ubuntu-1710" project = "ubuntu-os-cloud" }
  18. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler Google Compute Instance resource "google_compute_instance" "instance" { zone = "${data.google_compute_zones.zones.names[0]}" name = "t-800" machine_type = "f1-micro" boot_disk { initialize_params { image = "${data.google_compute_image.ubuntu.self_link}" } } network_interface { subnetwork = "${google_compute_subnetwork.skynet-subnet-10.self_link}" } }
  19. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • terraform init prepares the environment for execution • Syntactically validates the configuration files in the local directory • Downloads providers specified in configuration files • Populates a .terraform directory with all the goods Initialisation
  20. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler terraform init $ terraform init Initializing provider plugins... - Checking for available provider plugins on https://releases.hashicorp.com... - Downloading plugin for provider "google" (1.10.0)... The following providers do not have any version constraints in configuration, so the latest version was installed. To prevent automatic upgrades to new major versions that may contain breaking changes, it is recommended to add version = "..." constraints to the corresponding provider blocks in configuration, with the constraint strings suggested below. * provider.google: version = "~> 1.10" Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.
  21. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • terraform plan determines what needs to be done • 3 way comparison between configuration, state & actual • Builds a directed acyclic graph of changes • Outputs a plan confirming what will happen Planning
  22. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler terraform plan $ terraform plan -out myplan Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. data.google_compute_image.ubuntu: Refreshing state... data.google_compute_zones.zones: Refreshing state... ------------------------------------------------------------------------ An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: + google_compute_instance.instance id: <computed> … machine_type: "f1-micro" name: "t-800" ... zone: "europe-west2-a" ...
  23. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler terraform plan + google_compute_network.skynet id: <computed> auto_create_subnetworks: "false" gateway_ipv4: <computed> name: "skynet" ... + google_compute_subnetwork.skynet-subnet-10 id: <computed> ... ip_cidr_range: "192.168.10.0/24" name: "skynet-subnet-10" ... Plan: 3 to add, 0 to change, 0 to destroy. ------------------------------------------------------------------------ This plan was saved to: myplan To perform exactly these actions, run the following command to apply: terraform apply "myplan"
  24. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • terraform apply actually changes things • Will plan if you don’t provide one and prompt before applying • Takes anywhere from a few seconds to many minutes • Changes are written to the state file (default terraform.tfstate) – JSON document of infra as it existed after last apply – Previous state preserved in terraform.tfstate.backup Applying
  25. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler terraform apply $ terraform apply "myplan" google_compute_network.skynet: Creating... auto_create_subnetworks: "" => "false" gateway_ipv4: "" => "<computed>" name: "" => "skynet" project: "" => "<computed>" routing_mode: "" => "<computed>" self_link: "" => "<computed>" google_compute_network.skynet: Still creating... (10s elapsed) google_compute_network.skynet: Creation complete after 12s (ID: skynet) google_compute_subnetwork.skynet-subnet-10: Creating... fingerprint: "" => "<computed>" gateway_address: "" => "<computed>" ip_cidr_range: "" => "192.168.10.0/24" name: "" => "skynet-subnet-10" network: "" => "https://www.googleapis.com/compute/v1/projects/myproj/global/networks/skynet" project: "" => "<computed>" region: "" => "europe-west2" self_link: "" => "<computed>" google_compute_subnetwork.skynet-subnet-10: Still creating... (10s elapsed) google_compute_subnetwork.skynet-subnet-10: Creation complete after 14s (ID: europe-west2/skynet-subnet-10)
  26. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler terraform apply google_compute_instance.instance: Creating... boot_disk.#: "" => "1" boot_disk.0.auto_delete: "" => "true" ... boot_disk.0.initialize_params.#: "" => "1" boot_disk.0.initialize_params.0.image: "" => "https://www.googleapis.../ubuntu-1710-artful-v20180405" ... can_ip_forward: "" => "false" cpu_platform: "" => "<computed>" create_timeout: "" => "4" deletion_protection: "" => "false" ... machine_type: "" => "f1-micro" metadata_fingerprint: "" => "<computed>" name: "" => "t-800" network_interface.#: "" => "1" ... network_interface.0.subnetwork: "" => "https://www.googleapis.../subnetworks/skynet-subnet-10" network_interface.0.subnetwork_project: "" => "<computed>" ... zone: "" => "europe-west2-a" google_compute_instance.instance: Still creating... (10s elapsed) google_compute_instance.instance: Creation complete after 13s (ID: t-800) Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
  27. Success! Image source: https://www.roboticstomorrow.com/article/2017/07/artificial-intelligence-and-skynet/10324/

  28. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • A backend determines how Terraform will manage state • Default is local, simply a file on disk which could be versioned • Others allow remote storage and have lock semantics – Cloud buckets (azurerm, gcs, s3) – consul – etcd • Using a versioned bucket provides automatic backups of your state Backends (Managing State)
  29. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • Backends do not use providers and must be configured • The bucket must already exist, it is not created • Using terraform init will convert the local state to remote Configuring GCS Backend terraform { backend "gcs" { bucket = "flossuk-state" credentials = "/path/to/credentials.json" region = "europe-west2" project = "my-gce-project-id" } }
  30. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler terraform init $ terraform init Initializing the backend... Do you want to copy existing state to the new backend? Pre-existing state was found while migrating the previous "local" backend to the newly configured "gcs" backend. An existing non-empty state already exists in the new backend. The two states have been saved to temporary files that will be removed after responding to this query. Previous (type "local"): /tmp/terraform797635445/1-local.tfstate New (type "gcs"): /tmp/terraform797635445/2-gcs.tfstate Do you want to overwrite the state in the new backend with the previous state? Enter "yes" to copy and "no" to start with the existing state in the newly configured "gcs" backend. Enter a value: yes Successfully configured the backend "gcs"! Terraform will automatically use this backend unless the backend configuration changes. Initializing provider plugins...
  31. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • Declared with the variable keyword • Values can be declared explicitly or prompted during plan • With a terraform.tfvars file explict values can be overriden • plan allows individual variables to be specified • plan also allows naming a file other than terraform.tfvars • Three types of variable: string, boolean, list, map Variables
  32. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • default assigns (surprise) the default value and is optional • type is only required if the type cannot be inferred from the value • description optionally allows you to provide commentary • Interpolation performed with “${var.region}” Declaring a Variable variable "region" { default = "europe-west2" type = "string" description = "GCP region to be used by default" }
  33. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler Declaring Maps & Lists variable "zones" { default = ["europe-west-2a", "europe-west-2b"] type = "list" } ${var.zones[1]} variable "images" { type = "map" default = { "europe-west-2a" = "image-1234" "europe-west-2b" = "image-4567" } } ${var.images["europe-west-2a"]}
  34. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • Simply replace the hard coded europe-west2 with our variable • Run terraform plan to confirm nothing changed Refactoring Our Example resource "google_compute_subnetwork" "skynet-subnet-10" { name = "skynet-${var.region}" ip_cidr_range = "192.168.10.0/24" network = "${google_compute_network.skynet.self_link}" region = "${var.region}" } ------------------------------------------------------------ No changes. Infrastructure is up-to-date.
  35. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • A module is a reusable collection of resources and/or data sources • Accepts arguments, provides attributes • Source locates the module – Local directory – Git tags and revisions • Community modules exist in the registry https://registry.terraform.io/ Modules
  36. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • Just a bunch of configuration files in a directory • All variables are arguments • A variable with no default value becomes a mandatory argument • Attributes of resources created are made available with output Writing a Module output "skynet-subnet" { value = "${google_compute_subnetwork.skynet-subnet-10.self_link}" }
  37. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler Refactoring our Example variable "name" {} variable "cidr" {} variable "region" { default = "europe-west2" } resource "google_compute_network" "network" { name = "${var.name}" auto_create_subnetworks = "false" } resource "google_compute_subnetwork" "subnetwork" { name = "${var.name}-${var.region}" ip_cidr_range = "${var.cidr}" network = "${google_compute_network.network.self_link}" region = "${var.region}" } output "subnetwork" { value = "${google_compute_subnetwork.subnetwork.self_link}" }
  38. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • source tells terraform where to find it, retrieved with init • The same source can be used many times with different names • Attributes retrieved using the syntax module.name.attribute Using a Module module "network" { source = "gcp-network" name = "skynet" cidr = "192.168.10.0/24" } resource "google_compute_instance" "instance" { ... network_interface { subnetwork = "${module.network.subnetwork}" } }
  39. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • terraform destroy removes everything • Determines and lists what will be destroyed • Prompts for a yes before executing • You don’t need to run init to start again Destroying our Infrastructure
  40. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler terraform destroy $ terraform destroy google_compute_network.skynet: Refreshing state... (ID: skynet) ... An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: - destroy Terraform will perform the following actions: - google_compute_instance.instance - google_compute_network.skynet - google_compute_subnetwork.skynet-subnet-10 Plan: 0 to add, 0 to change, 3 to destroy. Do you really want to destroy? Terraform will destroy all your managed infrastructure, as shown above. There is no undo. Only 'yes' will be accepted to confirm. Enter a value: yes
  41. None
  42. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler terraform destroy google_compute_instance.instance: Destroying... (ID: t-800) google_compute_instance.instance: Still destroying... (ID: t-800, 10s elapsed) google_compute_instance.instance: Still destroying... (ID: t-800, 20s elapsed) google_compute_instance.instance: Destruction complete after 27s google_compute_subnetwork.skynet-subnet-10: Destroying... (ID: europe-west2/skynet-europe-west2) google_compute_subnetwork.skynet-subnet-10: Still destroying... (ID: europe-west2/..., 10s elapsed) google_compute_subnetwork.skynet-subnet-10: Destruction complete after 18s google_compute_network.skynet: Destroying... (ID: skynet) google_compute_network.skynet: Still destroying... (ID: skynet, 10s elapsed) google_compute_network.skynet: Still destroying... (ID: skynet, 20s elapsed) google_compute_network.skynet: Destruction complete after 26s Destroy complete! Resources: 3 destroyed.
  43. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • The backend used can be specified on the command line – Can use the same configuration multiple times – e.g. Same infrastructure in multiple regions – e.g. Same infrastructure, same region, different environments • State files can be a data source – Allows layering of “stacks” – Separate persistent items from disposable items Beyond the Basics
  44. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler Example Directory Tree . ├── application │ ├── application.tf │ ├── datasources.tf │ ├── network.tf │ ├── outputs.tf │ └── variables.tf ├── global │ ├── bastion.tf │ ├── buckets.tf │ ├── datasources.tf │ ├── dns.tf │ ├── outputs.tf │ ├── terraform.tf │ ├── variables.tf │ └── vpc.tf ├── modules │ ├── Terrafile ├── region ├── datasources.tf ├── grafana.tf ├── outputs.tf ├── prometheus.tf ├── terraform.tf ├── variables.tf
  45. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • Provisioners • Terrafile (https://github.com/claranet/python-terrafile) • tfenv (https://github.com/kamatama41/tfenv) Worthy Mentions
  46. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler • Terraform’s lifecycle is init, plan, apply and eventually destroy • Make use of data sources and variables to be generic as possible • Make use of a remote backend and many backends • Use modules to save time Summary
  47. Spring Conference, April 26th-27th 2018 Dynamic Earth, Edinburgh Mike Fowler

    mlfowler mike dot fowler at claranet dot uk gh-mlfowler Questions