What's new in Terraform 0.10 - HUG

In this talk we cover an introduction to Terraform, including it's primary use cases of Infrastructure as Code, composing multiple providers (such as IaaS, SaaS, and PaaS), and enabling self service of infrastructure. We show how Terraform works and the basics of the workflow. Then we talk about new features of Terraform 0.10, with a focus on the split between core and providers, the Kubernetes provider, and workspaces. Terraform 0.9 is reviewed with a focus on destroy provisioners, remote backends, state locking.

Transcript

2. @armon ARMON DADGAR
3. AGENDA HASHICORP USE CASES INTRO TO TERRAFORM NEW FEATURES Terraform 0.10 Terraform 0.9
4. PROVISION, SECURE AND RUN ANY INFRASTRUCTURE Nomad Consul Vault Vagrant Packer Terraform Consul Enterprise Terraform Enterprise Vault Enterprise PRODUCT SUITE OSS TOOL SUITE RUN Applications SECURE Application Infrastructure PROVISION Infrastructure FOR INDIVIDUALS FOR TEAMS Nomad Enterprise
5. SEP 18-20, 2017 | AUSTIN, TEXAS One day of training followed by two days of talks on the future of infrastructure. Discount Code: HUG20
6. 6 HashiCorp confidential do not distribute USE CASES Easily combine and build complex infrastructure Self-Serve Infrastructure Common Workflow for Any Infrastructure Hybrid Cloud Management Automate Infrastructure Provisioning Infrastructure as Code
7. 7 HashiCorp confidential do not distribute COLLABORATE on infrastructure as code using a version control system (VCS) VALIDATE with a common integrated workflow AUTOMATE public cloud, private cloud, and external services THE IDEAL STATE
8. Terraform
9. Terraform Goals • Unified view of infrastructure • Infrastructure as code • Compose multiple tiers (IaaS to PaaS to SaaS) • Safely change/iterate infrastructure over time • One workflow
10. Terraform Features • Open Source • Infrastructure as Code • Resource Providers • Plan and Apply • Collaboration, History [Enterprise]
11. Infrastructure as Code
12. File resource "google_compute_instance" "server" { name = "server" machine_type = "g1-small" zone = "us-central1-a" disk { image = "ubuntu-1404-trusty-v20160114e" } } resource "dnsimple_record" "hello" { domain = "example.com" name = "server" value = "${google_compute_instance.server.network_interface.0.address}" type = "A" }
13. File resource "google_compute_instance" "server" { name = "server" machine_type = "g1-small" zone = "us-central1-a" disk { image = "ubuntu-1404-trusty-v20160114e" } } resource "dnsimple_record" "hello" { domain = "example.com" name = "server" value = "${google_compute_instance.server.network_interface.0.address}" type = "A" }
14. File resource "google_compute_instance" "server" { name = "server" machine_type = "g1-small" zone = "us-central1-a" disk { image = "ubuntu-1404-trusty-v20160114e" } } resource "dnsimple_record" "hello" { domain = "example.com" name = "server" value = "${google_compute_instance.server.network_interface.0.address}" type = "A" }
15. Infrastructure as Code • Human-readable (HCL), Machine-editable (JSON) • Declarative • Text makes it versionable and VCS-friendly • Can be split into multiple files and modules
16. Resource Providers
17. Resource Providers • Integration point • Expose resources, such as "aws_instance", etc. • CRUD API to implement • Core vs. Providers simplifies logic for external contributors
18. Resource Providers Amazon BitBucket CenturyLink Cloud CloudFlare CloudStack Cobbler Consul Datadog DigitalOcean DNSMadeEasy DNSimple Docker Dyn GitHub Fastly Google Heroku Librato MicrosoB Azure MySQL OpenStack Packet PostgreSQL SoBLayer UltraDNS VMware Sphere and more...
19. Resource Providers Amazon BitBucket CenturyLink Cloud CloudFlare CloudStack Cobbler Consul Datadog DigitalOcean DNSMadeEasy DNSimple Docker Dyn GitHub Fastly Google Heroku Librato MicrosoB Azure MySQL OpenStack Packet PostgreSQL SoBLayer UltraDNS VMware Sphere and more...
20. Resource Providers Amazon BitBucket CenturyLink Cloud CloudFlare CloudStack Cobbler Consul Datadog DigitalOcean DNSMadeEasy DNSimple Docker Dyn GitHub Fastly Google Heroku Librato MicrosoB Azure MySQL OpenStack Packet PostgreSQL SoBLayer UltraDNS VMware Sphere and more...
21. Resource Providers Amazon BitBucket CenturyLink Cloud CloudFlare CloudStack Cobbler Consul Datadog DigitalOcean DNSMadeEasy DNSimple Docker Dyn GitHub Fastly Google Heroku Librato MicrosoB Azure MySQL OpenStack Packet PostgreSQL SoBLayer UltraDNS VMware Sphere and more...
22. Resource Providers Amazon BitBucket CenturyLink Cloud CloudFlare CloudStack Cobbler Consul Datadog DigitalOcean DNSMadeEasy DNSimple Docker Dyn GitHub Fastly Google Heroku Librato MicrosoB Azure MySQL OpenStack Packet PostgreSQL SoBLayer UltraDNS VMware Sphere and more...
23. Plan and Apply
24. File resource "google_compute_instance" "server" { name = "server" machine_type = "g1-small" zone = "us-central1-a" disk { image = "ubuntu-1404-trusty-v20160114e" } } resource "dnsimple_record" "hello" { domain = "example.com" name = "server" value = "${google_compute_instance.server.network_interface.0.address}" type = "A" }
25. Plan • Plan shows you what will happen • Plans can be saved to guarantee what will happen • Plans show reasons for certain actions (such as re-create) • Not equivalent to "noop" due to the ability to save a plan
26. Terminal $ terraform plan + google_compute_instance.server can_ip_forward: "false" disk.#: "1" disk.0.auto_delete: "true" disk.0.image: "ubuntu-1404-trusty-..." machine_type: "g1-small" metadata_fingerprint: "<computed>" name: "server" network_interface.#: "1" network_interface.0.address: "<computed>" network_interface.0.name: "<computed>" network_interface.0.network: "default" self_link: "<computed>" tags_fingerprint: "<computed>" zone: "us-central1-a" ... Plan: 3 to add, 0 to change, 0 to destroy.
27. Apply • Executes changes in order based on dependencies • Parallelizes changes when possible • Handles and recovers transient errors
28. Terminal $ terraform apply google_compute_instance.server: Creating... can_ip_forward: "" => "false" disk.#: "" => "1" disk.0.auto_delete: "" => "true" disk.0.image: "" => "ubuntu-1404-trusty..." machine_type: "" => "g1-small" metadata_fingerprint: "" => "<computed>" name: "" => "server" network_interface.#: "" => "1" network_interface.0.address: "" => "<computed>" network_interface.0.name: "" => "<computed>" network_interface.0.network: "" => "default" self_link: "" => "<computed>" tags_fingerprint: "" => "<computed>" zone: "" => "us-central1-a" ... Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
29. Apply for Changes • Not only creation, but changes over time • Plan will show you what will happen • The `-target` flag can be used for fine-grained change
30. NEW FEATURES IN TERRAFORM 0.10 AND 0.9
31. Terraform 0.10 (August 2, 2017) • Split of Terraform Core / Providers • Kubernetes Provider • Workspaces renamed from State Environments
32. Split of Core / Providers
33. Terraform Core vs Provider Core AWS Provider Fastly Provider HCL
 Files API Integration Terraform
34. Terraform Parts • Core Responsible for: • Parsing Configuration • Resource Graph • Interpolation • State Management • Execution • Providers responsible for: • Resource Definition & Validation • API Integration
35. Terraform Repository • Everything lived in a single repository • Released ~2 weeks • Version Sprawl (10+ releases per major version)
36. Mono Repo Issues • Terraform Core moves slowly • Weeks to implement new features • Terraform providers move fast • 70+ Providers today • 1000+ Resources • Hours to implement new resources
37. Mono Repo Issues • Hard to triage issues • 2000+ issues • Core Bugs to Provider Feature Requests • Hard to implement Core features • Hard to scale providers • Impossible to update Provider without updating Core
38. Divide and Conquer • Grand Split proposed months ago • Split Providers into individual repositories • Release Providers on independent cadence • Download Providers as needed • Version constrain Providers • "terraform init”
40. File provider aws { version = "~> v0.1.3" region = "us-west-2" } provider fastly { # Implicit: version = “latest” api_key = “exampleapikey" }
41. Terminal $ terraform init Initializing provider plugins... - Checking for available provider plugins on https:// releases.hashicorp.com... - Downloading plugin for provider "fastly" (0.1.2)... - Downloading plugin for provider "aws" (0.1.4)... The following providers do not have any version constraints in configuration, so the latest version was installed. To prevent automatic upgrades to new major versions that may contain breaking changes, it is recommended to add version = "..." constraints to the corresponding provider blocks in configuration, with the constraint strings suggested below. * provider.fastly: version = "~> 0.1" Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands
42. Terraform Core vs Provider Core 0.10 AWS Provider 0.1.4 Fastly Provider 0.1.2 Plugin Interface
43. End Goal • Independent Versioning and Release for Providers / Core • Live documentation for multiple versions • Make it easier to add and support Providers • Infrastructure as Code for all the things!
44. Kubernetes Provider
45. Kubernetes Provider • Terraform manages IaaS, SaaS, and PaaS • Kubernetes has sub-resources to manage (Pods, Controllers) • Compose other resources (load balancers, DNS, CDN) with K8S • Common workflow for everything
46. Kubernetes Provider • Data Source: kubernetes_service • Data Source: kubernetes_storage_class • Resource: kubernetes_config_map • Resource: kubernetes_horizontal_pod_autoscalar • Resource: kubernetes_limit_range • Resource: kubernetes_namespace • Resource: kubernetes_persistent_volume • Resource: kubernetes_persistent_volume_claim • Resource: kubernetes_pod • Resource: kubernetes_replication_controller • Resource: kubernetes_resource_quota • Resource: kubernetes_secret • Resource: kubernetes_service • Resource: kubernetes_service_account • Resource: kubernetes_storage_class • …
47. Composing Resources Provider Google google_container_cluster Provider Kubernetes kubernetes_namespace kubernetes_pod kubernetes_replication_controller consul_keys Provider Consul
48. Using Schedulers with Terraform • Kubernetes • Nomad • Docker Swarm • AWS ECS • Manage Application Lifecycle • Compose with other resources
49. Workspaces Formerly: State Environments
50. Workspaces • A state namespace • Allows single folder of TF config to manage multiple distinct sets of infrastructure resources
51. Terminal $ terraform workspace list * default $ terraform workspace new mitchellh-test Created and switched to workspace "mitchellh-test"! $ terraform workspace list default * mitchellh-test
52. File resource "aws_instance" "example" { count = “${terraform.workspace == "default" ? 5 : 1}" tags { Name = "web - ${terraform.workspace}" } # ... other fields }
53. Terraform 0.9 (March 15, 2017) • Destroy provisioners • Remote backends • State locking • Workspaces (formerly State Environments)
54. Destroy Provisioners
55. Provisioners (Terraform <= 0.8) • Run arbitrary code locally or remotely on resource creation • If provisioner fails, resource is tainted and scheduled for recreation on the next apply
56. File resource "null_resource" "example" { provisioner "local-exec" { command = "echo foo" } }
57. File resource "null_resource" "example" { provisioner "local-exec" { command = "echo foo" } provisioner "local-exec" { command = "echo destroying" when = "destroy" } }
58. Terminal $ terraform apply null_resource.example: Creating... null_resource.example: Provisioning with 'local-exec'... null_resource.example (local-exec): Executing: /bin/sh -c "echo foo" null_resource.example (local-exec): foo null_resource.example: Creation complete (ID: 1965091882910923448) Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
59. Terminal $ terraform destroy -force null_resource.example: Refreshing state... (ID: 1965091882910923448) null_resource.example: Destroying... (ID: 1965091882910923448) null_resource.example: Provisioning with 'local-exec'... null_resource.example (local-exec): Executing: /bin/sh -c "echo destroying" null_resource.example (local-exec): destroying null_resource.example: Destruction complete
60. Destroy Provisioners • Configured with when = "destroy" • Run on resource destroy (not just "terraform destroy") • Failure cancels physical resource destruction by default • Can allow failure with on_failure = "continue"
61. File resource "null_resource" "example" { provisioner "local-exec" { command = "exit 1" when = "destroy" } }
62. Terminal $ terraform destroy -force null_resource.example: Refreshing state... (ID: 8665586891184105369) null_resource.example: Destroying... (ID: 8665586891184105369) null_resource.example: Provisioning with 'local-exec'... null_resource.example (local-exec): Executing: /bin/sh -c "exit 1" Error applying plan: 1 error(s) occurred: * null_resource.example (destroy): 1 error(s) occurred: * Error running command 'exit 1': exit status 1.
63. File resource "null_resource" "example" { provisioner "local-exec" { command = "exit 1" when = "destroy" on_failure = "continue" } }
64. Terminal $ terraform destroy -force null_resource.example: Refreshing state... (ID: 8665586891184105369) null_resource.example: Destroying... (ID: 8665586891184105369) null_resource.example: Provisioning with 'local-exec'... null_resource.example (local-exec): Executing: /bin/sh -c "exit 1" null_resource.example: Destruction complete Destroy complete! Resources: 1 destroyed.
65. Destroy Provisioners • Useful for resource cleanup • Can SSH into machine (any machine!) prior to destruction • Recommend resource cleanup live as part of the resource itself, but destroy provisioners give you another option
66. Remote Backends
67. Before Remote Backends (TF <= 0.8) • Awkward "remote config" command • Users could accidentally run Terraform without remote init • Configuration only via CLI • Local cache of state stored in .terraform/terraform.tfstate • Changed remote configuration was manual
68. Terminal $ # TERRAFORM <= 0.8, BEFORE REMOTE BACKENDS $ terraform remote config \ -backend=S3 \ -backend-config="bucket=<bucket>" \ -backend-config="key=<path to file>" ...
69. Remote Backends • Subsumes "remote state", enables locking, workspaces, more • Configure from tf files, external configuration, or CLI • Detects configuration change • Forces new users of a TF configuration to initialize • One command to init them all: `terraform init`
70. File terraform { backend "s3" { bucket = "<bucket>" key = "<path to file>" } }
71. Terminal $ terraform init Initializing the backend... Successfully configured the backend "s3"! Terraform will automatically use this backend unless the backend configuration changes. Terraform has been successfully initialized!
72. Terminal $ # New user, didn't run init $ terraform console Backend reinitialization required. Please run "terraform init". Reason: Initial configuration of the requested backend "s3" ...
73. File terraform { backend "s3" { bucket = "CHANGED-THIS-SETTING" key = "<path to file>" } }
74. Terminal $ terraform console Backend reinitialization required. Please run "terraform init". Reason: Backend configuration changed for "s3" ...
75. Terminal $ terraform console Backend reinitialization required. Please run "terraform init". Reason: Unsetting the previously set backend "s3" ...
76. Remote Backends • One command to init: `terraform init` • Automatic detection of backend change (set, change, unset) • No state stored locally at all • Always gitignore ".terraform" folder
77. A Focus on Safety • Common complaint: easy to corrupt remote state • Remote backends add new layer of safety: detecting changes, checking "lineage", disallowing writing unsafe state, more.
78. A New "Init" • Init has existed since Terraform 0.1 • Used to just setup folder structure for new projects • Now the single source of init, safe to run multiple times • Initializes backend, downloads modules, creates folders • One day: downloads providers, verifies versions, more...
79. State Locking
80. State Locking • For supported backends, Terraform automatically locks state on write operations • If unlock fails, error is shown with lock ID to allow a force unlock • Doesn't lock against concurrent reads
81. File terraform { backend "consul" {} } resource "null_resource" "example" { provisioner "local-exec" { command = "sleep 10" } }
82. Terminal $ terraform apply null_resource.example: Creating... null_resource.example: Provisioning with 'local-exec'... null_resource.example (local-exec): Executing: /bin/sh -c "sleep 10"
83. Terminal $ terraform apply Error loading state: failed to lock state in Consul: Lock Info: ID: 5c0b66d6-018f-59b4-5536-499fec947fb2 Path: foo Operation: OperationTypeApply Who: mitchellh@Mitchells-iMac.lan Version: 0.9.1 Created: 2017-04-04 16:16:59.733058195 +0000 UTC Info: $ terraform console >
84. Terraform
85. SEP 18-20, 2017 | AUSTIN, TEXAS One day of training followed by two days of talks on the future of infrastructure. Discount Code: HUG20
86. THANKS! Q/A