What's new in Terraform 0.10 - HUG

In this talk we cover an introduction to Terraform, including it's primary use cases of Infrastructure as Code, composing multiple providers (such as IaaS, SaaS, and PaaS), and enabling self service of infrastructure. We show how Terraform works and the basics of the workflow. Then we talk about new features of Terraform 0.10, with a focus on the split between core and providers, the Kubernetes provider, and workspaces. Terraform 0.9 is reviewed with a focus on destroy provisioners, remote backends, state locking.

Transcript

1. View Slide

2. @armon
   ARMON DADGAR
   

   View Slide

3. AGENDA
   HASHICORP
   USE CASES
   INTRO TO TERRAFORM
   NEW FEATURES
   Terraform 0.10
   Terraform 0.9
   

   View Slide

4. PROVISION, SECURE AND RUN ANY INFRASTRUCTURE
   Nomad Consul
   Vault
   Vagrant Packer Terraform
   Consul Enterprise
   Terraform Enterprise
   Vault Enterprise
   PRODUCT SUITE
   OSS TOOL SUITE
   RUN
   Applications
   SECURE
   Application Infrastructure
   PROVISION
   Infrastructure
   FOR INDIVIDUALS FOR TEAMS
   Nomad Enterprise
   

   View Slide

5. SEP 18-20, 2017 | AUSTIN, TEXAS
   One day of training followed by two days
   of talks on the future of infrastructure.
   Discount Code: HUG20
   

   View Slide

6. 6
   HashiCorp confidential do not distribute
   USE CASES
   Easily combine and build
   complex infrastructure
   Self-Serve Infrastructure
   Common Workflow for Any
   Infrastructure
   Hybrid Cloud Management
   Automate Infrastructure
   Provisioning
   Infrastructure as Code
   

   View Slide

7. 7
   HashiCorp confidential do not distribute
   COLLABORATE
   on infrastructure as code
   using a version control system (VCS)
   VALIDATE
   with a common integrated
   workflow
   AUTOMATE
   public cloud, private cloud,
   and external services
   THE IDEAL STATE
   

   View Slide

8. Terraform
   

   View Slide

9. Terraform Goals
   • Unified view of infrastructure
   • Infrastructure as code
   • Compose multiple tiers (IaaS to PaaS to SaaS)
   • Safely change/iterate infrastructure over time
   • One workflow
   

   View Slide

10. Terraform Features
    • Open Source
    • Infrastructure as Code
    • Resource Providers
    • Plan and Apply
    • Collaboration, History [Enterprise]
    

    View Slide

11. Infrastructure as Code
    

    View Slide

12. File
    resource "google_compute_instance" "server" {
    name = "server"
    machine_type = "g1-small"
    zone = "us-central1-a"
    disk {
    image = "ubuntu-1404-trusty-v20160114e"
    }
    }
    resource "dnsimple_record" "hello" {
    domain = "example.com"
    name = "server"
    value = "${google_compute_instance.server.network_interface.0.address}"
    type = "A"
    }
    

    View Slide

13. File
    resource "google_compute_instance" "server" {
    name = "server"
    machine_type = "g1-small"
    zone = "us-central1-a"
    disk {
    image = "ubuntu-1404-trusty-v20160114e"
    }
    }
    resource "dnsimple_record" "hello" {
    domain = "example.com"
    name = "server"
    value = "${google_compute_instance.server.network_interface.0.address}"
    type = "A"
    }
    

    View Slide

14. File
    resource "google_compute_instance" "server" {
    name = "server"
    machine_type = "g1-small"
    zone = "us-central1-a"
    disk {
    image = "ubuntu-1404-trusty-v20160114e"
    }
    }
    resource "dnsimple_record" "hello" {
    domain = "example.com"
    name = "server"
    value = "${google_compute_instance.server.network_interface.0.address}"
    type = "A"
    }
    

    View Slide

15. Infrastructure as Code
    • Human-readable (HCL), Machine-editable (JSON)
    • Declarative
    • Text makes it versionable and VCS-friendly
    • Can be split into multiple files and modules
    

    View Slide

16. Resource Providers
    

    View Slide

17. Resource Providers
    • Integration point
    • Expose resources, such as "aws_instance", etc.
    • CRUD API to implement
    • Core vs. Providers simplifies logic for external contributors
    

    View Slide

18. Resource Providers
    Amazon BitBucket CenturyLink Cloud
    CloudFlare CloudStack Cobbler
    Consul Datadog DigitalOcean
    DNSMadeEasy DNSimple Docker
    Dyn GitHub Fastly
    Google Heroku Librato
    MicrosoB Azure MySQL OpenStack
    Packet PostgreSQL SoBLayer
    UltraDNS VMware Sphere and more...
    

    View Slide

19. Resource Providers
    Amazon BitBucket CenturyLink Cloud
    CloudFlare CloudStack Cobbler
    Consul Datadog DigitalOcean
    DNSMadeEasy DNSimple Docker
    Dyn GitHub Fastly
    Google Heroku Librato
    MicrosoB Azure MySQL OpenStack
    Packet PostgreSQL SoBLayer
    UltraDNS VMware Sphere and more...
    

    View Slide

20. Resource Providers
    Amazon BitBucket CenturyLink Cloud
    CloudFlare CloudStack Cobbler
    Consul Datadog DigitalOcean
    DNSMadeEasy DNSimple Docker
    Dyn GitHub Fastly
    Google Heroku Librato
    MicrosoB Azure MySQL OpenStack
    Packet PostgreSQL SoBLayer
    UltraDNS VMware Sphere and more...
    

    View Slide

21. Resource Providers
    Amazon BitBucket CenturyLink Cloud
    CloudFlare CloudStack Cobbler
    Consul Datadog DigitalOcean
    DNSMadeEasy DNSimple Docker
    Dyn GitHub Fastly
    Google Heroku Librato
    MicrosoB Azure MySQL OpenStack
    Packet PostgreSQL SoBLayer
    UltraDNS VMware Sphere and more...
    

    View Slide

22. Resource Providers
    Amazon BitBucket CenturyLink Cloud
    CloudFlare CloudStack Cobbler
    Consul Datadog DigitalOcean
    DNSMadeEasy DNSimple Docker
    Dyn GitHub Fastly
    Google Heroku Librato
    MicrosoB Azure MySQL OpenStack
    Packet PostgreSQL SoBLayer
    UltraDNS VMware Sphere and more...
    

    View Slide

23. Plan and Apply
    

    View Slide

24. File
    resource "google_compute_instance" "server" {
    name = "server"
    machine_type = "g1-small"
    zone = "us-central1-a"
    disk {
    image = "ubuntu-1404-trusty-v20160114e"
    }
    }
    resource "dnsimple_record" "hello" {
    domain = "example.com"
    name = "server"
    value = "${google_compute_instance.server.network_interface.0.address}"
    type = "A"
    }
    

    View Slide

25. Plan
    • Plan shows you what will happen
    • Plans can be saved to guarantee what will happen
    • Plans show reasons for certain actions (such as re-create)
    • Not equivalent to "noop" due to the ability to save a plan
    

    View Slide

26. Terminal
    $ terraform plan
    + google_compute_instance.server
    can_ip_forward: "false"
    disk.#: "1"
    disk.0.auto_delete: "true"
    disk.0.image: "ubuntu-1404-trusty-..."
    machine_type: "g1-small"
    metadata_fingerprint: ""
    name: "server"
    network_interface.#: "1"
    network_interface.0.address: ""
    network_interface.0.name: ""
    network_interface.0.network: "default"
    self_link: ""
    tags_fingerprint: ""
    zone: "us-central1-a"
    ...
    Plan: 3 to add, 0 to change, 0 to destroy.
    

    View Slide

27. Apply
    • Executes changes in order based on dependencies
    • Parallelizes changes when possible
    • Handles and recovers transient errors
    

    View Slide

28. Terminal
    $ terraform apply
    google_compute_instance.server: Creating...
    can_ip_forward: "" => "false"
    disk.#: "" => "1"
    disk.0.auto_delete: "" => "true"
    disk.0.image: "" => "ubuntu-1404-trusty..."
    machine_type: "" => "g1-small"
    metadata_fingerprint: "" => ""
    name: "" => "server"
    network_interface.#: "" => "1"
    network_interface.0.address: "" => ""
    network_interface.0.name: "" => ""
    network_interface.0.network: "" => "default"
    self_link: "" => ""
    tags_fingerprint: "" => ""
    zone: "" => "us-central1-a"
    ...
    Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
    

    View Slide

29. Apply for Changes
    • Not only creation, but changes over time
    • Plan will show you what will happen
    • The `-target` flag can be used for fine-grained change
    

    View Slide

30. NEW FEATURES IN
    TERRAFORM 0.10 AND 0.9
    

    View Slide

31. Terraform 0.10 (August 2, 2017)
    • Split of Terraform Core / Providers
    • Kubernetes Provider
    • Workspaces renamed from State Environments
    

    View Slide

32. Split of Core / Providers
    

    View Slide

33. Terraform Core vs Provider
    Core
    AWS
    Provider
    Fastly
    Provider
    HCL

    Files
    API
    Integration
    Terraform
    

    View Slide

34. Terraform Parts
    • Core Responsible for:
    • Parsing Configuration
    • Resource Graph
    • Interpolation
    • State Management
    • Execution
    • Providers responsible for:
    • Resource Definition &
    Validation
    • API Integration
    

    View Slide

35. Terraform Repository
    • Everything lived in a single repository
    • Released ~2 weeks
    • Version Sprawl (10+ releases per major version)
    

    View Slide

36. Mono Repo Issues
    • Terraform Core moves slowly
    • Weeks to implement new features
    • Terraform providers move fast
    • 70+ Providers today
    • 1000+ Resources
    • Hours to implement new resources
    

    View Slide

37. Mono Repo Issues
    • Hard to triage issues
    • 2000+ issues
    • Core Bugs to Provider Feature Requests
    • Hard to implement Core features
    • Hard to scale providers
    • Impossible to update Provider without updating Core
    

    View Slide

38. Divide and Conquer
    • Grand Split proposed months ago
    • Split Providers into individual repositories
    • Release Providers on independent cadence
    • Download Providers as needed
    • Version constrain Providers
    • "terraform init”
    

    View Slide

39. View Slide

40. File
    provider aws {
    version = "~> v0.1.3"
    region = "us-west-2"
    }
    provider fastly {
    # Implicit: version = “latest”
    api_key = “exampleapikey"
    }
    

    View Slide

41. Terminal
    $ terraform init
    Initializing provider plugins...
    - Checking for available provider plugins on https://
    releases.hashicorp.com...
    - Downloading plugin for provider "fastly" (0.1.2)...
    - Downloading plugin for provider "aws" (0.1.4)...
    The following providers do not have any version constraints in configuration,
    so the latest version was installed.
    To prevent automatic upgrades to new major versions that may contain breaking
    changes, it is recommended to add version = "..." constraints to the
    corresponding provider blocks in configuration, with the constraint strings
    suggested below.
    * provider.fastly: version = "~> 0.1"
    Terraform has been successfully initialized!
    You may now begin working with Terraform. Try running "terraform plan" to see
    any changes that are required for your infrastructure. All Terraform commands
    

    View Slide

42. Terraform Core vs Provider
    Core
    0.10
    AWS
    Provider
    0.1.4
    Fastly
    Provider
    0.1.2
    Plugin
    Interface
    

    View Slide

43. End Goal
    • Independent Versioning and Release for Providers / Core
    • Live documentation for multiple versions
    • Make it easier to add and support Providers
    • Infrastructure as Code for all the things!
    

    View Slide

44. Kubernetes
    Provider
    

    View Slide

45. Kubernetes Provider
    • Terraform manages IaaS, SaaS, and PaaS
    • Kubernetes has sub-resources to manage (Pods, Controllers)
    • Compose other resources (load balancers, DNS, CDN) with K8S
    • Common workflow for everything
    

    View Slide

46. Kubernetes Provider
    • Data Source: kubernetes_service
    • Data Source: kubernetes_storage_class
    • Resource: kubernetes_config_map
    • Resource: kubernetes_horizontal_pod_autoscalar
    • Resource: kubernetes_limit_range
    • Resource: kubernetes_namespace
    • Resource: kubernetes_persistent_volume
    • Resource: kubernetes_persistent_volume_claim
    • Resource: kubernetes_pod
    • Resource: kubernetes_replication_controller
    • Resource: kubernetes_resource_quota
    • Resource: kubernetes_secret
    • Resource: kubernetes_service
    • Resource: kubernetes_service_account
    • Resource: kubernetes_storage_class
    • …
    

    View Slide

47. Composing Resources
    Provider Google
    google_container_cluster
    Provider
    Kubernetes
    kubernetes_namespace
    kubernetes_pod
    kubernetes_replication_controller
    consul_keys
    Provider Consul
    

    View Slide

48. Using Schedulers with Terraform
    • Kubernetes
    • Nomad
    • Docker Swarm
    • AWS ECS
    • Manage Application Lifecycle
    • Compose with other resources
    

    View Slide

49. Workspaces
    Formerly: State Environments
    

    View Slide

50. Workspaces
    • A state namespace
    • Allows single folder of TF config to manage multiple distinct
    sets of infrastructure resources
    

    View Slide

51. Terminal
    $ terraform workspace list
    * default
    $ terraform workspace new mitchellh-test
    Created and switched to workspace "mitchellh-test"!
    $ terraform workspace list
    default
    * mitchellh-test
    

    View Slide

52. File
    resource "aws_instance" "example" {
    count = “${terraform.workspace == "default" ? 5 : 1}"
    tags { Name = "web - ${terraform.workspace}" }
    # ... other fields
    }
    

    View Slide

53. Terraform 0.9 (March 15, 2017)
    • Destroy provisioners
    • Remote backends
    • State locking
    • Workspaces (formerly State Environments)
    

    View Slide

54. Destroy Provisioners
    

    View Slide

55. Provisioners (Terraform <= 0.8)
    • Run arbitrary code locally or remotely on resource creation
    • If provisioner fails, resource is tainted and scheduled for
    recreation on the next apply
    

    View Slide

56. File
    resource "null_resource" "example" {
    provisioner "local-exec" {
    command = "echo foo"
    }
    }
    

    View Slide

57. File
    resource "null_resource" "example" {
    provisioner "local-exec" {
    command = "echo foo"
    }
    provisioner "local-exec" {
    command = "echo destroying"
    when = "destroy"
    }
    }
    

    View Slide

58. Terminal
    $ terraform apply
    null_resource.example: Creating...
    null_resource.example: Provisioning with 'local-exec'...
    null_resource.example (local-exec): Executing: /bin/sh -c "echo foo"
    null_resource.example (local-exec): foo
    null_resource.example: Creation complete (ID: 1965091882910923448)
    Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
    

    View Slide

59. Terminal
    $ terraform destroy -force
    null_resource.example: Refreshing state... (ID: 1965091882910923448)
    null_resource.example: Destroying... (ID: 1965091882910923448)
    null_resource.example: Provisioning with 'local-exec'...
    null_resource.example (local-exec): Executing: /bin/sh -c "echo destroying"
    null_resource.example (local-exec): destroying
    null_resource.example: Destruction complete
    

    View Slide

60. Destroy Provisioners
    • Configured with when = "destroy"
    • Run on resource destroy (not just "terraform destroy")
    • Failure cancels physical resource destruction by default
    • Can allow failure with on_failure = "continue"
    

    View Slide

61. File
    resource "null_resource" "example" {
    provisioner "local-exec" {
    command = "exit 1"
    when = "destroy"
    }
    }
    

    View Slide

62. Terminal
    $ terraform destroy -force
    null_resource.example: Refreshing state... (ID: 8665586891184105369)
    null_resource.example: Destroying... (ID: 8665586891184105369)
    null_resource.example: Provisioning with 'local-exec'...
    null_resource.example (local-exec): Executing: /bin/sh -c "exit 1"
    Error applying plan:
    1 error(s) occurred:
    * null_resource.example (destroy): 1 error(s) occurred:
    * Error running command 'exit 1': exit status 1.
    

    View Slide

63. File
    resource "null_resource" "example" {
    provisioner "local-exec" {
    command = "exit 1"
    when = "destroy"
    on_failure = "continue"
    }
    }
    

    View Slide

64. Terminal
    $ terraform destroy -force
    null_resource.example: Refreshing state... (ID: 8665586891184105369)
    null_resource.example: Destroying... (ID: 8665586891184105369)
    null_resource.example: Provisioning with 'local-exec'...
    null_resource.example (local-exec): Executing: /bin/sh -c "exit 1"
    null_resource.example: Destruction complete
    Destroy complete! Resources: 1 destroyed.
    

    View Slide

65. Destroy Provisioners
    • Useful for resource cleanup
    • Can SSH into machine (any machine!) prior to destruction
    • Recommend resource cleanup live as part of the resource itself,
    but destroy provisioners give you another option
    

    View Slide

66. Remote Backends
    

    View Slide

67. Before Remote Backends (TF <= 0.8)
    • Awkward "remote config" command
    • Users could accidentally run Terraform without remote init
    • Configuration only via CLI
    • Local cache of state stored in .terraform/terraform.tfstate
    • Changed remote configuration was manual
    

    View Slide

68. Terminal
    $ # TERRAFORM <= 0.8, BEFORE REMOTE BACKENDS
    $ terraform remote config \
    -backend=S3 \
    -backend-config="bucket=" \
    -backend-config="key="
    ...
    

    View Slide

69. Remote Backends
    • Subsumes "remote state", enables locking, workspaces, more
    • Configure from tf files, external configuration, or CLI
    • Detects configuration change
    • Forces new users of a TF configuration to initialize
    • One command to init them all: `terraform init`
    

    View Slide

70. File
    terraform {
    backend "s3" {
    bucket = ""
    key = ""
    }
    }
    

    View Slide

71. Terminal
    $ terraform init
    Initializing the backend...
    Successfully configured the backend "s3"! Terraform will automatically
    use this backend unless the backend configuration changes.
    Terraform has been successfully initialized!
    

    View Slide

72. Terminal
    $ # New user, didn't run init
    $ terraform console
    Backend reinitialization required. Please run "terraform init".
    Reason: Initial configuration of the requested backend "s3"
    ...
    

    View Slide

73. File
    terraform {
    backend "s3" {
    bucket = "CHANGED-THIS-SETTING"
    key = ""
    }
    }
    

    View Slide

74. Terminal
    $ terraform console
    Backend reinitialization required. Please run "terraform init".
    Reason: Backend configuration changed for "s3"
    ...
    

    View Slide

75. Terminal
    $ terraform console
    Backend reinitialization required. Please run "terraform init".
    Reason: Unsetting the previously set backend "s3"
    ...
    

    View Slide

76. Remote Backends
    • One command to init: `terraform init`
    • Automatic detection of backend change (set, change, unset)
    • No state stored locally at all
    • Always gitignore ".terraform" folder
    

    View Slide

77. A Focus on Safety
    • Common complaint: easy to corrupt remote state
    • Remote backends add new layer of safety: detecting changes,
    checking "lineage", disallowing writing unsafe state, more.
    

    View Slide

78. A New "Init"
    • Init has existed since Terraform 0.1
    • Used to just setup folder structure for new projects
    • Now the single source of init, safe to run multiple times
    • Initializes backend, downloads modules, creates folders
    • One day: downloads providers, verifies versions, more...
    

    View Slide

79. State Locking
    

    View Slide

80. State Locking
    • For supported backends, Terraform automatically locks state on
    write operations
    • If unlock fails, error is shown with lock ID to allow a force unlock
    • Doesn't lock against concurrent reads
    

    View Slide

81. File
    terraform { backend "consul" {} }
    resource "null_resource" "example" {
    provisioner "local-exec" {
    command = "sleep 10"
    }
    }
    

    View Slide

82. Terminal
    $ terraform apply
    null_resource.example: Creating...
    null_resource.example: Provisioning with 'local-exec'...
    null_resource.example (local-exec): Executing: /bin/sh -c "sleep 10"
    

    View Slide

83. Terminal
    $ terraform apply
    Error loading state: failed to lock state in Consul: Lock Info:
    ID: 5c0b66d6-018f-59b4-5536-499fec947fb2
    Path: foo
    Operation: OperationTypeApply
    Who: mitchellh@Mitchells-iMac.lan
    Version: 0.9.1
    Created: 2017-04-04 16:16:59.733058195 +0000 UTC
    Info:
    $ terraform console
    >
    

    View Slide

84. Terraform
    

    View Slide

85. SEP 18-20, 2017 | AUSTIN, TEXAS
    One day of training followed by two days
    of talks on the future of infrastructure.
    Discount Code: HUG20
    

    View Slide

86. THANKS!
    Q/A
    

    View Slide