$30 off During Our Annual Pro Sale. View Details »

What's new in Terraform 0.10 - HUG

What's new in Terraform 0.10 - HUG

In this talk we cover an introduction to Terraform, including it's primary use cases of Infrastructure as Code, composing multiple providers (such as IaaS, SaaS, and PaaS), and enabling self service of infrastructure. We show how Terraform works and the basics of the workflow. Then we talk about new features of Terraform 0.10, with a focus on the split between core and providers, the Kubernetes provider, and workspaces. Terraform 0.9 is reviewed with a focus on destroy provisioners, remote backends, state locking.

Armon Dadgar

August 16, 2017
Tweet

More Decks by Armon Dadgar

Other Decks in Technology

Transcript

  1. View Slide

  2. @armon
    ARMON DADGAR

    View Slide

  3. AGENDA
    HASHICORP
    USE CASES
    INTRO TO TERRAFORM
    NEW FEATURES
    Terraform 0.10
    Terraform 0.9

    View Slide

  4. PROVISION, SECURE AND RUN ANY INFRASTRUCTURE
    Nomad Consul
    Vault
    Vagrant Packer Terraform
    Consul Enterprise
    Terraform Enterprise
    Vault Enterprise
    PRODUCT SUITE
    OSS TOOL SUITE
    RUN
    Applications
    SECURE
    Application Infrastructure
    PROVISION
    Infrastructure
    FOR INDIVIDUALS FOR TEAMS
    Nomad Enterprise

    View Slide

  5. SEP 18-20, 2017 | AUSTIN, TEXAS
    One day of training followed by two days
    of talks on the future of infrastructure.
    Discount Code: HUG20

    View Slide

  6. 6
    HashiCorp confidential do not distribute
    USE CASES
    Easily combine and build
    complex infrastructure
    Self-Serve Infrastructure
    Common Workflow for Any
    Infrastructure
    Hybrid Cloud Management
    Automate Infrastructure
    Provisioning
    Infrastructure as Code

    View Slide

  7. 7
    HashiCorp confidential do not distribute
    COLLABORATE
    on infrastructure as code
    using a version control system (VCS)
    VALIDATE
    with a common integrated
    workflow
    AUTOMATE
    public cloud, private cloud,
    and external services
    THE IDEAL STATE

    View Slide

  8. Terraform

    View Slide

  9. Terraform Goals
    • Unified view of infrastructure
    • Infrastructure as code
    • Compose multiple tiers (IaaS to PaaS to SaaS)
    • Safely change/iterate infrastructure over time
    • One workflow

    View Slide

  10. Terraform Features
    • Open Source
    • Infrastructure as Code
    • Resource Providers
    • Plan and Apply
    • Collaboration, History [Enterprise]

    View Slide

  11. Infrastructure as Code

    View Slide

  12. File
    resource "google_compute_instance" "server" {
    name = "server"
    machine_type = "g1-small"
    zone = "us-central1-a"
    disk {
    image = "ubuntu-1404-trusty-v20160114e"
    }
    }
    resource "dnsimple_record" "hello" {
    domain = "example.com"
    name = "server"
    value = "${google_compute_instance.server.network_interface.0.address}"
    type = "A"
    }

    View Slide

  13. File
    resource "google_compute_instance" "server" {
    name = "server"
    machine_type = "g1-small"
    zone = "us-central1-a"
    disk {
    image = "ubuntu-1404-trusty-v20160114e"
    }
    }
    resource "dnsimple_record" "hello" {
    domain = "example.com"
    name = "server"
    value = "${google_compute_instance.server.network_interface.0.address}"
    type = "A"
    }

    View Slide

  14. File
    resource "google_compute_instance" "server" {
    name = "server"
    machine_type = "g1-small"
    zone = "us-central1-a"
    disk {
    image = "ubuntu-1404-trusty-v20160114e"
    }
    }
    resource "dnsimple_record" "hello" {
    domain = "example.com"
    name = "server"
    value = "${google_compute_instance.server.network_interface.0.address}"
    type = "A"
    }

    View Slide

  15. Infrastructure as Code
    • Human-readable (HCL), Machine-editable (JSON)
    • Declarative
    • Text makes it versionable and VCS-friendly
    • Can be split into multiple files and modules

    View Slide

  16. Resource Providers

    View Slide

  17. Resource Providers
    • Integration point
    • Expose resources, such as "aws_instance", etc.
    • CRUD API to implement
    • Core vs. Providers simplifies logic for external contributors

    View Slide

  18. Resource Providers
    Amazon BitBucket CenturyLink Cloud
    CloudFlare CloudStack Cobbler
    Consul Datadog DigitalOcean
    DNSMadeEasy DNSimple Docker
    Dyn GitHub Fastly
    Google Heroku Librato
    MicrosoB Azure MySQL OpenStack
    Packet PostgreSQL SoBLayer
    UltraDNS VMware Sphere and more...

    View Slide

  19. Resource Providers
    Amazon BitBucket CenturyLink Cloud
    CloudFlare CloudStack Cobbler
    Consul Datadog DigitalOcean
    DNSMadeEasy DNSimple Docker
    Dyn GitHub Fastly
    Google Heroku Librato
    MicrosoB Azure MySQL OpenStack
    Packet PostgreSQL SoBLayer
    UltraDNS VMware Sphere and more...

    View Slide

  20. Resource Providers
    Amazon BitBucket CenturyLink Cloud
    CloudFlare CloudStack Cobbler
    Consul Datadog DigitalOcean
    DNSMadeEasy DNSimple Docker
    Dyn GitHub Fastly
    Google Heroku Librato
    MicrosoB Azure MySQL OpenStack
    Packet PostgreSQL SoBLayer
    UltraDNS VMware Sphere and more...

    View Slide

  21. Resource Providers
    Amazon BitBucket CenturyLink Cloud
    CloudFlare CloudStack Cobbler
    Consul Datadog DigitalOcean
    DNSMadeEasy DNSimple Docker
    Dyn GitHub Fastly
    Google Heroku Librato
    MicrosoB Azure MySQL OpenStack
    Packet PostgreSQL SoBLayer
    UltraDNS VMware Sphere and more...

    View Slide

  22. Resource Providers
    Amazon BitBucket CenturyLink Cloud
    CloudFlare CloudStack Cobbler
    Consul Datadog DigitalOcean
    DNSMadeEasy DNSimple Docker
    Dyn GitHub Fastly
    Google Heroku Librato
    MicrosoB Azure MySQL OpenStack
    Packet PostgreSQL SoBLayer
    UltraDNS VMware Sphere and more...

    View Slide

  23. Plan and Apply

    View Slide

  24. File
    resource "google_compute_instance" "server" {
    name = "server"
    machine_type = "g1-small"
    zone = "us-central1-a"
    disk {
    image = "ubuntu-1404-trusty-v20160114e"
    }
    }
    resource "dnsimple_record" "hello" {
    domain = "example.com"
    name = "server"
    value = "${google_compute_instance.server.network_interface.0.address}"
    type = "A"
    }

    View Slide

  25. Plan
    • Plan shows you what will happen
    • Plans can be saved to guarantee what will happen
    • Plans show reasons for certain actions (such as re-create)
    • Not equivalent to "noop" due to the ability to save a plan

    View Slide

  26. Terminal
    $ terraform plan
    + google_compute_instance.server
    can_ip_forward: "false"
    disk.#: "1"
    disk.0.auto_delete: "true"
    disk.0.image: "ubuntu-1404-trusty-..."
    machine_type: "g1-small"
    metadata_fingerprint: ""
    name: "server"
    network_interface.#: "1"
    network_interface.0.address: ""
    network_interface.0.name: ""
    network_interface.0.network: "default"
    self_link: ""
    tags_fingerprint: ""
    zone: "us-central1-a"
    ...
    Plan: 3 to add, 0 to change, 0 to destroy.

    View Slide

  27. Apply
    • Executes changes in order based on dependencies
    • Parallelizes changes when possible
    • Handles and recovers transient errors

    View Slide

  28. Terminal
    $ terraform apply
    google_compute_instance.server: Creating...
    can_ip_forward: "" => "false"
    disk.#: "" => "1"
    disk.0.auto_delete: "" => "true"
    disk.0.image: "" => "ubuntu-1404-trusty..."
    machine_type: "" => "g1-small"
    metadata_fingerprint: "" => ""
    name: "" => "server"
    network_interface.#: "" => "1"
    network_interface.0.address: "" => ""
    network_interface.0.name: "" => ""
    network_interface.0.network: "" => "default"
    self_link: "" => ""
    tags_fingerprint: "" => ""
    zone: "" => "us-central1-a"
    ...
    Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

    View Slide

  29. Apply for Changes
    • Not only creation, but changes over time
    • Plan will show you what will happen
    • The `-target` flag can be used for fine-grained change

    View Slide

  30. NEW FEATURES IN
    TERRAFORM 0.10 AND 0.9

    View Slide

  31. Terraform 0.10 (August 2, 2017)
    • Split of Terraform Core / Providers
    • Kubernetes Provider
    • Workspaces renamed from State Environments

    View Slide

  32. Split of Core / Providers

    View Slide

  33. Terraform Core vs Provider
    Core
    AWS
    Provider
    Fastly
    Provider
    HCL

    Files
    API
    Integration
    Terraform

    View Slide

  34. Terraform Parts
    • Core Responsible for:
    • Parsing Configuration
    • Resource Graph
    • Interpolation
    • State Management
    • Execution
    • Providers responsible for:
    • Resource Definition &
    Validation
    • API Integration

    View Slide

  35. Terraform Repository
    • Everything lived in a single repository
    • Released ~2 weeks
    • Version Sprawl (10+ releases per major version)

    View Slide

  36. Mono Repo Issues
    • Terraform Core moves slowly
    • Weeks to implement new features
    • Terraform providers move fast
    • 70+ Providers today
    • 1000+ Resources
    • Hours to implement new resources

    View Slide

  37. Mono Repo Issues
    • Hard to triage issues
    • 2000+ issues
    • Core Bugs to Provider Feature Requests
    • Hard to implement Core features
    • Hard to scale providers
    • Impossible to update Provider without updating Core

    View Slide

  38. Divide and Conquer
    • Grand Split proposed months ago
    • Split Providers into individual repositories
    • Release Providers on independent cadence
    • Download Providers as needed
    • Version constrain Providers
    • "terraform init”

    View Slide

  39. View Slide

  40. File
    provider aws {
    version = "~> v0.1.3"
    region = "us-west-2"
    }
    provider fastly {
    # Implicit: version = “latest”
    api_key = “exampleapikey"
    }

    View Slide

  41. Terminal
    $ terraform init
    Initializing provider plugins...
    - Checking for available provider plugins on https://
    releases.hashicorp.com...
    - Downloading plugin for provider "fastly" (0.1.2)...
    - Downloading plugin for provider "aws" (0.1.4)...
    The following providers do not have any version constraints in configuration,
    so the latest version was installed.
    To prevent automatic upgrades to new major versions that may contain breaking
    changes, it is recommended to add version = "..." constraints to the
    corresponding provider blocks in configuration, with the constraint strings
    suggested below.
    * provider.fastly: version = "~> 0.1"
    Terraform has been successfully initialized!
    You may now begin working with Terraform. Try running "terraform plan" to see
    any changes that are required for your infrastructure. All Terraform commands

    View Slide

  42. Terraform Core vs Provider
    Core
    0.10
    AWS
    Provider
    0.1.4
    Fastly
    Provider
    0.1.2
    Plugin
    Interface

    View Slide

  43. End Goal
    • Independent Versioning and Release for Providers / Core
    • Live documentation for multiple versions
    • Make it easier to add and support Providers
    • Infrastructure as Code for all the things!

    View Slide

  44. Kubernetes
    Provider

    View Slide

  45. Kubernetes Provider
    • Terraform manages IaaS, SaaS, and PaaS
    • Kubernetes has sub-resources to manage (Pods, Controllers)
    • Compose other resources (load balancers, DNS, CDN) with K8S
    • Common workflow for everything

    View Slide

  46. Kubernetes Provider
    • Data Source: kubernetes_service
    • Data Source: kubernetes_storage_class
    • Resource: kubernetes_config_map
    • Resource: kubernetes_horizontal_pod_autoscalar
    • Resource: kubernetes_limit_range
    • Resource: kubernetes_namespace
    • Resource: kubernetes_persistent_volume
    • Resource: kubernetes_persistent_volume_claim
    • Resource: kubernetes_pod
    • Resource: kubernetes_replication_controller
    • Resource: kubernetes_resource_quota
    • Resource: kubernetes_secret
    • Resource: kubernetes_service
    • Resource: kubernetes_service_account
    • Resource: kubernetes_storage_class
    • …

    View Slide

  47. Composing Resources
    Provider Google
    google_container_cluster
    Provider
    Kubernetes
    kubernetes_namespace
    kubernetes_pod
    kubernetes_replication_controller
    consul_keys
    Provider Consul

    View Slide

  48. Using Schedulers with Terraform
    • Kubernetes
    • Nomad
    • Docker Swarm
    • AWS ECS
    • Manage Application Lifecycle
    • Compose with other resources

    View Slide

  49. Workspaces
    Formerly: State Environments

    View Slide

  50. Workspaces
    • A state namespace
    • Allows single folder of TF config to manage multiple distinct
    sets of infrastructure resources

    View Slide

  51. Terminal
    $ terraform workspace list
    * default
    $ terraform workspace new mitchellh-test
    Created and switched to workspace "mitchellh-test"!
    $ terraform workspace list
    default
    * mitchellh-test

    View Slide

  52. File
    resource "aws_instance" "example" {
    count = “${terraform.workspace == "default" ? 5 : 1}"
    tags { Name = "web - ${terraform.workspace}" }
    # ... other fields
    }

    View Slide

  53. Terraform 0.9 (March 15, 2017)
    • Destroy provisioners
    • Remote backends
    • State locking
    • Workspaces (formerly State Environments)

    View Slide

  54. Destroy Provisioners

    View Slide

  55. Provisioners (Terraform <= 0.8)
    • Run arbitrary code locally or remotely on resource creation
    • If provisioner fails, resource is tainted and scheduled for
    recreation on the next apply

    View Slide

  56. File
    resource "null_resource" "example" {
    provisioner "local-exec" {
    command = "echo foo"
    }
    }

    View Slide

  57. File
    resource "null_resource" "example" {
    provisioner "local-exec" {
    command = "echo foo"
    }
    provisioner "local-exec" {
    command = "echo destroying"
    when = "destroy"
    }
    }

    View Slide

  58. Terminal
    $ terraform apply
    null_resource.example: Creating...
    null_resource.example: Provisioning with 'local-exec'...
    null_resource.example (local-exec): Executing: /bin/sh -c "echo foo"
    null_resource.example (local-exec): foo
    null_resource.example: Creation complete (ID: 1965091882910923448)
    Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

    View Slide

  59. Terminal
    $ terraform destroy -force
    null_resource.example: Refreshing state... (ID: 1965091882910923448)
    null_resource.example: Destroying... (ID: 1965091882910923448)
    null_resource.example: Provisioning with 'local-exec'...
    null_resource.example (local-exec): Executing: /bin/sh -c "echo destroying"
    null_resource.example (local-exec): destroying
    null_resource.example: Destruction complete

    View Slide

  60. Destroy Provisioners
    • Configured with when = "destroy"
    • Run on resource destroy (not just "terraform destroy")
    • Failure cancels physical resource destruction by default
    • Can allow failure with on_failure = "continue"

    View Slide

  61. File
    resource "null_resource" "example" {
    provisioner "local-exec" {
    command = "exit 1"
    when = "destroy"
    }
    }

    View Slide

  62. Terminal
    $ terraform destroy -force
    null_resource.example: Refreshing state... (ID: 8665586891184105369)
    null_resource.example: Destroying... (ID: 8665586891184105369)
    null_resource.example: Provisioning with 'local-exec'...
    null_resource.example (local-exec): Executing: /bin/sh -c "exit 1"
    Error applying plan:
    1 error(s) occurred:
    * null_resource.example (destroy): 1 error(s) occurred:
    * Error running command 'exit 1': exit status 1.

    View Slide

  63. File
    resource "null_resource" "example" {
    provisioner "local-exec" {
    command = "exit 1"
    when = "destroy"
    on_failure = "continue"
    }
    }

    View Slide

  64. Terminal
    $ terraform destroy -force
    null_resource.example: Refreshing state... (ID: 8665586891184105369)
    null_resource.example: Destroying... (ID: 8665586891184105369)
    null_resource.example: Provisioning with 'local-exec'...
    null_resource.example (local-exec): Executing: /bin/sh -c "exit 1"
    null_resource.example: Destruction complete
    Destroy complete! Resources: 1 destroyed.

    View Slide

  65. Destroy Provisioners
    • Useful for resource cleanup
    • Can SSH into machine (any machine!) prior to destruction
    • Recommend resource cleanup live as part of the resource itself,
    but destroy provisioners give you another option

    View Slide

  66. Remote Backends

    View Slide

  67. Before Remote Backends (TF <= 0.8)
    • Awkward "remote config" command
    • Users could accidentally run Terraform without remote init
    • Configuration only via CLI
    • Local cache of state stored in .terraform/terraform.tfstate
    • Changed remote configuration was manual

    View Slide

  68. Terminal
    $ # TERRAFORM <= 0.8, BEFORE REMOTE BACKENDS
    $ terraform remote config \
    -backend=S3 \
    -backend-config="bucket=" \
    -backend-config="key="
    ...

    View Slide

  69. Remote Backends
    • Subsumes "remote state", enables locking, workspaces, more
    • Configure from tf files, external configuration, or CLI
    • Detects configuration change
    • Forces new users of a TF configuration to initialize
    • One command to init them all: `terraform init`

    View Slide

  70. File
    terraform {
    backend "s3" {
    bucket = ""
    key = ""
    }
    }

    View Slide

  71. Terminal
    $ terraform init
    Initializing the backend...
    Successfully configured the backend "s3"! Terraform will automatically
    use this backend unless the backend configuration changes.
    Terraform has been successfully initialized!

    View Slide

  72. Terminal
    $ # New user, didn't run init
    $ terraform console
    Backend reinitialization required. Please run "terraform init".
    Reason: Initial configuration of the requested backend "s3"
    ...

    View Slide

  73. File
    terraform {
    backend "s3" {
    bucket = "CHANGED-THIS-SETTING"
    key = ""
    }
    }

    View Slide

  74. Terminal
    $ terraform console
    Backend reinitialization required. Please run "terraform init".
    Reason: Backend configuration changed for "s3"
    ...

    View Slide

  75. Terminal
    $ terraform console
    Backend reinitialization required. Please run "terraform init".
    Reason: Unsetting the previously set backend "s3"
    ...

    View Slide

  76. Remote Backends
    • One command to init: `terraform init`
    • Automatic detection of backend change (set, change, unset)
    • No state stored locally at all
    • Always gitignore ".terraform" folder

    View Slide

  77. A Focus on Safety
    • Common complaint: easy to corrupt remote state
    • Remote backends add new layer of safety: detecting changes,
    checking "lineage", disallowing writing unsafe state, more.

    View Slide

  78. A New "Init"
    • Init has existed since Terraform 0.1
    • Used to just setup folder structure for new projects
    • Now the single source of init, safe to run multiple times
    • Initializes backend, downloads modules, creates folders
    • One day: downloads providers, verifies versions, more...

    View Slide

  79. State Locking

    View Slide

  80. State Locking
    • For supported backends, Terraform automatically locks state on
    write operations
    • If unlock fails, error is shown with lock ID to allow a force unlock
    • Doesn't lock against concurrent reads

    View Slide

  81. File
    terraform { backend "consul" {} }
    resource "null_resource" "example" {
    provisioner "local-exec" {
    command = "sleep 10"
    }
    }

    View Slide

  82. Terminal
    $ terraform apply
    null_resource.example: Creating...
    null_resource.example: Provisioning with 'local-exec'...
    null_resource.example (local-exec): Executing: /bin/sh -c "sleep 10"

    View Slide

  83. Terminal
    $ terraform apply
    Error loading state: failed to lock state in Consul: Lock Info:
    ID: 5c0b66d6-018f-59b4-5536-499fec947fb2
    Path: foo
    Operation: OperationTypeApply
    Who: [email protected]
    Version: 0.9.1
    Created: 2017-04-04 16:16:59.733058195 +0000 UTC
    Info:
    $ terraform console
    >

    View Slide

  84. Terraform

    View Slide

  85. SEP 18-20, 2017 | AUSTIN, TEXAS
    One day of training followed by two days
    of talks on the future of infrastructure.
    Discount Code: HUG20

    View Slide

  86. THANKS!
    Q/A

    View Slide