$30 off During Our Annual Pro Sale. View Details »

Build, provision & deploy in the Cloud with Packer, Ansible & Terraform – PHPUK 2018

Thijs Feryn
February 15, 2018

Build, provision & deploy in the Cloud with Packer, Ansible & Terraform – PHPUK 2018

More information about this Cloud automation talk I did at PHPUK Conference 2018 in London can be found on https://feryn.eu/speaking/build-provision-deploy-cloud-packer-ansible-phpuk-2018/

Thijs Feryn

February 15, 2018
Tweet

More Decks by Thijs Feryn

Other Decks in Technology

Transcript

  1. By Thijs Feryn
    Build, provision & deploy
    in the Cloud with
    Packer, Ansible & Terraform
    PHP UK CONFERENCE

    View Slide

  2. I’m so sick of Cloud!

    View Slide

  3. View Slide

  4. Automation

    View Slide

  5. Many platforms

    View Slide

  6. View Slide

  7. View Slide

  8. ✓ Less lock-in
    ✓ Faster deployments
    ✓ Better scalability
    ✓ Better reproducibility
    ✓ Less human error
    ✓ Lower cost
    ✓ Operational stability
    Goals

    View Slide

  9. Hi, I’m Thijs

    View Slide

  10. I’m
    @ThijsFeryn
    on Twitter

    View Slide

  11. I’m an
    Evangelist
    At

    View Slide

  12. I’m an
    Evangelist
    At

    View Slide

  13. View Slide

  14. 185 17
    11 4

    View Slide

  15. https://joind.in/talk/ae063

    View Slide

  16. Build Provision Orchestrate

    View Slide

  17. App code Bake image
    Build

    View Slide

  18. View Slide

  19. App code
    Provision
    Bake image Pour “sauce”
    Build

    View Slide

  20. View Slide

  21. App code
    Provision Orchestrate
    Bake image Pour “sauce” Serve
    Build

    View Slide

  22. View Slide

  23. Infrastructure as code

    View Slide

  24. Disposable stacks

    View Slide

  25. Reproducible environments

    View Slide

  26. View Slide

  27. ✓ Alicloud ECS
    ✓ Amazon EC2
    ✓ Azure
    ✓ CloudStack
    ✓ DigitalOcean
    ✓ Docker
    ✓ File
    ✓ Google Cloud
    ✓ Hyper-V
    ✓ LXC
    ✓ LXD
    ✓ Null
    ✓ 1&1
    ✓ OpenStack
    ✓ Oracle OCI
    ✓ Parallels
    ✓ ProfitBricks
    ✓ QEMU
    ✓ Triton
    ✓ VirtualBox
    ✓ VMware
    ✓ Custom
    Packer builders

    View Slide

  28. View Slide

  29. {
    "builders": [
    {
    "type": "openstack",
    "identity_endpoint": "https://osp.combell.com:5000/v3",
    "tenant_name": "",
    "username": "",
    "password": "",
    "image_name": "thijsferyn_{{isotime \"2006_01_02__03_04_05\"}}",
    "source_image": "1b9fe43e-4cf2-4986-b7f2-51f5706b120b",
    "ssh_username": "debian",
    "domain_name" : "default",
    "flavor": "m1.medium",
    "networks": ["a1175c54-cb12-436d-ac7b-d327499dc39b"],
    "floating_ip_pool" : "public"
    }
    ],
    "provisioners": [
    {
    "type": "shell",
    "inline": ["apt-get update","apt-get install -y nginx"]
    }
    ]
    }

    View Slide

  30. View Slide

  31. {
    "builders": [
    {
    "type": "amazon-ebs",
    "access_key": "",
    "secret_key": "",
    "region": "eu-west-2",
    "instance_type": "t2.micro",
    "ssh_username": "admin",
    "associate_public_ip_address": true,
    "subnet_id": “subnet-11d3d56a",
    "source_ami": "ami-e1e8f085",
    "ami_name": "thijsferyn_{{isotime \"2006_01_02__03_04_05\"}}"
    }
    ],
    "provisioners": [
    {
    "type": "shell",
    "inline": ["apt-get update","apt-get install -y nginx"]
    }
    ]
    }

    View Slide

  32. $ packer build packer.json

    View Slide

  33. Packer
    build
    Cloud API
    Boot VM using
    source image
    Take VM
    snapshot
    Image
    registry
    Store VM
    snapshot as
    new image
    Shell
    provisioning
    run
    Tear
    down
    VM

    View Slide

  34. $ openstack image list
    +--------------------------------------+----------------------------------+--------+
    | ID | Name | Status |
    +--------------------------------------+----------------------------------+--------+
    | ef90ad1c-ed64-4ae0-8283-e446e95c33e6 | thijsferyn_2018_01_31__02_26_27 | active |
    | aa0a5244-0d57-477d-b985-653d6b881292 | Windows-2012-R2 | active |
    | 31e84bbb-abbb-4d5a-b620-ecebf4697d16 | centos-6-64bit | active |
    | 9eac74f9-6e6c-45a9-bd93-915d3e390687 | centos-7-64bit | active |
    | 3a69a110-775f-41ad-9d63-32079db57203 | cirros-0.3.4-64bit | active |
    | 244cbbde-77b6-4f1e-837f-9250520ee78b | coreos-stable | active |
    | 1b9fe43e-4cf2-4986-b7f2-51f5706b120b | debian-jessie-64bit | active |
    | 690b063b-f239-4032-ad23-9ae6337f248f | debian-stretch-64bit | active |
    | d8034a10-9b15-46b1-9c68-bd98f92d0ffb | ubuntu-server-precise-lts-64-bit | active |
    | 040f9d1d-bb18-4466-abd7-9f2172e6db70 | ubuntu-server-trusty-lts-64-bit | active |
    | 5be2b652-a965-4ecb-a2b4-b9f83d7779e6 | ubuntu-server-xenial-lts-64-bit | active |
    +--------------------------------------+----------------------------------+--------+

    View Slide

  35. $ aws ec2 describe-images --owners=self --query "Images[*].
    {ID:ImageId,Name:Name,Status:State}" --output=table
    +--------------+-----------------------------------+-------------+
    | ID | Name | Status |
    +--------------+-----------------------------------+-------------+
    | ami-82ccd6e6| thijsferyn_2018_01_31__02_26_27 | available |
    +--------------+-----------------------------------+-------------+

    View Slide

  36. View Slide

  37. Configuration
    management
    system

    View Slide

  38. View Slide

  39. ✓ Written in Python
    ✓ Agentless
    ✓ Standalone
    ✓ SSH-based
    ✓ Runs playbook (yml files)
    ✓ Groups playbook in roles
    ✓ Install software
    ✓ Configure your server
    Ansible

    View Slide

  40. ---
    - hosts: all
    vars:
    code_folders:
    - "web"
    become: true
    tasks:
    - name: update apt cache
    apt: update_cache=yes cache_valid_time=86400
    - name: install required packages
    apt: name={{ item }} state=present
    with_items:
    - vim
    - curl
    - nginx
    - php7.0-fpm
    - rsync
    - name: create /var/www/html/ folder
    file:
    path: /var/www/html/
    state: directory
    mode: 0755
    owner: www-data
    playbook.yml

    View Slide

  41. - name: create /var/www/html/ folder
    file:
    path: /var/www/html/
    state: directory
    mode: 0755
    owner: www-data
    - name: Copy code
    synchronize:
    src: ../{{ item }}
    dest: /var/www/html
    delete: yes
    recursive: yes
    group: no
    owner: no
    perms: no
    with_items: "{{code_folders}}"
    - name: Chown code to www-data
    file:
    path: /var/www/html/{{ item }}
    group: www-data
    owner: www-data
    mode: u=rwx,g=r,o=r
    recurse: yes
    playbook.yml

    View Slide

  42. Organize
    your
    playbooks

    View Slide

  43. .
    !"" files
    # $"" db.sql
    !"" host_vars
    # $"" default.yml
    !"" group_vars
    # $"" all.yml
    !"" inventory
    !"" playbook.yml
    !"" roles
    # $"" my-role
    # !"" defaults
    # # $"" main.yml
    # !"" handlers
    # # $"" main.yml
    # !"" meta
    # # $"" main.yml
    # !"" tasks
    # # $"" main.yml
    # !"" templates
    # # $"" template.j2
    # $"" vars
    # $"" vars.yml
    !"" tasks
    # !"" task1.yml
    # !"" task2.yml
    $"" templates
    $"" template.j2

    View Slide

  44. ---
    - hosts: all
    tasks:
    - include_tasks: task1.yml
    - include_tasks: task2.yml
    roles:
    - my-role

    View Slide

  45. $ ansible-playbook playbook.yml

    View Slide

  46. {
    "builders": [
    {
    "type": "openstack",
    "identity_endpoint": "https://osp.combell.com:5000/v3",
    "tenant_name": "18324-1",
    "username": "18324",
    "password": "",
    "image_name": "MyBuild-{{isotime \"2006-01-02 03:04:05\"}}",
    "source_image": "1b9fe43e-4cf2-4986-b7f2-51f5706b120b",
    "ssh_username": "debian",
    "domain_name" : "default",
    "flavor": "m1.medium",
    "networks": ["a1175c54-cb12-436d-ac7b-d327499dc39b"],
    "floating_ip_pool" : "public"
    }
    ],
    "provisioners": [
    {
    "type": "ansible",
    "playbook_file": "./ansible/playbook.yml"
    }
    ]
    }

    View Slide

  47. $ ansible-playbook --extra-vars packer_build_name=openstack
    packer_builder_type=openstack -i /var/folders/j7/
    kpxjpxq92vs_t97gg6tqtttm0000gn/T/packer-provisioner-
    ansible972172074 /Users/thijsferyn/Dropbox/Sites/openstack/
    packer-ansible-terraform/ansible/playbook.yml --private-key /
    var/folders/j7/kpxjpxq92vs_t97gg6tqtttm0000gn/T/ansible-
    key333617975

    View Slide

  48. Packer
    build
    Cloud API
    Boot VM using
    source image
    Take VM
    snapshot
    Image
    registry
    Store VM
    snapshot as
    new image
    Ansible
    provisioning
    run
    Tear
    down
    VM

    View Slide

  49. "image_name": "MyBuild-{{isotime \"2006-01-02 03:04:05\"}}",
    "source_image": "1b9fe43e-4cf2-4986-b7f2-51f5706b120b",
    "ssh_username": "debian",
    "domain_name" : "default",
    "flavor": "m1.medium",
    "networks": ["a1175c54-cb12-436d-ac7b-d327499dc39b"],
    "floating_ip_pool" : "public"
    }
    ],
    "provisioners": [
    {
    "type": "ansible",
    "playbook_file": "./ansible/playbook.yml"
    }
    ],
    "post-processors": [
    {
    "type": "manifest",
    "output": "manifest.json"
    }
    ]
    }

    View Slide

  50. Store build
    results
    Packer
    build
    Cloud API
    Boot VM using
    source image
    Take VM
    snapshot
    Image
    registry
    Store VM
    snapshot as
    new image
    Ansible
    provisioning
    run
    Tear
    down
    VM
    manifest.json

    View Slide

  51. {
    "builds": [
    {
    "name": "amazon-ebs",
    "builder_type": "amazon-ebs",
    "build_time": 1517408990,
    "files": null,
    "artifact_id": "eu-west-2:ami-82ccd6e6",
    "packer_run_uuid": "ba98283c-08ca-a9fc-ab10-e49502ce4ab6"
    },
    {
    "name": "openstack",
    "builder_type": "openstack",
    "build_time": 1509053610,
    "files": null,
    "artifact_id": "ef90ad1c-ed64-4ae0-8283-e446e95c33e6",
    "packer_run_uuid": "b6000539-f3e6-85dd-2733-9fb3ae443e9f"
    }
    ],
    "last_run_uuid": "b6000539-f3e6-85dd-2733-9fb3ae443e9f"
    }

    View Slide

  52. $ cat manifest.json | \
    jq '.builds[-1].artifact_id'

    View Slide

  53. $ openstack image list
    +--------------------------------------+----------------------------------+--------+
    | ID | Name | Status |
    +--------------------------------------+----------------------------------+--------+
    | ef90ad1c-ed64-4ae0-8283-e446e95c33e6 | thijsferyn_2018_01_31__02_26_27 | active |
    | aa0a5244-0d57-477d-b985-653d6b881292 | Windows-2012-R2 | active |
    | 31e84bbb-abbb-4d5a-b620-ecebf4697d16 | centos-6-64bit | active |
    | 9eac74f9-6e6c-45a9-bd93-915d3e390687 | centos-7-64bit | active |
    | 3a69a110-775f-41ad-9d63-32079db57203 | cirros-0.3.4-64bit | active |
    | 244cbbde-77b6-4f1e-837f-9250520ee78b | coreos-stable | active |
    | 1b9fe43e-4cf2-4986-b7f2-51f5706b120b | debian-jessie-64bit | active |
    | 690b063b-f239-4032-ad23-9ae6337f248f | debian-stretch-64bit | active |
    | d8034a10-9b15-46b1-9c68-bd98f92d0ffb | ubuntu-server-precise-lts-64-bit | active |
    | 040f9d1d-bb18-4466-abd7-9f2172e6db70 | ubuntu-server-trusty-lts-64-bit | active |
    | 5be2b652-a965-4ecb-a2b4-b9f83d7779e6 | ubuntu-server-xenial-lts-64-bit | active |
    +--------------------------------------+----------------------------------+--------+

    View Slide

  54. View Slide

  55. Deploy
    the VM
    image

    View Slide

  56. Orchestration

    View Slide

  57. ✓ Launch virtual network
    ✓ Boot up webservers
    ✓ Configure firewalls
    ✓ Assign loadbalancers
    ✓ Create DNS-records
    ✓ Assign SSH keys
    ✓ Create autoscaling group
    Orchestration

    View Slide

  58. ✓ CloudFormation (AWS)
    ✓ Heat (OpenStack)
    ✓ Resource manager (Azure)
    Vendor-specific orchestration tools

    View Slide

  59. View Slide

  60. ✓ Infrastructure As Code
    ✓ Plan, graph, execute
    ✓ State management
    ✓ Multiple providers
    ✓ Incremental changes
    ✓ Interacts with Cloud vendor APIs
    Terraform

    View Slide

  61. ✓ Alicloud
    ✓ Archive
    ✓ Arukas
    ✓ AWS
    ✓ Bitbucket
    ✓ CenturyLinkCloud
    ✓ Chef
    ✓ Circonus
    ✓ Cloudflare
    ✓ CloudStack
    ✓ Cobbler
    ✓ Consul
    ✓ Datadog
    ✓ DigitalOcean
    ✓ DNS
    ✓ DNSMadeEasy
    ✓ DNSimple
    ✓ Docker
    ✓ Dyn
    ✓ External
    ✓ Fastly
    ✓ GitHub
    ✓ Gitlab
    ✓ Google Cloud
    ✓ Grafana
    ✓ Heroku
    ✓ HTTP
    ✓ Icinga2
    ✓ Ignition
    ✓ InfluxDB
    ✓ Kubernetes
    ✓ Librato
    ✓ Local
    ✓ Logentries
    ✓ LogicMonitor
    ✓ Mailgun
    ✓ Microsoft Azure
    ✓ Microsoft Azure
    (Legacy ASM)
    ✓ MySQL
    ✓ New Relic
    ✓ Nomad
    ✓ NS1
    ✓ 1&1
    ✓ Oracle Public Cloud
    ✓ OpenStack
    ✓ OpsGenie
    ✓ OVH
    ✓ Packet
    ✓ PagerDuty
    ✓ PostgreSQL
    ✓ PowerDNS
    ✓ ProfitBricks
    ✓ RabbitMQ
    ✓ Rancher
    ✓ Random
    ✓ Rundeck
    ✓ Scaleway
    ✓ SoftLayer
    ✓ StatusCake
    ✓ Spotinst
    ✓ Template
    ✓ Terraform
    ✓ Terraform Enterprise
    ✓ TLS
    ✓ Triton
    ✓ UltraDNS
    ✓ Vault
    ✓ VMware vCloud
    Director
    ✓ VMware vSphere
    Terraform providers

    View Slide

  62. main.tf

    View Slide

  63. variable "ami" {
    type = "string"
    default = "ami-e1e8f085" #Stock Debian Stretch
    }
    resource "aws_instance" "web" {
    ami = "${var.ami}"
    instance_type = "t2.micro"
    security_groups = ["Web"]
    tags {
    Name = "web"
    }
    }
    data "aws_route53_zone" "web" {
    name = "aws.combell.com."
    }
    resource "aws_route53_record" "web" {
    zone_id = "${data.aws_route53_zone.web.zone_id}"
    name = "thijsferyn-web.${data.aws_route53_zone.web.name}"
    type = "A"
    ttl = "60"
    records = ["${aws_instance.web.public_ip}"]
    }
    output "public_ip" {
    value = "${aws_instance.web.public_ip}"
    }
    output "hostname" {
    value = "${aws_route53_record.web.name}"
    }

    View Slide

  64. $ terraform init

    View Slide

  65. $ terraform init
    Initializing provider plugins...
    - Checking for available provider plugins on https://releases.hashicorp.com...
    - Downloading plugin for provider "aws" (1.8.0)...
    The following providers do not have any version constraints in configuration,
    so the latest version was installed.
    To prevent automatic upgrades to new major versions that may contain breaking
    changes, it is recommended to add version = "..." constraints to the
    corresponding provider blocks in configuration, with the constraint strings
    suggested below.
    * provider.aws: version = "~> 1.8"
    Terraform has been successfully initialized!

    View Slide

  66. $ terraform plan

    View Slide

  67. $ terraform plan
    Refreshing Terraform state in-memory prior to plan...
    The refreshed state will be used to calculate this plan, but will not be
    persisted to local or remote state storage.
    data.aws_route53_zone.web: Refreshing state...
    ------------------------------------------------------------------------
    An execution plan has been generated and is shown below.
    Resource actions are indicated with the following symbols:

    View Slide

  68. An execution plan has been generated and is shown below.
    Resource actions are indicated with the following symbols:
    + create
    Terraform will perform the following actions:
    + aws_instance.web
    id:
    ami: "ami-e1e8f085"
    associate_public_ip_address:
    availability_zone:
    ebs_block_device.#:
    ephemeral_block_device.#:
    instance_state:
    instance_type: "t2.micro"
    ipv6_address_count:
    ipv6_addresses.#:
    key_name:
    network_interface.#:
    network_interface_id:
    placement_group:
    primary_network_interface_id:
    private_dns:
    private_ip:
    public_dns:
    public_ip:
    root_block_device.#:
    security_groups.#: "1"
    security_groups.2661672386: "Web"
    source_dest_check: "true"
    subnet_id:
    tags.%: "1"
    tags.Name: "web"
    tenancy:
    volume_tags.%:
    vpc_security_group_ids.#:

    View Slide

  69. + aws_route53_record.web
    id:
    fqdn:
    name: "thijsferyn-web.aws.combell.com"
    records.#:
    ttl: "60"
    type: "A"
    zone_id: "Z3K2HG3W48B0MR"
    Plan: 2 to add, 0 to change, 0 to destroy.

    View Slide

  70. $ terraform apply

    View Slide

  71. $ terraform plan -out=bla
    $ terraform apply bla

    View Slide

  72. View Slide

  73. $ terraform output
    hostname = thijsferyn-web.aws.combell.com
    public_ip = 35.178.38.182

    View Slide

  74. $ terraform output -json
    {
    "hostname": {
    "sensitive": false,
    "type": "string",
    "value": "thijsferyn-web.aws.combell.com"
    },
    "public_ip": {
    "sensitive": false,
    "type": "string",
    "value": "35.178.38.182"
    }
    }

    View Slide

  75. $ terraform show

    View Slide

  76. $ terraform show
    aws_instance.web:
    id = i-0333e97ea6bdbc5ae
    ami = ami-e1e8f085
    associate_public_ip_address = true
    availability_zone = eu-west-2a
    disable_api_termination = false
    ebs_block_device.# = 0
    ebs_optimized = false
    ephemeral_block_device.# = 0
    iam_instance_profile =
    instance_state = running
    instance_type = t2.micro
    ipv6_addresses.# = 0
    key_name =
    monitoring = false
    network_interface.# = 0
    network_interface_id = eni-c62e7193
    placement_group =
    primary_network_interface_id = eni-c62e7193
    private_dns = ip-172-31-19-7.eu-west-2.compute.internal

    View Slide

  77. network_interface_id = eni-c62e7193
    placement_group =
    primary_network_interface_id = eni-c62e7193
    private_dns = ip-172-31-19-7.eu-west-2.compute.internal
    private_ip = 172.31.19.7
    public_dns = ec2-35-178-38-182.eu-west-2.compute.amazonaws.com
    public_ip = 35.178.38.182
    root_block_device.# = 1
    root_block_device.0.delete_on_termination = true
    root_block_device.0.iops = 100
    root_block_device.0.volume_id = vol-0b9364527ec09549e
    root_block_device.0.volume_size = 8
    root_block_device.0.volume_type = gp2
    security_groups.# = 1
    security_groups.763657905 = Web
    source_dest_check = true
    subnet_id = subnet-11d3d56a
    tags.% = 1
    tags.Name = web
    tenancy = default
    volume_tags.% = 0
    vpc_security_group_ids.# = 0

    View Slide

  78. aws_route53_record.web:
    id = Z3K2HG3W48B0MR_thijsferyn-web.aws.combell.com._A
    fqdn = thijsferyn-web.aws.combell.com
    health_check_id =
    name = thijsferyn-web.aws.combell.com
    records.# = 1
    records.1088376871 = 35.178.38.182
    set_identifier =
    ttl = 60
    type = A
    zone_id = Z3K2HG3W48B0MR
    data.aws_route53_zone.web:
    id = Z3K2HG3W48B0MR
    caller_reference = F7596F57-9ADE-9B0D-8F53-53DF42E03DB3
    comment =
    name = aws.combell.com.
    private_zone = false
    resource_record_set_count = 3
    zone_id = Z3K2HG3W48B0MR
    Outputs:
    hostname = thijsferyn-web.aws.combell.com
    public_ip = 35.178.38.182

    View Slide

  79. State

    View Slide

  80. State
    backend Terraform Cloud API

    View Slide

  81. terraform.tfstate
    terraform.tfstate.backup

    View Slide

  82. JSON data

    View Slide

  83. ✓ Artifactory
    ✓ Azurerm
    ✓ Consul
    ✓ Etcd
    ✓ Gcs
    ✓ HTTP
    ✓ Manta
    ✓ S3
    ✓ Swift
    ✓ Terraform
    Enterprise
    Remote state

    View Slide

  84. Changes

    View Slide

  85. $ terraform plan -var 'ami=ami-20ccd644'

    View Slide

  86. ~ update in-place
    -/+ destroy and then create replacement
    Terraform will perform the following actions:
    -/+ aws_instance.web (new resource required)
    id: "i-0333e97ea6bdbc5ae" => (forces new
    resource)
    ami: "ami-e1e8f085" => "ami-20ccd644" (forces new
    resource)
    associate_public_ip_address: "true" =>
    availability_zone: "eu-west-2a" =>
    ebs_block_device.#: "0" =>
    ephemeral_block_device.#: "0" =>
    instance_state: "running" =>
    instance_type: "t2.micro" => "t2.micro"
    ipv6_address_count: "" =>
    ipv6_addresses.#: "0" =>
    key_name: "" =>
    network_interface.#: "0" =>
    network_interface_id: "eni-c62e7193" =>
    placement_group: "" =>
    primary_network_interface_id: "eni-c62e7193" =>
    private_dns: "ip-172-31-19-7.eu-west-2.compute.internal" =>

    private_ip: "172.31.19.7" =>

    View Slide

  87. key_name: "" =>
    network_interface.#: "0" =>
    network_interface_id: "eni-c62e7193" =>
    placement_group: "" =>
    primary_network_interface_id: "eni-c62e7193" =>
    private_dns: "ip-172-31-19-7.eu-west-2.compute.internal" =>

    private_ip: "172.31.19.7" =>
    public_dns: "ec2-35-178-38-182.eu-
    west-2.compute.amazonaws.com" =>
    public_ip: "35.178.38.182" =>
    root_block_device.#: "1" =>
    security_groups.#: "1" => "1"
    security_groups.763657905: "Web" => "Web"
    source_dest_check: "true" => "true"
    subnet_id: "subnet-11d3d56a" =>
    tags.%: "1" => "1"
    tags.Name: "web" => "web"
    tenancy: "default" =>
    volume_tags.%: "0" =>
    vpc_security_group_ids.#: "0" =>
    ~ aws_route53_record.web
    records.#: "" =>
    Plan: 1 to add, 1 to change, 1 to destroy.

    View Slide

  88. $ terraform apply -var 'ami=ami-20ccd644'

    View Slide

  89. Downtime

    View Slide

  90. High availability plan

    View Slide

  91. Workspaces

    View Slide

  92. One Terraform project
    Multiple workspaces
    State per workspace
    Multiple environments

    View Slide

  93. Blue
    Green

    View Slide

  94. Workspace 1 Workspace 2
    Internet
    Entry
    point
    BLUE
    GREEN

    View Slide

  95. $ terraform init
    $ terraform workspace new wsp1
    $ terraform workspace new wsp2
    $ terraform workspace select wsp1
    $ terraform plan
    $ terraform apply
    $ terraform workspace select wsp2
    $ terraform plan
    $ terraform apply

    View Slide

  96. .
    !"" main.tf
    !"" terraform.tfstate
    !"" terraform.tfstate.backup
    $"" terraform.tfstate.d
    !"" wsp1
    $"" wsp2

    View Slide

  97. variable "ami" {
    type = "string"
    default = "ami-e1e8f085" #Stock Debian Stretch
    }
    resource "aws_instance" "web" {
    ami = "${var.ami}"
    instance_type = "t2.micro"
    security_groups = ["Web"]
    tags {
    Name = "web-${terraform.workspace}"
    }
    }
    output "public_ip" {
    value = "${aws_instance.web.public_ip}"
    }
    Remove DNS
    resource
    Interpolate
    workspace
    info

    View Slide

  98. variable "ip" {
    type = "string"
    }
    data "aws_route53_zone" "web" {
    name = "aws.combell.com."
    }
    resource "aws_route53_record" "web" {
    zone_id = "${data.aws_route53_zone.web.zone_id}"
    name = "thijsferyn-web.${data.aws_route53_zone.web.name}"
    type = "A"
    ttl = "60"
    records = ["${var.ip}"]
    }
    output "hostname" {
    value = "${aws_route53_record.web.name}"
    }
    Separate
    Terraform
    project

    View Slide

  99. $ terraform apply -var 'ip="35.176.171.168"'

    View Slide

  100. DNS round robin

    View Slide

  101. variable "ami" {
    default = "ami-e1e8f085" #Stock Debian Stretch
    }
    variable "instances" {
    default = "2"
    }
    resource "aws_instance" "web" {
    ami = "${var.ami}"
    instance_type = "t2.micro"
    security_groups = ["Web"]
    count = "${var.instances}"
    tags {
    Name = "web-${format("%02d", count.index+1)}-${terraform.workspace}"
    }
    }
    output "public_ip" {
    value = ["${aws_instance.web.*.public_ip}"]
    }
    Create
    multiple
    instances at
    once

    View Slide

  102. variable "ip" {
    type = "list"
    }
    data "aws_route53_zone" "web" {
    name = "aws.combell.com."
    }
    resource "aws_route53_record" "web" {
    zone_id = "${data.aws_route53_zone.web.zone_id}"
    name = "thijsferyn-web.${data.aws_route53_zone.web.name}"
    type = "A"
    ttl = "60"
    records = ["${var.ip}"]
    }
    output "hostname" {
    value = "${aws_route53_record.web.name}"
    }

    View Slide

  103. $ terraform apply -var 'ip=["35.176.171.168","35.177.208.38"]'

    View Slide

  104. Rolling
    updates

    View Slide

  105. Deployment B
    Internet
    Entry
    point
    Drain connections
    and gradually switch
    from deployment A
    to deployment B
    Deployment A
    ROLLING
    UPDATES

    View Slide

  106. Autoscaling
    group

    View Slide

  107. Custom AMI
    V
    P
    C
    ALB
    Private
    subnet 1
    Private
    subnet 2
    Public
    subnet 1
    Public
    subnet 2
    Internet
    gateway
    Route
    table
    Target
    group
    Internet
    Launch
    config
    Autoscaling
    group
    Metrics
    Security
    group

    View Slide

  108. resource "aws_vpc" "thijsferyn_terraform" {
    cidr_block = "10.0.0.0/16"
    tags {
    Name = "thijsferyn_${terraform.workspace}_terraform"
    }
    }
    resource "aws_subnet" "thijsferyn_terraform_public" {
    availability_zone = "eu-west-2a"
    vpc_id = "${aws_vpc.thijsferyn_terraform.id}"
    cidr_block = "10.0.0.0/24"
    tags {
    Name = "thijsferyn_terraform_${terraform.workspace}_public"
    }
    }
    resource "aws_subnet" "thijsferyn_terraform_private" {
    availability_zone = "eu-west-2a"
    vpc_id = "${aws_vpc.thijsferyn_terraform.id}"
    cidr_block = "10.0.1.0/24"
    tags {
    Name = "thijsferyn_terraform_${terraform.workspace}_private"
    }
    }
    Virtual
    network

    View Slide

  109. resource "aws_security_group" "thijsferyn_terraform" {
    name = "thijsferyn-terraform-${terraform.workspace}"
    description = "Allow HTTP(S) & SSH"
    vpc_id = "${aws_vpc.thijsferyn_terraform.id}"
    ingress {
    from_port = 80
    to_port = 80
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    }
    ingress {
    from_port = 443
    to_port = 443
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    }
    ingress {
    from_port = 22
    to_port = 22
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    }
    egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
    }
    tags {
    Name = "thijsferyn_terraform-${terraform.workspace}"
    }
    }
    Firewall

    View Slide

  110. data "aws_ami" "thijsferyn_terraform" {
    most_recent = true
    filter {
    name = "name"
    values = ["thijsferyn_*"]
    }
    filter {
    name = "virtualization-type"
    values = ["hvm"]
    }
    owners = ["826481595599"]
    }
    resource "aws_launch_configuration" "thijsferyn_terraform" {
    image_id = "${data.aws_ami.thijsferyn_terraform.id}"
    instance_type = "t2.micro"
    security_groups = ["${aws_security_group.thijsferyn_terraform.id}"]
    lifecycle {
    create_before_destroy = true
    }
    }
    Prepare
    VMs

    View Slide

  111. resource "aws_lb" "thijsferyn_terraform" {
    name = "thijsferyn-terraform-${terraform.workspace}"
    internal = false
    security_groups = ["${aws_security_group.thijsferyn_terraform.id}"]
    subnets = ["${aws_subnet.thijsferyn_terraform_public.*.id}","$
    {aws_subnet.thijsferyn_terraform_public2.*.id}"]
    }
    resource "aws_lb_listener" "thijsferyn_terraform" {
    load_balancer_arn = "${aws_lb.thijsferyn_terraform.arn}"
    port = "80"
    protocol = "HTTP"
    default_action {
    target_group_arn = "${aws_lb_target_group.thijsferyn_terraform.arn}"
    type = "forward"
    }
    }
    resource "aws_lb_target_group" "thijsferyn_terraform" {
    name = "thijsferyn-terraform-${terraform.workspace}"
    port = 80
    protocol = "HTTP"
    vpc_id = "${aws_vpc.thijsferyn_terraform.id}"
    }
    Load
    balancing

    View Slide

  112. resource "aws_autoscaling_group" "thijsferyn_terraform" {
    availability_zones = ["eu-west-2a"]
    name = "thijsferyn-terraform-${terraform.workspace}-$
    {aws_launch_configuration.thijsferyn_terraform.name}"
    max_size = 10
    min_size = 3
    launch_configuration = "${aws_launch_configuration.thijsferyn_terraform.name}"
    target_group_arns = ["${aws_lb_target_group.thijsferyn_terraform.id}"]
    health_check_grace_period = 10
    health_check_type = "ELB"
    vpc_zone_identifier = ["${aws_subnet.thijsferyn_terraform_private.id}"]
    default_cooldown = 60
    depends_on = ["aws_launch_configuration.thijsferyn_terraform"]
    lifecycle {
    create_before_destroy = true
    }
    tag {
    key = "Name"
    value = "thijsferyn-terraform-${terraform.workspace}-$
    {aws_launch_configuration.thijsferyn_terraform.name}"
    propagate_at_launch = true
    }
    }
    Autoscaling
    group

    View Slide

  113. resource "aws_autoscaling_policy" "thijsferyn_terraform_high" {
    name = "thijsferyn-terraform-${terraform.workspace}-high"
    scaling_adjustment = 1
    adjustment_type = "ChangeInCapacity"
    cooldown = 10
    autoscaling_group_name = "${aws_autoscaling_group.thijsferyn_terraform.name}"
    }
    resource "aws_autoscaling_policy" "thijsferyn_terraform_low" {
    name = "thijsferyn-terraform-${terraform.workspace}low"
    scaling_adjustment = -1
    adjustment_type = "ChangeInCapacity"
    cooldown = 10
    autoscaling_group_name = "${aws_autoscaling_group.thijsferyn_terraform.name}"
    }
    Scale up
    scale down

    View Slide

  114. resource "aws_cloudwatch_metric_alarm" "thijsferyn_terraform_high" {
    alarm_name = "thijsferyn-terraform-${terraform.workspace}-high"
    comparison_operator = "GreaterThanOrEqualToThreshold"
    evaluation_periods = "1"
    metric_name = "RequestCountPerTarget"
    namespace = "AWS/ApplicationELB"
    period = "60"
    statistic = "Sum"
    threshold = "20"
    dimensions {
    TargetGroup = "${aws_lb_target_group.thijsferyn_terraform.name}"
    LoadBalancer = "${aws_lb.thijsferyn_terraform.name}"
    }
    alarm_description = "More than 20 requests per target causes scaleout"
    alarm_actions = ["${aws_autoscaling_policy.thijsferyn_terraform_high.arn}"]
    }
    High
    water mark

    View Slide

  115. $ terraform plan
    $ terraform apply

    View Slide

  116. $ terraform graph | dot -Tpng > graph.png

    View Slide

  117. View Slide

  118. View Slide

  119. ✓ vars.tf
    ✓ output.tf
    ✓ secgroup.tf
    ✓ keypair.tf
    ✓ compute.tf
    ✓ network.tf
    ✓ provider.tf
    Organize Terraform files

    View Slide

  120. Modules

    View Slide

  121. !"" autoscaling
    # !"" README.MD
    # !"" main.tf
    # !"" outputs.tf
    # $"" variables.tf
    !"" dns
    # !"" README.MD
    # !"" main.tf
    # !"" outputs.tf
    # $"" variables.tf
    !"" launch
    # !"" README.MD
    # !"" main.tf
    # !"" outputs.tf
    # $"" variables.tf
    !"" loadbalancing
    # !"" README.MD
    # !"" main.tf
    # !"" outputs.tf
    # $"" variables.tf
    !"" main.tf
    !"" networking
    # !"" README.MD
    # !"" main.tf
    # !"" outputs.tf
    # $"" variables.tf

    View Slide

  122. variable "zone_name" {
    default = "aws.combell.com."
    }
    variable "record_name" {
    default = ""
    }
    variable "records" {
    type = "list"
    }
    output "dns_endpoint" {
    value = "${aws_route53_record.thijsferyn_terraform_aws_combell_com.fqdn}"
    }
    data "aws_route53_zone" "thijsferyn_terraform_aws_combell_com" {
    name = "${var.zone_name}"
    }
    resource "aws_route53_record" "thijsferyn_terraform_aws_combell_com" {
    zone_id = "${data.aws_route53_zone.thijsferyn_terraform_aws_combell_com.zone_id}"
    name = "${var.record_name == "" ? "thijsferyn-terraform${terraform.workspace ==
    "default" ? "" : "-${terraform.workspace}"}": var.record_name}.$
    {data.aws_route53_zone.thijsferyn_terraform_aws_combell_com.name}"
    type = "CNAME"
    ttl = "60"
    records = ["${var.records}"]
    }
    Variables.tf
    module input
    args
    Output.tf
    module
    output
    Main.tf
    processing

    View Slide

  123. module "dns" {
    source = "./dns"
    record_name = "www"
    records = ["bla.domain.com"]
    }
    output "dns_endpoint" {
    value = "${module.dns.dns_endpoint}"
    }
    Root
    Terraform
    file

    View Slide

  124. module "networking" {
    source = "./networking"
    }
    module "loadbalancing" {
    source = "./loadbalancing"
    security_groups = ["${module.networking.security_group_id}"]
    vpc_id = "${module.networking.vpc_id}"
    subnets = ["${module.networking.subnet_public1_id}","${module.networking.subnet_public2_id}"]
    }
    module "launch" {
    source = "./launch"
    instance_type = "t2.small"
    security_groups = ["${module.networking.security_group_id}"]
    }
    module "autoscaling" {
    source = "./autoscaling"
    target_group = "${module.loadbalancing.target_group_name}"
    loadbalancer = "${module.loadbalancing.loadbalancer_name}"
    min_size = "4"
    subnets = ["${module.networking.subnet_private1_id}","${module.networking.subnet_private2_id}"]
    launch_configuration = "${module.launch.launch_configuration_name}"
    target_group_arn = "${module.loadbalancing.target_group_id}"
    }
    module "dns" {
    source = "./dns"
    records = ["${module.loadbalancing.dns_name}"]
    }
    output "dns_endpoint" {
    value = "${module.dns.dns_endpoint}"
    }

    View Slide

  125. https://registry.terraform.io

    View Slide

  126. View Slide

  127. module "consul" {
    source = "hashicorp/consul/aws"
    }

    View Slide

  128. $ terraform init

    View Slide

  129. Downloading modules...
    Initializing provider plugins...
    - Checking for available provider plugins on https://releases.hashicorp.com...
    - Downloading plugin for provider "aws" (1.9.0)...
    - Downloading plugin for provider "template" (1.0.0)...
    The following providers do not have any version constraints in configuration,
    so the latest version was installed.
    To prevent automatic upgrades to new major versions that may contain breaking
    changes, it is recommended to add version = "..." constraints to the
    corresponding provider blocks in configuration, with the constraint strings
    suggested below.
    * provider.aws: version = "~> 1.9"
    * provider.template: version = "~> 1.0"
    Terraform has been successfully initialized!
    You may now begin working with Terraform. Try running "terraform plan" to see
    any changes that are required for your infrastructure. All Terraform commands
    should now work.
    If you ever set or change modules or backend configuration for Terraform,
    rerun this command to reinitialize your working directory. If you forget, other
    commands will detect it and remind you to do so if necessary.

    View Slide

  130. module “example" {
    source = "github.com/hashicorp/example"
    }
    module "consul" {
    source = "git::https://hashicorp.com/consul.git"
    }
    module "ami" {
    source = "git::ssh://[email protected]/owner/repo.git"
    }

    View Slide

  131. ✓Local files
    ✓Terraform Registry
    ✓GitHub
    ✓Bitbucket
    ✓Generic Git Repo
    ✓Mercurial
    ✓HTTP
    ✓S3 buckets

    View Slide

  132. Remote state

    View Slide

  133. View Slide

  134. $ consul agent -data-dir=/tmp/consul \
    -server -ui -dev

    View Slide

  135. terraform {
    backend "consul" {
    address = "127.0.0.1:8500"
    path = "example/terraform_state"
    }
    }

    View Slide

  136. $ terraform apply
    Error locking state: Error acquiring the state lock:
    resource temporarily unavailable
    Lock Info:
    ID: d2c36deb-32c0-ecc7-b1f1-10068c2ed93b
    Path: terraform.tfstate
    Operation: OperationTypeApply
    Who: [email protected]
    Version: 0.10.7
    Created: 2018-02-02 16:03:17.005327423 +0000 UTC
    Info:
    Terraform acquires a state lock to protect the state
    from being written
    by multiple users at the same time. Please resolve
    the issue above and try
    again. For most commands, you can disable locking
    with the "-lock=false"
    flag, but this is not recommended.

    View Slide

  137. data "terraform_remote_state" "web" {
    backend = "consul"
    config {
    address = "127.0.0.1:8500"
    path = "thijsferyn/terraform/production"
    }
    }
    data "aws_route53_zone" "web" {
    name = "aws.combell.com."
    }
    resource "aws_route53_record" "web" {
    zone_id = "${data.aws_route53_zone.web.zone_id}"
    name = "thijsferyn-web.${data.aws_route53_zone.web.name}"
    type = "A"
    ttl = "60"
    records = ["${data.terraform_remote_state.web.public_ip}"]
    }
    output "hostname" {
    value = "${aws_route53_record.web.name}"
    }

    View Slide

  138. Instead of using CLI
    variables, we can use
    remote state to
    communicate between
    Terraform projects

    View Slide

  139. Hybrid Cloud?

    View Slide

  140. https://feryn.eu
    https://twitter.com/ThijsFeryn
    https://instagram.com/ThijsFeryn

    View Slide

  141. https://joind.in/talk/ae063

    View Slide

  142. View Slide