Build, provision & deploy in the Cloud with Packer, Ansible & Terraform – PHPUK 2018

More information about this Cloud automation talk I did at PHPUK Conference 2018 in London can be found on https://feryn.eu/speaking/build-provision-deploy-cloud-packer-ansible-phpuk-2018/

Transcript

1. By Thijs Feryn Build, provision & deploy in the Cloud with Packer, Ansible & Terraform PHP UK CONFERENCE
2. I’m so sick of Cloud!
4. Automation
5. Many platforms
8. ✓ Less lock-in ✓ Faster deployments ✓ Better scalability ✓ Better reproducibility ✓ Less human error ✓ Lower cost ✓ Operational stability Goals
9. Hi, I’m Thijs
10. I’m @ThijsFeryn on Twitter
11. I’m an Evangelist At
12. I’m an Evangelist At
14. 185 17 11 4
15. https://joind.in/talk/ae063
16. Build Provision Orchestrate
17. App code Bake image Build
19. App code Provision Bake image Pour “sauce” Build
21. App code Provision Orchestrate Bake image Pour “sauce” Serve Build
23. Infrastructure as code
24. Disposable stacks
25. Reproducible environments
27. ✓ Alicloud ECS ✓ Amazon EC2 ✓ Azure ✓ CloudStack ✓ DigitalOcean ✓ Docker ✓ File ✓ Google Cloud ✓ Hyper-V ✓ LXC ✓ LXD ✓ Null ✓ 1&1 ✓ OpenStack ✓ Oracle OCI ✓ Parallels ✓ ProfitBricks ✓ QEMU ✓ Triton ✓ VirtualBox ✓ VMware ✓ Custom Packer builders
29. { "builders": [ { "type": "openstack", "identity_endpoint": "https://osp.combell.com:5000/v3", "tenant_name": "", "username": "", "password": "", "image_name": "thijsferyn_{{isotime \"2006_01_02__03_04_05\"}}", "source_image": "1b9fe43e-4cf2-4986-b7f2-51f5706b120b", "ssh_username": "debian", "domain_name" : "default", "flavor": "m1.medium", "networks": ["a1175c54-cb12-436d-ac7b-d327499dc39b"], "floating_ip_pool" : "public" } ], "provisioners": [ { "type": "shell", "inline": ["apt-get update","apt-get install -y nginx"] } ] }
31. { "builders": [ { "type": "amazon-ebs", "access_key": "", "secret_key": "", "region": "eu-west-2", "instance_type": "t2.micro", "ssh_username": "admin", "associate_public_ip_address": true, "subnet_id": “subnet-11d3d56a", "source_ami": "ami-e1e8f085", "ami_name": "thijsferyn_{{isotime \"2006_01_02__03_04_05\"}}" } ], "provisioners": [ { "type": "shell", "inline": ["apt-get update","apt-get install -y nginx"] } ] }
32. $ packer build packer.json
33. Packer build Cloud API Boot VM using source image Take VM snapshot Image registry Store VM snapshot as new image Shell provisioning run Tear down VM
34. $ openstack image list +--------------------------------------+----------------------------------+--------+ | ID | Name | Status | +--------------------------------------+----------------------------------+--------+ | ef90ad1c-ed64-4ae0-8283-e446e95c33e6 | thijsferyn_2018_01_31__02_26_27 | active | | aa0a5244-0d57-477d-b985-653d6b881292 | Windows-2012-R2 | active | | 31e84bbb-abbb-4d5a-b620-ecebf4697d16 | centos-6-64bit | active | | 9eac74f9-6e6c-45a9-bd93-915d3e390687 | centos-7-64bit | active | | 3a69a110-775f-41ad-9d63-32079db57203 | cirros-0.3.4-64bit | active | | 244cbbde-77b6-4f1e-837f-9250520ee78b | coreos-stable | active | | 1b9fe43e-4cf2-4986-b7f2-51f5706b120b | debian-jessie-64bit | active | | 690b063b-f239-4032-ad23-9ae6337f248f | debian-stretch-64bit | active | | d8034a10-9b15-46b1-9c68-bd98f92d0ffb | ubuntu-server-precise-lts-64-bit | active | | 040f9d1d-bb18-4466-abd7-9f2172e6db70 | ubuntu-server-trusty-lts-64-bit | active | | 5be2b652-a965-4ecb-a2b4-b9f83d7779e6 | ubuntu-server-xenial-lts-64-bit | active | +--------------------------------------+----------------------------------+--------+
35. $ aws ec2 describe-images --owners=self --query "Images[*]. {ID:ImageId,Name:Name,Status:State}" --output=table +--------------+-----------------------------------+-------------+ | ID | Name | Status | +--------------+-----------------------------------+-------------+ | ami-82ccd6e6| thijsferyn_2018_01_31__02_26_27 | available | +--------------+-----------------------------------+-------------+
37. Configuration management system
39. ✓ Written in Python ✓ Agentless ✓ Standalone ✓ SSH-based ✓ Runs playbook (yml files) ✓ Groups playbook in roles ✓ Install software ✓ Configure your server Ansible
40. --- - hosts: all vars: code_folders: - "web" become: true tasks: - name: update apt cache apt: update_cache=yes cache_valid_time=86400 - name: install required packages apt: name={{ item }} state=present with_items: - vim - curl - nginx - php7.0-fpm - rsync - name: create /var/www/html/ folder file: path: /var/www/html/ state: directory mode: 0755 owner: www-data playbook.yml
41. - name: create /var/www/html/ folder file: path: /var/www/html/ state: directory mode: 0755 owner: www-data - name: Copy code synchronize: src: ../{{ item }} dest: /var/www/html delete: yes recursive: yes group: no owner: no perms: no with_items: "{{code_folders}}" - name: Chown code to www-data file: path: /var/www/html/{{ item }} group: www-data owner: www-data mode: u=rwx,g=r,o=r recurse: yes playbook.yml
42. Organize your playbooks
43. . !"" files # $"" db.sql !"" host_vars # $"" default.yml !"" group_vars # $"" all.yml !"" inventory !"" playbook.yml !"" roles # $"" my-role # !"" defaults # # $"" main.yml # !"" handlers # # $"" main.yml # !"" meta # # $"" main.yml # !"" tasks # # $"" main.yml # !"" templates # # $"" template.j2 # $"" vars # $"" vars.yml !"" tasks # !"" task1.yml # !"" task2.yml $"" templates $"" template.j2
44. --- - hosts: all tasks: - include_tasks: task1.yml - include_tasks: task2.yml roles: - my-role
45. $ ansible-playbook playbook.yml
46. { "builders": [ { "type": "openstack", "identity_endpoint": "https://osp.combell.com:5000/v3", "tenant_name": "18324-1", "username": "18324", "password": "", "image_name": "MyBuild-{{isotime \"2006-01-02 03:04:05\"}}", "source_image": "1b9fe43e-4cf2-4986-b7f2-51f5706b120b", "ssh_username": "debian", "domain_name" : "default", "flavor": "m1.medium", "networks": ["a1175c54-cb12-436d-ac7b-d327499dc39b"], "floating_ip_pool" : "public" } ], "provisioners": [ { "type": "ansible", "playbook_file": "./ansible/playbook.yml" } ] }
47. $ ansible-playbook --extra-vars packer_build_name=openstack packer_builder_type=openstack -i /var/folders/j7/ kpxjpxq92vs_t97gg6tqtttm0000gn/T/packer-provisioner- ansible972172074 /Users/thijsferyn/Dropbox/Sites/openstack/ packer-ansible-terraform/ansible/playbook.yml --private-key / var/folders/j7/kpxjpxq92vs_t97gg6tqtttm0000gn/T/ansible- key333617975
48. Packer build Cloud API Boot VM using source image Take VM snapshot Image registry Store VM snapshot as new image Ansible provisioning run Tear down VM
49. "image_name": "MyBuild-{{isotime \"2006-01-02 03:04:05\"}}", "source_image": "1b9fe43e-4cf2-4986-b7f2-51f5706b120b", "ssh_username": "debian", "domain_name" : "default", "flavor": "m1.medium", "networks": ["a1175c54-cb12-436d-ac7b-d327499dc39b"], "floating_ip_pool" : "public" } ], "provisioners": [ { "type": "ansible", "playbook_file": "./ansible/playbook.yml" } ], "post-processors": [ { "type": "manifest", "output": "manifest.json" } ] }
50. Store build results Packer build Cloud API Boot VM using source image Take VM snapshot Image registry Store VM snapshot as new image Ansible provisioning run Tear down VM manifest.json
51. { "builds": [ { "name": "amazon-ebs", "builder_type": "amazon-ebs", "build_time": 1517408990, "files": null, "artifact_id": "eu-west-2:ami-82ccd6e6", "packer_run_uuid": "ba98283c-08ca-a9fc-ab10-e49502ce4ab6" }, { "name": "openstack", "builder_type": "openstack", "build_time": 1509053610, "files": null, "artifact_id": "ef90ad1c-ed64-4ae0-8283-e446e95c33e6", "packer_run_uuid": "b6000539-f3e6-85dd-2733-9fb3ae443e9f" } ], "last_run_uuid": "b6000539-f3e6-85dd-2733-9fb3ae443e9f" }
52. $ cat manifest.json | \ jq '.builds[-1].artifact_id'
53. $ openstack image list +--------------------------------------+----------------------------------+--------+ | ID | Name | Status | +--------------------------------------+----------------------------------+--------+ | ef90ad1c-ed64-4ae0-8283-e446e95c33e6 | thijsferyn_2018_01_31__02_26_27 | active | | aa0a5244-0d57-477d-b985-653d6b881292 | Windows-2012-R2 | active | | 31e84bbb-abbb-4d5a-b620-ecebf4697d16 | centos-6-64bit | active | | 9eac74f9-6e6c-45a9-bd93-915d3e390687 | centos-7-64bit | active | | 3a69a110-775f-41ad-9d63-32079db57203 | cirros-0.3.4-64bit | active | | 244cbbde-77b6-4f1e-837f-9250520ee78b | coreos-stable | active | | 1b9fe43e-4cf2-4986-b7f2-51f5706b120b | debian-jessie-64bit | active | | 690b063b-f239-4032-ad23-9ae6337f248f | debian-stretch-64bit | active | | d8034a10-9b15-46b1-9c68-bd98f92d0ffb | ubuntu-server-precise-lts-64-bit | active | | 040f9d1d-bb18-4466-abd7-9f2172e6db70 | ubuntu-server-trusty-lts-64-bit | active | | 5be2b652-a965-4ecb-a2b4-b9f83d7779e6 | ubuntu-server-xenial-lts-64-bit | active | +--------------------------------------+----------------------------------+--------+
55. Deploy the VM image
56. Orchestration
57. ✓ Launch virtual network ✓ Boot up webservers ✓ Configure firewalls ✓ Assign loadbalancers ✓ Create DNS-records ✓ Assign SSH keys ✓ Create autoscaling group Orchestration
58. ✓ CloudFormation (AWS) ✓ Heat (OpenStack) ✓ Resource manager (Azure) Vendor-specific orchestration tools
60. ✓ Infrastructure As Code ✓ Plan, graph, execute ✓ State management ✓ Multiple providers ✓ Incremental changes ✓ Interacts with Cloud vendor APIs Terraform
61. ✓ Alicloud ✓ Archive ✓ Arukas ✓ AWS ✓ Bitbucket ✓ CenturyLinkCloud ✓ Chef ✓ Circonus ✓ Cloudflare ✓ CloudStack ✓ Cobbler ✓ Consul ✓ Datadog ✓ DigitalOcean ✓ DNS ✓ DNSMadeEasy ✓ DNSimple ✓ Docker ✓ Dyn ✓ External ✓ Fastly ✓ GitHub ✓ Gitlab ✓ Google Cloud ✓ Grafana ✓ Heroku ✓ HTTP ✓ Icinga2 ✓ Ignition ✓ InfluxDB ✓ Kubernetes ✓ Librato ✓ Local ✓ Logentries ✓ LogicMonitor ✓ Mailgun ✓ Microsoft Azure ✓ Microsoft Azure (Legacy ASM) ✓ MySQL ✓ New Relic ✓ Nomad ✓ NS1 ✓ 1&1 ✓ Oracle Public Cloud ✓ OpenStack ✓ OpsGenie ✓ OVH ✓ Packet ✓ PagerDuty ✓ PostgreSQL ✓ PowerDNS ✓ ProfitBricks ✓ RabbitMQ ✓ Rancher ✓ Random ✓ Rundeck ✓ Scaleway ✓ SoftLayer ✓ StatusCake ✓ Spotinst ✓ Template ✓ Terraform ✓ Terraform Enterprise ✓ TLS ✓ Triton ✓ UltraDNS ✓ Vault ✓ VMware vCloud Director ✓ VMware vSphere Terraform providers
62. main.tf
63. variable "ami" { type = "string" default = "ami-e1e8f085" #Stock Debian Stretch } resource "aws_instance" "web" { ami = "${var.ami}" instance_type = "t2.micro" security_groups = ["Web"] tags { Name = "web" } } data "aws_route53_zone" "web" { name = "aws.combell.com." } resource "aws_route53_record" "web" { zone_id = "${data.aws_route53_zone.web.zone_id}" name = "thijsferyn-web.${data.aws_route53_zone.web.name}" type = "A" ttl = "60" records = ["${aws_instance.web.public_ip}"] } output "public_ip" { value = "${aws_instance.web.public_ip}" } output "hostname" { value = "${aws_route53_record.web.name}" }
64. $ terraform init
65. $ terraform init Initializing provider plugins... - Checking for available provider plugins on https://releases.hashicorp.com... - Downloading plugin for provider "aws" (1.8.0)... The following providers do not have any version constraints in configuration, so the latest version was installed. To prevent automatic upgrades to new major versions that may contain breaking changes, it is recommended to add version = "..." constraints to the corresponding provider blocks in configuration, with the constraint strings suggested below. * provider.aws: version = "~> 1.8" Terraform has been successfully initialized!
66. $ terraform plan
67. $ terraform plan Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. data.aws_route53_zone.web: Refreshing state... ------------------------------------------------------------------------ An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols:
68. An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: + aws_instance.web id: <computed> ami: "ami-e1e8f085" associate_public_ip_address: <computed> availability_zone: <computed> ebs_block_device.#: <computed> ephemeral_block_device.#: <computed> instance_state: <computed> instance_type: "t2.micro" ipv6_address_count: <computed> ipv6_addresses.#: <computed> key_name: <computed> network_interface.#: <computed> network_interface_id: <computed> placement_group: <computed> primary_network_interface_id: <computed> private_dns: <computed> private_ip: <computed> public_dns: <computed> public_ip: <computed> root_block_device.#: <computed> security_groups.#: "1" security_groups.2661672386: "Web" source_dest_check: "true" subnet_id: <computed> tags.%: "1" tags.Name: "web" tenancy: <computed> volume_tags.%: <computed> vpc_security_group_ids.#: <computed>
69. + aws_route53_record.web id: <computed> fqdn: <computed> name: "thijsferyn-web.aws.combell.com" records.#: <computed> ttl: "60" type: "A" zone_id: "Z3K2HG3W48B0MR" Plan: 2 to add, 0 to change, 0 to destroy.
70. $ terraform apply
71. $ terraform plan -out=bla $ terraform apply bla
73. $ terraform output hostname = thijsferyn-web.aws.combell.com public_ip = 35.178.38.182
74. $ terraform output -json { "hostname": { "sensitive": false, "type": "string", "value": "thijsferyn-web.aws.combell.com" }, "public_ip": { "sensitive": false, "type": "string", "value": "35.178.38.182" } }
75. $ terraform show
76. $ terraform show aws_instance.web: id = i-0333e97ea6bdbc5ae ami = ami-e1e8f085 associate_public_ip_address = true availability_zone = eu-west-2a disable_api_termination = false ebs_block_device.# = 0 ebs_optimized = false ephemeral_block_device.# = 0 iam_instance_profile = instance_state = running instance_type = t2.micro ipv6_addresses.# = 0 key_name = monitoring = false network_interface.# = 0 network_interface_id = eni-c62e7193 placement_group = primary_network_interface_id = eni-c62e7193 private_dns = ip-172-31-19-7.eu-west-2.compute.internal
77. network_interface_id = eni-c62e7193 placement_group = primary_network_interface_id = eni-c62e7193 private_dns = ip-172-31-19-7.eu-west-2.compute.internal private_ip = 172.31.19.7 public_dns = ec2-35-178-38-182.eu-west-2.compute.amazonaws.com public_ip = 35.178.38.182 root_block_device.# = 1 root_block_device.0.delete_on_termination = true root_block_device.0.iops = 100 root_block_device.0.volume_id = vol-0b9364527ec09549e root_block_device.0.volume_size = 8 root_block_device.0.volume_type = gp2 security_groups.# = 1 security_groups.763657905 = Web source_dest_check = true subnet_id = subnet-11d3d56a tags.% = 1 tags.Name = web tenancy = default volume_tags.% = 0 vpc_security_group_ids.# = 0
78. aws_route53_record.web: id = Z3K2HG3W48B0MR_thijsferyn-web.aws.combell.com._A fqdn = thijsferyn-web.aws.combell.com health_check_id = name = thijsferyn-web.aws.combell.com records.# = 1 records.1088376871 = 35.178.38.182 set_identifier = ttl = 60 type = A zone_id = Z3K2HG3W48B0MR data.aws_route53_zone.web: id = Z3K2HG3W48B0MR caller_reference = F7596F57-9ADE-9B0D-8F53-53DF42E03DB3 comment = name = aws.combell.com. private_zone = false resource_record_set_count = 3 zone_id = Z3K2HG3W48B0MR Outputs: hostname = thijsferyn-web.aws.combell.com public_ip = 35.178.38.182
79. State
80. State backend Terraform Cloud API
81. terraform.tfstate terraform.tfstate.backup
82. JSON data
83. ✓ Artifactory ✓ Azurerm ✓ Consul ✓ Etcd ✓ Gcs ✓ HTTP ✓ Manta ✓ S3 ✓ Swift ✓ Terraform Enterprise Remote state
84. Changes
85. $ terraform plan -var 'ami=ami-20ccd644'
86. ~ update in-place -/+ destroy and then create replacement Terraform will perform the following actions: -/+ aws_instance.web (new resource required) id: "i-0333e97ea6bdbc5ae" => <computed> (forces new resource) ami: "ami-e1e8f085" => "ami-20ccd644" (forces new resource) associate_public_ip_address: "true" => <computed> availability_zone: "eu-west-2a" => <computed> ebs_block_device.#: "0" => <computed> ephemeral_block_device.#: "0" => <computed> instance_state: "running" => <computed> instance_type: "t2.micro" => "t2.micro" ipv6_address_count: "" => <computed> ipv6_addresses.#: "0" => <computed> key_name: "" => <computed> network_interface.#: "0" => <computed> network_interface_id: "eni-c62e7193" => <computed> placement_group: "" => <computed> primary_network_interface_id: "eni-c62e7193" => <computed> private_dns: "ip-172-31-19-7.eu-west-2.compute.internal" => <computed> private_ip: "172.31.19.7" => <computed>
87. key_name: "" => <computed> network_interface.#: "0" => <computed> network_interface_id: "eni-c62e7193" => <computed> placement_group: "" => <computed> primary_network_interface_id: "eni-c62e7193" => <computed> private_dns: "ip-172-31-19-7.eu-west-2.compute.internal" => <computed> private_ip: "172.31.19.7" => <computed> public_dns: "ec2-35-178-38-182.eu- west-2.compute.amazonaws.com" => <computed> public_ip: "35.178.38.182" => <computed> root_block_device.#: "1" => <computed> security_groups.#: "1" => "1" security_groups.763657905: "Web" => "Web" source_dest_check: "true" => "true" subnet_id: "subnet-11d3d56a" => <computed> tags.%: "1" => "1" tags.Name: "web" => "web" tenancy: "default" => <computed> volume_tags.%: "0" => <computed> vpc_security_group_ids.#: "0" => <computed> ~ aws_route53_record.web records.#: "" => <computed> Plan: 1 to add, 1 to change, 1 to destroy.
88. $ terraform apply -var 'ami=ami-20ccd644'
89. Downtime
90. High availability plan
91. Workspaces
92. One Terraform project Multiple workspaces State per workspace Multiple environments
93. Blue Green
94. Workspace 1 Workspace 2 Internet Entry point BLUE GREEN
95. $ terraform init $ terraform workspace new wsp1 $ terraform workspace new wsp2 $ terraform workspace select wsp1 $ terraform plan $ terraform apply $ terraform workspace select wsp2 $ terraform plan $ terraform apply
96. . !"" main.tf !"" terraform.tfstate !"" terraform.tfstate.backup $"" terraform.tfstate.d !"" wsp1 $"" wsp2
97. variable "ami" { type = "string" default = "ami-e1e8f085" #Stock Debian Stretch } resource "aws_instance" "web" { ami = "${var.ami}" instance_type = "t2.micro" security_groups = ["Web"] tags { Name = "web-${terraform.workspace}" } } output "public_ip" { value = "${aws_instance.web.public_ip}" } Remove DNS resource Interpolate workspace info
98. variable "ip" { type = "string" } data "aws_route53_zone" "web" { name = "aws.combell.com." } resource "aws_route53_record" "web" { zone_id = "${data.aws_route53_zone.web.zone_id}" name = "thijsferyn-web.${data.aws_route53_zone.web.name}" type = "A" ttl = "60" records = ["${var.ip}"] } output "hostname" { value = "${aws_route53_record.web.name}" } Separate Terraform project
99. $ terraform apply -var 'ip="35.176.171.168"'
100. DNS round robin
101. variable "ami" { default = "ami-e1e8f085" #Stock Debian Stretch } variable "instances" { default = "2" } resource "aws_instance" "web" { ami = "${var.ami}" instance_type = "t2.micro" security_groups = ["Web"] count = "${var.instances}" tags { Name = "web-${format("%02d", count.index+1)}-${terraform.workspace}" } } output "public_ip" { value = ["${aws_instance.web.*.public_ip}"] } Create multiple instances at once
102. variable "ip" { type = "list" } data "aws_route53_zone" "web" { name = "aws.combell.com." } resource "aws_route53_record" "web" { zone_id = "${data.aws_route53_zone.web.zone_id}" name = "thijsferyn-web.${data.aws_route53_zone.web.name}" type = "A" ttl = "60" records = ["${var.ip}"] } output "hostname" { value = "${aws_route53_record.web.name}" }
103. $ terraform apply -var 'ip=["35.176.171.168","35.177.208.38"]'
104. Rolling updates
105. Deployment B Internet Entry point Drain connections and gradually switch from deployment A to deployment B Deployment A ROLLING UPDATES
106. Autoscaling group
107. Custom AMI V P C ALB Private subnet 1 Private subnet 2 Public subnet 1 Public subnet 2 Internet gateway Route table Target group Internet Launch config Autoscaling group Metrics Security group
108. resource "aws_vpc" "thijsferyn_terraform" { cidr_block = "10.0.0.0/16" tags { Name = "thijsferyn_${terraform.workspace}_terraform" } } resource "aws_subnet" "thijsferyn_terraform_public" { availability_zone = "eu-west-2a" vpc_id = "${aws_vpc.thijsferyn_terraform.id}" cidr_block = "10.0.0.0/24" tags { Name = "thijsferyn_terraform_${terraform.workspace}_public" } } resource "aws_subnet" "thijsferyn_terraform_private" { availability_zone = "eu-west-2a" vpc_id = "${aws_vpc.thijsferyn_terraform.id}" cidr_block = "10.0.1.0/24" tags { Name = "thijsferyn_terraform_${terraform.workspace}_private" } } Virtual network
109. resource "aws_security_group" "thijsferyn_terraform" { name = "thijsferyn-terraform-${terraform.workspace}" description = "Allow HTTP(S) & SSH" vpc_id = "${aws_vpc.thijsferyn_terraform.id}" ingress { from_port = 80 to_port = 80 protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] } ingress { from_port = 443 to_port = 443 protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] } ingress { from_port = 22 to_port = 22 protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] } egress { from_port = 0 to_port = 0 protocol = "-1" cidr_blocks = ["0.0.0.0/0"] } tags { Name = "thijsferyn_terraform-${terraform.workspace}" } } Firewall
110. data "aws_ami" "thijsferyn_terraform" { most_recent = true filter { name = "name" values = ["thijsferyn_*"] } filter { name = "virtualization-type" values = ["hvm"] } owners = ["826481595599"] } resource "aws_launch_configuration" "thijsferyn_terraform" { image_id = "${data.aws_ami.thijsferyn_terraform.id}" instance_type = "t2.micro" security_groups = ["${aws_security_group.thijsferyn_terraform.id}"] lifecycle { create_before_destroy = true } } Prepare VMs
111. resource "aws_lb" "thijsferyn_terraform" { name = "thijsferyn-terraform-${terraform.workspace}" internal = false security_groups = ["${aws_security_group.thijsferyn_terraform.id}"] subnets = ["${aws_subnet.thijsferyn_terraform_public.*.id}","$ {aws_subnet.thijsferyn_terraform_public2.*.id}"] } resource "aws_lb_listener" "thijsferyn_terraform" { load_balancer_arn = "${aws_lb.thijsferyn_terraform.arn}" port = "80" protocol = "HTTP" default_action { target_group_arn = "${aws_lb_target_group.thijsferyn_terraform.arn}" type = "forward" } } resource "aws_lb_target_group" "thijsferyn_terraform" { name = "thijsferyn-terraform-${terraform.workspace}" port = 80 protocol = "HTTP" vpc_id = "${aws_vpc.thijsferyn_terraform.id}" } Load balancing
112. resource "aws_autoscaling_group" "thijsferyn_terraform" { availability_zones = ["eu-west-2a"] name = "thijsferyn-terraform-${terraform.workspace}-$ {aws_launch_configuration.thijsferyn_terraform.name}" max_size = 10 min_size = 3 launch_configuration = "${aws_launch_configuration.thijsferyn_terraform.name}" target_group_arns = ["${aws_lb_target_group.thijsferyn_terraform.id}"] health_check_grace_period = 10 health_check_type = "ELB" vpc_zone_identifier = ["${aws_subnet.thijsferyn_terraform_private.id}"] default_cooldown = 60 depends_on = ["aws_launch_configuration.thijsferyn_terraform"] lifecycle { create_before_destroy = true } tag { key = "Name" value = "thijsferyn-terraform-${terraform.workspace}-$ {aws_launch_configuration.thijsferyn_terraform.name}" propagate_at_launch = true } } Autoscaling group
113. resource "aws_autoscaling_policy" "thijsferyn_terraform_high" { name = "thijsferyn-terraform-${terraform.workspace}-high" scaling_adjustment = 1 adjustment_type = "ChangeInCapacity" cooldown = 10 autoscaling_group_name = "${aws_autoscaling_group.thijsferyn_terraform.name}" } resource "aws_autoscaling_policy" "thijsferyn_terraform_low" { name = "thijsferyn-terraform-${terraform.workspace}low" scaling_adjustment = -1 adjustment_type = "ChangeInCapacity" cooldown = 10 autoscaling_group_name = "${aws_autoscaling_group.thijsferyn_terraform.name}" } Scale up scale down
114. resource "aws_cloudwatch_metric_alarm" "thijsferyn_terraform_high" { alarm_name = "thijsferyn-terraform-${terraform.workspace}-high" comparison_operator = "GreaterThanOrEqualToThreshold" evaluation_periods = "1" metric_name = "RequestCountPerTarget" namespace = "AWS/ApplicationELB" period = "60" statistic = "Sum" threshold = "20" dimensions { TargetGroup = "${aws_lb_target_group.thijsferyn_terraform.name}" LoadBalancer = "${aws_lb.thijsferyn_terraform.name}" } alarm_description = "More than 20 requests per target causes scaleout" alarm_actions = ["${aws_autoscaling_policy.thijsferyn_terraform_high.arn}"] } High water mark
115. $ terraform plan $ terraform apply
116. $ terraform graph | dot -Tpng > graph.png
119. ✓ vars.tf ✓ output.tf ✓ secgroup.tf ✓ keypair.tf ✓ compute.tf ✓ network.tf ✓ provider.tf Organize Terraform files
120. Modules
121. !"" autoscaling # !"" README.MD # !"" main.tf # !"" outputs.tf # $"" variables.tf !"" dns # !"" README.MD # !"" main.tf # !"" outputs.tf # $"" variables.tf !"" launch # !"" README.MD # !"" main.tf # !"" outputs.tf # $"" variables.tf !"" loadbalancing # !"" README.MD # !"" main.tf # !"" outputs.tf # $"" variables.tf !"" main.tf !"" networking # !"" README.MD # !"" main.tf # !"" outputs.tf # $"" variables.tf
122. variable "zone_name" { default = "aws.combell.com." } variable "record_name" { default = "" } variable "records" { type = "list" } output "dns_endpoint" { value = "${aws_route53_record.thijsferyn_terraform_aws_combell_com.fqdn}" } data "aws_route53_zone" "thijsferyn_terraform_aws_combell_com" { name = "${var.zone_name}" } resource "aws_route53_record" "thijsferyn_terraform_aws_combell_com" { zone_id = "${data.aws_route53_zone.thijsferyn_terraform_aws_combell_com.zone_id}" name = "${var.record_name == "" ? "thijsferyn-terraform${terraform.workspace == "default" ? "" : "-${terraform.workspace}"}": var.record_name}.$ {data.aws_route53_zone.thijsferyn_terraform_aws_combell_com.name}" type = "CNAME" ttl = "60" records = ["${var.records}"] } Variables.tf module input args Output.tf module output Main.tf processing
123. module "dns" { source = "./dns" record_name = "www" records = ["bla.domain.com"] } output "dns_endpoint" { value = "${module.dns.dns_endpoint}" } Root Terraform file
124. module "networking" { source = "./networking" } module "loadbalancing" { source = "./loadbalancing" security_groups = ["${module.networking.security_group_id}"] vpc_id = "${module.networking.vpc_id}" subnets = ["${module.networking.subnet_public1_id}","${module.networking.subnet_public2_id}"] } module "launch" { source = "./launch" instance_type = "t2.small" security_groups = ["${module.networking.security_group_id}"] } module "autoscaling" { source = "./autoscaling" target_group = "${module.loadbalancing.target_group_name}" loadbalancer = "${module.loadbalancing.loadbalancer_name}" min_size = "4" subnets = ["${module.networking.subnet_private1_id}","${module.networking.subnet_private2_id}"] launch_configuration = "${module.launch.launch_configuration_name}" target_group_arn = "${module.loadbalancing.target_group_id}" } module "dns" { source = "./dns" records = ["${module.loadbalancing.dns_name}"] } output "dns_endpoint" { value = "${module.dns.dns_endpoint}" }
125. https://registry.terraform.io
127. module "consul" { source = "hashicorp/consul/aws" }
128. $ terraform init
129. Downloading modules... Initializing provider plugins... - Checking for available provider plugins on https://releases.hashicorp.com... - Downloading plugin for provider "aws" (1.9.0)... - Downloading plugin for provider "template" (1.0.0)... The following providers do not have any version constraints in configuration, so the latest version was installed. To prevent automatic upgrades to new major versions that may contain breaking changes, it is recommended to add version = "..." constraints to the corresponding provider blocks in configuration, with the constraint strings suggested below. * provider.aws: version = "~> 1.9" * provider.template: version = "~> 1.0" Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.
130. module “example" { source = "github.com/hashicorp/example" } module "consul" { source = "git::https://hashicorp.com/consul.git" } module "ami" { source = "git::ssh://git@github.com/owner/repo.git" }
131. ✓Local files ✓Terraform Registry ✓GitHub ✓Bitbucket ✓Generic Git Repo ✓Mercurial ✓HTTP ✓S3 buckets
132. Remote state
134. $ consul agent -data-dir=/tmp/consul \ -server -ui -dev
135. terraform { backend "consul" { address = "127.0.0.1:8500" path = "example/terraform_state" } }
136. $ terraform apply Error locking state: Error acquiring the state lock: resource temporarily unavailable Lock Info: ID: d2c36deb-32c0-ecc7-b1f1-10068c2ed93b Path: terraform.tfstate Operation: OperationTypeApply Who: thijsferyn@MacBook-Pro-van-Thijs.local Version: 0.10.7 Created: 2018-02-02 16:03:17.005327423 +0000 UTC Info: Terraform acquires a state lock to protect the state from being written by multiple users at the same time. Please resolve the issue above and try again. For most commands, you can disable locking with the "-lock=false" flag, but this is not recommended.
137. data "terraform_remote_state" "web" { backend = "consul" config { address = "127.0.0.1:8500" path = "thijsferyn/terraform/production" } } data "aws_route53_zone" "web" { name = "aws.combell.com." } resource "aws_route53_record" "web" { zone_id = "${data.aws_route53_zone.web.zone_id}" name = "thijsferyn-web.${data.aws_route53_zone.web.name}" type = "A" ttl = "60" records = ["${data.terraform_remote_state.web.public_ip}"] } output "hostname" { value = "${aws_route53_record.web.name}" }
138. Instead of using CLI variables, we can use remote state to communicate between Terraform projects
139. Hybrid Cloud?
140. https://feryn.eu https://twitter.com/ThijsFeryn https://instagram.com/ThijsFeryn
141. https://joind.in/talk/ae063