Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Taming Infrastructure Workflow at Scale with Terraform

Taming Infrastructure Workflow at Scale with Terraform

Today’s cloud infrastructure is really complex. What if you could truly make this infrastructure a black box? What if you could mutate this infrastructure safely and easily? See how to use Terraform efficiently across hundreds of developers leveraging infrastructure as code and Terraform modules.

As more operations choices are added to your data center, whether through company acquisitions, a growing development team, or general technical debt, managing infrastructure complexity becomes a nightmare. Yet the end goal is still the same — safely deploy your application to your infrastructure. We need to tame our data centers by managing change across systems, enforcing policies, and by establishing a workflow for both developers and operations engineers to build in a collaborative environment.

This talk will discuss the problems faced in managing a modern cloud infrastructure, and how a set of innovative open source tools like Terraform can be used to tame the rising complexity curve. Terraform builds on years of research on graph theory to model the relationships between infrastructure resources so operators can safely manage and change infrastructure resources across bare metal, IaaS, PasS, and SaaS. Terraform models and potentially prevents that simple change with unforeseen consequences so operators don't have to.

Join me as I take you on a journey of using Terraform as we take control of our cloud infrastructure. This goal of this demo driven talk is to showcase how Terraform can help build multi-tier application infrastructure supporting multiple cloud platforms and services.

This talk was given at NDC Oslo 2018. For more details visit: https://ndcoslo.com/talk/taming-infrastructure-workflow-at-scale/

Github repository for the live demo done during the talk: https://github.com/anubhavmishra/terraform-workflow-examples

Anubhav Mishra

June 13, 2018
Tweet

More Decks by Anubhav Mishra

Other Decks in Technology

Transcript

  1. PROVISION, SECURE AND RUN ANY INFRASTRUCTURE Nomad Consul Vault Vagrant

    Packer Terraform Consul Enterprise Terraform Enterprise Vault Enterprise PRODUCT SUITE OSS TOOL SUITE RUN Applications SECURE Application Infrastructure PROVISION Infrastructure FOR INDIVIDUALS FOR TEAMS Nomad Enterprise
  2. $ ssh [email protected]
 $ sudo apt-get install default-jre $ echo

    “the oracle way” $ sudo add-apt-repository ppa:webupd8team/java $ sudo apt-get update $ sudo apt-get install oracle-java8-installer $ ./setup-java.sh Terminal
  3. $ aws usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]

    To see help text, you can run: aws help aws <command> help aws <command> <subcommand> help Terminal
  4. Copyright © 2017 HashiCorp @anubhavm  Why Terraform? !38 How

    Do We Provision Resources? • Compute (Cloud Servers) • Network (VPCs, ACLs, Firewalls) • Storage (Databases, Object Stores)
  5. Copyright © 2017 HashiCorp @anubhavm  Why Terraform? !39 How

    Do We Manage the Lifecycle of Resources? • Compute (Cloud Servers) • Network (VPCs, ACLs, Firewalls) • Storage (Databases, Object Stores)
  6. Copyright © 2017 HashiCorp @anubhavm  Why Terraform? !40 How

    Do We Enforce Policies Across the Resources? • Compute (Cloud Servers) • Network (VPCs, ACLs, Firewalls) • Storage (Databases, Object Stores)
  7. Copyright © 2017 HashiCorp @anubhavm  Why Terraform? !41 How

    Do We Automate and Share the Configurations? • Compute (Cloud Servers) • Network (VPCs, ACLs, Firewalls) • Storage (Databases, Object Stores)
  8. Copyright © 2017 HashiCorp @anubhavm  Goal !43 Provide a

    Unified Workflow Using Infrastructure as Code to Iterate Over Infrastructure Safely that is Capable of Provisioning Anything Anywhere.
  9. Copyright © 2017 HashiCorp @anubhavm  !45 Terminal resource "azurerm_virtual_machine"

    "web" { storage_image_reference { publisher = "Canonical" offer = "UbuntuServer" sku = "16.04-LTS" version = "latest" } } resource "aws_instance" "web" { ami = "ami-b123125" } main.tf
  10. Copyright © 2017 HashiCorp @anubhavm  !46 Terminal resource "azurerm_public_ip"

    "web" { name = "webserver-ip" location = "West US" resource_group_name = "webserver-rg" public_ip_address_allocation = "static" } resource "dnsimple_record" "hello" { domain = "example.com" name = "test" value = "${azurerm_public_ip.web.ip_address}" type = "A" } main.tf
  11. Copyright © 2017 HashiCorp @anubhavm  !47 Terminal resource "azurerm_public_ip"

    "web" { name = "webserver-ip" location = "West US" resource_group_name = "webserver-rg" public_ip_address_allocation = "static" } resource "dnsimple_record" "hello" { domain = "example.com" name = "test" value = "${azurerm_public_ip.web.ip_address}" type = "A" } main.tf
  12. Copyright © 2017 HashiCorp @anubhavm  !48 Terminal resource "cloudflare_page_rule"

    "www" { # ... } resource "fastly_service_v1" "myservice" { name = "myawesometestservice" # ... } main.tf
  13. Copyright © 2017 HashiCorp @anubhavm  !49 Terminal resource "github_membership"

    "membership_for_some_user" { username = "admin" role = "owner" } resource "gitlab_project" "example" { name = "example" description = "My awesome codebase" visibility_level = "public" } main.tf
  14. Copyright © 2017 HashiCorp @anubhavm  !59 Terminal resource "azurerm_public_ip"

    "web" { resource_group_name = "webserver-rg" public_ip_address_allocation = "static" } resource "azurerm_network_interface" "web" { resource_group_name = "webserver-rg" ip_configuration { ..... public_ip_address_id = "${azurerm_public_ip.web.id}" } } resource "azurerm_virtual_machine" "web" { count = 1 network_interface_ids = ["${azurerm_network_interface.web.id}"] storage_image_reference { offer = "UbuntuServer" ..... } } resource "dnsimple_record" "web" { domain = "example.com" name = "webs" value = "${azurerm_public_ip.web.ip_address}" type = "A" } main.tf
  15. Copyright © 2017 HashiCorp @anubhavm  !60 Terminal resource "azurerm_public_ip"

    "web" { resource_group_name = "webserver-rg" public_ip_address_allocation = "static" } resource "azurerm_network_interface" "web" { resource_group_name = "webserver-rg" ip_configuration { ..... public_ip_address_id = "${azurerm_public_ip.web.id}" } } resource "azurerm_virtual_machine" "web" { count = 1 network_interface_ids = ["${azurerm_network_interface.web.id}"] storage_image_reference { offer = "UbuntuServer" ..... } } resource "dnsimple_record" "web" { domain = "example.com" name = "webs" value = "${azurerm_public_ip.web.ip_address}" type = "A" } main.tf
  16. Copyright © 2017 HashiCorp @anubhavm  !62 “Shows You What

    Will Happen Before It Actually Happens” TERRAFORM PLAN Anubhav Mishra NDC Oslo 2018
  17. Copyright © 2017 HashiCorp @anubhavm  !63 Terminal + azurerm_public_ip.web

    ..... + azurerm_network_interface.web ..... + azurerm_virtual_machine.webserver ..... + dnsimple_record.hello id: <computed> domain: "example.com" domain_id: <computed> hostname: <computed> name: "test" priority: <computed> ttl: "3600" type: "A" value: "${azurerm_public_ip.web.ip_address}" shell
  18. Copyright © 2017 HashiCorp @anubhavm  !64 Terminal + azurerm_public_ip.web

    ..... + azurerm_network_interface.web ..... + azurerm_virtual_machine.webserver ..... + dnsimple_record.hello id: <computed> domain: "example.com" domain_id: <computed> hostname: <computed> name: "test" priority: <computed> ttl: "3600" type: "A" value: "${azurerm_public_ip.web.ip_address}" shell
  19. Copyright © 2017 HashiCorp @anubhavm  !65 Terminal + azurerm_public_ip.web

    ..... + azurerm_network_interface.web ..... + azurerm_virtual_machine.webserver ..... + dnsimple_record.hello id: <computed> domain: "example.com" domain_id: <computed> hostname: <computed> name: "test" priority: <computed> ttl: "3600" type: "A" value: "${azurerm_public_ip.web.ip_address}" shell
  20. Copyright © 2017 HashiCorp @anubhavm  !66 Terminal + azurerm_public_ip.web

    ..... + azurerm_network_interface.web ..... + azurerm_virtual_machine.webserver ..... + dnsimple_record.hello id: <computed> domain: "example.com" domain_id: <computed> hostname: <computed> name: "test" priority: <computed> ttl: "3600" type: "A" value: "${azurerm_public_ip.web.ip_address}" shell
  21. Copyright © 2017 HashiCorp @anubhavm  !71 Map of Real

    World Resources to Your Configuration. TERRAFORM STATE
  22. Copyright © 2017 HashiCorp @anubhavm  !72 Terminal { "version":

    3, "terraform_version": "0.11.5", "serial": 4, "lineage": "af985fb6-6e75-66bc-984a-7635ea4249c7", "modules": [ { "path": [ "root" ], "outputs": {}, "resources": { "azurerm_resource_group.default": { "type": "azurerm_resource_group", "depends_on": [], "primary": { "id": "", }, }, "deposed": [], "provider": "provider.azurerm" } }, "depends_on": [] }, terraform.tfstate
  23. Copyright © 2017 HashiCorp @anubhavm  !73 Terminal { "version":

    3, "terraform_version": "0.11.5", "serial": 4, "lineage": "af985fb6-6e75-66bc-984a-7635ea4249c7", "modules": [ { "path": [ "root" ], "outputs": {}, "resources": { "azurerm_resource_group.default": { "type": "azurerm_resource_group", "depends_on": [], "primary": { "id": "", }, }, "deposed": [], "provider": "provider.azurerm" } }, "depends_on": [] }, terraform.tfstate
  24. Copyright © 2017 HashiCorp @anubhavm  !73 Terminal { "version":

    3, "terraform_version": "0.11.5", "serial": 4, "lineage": "af985fb6-6e75-66bc-984a-7635ea4249c7", "modules": [ { "path": [ "root" ], "outputs": {}, "resources": { "azurerm_resource_group.default": { "type": "azurerm_resource_group", "depends_on": [], "primary": { "id": "", }, }, "deposed": [], "provider": "provider.azurerm" } }, "depends_on": [] }, terraform.tfstate
  25. Copyright © 2017 HashiCorp @anubhavm  !73 Terminal { "version":

    3, "terraform_version": "0.11.5", "serial": 4, "lineage": "af985fb6-6e75-66bc-984a-7635ea4249c7", "modules": [ { "path": [ "root" ], "outputs": {}, "resources": { "azurerm_resource_group.default": { "type": "azurerm_resource_group", "depends_on": [], "primary": { "id": "", }, }, "deposed": [], "provider": "provider.azurerm" } }, "depends_on": [] }, terraform.tfstate Storage: file, consul, azurerm, s3, gcs, etc.
  26. Copyright © 2017 HashiCorp @anubhavm  !82 Terminal resource "azurerm_public_ip"

    "web" { ..... } resource "azurerm_network_interface" "web" { ip_configuration { ..... public_ip_address_id = "${azurerm_public_ip.web.id}" } } resource "azurerm_virtual_machine" "web" { count = 1 network_interface_ids = ["${azurerm_network_interface.web.id}"] storage_image_reference { publisher = "Canonical" offer = "UbuntuServer" sku = "16.04-LTS" version = "latest" } } resource "dnsimple_record" "web" { domain = "example.com" name = "test" value = "${azurerm_public_ip.web.ip_address}" type = "A" } main.tf
  27. Copyright © 2017 HashiCorp @anubhavm  !83 Terminal resource "azurerm_public_ip"

    "web" { ..... } resource "azurerm_network_interface" "web" { ip_configuration { ..... public_ip_address_id = "${azurerm_public_ip.web.id}" } } resource "azurerm_virtual_machine" "web" { count = "${var.count}" network_interface_ids = ["${azurerm_network_interface.web.id}"] storage_image_reference { publisher = "Canonical" offer = "${var.os_name}" sku = "${var.sku}" version = "latest" } } resource "dnsimple_record" "web" { domain = "example.com" name = "test" value = "${azurerm_public_ip.web.ip_address}" type = "A" } main.tf
  28. Copyright © 2017 HashiCorp @anubhavm  !84 Terminal resource "azurerm_public_ip"

    "web" { ..... } resource "azurerm_network_interface" "web" { ip_configuration { ..... public_ip_address_id = "${azurerm_public_ip.web.id}" } } resource "azurerm_virtual_machine" "web" { count = "${var.count}" network_interface_ids = ["${azurerm_network_interface.web.id}"] storage_image_reference { publisher = "Canonical" offer = "${var.os_name}" sku = “${var.sku}" version = "latest" } } resource "dnsimple_record" "web" { domain = "example.com" name = "test" value = "${azurerm_public_ip.web.ip_address}" type = "A" } main.tf
  29. Copyright © 2017 HashiCorp @anubhavm  !85 Terminal module "webserver"

    { source = "mishracorp/webserver" count = 10 os_name = "UbuntuServer" } main.tf
  30. Copyright © 2017 HashiCorp @anubhavm  !87 Terminal module "webserver"

    { source = "mishracorp/webserver" count = 10 os_name = "UbuntuServer" } main.tf
  31. Copyright © 2017 HashiCorp @anubhavm  !88 Terminal module "webserver"

    { source = "mishracorp/webserver" count = 10 os_name = "UbuntuServer" } main.tf OPERATIONS ENGINEER
  32. Copyright © 2017 HashiCorp @anubhavm  !89 Terminal module "webserver"

    { source = "mishracorp/webserver" count = 10 os_name = "UbuntuServer" } main.tf OPERATIONS ENGINEER SOFTWARE ENGINEER
  33. Copyright © 2017 HashiCorp @anubhavm  !90 Terminal module "webserver"

    { source = "mishracorp/webserver" count = 10 os_name = "UbuntuServer" } main.tf OPERATIONS ENGINEER SOFTWARE ENGINEER WEBSERVERS