Code PaLOUsa 2020 - TF v Pulumi

Transcript

1. IaC 2 WAYS: TF v PULUMI Code PaLOUsa 2020 August 21 - Kevin Tinn
2. • Cloud Application Architect Practice Lead at World Wide Technology • Over a decade of experience in the industry • Software Development • .NET (C# and VB.NET) • JVM (Scala) • JavaScript • Data Streaming Architectures • Kafka, Kinesis, Event Hubs • Application Architecture • AWS, Azure, and on-prem solutions • Incessant traveler with a new-found skiing sailing addiction • I Live in Denver, by way of St Louis, and grew up in TN INTRO: KEVIN TINN 2
3. • IaC overview • Intro to Terraform and first deployment • Overview of IaaS Stack components • Complex Terraform stack overview & deployment • Pulumi overview • Pulumi deployment & tf2pulumi • Picking your tool AGENDA 3
4. • We’ll be working from a github repo, lots of good info in the README • https://github.com/kevasync/aws-meetup-group- terraform/blob/master/README.md • Session Link: • https://app.hopin.to/events/cpl20/sessions/b926a36a-3a04-429e-a6fd- 13004d6900db REPO INFO 4
5. • Declarative representation of known good state of resources • Repeatable process allows for high-fidelity duplication of production environments for testing and development purposes with minimal efforts • In the event of non-trivial environment issues, redeployment of resources to known good state is far more timely than troubleshooting • Not that you shouldn’t troubleshoot, as discoveries can be folded into known good state that is deployed via coded infrastructure • Allows for source-controlled storage and CI/CD pipeline integration • Recap CloudFormation Meetup • https://github.com/kevasync/aws-meetup-group-cloud-formation-meetup BENEFITS OF CODED INFRA 5
6. • Open source IaC software tool created by Hashicorp • https://github.com/hashicorp/terraform • Released in 2014 and widely adopted by the industry • Enables definition and provisioning of datacenter and cloud infrastructure using high-level config language called Hashicorp Configuration Language (HCL) or JSON • Goal is to enable building, changing, and versioning infrastructure safely • Can be used to provision on many platforms • AWS, Azure, GCP, Alibaba Cloud, IBM Cloud (Bluemix), DigitalOcean, Linode, Oracle Cloud Infrastructure, OVH, VMware vSphere, OpenNebula, OpenStack INTRO TO TERRAFORM 6
7. • Two flavors of Terraform exist • Terraform Cloud • Terraform Enterprise • Terraform Cloud: Free to use SaaS (Software as a Service) offering • Keeps track of state for things that have been deployed • Terraform Enterprise: Self-hosted distribution of Terraform Cloud. It offers enterprises a private instance of the Terraform Cloud application, with no resource limits and with additional enterprise-grade architectural features like audit logging and SAML single sign-on. • For this lab, we will just be using the TF CLI, which stores state locally, thus we are not using either • In a production scenario you should not use the CLI locally to deploy • Tie your CI/CD into TF cloud to Enterprise FLAVORS OF TERRAFORM 7
8. • Provider: A provider is responsible for understanding API interactions and exposing resources. Providers generally are an IaaS (AWS, VMWare, Azure, GCP, Alibaba Cloud), PaaS (Heroku, GitHub), or SaaS services (Terraform Cloud, DataDog) • The combination of IaaS, PaaS, and SaaS offerings allows the creation so a complete tech stack • Repos • CI/CD pipelines • Cloud and Data center infrastructure • Serverless cloud components • List of providers: https://www.terraform.io/docs/providers/index.html • It should be noted that different providers can require different syntaxes for similar components, e.g. a VM • This results in vender lock-in TERRAFORM PROVIDERS 8
9. • Approach - 3 Phases: Write, Plan, Apply • Configuration files: Write - Describe to Terraform the components needed, from a single application to an entire datacenter • Execution plans: Plan - During its planning step, Terraform creates an execution plan that defines what it will do when applying configuration files • Resource Graph: Plan -Terraform builds a graph of all your resources, and parallelizes the creation and modification of any non-dependent resources • Change Automation: Apply - Changesets are created and applied to your environment based on the exec TERRAFORM APPROACH & COMPONENTS 9
10. • Install TF CLI • Using package mangers • Homebrew (Mac): brew install terraform • Chocolatey (Windows): choco install terraform • Manual downloads • https://www.terraform.io/downloads.html • There is also a “Download CLI” link to download on the terraform.io homepage • You’ll have to add the binary to your PATH or execute everything from the directory you download/unzip into INSTALL TERRAFORM CLI & CLONE REPO 10
11. • In the AWS Portal navigate to IAM > Users > New User • Name user and select programmatic access, add to security group with admin privs, and get user credentials CREATE AWS CLI ACCOUNT 11
12. • Install AWS CLI • Mac: • Homebrew: brew install awscli • Bundled installer: curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip" unzip awscli-bundle.zip sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws • Windows: MSI Installer https://docs.aws.amazon.com/cli/latest/userguide/install- windows.html#install-msi-on-windows INSTALL AWS CLI 12
13. • Type command: aws configure • Enter AWS Access Secret ID, Access Secret, default region, and format CONFIGURE LOCAL AWS CREDENTIALS 13 You can verify by viewing the file contents
14. • Let’s start from scratch, in this exercise we’ll create a VPC and subnet • After installing Terraform and configuring AWS credentials, create a directory called tf-helloworld • In it create a file called stack.tf • Add contents from next slide and save • Also available in README from https://github.com/kevasync/aws-meetup-group- terraform HELLO WORLD – TERRAFORM-STYLE 14
15. provider "aws" { version = "~> 2.0" region = "us-east-1" } resource "aws_vpc" "main" { cidr_block = "192.168.225.0/24" enable_dns_support = true enable_dns_hostnames = true tags = { CostCenter = "tf-helloworld" } } resource "aws_subnet" "main-subnet" { vpc_id = "${aws_vpc.main.id}" cidr_block = "192.168.225.0/25" availability_zone = "us-east-1a" tags = { CostCenter = "tf-helloworld" } } HELLO WORLD – TF 15
16. • Initialize the project: terraform init • Get a preview of what will be created: terraform plan • Validate that terraform will create a VPC and a subnet • Deploy the resources: terraform apply • View deployed resources: terraform show HELLO WORLD – TF 16
17. • The first command that must be executed is terraform init • must be in directory with .tf files TERRAFORM INIT 17
18. • Execute is terraform plan to see what changes TF will need to make in order to create or modify the resources specified in your configuration files. Here is a simple example for a VPC TERRAFORM PLAN 18
19. • Execute is terraform apply and enter yes at the prompt to provision resources TERRAFORM APPLY 19
20. • Execute is terraform show to see details about what has been deployed TERRAFORM SHOW 20
21. • In AWS portal navigate to Resource Groups > Tag Editor and search for CostCenter: tf-helloworld to see your VPC and subnet: VIEW RESOURCES VIA TAG 21
22. • Congrats, you’ve created your first resources with Terraform • Clean up the resources: terraform destroy • Check out the Terraform docs for examples of how to create other AWS resources: https://www.terraform.io/docs/providers/aws/ HELLO WORLD – WRAP UP 22
23. • Clone AWS Meetup Group repo: • git clone https://github.com/kevasync/aws-meetup-group- terraform.git • Can also be manually downloaded from https://github.com/kevasync/aws- meetup-group-terraform • Files • vars.tf: Contains variables used across all .tf files • provider.tf: Containes provider(s) used across all .tf files • network.tf: Defines networking resources • ec2.tf: Defines EC2 web hosting resources • rds.tf: Defines RDS database resources • Terraform looks for all files with .tf in the directory the Terraform commands are within, e.g. init, plan, apply, show OVERVIEW OF IaaS HOSTING TF STACK 23
24. IAAS HOSTING ARCHITECTURE 24
25. • VPC: Virtual Private Cloud - lets you provision a logically isolated section of the AWS Cloud where you can launch AWS resources in a virtual network that you define • Public Subnet: Subnet in VPC that allows access from the internet to an EC2 instance • Route Table: Controls where network traffic is directed, used to direct traffic to the internet gateway • Internet Gateway: Allows traffic in and out of VPC • Route Table Association: Enforces routes in route table upon VPC • VPC Default Route: Route that undefined traffic is subjected to, used to route egress traffic out of VPC NETWORKING COMPONENTS IN STACK 25
26. • EC2 Instance: Virtual Machine running Linux used to host web server • EC2 Instance Security Group: Defines what source/destination IPs/ports communication to/from EC2 can occur EC2 COMPONENTS IN STACK 26
27. • RDS Database: Relational Database Service - PaaS database instance • RDS Subnet 1: Subnet that primary RDS database resides in • RDS Subnet 2: Subnet that failover RDS database resides in; must be different Availability Zone than RDS Subnet 1 • RDS Subnet Group: Grouping of subnets used to implement high- availability of RDS Database • RDS Security Group: Defines what source/destination IPs/ports communication to/from RDS Database can occur RDS COMPONENTS IN STACK 27
28. • provider: This block is used to configure the named provider. Multiple providers can be used within a single file. As an example, the AWS Provider specifies which credentials to use and the region to create resources in: provider "aws" { profile = "default” region = "us-east-1” } • resource: This block defines a resource that exists within the infrastructure. A resource might be a physical component such as an EC2 instance, or it can be a logical resource such as a Heroku application. TF FILE BLOCKS 28
29. • data: A block that accesses a data source • Get latest Amazon Machine Image (AMI) Id for a specific kind of image • In our case, the latest ID for the AMZN provided Linux image • Each data resource is associated with a single data source, which determines the kind of object (or objects) it reads and what query constraint arguments are available. • Each data source in turn belongs to a provider, which is a plugin for Terraform that offers a collection of resource types and data sources that most often belong to a single cloud or on-premises infrastructure platform. TF FILE BLOCKS 29
30. • After cloning the repo, run • terraform init • terraform plan • terraform apply • Check out resources searching for CostCenter: aws-meetup-group tag • Delete the repo • terraform destroy DEPLOY RESOURCES FROM REPO 30
31. • “Pulumi is an open source platform for building and deploying cloud infrastructure and applications in your favorite language on any cloud” • Allows IaC using programming languages. Allowing developers to code infra without having to learn a new language • Also allows for better test-ability • Supported Languages • JavaScript • Typescript • Python • .NET Core • Go PULUMI OVERVIEW 31
32. • Create Pulumi account • https://app.pulumi.com/account/tokens • Can create via github account, email, etc. • Create an access token and copy it CREATE PULUMI ACCOUNT 32
33. • Mac • brew install pulumi • Windows • choco install pulumi • @"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershe ll.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; iex ((New-Object System.Net.WebClient).DownloadString('https://get.pulum i.com/install.ps1'))" && SET "PATH=%PATH%;%USERPROFILE%\.pulumi\bin” • Install pulumi aws plugin pulumi plugin install resource aws 1.18.0 INSTALLING PULUMI 33
34. • Change directory into pulumi-iaas-hosting-stack • Initialize pulumi stack: pulumi stack init • Enter access token: • Install AWS pulumi module: npm install @pulumi/aws • Deploy resources: pulumi up INIT PULUMI AND DEPLOY 34
35. PULUMI DEPLOYMENT 35
36. • Destroy Pulumi resources pulumi destroy • Delete the stack: pulumi stack rm <account>/<stack>/<env> --force • Be careful, if you delete the stack without destroying the resources they will be orphaned, and you will have to delete resources manually • Hooray tags! DESTROY RESOURCES & DELETE PULUMI STACK 36
37. • Not part of hands-on lab, but pretty sweet so let’s take a look • Requires Go and dep to be installed • brew install go • brew install dep • Install tf2pulumi • Get binary from https://github.com/pulumi/tf2pulumi/releases • Get darwin version for Mac • Unzip and add this to location included in your path, or put it in your root (~) • cp ~/Downloads/tf2pulumi ~/ • chmod +x ~/tf2pulumi CONVERTING TF TO PULUMI WITH TF2PULUMI 37
38. • Create a Pulumi stack and convert from existing TF configuration files • pulumi new typescript --dir pulumi-iaas-hosting-stack • If not logged in, enter pulumi access token created earlier • Enter project name: pulumi-iaas-hosting-stack • Enter project description: AWS IaaS/RDS Hosting Stack • Enter stack name: dev CONVERTING TO PULUMI 38
39. • Requires configuration to be TF 0.11.* compliance • Quotes around variable data types • String interpolation required for access of variables • “${varName}” • Use tfenv to be able to switch between version of TF • brew uninstall terraform • brew install tfenv • tfenv install 0.11.14 • tfenv install 0.12.19 • tfenv use 0.11.14 • Convert Terraform to pulumi typscrit file • ~/tf2pulumi >pulumi-iaas-hosting-stack/index.ts TF2PULUMI 39
40. • Get AWS Pulumi package: npm install @pulumi/aws • Update line 11 to strongly type InstnaceType: const instance_type = (config.get("instance_type") || "t2.small") as aws.ec2.InstanceType; • Set Pulumi region: pulumi config set aws:region us- east-1 • Create Pulumi stack: pulumi up • Update Pulumi stack: pulumi update DEPLOY WITH PULUMI AFTER CONVERSION 40
41. PULUMI UP OUTPUT 41
42. Destroy Pulumi stack: pulumi detroy DESTROY PULUMI STACK RESOURCES 42
43. • Both are awesome tools, so let’s start with your role • Do you come from a dev background or an admin/ops bg? • Find admin/ops folks tend to gravitate towards TF • Devs tend to prefer Pulumi • IDE “go to source” provides ability to dig into code and use it as docs • Do you have on-prem components to deploy? • Do you have requirements that requires a non cloud-based management portal PICKING YOUR TOOL 43
44. • In this session, we have covered: • Intro to Terraform • Getting started with Terraform • Deploying a complex, multi-file Terraform stack • Intro to Pulumi • Deploying a complex stack with Pulumi • Converting a Terraform project to Pulumi using tf2pulumi • Thank you for joining • Thanks to Code PaLOUsa for organizing!! • Please come talk to me if you have further questions CONCLUSION 44