From YAML to TypeScript: Developer’s View on Cloud Automation Mikhail Shilkov – Software engineer @ Pulumi 24-25-26 Novembre, 2020

Why Infrastructure as Code General- purpose Languages

Cloud Automation

Unified Management API

REST calls Content-Type: application/x-www-form-urlencoded; charset=UTF-8 X-Amz-Date: 20130813T150211Z Host: ec2.amazonaws.com Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20130813/us-east- 1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz- date, Signature=ced6826de92d2bdeed8f846f0bf508e8559e98e4b0194b84example54174deb456c http://ec2.amazonaws.com/?Action=RunInstances ImageId=ami-2bb65342 &MaxCount=3 &MinCount=1 &Monitoring.Enabled=true &Placement.AvailabilityZone=us-east-1a &Version=2016-11-15

var launchRequest = new RunInstancesRequest() { ImageId = amiID, InstanceType = "t1.micro", MinCount = 1, MaxCount = 1, KeyName = keyPairName, SecurityGroupIds = groups }; var launchResponse = ec2Client.RunInstances(launchRequest); var instances = launchResponse.Reservation.Instances; SDKs to call those REST endpoints

az storage account create \ --location westus \ --name samplesa \ --resource-group myrg \ --sku LRS Command-line Tools

Step-by-step provisioning process Depend on both the current and the target state How to handle failures? Imperative Complex Error-prone Scripting: Challenges

Desired State Configuration

Desired State Configuration Target Current Tool

Managing Resource Graphs Target Current Tool

Be able to describe the desired state in full, concisely, and correctly Determine the correct order of creation and destruction Preview the changes before running the deployment Expressiveness Dependencies Transparency Challenges

AWS CloudFormation Azure Resource Manager Google Cloud Deployment Manager Desired State Provisioning

Markup Languages YET ANOTHER MARKUP LANGUAGE

Azure ARM Templates: JSON "resources": [ { "apiVersion": "2016-01-01", "type": "Microsoft.Storage/storageAccounts", "name": "mystorageaccount", "location": "westus", "sku": { "name": "Standard_LRS" }, "kind": "Storage", "properties": { } } ]

AWS CloudFormation: YAML Resources: S3BucketForURLs: Type: "AWS::S3::Bucket" DeletionPolicy: Delete Properties: BucketName: !If [ "CreateNewBucket", !Ref "AWS::NoValue", !Ref S3BucketName ] WebsiteConfiguration: IndexDocument: "index.html" LifecycleConfiguration: Rules: - Id: DisposeShortUrls ExpirationInDays: !Ref URLExpiration Prefix: "u" Status: Enabled

resource "azurerm_storage_account" "sa" { name = "mysa" location = azurerm_resource_group.rg.location resource_group_name = azurerm_resource_group.rg.name account_tier = "Standard" account_replication_type = "LRS" } Terraform: DSL (HashiCorp Configuration Language)

Lack of fast feedback about errors Which options need to be filled and which values are possible Thousands of lines of markup are hard to write, read, and navigate Tooling Discovery Size Markup Authoring Experience

Example: Code to infrastructure ratio KV Store Table “URLs” Function “Add URL” Function “Open URL” Lines of code Function Code ~20 CloudFormation ~240 ARM Templates ~160 Terraform ~220 Estimates based on public examples

{ "Resources": { "InstanceSecurityGroup": { "Type": "AWS::EC2::SecurityGroup", "Properties": { "SecurityGroupIngress": [ ... ] } }, "EC2Instance": { "Type": "AWS::EC2::Instance", "Properties": { "InstanceType": "t2.micro", "SecurityGroups": [ { "Ref": "InstanceSecurityGroup" } ], "ImageId": "ami-0080e4c5bc078760e" } } } } References Example for AWS CloudFormation

{ "condition": "[equals(parameters('production'), 'Yes')]", "type": "Microsoft.Compute/availabilitySets", "apiVersion": "2017-03-30", "name": "[variables('availabilitySetName')]", "location": "[resourceGroup().location]", "properties": { "platformFaultDomainCount": 2, "platformUpdateDomainCount": 3 } } Conditionals Example for Azure RM Templates

locals { files = ["index.php","main.php"] } resource "aws_s3_bucket_object" "files" { count = "${length(local.files)}" key = "${local.files[count.index]}" bucket = "${aws_s3_bucket.examplebucket.id}" source = "./src/${local.files[count.index]}" } Loops Example for Terraform

Different shape of infrastructure for different environments Bundling several related resources together for a higher-level component Sharing libraries of components within the company or in community Configuration Abstraction Packages Reusability

Test a single component against mocks Inspect the resulting resource tree before deployment Test the result of a deployment Unit Integration End-to-end Testability

I think we solved these problems before?

IDEs Code completion Unit tests Types Functions Package managers Versioning Classes Code reviews Contracts SDKs Workflows Refactoring

Pulumi Program the cloud

General-Purpose Programming Languages Node.js Go Python .NET

const resourceGroup = new azure.core.ResourceGroup("rg"); const storageAccount = new azure.storage.Account("storage", { resourceGroupName: resourceGroup.name, accountReplicationType: "LRS", accountTier: "Standard", }); TypeScript example

const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); Example: Serverless HTTP

const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); Example: Serverless HTTP

const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); Example: Serverless HTTP

const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); Still, Desired State!

CLI drives the deployment How Pulumi works CLI and Engine Last deployed state index.ts Language host AWS Azure GCP Kubernetes new Resource() Create, update, delete

• AWS • Azure • GCP • Digital Ocean • Cloudflare … and more Providers • Docker • Kubernetes • OpenStack • PostgreSQL • New Relic

Familiar language, experience, toolchain, packages Conditionals, loops, functions, classes, components, modules Reusable components that encapsulate complex logic and provide the right level of abstraction Developer-friendly Logic Abstractions Benefits of Real Code

Auto- completion

Compile Errors 38

for (const az of aws.getAvailabilityZones().names) { const server = new aws.ec2.Instance(`web-${az}`, { instanceType: "t2.micro", securityGroups: [ group.id ], ami: "ami-0080e4c5bc078760e", availabilityZone: az, }); } Conditionals, loops

class AksCluster extends pulumi.ComponentResource { constructor(name: string, args: AksClusterArgs, opts: pulumi.ComponentResourceOptions = {}) { super("examples:aks:AksCluster", name, args, opts); const password = new random.RandomPassword("password"/*, {...}*/); const sshPublicKey = new tls.PrivateKey("key"/*, {...}*/); const adApp = new azuread.Application("aks"/*, {...}*/); const adSp = new azuread.ServicePrincipal("aksSp"/*, {...}*/); const vnet = new azure.network.VirtualNetwork("vnet"/*, {...}*/); const subnet = new azure.network.Subnet("subnet"/*, {...}*/); const cluster = new azure.containerservice.KubernetesCluster("aksCluster"/*, {...}*/) } } Components

Conclusions

Fine-grained components Better Automation

43 Infrastructure is a part of your applications. Use your developer skills to build it.

@MikhailShilkov https://mikhail.io