From YAML to TypeScript: Developer’s View on Cloud Automation

Modern cloud platforms offer amazing capabilities for application developers. Cloud apps utilize multiple services and consist of many components, so they are hard to manage without employing Infrastructure as Code. Traditional tools like CloudFormation, ARM templates, and Terraform use text-based formats, which tend to be tedious, repetitive, and cumbersome to reuse. What if instead of configuration files you could use a language like TypeScript? See how you can bring your favorite developer tools like code completion, types, components, and abstractions to cloud infrastructure definition.

Transcript

1. From YAML to TypeScript: Developer’s View on Cloud Automation Mikhail Shilkov – Software engineer @ Pulumi 24-25-26 Novembre, 2020
2. Why Infrastructure as Code General- purpose Languages
3. Cloud Automation
4. Unified Management API
5. REST calls Content-Type: application/x-www-form-urlencoded; charset=UTF-8 X-Amz-Date: 20130813T150211Z Host: ec2.amazonaws.com Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20130813/us-east- 1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz- date, Signature=ced6826de92d2bdeed8f846f0bf508e8559e98e4b0194b84example54174deb456c http://ec2.amazonaws.com/?Action=RunInstances ImageId=ami-2bb65342 &MaxCount=3 &MinCount=1 &Monitoring.Enabled=true &Placement.AvailabilityZone=us-east-1a &Version=2016-11-15
6. var launchRequest = new RunInstancesRequest() { ImageId = amiID, InstanceType = "t1.micro", MinCount = 1, MaxCount = 1, KeyName = keyPairName, SecurityGroupIds = groups }; var launchResponse = ec2Client.RunInstances(launchRequest); var instances = launchResponse.Reservation.Instances; SDKs to call those REST endpoints
7. az storage account create \ --location westus \ --name samplesa \ --resource-group myrg \ --sku LRS Command-line Tools
8. Step-by-step provisioning process Depend on both the current and the target state How to handle failures? Imperative Complex Error-prone Scripting: Challenges
9. Desired State Configuration
10. Desired State Configuration Target Current Tool
11. Managing Resource Graphs Target Current Tool
12. Be able to describe the desired state in full, concisely, and correctly Determine the correct order of creation and destruction Preview the changes before running the deployment Expressiveness Dependencies Transparency Challenges
13. AWS CloudFormation Azure Resource Manager Google Cloud Deployment Manager Desired State Provisioning
14. Markup Languages YET ANOTHER MARKUP LANGUAGE
15. Azure ARM Templates: JSON "resources": [ { "apiVersion": "2016-01-01", "type": "Microsoft.Storage/storageAccounts", "name": "mystorageaccount", "location": "westus", "sku": { "name": "Standard_LRS" }, "kind": "Storage", "properties": { } } ]
16. AWS CloudFormation: YAML Resources: S3BucketForURLs: Type: "AWS::S3::Bucket" DeletionPolicy: Delete Properties: BucketName: !If [ "CreateNewBucket", !Ref "AWS::NoValue", !Ref S3BucketName ] WebsiteConfiguration: IndexDocument: "index.html" LifecycleConfiguration: Rules: - Id: DisposeShortUrls ExpirationInDays: !Ref URLExpiration Prefix: "u" Status: Enabled
17. resource "azurerm_storage_account" "sa" { name = "mysa" location = azurerm_resource_group.rg.location resource_group_name = azurerm_resource_group.rg.name account_tier = "Standard" account_replication_type = "LRS" } Terraform: DSL (HashiCorp Configuration Language)
18. Lack of fast feedback about errors Which options need to be filled and which values are possible Thousands of lines of markup are hard to write, read, and navigate Tooling Discovery Size Markup Authoring Experience
19. Example: Code to infrastructure ratio KV Store Table “URLs” Function “Add URL” Function “Open URL” Lines of code Function Code ~20 CloudFormation ~240 ARM Templates ~160 Terraform ~220 Estimates based on public examples
20. { "Resources": { "InstanceSecurityGroup": { "Type": "AWS::EC2::SecurityGroup", "Properties": { "SecurityGroupIngress": [ ... ] } }, "EC2Instance": { "Type": "AWS::EC2::Instance", "Properties": { "InstanceType": "t2.micro", "SecurityGroups": [ { "Ref": "InstanceSecurityGroup" } ], "ImageId": "ami-0080e4c5bc078760e" } } } } References Example for AWS CloudFormation
21. { "condition": "[equals(parameters('production'), 'Yes')]", "type": "Microsoft.Compute/availabilitySets", "apiVersion": "2017-03-30", "name": "[variables('availabilitySetName')]", "location": "[resourceGroup().location]", "properties": { "platformFaultDomainCount": 2, "platformUpdateDomainCount": 3 } } Conditionals Example for Azure RM Templates
22. locals { files = ["index.php","main.php"] } resource "aws_s3_bucket_object" "files" { count = "${length(local.files)}" key = "${local.files[count.index]}" bucket = "${aws_s3_bucket.examplebucket.id}" source = "./src/${local.files[count.index]}" } Loops Example for Terraform
23. Different shape of infrastructure for different environments Bundling several related resources together for a higher-level component Sharing libraries of components within the company or in community Configuration Abstraction Packages Reusability
24. Test a single component against mocks Inspect the resulting resource tree before deployment Test the result of a deployment Unit Integration End-to-end Testability
25. I think we solved these problems before?
26. IDEs Code completion Unit tests Types Functions Package managers Versioning Classes Code reviews Contracts SDKs Workflows Refactoring
27. Pulumi Program the cloud
28. General-Purpose Programming Languages Node.js Go Python .NET
29. const resourceGroup = new azure.core.ResourceGroup("rg"); const storageAccount = new azure.storage.Account("storage", { resourceGroupName: resourceGroup.name, accountReplicationType: "LRS", accountTier: "Standard", }); TypeScript example
30. const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); Example: Serverless HTTP
31. const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); Example: Serverless HTTP
32. const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); Example: Serverless HTTP
33. const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); Still, Desired State!
34. CLI drives the deployment How Pulumi works CLI and Engine Last deployed state index.ts Language host AWS Azure GCP Kubernetes new Resource() Create, update, delete
35. • AWS • Azure • GCP • Digital Ocean • Cloudflare … and more Providers • Docker • Kubernetes • OpenStack • PostgreSQL • New Relic
36. Familiar language, experience, toolchain, packages Conditionals, loops, functions, classes, components, modules Reusable components that encapsulate complex logic and provide the right level of abstraction Developer-friendly Logic Abstractions Benefits of Real Code
37. Auto- completion
38. Compile Errors 38
39. for (const az of aws.getAvailabilityZones().names) { const server = new aws.ec2.Instance(`web-${az}`, { instanceType: "t2.micro", securityGroups: [ group.id ], ami: "ami-0080e4c5bc078760e", availabilityZone: az, }); } Conditionals, loops
40. class AksCluster extends pulumi.ComponentResource { constructor(name: string, args: AksClusterArgs, opts: pulumi.ComponentResourceOptions = {}) { super("examples:aks:AksCluster", name, args, opts); const password = new random.RandomPassword("password"/*, {...}*/); const sshPublicKey = new tls.PrivateKey("key"/*, {...}*/); const adApp = new azuread.Application("aks"/*, {...}*/); const adSp = new azuread.ServicePrincipal("aksSp"/*, {...}*/); const vnet = new azure.network.VirtualNetwork("vnet"/*, {...}*/); const subnet = new azure.network.Subnet("subnet"/*, {...}*/); const cluster = new azure.containerservice.KubernetesCluster("aksCluster"/*, {...}*/) } } Components
41. Conclusions
42. Fine-grained components Better Automation
43. 43 Infrastructure is a part of your applications. Use your developer skills to build it.
44. @MikhailShilkov https://mikhail.io