Slide 1

Slide 1 text

From YAML to TypeScript: Developer’s View on Cloud Automation Mikhail Shilkov – Software engineer @ Pulumi 24-25-26 Novembre, 2020

Slide 2

Slide 2 text

Why Infrastructure as Code General- purpose Languages

Slide 3

Slide 3 text

Cloud Automation

Slide 4

Slide 4 text

Unified Management API

Slide 5

Slide 5 text

REST calls Content-Type: application/x-www-form-urlencoded; charset=UTF-8 X-Amz-Date: 20130813T150211Z Host: ec2.amazonaws.com Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20130813/us-east- 1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz- date, Signature=ced6826de92d2bdeed8f846f0bf508e8559e98e4b0194b84example54174deb456c http://ec2.amazonaws.com/?Action=RunInstances ImageId=ami-2bb65342 &MaxCount=3 &MinCount=1 &Monitoring.Enabled=true &Placement.AvailabilityZone=us-east-1a &Version=2016-11-15

Slide 6

Slide 6 text

var launchRequest = new RunInstancesRequest() { ImageId = amiID, InstanceType = "t1.micro", MinCount = 1, MaxCount = 1, KeyName = keyPairName, SecurityGroupIds = groups }; var launchResponse = ec2Client.RunInstances(launchRequest); var instances = launchResponse.Reservation.Instances; SDKs to call those REST endpoints

Slide 7

Slide 7 text

az storage account create \ --location westus \ --name samplesa \ --resource-group myrg \ --sku LRS Command-line Tools

Slide 8

Slide 8 text

Step-by-step provisioning process Depend on both the current and the target state How to handle failures? Imperative Complex Error-prone Scripting: Challenges

Slide 9

Slide 9 text

Desired State Configuration

Slide 10

Slide 10 text

Desired State Configuration Target Current Tool

Slide 11

Slide 11 text

Managing Resource Graphs Target Current Tool

Slide 12

Slide 12 text

Be able to describe the desired state in full, concisely, and correctly Determine the correct order of creation and destruction Preview the changes before running the deployment Expressiveness Dependencies Transparency Challenges

Slide 13

Slide 13 text

AWS CloudFormation Azure Resource Manager Google Cloud Deployment Manager Desired State Provisioning

Slide 14

Slide 14 text

Markup Languages YET ANOTHER MARKUP LANGUAGE

Slide 15

Slide 15 text

Azure ARM Templates: JSON "resources": [ { "apiVersion": "2016-01-01", "type": "Microsoft.Storage/storageAccounts", "name": "mystorageaccount", "location": "westus", "sku": { "name": "Standard_LRS" }, "kind": "Storage", "properties": { } } ]

Slide 16

Slide 16 text

AWS CloudFormation: YAML Resources: S3BucketForURLs: Type: "AWS::S3::Bucket" DeletionPolicy: Delete Properties: BucketName: !If [ "CreateNewBucket", !Ref "AWS::NoValue", !Ref S3BucketName ] WebsiteConfiguration: IndexDocument: "index.html" LifecycleConfiguration: Rules: - Id: DisposeShortUrls ExpirationInDays: !Ref URLExpiration Prefix: "u" Status: Enabled

Slide 17

Slide 17 text

resource "azurerm_storage_account" "sa" { name = "mysa" location = azurerm_resource_group.rg.location resource_group_name = azurerm_resource_group.rg.name account_tier = "Standard" account_replication_type = "LRS" } Terraform: DSL (HashiCorp Configuration Language)

Slide 18

Slide 18 text

Lack of fast feedback about errors Which options need to be filled and which values are possible Thousands of lines of markup are hard to write, read, and navigate Tooling Discovery Size Markup Authoring Experience

Slide 19

Slide 19 text

Example: Code to infrastructure ratio KV Store Table “URLs” Function “Add URL” Function “Open URL” Lines of code Function Code ~20 CloudFormation ~240 ARM Templates ~160 Terraform ~220 Estimates based on public examples

Slide 20

Slide 20 text

{ "Resources": { "InstanceSecurityGroup": { "Type": "AWS::EC2::SecurityGroup", "Properties": { "SecurityGroupIngress": [ ... ] } }, "EC2Instance": { "Type": "AWS::EC2::Instance", "Properties": { "InstanceType": "t2.micro", "SecurityGroups": [ { "Ref": "InstanceSecurityGroup" } ], "ImageId": "ami-0080e4c5bc078760e" } } } } References Example for AWS CloudFormation

Slide 21

Slide 21 text

{ "condition": "[equals(parameters('production'), 'Yes')]", "type": "Microsoft.Compute/availabilitySets", "apiVersion": "2017-03-30", "name": "[variables('availabilitySetName')]", "location": "[resourceGroup().location]", "properties": { "platformFaultDomainCount": 2, "platformUpdateDomainCount": 3 } } Conditionals Example for Azure RM Templates

Slide 22

Slide 22 text

locals { files = ["index.php","main.php"] } resource "aws_s3_bucket_object" "files" { count = "${length(local.files)}" key = "${local.files[count.index]}" bucket = "${aws_s3_bucket.examplebucket.id}" source = "./src/${local.files[count.index]}" } Loops Example for Terraform

Slide 23

Slide 23 text

Different shape of infrastructure for different environments Bundling several related resources together for a higher-level component Sharing libraries of components within the company or in community Configuration Abstraction Packages Reusability

Slide 24

Slide 24 text

Test a single component against mocks Inspect the resulting resource tree before deployment Test the result of a deployment Unit Integration End-to-end Testability

Slide 25

Slide 25 text

I think we solved these problems before?

Slide 26

Slide 26 text

IDEs Code completion Unit tests Types Functions Package managers Versioning Classes Code reviews Contracts SDKs Workflows Refactoring

Slide 27

Slide 27 text

Pulumi Program the cloud

Slide 28

Slide 28 text

General-Purpose Programming Languages Node.js Go Python .NET

Slide 29

Slide 29 text

const resourceGroup = new azure.core.ResourceGroup("rg"); const storageAccount = new azure.storage.Account("storage", { resourceGroupName: resourceGroup.name, accountReplicationType: "LRS", accountTier: "Standard", }); TypeScript example

Slide 30

Slide 30 text

const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); Example: Serverless HTTP

Slide 31

Slide 31 text

const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); Example: Serverless HTTP

Slide 32

Slide 32 text

const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); Example: Serverless HTTP

Slide 33

Slide 33 text

const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); Still, Desired State!

Slide 34

Slide 34 text

CLI drives the deployment How Pulumi works CLI and Engine Last deployed state index.ts Language host AWS Azure GCP Kubernetes new Resource() Create, update, delete

Slide 35

Slide 35 text

• AWS • Azure • GCP • Digital Ocean • Cloudflare … and more Providers • Docker • Kubernetes • OpenStack • PostgreSQL • New Relic

Slide 36

Slide 36 text

Familiar language, experience, toolchain, packages Conditionals, loops, functions, classes, components, modules Reusable components that encapsulate complex logic and provide the right level of abstraction Developer-friendly Logic Abstractions Benefits of Real Code

Slide 37

Slide 37 text

Auto- completion

Slide 38

Slide 38 text

Compile Errors 38

Slide 39

Slide 39 text

for (const az of aws.getAvailabilityZones().names) { const server = new aws.ec2.Instance(`web-${az}`, { instanceType: "t2.micro", securityGroups: [ group.id ], ami: "ami-0080e4c5bc078760e", availabilityZone: az, }); } Conditionals, loops

Slide 40

Slide 40 text

class AksCluster extends pulumi.ComponentResource { constructor(name: string, args: AksClusterArgs, opts: pulumi.ComponentResourceOptions = {}) { super("examples:aks:AksCluster", name, args, opts); const password = new random.RandomPassword("password"/*, {...}*/); const sshPublicKey = new tls.PrivateKey("key"/*, {...}*/); const adApp = new azuread.Application("aks"/*, {...}*/); const adSp = new azuread.ServicePrincipal("aksSp"/*, {...}*/); const vnet = new azure.network.VirtualNetwork("vnet"/*, {...}*/); const subnet = new azure.network.Subnet("subnet"/*, {...}*/); const cluster = new azure.containerservice.KubernetesCluster("aksCluster"/*, {...}*/) } } Components

Slide 41

Slide 41 text

Conclusions

Slide 42

Slide 42 text

Fine-grained components Better Automation

Slide 43

Slide 43 text

43 Infrastructure is a part of your applications. Use your developer skills to build it.

Slide 44

Slide 44 text

@MikhailShilkov https://mikhail.io