From YAML to TypeScript: Developer’s View on Cloud Automation

Modern cloud platforms offer amazing capabilities for application developers. Cloud apps utilize multiple services and consist of many components, so they are hard to manage without employing Infrastructure as Code. Traditional tools like CloudFormation, ARM templates, and Terraform use text-based formats, which tend to be tedious, repetitive, and cumbersome to reuse. What if instead of configuration files you could use a language like TypeScript? See how you can bring your favorite developer tools like code completion, types, components, and abstractions to cloud infrastructure definition.

Transcript

1. From YAML to TypeScript:
   Developer’s View on Cloud Automation
   Mikhail Shilkov – Software engineer @ Pulumi
   24-25-26 Novembre, 2020
   

   View Slide

2. Why
   Infrastructure
   as Code
   General-
   purpose
   Languages
   

   View Slide

3. Cloud
   Automation
   

   View Slide

4. Unified Management API
   

   View Slide

5. REST calls
   Content-Type: application/x-www-form-urlencoded; charset=UTF-8
   X-Amz-Date: 20130813T150211Z
   Host: ec2.amazonaws.com
   Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20130813/us-east-
   1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-
   date, Signature=ced6826de92d2bdeed8f846f0bf508e8559e98e4b0194b84example54174deb456c
   http://ec2.amazonaws.com/?Action=RunInstances
   ImageId=ami-2bb65342
   &MaxCount=3
   &MinCount=1
   &Monitoring.Enabled=true
   &Placement.AvailabilityZone=us-east-1a
   &Version=2016-11-15
   

   View Slide

6. var launchRequest = new RunInstancesRequest()
   {
   ImageId = amiID,
   InstanceType = "t1.micro",
   MinCount = 1,
   MaxCount = 1,
   KeyName = keyPairName,
   SecurityGroupIds = groups
   };
   var launchResponse = ec2Client.RunInstances(launchRequest);
   var instances = launchResponse.Reservation.Instances;
   SDKs to call those REST endpoints
   

   View Slide

7. az storage account create \
   --location westus \
   --name samplesa \
   --resource-group myrg \
   --sku LRS
   Command-line Tools
   

   View Slide

8. Step-by-step
   provisioning
   process
   Depend on both
   the current and
   the target state
   How to handle
   failures?
   Imperative Complex Error-prone
   Scripting: Challenges
   

   View Slide

9. Desired State
   Configuration
   

   View Slide

10. Desired State Configuration
    Target Current
    Tool
    

    View Slide

11. Managing Resource Graphs
    Target Current
    Tool
    

    View Slide

12. Be able to describe
    the desired state in
    full, concisely, and
    correctly
    Determine the
    correct order of
    creation and
    destruction
    Preview the changes
    before running the
    deployment
    Expressiveness Dependencies Transparency
    Challenges
    

    View Slide

13. AWS
    CloudFormation
    Azure
    Resource
    Manager
    Google Cloud
    Deployment
    Manager
    Desired State Provisioning
    

    View Slide

14. Markup
    Languages
    YET
    ANOTHER
    MARKUP
    LANGUAGE
    

    View Slide

15. Azure ARM Templates: JSON
    "resources": [
    {
    "apiVersion": "2016-01-01",
    "type": "Microsoft.Storage/storageAccounts",
    "name": "mystorageaccount",
    "location": "westus",
    "sku": {
    "name": "Standard_LRS"
    },
    "kind": "Storage",
    "properties": {
    }
    }
    ]
    

    View Slide

16. AWS CloudFormation: YAML
    Resources:
    S3BucketForURLs:
    Type: "AWS::S3::Bucket"
    DeletionPolicy: Delete
    Properties:
    BucketName: !If [ "CreateNewBucket", !Ref "AWS::NoValue", !Ref S3BucketName ]
    WebsiteConfiguration:
    IndexDocument: "index.html"
    LifecycleConfiguration:
    Rules:
    -
    Id: DisposeShortUrls
    ExpirationInDays: !Ref URLExpiration
    Prefix: "u"
    Status: Enabled
    

    View Slide

17. resource "azurerm_storage_account" "sa" {
    name = "mysa"
    location = azurerm_resource_group.rg.location
    resource_group_name = azurerm_resource_group.rg.name
    account_tier = "Standard"
    account_replication_type = "LRS"
    }
    Terraform: DSL
    (HashiCorp Configuration Language)
    

    View Slide

18. Lack of fast
    feedback about
    errors
    Which options need
    to be filled and
    which values are
    possible
    Thousands of lines of
    markup are hard to
    write, read, and
    navigate
    Tooling Discovery Size
    Markup Authoring Experience
    

    View Slide

19. Example: Code to infrastructure ratio
    KV Store
    Table “URLs”
    Function
    “Add URL”
    Function
    “Open URL”
    Lines of code
    Function Code ~20
    CloudFormation ~240
    ARM Templates ~160
    Terraform ~220
    Estimates based on public examples
    

    View Slide

20. {
    "Resources": {
    "InstanceSecurityGroup": {
    "Type": "AWS::EC2::SecurityGroup",
    "Properties": {
    "SecurityGroupIngress": [ ... ]
    }
    },
    "EC2Instance": {
    "Type": "AWS::EC2::Instance",
    "Properties": {
    "InstanceType": "t2.micro",
    "SecurityGroups": [ { "Ref": "InstanceSecurityGroup" } ],
    "ImageId": "ami-0080e4c5bc078760e"
    }
    }
    }
    }
    References
    Example for AWS CloudFormation
    

    View Slide

21. {
    "condition": "[equals(parameters('production'), 'Yes')]",
    "type": "Microsoft.Compute/availabilitySets",
    "apiVersion": "2017-03-30",
    "name": "[variables('availabilitySetName')]",
    "location": "[resourceGroup().location]",
    "properties": {
    "platformFaultDomainCount": 2,
    "platformUpdateDomainCount": 3
    }
    }
    Conditionals
    Example for Azure RM Templates
    

    View Slide

22. locals {
    files = ["index.php","main.php"]
    }
    resource "aws_s3_bucket_object" "files" {
    count = "${length(local.files)}"
    key = "${local.files[count.index]}"
    bucket = "${aws_s3_bucket.examplebucket.id}"
    source = "./src/${local.files[count.index]}"
    }
    Loops
    Example for Terraform
    

    View Slide

23. Different shape of
    infrastructure for
    different
    environments
    Bundling several
    related resources
    together for a
    higher-level
    component
    Sharing libraries of
    components within
    the company or in
    community
    Configuration Abstraction Packages
    Reusability
    

    View Slide

24. Test a single
    component against
    mocks
    Inspect the resulting
    resource tree before
    deployment
    Test the result of
    a deployment
    Unit Integration End-to-end
    Testability
    

    View Slide

25. I think we
    solved
    these
    problems
    before?
    

    View Slide

26. IDEs
    Code completion
    Unit tests
    Types
    Functions
    Package managers
    Versioning
    Classes
    Code reviews
    Contracts
    SDKs
    Workflows
    Refactoring
    

    View Slide

27. Pulumi
    Program the cloud
    

    View Slide

28. General-Purpose Programming Languages
    Node.js
    Go
    Python
    .NET
    

    View Slide

29. const resourceGroup = new azure.core.ResourceGroup("rg");
    const storageAccount = new azure.storage.Account("storage", {
    resourceGroupName: resourceGroup.name,
    accountReplicationType: "LRS",
    accountTier: "Standard",
    });
    TypeScript example
    

    View Slide

30. const counterTable = new aws.dynamodb.Table("url-table", {
    attributes: [{ name: "id", type: "S" }],
    hashKey: "id",
    readCapacity: 5,
    });
    const lambda = new aws.lambda.Function("mylambda", {
    code: new pulumi.asset.FileArchive("./app"),
    handler: "index.handler",
    environment: { variables: {
    "COUNTER_TABLE": counterTable.name,
    }},
    });
    const endpoint = new awsx.apigateway.API("short-url", {
    routes: [{
    path: "/{route+}",
    method: "GET",
    eventHandler: lambda,
    }],
    });
    Example:
    Serverless
    HTTP
    

    View Slide

31. const counterTable = new aws.dynamodb.Table("url-table", {
    attributes: [{ name: "id", type: "S" }],
    hashKey: "id",
    readCapacity: 5,
    });
    const lambda = new aws.lambda.Function("mylambda", {
    code: new pulumi.asset.FileArchive("./app"),
    handler: "index.handler",
    environment: { variables: {
    "COUNTER_TABLE": counterTable.name,
    }},
    });
    const endpoint = new awsx.apigateway.API("short-url", {
    routes: [{
    path: "/{route+}",
    method: "GET",
    eventHandler: lambda,
    }],
    });
    Example:
    Serverless
    HTTP
    

    View Slide

32. const counterTable = new aws.dynamodb.Table("url-table", {
    attributes: [{ name: "id", type: "S" }],
    hashKey: "id",
    readCapacity: 5,
    });
    const lambda = new aws.lambda.Function("mylambda", {
    code: new pulumi.asset.FileArchive("./app"),
    handler: "index.handler",
    environment: { variables: {
    "COUNTER_TABLE": counterTable.name,
    }},
    });
    const endpoint = new awsx.apigateway.API("short-url", {
    routes: [{
    path: "/{route+}",
    method: "GET",
    eventHandler: lambda,
    }],
    });
    Example:
    Serverless
    HTTP
    

    View Slide

33. const counterTable = new aws.dynamodb.Table("url-table", {
    attributes: [{ name: "id", type: "S" }],
    hashKey: "id",
    readCapacity: 5,
    });
    const lambda = new aws.lambda.Function("mylambda", {
    code: new pulumi.asset.FileArchive("./app"),
    handler: "index.handler",
    environment: { variables: {
    "COUNTER_TABLE": counterTable.name,
    }},
    });
    const endpoint = new awsx.apigateway.API("short-url", {
    routes: [{
    path: "/{route+}",
    method: "GET",
    eventHandler: lambda,
    }],
    });
    Still,
    Desired
    State!
    

    View Slide

34. CLI drives the deployment
    How Pulumi
    works
    CLI and Engine
    Last
    deployed
    state
    index.ts
    Language host
    AWS
    Azure
    GCP
    Kubernetes
    new Resource()
    Create, update,
    delete
    

    View Slide

35. • AWS
    • Azure
    • GCP
    • Digital Ocean
    • Cloudflare
    … and more
    Providers
    • Docker
    • Kubernetes
    • OpenStack
    • PostgreSQL
    • New Relic
    

    View Slide

36. Familiar language,
    experience,
    toolchain, packages
    Conditionals, loops,
    functions, classes,
    components,
    modules
    Reusable
    components that
    encapsulate
    complex logic and
    provide the right
    level of abstraction
    Developer-friendly Logic Abstractions
    Benefits of Real Code
    

    View Slide

37. Auto-
    completion
    

    View Slide

38. Compile
    Errors
    38
    

    View Slide

39. for (const az of aws.getAvailabilityZones().names) {
    const server = new aws.ec2.Instance(`web-${az}`, {
    instanceType: "t2.micro",
    securityGroups: [ group.id ],
    ami: "ami-0080e4c5bc078760e",
    availabilityZone: az,
    });
    }
    Conditionals, loops
    

    View Slide

40. class AksCluster extends pulumi.ComponentResource {
    constructor(name: string,
    args: AksClusterArgs,
    opts: pulumi.ComponentResourceOptions = {}) {
    super("examples:aks:AksCluster", name, args, opts);
    const password = new random.RandomPassword("password"/*, {...}*/);
    const sshPublicKey = new tls.PrivateKey("key"/*, {...}*/);
    const adApp = new azuread.Application("aks"/*, {...}*/);
    const adSp = new azuread.ServicePrincipal("aksSp"/*, {...}*/);
    const vnet = new azure.network.VirtualNetwork("vnet"/*, {...}*/);
    const subnet = new azure.network.Subnet("subnet"/*, {...}*/);
    const cluster = new azure.containerservice.KubernetesCluster("aksCluster"/*, {...}*/)
    }
    }
    Components
    

    View Slide

41. Conclusions
    

    View Slide

42. Fine-grained
    components
    Better
    Automation
    

    View Slide

43. 43
    Infrastructure is a part of your applications.
    Use your developer skills to build it.
    

    View Slide

44. @MikhailShilkov
    https://mikhail.io
    

    View Slide