Upgrade to Pro — share decks privately, control downloads, hide ads and more …

From YAML to TypeScript: Developer’s View on Cloud Automation

From YAML to TypeScript: Developer’s View on Cloud Automation

Modern cloud platforms offer amazing capabilities for application developers. Cloud apps utilize multiple services and consist of many components, so they are hard to manage without employing Infrastructure as Code. Traditional tools like CloudFormation, ARM templates, and Terraform use text-based formats, which tend to be tedious, repetitive, and cumbersome to reuse. What if instead of configuration files you could use a language like TypeScript? See how you can bring your favorite developer tools like code completion, types, components, and abstractions to cloud infrastructure definition.

Mikhail Shilkov

November 24, 2020
Tweet

More Decks by Mikhail Shilkov

Other Decks in Programming

Transcript

  1. From YAML to TypeScript:
    Developer’s View on Cloud Automation
    Mikhail Shilkov – Software engineer @ Pulumi
    24-25-26 Novembre, 2020

    View Slide

  2. Why
    Infrastructure
    as Code
    General-
    purpose
    Languages

    View Slide

  3. Cloud
    Automation

    View Slide

  4. Unified Management API

    View Slide

  5. REST calls
    Content-Type: application/x-www-form-urlencoded; charset=UTF-8
    X-Amz-Date: 20130813T150211Z
    Host: ec2.amazonaws.com
    Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20130813/us-east-
    1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-
    date, Signature=ced6826de92d2bdeed8f846f0bf508e8559e98e4b0194b84example54174deb456c
    http://ec2.amazonaws.com/?Action=RunInstances
    ImageId=ami-2bb65342
    &MaxCount=3
    &MinCount=1
    &Monitoring.Enabled=true
    &Placement.AvailabilityZone=us-east-1a
    &Version=2016-11-15

    View Slide

  6. var launchRequest = new RunInstancesRequest()
    {
    ImageId = amiID,
    InstanceType = "t1.micro",
    MinCount = 1,
    MaxCount = 1,
    KeyName = keyPairName,
    SecurityGroupIds = groups
    };
    var launchResponse = ec2Client.RunInstances(launchRequest);
    var instances = launchResponse.Reservation.Instances;
    SDKs to call those REST endpoints

    View Slide

  7. az storage account create \
    --location westus \
    --name samplesa \
    --resource-group myrg \
    --sku LRS
    Command-line Tools

    View Slide

  8. Step-by-step
    provisioning
    process
    Depend on both
    the current and
    the target state
    How to handle
    failures?
    Imperative Complex Error-prone
    Scripting: Challenges

    View Slide

  9. Desired State
    Configuration

    View Slide

  10. Desired State Configuration
    Target Current
    Tool

    View Slide

  11. Managing Resource Graphs
    Target Current
    Tool

    View Slide

  12. Be able to describe
    the desired state in
    full, concisely, and
    correctly
    Determine the
    correct order of
    creation and
    destruction
    Preview the changes
    before running the
    deployment
    Expressiveness Dependencies Transparency
    Challenges

    View Slide

  13. AWS
    CloudFormation
    Azure
    Resource
    Manager
    Google Cloud
    Deployment
    Manager
    Desired State Provisioning

    View Slide

  14. Markup
    Languages
    YET
    ANOTHER
    MARKUP
    LANGUAGE

    View Slide

  15. Azure ARM Templates: JSON
    "resources": [
    {
    "apiVersion": "2016-01-01",
    "type": "Microsoft.Storage/storageAccounts",
    "name": "mystorageaccount",
    "location": "westus",
    "sku": {
    "name": "Standard_LRS"
    },
    "kind": "Storage",
    "properties": {
    }
    }
    ]

    View Slide

  16. AWS CloudFormation: YAML
    Resources:
    S3BucketForURLs:
    Type: "AWS::S3::Bucket"
    DeletionPolicy: Delete
    Properties:
    BucketName: !If [ "CreateNewBucket", !Ref "AWS::NoValue", !Ref S3BucketName ]
    WebsiteConfiguration:
    IndexDocument: "index.html"
    LifecycleConfiguration:
    Rules:
    -
    Id: DisposeShortUrls
    ExpirationInDays: !Ref URLExpiration
    Prefix: "u"
    Status: Enabled

    View Slide

  17. resource "azurerm_storage_account" "sa" {
    name = "mysa"
    location = azurerm_resource_group.rg.location
    resource_group_name = azurerm_resource_group.rg.name
    account_tier = "Standard"
    account_replication_type = "LRS"
    }
    Terraform: DSL
    (HashiCorp Configuration Language)

    View Slide

  18. Lack of fast
    feedback about
    errors
    Which options need
    to be filled and
    which values are
    possible
    Thousands of lines of
    markup are hard to
    write, read, and
    navigate
    Tooling Discovery Size
    Markup Authoring Experience

    View Slide

  19. Example: Code to infrastructure ratio
    KV Store
    Table “URLs”
    Function
    “Add URL”
    Function
    “Open URL”
    Lines of code
    Function Code ~20
    CloudFormation ~240
    ARM Templates ~160
    Terraform ~220
    Estimates based on public examples

    View Slide

  20. {
    "Resources": {
    "InstanceSecurityGroup": {
    "Type": "AWS::EC2::SecurityGroup",
    "Properties": {
    "SecurityGroupIngress": [ ... ]
    }
    },
    "EC2Instance": {
    "Type": "AWS::EC2::Instance",
    "Properties": {
    "InstanceType": "t2.micro",
    "SecurityGroups": [ { "Ref": "InstanceSecurityGroup" } ],
    "ImageId": "ami-0080e4c5bc078760e"
    }
    }
    }
    }
    References
    Example for AWS CloudFormation

    View Slide

  21. {
    "condition": "[equals(parameters('production'), 'Yes')]",
    "type": "Microsoft.Compute/availabilitySets",
    "apiVersion": "2017-03-30",
    "name": "[variables('availabilitySetName')]",
    "location": "[resourceGroup().location]",
    "properties": {
    "platformFaultDomainCount": 2,
    "platformUpdateDomainCount": 3
    }
    }
    Conditionals
    Example for Azure RM Templates

    View Slide

  22. locals {
    files = ["index.php","main.php"]
    }
    resource "aws_s3_bucket_object" "files" {
    count = "${length(local.files)}"
    key = "${local.files[count.index]}"
    bucket = "${aws_s3_bucket.examplebucket.id}"
    source = "./src/${local.files[count.index]}"
    }
    Loops
    Example for Terraform

    View Slide

  23. Different shape of
    infrastructure for
    different
    environments
    Bundling several
    related resources
    together for a
    higher-level
    component
    Sharing libraries of
    components within
    the company or in
    community
    Configuration Abstraction Packages
    Reusability

    View Slide

  24. Test a single
    component against
    mocks
    Inspect the resulting
    resource tree before
    deployment
    Test the result of
    a deployment
    Unit Integration End-to-end
    Testability

    View Slide

  25. I think we
    solved
    these
    problems
    before?

    View Slide

  26. IDEs
    Code completion
    Unit tests
    Types
    Functions
    Package managers
    Versioning
    Classes
    Code reviews
    Contracts
    SDKs
    Workflows
    Refactoring

    View Slide

  27. Pulumi
    Program the cloud

    View Slide

  28. General-Purpose Programming Languages
    Node.js
    Go
    Python
    .NET

    View Slide

  29. const resourceGroup = new azure.core.ResourceGroup("rg");
    const storageAccount = new azure.storage.Account("storage", {
    resourceGroupName: resourceGroup.name,
    accountReplicationType: "LRS",
    accountTier: "Standard",
    });
    TypeScript example

    View Slide

  30. const counterTable = new aws.dynamodb.Table("url-table", {
    attributes: [{ name: "id", type: "S" }],
    hashKey: "id",
    readCapacity: 5,
    });
    const lambda = new aws.lambda.Function("mylambda", {
    code: new pulumi.asset.FileArchive("./app"),
    handler: "index.handler",
    environment: { variables: {
    "COUNTER_TABLE": counterTable.name,
    }},
    });
    const endpoint = new awsx.apigateway.API("short-url", {
    routes: [{
    path: "/{route+}",
    method: "GET",
    eventHandler: lambda,
    }],
    });
    Example:
    Serverless
    HTTP

    View Slide

  31. const counterTable = new aws.dynamodb.Table("url-table", {
    attributes: [{ name: "id", type: "S" }],
    hashKey: "id",
    readCapacity: 5,
    });
    const lambda = new aws.lambda.Function("mylambda", {
    code: new pulumi.asset.FileArchive("./app"),
    handler: "index.handler",
    environment: { variables: {
    "COUNTER_TABLE": counterTable.name,
    }},
    });
    const endpoint = new awsx.apigateway.API("short-url", {
    routes: [{
    path: "/{route+}",
    method: "GET",
    eventHandler: lambda,
    }],
    });
    Example:
    Serverless
    HTTP

    View Slide

  32. const counterTable = new aws.dynamodb.Table("url-table", {
    attributes: [{ name: "id", type: "S" }],
    hashKey: "id",
    readCapacity: 5,
    });
    const lambda = new aws.lambda.Function("mylambda", {
    code: new pulumi.asset.FileArchive("./app"),
    handler: "index.handler",
    environment: { variables: {
    "COUNTER_TABLE": counterTable.name,
    }},
    });
    const endpoint = new awsx.apigateway.API("short-url", {
    routes: [{
    path: "/{route+}",
    method: "GET",
    eventHandler: lambda,
    }],
    });
    Example:
    Serverless
    HTTP

    View Slide

  33. const counterTable = new aws.dynamodb.Table("url-table", {
    attributes: [{ name: "id", type: "S" }],
    hashKey: "id",
    readCapacity: 5,
    });
    const lambda = new aws.lambda.Function("mylambda", {
    code: new pulumi.asset.FileArchive("./app"),
    handler: "index.handler",
    environment: { variables: {
    "COUNTER_TABLE": counterTable.name,
    }},
    });
    const endpoint = new awsx.apigateway.API("short-url", {
    routes: [{
    path: "/{route+}",
    method: "GET",
    eventHandler: lambda,
    }],
    });
    Still,
    Desired
    State!

    View Slide

  34. CLI drives the deployment
    How Pulumi
    works
    CLI and Engine
    Last
    deployed
    state
    index.ts
    Language host
    AWS
    Azure
    GCP
    Kubernetes
    new Resource()
    Create, update,
    delete

    View Slide

  35. • AWS
    • Azure
    • GCP
    • Digital Ocean
    • Cloudflare
    … and more
    Providers
    • Docker
    • Kubernetes
    • OpenStack
    • PostgreSQL
    • New Relic

    View Slide

  36. Familiar language,
    experience,
    toolchain, packages
    Conditionals, loops,
    functions, classes,
    components,
    modules
    Reusable
    components that
    encapsulate
    complex logic and
    provide the right
    level of abstraction
    Developer-friendly Logic Abstractions
    Benefits of Real Code

    View Slide

  37. Auto-
    completion

    View Slide

  38. Compile
    Errors
    38

    View Slide

  39. for (const az of aws.getAvailabilityZones().names) {
    const server = new aws.ec2.Instance(`web-${az}`, {
    instanceType: "t2.micro",
    securityGroups: [ group.id ],
    ami: "ami-0080e4c5bc078760e",
    availabilityZone: az,
    });
    }
    Conditionals, loops

    View Slide

  40. class AksCluster extends pulumi.ComponentResource {
    constructor(name: string,
    args: AksClusterArgs,
    opts: pulumi.ComponentResourceOptions = {}) {
    super("examples:aks:AksCluster", name, args, opts);
    const password = new random.RandomPassword("password"/*, {...}*/);
    const sshPublicKey = new tls.PrivateKey("key"/*, {...}*/);
    const adApp = new azuread.Application("aks"/*, {...}*/);
    const adSp = new azuread.ServicePrincipal("aksSp"/*, {...}*/);
    const vnet = new azure.network.VirtualNetwork("vnet"/*, {...}*/);
    const subnet = new azure.network.Subnet("subnet"/*, {...}*/);
    const cluster = new azure.containerservice.KubernetesCluster("aksCluster"/*, {...}*/)
    }
    }
    Components

    View Slide

  41. Conclusions

    View Slide

  42. Fine-grained
    components
    Better
    Automation

    View Slide

  43. 43
    Infrastructure is a part of your applications.
    Use your developer skills to build it.

    View Slide

  44. @MikhailShilkov
    https://mikhail.io

    View Slide