Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Pulumi: Cloud Infrastructure as C# and F#

Pulumi: Cloud Infrastructure as C# and F#

Modern cloud platforms offer amazing capabilities for application developers. However, cloud services are hard to manage without employing Infrastructure as Code tools for automation.

Traditional tools like CloudFormation, ARM templates, and Terraform use text-based formats, which tend to be tedious, repetitive, and cumbersome to reuse.

What if instead of configuration files, you could use your favorite programming languages like C# and F#? In this talk, Mikhail will introduce you to Pulumi, which uses actual code to manage infrastructure! See how you can bring your developer tools like code completion, types, components, and abstractions to cloud infrastructure definition.

Mikhail Shilkov

February 18, 2020
Tweet

More Decks by Mikhail Shilkov

Other Decks in Programming

Transcript

  1. Cloud Infrastructure
    as C# and F#
    with Pulumi
    Mikhail Shilkov

    View Slide

  2. • Software engineer at Pulumi:
    .NET SDK, Azure, Core platform
    • Cloud, Serverless
    • Functional programming, F#
    • Microsoft Azure MVP
    https://mikhail.io
    @MikhailShilkov
    About me

    View Slide

  3. Agenda

    View Slide

  4. Agenda

    View Slide

  5. Agenda

    View Slide

  6. Cloud
    Automation
    5

    View Slide

  7. Unified Management API
    Management API (e.g., Azure Resource Manager)
    Cloud Service Cloud Service Cloud Service
    Web Portal CLI SDK 3rd Party

    View Slide

  8. REST calls
    Content-Type: application/x-www-form-urlencoded; charset=UTF-8
    X-Amz-Date: 20130813T150211Z
    Host: ec2.amazonaws.com
    Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20130813/us-east-
    1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-
    date, Signature=ced6826de92d2bdeed8f846f0bf508e8559e98e4b0194b84example54174deb456c
    http://ec2.amazonaws.com/?Action=RunInstances
    ImageId=ami-2bb65342
    &MaxCount=3
    &MinCount=1
    &Monitoring.Enabled=true
    &Placement.AvailabilityZone=us-east-1a
    &Version=2016-11-15

    View Slide

  9. var launchRequest = new RunInstancesRequest
    {
    ImageId = amiID,
    InstanceType = "t1.micro",
    MinCount = 1,
    MaxCount = 1,
    KeyName = keyPairName,
    SecurityGroupIds = groups
    };
    var launchResponse = ec2Client.RunInstances(launchRequest);
    var instances = launchResponse.Reservation.Instances;
    SDKs to call those REST endpoints

    View Slide

  10. aws ec2 run-instances \
    --image-id ami-5ec1673e \
    --key-name MyKey \
    --security-groups EC2SecurityGroup \
    --instance-type t2.micro \
    --placement AvailabilityZone=us-west-2b
    Command-line Tools

    View Slide

  11. Step-by-step
    provisioning
    process
    Depend on both
    the current and
    the target state
    How to handle
    failures?
    Imperative Complex Error-prone
    SDK and Scripting: Challenges

    View Slide

  12. Desired State
    Configuration
    10

    View Slide

  13. Desired State Configuration
    Target Current
    Tool

    View Slide

  14. Managing Resource Graphs
    Target Current
    Tool

    View Slide

  15. Be able to describe
    the desired state in
    full, concisely, and
    correctly
    Determine the
    correct order of
    creation and
    destruction
    Preview the changes
    before running the
    deployment
    Expressiveness Correctness Transparency
    Challenges

    View Slide

  16. AWS
    CloudFormation
    Azure
    Resource
    Manager
    Google Cloud
    Deployment
    Manager
    Vendor Tools

    View Slide

  17. Markup
    Languages
    14
    YET
    ANOTHER
    MARKUP
    LANGUAGE

    View Slide

  18. Azure ARM Templates: JSON
    "resources": [
    {
    "apiVersion": "2016-01-01",
    "type": "Microsoft.Storage/storageAccounts",
    "name": "mystorageaccount",
    "location": "westus",
    "sku": {
    "name": "Standard_LRS"
    },
    "kind": "Storage",
    "properties": {
    }
    }
    ]

    View Slide

  19. AWS CloudFormation: YAML
    Resources:
    S3BucketForURLs:
    Type: "AWS::S3::Bucket"
    DeletionPolicy: Delete
    Properties:
    BucketName: !If [ "CreateNewBucket", !Ref "AWS::NoValue", !Ref S3BucketName ]
    WebsiteConfiguration:
    IndexDocument: "index.html"
    LifecycleConfiguration:
    Rules:
    -
    Id: DisposeShortUrls
    ExpirationInDays: !Ref URLExpiration
    Prefix: "u"
    Status: Enabled

    View Slide

  20. resource "azurerm_storage_account" "sa" {
    name = "mysa"
    location = azurerm_resource_group.rg.location
    resource_group_name = azurerm_resource_group.rg.name
    account_tier = "Standard"
    account_replication_type = "LRS"
    }
    Terraform: DSL
    (HashiCorp Configuration Language)

    View Slide

  21. Lack of fast
    feedback about
    errors
    Which options need
    to be filled and
    which values are
    possible
    Thousands of lines of
    markup are hard to
    write, read, and
    navigate
    Tooling Discovery Size
    Markup Authoring Experience

    View Slide

  22. Example: Code to infrastructure ratio
    KV Store
    Table “URLs”
    Function
    “Add URL”
    Function
    “Open URL”
    Lines of code
    Function Code ~20
    CloudFormation ~240
    ARM Templates ~160
    Terraform ~220
    Estimates based on public examples

    View Slide

  23. {
    "Resources": {
    "InstanceSecurityGroup": {
    "Type": "AWS::EC2::SecurityGroup",
    "Properties": {
    "SecurityGroupIngress": [ ... ]
    }
    },
    "EC2Instance": {
    "Type": "AWS::EC2::Instance",
    "Properties": {
    "InstanceType": "t2.micro",
    "SecurityGroups": [ { "Ref": "InstanceSecurityGroup" } ],
    "ImageId": "ami-0080e4c5bc078760e"
    }
    }
    }
    }
    References
    Example for AWS CloudFormation

    View Slide

  24. {
    "condition": "[equals(parameters('production'), 'Yes')]",
    "type": "Microsoft.Compute/availabilitySets",
    "apiVersion": "2017-03-30",
    "name": "[variables('availabilitySetName')]",
    "location": "[resourceGroup().location]",
    "properties": {
    "platformFaultDomainCount": 2,
    "platformUpdateDomainCount": 3
    }
    }
    Conditionals
    Example for Azure RM Templates

    View Slide

  25. locals {
    files = ["index.php","main.php"]
    }
    resource "aws_s3_bucket_object" "files" {
    count = "${length(local.files)}"
    key = "${local.files[count.index]}"
    bucket = "${aws_s3_bucket.examplebucket.id}"
    source = "./src/${local.files[count.index]}"
    }
    Loops
    Example for Terraform

    View Slide

  26. Different shape of
    infrastructure for
    different
    environments
    Bundling several
    related resources
    together for a
    higher-level
    component
    Sharing libraries of
    components within
    the company or in
    community
    Configuration Abstraction Packages
    Reusability

    View Slide

  27. Inspect the resulting
    resource tree before
    deployment
    (against mocks)
    Deploy a part of
    your graph to a test
    environment, inspect
    the result
    Test the result of
    a complete
    deployment
    Unit Integration End-to-end
    Testability

    View Slide

  28. I think we
    solved
    these
    problems
    before?

    View Slide

  29. IDEs
    Code completion
    Unit tests
    Types
    Functions
    Package managers
    Versioning
    Classes
    Code reviews
    Contracts
    SDKs
    Workflows
    Refactoring

    View Slide

  30. Pulumi
    Cloud infrastructure using real languages
    Developers and operators working
    together
    21

    View Slide

  31. General-Purpose Programming Languages
    Node.js
    Go
    Python
    .NET

    View Slide

  32. • AWS
    • Azure
    • GCP
    • Digital Ocean
    • Cloudflare
    … and more
    Providers
    • Docker
    • Kubernetes
    • OpenStack
    • PostgreSQL
    • New Relic

    View Slide

  33. var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard",
    });
    C# example

    View Slide

  34. Still,
    Desired
    State!
    var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard",
    });

    View Slide

  35. CLI drives the deployment
    How Pulumi
    works
    CLI & engine
    Last
    deployed
    state
    Program.cs
    Language host
    AWS
    Azure
    GCP
    Kubernetes
    new Resource()
    CRUD

    View Slide

  36. Static Website
    on AWS
    Demo
    26

    View Slide

  37. Serverless
    Functions
    on Azure
    Infrastructure Components
    33

    View Slide

  38. Resources to Provision an Azure Function
    Blob
    Plan
    Container
    Storage
    Account
    Function
    App

    View Slide

  39. • Encapsulate low-level resources into a higher-level abstraction
    • Reuse, shared abstractions, versioning, package management
    • Grouping and structure
    Component Resources

    View Slide

  40. // A resource group to contain our Azure Functions
    var resourceGroup = new ResourceGroup("functions-rg");
    // A .NET Azure Function
    var dotnet = new ArchiveFunctionApp("http", new ArchiveFunctionAppArgs
    {
    ResourceGroup = resourceGroup,
    Archive = new FileArchive("./dotnet/bin/Debug/netcoreapp3.1/publish")
    });
    .NET Azure Function

    View Slide

  41. Let’s run pulumi up …
    Type Name Plan
    + pulumi:pulumi:Stack functions-dev create
    + ├─ azure:appservice:ArchiveFunctionApp http create
    + │ ├─ azure:storage:Account http create
    + │ ├─ azure:appservice:Plan http create
    + │ ├─ azure:storage:Container http create
    + │ ├─ azure:storage:ZipBlob http create
    + │ └─ azure:appservice:FunctionApp http create
    + └─ azure:core:ResourceGroup rg create

    View Slide

  42. // A Node.js Azure Function
    var js = new ArchiveFunctionApp("js", new ArchiveFunctionAppArgs
    {
    ResourceGroup = resourceGroup,
    Archive = new FileArchive("./javascript"),
    Runtime = "node"
    });
    Node.js Azure Function

    View Slide

  43. var linuxResourceGroup = new ResourceGroup("linux-rg");
    var linuxPlan = new Plan("linux-asp", new PlanArgs
    {
    ResourceGroupName = linuxResourceGroup.Name,
    Kind = "Linux",
    Reserved = true,
    Sku = new PlanSkuArgs { Tier = "Dynamic", Size = "Y1" }
    });
    var python = new ArchiveFunctionApp(...);
    Python Azure Function on Linux

    View Slide

  44. var linuxResourceGroup = new ResourceGroup("linux-rg");
    var linuxPlan = new Plan("linux-asp", new PlanArgs { ... });
    var python = new ArchiveFunctionApp("py", new ArchiveFunctionAppArgs
    {
    ResourceGroup = linuxResourceGroup,
    Archive = new FileArchive("./python"),
    Plan = linuxPlan,
    Runtime = "python"
    });
    Python Azure Function on Linux

    View Slide

  45. Cosmos App
    Creating custom abstractions
    37

    View Slide

  46. Cosmos App
    (1)

    View Slide

  47. Cosmos App
    (2)

    View Slide

  48. Cosmos App
    (3)

    View Slide

  49. Cosmos App
    (4)

    View Slide

  50. var app = new CosmosApp("functions", new CosmosAppArgs
    {
    ResourceGroup = resourceGroup,
    Locations = ["WestUS", "EastUS", "WestEurope"],
    Factory = global => region =>
    {
    var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs
    {
    ResourceGroupName = resourceGroup.Name,
    Location = region.Location,
    Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"),
    AppSettings = { { "CosmosDBConnection", global.ConnectionString } }
    });
    return new AzureEndpoint(func.AppId);
    }
    });
    Global Applications

    View Slide

  51. var app = new CosmosApp("functions", new CosmosAppArgs
    {
    ResourceGroup = resourceGroup,
    Locations = ["WestUS", "EastUS", "WestEurope"],
    Factory = global => region =>
    {
    var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs
    {
    ResourceGroupName = resourceGroup.Name,
    Location = region.Location,
    Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"),
    AppSettings = { { "CosmosDBConnection", global.ConnectionString } }
    });
    return new AzureEndpoint(func.AppId);
    }
    });
    Global Applications

    View Slide

  52. How it works
    Behind the scenes
    41

    View Slide

  53. N providers, M runtimes

    View Slide

  54. N providers, M runtimes

    View Slide

  55. O(N*M) complexity

    View Slide

  56. O(N+M) complexity
    Intermediate
    Representation

    View Slide

  57. O(N+M) complexity

    View Slide

  58. State
    Management
    and Backends
    41

    View Slide

  59. State is a persisted graph
    State
    Management
    CLI and
    Engine
    Last
    deployed
    state
    Program.cs
    Language host
    AWS
    Azure
    GCP
    Kubernetes
    new Resource()
    CRUD
    Pulumi Service
    S3 / Blob Storage
    Local disk

    View Slide

  60. Open-source
    Free for any use
    Cross-platform
    Written in Go
    Open-source
    Free for any use
    Can be created by
    1st or 3rd party
    Web API + portal
    “GitHub for infra”
    Free for individuals
    Paid plans for teams
    Useful but not
    required
    CLI Providers SaaS backend
    Pulumi components

    View Slide

  61. Cloud credentials
    must be configured
    for CLI to run
    AWS profiles
    az login
    Tokens can be set
    via environment
    variables
    CLI Local logins Environment
    Cloud Authentication

    View Slide

  62. Deployment process
    Allow developers to
    iterate quickly on dev
    environments.
    Gain visibility and change
    management with
    Continuous Delivery.

    View Slide

  63. Track Infrastructure
    Changes from
    Commit to
    Deployment
    End to End Visibility

    View Slide

  64. Dependency Visualization
    Understand
    Dependencies
    Between Resources

    View Slide

  65. APIs and Webhooks
    Integrate
    Infrastructure
    Changes via
    Webhooks

    View Slide

  66. Role-Based Access Controls
    Manage Access
    Based on Stacks and
    Teams

    View Slide

  67. Kubernetes
    Deploy to multiple clouds
    45

    View Slide

  68. Kubernetes layers
    Managed Kubernetes cluster
    Infrastructure Resources (networking, storage, identity)
    Managed Service Managed Service
    Application Application Application

    View Slide

  69. Organizations, Projects, and Stacks
    Org: acme-corp
    Project: vpc
    Stack: dev
    env: dev
    region: us-east-1
    Stack: prod
    env: prod
    region: us-west-2
    Project: k8s-cluster
    Stack: dev
    env: dev
    region: us-east-1
    Stack: prod
    env: prod
    region: us-west-2
    Project: svc-userprofile
    Stack: dev
    env: dev
    region: us-east-1
    Stack: prod
    env: prod
    region: us-west-2
    Project: svc-email
    Stack: dev
    env: dev
    region: us-east-1
    Stack: prod
    env: prod
    region: us-west-2

    View Slide

  70. Stack References
    Org: acme-corp
    vpc
    Stack: dev
    env: dev
    region: us-east-1
    k8s-cluster
    Stack: dev
    env: dev
    region: us-east-1
    svc-userprofile
    Stack: dev
    env: dev
    region: us-east-1
    svc-email
    Stack: dev
    env: dev
    region: us-east-1

    View Slide

  71. Advanced
    scenarios
    Demo
    46

    View Slide

  72. F#
    48

    View Slide

  73. let storageAccount =
    Account("sa",
    AccountArgs
    (ResourceGroupName = io resourceGroup.Name,
    AccountReplicationType = input "LRS",
    AccountTier = input "Standard"))
    let sku = PlanSkuArgs(Tier = input "Basic", Size = input "B1")
    let appServicePlan =
    Plan("asp",
    PlanArgs
    (ResourceGroupName = io resourceGroup.Name,
    Kind = input "App",
    Sku = input sku))
    Resources in F#

    View Slide

  74. let d = deployment "mydep" {
    deploymentSpec {
    replicas 3
    selector { matchLabels appLabels }
    podTemplateSpec {
    metadata { labels appLabels }
    podSpec {
    container {
    name "nginx"
    image "nginx"
    containerPort 80
    }
    }
    }
    }
    }
    Computation Expression DSL?
    https://github.com/pulumi/pulumi/issues/3644

    View Slide

  75. Quality Control
    50

    View Slide

  76. Programming
    Languages for
    Infrastructure?
    You sure?

    View Slide

  77. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var result = await Deployment.TestAsync();
    var resource = result.Resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing

    View Slide

  78. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var result = await Deployment.TestAsync();
    var resource = result.Resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing

    View Slide

  79. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var result = await Deployment.TestAsync();
    var resource = result.Resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing

    View Slide

  80. Policy as Code (.NET coming soon)
    const policies = new PolicyPack("aws", {
    policies: [
    {
    name: "prohibited-public-internet",
    description: "Ingress rules with public internet access are prohibited.",
    enforcementLevel: "mandatory",
    validateResource: validateTypedResource(aws.ec2.SecurityGroup, (sg, args, report) => {
    const publicInternetRules = (sg.ingress || []).find(ingressRule =>
    (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0"));
    if (publicInternetRules) {
    report("Ingress rules with public internet access are prohibited.");
    }
    }),
    },
    ],
    });

    View Slide

  81. Alternatives
    53

    View Slide

  82. TypeScript
    Python
    Java
    .NET
    Transpiled to and
    deployed by
    AWS
    CloudFormation
    AWS only Languages CloudFormation
    AWS Cloud Development Kit (CDK)

    View Slide

  83. var queue = new Queue(this, "MyFirstQueue", new QueueProps
    {
    VisibilityTimeoutSec = 300
    });
    var topic = new Topic(this, "MyFirstTopic", new TopicProps
    {
    DisplayName = "My First Topic Yeah"
    });
    AWS CDK example

    View Slide

  84. // This is F#
    let myStorage = storageAccount {
    name "mystorage" // set account name
    sku Storage.Sku.PremiumLRS // use Premium LRS
    }
    let myWebApp = webApp {
    name "mysuperwebapp" // set web app name
    sku WebApp.Sku.S1 // use S1 size
    setting "storage_key" myStorage.Key // app setting
    depends_on myStorage // webapp is dependent on storage
    }
    Azure? ARM Generators (ex: Farmer)
    https://github.com/CompositionalIT/farmer

    View Slide

  85. Conclusions
    55

    View Slide

  86. Define
    Infrastructure
    as Code
    Making
    Cloud
    Apps?

    View Slide

  87. Fine-grained
    components
    Better
    Automation

    View Slide

  88. Familiar language,
    experience,
    toolchain, packages
    Conditionals, loops,
    functions, classes,
    components,
    modules
    Reusable
    components that
    encapsulate
    complex logic and
    provide the right
    level of abstraction
    Developer-friendly Logic Abstractions
    Benefits of Real Code

    View Slide

  89. м
    Infrastructure
    as Code
    Architecture
    as Code

    View Slide

  90. • Get Started with Pulumi:
    https://pulumi.com/dotnet
    • Pulumi + .NET blog post:
    “Building Modern Cloud Applications using Pulumi and .NET Core”
    https://devblogs.microsoft.com/dotnet/building-modern-cloud-
    applications-using-pulumi-and-net-core/
    • Pulumi + Cosmos blog post:
    “How to build globally distributed applications with Azure Cosmos
    DB and Pulumi”
    https://azure.microsoft.com/en-gb/blog/how-to-build-globally-
    distributed-applications-with-azure-cosmos-db-and-pulumi/
    Useful Links

    View Slide

  91. Meetups, Talks and Workshops with Pulumi
    Upcoming Events
    @PulumiCorp
    pulumi
    http://bit.ly/2OQ00Tt
    18 Feb Copenhagen, DK
    18 Feb Kaunas, LT
    19 Feb Malmo, SE
    20 Feb Amsterdam, NL
    21 Feb Madrid, ES
    25 Feb Prague, CZ
    26 Feb Frankfurt, DE
    30 Mar Amsterdam, NL

    View Slide

  92. Amsterdam – 30 March
    The Pulumi team will provide hands-on instruction
    in a 4-hour workshop showing you how to setup
    your infrastructure for Kubernetes and other useful
    configurations.
    Includes lunch, workshop, labs and happy hour.
    Coming to a city near you
    Pulumi IaC Workshops
    @PulumiCorp
    pulumi
    http://bit.ly/2HgD9wd
    You don’t have to attend KubeCon to join this workshop

    View Slide

  93. @MikhailShilkov
    https://mikhail.io

    View Slide