Pulumi: Cloud Infrastructure as C# and F#

Modern cloud platforms offer amazing capabilities for application developers. However, cloud services are hard to manage without employing Infrastructure as Code tools for automation.

Traditional tools like CloudFormation, ARM templates, and Terraform use text-based formats, which tend to be tedious, repetitive, and cumbersome to reuse.

What if instead of configuration files, you could use your favorite programming languages like C# and F#? In this talk, Mikhail will introduce you to Pulumi, which uses actual code to manage infrastructure! See how you can bring your developer tools like code completion, types, components, and abstractions to cloud infrastructure definition.

Transcript

1. Cloud Infrastructure
   as C# and F#
   with Pulumi
   Mikhail Shilkov
   

   View Slide

2. • Software engineer at Pulumi:
   .NET SDK, Azure, Core platform
   • Cloud, Serverless
   • Functional programming, F#
   • Microsoft Azure MVP
   https://mikhail.io
   @MikhailShilkov
   About me
   

   View Slide

3. Agenda
   

   View Slide

4. Agenda
   

   View Slide

5. Agenda
   

   View Slide

6. Cloud
   Automation
   5
   

   View Slide

7. Unified Management API
   Management API (e.g., Azure Resource Manager)
   Cloud Service Cloud Service Cloud Service
   Web Portal CLI SDK 3rd Party
   

   View Slide

8. REST calls
   Content-Type: application/x-www-form-urlencoded; charset=UTF-8
   X-Amz-Date: 20130813T150211Z
   Host: ec2.amazonaws.com
   Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20130813/us-east-
   1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-
   date, Signature=ced6826de92d2bdeed8f846f0bf508e8559e98e4b0194b84example54174deb456c
   http://ec2.amazonaws.com/?Action=RunInstances
   ImageId=ami-2bb65342
   &MaxCount=3
   &MinCount=1
   &Monitoring.Enabled=true
   &Placement.AvailabilityZone=us-east-1a
   &Version=2016-11-15
   

   View Slide

9. var launchRequest = new RunInstancesRequest
   {
   ImageId = amiID,
   InstanceType = "t1.micro",
   MinCount = 1,
   MaxCount = 1,
   KeyName = keyPairName,
   SecurityGroupIds = groups
   };
   var launchResponse = ec2Client.RunInstances(launchRequest);
   var instances = launchResponse.Reservation.Instances;
   SDKs to call those REST endpoints
   

   View Slide

10. aws ec2 run-instances \
    --image-id ami-5ec1673e \
    --key-name MyKey \
    --security-groups EC2SecurityGroup \
    --instance-type t2.micro \
    --placement AvailabilityZone=us-west-2b
    Command-line Tools
    

    View Slide

11. Step-by-step
    provisioning
    process
    Depend on both
    the current and
    the target state
    How to handle
    failures?
    Imperative Complex Error-prone
    SDK and Scripting: Challenges
    

    View Slide

12. Desired State
    Configuration
    10
    

    View Slide

13. Desired State Configuration
    Target Current
    Tool
    

    View Slide

14. Managing Resource Graphs
    Target Current
    Tool
    

    View Slide

15. Be able to describe
    the desired state in
    full, concisely, and
    correctly
    Determine the
    correct order of
    creation and
    destruction
    Preview the changes
    before running the
    deployment
    Expressiveness Correctness Transparency
    Challenges
    

    View Slide

16. AWS
    CloudFormation
    Azure
    Resource
    Manager
    Google Cloud
    Deployment
    Manager
    Vendor Tools
    

    View Slide

17. Markup
    Languages
    14
    YET
    ANOTHER
    MARKUP
    LANGUAGE
    

    View Slide

18. Azure ARM Templates: JSON
    "resources": [
    {
    "apiVersion": "2016-01-01",
    "type": "Microsoft.Storage/storageAccounts",
    "name": "mystorageaccount",
    "location": "westus",
    "sku": {
    "name": "Standard_LRS"
    },
    "kind": "Storage",
    "properties": {
    }
    }
    ]
    

    View Slide

19. AWS CloudFormation: YAML
    Resources:
    S3BucketForURLs:
    Type: "AWS::S3::Bucket"
    DeletionPolicy: Delete
    Properties:
    BucketName: !If [ "CreateNewBucket", !Ref "AWS::NoValue", !Ref S3BucketName ]
    WebsiteConfiguration:
    IndexDocument: "index.html"
    LifecycleConfiguration:
    Rules:
    -
    Id: DisposeShortUrls
    ExpirationInDays: !Ref URLExpiration
    Prefix: "u"
    Status: Enabled
    

    View Slide

20. resource "azurerm_storage_account" "sa" {
    name = "mysa"
    location = azurerm_resource_group.rg.location
    resource_group_name = azurerm_resource_group.rg.name
    account_tier = "Standard"
    account_replication_type = "LRS"
    }
    Terraform: DSL
    (HashiCorp Configuration Language)
    

    View Slide

21. Lack of fast
    feedback about
    errors
    Which options need
    to be filled and
    which values are
    possible
    Thousands of lines of
    markup are hard to
    write, read, and
    navigate
    Tooling Discovery Size
    Markup Authoring Experience
    

    View Slide

22. Example: Code to infrastructure ratio
    KV Store
    Table “URLs”
    Function
    “Add URL”
    Function
    “Open URL”
    Lines of code
    Function Code ~20
    CloudFormation ~240
    ARM Templates ~160
    Terraform ~220
    Estimates based on public examples
    

    View Slide

23. {
    "Resources": {
    "InstanceSecurityGroup": {
    "Type": "AWS::EC2::SecurityGroup",
    "Properties": {
    "SecurityGroupIngress": [ ... ]
    }
    },
    "EC2Instance": {
    "Type": "AWS::EC2::Instance",
    "Properties": {
    "InstanceType": "t2.micro",
    "SecurityGroups": [ { "Ref": "InstanceSecurityGroup" } ],
    "ImageId": "ami-0080e4c5bc078760e"
    }
    }
    }
    }
    References
    Example for AWS CloudFormation
    

    View Slide

24. {
    "condition": "[equals(parameters('production'), 'Yes')]",
    "type": "Microsoft.Compute/availabilitySets",
    "apiVersion": "2017-03-30",
    "name": "[variables('availabilitySetName')]",
    "location": "[resourceGroup().location]",
    "properties": {
    "platformFaultDomainCount": 2,
    "platformUpdateDomainCount": 3
    }
    }
    Conditionals
    Example for Azure RM Templates
    

    View Slide

25. locals {
    files = ["index.php","main.php"]
    }
    resource "aws_s3_bucket_object" "files" {
    count = "${length(local.files)}"
    key = "${local.files[count.index]}"
    bucket = "${aws_s3_bucket.examplebucket.id}"
    source = "./src/${local.files[count.index]}"
    }
    Loops
    Example for Terraform
    

    View Slide

26. Different shape of
    infrastructure for
    different
    environments
    Bundling several
    related resources
    together for a
    higher-level
    component
    Sharing libraries of
    components within
    the company or in
    community
    Configuration Abstraction Packages
    Reusability
    

    View Slide

27. Inspect the resulting
    resource tree before
    deployment
    (against mocks)
    Deploy a part of
    your graph to a test
    environment, inspect
    the result
    Test the result of
    a complete
    deployment
    Unit Integration End-to-end
    Testability
    

    View Slide

28. I think we
    solved
    these
    problems
    before?
    

    View Slide

29. IDEs
    Code completion
    Unit tests
    Types
    Functions
    Package managers
    Versioning
    Classes
    Code reviews
    Contracts
    SDKs
    Workflows
    Refactoring
    

    View Slide

30. Pulumi
    Cloud infrastructure using real languages
    Developers and operators working
    together
    21
    

    View Slide

31. General-Purpose Programming Languages
    Node.js
    Go
    Python
    .NET
    

    View Slide

32. • AWS
    • Azure
    • GCP
    • Digital Ocean
    • Cloudflare
    … and more
    Providers
    • Docker
    • Kubernetes
    • OpenStack
    • PostgreSQL
    • New Relic
    

    View Slide

33. var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard",
    });
    C# example
    

    View Slide

34. Still,
    Desired
    State!
    var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard",
    });
    

    View Slide

35. CLI drives the deployment
    How Pulumi
    works
    CLI & engine
    Last
    deployed
    state
    Program.cs
    Language host
    AWS
    Azure
    GCP
    Kubernetes
    new Resource()
    CRUD
    

    View Slide

36. Static Website
    on AWS
    Demo
    26
    

    View Slide

37. Serverless
    Functions
    on Azure
    Infrastructure Components
    33
    

    View Slide

38. Resources to Provision an Azure Function
    Blob
    Plan
    Container
    Storage
    Account
    Function
    App
    

    View Slide

39. • Encapsulate low-level resources into a higher-level abstraction
    • Reuse, shared abstractions, versioning, package management
    • Grouping and structure
    Component Resources
    

    View Slide

40. // A resource group to contain our Azure Functions
    var resourceGroup = new ResourceGroup("functions-rg");
    // A .NET Azure Function
    var dotnet = new ArchiveFunctionApp("http", new ArchiveFunctionAppArgs
    {
    ResourceGroup = resourceGroup,
    Archive = new FileArchive("./dotnet/bin/Debug/netcoreapp3.1/publish")
    });
    .NET Azure Function
    

    View Slide

41. Let’s run pulumi up …
    Type Name Plan
    + pulumi:pulumi:Stack functions-dev create
    + ├─ azure:appservice:ArchiveFunctionApp http create
    + │ ├─ azure:storage:Account http create
    + │ ├─ azure:appservice:Plan http create
    + │ ├─ azure:storage:Container http create
    + │ ├─ azure:storage:ZipBlob http create
    + │ └─ azure:appservice:FunctionApp http create
    + └─ azure:core:ResourceGroup rg create
    

    View Slide

42. // A Node.js Azure Function
    var js = new ArchiveFunctionApp("js", new ArchiveFunctionAppArgs
    {
    ResourceGroup = resourceGroup,
    Archive = new FileArchive("./javascript"),
    Runtime = "node"
    });
    Node.js Azure Function
    

    View Slide

43. var linuxResourceGroup = new ResourceGroup("linux-rg");
    var linuxPlan = new Plan("linux-asp", new PlanArgs
    {
    ResourceGroupName = linuxResourceGroup.Name,
    Kind = "Linux",
    Reserved = true,
    Sku = new PlanSkuArgs { Tier = "Dynamic", Size = "Y1" }
    });
    var python = new ArchiveFunctionApp(...);
    Python Azure Function on Linux
    

    View Slide

44. var linuxResourceGroup = new ResourceGroup("linux-rg");
    var linuxPlan = new Plan("linux-asp", new PlanArgs { ... });
    var python = new ArchiveFunctionApp("py", new ArchiveFunctionAppArgs
    {
    ResourceGroup = linuxResourceGroup,
    Archive = new FileArchive("./python"),
    Plan = linuxPlan,
    Runtime = "python"
    });
    Python Azure Function on Linux
    

    View Slide

45. Cosmos App
    Creating custom abstractions
    37
    

    View Slide

46. Cosmos App
    (1)
    

    View Slide

47. Cosmos App
    (2)
    

    View Slide

48. Cosmos App
    (3)
    

    View Slide

49. Cosmos App
    (4)
    

    View Slide

50. var app = new CosmosApp("functions", new CosmosAppArgs
    {
    ResourceGroup = resourceGroup,
    Locations = ["WestUS", "EastUS", "WestEurope"],
    Factory = global => region =>
    {
    var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs
    {
    ResourceGroupName = resourceGroup.Name,
    Location = region.Location,
    Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"),
    AppSettings = { { "CosmosDBConnection", global.ConnectionString } }
    });
    return new AzureEndpoint(func.AppId);
    }
    });
    Global Applications
    

    View Slide

51. var app = new CosmosApp("functions", new CosmosAppArgs
    {
    ResourceGroup = resourceGroup,
    Locations = ["WestUS", "EastUS", "WestEurope"],
    Factory = global => region =>
    {
    var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs
    {
    ResourceGroupName = resourceGroup.Name,
    Location = region.Location,
    Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"),
    AppSettings = { { "CosmosDBConnection", global.ConnectionString } }
    });
    return new AzureEndpoint(func.AppId);
    }
    });
    Global Applications
    

    View Slide

52. How it works
    Behind the scenes
    41
    

    View Slide

53. N providers, M runtimes
    

    View Slide

54. N providers, M runtimes
    

    View Slide

55. O(N*M) complexity
    

    View Slide

56. O(N+M) complexity
    Intermediate
    Representation
    

    View Slide

57. O(N+M) complexity
    

    View Slide

58. State
    Management
    and Backends
    41
    

    View Slide

59. State is a persisted graph
    State
    Management
    CLI and
    Engine
    Last
    deployed
    state
    Program.cs
    Language host
    AWS
    Azure
    GCP
    Kubernetes
    new Resource()
    CRUD
    Pulumi Service
    S3 / Blob Storage
    Local disk
    

    View Slide

60. Open-source
    Free for any use
    Cross-platform
    Written in Go
    Open-source
    Free for any use
    Can be created by
    1st or 3rd party
    Web API + portal
    “GitHub for infra”
    Free for individuals
    Paid plans for teams
    Useful but not
    required
    CLI Providers SaaS backend
    Pulumi components
    

    View Slide

61. Cloud credentials
    must be configured
    for CLI to run
    AWS profiles
    az login
    Tokens can be set
    via environment
    variables
    CLI Local logins Environment
    Cloud Authentication
    

    View Slide

62. Deployment process
    Allow developers to
    iterate quickly on dev
    environments.
    Gain visibility and change
    management with
    Continuous Delivery.
    

    View Slide

63. Track Infrastructure
    Changes from
    Commit to
    Deployment
    End to End Visibility
    

    View Slide

64. Dependency Visualization
    Understand
    Dependencies
    Between Resources
    

    View Slide

65. APIs and Webhooks
    Integrate
    Infrastructure
    Changes via
    Webhooks
    

    View Slide

66. Role-Based Access Controls
    Manage Access
    Based on Stacks and
    Teams
    

    View Slide

67. Kubernetes
    Deploy to multiple clouds
    45
    

    View Slide

68. Kubernetes layers
    Managed Kubernetes cluster
    Infrastructure Resources (networking, storage, identity)
    Managed Service Managed Service
    Application Application Application
    

    View Slide

69. Organizations, Projects, and Stacks
    Org: acme-corp
    Project: vpc
    Stack: dev
    env: dev
    region: us-east-1
    Stack: prod
    env: prod
    region: us-west-2
    Project: k8s-cluster
    Stack: dev
    env: dev
    region: us-east-1
    Stack: prod
    env: prod
    region: us-west-2
    Project: svc-userprofile
    Stack: dev
    env: dev
    region: us-east-1
    Stack: prod
    env: prod
    region: us-west-2
    Project: svc-email
    Stack: dev
    env: dev
    region: us-east-1
    Stack: prod
    env: prod
    region: us-west-2
    

    View Slide

70. Stack References
    Org: acme-corp
    vpc
    Stack: dev
    env: dev
    region: us-east-1
    k8s-cluster
    Stack: dev
    env: dev
    region: us-east-1
    svc-userprofile
    Stack: dev
    env: dev
    region: us-east-1
    svc-email
    Stack: dev
    env: dev
    region: us-east-1
    

    View Slide

71. Advanced
    scenarios
    Demo
    46
    

    View Slide

72. F#
    48
    

    View Slide

73. let storageAccount =
    Account("sa",
    AccountArgs
    (ResourceGroupName = io resourceGroup.Name,
    AccountReplicationType = input "LRS",
    AccountTier = input "Standard"))
    let sku = PlanSkuArgs(Tier = input "Basic", Size = input "B1")
    let appServicePlan =
    Plan("asp",
    PlanArgs
    (ResourceGroupName = io resourceGroup.Name,
    Kind = input "App",
    Sku = input sku))
    Resources in F#
    

    View Slide

74. let d = deployment "mydep" {
    deploymentSpec {
    replicas 3
    selector { matchLabels appLabels }
    podTemplateSpec {
    metadata { labels appLabels }
    podSpec {
    container {
    name "nginx"
    image "nginx"
    containerPort 80
    }
    }
    }
    }
    }
    Computation Expression DSL?
    https://github.com/pulumi/pulumi/issues/3644
    

    View Slide

75. Quality Control
    50
    

    View Slide

76. Programming
    Languages for
    Infrastructure?
    You sure?
    

    View Slide

77. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var result = await Deployment.TestAsync();
    var resource = result.Resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing
    

    View Slide

78. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var result = await Deployment.TestAsync();
    var resource = result.Resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing
    

    View Slide

79. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var result = await Deployment.TestAsync();
    var resource = result.Resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing
    

    View Slide

80. Policy as Code (.NET coming soon)
    const policies = new PolicyPack("aws", {
    policies: [
    {
    name: "prohibited-public-internet",
    description: "Ingress rules with public internet access are prohibited.",
    enforcementLevel: "mandatory",
    validateResource: validateTypedResource(aws.ec2.SecurityGroup, (sg, args, report) => {
    const publicInternetRules = (sg.ingress || []).find(ingressRule =>
    (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0"));
    if (publicInternetRules) {
    report("Ingress rules with public internet access are prohibited.");
    }
    }),
    },
    ],
    });
    

    View Slide

81. Alternatives
    53
    

    View Slide

82. TypeScript
    Python
    Java
    .NET
    Transpiled to and
    deployed by
    AWS
    CloudFormation
    AWS only Languages CloudFormation
    AWS Cloud Development Kit (CDK)
    

    View Slide

83. var queue = new Queue(this, "MyFirstQueue", new QueueProps
    {
    VisibilityTimeoutSec = 300
    });
    var topic = new Topic(this, "MyFirstTopic", new TopicProps
    {
    DisplayName = "My First Topic Yeah"
    });
    AWS CDK example
    

    View Slide

84. // This is F#
    let myStorage = storageAccount {
    name "mystorage" // set account name
    sku Storage.Sku.PremiumLRS // use Premium LRS
    }
    let myWebApp = webApp {
    name "mysuperwebapp" // set web app name
    sku WebApp.Sku.S1 // use S1 size
    setting "storage_key" myStorage.Key // app setting
    depends_on myStorage // webapp is dependent on storage
    }
    Azure? ARM Generators (ex: Farmer)
    https://github.com/CompositionalIT/farmer
    

    View Slide

85. Conclusions
    55
    

    View Slide

86. Define
    Infrastructure
    as Code
    Making
    Cloud
    Apps?
    

    View Slide

87. Fine-grained
    components
    Better
    Automation
    

    View Slide

88. Familiar language,
    experience,
    toolchain, packages
    Conditionals, loops,
    functions, classes,
    components,
    modules
    Reusable
    components that
    encapsulate
    complex logic and
    provide the right
    level of abstraction
    Developer-friendly Logic Abstractions
    Benefits of Real Code
    

    View Slide

89. м
    Infrastructure
    as Code
    Architecture
    as Code
    

    View Slide

90. • Get Started with Pulumi:
    https://pulumi.com/dotnet
    • Pulumi + .NET blog post:
    “Building Modern Cloud Applications using Pulumi and .NET Core”
    https://devblogs.microsoft.com/dotnet/building-modern-cloud-
    applications-using-pulumi-and-net-core/
    • Pulumi + Cosmos blog post:
    “How to build globally distributed applications with Azure Cosmos
    DB and Pulumi”
    https://azure.microsoft.com/en-gb/blog/how-to-build-globally-
    distributed-applications-with-azure-cosmos-db-and-pulumi/
    Useful Links
    

    View Slide

91. Meetups, Talks and Workshops with Pulumi
    Upcoming Events
    @PulumiCorp
    pulumi
    http://bit.ly/2OQ00Tt
    18 Feb Copenhagen, DK
    18 Feb Kaunas, LT
    19 Feb Malmo, SE
    20 Feb Amsterdam, NL
    21 Feb Madrid, ES
    25 Feb Prague, CZ
    26 Feb Frankfurt, DE
    30 Mar Amsterdam, NL
    

    View Slide

92. Amsterdam – 30 March
    The Pulumi team will provide hands-on instruction
    in a 4-hour workshop showing you how to setup
    your infrastructure for Kubernetes and other useful
    configurations.
    Includes lunch, workshop, labs and happy hour.
    Coming to a city near you
    Pulumi IaC Workshops
    @PulumiCorp
    pulumi
    http://bit.ly/2HgD9wd
    You don’t have to attend KubeCon to join this workshop
    

    View Slide

93. @MikhailShilkov
    https://mikhail.io
    

    View Slide