Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Azure Infrastructure as C# and F#

Azure Infrastructure as C# and F#

Azure cloud offers amazing capabilities for application developers. Cloud applications utilize multiple services and consist of many components, so they are hard to manage without employing Infrastructure as Code. Traditional tools like CLI scripts, ARM templates, and Terraform use text-based formats, which tend to be tedious, repetitive, and cumbersome to reuse. What if instead of configuration files you could use your favorite programming languages like C# or F#? See how you can bring your developer tools like code completion, types, components, and abstractions to cloud infrastructure definition.

Mikhail Shilkov

February 25, 2020
Tweet

More Decks by Mikhail Shilkov

Other Decks in Programming

Transcript

  1. View Slide

  2. View Slide

  3. DevOps in Cloud age
    Developer
    Cloud Infrastructure
    User
    IT

    View Slide

  4. About me

    View Slide

  5. Agenda

    View Slide

  6. Agenda

    View Slide

  7. Agenda

    View Slide

  8. Cloud Automation
    49

    View Slide

  9. Unified Management API
    Management API (Azure Resource Manager)
    Cloud Service Cloud Service Cloud Service
    Web Portal CLI SDK 3rd Party

    View Slide

  10. REST calls
    PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/
    {resourceGroupName}/providers/Microsoft.Storage/storageAccounts/mystorageaccount
    ?api-version=2016-01-01
    {
    "location": "westus",
    "properties": {
    }
    "sku": {
    "name": "Standard_LRS"
    },
    "kind": "Storage"
    }

    View Slide

  11. var storageAccount = az.StorageAccounts.Define(storageAccountName)
    .WithRegion(Region.US_EAST)
    .WithNewResourceGroup(resourceGroupName)
    .Create();
    SDKs to call those REST endpoints

    View Slide

  12. az storage account create \
    --location westus \
    --name samplesa \
    --resource-group myrg \
    --sku LRS
    Command-line Tools

    View Slide

  13. Step-by-step
    provisioning
    process
    Depend on both
    the current and
    the target state
    How to handle
    failures?
    Imperative Complex Error-prone
    SDK and Scripting: Challenges

    View Slide

  14. Desired State Configuration
    51

    View Slide

  15. Desired State Configuration
    Target Current
    Tool

    View Slide

  16. Managing Resource Graphs
    Target Current
    Tool

    View Slide

  17. Be able to describe
    the desired state in
    full, concisely, and
    correctly
    Determine the
    correct order of
    creation and
    destruction
    Preview the changes
    before running the
    deployment
    Expressiveness Correctness Transparency
    Challenges

    View Slide

  18. ARM Templates: JSON
    "resources": [
    {
    "apiVersion": "2016-01-01",
    "type": "Microsoft.Storage/storageAccounts",
    "name": "mystorageaccount",
    "location": "westus",
    "sku": {
    "name": "Standard_LRS"
    },
    "kind": "Storage",
    "properties": {
    }
    }
    ]

    View Slide

  19. resource "azurerm_storage_account" "sa" {
    name = "mysa"
    location = azurerm_resource_group.rg.location
    resource_group_name = azurerm_resource_group.rg.name
    account_tier = "Standard"
    account_replication_type = "LRS"
    }
    Terraform: DSL (HashiCorp Configuration Language)

    View Slide

  20. Lack of fast feedback
    about errors
    Which options need
    to be filled and
    which values are
    possible
    Thousands of lines of
    markup are hard to
    write, read, and
    navigate
    Tooling Discovery Size
    Markup Authoring Experience

    View Slide

  21. Example: Code to infrastructure ratio
    Storage
    Table “URLs”
    Function
    “Add URL”
    Function
    “Open URL”
    Lines of code
    Function Code ~20
    ARM Templates ~160
    Terraform ~140
    Estimates based on public examples

    View Slide

  22. {
    "condition": "[equals(parameters('production'), 'Yes')]",
    "type": "Microsoft.Compute/availabilitySets",
    "apiVersion": "2017-03-30",
    "name": "[variables('availabilitySetName')]",
    "location": "[resourceGroup().location]",
    "properties": {
    "platformFaultDomainCount": 2,
    "platformUpdateDomainCount": 3
    }
    }
    Conditionals
    Example for Azure RM Templates

    View Slide

  23. locals {
    files = ["index.php","main.php"]
    }
    resource "aws_s3_bucket_object" "files" {
    count = "${length(local.files)}"
    key = "${local.files[count.index]}"
    bucket = "${aws_s3_bucket.examplebucket.id}"
    source = "./src/${local.files[count.index]}"
    }
    Loops
    Example for Terraform

    View Slide

  24. Developer
    Tools

    View Slide

  25. Pulumi
    56

    View Slide

  26. General-Purpose Programming Languages
    Node.js
    Go
    Python
    .NET

    View Slide

  27. Providers

    View Slide

  28. var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard",
    });
    C# example

    View Slide

  29. Still,
    Desired
    State!
    var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard",
    });

    View Slide

  30. How Pulumi works
    CLI & engine
    Last
    deployed
    state
    Program.cs
    Language host
    Azure
    AWS
    GCP
    Kubernetes
    new
    Resource()
    CRUD

    View Slide

  31. Static Website
    on Azure
    59

    View Slide

  32. Components:
    Serverless Functions
    04

    View Slide

  33. Resources to Provision an Azure Function
    Blob
    Plan
    Container
    Storage
    Account
    Function
    App

    View Slide

  34. Component Resources

    View Slide

  35. // A resource group to contain our Azure Functions
    var resourceGroup = new ResourceGroup("functions-rg");
    // A .NET Azure Function
    var dotnet = new ArchiveFunctionApp("http", new ArchiveFunctionAppArgs
    {
    ResourceGroup = resourceGroup,
    Archive = new FileArchive("./dotnet/bin/Debug/netcoreapp3.1/publish")
    });
    .NET Azure Function

    View Slide

  36. Let’s run pulumi up …
    Type Name Plan
    + pulumi:pulumi:Stack functions-dev create
    + ├─ azure:appservice:ArchiveFunctionApp http create
    + │ ├─ azure:storage:Account http create
    + │ ├─ azure:appservice:Plan http create
    + │ ├─ azure:storage:Container http create
    + │ ├─ azure:storage:ZipBlob http create
    + │ └─ azure:appservice:FunctionApp http create
    + └─ azure:core:ResourceGroup rg create

    View Slide

  37. // A Node.js Azure Function
    var js = new ArchiveFunctionApp("js", new ArchiveFunctionAppArgs
    {
    ResourceGroup = resourceGroup,
    Archive = new FileArchive("./javascript"),
    Runtime = "node"
    });
    Node.js Azure Function

    View Slide

  38. var linuxResourceGroup = new ResourceGroup("linux-rg");
    var linuxPlan = new Plan("linux-asp", new PlanArgs
    {
    ResourceGroupName = linuxResourceGroup.Name,
    Kind = "Linux",
    Reserved = true,
    Sku = new PlanSkuArgs { Tier = "Dynamic", Size = "Y1" }
    });
    var python = new ArchiveFunctionApp(...);
    Python Azure Function on Linux

    View Slide

  39. var linuxResourceGroup = new ResourceGroup("linux-rg");
    var linuxPlan = new Plan("linux-asp", new PlanArgs { ... });
    var python = new ArchiveFunctionApp("py", new ArchiveFunctionAppArgs
    {
    ResourceGroup = linuxResourceGroup,
    Archive = new FileArchive("./python"),
    Plan = linuxPlan,
    Runtime = "python"
    });
    Python Azure Function on Linux

    View Slide

  40. Cosmos App
    08

    View Slide

  41. Cosmos App (1)

    View Slide

  42. Cosmos App (2)

    View Slide

  43. Cosmos App (3)

    View Slide

  44. Cosmos App (4)

    View Slide

  45. var app = new CosmosApp("functions", new CosmosAppArgs
    {
    ResourceGroup = resourceGroup,
    Locations = ["WestUS", "EastUS", "WestEurope"],
    Factory = global => region =>
    {
    var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs
    {
    ResourceGroupName = resourceGroup.Name,
    Location = region.Location,
    Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"),
    AppSettings = { { "CosmosDBConnection", global.ConnectionString } }
    });
    return new AzureEndpoint(func.AppId);
    }
    });
    Global Applications

    View Slide

  46. var app = new CosmosApp("functions", new CosmosAppArgs
    {
    ResourceGroup = resourceGroup,
    Locations = ["WestUS", "EastUS", "WestEurope"],
    Factory = global => region =>
    {
    var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs
    {
    ResourceGroupName = resourceGroup.Name,
    Location = region.Location,
    Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"),
    AppSettings = { { "CosmosDBConnection", global.ConnectionString } }
    });
    return new AzureEndpoint(func.AppId);
    }
    });
    Global Applications

    View Slide

  47. Provider models
    12

    View Slide

  48. N providers, M runtimes

    View Slide

  49. N providers, M runtimes

    View Slide

  50. O(N*M) complexity

    View Slide

  51. O(N+M) complexity
    Intermediate
    Representation

    View Slide

  52. O(N+M) complexity

    View Slide

  53. State Management and
    Backends
    14

    View Slide

  54. State Management
    CLI and
    Engine
    Last
    deployed
    state
    Program.cs
    Language host
    AWS
    Azure
    GCP
    Kubernetes
    new Resource()
    CRUD
    Pulumi Service
    Blob Storage / S3
    Local disk

    View Slide

  55. Open-source
    Free for any use
    Cross-platform
    Written in Go
    Open-source
    Free for any use
    Can be created by
    1st or 3rd party
    Web API + portal
    “GitHub for infra”
    Free for individuals
    Paid plans for teams
    Useful but not
    required
    CLI Providers SaaS backend
    Pulumi components

    View Slide

  56. Cloud credentials
    must be configured
    for CLI to run
    az login Tokens can be set via
    environment
    variables
    CLI Local logins Environment
    Cloud Authentication

    View Slide

  57. Deployment process
    Allow developers to
    iterate quickly on dev
    environments.
    Gain visibility and change
    management with
    Continuous Delivery.

    View Slide

  58. Kubernetes
    17

    View Slide

  59. Kubernetes layers
    Managed Kubernetes cluster (AKS)
    Infrastructure Resources (networking, storage, identity)
    Managed Service Managed Service
    Application Application Application

    View Slide

  60. Organizations, Projects, and Stacks
    Org: acme-corp
    Project: vpc
    Stack: dev
    env: dev
    region: us-east-1
    Stack: prod
    env: prod
    region: us-west-2
    Project: k8s-cluster
    Stack: dev
    env: dev
    region: us-east-1
    Stack: prod
    env: prod
    region: us-west-2
    Project: svc-userprofile
    Stack: dev
    env: dev
    region: us-east-1
    Stack: prod
    env: prod
    region: us-west-2
    Project: svc-email
    Stack: dev
    env: dev
    region: us-east-1
    Stack: prod
    env: prod
    region: us-west-2

    View Slide

  61. Stack References
    Org: acme-corp
    vpc
    Stack: dev
    env: dev
    region: us-east-1
    k8s-cluster
    Stack: dev
    env: dev
    region: us-east-1
    svc-userprofile
    Stack: dev
    env: dev
    region: us-east-1
    svc-email
    Stack: dev
    env: dev
    region: us-east-1

    View Slide

  62. Advanced scenarios
    19

    View Slide

  63. F#
    22

    View Slide

  64. let storageAccount =
    Account("sa",
    AccountArgs
    (ResourceGroupName = io resourceGroup.Name,
    AccountReplicationType = input "LRS",
    AccountTier = input "Standard"))
    let sku = PlanSkuArgs(Tier = input "Basic", Size = input "B1")
    let appServicePlan =
    Plan("asp",
    PlanArgs
    (ResourceGroupName = io resourceGroup.Name,
    Kind = input "App",
    Sku = input sku))
    Resources in F#

    View Slide

  65. let d = deployment "mydep" {
    deploymentSpec {
    replicas 3
    selector { matchLabels appLabels }
    podTemplateSpec {
    metadata { labels appLabels }
    podSpec {
    container {
    name "nginx"
    image "nginx"
    containerPort 80
    }
    }
    }
    }
    }
    Computation Expression DSL?
    https://github.com/pulumi/pulumi/issues/3644

    View Slide

  66. Quality Control
    24

    View Slide

  67. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var result = await Deployment.TestAsync();
    var resource = result.Resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing

    View Slide

  68. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var result = await Deployment.TestAsync();
    var resource = result.Resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing

    View Slide

  69. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var result = await Deployment.TestAsync();
    var resource = result.Resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing

    View Slide

  70. Policy as Code (.NET coming later)
    const policies = new PolicyPack("aws", {
    policies: [
    {
    name: "prohibited-public-internet",
    description: "Ingress rules with public internet access are prohibited.",
    enforcementLevel: "mandatory",
    validateResource: validateTypedResource(aws.ec2.SecurityGroup, (sg, args, report) => {
    const publicInternetRules = (sg.ingress || []).find(ingressRule =>
    (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0"));
    if (publicInternetRules) {
    report("Ingress rules with public internet access are prohibited.");
    }
    }),
    },
    ],
    });

    View Slide

  71. Conclusions
    27

    View Slide

  72. Working on
    Cloud Apps?
    Define your
    infrastructure
    as code!

    View Slide

  73. Fine-grained
    components
    Better
    Automation

    View Slide

  74. Familiar language,
    experience,
    toolchain, packages
    Conditionals, loops,
    functions, classes,
    components,
    modules
    Reusable
    components that
    encapsulate complex
    logic and provide the
    right level of
    abstraction
    Developer-friendly Logic Abstractions
    Benefits of Real Code

    View Slide

  75. м
    Infrastructure as
    Code
    Architecture as
    Code

    View Slide

  76. Useful Links
    http://bit.ly/pulumilinks

    View Slide