Azure Infrastructure as C# and F#

Azure cloud offers amazing capabilities for application developers. Cloud applications utilize multiple services and consist of many components, so they are hard to manage without employing Infrastructure as Code. Traditional tools like CLI scripts, ARM templates, and Terraform use text-based formats, which tend to be tedious, repetitive, and cumbersome to reuse. What if instead of configuration files you could use your favorite programming languages like C# or F#? See how you can bring your developer tools like code completion, types, components, and abstractions to cloud infrastructure definition.

Transcript

1. View Slide

2. View Slide

3. DevOps in Cloud age
   Developer
   Cloud Infrastructure
   User
   IT
   

   View Slide

4. About me
   

   View Slide

5. Agenda
   

   View Slide

6. Agenda
   

   View Slide

7. Agenda
   

   View Slide

8. Cloud Automation
   49
   

   View Slide

9. Unified Management API
   Management API (Azure Resource Manager)
   Cloud Service Cloud Service Cloud Service
   Web Portal CLI SDK 3rd Party
   

   View Slide

10. REST calls
    PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/
    {resourceGroupName}/providers/Microsoft.Storage/storageAccounts/mystorageaccount
    ?api-version=2016-01-01
    {
    "location": "westus",
    "properties": {
    }
    "sku": {
    "name": "Standard_LRS"
    },
    "kind": "Storage"
    }
    

    View Slide

11. var storageAccount = az.StorageAccounts.Define(storageAccountName)
    .WithRegion(Region.US_EAST)
    .WithNewResourceGroup(resourceGroupName)
    .Create();
    SDKs to call those REST endpoints
    

    View Slide

12. az storage account create \
    --location westus \
    --name samplesa \
    --resource-group myrg \
    --sku LRS
    Command-line Tools
    

    View Slide

13. Step-by-step
    provisioning
    process
    Depend on both
    the current and
    the target state
    How to handle
    failures?
    Imperative Complex Error-prone
    SDK and Scripting: Challenges
    

    View Slide

14. Desired State Configuration
    51
    

    View Slide

15. Desired State Configuration
    Target Current
    Tool
    

    View Slide

16. Managing Resource Graphs
    Target Current
    Tool
    

    View Slide

17. Be able to describe
    the desired state in
    full, concisely, and
    correctly
    Determine the
    correct order of
    creation and
    destruction
    Preview the changes
    before running the
    deployment
    Expressiveness Correctness Transparency
    Challenges
    

    View Slide

18. ARM Templates: JSON
    "resources": [
    {
    "apiVersion": "2016-01-01",
    "type": "Microsoft.Storage/storageAccounts",
    "name": "mystorageaccount",
    "location": "westus",
    "sku": {
    "name": "Standard_LRS"
    },
    "kind": "Storage",
    "properties": {
    }
    }
    ]
    

    View Slide

19. resource "azurerm_storage_account" "sa" {
    name = "mysa"
    location = azurerm_resource_group.rg.location
    resource_group_name = azurerm_resource_group.rg.name
    account_tier = "Standard"
    account_replication_type = "LRS"
    }
    Terraform: DSL (HashiCorp Configuration Language)
    

    View Slide

20. Lack of fast feedback
    about errors
    Which options need
    to be filled and
    which values are
    possible
    Thousands of lines of
    markup are hard to
    write, read, and
    navigate
    Tooling Discovery Size
    Markup Authoring Experience
    

    View Slide

21. Example: Code to infrastructure ratio
    Storage
    Table “URLs”
    Function
    “Add URL”
    Function
    “Open URL”
    Lines of code
    Function Code ~20
    ARM Templates ~160
    Terraform ~140
    Estimates based on public examples
    

    View Slide

22. {
    "condition": "[equals(parameters('production'), 'Yes')]",
    "type": "Microsoft.Compute/availabilitySets",
    "apiVersion": "2017-03-30",
    "name": "[variables('availabilitySetName')]",
    "location": "[resourceGroup().location]",
    "properties": {
    "platformFaultDomainCount": 2,
    "platformUpdateDomainCount": 3
    }
    }
    Conditionals
    Example for Azure RM Templates
    

    View Slide

23. locals {
    files = ["index.php","main.php"]
    }
    resource "aws_s3_bucket_object" "files" {
    count = "${length(local.files)}"
    key = "${local.files[count.index]}"
    bucket = "${aws_s3_bucket.examplebucket.id}"
    source = "./src/${local.files[count.index]}"
    }
    Loops
    Example for Terraform
    

    View Slide

24. Developer
    Tools
    

    View Slide

25. Pulumi
    56
    

    View Slide

26. General-Purpose Programming Languages
    Node.js
    Go
    Python
    .NET
    

    View Slide

27. Providers
    

    View Slide

28. var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard",
    });
    C# example
    

    View Slide

29. Still,
    Desired
    State!
    var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard",
    });
    

    View Slide

30. How Pulumi works
    CLI & engine
    Last
    deployed
    state
    Program.cs
    Language host
    Azure
    AWS
    GCP
    Kubernetes
    new
    Resource()
    CRUD
    

    View Slide

31. Static Website
    on Azure
    59
    

    View Slide

32. Components:
    Serverless Functions
    04
    

    View Slide

33. Resources to Provision an Azure Function
    Blob
    Plan
    Container
    Storage
    Account
    Function
    App
    

    View Slide

34. Component Resources
    

    View Slide

35. // A resource group to contain our Azure Functions
    var resourceGroup = new ResourceGroup("functions-rg");
    // A .NET Azure Function
    var dotnet = new ArchiveFunctionApp("http", new ArchiveFunctionAppArgs
    {
    ResourceGroup = resourceGroup,
    Archive = new FileArchive("./dotnet/bin/Debug/netcoreapp3.1/publish")
    });
    .NET Azure Function
    

    View Slide

36. Let’s run pulumi up …
    Type Name Plan
    + pulumi:pulumi:Stack functions-dev create
    + ├─ azure:appservice:ArchiveFunctionApp http create
    + │ ├─ azure:storage:Account http create
    + │ ├─ azure:appservice:Plan http create
    + │ ├─ azure:storage:Container http create
    + │ ├─ azure:storage:ZipBlob http create
    + │ └─ azure:appservice:FunctionApp http create
    + └─ azure:core:ResourceGroup rg create
    

    View Slide

37. // A Node.js Azure Function
    var js = new ArchiveFunctionApp("js", new ArchiveFunctionAppArgs
    {
    ResourceGroup = resourceGroup,
    Archive = new FileArchive("./javascript"),
    Runtime = "node"
    });
    Node.js Azure Function
    

    View Slide

38. var linuxResourceGroup = new ResourceGroup("linux-rg");
    var linuxPlan = new Plan("linux-asp", new PlanArgs
    {
    ResourceGroupName = linuxResourceGroup.Name,
    Kind = "Linux",
    Reserved = true,
    Sku = new PlanSkuArgs { Tier = "Dynamic", Size = "Y1" }
    });
    var python = new ArchiveFunctionApp(...);
    Python Azure Function on Linux
    

    View Slide

39. var linuxResourceGroup = new ResourceGroup("linux-rg");
    var linuxPlan = new Plan("linux-asp", new PlanArgs { ... });
    var python = new ArchiveFunctionApp("py", new ArchiveFunctionAppArgs
    {
    ResourceGroup = linuxResourceGroup,
    Archive = new FileArchive("./python"),
    Plan = linuxPlan,
    Runtime = "python"
    });
    Python Azure Function on Linux
    

    View Slide

40. Cosmos App
    08
    

    View Slide

41. Cosmos App (1)
    

    View Slide

42. Cosmos App (2)
    

    View Slide

43. Cosmos App (3)
    

    View Slide

44. Cosmos App (4)
    

    View Slide

45. var app = new CosmosApp("functions", new CosmosAppArgs
    {
    ResourceGroup = resourceGroup,
    Locations = ["WestUS", "EastUS", "WestEurope"],
    Factory = global => region =>
    {
    var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs
    {
    ResourceGroupName = resourceGroup.Name,
    Location = region.Location,
    Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"),
    AppSettings = { { "CosmosDBConnection", global.ConnectionString } }
    });
    return new AzureEndpoint(func.AppId);
    }
    });
    Global Applications
    

    View Slide

46. var app = new CosmosApp("functions", new CosmosAppArgs
    {
    ResourceGroup = resourceGroup,
    Locations = ["WestUS", "EastUS", "WestEurope"],
    Factory = global => region =>
    {
    var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs
    {
    ResourceGroupName = resourceGroup.Name,
    Location = region.Location,
    Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"),
    AppSettings = { { "CosmosDBConnection", global.ConnectionString } }
    });
    return new AzureEndpoint(func.AppId);
    }
    });
    Global Applications
    

    View Slide

47. Provider models
    12
    

    View Slide

48. N providers, M runtimes
    

    View Slide

49. N providers, M runtimes
    

    View Slide

50. O(N*M) complexity
    

    View Slide

51. O(N+M) complexity
    Intermediate
    Representation
    

    View Slide

52. O(N+M) complexity
    

    View Slide

53. State Management and
    Backends
    14
    

    View Slide

54. State Management
    CLI and
    Engine
    Last
    deployed
    state
    Program.cs
    Language host
    AWS
    Azure
    GCP
    Kubernetes
    new Resource()
    CRUD
    Pulumi Service
    Blob Storage / S3
    Local disk
    

    View Slide

55. Open-source
    Free for any use
    Cross-platform
    Written in Go
    Open-source
    Free for any use
    Can be created by
    1st or 3rd party
    Web API + portal
    “GitHub for infra”
    Free for individuals
    Paid plans for teams
    Useful but not
    required
    CLI Providers SaaS backend
    Pulumi components
    

    View Slide

56. Cloud credentials
    must be configured
    for CLI to run
    az login Tokens can be set via
    environment
    variables
    CLI Local logins Environment
    Cloud Authentication
    

    View Slide

57. Deployment process
    Allow developers to
    iterate quickly on dev
    environments.
    Gain visibility and change
    management with
    Continuous Delivery.
    

    View Slide

58. Kubernetes
    17
    

    View Slide

59. Kubernetes layers
    Managed Kubernetes cluster (AKS)
    Infrastructure Resources (networking, storage, identity)
    Managed Service Managed Service
    Application Application Application
    

    View Slide

60. Organizations, Projects, and Stacks
    Org: acme-corp
    Project: vpc
    Stack: dev
    env: dev
    region: us-east-1
    Stack: prod
    env: prod
    region: us-west-2
    Project: k8s-cluster
    Stack: dev
    env: dev
    region: us-east-1
    Stack: prod
    env: prod
    region: us-west-2
    Project: svc-userprofile
    Stack: dev
    env: dev
    region: us-east-1
    Stack: prod
    env: prod
    region: us-west-2
    Project: svc-email
    Stack: dev
    env: dev
    region: us-east-1
    Stack: prod
    env: prod
    region: us-west-2
    

    View Slide

61. Stack References
    Org: acme-corp
    vpc
    Stack: dev
    env: dev
    region: us-east-1
    k8s-cluster
    Stack: dev
    env: dev
    region: us-east-1
    svc-userprofile
    Stack: dev
    env: dev
    region: us-east-1
    svc-email
    Stack: dev
    env: dev
    region: us-east-1
    

    View Slide

62. Advanced scenarios
    19
    

    View Slide

63. F#
    22
    

    View Slide

64. let storageAccount =
    Account("sa",
    AccountArgs
    (ResourceGroupName = io resourceGroup.Name,
    AccountReplicationType = input "LRS",
    AccountTier = input "Standard"))
    let sku = PlanSkuArgs(Tier = input "Basic", Size = input "B1")
    let appServicePlan =
    Plan("asp",
    PlanArgs
    (ResourceGroupName = io resourceGroup.Name,
    Kind = input "App",
    Sku = input sku))
    Resources in F#
    

    View Slide

65. let d = deployment "mydep" {
    deploymentSpec {
    replicas 3
    selector { matchLabels appLabels }
    podTemplateSpec {
    metadata { labels appLabels }
    podSpec {
    container {
    name "nginx"
    image "nginx"
    containerPort 80
    }
    }
    }
    }
    }
    Computation Expression DSL?
    https://github.com/pulumi/pulumi/issues/3644
    

    View Slide

66. Quality Control
    24
    

    View Slide

67. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var result = await Deployment.TestAsync();
    var resource = result.Resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing
    

    View Slide

68. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var result = await Deployment.TestAsync();
    var resource = result.Resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing
    

    View Slide

69. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var result = await Deployment.TestAsync();
    var resource = result.Resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing
    

    View Slide

70. Policy as Code (.NET coming later)
    const policies = new PolicyPack("aws", {
    policies: [
    {
    name: "prohibited-public-internet",
    description: "Ingress rules with public internet access are prohibited.",
    enforcementLevel: "mandatory",
    validateResource: validateTypedResource(aws.ec2.SecurityGroup, (sg, args, report) => {
    const publicInternetRules = (sg.ingress || []).find(ingressRule =>
    (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0"));
    if (publicInternetRules) {
    report("Ingress rules with public internet access are prohibited.");
    }
    }),
    },
    ],
    });
    

    View Slide

71. Conclusions
    27
    

    View Slide

72. Working on
    Cloud Apps?
    Define your
    infrastructure
    as code!
    

    View Slide

73. Fine-grained
    components
    Better
    Automation
    

    View Slide

74. Familiar language,
    experience,
    toolchain, packages
    Conditionals, loops,
    functions, classes,
    components,
    modules
    Reusable
    components that
    encapsulate complex
    logic and provide the
    right level of
    abstraction
    Developer-friendly Logic Abstractions
    Benefits of Real Code
    

    View Slide

75. м
    Infrastructure as
    Code
    Architecture as
    Code
    

    View Slide

76. Useful Links
    http://bit.ly/pulumilinks
    

    View Slide