Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Azure Infrastructure as C# and F#

Azure Infrastructure as C# and F#

Azure cloud platform offers amazing capabilities for application developers. Cloud apps utilize multiple services and consist of many components, so they are hard to manage without employing Infrastructure as Code.

Traditional tools like ARM templates and Terraform use text-based formats, which tend to be tedious, repetitive, and cumbersome to reuse.

What if instead of configuration files you could use your favourite programming languages like C# or F#? See how you can bring your developer tools like code completion, types, components, and abstractions to cloud infrastructure definition.

Mikhail Shilkov

March 31, 2020
Tweet

More Decks by Mikhail Shilkov

Other Decks in Programming

Transcript

  1. Azure Infrastructure as C# and F#
    Mikhail Shilkov

    View Slide

  2. DevOps in the cloud age
    Developer
    Cloud Infrastructure
    User
    IT

    View Slide

  3. ● Pulumi: Modern Infrastructure as Code
    ● Open source and free
    ● Use .NET languages to manage cloud resources
    ● Desired state configuration
    ● Tools that you love
    ● Reusable abstractions
    ● Automated testing
    ● Multi-cloud
    Agenda

    View Slide

  4. About me
    • Software engineer at Pulumi
    .NET SDK, Azure, Core platform
    • Serverless
    • Functional programming, F#
    • Microsoft Azure MVP
    @MikhailShilkov
    [email protected]
    https://mikhail.io

    View Slide

  5. .NET for Cloud Resources
    Provision infrastructure anywhere using C#, F#, or VB.NET

    View Slide

  6. General-purpose Programming Languages

    View Slide

  7. Providers
    ● Azure
    ● AWS
    ● GCP
    ● Digital Ocean
    ● Cloudflare
    … and more
    ● Docker
    ● Kubernetes
    ● OpenStack
    ● PostgreSQL
    ● New Relic

    View Slide

  8. Static Website
    on Azure
    Demo

    View Slide

  9. Desired State Configuration
    Maintainable infrastructure
    10

    View Slide

  10. Unified Management API
    Management API (Azure Resource Manager)
    Cloud Service Cloud Service Cloud Service
    Web Portal CLI SDK 3rd Party

    View Slide

  11. REST calls
    PUT
    https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/
    {resourceGroupName}/providers/Microsoft.Storage/storageAccounts/mystorageac
    count?api-version=2016-01-01
    {
    "location": "westus",
    "properties": {
    }
    "sku": {
    "name": "Standard_LRS"
    },
    "kind": "Storage"
    }

    View Slide

  12. var storageAccount = az.StorageAccounts.Define(storageAccountName)
    .WithRegion(Region.US_EAST)
    .WithNewResourceGroup(resourceGroupName)
    .Create();
    SDKs to call those REST endpoints

    View Slide

  13. az storage account create \
    --location westus \
    --name samplesa \
    --resource-group myrg \
    --sku LRS
    Command-line Tools

    View Slide

  14. SDK and Scripting: Challenges
    14
    Imperative
    Step-by-step
    provisioning process
    Complex
    Depend on both the
    current and the target
    state
    Error-prone
    How to handle failures?

    View Slide

  15. Desired State Configuration
    Target Current
    Tool

    View Slide

  16. Managing Resource Graphs
    Target Current
    Tool

    View Slide

  17. Markup-based Languages
    17
    AWS CloudFormation
    YAML
    Azure RM Templates
    JSON
    Terraform
    HCL
    "resources": [
    {
    "apiVersion": "2016-01-01",
    "type": "Microsoft.Storage/storageAccounts",
    "name": "mystorageaccount",
    "location": "westus",
    "sku": {
    "name": "Standard_LRS"
    },
    "kind": "Storage",
    "properties": {
    }
    }
    ]
    Resources:
    S3BucketForURLs:
    Type: "AWS::S3::Bucket"
    DeletionPolicy: Delete
    Properties:
    BucketName: !If [ "CreateNewBucket", !Ref
    WebsiteConfiguration:
    IndexDocument: "index.html"
    LifecycleConfiguration:
    Rules:
    -
    Id: DisposeShortUrls
    ExpirationInDays: !Ref URLExpiration
    Prefix: "u"
    Status: Enabled
    resource "azurerm_storage_account" "sa" {
    name = "mysa"
    location = azurerm_resource_group
    .rg.location
    resource_group_name = azurerm_resource_group
    .rg.name
    account_tier = "Standard"
    account_replication_type = "LRS"
    }
    Every tool has its own language,
    own tools, own conventions

    View Slide

  18. Markup Authoring Experience
    18
    Tooling
    Lack of tooling on day 1.
    No fast feedback from
    editors.
    Discovery
    Which properties can be
    set, and which values are
    valid?
    Size
    Thousands of lines of
    markup are hard to write,
    read, and maintain

    View Slide

  19. Example App: Serverless URL Shortener
    KV Store
    Table “URLs”
    Function
    “Add URL”
    Function
    “Open URL”
    Tool Lines of code
    Function Code ~20
    CloudFormation ~240
    ARM Templates ~160
    Terraform ~220
    Estimates based on public examples

    View Slide

  20. var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard"
    });
    C# Example

    View Slide

  21. Desired
    State!
    var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard"
    });

    View Slide

  22. How Pulumi Works
    CLI & engine
    Last
    deployed
    state
    Program.cs
    Language host
    AWS
    Azure
    GCP
    Kubernetes
    new
    Resource()
    CRUD

    View Slide

  23. Tools That You Love
    Developers can apply their existing skills to infrastructure

    View Slide

  24. Pulumi ❤ .NET
    ● .NET Core 3.1
    ● OS: Windows, macOS, Linux
    ● Language: C#, F#, VB.NET
    ● Editor and IDE: Visual Studio, Code, Rider
    ● IntelliSense, ReSharper, StyleCop, DocFX
    ● Package Manager: NuGet, MyGet, Paket
    ● NUnit, xUnit.net, Moq

    View Slide

  25. var storageAccount = new Account("sa", new AccountArgs {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard"
    });
    var appServicePlan = new Plan("asp", new PlanArgs {
    ResourceGroupName = resourceGroup.Name,
    Kind = "App",
    Sku = new PlanSkuArgs { Tier = "Basic", Size = "B1” }
    });
    Cloud Resources with C#

    View Slide

  26. let storageAccount =
    Account("sa",
    AccountArgs
    (ResourceGroupName = io resourceGroup.Name,
    AccountReplicationType = input "LRS",
    AccountTier = input "Standard"))
    let sku = PlanSkuArgs(Tier = input "Basic", Size = input "B1")
    let appServicePlan =
    Plan("asp",
    PlanArgs
    (ResourceGroupName = io resourceGroup.Name,
    Kind = input "App",
    Sku = input sku))
    Cloud Resources with F#

    View Slide

  27. ' Create an Azure Resource Group
    Dim resourceGroup = New ResourceGroup("resourceGroup")
    ' Create an Azure Storage Account
    Dim storageAccount = New Account("storage", New AccountArgs With {
    .ResourceGroupName = resourceGroup.Name,
    .AccountReplicationType = "LRS",
    .AccountTier = "Standard"
    })
    Cloud Resources with VB.NET

    View Slide

  28. Reusable Abstractions
    Infrastructure Components

    View Slide

  29. Resources to Provision an Azure Function
    Blob
    App
    Plan
    Container
    Storage
    Account
    Function
    App

    View Slide

  30. // A resource group to contain our Azure Functions
    var resourceGroup = new ResourceGroup("functions-rg");
    // A .NET Azure Function
    var dotnet = new ArchiveFunctionApp("http", new ArchiveFunctionAppArgs
    {
    ResourceGroup = resourceGroup,
    Archive = new FileArchive("./dotnet/bin/Debug/netcoreapp3.1/publish")
    });
    .NET Azure Function

    View Slide

  31. Let’s run pulumi up …
    Type Name Plan
    + pulumi:pulumi:Stack functions-dev create
    + ├─ azure:appservice:ArchiveFunctionApp http create
    + │ ├─ azure:storage:Account http create
    + │ ├─ azure:appservice:Plan http create
    + │ ├─ azure:storage:Container http create
    + │ ├─ azure:storage:ZipBlob http create
    + │ └─ azure:appservice:FunctionApp http create
    + └─ azure:core:ResourceGroup rg create

    View Slide

  32. Serverless:
    Azure Functions
    Demo

    View Slide

  33. Building a Cosmos App
    Create custom abstractions

    View Slide

  34. Cosmos App (1)

    View Slide

  35. Cosmos App (2)

    View Slide

  36. Cosmos App (3)

    View Slide

  37. Cosmos App (4)

    View Slide

  38. var app = new CosmosApp("functions", new CosmosAppArgs
    {
    ResourceGroup = resourceGroup,
    Locations = ["WestUS", "EastUS", "WestEurope"],
    Factory = global => region =>
    {
    var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs
    {
    ResourceGroupName = resourceGroup.Name,
    Location = region.Location,
    Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"),
    AppSettings = { { "CosmosDBConnection", global.ConnectionString } }
    });
    return new AzureEndpoint(func.AppId);
    }
    });
    Global Applications

    View Slide

  39. var app = new CosmosApp("functions", new CosmosAppArgs
    {
    ResourceGroup = resourceGroup,
    Locations = ["WestUS", "EastUS", "WestEurope"],
    Factory = global => region =>
    {
    var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs
    {
    ResourceGroupName = resourceGroup.Name,
    Location = region.Location,
    Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"),
    AppSettings = { { "CosmosDBConnection", global.ConnectionString } }
    });
    return new AzureEndpoint(func.AppId);
    }
    });
    Global Applications

    View Slide

  40. Automated Testing
    Unit testing and policies

    View Slide

  41. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var resources = await Deployment.TestAsync();
    var resource = resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing

    View Slide

  42. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var resources = await Deployment.TestAsync();
    var resource = resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing

    View Slide

  43. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var resources = await Deployment.TestAsync();
    var resource = resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing

    View Slide

  44. Policy as Code (.NET coming later)
    const policies = new PolicyPack("network-policies", {
    policies: [
    {
    name: "prohibited-public-internet",
    description: "Ingress rules with public internet access are prohibited.",
    enforcementLevel: "mandatory",
    validateResource: validateTypedResource(SecurityGroup, (sg, args, report) => {
    const publicInternetRules = (sg.ingress || []).find(ingressRule =>
    (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0"));
    if (publicInternetRules) {
    report("Ingress rules with public internet access are prohibited.");
    }
    }),
    },
    ],
    });

    View Slide

  45. Kubernetes
    Deploy to multiple clouds

    View Slide

  46. Cloud Transition
    Kubernetes
    Azure
    Functions
    Azure
    Storage
    Azure
    Analytics
    Azure ML
    MySQL
    DataDog
    App
    Docker
    DataDog
    New Relic
    App
    MySQL
    EARLY CLOUD
    Mostly Static
    N-Tier Apps
    VMs
    Private Cloud
    CURRENT CLOUD
    Partly Dynamic
    Less Monolithic
    VMs and Containers*
    Hybrid - Public/Private
    *Experimentation
    FUTURE CLOUD
    Fully Dynamic
    Hyper-Connected Services
    Containers and Serverless
    Mostly Public Cloud

    View Slide

  47. Infrastructure Landscape
    Foundation
    Security
    IAM
    KMS
    Networking
    VPC
    Subnets
    Firewalls
    Load Balancing
    DNS
    Compute
    VMs
    Containers
    Clusters
    Registries
    APM
    Monitoring
    Logging
    AlerKng
    Serverless
    Functions
    API Gateways
    Data
    Object Stores
    Databases
    SQL
    NoSQL
    MQ
    Queues
    Pub/Sub
    Applications
    Images Container Images Code Packaging
    CI/CD

    View Slide

  48. Kubernetes layers
    Managed Kubernetes cluster
    Infrastructure Resources (networking, storage, identity)
    Managed Service Managed Service
    Application Application Application

    View Slide

  49. Stack References
    Org: acme-corp
    vpc
    Stack: dev
    env: dev
    region: us-east-1
    k8s-cluster
    Stack: dev
    env: dev
    region: us-east-1
    svc-userprofile
    Stack: dev
    env: dev
    region: us-east-1
    svc-email
    Stack: dev
    env: dev
    region: us-east-1

    View Slide

  50. Kubernetes &
    Multi-stack
    Solutions
    Demo

    View Slide

  51. Don’t miss the next session

    View Slide

  52. Conclusions
    55

    View Slide

  53. PROVISIONING
    Developer-friendly
    Familiar language experience, toolchain,
    packages – applied to cloud infrastructure.
    Developers and operators working in a
    team.
    Cloud Engineering Transformed
    TESTING
    Confidence and quality
    Unit testing and TDD with battle-tested
    tools to ensure correctness.
    Policy as Code for compliance, cost
    control, and company-wide best
    practices.
    ARCHITECTURE
    Logic and abstractions
    Conditionals, loops, functions, classes,
    and packages out of the box.
    Reusable components that encapsulate
    complex logic and provider the right
    level of abstraction.
    Modern Infrastructure as Code Capabilities
    to ship faster and with confidence

    View Slide

  54. Useful Links
    https://bit.ly/pulumilinks

    View Slide