Azure Infrastructure as C# and F#

Azure cloud platform offers amazing capabilities for application developers. Cloud apps utilize multiple services and consist of many components, so they are hard to manage without employing Infrastructure as Code.

Traditional tools like ARM templates and Terraform use text-based formats, which tend to be tedious, repetitive, and cumbersome to reuse.

What if instead of configuration files you could use your favourite programming languages like C# or F#? See how you can bring your developer tools like code completion, types, components, and abstractions to cloud infrastructure definition.

Transcript

1. Azure Infrastructure as C# and F#
   Mikhail Shilkov
   

   View Slide

2. DevOps in the cloud age
   Developer
   Cloud Infrastructure
   User
   IT
   

   View Slide

3. ● Pulumi: Modern Infrastructure as Code
   ● Open source and free
   ● Use .NET languages to manage cloud resources
   ● Desired state configuration
   ● Tools that you love
   ● Reusable abstractions
   ● Automated testing
   ● Multi-cloud
   Agenda
   

   View Slide

4. About me
   • Software engineer at Pulumi
   .NET SDK, Azure, Core platform
   • Serverless
   • Functional programming, F#
   • Microsoft Azure MVP
   @MikhailShilkov
   mikhail@pulumi.com
   https://mikhail.io
   

   View Slide

5. .NET for Cloud Resources
   Provision infrastructure anywhere using C#, F#, or VB.NET
   

   View Slide

6. General-purpose Programming Languages
   

   View Slide

7. Providers
   ● Azure
   ● AWS
   ● GCP
   ● Digital Ocean
   ● Cloudflare
   … and more
   ● Docker
   ● Kubernetes
   ● OpenStack
   ● PostgreSQL
   ● New Relic
   

   View Slide

8. Static Website
   on Azure
   Demo
   

   View Slide

9. Desired State Configuration
   Maintainable infrastructure
   10
   

   View Slide

10. Unified Management API
    Management API (Azure Resource Manager)
    Cloud Service Cloud Service Cloud Service
    Web Portal CLI SDK 3rd Party
    

    View Slide

11. REST calls
    PUT
    https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/
    {resourceGroupName}/providers/Microsoft.Storage/storageAccounts/mystorageac
    count?api-version=2016-01-01
    {
    "location": "westus",
    "properties": {
    }
    "sku": {
    "name": "Standard_LRS"
    },
    "kind": "Storage"
    }
    

    View Slide

12. var storageAccount = az.StorageAccounts.Define(storageAccountName)
    .WithRegion(Region.US_EAST)
    .WithNewResourceGroup(resourceGroupName)
    .Create();
    SDKs to call those REST endpoints
    

    View Slide

13. az storage account create \
    --location westus \
    --name samplesa \
    --resource-group myrg \
    --sku LRS
    Command-line Tools
    

    View Slide

14. SDK and Scripting: Challenges
    14
    Imperative
    Step-by-step
    provisioning process
    Complex
    Depend on both the
    current and the target
    state
    Error-prone
    How to handle failures?
    

    View Slide

15. Desired State Configuration
    Target Current
    Tool
    

    View Slide

16. Managing Resource Graphs
    Target Current
    Tool
    

    View Slide

17. Markup-based Languages
    17
    AWS CloudFormation
    YAML
    Azure RM Templates
    JSON
    Terraform
    HCL
    "resources": [
    {
    "apiVersion": "2016-01-01",
    "type": "Microsoft.Storage/storageAccounts",
    "name": "mystorageaccount",
    "location": "westus",
    "sku": {
    "name": "Standard_LRS"
    },
    "kind": "Storage",
    "properties": {
    }
    }
    ]
    Resources:
    S3BucketForURLs:
    Type: "AWS::S3::Bucket"
    DeletionPolicy: Delete
    Properties:
    BucketName: !If [ "CreateNewBucket", !Ref
    WebsiteConfiguration:
    IndexDocument: "index.html"
    LifecycleConfiguration:
    Rules:
    -
    Id: DisposeShortUrls
    ExpirationInDays: !Ref URLExpiration
    Prefix: "u"
    Status: Enabled
    resource "azurerm_storage_account" "sa" {
    name = "mysa"
    location = azurerm_resource_group
    .rg.location
    resource_group_name = azurerm_resource_group
    .rg.name
    account_tier = "Standard"
    account_replication_type = "LRS"
    }
    Every tool has its own language,
    own tools, own conventions
    

    View Slide

18. Markup Authoring Experience
    18
    Tooling
    Lack of tooling on day 1.
    No fast feedback from
    editors.
    Discovery
    Which properties can be
    set, and which values are
    valid?
    Size
    Thousands of lines of
    markup are hard to write,
    read, and maintain
    

    View Slide

19. Example App: Serverless URL Shortener
    KV Store
    Table “URLs”
    Function
    “Add URL”
    Function
    “Open URL”
    Tool Lines of code
    Function Code ~20
    CloudFormation ~240
    ARM Templates ~160
    Terraform ~220
    Estimates based on public examples
    

    View Slide

20. var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard"
    });
    C# Example
    

    View Slide

21. Desired
    State!
    var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard"
    });
    

    View Slide

22. How Pulumi Works
    CLI & engine
    Last
    deployed
    state
    Program.cs
    Language host
    AWS
    Azure
    GCP
    Kubernetes
    new
    Resource()
    CRUD
    

    View Slide

23. Tools That You Love
    Developers can apply their existing skills to infrastructure
    

    View Slide

24. Pulumi ❤ .NET
    ● .NET Core 3.1
    ● OS: Windows, macOS, Linux
    ● Language: C#, F#, VB.NET
    ● Editor and IDE: Visual Studio, Code, Rider
    ● IntelliSense, ReSharper, StyleCop, DocFX
    ● Package Manager: NuGet, MyGet, Paket
    ● NUnit, xUnit.net, Moq
    

    View Slide

25. var storageAccount = new Account("sa", new AccountArgs {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard"
    });
    var appServicePlan = new Plan("asp", new PlanArgs {
    ResourceGroupName = resourceGroup.Name,
    Kind = "App",
    Sku = new PlanSkuArgs { Tier = "Basic", Size = "B1” }
    });
    Cloud Resources with C#
    

    View Slide

26. let storageAccount =
    Account("sa",
    AccountArgs
    (ResourceGroupName = io resourceGroup.Name,
    AccountReplicationType = input "LRS",
    AccountTier = input "Standard"))
    let sku = PlanSkuArgs(Tier = input "Basic", Size = input "B1")
    let appServicePlan =
    Plan("asp",
    PlanArgs
    (ResourceGroupName = io resourceGroup.Name,
    Kind = input "App",
    Sku = input sku))
    Cloud Resources with F#
    

    View Slide

27. ' Create an Azure Resource Group
    Dim resourceGroup = New ResourceGroup("resourceGroup")
    ' Create an Azure Storage Account
    Dim storageAccount = New Account("storage", New AccountArgs With {
    .ResourceGroupName = resourceGroup.Name,
    .AccountReplicationType = "LRS",
    .AccountTier = "Standard"
    })
    Cloud Resources with VB.NET
    

    View Slide

28. Reusable Abstractions
    Infrastructure Components
    

    View Slide

29. Resources to Provision an Azure Function
    Blob
    App
    Plan
    Container
    Storage
    Account
    Function
    App
    

    View Slide

30. // A resource group to contain our Azure Functions
    var resourceGroup = new ResourceGroup("functions-rg");
    // A .NET Azure Function
    var dotnet = new ArchiveFunctionApp("http", new ArchiveFunctionAppArgs
    {
    ResourceGroup = resourceGroup,
    Archive = new FileArchive("./dotnet/bin/Debug/netcoreapp3.1/publish")
    });
    .NET Azure Function
    

    View Slide

31. Let’s run pulumi up …
    Type Name Plan
    + pulumi:pulumi:Stack functions-dev create
    + ├─ azure:appservice:ArchiveFunctionApp http create
    + │ ├─ azure:storage:Account http create
    + │ ├─ azure:appservice:Plan http create
    + │ ├─ azure:storage:Container http create
    + │ ├─ azure:storage:ZipBlob http create
    + │ └─ azure:appservice:FunctionApp http create
    + └─ azure:core:ResourceGroup rg create
    

    View Slide

32. Serverless:
    Azure Functions
    Demo
    

    View Slide

33. Building a Cosmos App
    Create custom abstractions
    

    View Slide

34. Cosmos App (1)
    

    View Slide

35. Cosmos App (2)
    

    View Slide

36. Cosmos App (3)
    

    View Slide

37. Cosmos App (4)
    

    View Slide

38. var app = new CosmosApp("functions", new CosmosAppArgs
    {
    ResourceGroup = resourceGroup,
    Locations = ["WestUS", "EastUS", "WestEurope"],
    Factory = global => region =>
    {
    var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs
    {
    ResourceGroupName = resourceGroup.Name,
    Location = region.Location,
    Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"),
    AppSettings = { { "CosmosDBConnection", global.ConnectionString } }
    });
    return new AzureEndpoint(func.AppId);
    }
    });
    Global Applications
    

    View Slide

39. var app = new CosmosApp("functions", new CosmosAppArgs
    {
    ResourceGroup = resourceGroup,
    Locations = ["WestUS", "EastUS", "WestEurope"],
    Factory = global => region =>
    {
    var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs
    {
    ResourceGroupName = resourceGroup.Name,
    Location = region.Location,
    Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"),
    AppSettings = { { "CosmosDBConnection", global.ConnectionString } }
    });
    return new AzureEndpoint(func.AppId);
    }
    });
    Global Applications
    

    View Slide

40. Automated Testing
    Unit testing and policies
    

    View Slide

41. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var resources = await Deployment.TestAsync();
    var resource = resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing
    

    View Slide

42. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var resources = await Deployment.TestAsync();
    var resource = resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing
    

    View Slide

43. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var resources = await Deployment.TestAsync();
    var resource = resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing
    

    View Slide

44. Policy as Code (.NET coming later)
    const policies = new PolicyPack("network-policies", {
    policies: [
    {
    name: "prohibited-public-internet",
    description: "Ingress rules with public internet access are prohibited.",
    enforcementLevel: "mandatory",
    validateResource: validateTypedResource(SecurityGroup, (sg, args, report) => {
    const publicInternetRules = (sg.ingress || []).find(ingressRule =>
    (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0"));
    if (publicInternetRules) {
    report("Ingress rules with public internet access are prohibited.");
    }
    }),
    },
    ],
    });
    

    View Slide

45. Kubernetes
    Deploy to multiple clouds
    

    View Slide

46. Cloud Transition
    Kubernetes
    Azure
    Functions
    Azure
    Storage
    Azure
    Analytics
    Azure ML
    MySQL
    DataDog
    App
    Docker
    DataDog
    New Relic
    App
    MySQL
    EARLY CLOUD
    Mostly Static
    N-Tier Apps
    VMs
    Private Cloud
    CURRENT CLOUD
    Partly Dynamic
    Less Monolithic
    VMs and Containers*
    Hybrid - Public/Private
    *Experimentation
    FUTURE CLOUD
    Fully Dynamic
    Hyper-Connected Services
    Containers and Serverless
    Mostly Public Cloud
    

    View Slide

47. Infrastructure Landscape
    Foundation
    Security
    IAM
    KMS
    Networking
    VPC
    Subnets
    Firewalls
    Load Balancing
    DNS
    Compute
    VMs
    Containers
    Clusters
    Registries
    APM
    Monitoring
    Logging
    AlerKng
    Serverless
    Functions
    API Gateways
    Data
    Object Stores
    Databases
    SQL
    NoSQL
    MQ
    Queues
    Pub/Sub
    Applications
    Images Container Images Code Packaging
    CI/CD
    

    View Slide

48. Kubernetes layers
    Managed Kubernetes cluster
    Infrastructure Resources (networking, storage, identity)
    Managed Service Managed Service
    Application Application Application
    

    View Slide

49. Stack References
    Org: acme-corp
    vpc
    Stack: dev
    env: dev
    region: us-east-1
    k8s-cluster
    Stack: dev
    env: dev
    region: us-east-1
    svc-userprofile
    Stack: dev
    env: dev
    region: us-east-1
    svc-email
    Stack: dev
    env: dev
    region: us-east-1
    

    View Slide

50. Kubernetes &
    Multi-stack
    Solutions
    Demo
    

    View Slide

51. Don’t miss the next session
    

    View Slide

52. Conclusions
    55
    

    View Slide

53. PROVISIONING
    Developer-friendly
    Familiar language experience, toolchain,
    packages – applied to cloud infrastructure.
    Developers and operators working in a
    team.
    Cloud Engineering Transformed
    TESTING
    Confidence and quality
    Unit testing and TDD with battle-tested
    tools to ensure correctness.
    Policy as Code for compliance, cost
    control, and company-wide best
    practices.
    ARCHITECTURE
    Logic and abstractions
    Conditionals, loops, functions, classes,
    and packages out of the box.
    Reusable components that encapsulate
    complex logic and provider the right
    level of abstraction.
    Modern Infrastructure as Code Capabilities
    to ship faster and with confidence
    

    View Slide

54. Useful Links
    https://bit.ly/pulumilinks
    

    View Slide