Managing Any Cloud with .NET

All Pulumi capabilities for defining cloud resources are now available for .NET developers. Pulumi engineer Mikhail Shilkov will show you how you can use C#, F#, and VB.NET to define the infrastructure for Azure, AWS, or even on-prem Kubernetes clusters.

Transcript

1. Managing Any Cloud with .NET
   Mikhail Shilkov
   

   View Slide

2. About me
   • Mikhail Shilkov
   • Software engineer at Pulumi
   .NET SDK, Azure, Core platform
   • Serverless
   • Functional programming, F#
   • Microsoft Azure MVP
   @MikhailShilkov
   mikhail@pulumi.com
   

   View Slide

3. ● Pulumi: Modern Infrastructure as Code
   ● Use .NET languages to manage cloud resources
   ● Desired state configuration
   ● Tools that you love
   ● Reusable abstractions
   ● Automated testing
   ● Multi-cloud
   Agenda
   

   View Slide

4. .NET for All the Clouds
   Provision infrastructure anywhere using C#, F#, or VB.NET
   

   View Slide

5. General-purpose Programming Languages
   

   View Slide

6. Providers
   ● AWS
   ● Azure
   ● GCP
   ● Digital Ocean
   ● Cloudflare
   … and more
   ● Docker
   ● Kubernetes
   ● OpenStack
   ● PostgreSQL
   ● New Relic
   

   View Slide

7. Static Website
   on AWS
   Demo
   

   View Slide

8. Desired State Configuration
   Maintainable infrastructure
   10
   

   View Slide

9. Unified Management API
   Management API (e.g., Azure Resource Manager)
   Cloud Service Cloud Service Cloud Service
   Web Portal CLI SDK 3rd Party
   

   View Slide

10. var launchRequest = new RunInstancesRequest
    {
    ImageId = amiID,
    InstanceType = "t1.micro",
    MinCount = 1,
    MaxCount = 1,
    KeyName = keyPairName,
    SecurityGroupIds = groups
    };
    var launchResponse = ec2Client.RunInstances(launchRequest);
    var instances = launchResponse.Reservation.Instances;
    SDKs to call those REST endpoints
    

    View Slide

11. aws ec2 run-instances \
    --image-id ami-5ec1673e \
    --key-name MyKey \
    --security-groups EC2SecurityGroup \
    --instance-type t2.micro \
    --placement AvailabilityZone=us-west-2b
    Command-line Tools
    

    View Slide

12. SDK and Scripting: Challenges
    12
    Imperative
    Step-by-step
    provisioning process
    Complex
    Depend on both the
    current and the target
    state
    Error-prone
    How to handle failures?
    

    View Slide

13. Desired State Configuration
    Target Current
    Tool
    

    View Slide

14. Managing Resource Graphs
    Target Current
    Tool
    

    View Slide

15. var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard",
    });
    C# Example
    

    View Slide

16. Desired
    State!
    var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard",
    });
    

    View Slide

17. How Pulumi Works
    CLI & engine
    Last
    deployed
    state
    Program.cs
    Language host
    AWS
    Azure
    GCP
    Kubernetes
    new
    Resource()
    CRUD
    

    View Slide

18. Markup-based Languages
    18
    AWS CloudFormation
    YAML
    Azure RM Templates
    JSON
    Terraform
    HCL
    "resources": [
    {
    "apiVersion": "2016-01-01",
    "type": "Microsoft.Storage/storageAccounts",
    "name": "mystorageaccount",
    "location": "westus",
    "sku": {
    "name": "Standard_LRS"
    },
    "kind": "Storage",
    "properties": {
    }
    }
    ]
    Resources:
    S3BucketForURLs:
    Type: "AWS::S3::Bucket"
    DeletionPolicy: Delete
    Properties:
    BucketName: !If [ "CreateNewBucket", !Ref
    WebsiteConfiguration:
    IndexDocument: "index.html"
    LifecycleConfiguration:
    Rules:
    -
    Id: DisposeShortUrls
    ExpirationInDays: !Ref URLExpiration
    Prefix: "u"
    Status: Enabled
    resource "azurerm_storage_account" "sa" {
    name = "mysa"
    location = azurerm_resource_group
    .rg.location
    resource_group_name = azurerm_resource_group
    .rg.name
    account_tier = "Standard"
    account_replication_type = "LRS"
    }
    Every alternative has its own language,
    own tools, own conventions
    

    View Slide

19. Markup Authoring Experience
    19
    Tooling
    Lack of tooling on day 1.
    No fast feedback from
    editors.
    Discovery
    Which properties can be
    set, and which values are
    valid?
    Size
    Thousands of lines of
    markup are hard to write,
    read, and maintain
    

    View Slide

20. Example App: Serverless URL Shortener
    KV Store
    Table “URLs”
    Function
    “Add URL”
    Function
    “Open URL”
    Tool Lines of code
    Function Code ~20
    CloudFormation ~240
    ARM Templates ~160
    Terraform ~220
    Estimates based on public examples
    

    View Slide

21. Tools That You Love
    Developers can apply their existing skills to infrastructure
    

    View Slide

22. Pulumi ❤ .NET
    ● .NET Core 3.1
    ● OS: Windows, macOS, Linux
    ● Language: C#, F#, VB.NET
    ● Editor and IDE: Visual Studio, Code, Rider
    ● IntelliSense, ReSharper, StyleCop, DocFX
    ● Package Manager: NuGet, MyGet, Paket
    ● NUnit, xUnit.net, Moq
    

    View Slide

23. var storageAccount = new Account("sa", new AccountArgs {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard"
    });
    var appServicePlan = new Plan("asp", new PlanArgs {
    ResourceGroupName = resourceGroup.Name,
    Kind = "App",
    Sku = new PlanSkuArgs { Tier = "Basic", Size = "B1” }
    });
    Cloud Resources with C#
    

    View Slide

24. let storageAccount =
    Account("sa",
    AccountArgs
    (ResourceGroupName = io resourceGroup.Name,
    AccountReplicationType = input "LRS",
    AccountTier = input "Standard"))
    let sku = PlanSkuArgs(Tier = input "Basic", Size = input "B1")
    let appServicePlan =
    Plan("asp",
    PlanArgs
    (ResourceGroupName = io resourceGroup.Name,
    Kind = input "App",
    Sku = input sku))
    Cloud Resources with F#
    

    View Slide

25. ' Create an Azure Resource Group
    Dim resourceGroup = New ResourceGroup("resourceGroup")
    ' Create an Azure Storage Account
    Dim storageAccount = New Account("storage", New AccountArgs With {
    .ResourceGroupName = resourceGroup.Name,
    .AccountReplicationType = "LRS",
    .AccountTier = "Standard"
    })
    Cloud Resources with VB.NET
    

    View Slide

26. Reusable Abstractions
    Infrastructure Components
    

    View Slide

27. Resources to Provision an Azure Function
    Blob
    App
    Plan
    Container
    Storage
    Account
    Function
    App
    

    View Slide

28. // A resource group to contain our Azure Functions
    var resourceGroup = new ResourceGroup("functions-rg");
    // A .NET Azure Function
    var dotnet = new ArchiveFunctionApp("http", new ArchiveFunctionAppArgs
    {
    ResourceGroup = resourceGroup,
    Archive = new FileArchive("./dotnet/bin/Debug/netcoreapp3.1/publish")
    });
    .NET Azure Function
    

    View Slide

29. Let’s run pulumi up …
    Type Name Plan
    + pulumi:pulumi:Stack functions-dev create
    + ├─ azure:appservice:ArchiveFunctionApp http create
    + │ ├─ azure:storage:Account http create
    + │ ├─ azure:appservice:Plan http create
    + │ ├─ azure:storage:Container http create
    + │ ├─ azure:storage:ZipBlob http create
    + │ └─ azure:appservice:FunctionApp http create
    + └─ azure:core:ResourceGroup rg create
    

    View Slide

30. Serverless:
    Azure Functions
    Demo
    

    View Slide

31. Building a Cosmos App
    Create custom abstractions
    

    View Slide

32. Cosmos App (1)
    

    View Slide

33. Cosmos App (2)
    

    View Slide

34. Cosmos App (3)
    

    View Slide

35. Cosmos App (4)
    

    View Slide

36. var app = new CosmosApp("functions", new CosmosAppArgs
    {
    ResourceGroup = resourceGroup,
    Locations = ["WestUS", "EastUS", "WestEurope"],
    Factory = global => region =>
    {
    var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs
    {
    ResourceGroupName = resourceGroup.Name,
    Location = region.Location,
    Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"),
    AppSettings = { { "CosmosDBConnection", global.ConnectionString } }
    });
    return new AzureEndpoint(func.AppId);
    }
    });
    Global Applications
    

    View Slide

37. var app = new CosmosApp("functions", new CosmosAppArgs
    {
    ResourceGroup = resourceGroup,
    Locations = ["WestUS", "EastUS", "WestEurope"],
    Factory = global => region =>
    {
    var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs
    {
    ResourceGroupName = resourceGroup.Name,
    Location = region.Location,
    Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"),
    AppSettings = { { "CosmosDBConnection", global.ConnectionString } }
    });
    return new AzureEndpoint(func.AppId);
    }
    });
    Global Applications
    

    View Slide

38. Automated Testing
    Unit testing and policies
    

    View Slide

39. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var resources = await Deployment.TestAsync();
    var resource = resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing
    

    View Slide

40. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var resources = await Deployment.TestAsync();
    var resource = resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing
    

    View Slide

41. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var resources = await Deployment.TestAsync();
    var resource = resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing
    

    View Slide

42. Policy as Code (.NET coming later)
    const policies = new PolicyPack("aws", {
    policies: [
    {
    name: "prohibited-public-internet",
    description: "Ingress rules with public internet access are prohibited.",
    enforcementLevel: "mandatory",
    validateResource: validateTypedResource(aws.ec2.SecurityGroup, (sg, args, report) => {
    const publicInternetRules = (sg.ingress || []).find(ingressRule =>
    (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0"));
    if (publicInternetRules) {
    report("Ingress rules with public internet access are prohibited.");
    }
    }),
    },
    ],
    });
    

    View Slide

43. Kubernetes
    Deploy to multiple clouds
    

    View Slide

44. Cloud Transition
    44
    Kubernetes
    Azure
    Functions
    AWS S3
    Azure
    Analytics
    Google ML
    MySQL
    DataDog
    App
    Docker
    DataDog
    New Relic
    App
    MySQL
    EARLY CLOUD
    Mostly Static
    N-Tier Apps
    VMs
    Private Cloud
    CURRENT CLOUD
    Partly Dynamic
    Less Monolithic
    VMs and Containers*
    Hybrid - Public/Private
    *Experimentation
    FUTURE CLOUD
    Fully Dynamic
    Hyper-Connected Services
    Containers and Serverless
    Mostly Public Cloud
    

    View Slide

45. Infrastructure Landscape
    45
    Founda3on
    Security
    IAM
    KMS
    Networking
    VPC
    Subnets
    Firewalls
    Load Balancing
    DNS
    Compute
    VMs
    Containers
    Clusters
    Registries
    APM
    Monitoring
    Logging
    AlerKng
    Serverless
    Functions
    API Gateways
    Data
    Object Stores
    Databases
    SQL
    NoSQL
    MQ
    Queues
    Pub/Sub
    Applications
    Images Container Images Code Packaging
    CI/CD
    

    View Slide

46. Kubernetes layers
    Managed Kubernetes cluster
    Infrastructure Resources (networking, storage, identity)
    Managed Service Managed Service
    Application Application Application
    

    View Slide

47. Stack References
    Org: acme-corp
    vpc
    Stack: dev
    env: dev
    region: us-east-1
    k8s-cluster
    Stack: dev
    env: dev
    region: us-east-1
    svc-userprofile
    Stack: dev
    env: dev
    region: us-east-1
    svc-email
    Stack: dev
    env: dev
    region: us-east-1
    

    View Slide

48. Kubernetes &
    Multi-stack
    Solutions
    Demo
    

    View Slide

49. Conclusions
    55
    

    View Slide

50. PROVISIONING
    Developer-friendly
    Familiar language experience, toolchain,
    packages – applied to cloud infrastructure.
    Developers and operators working in a
    team.
    Cloud Engineering Transformed
    TESTING
    Confidence and quality
    Unit testing and TDD with battle-tested
    tools to ensure correctness.
    Policy as Code for compliance, cost
    control, and company-wide best
    practices.
    ARCHITECTURE
    Logic and abstractions
    Conditionals, loops, functions, classes,
    and packages out of the box.
    Reusable components that encapsulate
    complex logic and provider the right
    level of abstraction.
    Modern Infrastructure as Code Capabilities
    to ship faster and with confidence
    

    View Slide

51. Useful Links
    http://bit.ly/pulumilinks
    

    View Slide

52. Thank you!
    Join slack.pulumi.com for help after this session
    Submit requests and bugs on github.com/pulumi/pulumi
    Subscribe for weekly tutorials on youtube.com/pulumiTV
    Follow @PulumiCorp for updates, or hit me up directly @MikhailShilkov
    Here are some more ways to learn about Pulumi and join the community
    Upcoming talks and workshops at pulumi.com/webinars
    

    View Slide