Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Management Superpowers with Pulumi and .NET

Cloud Management Superpowers with Pulumi and .NET

Mikhail Shilkov

June 02, 2020
Tweet

More Decks by Mikhail Shilkov

Other Decks in Programming

Transcript

  1. Cloud Management Superpowers
    with .NET
    Mikhail Shilkov

    View Slide

  2. DevOps
    Developer
    IT Infrastructure
    User

    View Slide

  3. DevOps in Cloud age
    Developer
    Cloud Infrastructure
    User
    IT

    View Slide

  4. Cloud tooling
    Developer

    View Slide

  5. About me
    • Mikhail Shilkov
    • Software engineer at Pulumi
    .NET SDK, Azure, Core platform
    • Serverless
    • Functional programming, F#
    • Microsoft Azure MVP
    @MikhailShilkov
    [email protected]

    View Slide

  6. Intro
    ● Pulumi: Modern Infrastructure as Code
    ● .NET languages to manage cloud resources
    Cloud Superpowers
    ● Provisioning
    ● Architecture
    ● Testing
    ● Policy
    ● Delivery
    Agenda

    View Slide

  7. .NET for All the Clouds
    Provision infrastructure anywhere using C#, F#, or VB.NET

    View Slide

  8. General-purpose Programming Languages

    View Slide

  9. Providers
    ● AWS
    ● Azure
    ● GCP
    ● Digital Ocean
    ● Cloudflare
    … and more
    ● Docker
    ● Kubernetes
    ● OpenStack
    ● PostgreSQL
    ● New Relic

    View Slide

  10. Static Website
    on AWS
    Demo

    View Slide

  11. Provisioning
    with Desired State Configuration
    10

    View Slide

  12. Cloud Management API
    Management API (e.g., Azure Resource Manager)
    Cloud Service Cloud Service Cloud Service
    Web Portal CLI SDK 3rd Party

    View Slide

  13. var launchRequest = new RunInstancesRequest
    {
    ImageId = amiID,
    InstanceType = "t1.micro",
    MinCount = 1,
    MaxCount = 1,
    KeyName = keyPairName,
    SecurityGroupIds = groups
    };
    var launchResponse = ec2Client.RunInstances(launchRequest);
    var instances = launchResponse.Reservation.Instances;
    SDKs to call those REST endpoints

    View Slide

  14. aws ec2 run-instances \
    --image-id ami-5ec1673e \
    --key-name MyKey \
    --security-groups EC2SecurityGroup \
    --instance-type t2.micro \
    --placement AvailabilityZone=us-west-2b
    Command-line Tools

    View Slide

  15. SDK and Scripting: Challenges
    15
    Imperative
    Step-by-step
    provisioning process
    Complex
    Depend on both the
    current and the target
    state
    Error-prone
    How to handle failures?

    View Slide

  16. Desired State Configuration
    Target Current
    Tool

    View Slide

  17. Managing Resource Graphs
    Target Current
    Tool

    View Slide

  18. var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard",
    });
    C# Example

    View Slide

  19. Desired
    State!
    var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard",
    });

    View Slide

  20. How Pulumi Works
    CLI & engine
    Last
    deployed
    state
    Program.cs
    Language host
    AWS
    Azure
    GCP
    Kubernetes
    new
    Resource()
    CRUD

    View Slide

  21. Markup-based Languages
    21
    AWS CloudFormation
    YAML
    Azure RM Templates
    JSON
    Terraform
    HCL
    "resources": [
    {
    "apiVersion": "2016-01-01",
    "type": "Microsoft.Storage/storageAccounts",
    "name": "mystorageaccount",
    "location": "westus",
    "sku": {
    "name": "Standard_LRS"
    },
    "kind": "Storage",
    "properties": {
    }
    }
    ]
    Resources:
    S3BucketForURLs:
    Type: "AWS::S3::Bucket"
    DeletionPolicy: Delete
    Properties:
    BucketName: !If [ "CreateNewBucket", !Ref
    WebsiteConfiguration:
    IndexDocument: "index.html"
    LifecycleConfiguration:
    Rules:
    -
    Id: DisposeShortUrls
    ExpirationInDays: !Ref URLExpiration
    Prefix: "u"
    Status: Enabled
    resource "azurerm_storage_account" "sa" {
    name = "mysa"
    location = azurerm_resource_group
    .rg.location
    resource_group_name = azurerm_resource_group
    .rg.name
    account_tier = "Standard"
    account_replication_type = "LRS"
    }
    Every alternative has its own language,
    own tools, own conventions

    View Slide

  22. Markup Authoring Experience
    22
    Tooling
    Lack of tooling on day 1.
    No fast feedback from
    editors.
    Discovery
    Which properties can be
    set, and which values are
    valid?
    Size
    Thousands of lines of
    markup are hard to write,
    read, and maintain

    View Slide

  23. Example App: Serverless URL Shortener
    KV Store
    Table “URLs”
    Function
    “Add URL”
    Function
    “Open URL”
    Tool Lines of code
    Function Code ~20
    CloudFormation ~240
    ARM Templates ~160
    Terraform ~220
    Estimates based on public examples

    View Slide

  24. Tools That You Love
    Developers can apply their existing skills to infrastructure

    View Slide

  25. Pulumi ❤ .NET
    ● .NET Core 3.1
    ● OS: Windows, macOS, Linux
    ● Language: C#, F#, VB.NET
    ● Editor and IDE: Visual Studio, Code, Rider
    ● IntelliSense, ReSharper, StyleCop, DocFX
    ● Package Manager: NuGet, MyGet, Paket
    ● NUnit, xUnit.net, Moq

    View Slide

  26. var storageAccount = new Account("sa", new AccountArgs {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard"
    });
    var appServicePlan = new Plan("asp", new PlanArgs {
    ResourceGroupName = resourceGroup.Name,
    Kind = "App",
    Sku = new PlanSkuArgs { Tier = "Basic", Size = "B1” }
    });
    Cloud Resources with C#

    View Slide

  27. let storageAccount =
    Account("sa",
    AccountArgs
    (ResourceGroupName = io resourceGroup.Name,
    AccountReplicationType = input "LRS",
    AccountTier = input "Standard"))
    let sku = PlanSkuArgs(Tier = input "Basic", Size = input "B1")
    let appServicePlan =
    Plan("asp",
    PlanArgs
    (ResourceGroupName = io resourceGroup.Name,
    Kind = input "App",
    Sku = input sku))
    Cloud Resources with F#

    View Slide

  28. ' Create an Azure Resource Group
    Dim resourceGroup = New ResourceGroup("resourceGroup")
    ' Create an Azure Storage Account
    Dim storageAccount = New Account("storage", New AccountArgs With {
    .ResourceGroupName = resourceGroup.Name,
    .AccountReplicationType = "LRS",
    .AccountTier = "Standard"
    })
    Cloud Resources with VB.NET

    View Slide

  29. Architecture
    Reusable Abstractions

    View Slide

  30. Resources to Provision an Azure Function
    Blob
    App
    Plan
    Container
    Storage
    Account
    Function
    App

    View Slide

  31. // A resource group to contain our Azure Functions
    var resourceGroup = new ResourceGroup("functions-rg");
    // A .NET Azure Function
    var dotnet = new ArchiveFunctionApp("http", new ArchiveFunctionAppArgs
    {
    ResourceGroup = resourceGroup,
    Archive = new FileArchive("./dotnet/bin/Debug/netcoreapp3.1/publish")
    });
    .NET Azure Function

    View Slide

  32. Let’s run pulumi up …
    Type Name Plan
    + pulumi:pulumi:Stack functions-dev create
    + ├─ azure:appservice:ArchiveFunctionApp http create
    + │ ├─ azure:storage:Account http create
    + │ ├─ azure:appservice:Plan http create
    + │ ├─ azure:storage:Container http create
    + │ ├─ azure:storage:ZipBlob http create
    + │ └─ azure:appservice:FunctionApp http create
    + └─ azure:core:ResourceGroup rg create

    View Slide

  33. // A Node.js Azure Function
    var js = new ArchiveFunctionApp("js", new ArchiveFunctionAppArgs
    {
    ResourceGroup = resourceGroup,
    Archive = new FileArchive("./javascript"),
    Runtime = "node"
    });
    Node.js Azure Function

    View Slide

  34. var linuxResourceGroup = new ResourceGroup("linux-rg");
    var linuxPlan = new Plan("linux-asp", new PlanArgs
    {
    ResourceGroupName = linuxResourceGroup.Name,
    Kind = "Linux",
    Reserved = true,
    Sku = new PlanSkuArgs { Tier = "Dynamic", Size = "Y1" }
    });
    var python = new ArchiveFunctionApp(...);
    Python Azure Function on Linux

    View Slide

  35. var linuxResourceGroup = new ResourceGroup("linux-rg");
    var linuxPlan = new Plan("linux-asp", new PlanArgs { ... });
    var python = new ArchiveFunctionApp("py", new ArchiveFunctionAppArgs
    {
    ResourceGroup = linuxResourceGroup,
    Archive = new FileArchive("./python"),
    Plan = linuxPlan,
    Runtime = "python"
    });
    Python Azure Function on Linux

    View Slide

  36. Building a Cosmos App
    Create custom abstractions

    View Slide

  37. Cosmos App (1)

    View Slide

  38. Cosmos App (2)

    View Slide

  39. Cosmos App (3)

    View Slide

  40. Cosmos App (4)

    View Slide

  41. var app = new CosmosApp("functions", new CosmosAppArgs
    {
    ResourceGroup = resourceGroup,
    Locations = ["WestUS", "EastUS", "WestEurope"],
    Factory = global => region =>
    {
    var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs
    {
    ResourceGroupName = resourceGroup.Name,
    Location = region.Location,
    Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"),
    AppSettings = { { "CosmosDBConnection", global.ConnectionString } }
    });
    return new AzureEndpoint(func.AppId);
    }
    });
    Global Applications

    View Slide

  42. var app = new CosmosApp("functions", new CosmosAppArgs
    {
    ResourceGroup = resourceGroup,
    Locations = ["WestUS", "EastUS", "WestEurope"],
    Factory = global => region =>
    {
    var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs
    {
    ResourceGroupName = resourceGroup.Name,
    Location = region.Location,
    Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"),
    AppSettings = { { "CosmosDBConnection", global.ConnectionString } }
    });
    return new AzureEndpoint(func.AppId);
    }
    });
    Global Applications

    View Slide

  43. Testing and Policy
    Validate deployments

    View Slide

  44. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var resources = await Deployment.TestAsync();
    var resource = resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing

    View Slide

  45. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var resources = await Deployment.TestAsync();
    var resource = resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing

    View Slide

  46. [Fact]
    public async Task InstancesMustNotHaveSshPortOpenToInternet()
    {
    var resources = await Deployment.TestAsync();
    var resource = resources.OfType().Single();
    var ingress = await resource.GetAsync(sg => sg.Ingress);
    foreach (var rule in ingress)
    foreach (var cidrBlock in rule.CidrBlocks)
    {
    Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0",
    "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)");
    }
    }
    Unit Testing

    View Slide

  47. View Slide

  48. Policy as Code (.NET coming later)
    const policies = new PolicyPack("aws", {
    policies: [
    {
    name: "prohibited-public-internet",
    description: "Ingress rules with public internet access are prohibited.",
    enforcementLevel: "mandatory",
    validateResource: validateTypedResource(aws.ec2.SecurityGroup, (sg, args, report) => {
    const publicInternetRules = (sg.ingress || []).find(ingressRule =>
    (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0"));
    if (publicInternetRules) {
    report("Ingress rules with public internet access are prohibited.");
    }
    }),
    },
    ],
    });

    View Slide

  49. Delivery
    Multi-cloud cross-stack automation

    View Slide

  50. State Management
    CLI and
    Engine
    Last
    deployed
    state
    Program.cs
    Language host
    AWS
    Azure
    GCP
    Kubernetes
    new Resource()
    CRUD
    Pulumi Service
    S3 / Blob Storage
    Local disk

    View Slide

  51. Pulumi Components
    51
    CLI
    Open-source
    Free for any use
    Cross-platform
    Written in Go
    Providers
    Open-source
    Free for any use
    Can be created by 1st or
    3rd party
    SaaS backend
    Web API + portal
    “GitHub for infra”
    Free for individuals
    Paid plans for teams
    Useful but not required

    View Slide

  52. Deployment process
    Allow developers to
    iterate quickly on dev
    environments.
    Gain visibility and change
    management with
    Continuous Delivery.

    View Slide

  53. Cloud Transition
    53
    Kubernetes
    Azure
    Functions
    AWS S3
    Azure
    Analytics
    Google ML
    MySQL
    DataDog
    App
    Docker
    DataDog
    New Relic
    App
    MySQL
    EARLY CLOUD
    Mostly Static
    N-Tier Apps
    VMs
    Private Cloud
    CURRENT CLOUD
    Partly Dynamic
    Less Monolithic
    VMs and Containers*
    Hybrid - Public/Private
    *ExperimentaDon
    FUTURE CLOUD
    Fully Dynamic
    Hyper-Connected Services
    Containers and Serverless
    Mostly Public Cloud

    View Slide

  54. Infrastructure Landscape
    54
    Foundation
    Security
    IAM
    KMS
    Networking
    VPC
    Subnets
    Firewalls
    Load Balancing
    DNS
    Compute
    VMs
    Containers
    Clusters
    Registries
    APM
    Monitoring
    Logging
    Alerting
    Serverless
    Functions
    API Gateways
    Data
    Object Stores
    Databases
    SQL
    NoSQL
    MQ
    Queues
    Pub/Sub
    Applications
    Images Container Images Code Packaging
    CI/CD

    View Slide

  55. Kubernetes layers
    Managed Kubernetes cluster
    Infrastructure Resources (networking, storage, identity)
    Managed Service Managed Service
    Application Application Application

    View Slide

  56. Stack References
    Org: acme-corp
    vpc
    Stack: dev
    env: dev
    region: us-east-1
    k8s-cluster
    Stack: dev
    env: dev
    region: us-east-1
    svc-userprofile
    Stack: dev
    env: dev
    region: us-east-1
    svc-email
    Stack: dev
    env: dev
    region: us-east-1

    View Slide

  57. Kubernetes &
    Multi-stack
    Solutions
    Demo

    View Slide

  58. Conclusions

    View Slide

  59. PROVISIONING
    Developer-friendly
    Familiar language experience, toolchain,
    packages – applied to cloud infrastructure.
    Developers and operators working in a
    team.
    Cloud Engineering Transformed
    TESTING
    Confidence and quality
    Unit testing and TDD with battle-tested
    tools to ensure correctness.
    Policy as Code for compliance, cost
    control, and company-wide best
    practices.
    ARCHITECTURE
    Logic and abstractions
    Conditionals, loops, functions, classes,
    and packages out of the box.
    Reusable components that encapsulate
    complex logic and provider the right
    level of abstraction.
    Modern Infrastructure as Code Capabilities
    to ship faster and with confidence

    View Slide

  60. Useful Links
    http://bit.ly/pulumilinks

    View Slide

  61. Q&A

    View Slide