Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cloud Management Superpowers with Pulumi

Cloud Management Superpowers with Pulumi

n this talk, Mikhail will introduce Pulumi, an open-source tool that leverages programming languages like C#, TypeScript, and Python to manage cloud resources yet in a declarative manner! Using developer tools that you know and love, you can now apply unit testing, reusable architecture, deployment policies, and continuous delivery while provisioning resources in the cloud.

Mikhail Shilkov

September 08, 2020
Tweet

More Decks by Mikhail Shilkov

Other Decks in Technology

Transcript

  1. Cloud Management Superpowers
    with Pulumi
    Mikhail Shilkov

    View Slide

  2. About me
    • Mikhail Shilkov
    • Software engineer at Pulumi
    Azure, .NET SDK, Core platform
    • Microsoft Azure MVP
    @MikhailShilkov
    https://mikhail.io
    [email protected]

    View Slide

  3. Intro
    ● Cloud Engineering
    ● Modern Infrastructure as Code
    Cloud Superpowers
    ● Provisioning
    ● Architecture
    ● Testing
    ● Policy
    ● Automation
    Agenda

    View Slide

  4. Cloud Engineering
    Provision cloud infrastructure using C#, TypeScript, Python, Go

    View Slide

  5. Infrastructure Landscape
    Foundation
    Security
    IAM
    KMS
    Networking
    VPC
    Subnets
    Firewalls
    Load Balancing
    DNS
    Compute
    VMs
    Containers
    Clusters
    Registries
    APM
    Monitoring
    Logging
    Alerting
    Serverless
    Functions
    API Gateways
    Data
    Object Stores
    Databases
    SQL
    NoSQL
    MQ
    Queues
    Pub/Sub
    Applications
    Images Container Images Code Packaging
    CI/CD

    View Slide

  6. Azure API in numbers
    ● 140+ resource providers
    ● 900+ resource types
    ● 13.000+ properties to manage
    ● 10.000+ PRs and issues in the specifications repo

    View Slide

  7. Kubernetes automation
    ● Open specifications
    ● Desired state configuration
    ● Reconciliation loop
    ● Operators
    ● Rolling updates
    ● Automation in DNA

    View Slide

  8. PAST
    Lift-and-shift
    Virtual machines on demand. Rapid
    procurement cycles.
    Manual provisioning.
    Cloud Evolution
    FUTURE
    Cloud engineering
    Convergence with application
    development and software engineering.
    Abstractions for “the most powerful
    computer ever”. Architecture as code.
    PRESENT
    Cloud native
    Hundreds of managed services.
    Specialized solutions for broad set of
    problems.
    Infrastructure-as-code, desired state
    configuration.
    Cloud has been changing the world
    and it’s not done yet

    View Slide

  9. Modern Applications
    ● Designed for the cloud
    ● Core business differenciator
    ● Fast time to market
    ● Broad footprint of resource types
    ● Resources under management grow fast

    View Slide

  10. Modern Teams & Workflows
    ● Collaboration between Dev, IT, SRE, Security
    ● Cloud as a first-class target for developers
    ● Frequent delivery of value
    ● Automation from commit to production

    View Slide

  11. Provisioning
    Cloud infrastructure using C#, TypeScript, Python, Go
    10

    View Slide

  12. Desired State Configuration
    Target Current
    Tool

    View Slide

  13. Managing Resource Graphs
    Target Current
    Tool

    View Slide

  14. Managing Resource Graphs
    Target Current
    Tool

    View Slide

  15. General-purpose Programming Languages

    View Slide

  16. Providers
    ● AWS
    ● Azure
    ● GCP
    ● Digital Ocean
    ● Cloudflare
    … and more
    ● Docker
    ● Kubernetes
    ● OpenStack
    ● PostgreSQL
    ● New Relic

    View Slide

  17. var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard",
    });
    C# Example

    View Slide

  18. Desired
    State!
    var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard",
    });

    View Slide

  19. Sample Pulumi
    Application
    Demo

    View Slide

  20. How Pulumi Works
    CLI & engine
    Last
    deployed
    state
    index.ts
    Language host
    AWS
    Azure
    GCP
    Kubernetes
    new
    Resource()
    CRUD

    View Slide

  21. Tools That You Love
    Developers can apply their existing skills to infrastructure

    View Slide

  22. Pulumi relies on existing tools
    ● Compiler: tsc, dotnet, python3, go
    ● Language: TS, JS, C#, Python, Go, F#, VB.NET
    ● Editor and IDE: Visual Studio, Code, Rider, …
    ● IntelliSense, ReSharper, StyleCop, DocFX
    ● Package Manager: npm, NuGet, PyPi, Paket, …
    ● Unit Testing: mocha, NUnit, xUnit.net, Moq, …

    View Slide

  23. Architecture
    Reusable Abstractions

    View Slide

  24. Components Demo

    View Slide

  25. Testing and Policy
    Validate deployments

    View Slide

  26. [Test]
    public async Task ResourceGroupHasEnvironmentTag()
    {
    var resources = await Deployment.TestAsync();
    var resourceGroup = resources.OfType().First();
    var tags = await resourceGroup.Tags.GetValueAsync();
    tags.Should().NotBeNull("Tags must be defined");
    tags.Should().ContainKey("Environment");
    }
    Unit Testing

    View Slide

  27. it("Max distance between regions is at least 500 km", (done) => {
    sut.cosmosdbAccount.id.apply(id => {
    let max = 0;
    // Iterate through all pairs of regions and calculate locations.
    for (const regionA of accountLocations) {
    for (const regionB of accountLocations) {
    const distance = distanceBetweenRegions(regionA, regionB);
    if (distance > 500) {
    done();
    return;
    }
    max = Math.max(max, distance);
    }
    }
    done(new Error(`No regions are at least 500 km apart: max is ${max} km`));
    });
    });
    Unit Testing

    View Slide

  28. Policy as Code
    const policies = new PolicyPack("azure", {
    policies: [
    {
    name: "prohibited-public-internet",
    description: "Inbound rules with public internet access are prohibited.",
    enforcementLevel: "mandatory",
    validateResource: validateResourceOfType(
    azure.network.NetworkSecurityRule, (securityRule, args, reportViolation) => {
    if (securityRule.sourceAddressPrefix === "*") {
    reportViolation("Inbound public internet access rules are prohibited.");
    }
    }),
    }],
    });

    View Slide

  29. Management
    Multi-cloud cross-stack automation

    View Slide

  30. Transformations
    ● Apply consistent changes across resources in your stack
    ● The full power of general-purpose languages

    View Slide

  31. const autoTags = {
    "user:Project": pulumi.getProject(),
    "user:Stack": pulumi.getStack(),
    "user:Cost Center": config.require("costCenter"),
    };
    pulumi.runtime.registerStackTransformation((args) => {
    if (isTaggable(args.type)) {
    args.props["tags"] = {
    ...args.props["tags"],
    ...autoTags };
    return { props: args.props, opts: args.opts };
    }
    return undefined;
    });
    Example: Auto tagging resources

    View Slide

  32. const autoTags = {
    "user:Project": pulumi.getProject(),
    "user:Stack": pulumi.getStack(),
    "user:Cost Center": config.require("costCenter"),
    };
    pulumi.runtime.registerStackTransformation((args) => {
    if (isTaggable(args.type)) {
    args.props["tags"] = {
    ...args.props["tags"],
    ...autoTags };
    return { props: args.props, opts: args.opts };
    }
    return undefined;
    });
    Example: Auto tagging resources

    View Slide

  33. const autoTags = {
    "user:Project": pulumi.getProject(),
    "user:Stack": pulumi.getStack(),
    "user:Cost Center": config.require("costCenter"),
    };
    pulumi.runtime.registerStackTransformation((args) => {
    if (isTaggable(args.type)) {
    args.props["tags"] = {
    ...args.props["tags"],
    ...autoTags };
    return { props: args.props, opts: args.opts };
    }
    return undefined;
    });
    Example: Auto tagging resources

    View Slide

  34. Secret management
    ● Mark any input, output, or internal value as secret
    ● Encrypt with AWS KMS, Azure KeyVault, Google Cloud KMS,
    HashiCorp Vault, Pulumi Service, or self-managed key
    ● Automatic secret flow

    View Slide

  35. // Create a new KMS key
    const key = new aws.kms.Key("stack-encryption-key", {
    deletionWindowInDays: 10,
    description: "KMS key for encrypting secret values",
    });
    // Create a new alias to the key
    const alias = new aws.kms.Alias("alias", {
    targetKeyId: key.keyId,
    });
    export const aliasArn = alias.arn;
    Example: Create a KMS Key

    View Slide

  36. # In CLI
    pulumi new ...
    --secrets-provider="awskms://alias/${KEY_ALIAS}?region=us-west-2"
    // In code
    const superSecret = config.requireSecret("supersecret");
    const anotherSecret = pulumi.secret("a secret value");
    Example: Use the Key

    View Slide

  37. Stack References
    Org: acme-corp
    vpc
    Stack: dev
    env: dev
    region: us-east-1
    k8s-cluster
    Stack: dev
    env: dev
    region: us-east-1
    svc-userprofile
    Stack: dev
    env: dev
    region: us-east-1
    svc-email
    Stack: dev
    env: dev
    region: us-east-1

    View Slide

  38. Kubernetes layers
    Managed Kubernetes cluster
    Infrastructure Resources (networking, storage, identity)
    Managed Service Managed Service
    Application Application Application

    View Slide

  39. Kubernetes &
    Multi-stack
    Solutions
    Demo

    View Slide

  40. Automation API
    Orchestrate deployments from code

    View Slide

  41. What if I want to…
    ● Drive deployment workflows within CI/CD
    ● Test on ephemeral environments
    ● Multi-stage deployments (blue-green)
    ● Deploy application code and database migrations
    ● Build higher level tools, custom CLIs, application frameworks
    ● Use Pulumi behind a REST or gRPC API
    ● Debug programs as they execute

    View Slide

  42. Automation API Demo

    View Slide

  43. Conclusions

    View Slide

  44. PROVISIONING
    Developer-friendly
    Familiar language experience, toolchain,
    packages – applied to cloud infrastructure.
    Developers and operators working in a
    team.
    Cloud Engineering Transformed
    TESTING
    Confidence and quality
    Unit testing and TDD with battle-tested
    tools to ensure correctness.
    Policy as Code for compliance, cost
    control, and company-wide best
    practices.
    ARCHITECTURE
    Logic and abstractions
    Conditionals, loops, functions, classes,
    and packages out of the box.
    Reusable components that encapsulate
    complex logic and provider the right
    level of abstraction.
    Modern Infrastructure as Code Capabilities
    to ship faster and with confidence

    View Slide

  45. Inspiring Use Cases
    45
    Patterns
    Codified best practices
    shared as libraries
    Platforms
    Central team managing
    building blocks for other
    teams
    SaaS
    Provision infrastructure
    on-demand for every
    tenant

    View Slide

  46. Useful Links
    http://bit.ly/pulumilinks

    View Slide

  47. Q&A

    View Slide