Cloud Management Superpowers with Pulumi

n this talk, Mikhail will introduce Pulumi, an open-source tool that leverages programming languages like C#, TypeScript, and Python to manage cloud resources yet in a declarative manner! Using developer tools that you know and love, you can now apply unit testing, reusable architecture, deployment policies, and continuous delivery while provisioning resources in the cloud.

Transcript

1. Cloud Management Superpowers
   with Pulumi
   Mikhail Shilkov
   

   View Slide

2. About me
   • Mikhail Shilkov
   • Software engineer at Pulumi
   Azure, .NET SDK, Core platform
   • Microsoft Azure MVP
   @MikhailShilkov
   https://mikhail.io
   mikhail@pulumi.com
   

   View Slide

3. Intro
   ● Cloud Engineering
   ● Modern Infrastructure as Code
   Cloud Superpowers
   ● Provisioning
   ● Architecture
   ● Testing
   ● Policy
   ● Automation
   Agenda
   

   View Slide

4. Cloud Engineering
   Provision cloud infrastructure using C#, TypeScript, Python, Go
   

   View Slide

5. Infrastructure Landscape
   Foundation
   Security
   IAM
   KMS
   Networking
   VPC
   Subnets
   Firewalls
   Load Balancing
   DNS
   Compute
   VMs
   Containers
   Clusters
   Registries
   APM
   Monitoring
   Logging
   Alerting
   Serverless
   Functions
   API Gateways
   Data
   Object Stores
   Databases
   SQL
   NoSQL
   MQ
   Queues
   Pub/Sub
   Applications
   Images Container Images Code Packaging
   CI/CD
   

   View Slide

6. Azure API in numbers
   ● 140+ resource providers
   ● 900+ resource types
   ● 13.000+ properties to manage
   ● 10.000+ PRs and issues in the specifications repo
   

   View Slide

7. Kubernetes automation
   ● Open specifications
   ● Desired state configuration
   ● Reconciliation loop
   ● Operators
   ● Rolling updates
   ● Automation in DNA
   

   View Slide

8. PAST
   Lift-and-shift
   Virtual machines on demand. Rapid
   procurement cycles.
   Manual provisioning.
   Cloud Evolution
   FUTURE
   Cloud engineering
   Convergence with application
   development and software engineering.
   Abstractions for “the most powerful
   computer ever”. Architecture as code.
   PRESENT
   Cloud native
   Hundreds of managed services.
   Specialized solutions for broad set of
   problems.
   Infrastructure-as-code, desired state
   configuration.
   Cloud has been changing the world
   and it’s not done yet
   

   View Slide

9. Modern Applications
   ● Designed for the cloud
   ● Core business differenciator
   ● Fast time to market
   ● Broad footprint of resource types
   ● Resources under management grow fast
   

   View Slide

10. Modern Teams & Workflows
    ● Collaboration between Dev, IT, SRE, Security
    ● Cloud as a first-class target for developers
    ● Frequent delivery of value
    ● Automation from commit to production
    

    View Slide

11. Provisioning
    Cloud infrastructure using C#, TypeScript, Python, Go
    10
    

    View Slide

12. Desired State Configuration
    Target Current
    Tool
    

    View Slide

13. Managing Resource Graphs
    Target Current
    Tool
    

    View Slide

14. Managing Resource Graphs
    Target Current
    Tool
    

    View Slide

15. General-purpose Programming Languages
    

    View Slide

16. Providers
    ● AWS
    ● Azure
    ● GCP
    ● Digital Ocean
    ● Cloudflare
    … and more
    ● Docker
    ● Kubernetes
    ● OpenStack
    ● PostgreSQL
    ● New Relic
    

    View Slide

17. var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard",
    });
    C# Example
    

    View Slide

18. Desired
    State!
    var resourceGroup = new ResourceGroup("rg");
    var storageAccount = new Account("storage", new AccountArgs
    {
    ResourceGroupName = resourceGroup.Name,
    AccountReplicationType = "LRS",
    AccountTier = "Standard",
    });
    

    View Slide

19. Sample Pulumi
    Application
    Demo
    

    View Slide

20. How Pulumi Works
    CLI & engine
    Last
    deployed
    state
    index.ts
    Language host
    AWS
    Azure
    GCP
    Kubernetes
    new
    Resource()
    CRUD
    

    View Slide

21. Tools That You Love
    Developers can apply their existing skills to infrastructure
    

    View Slide

22. Pulumi relies on existing tools
    ● Compiler: tsc, dotnet, python3, go
    ● Language: TS, JS, C#, Python, Go, F#, VB.NET
    ● Editor and IDE: Visual Studio, Code, Rider, …
    ● IntelliSense, ReSharper, StyleCop, DocFX
    ● Package Manager: npm, NuGet, PyPi, Paket, …
    ● Unit Testing: mocha, NUnit, xUnit.net, Moq, …
    

    View Slide

23. Architecture
    Reusable Abstractions
    

    View Slide

24. Components Demo
    

    View Slide

25. Testing and Policy
    Validate deployments
    

    View Slide

26. [Test]
    public async Task ResourceGroupHasEnvironmentTag()
    {
    var resources = await Deployment.TestAsync();
    var resourceGroup = resources.OfType().First();
    var tags = await resourceGroup.Tags.GetValueAsync();
    tags.Should().NotBeNull("Tags must be defined");
    tags.Should().ContainKey("Environment");
    }
    Unit Testing
    

    View Slide

27. it("Max distance between regions is at least 500 km", (done) => {
    sut.cosmosdbAccount.id.apply(id => {
    let max = 0;
    // Iterate through all pairs of regions and calculate locations.
    for (const regionA of accountLocations) {
    for (const regionB of accountLocations) {
    const distance = distanceBetweenRegions(regionA, regionB);
    if (distance > 500) {
    done();
    return;
    }
    max = Math.max(max, distance);
    }
    }
    done(new Error(`No regions are at least 500 km apart: max is ${max} km`));
    });
    });
    Unit Testing
    

    View Slide

28. Policy as Code
    const policies = new PolicyPack("azure", {
    policies: [
    {
    name: "prohibited-public-internet",
    description: "Inbound rules with public internet access are prohibited.",
    enforcementLevel: "mandatory",
    validateResource: validateResourceOfType(
    azure.network.NetworkSecurityRule, (securityRule, args, reportViolation) => {
    if (securityRule.sourceAddressPrefix === "*") {
    reportViolation("Inbound public internet access rules are prohibited.");
    }
    }),
    }],
    });
    

    View Slide

29. Management
    Multi-cloud cross-stack automation
    

    View Slide

30. Transformations
    ● Apply consistent changes across resources in your stack
    ● The full power of general-purpose languages
    

    View Slide

31. const autoTags = {
    "user:Project": pulumi.getProject(),
    "user:Stack": pulumi.getStack(),
    "user:Cost Center": config.require("costCenter"),
    };
    pulumi.runtime.registerStackTransformation((args) => {
    if (isTaggable(args.type)) {
    args.props["tags"] = {
    ...args.props["tags"],
    ...autoTags };
    return { props: args.props, opts: args.opts };
    }
    return undefined;
    });
    Example: Auto tagging resources
    

    View Slide

32. const autoTags = {
    "user:Project": pulumi.getProject(),
    "user:Stack": pulumi.getStack(),
    "user:Cost Center": config.require("costCenter"),
    };
    pulumi.runtime.registerStackTransformation((args) => {
    if (isTaggable(args.type)) {
    args.props["tags"] = {
    ...args.props["tags"],
    ...autoTags };
    return { props: args.props, opts: args.opts };
    }
    return undefined;
    });
    Example: Auto tagging resources
    

    View Slide

33. const autoTags = {
    "user:Project": pulumi.getProject(),
    "user:Stack": pulumi.getStack(),
    "user:Cost Center": config.require("costCenter"),
    };
    pulumi.runtime.registerStackTransformation((args) => {
    if (isTaggable(args.type)) {
    args.props["tags"] = {
    ...args.props["tags"],
    ...autoTags };
    return { props: args.props, opts: args.opts };
    }
    return undefined;
    });
    Example: Auto tagging resources
    

    View Slide

34. Secret management
    ● Mark any input, output, or internal value as secret
    ● Encrypt with AWS KMS, Azure KeyVault, Google Cloud KMS,
    HashiCorp Vault, Pulumi Service, or self-managed key
    ● Automatic secret flow
    

    View Slide

35. // Create a new KMS key
    const key = new aws.kms.Key("stack-encryption-key", {
    deletionWindowInDays: 10,
    description: "KMS key for encrypting secret values",
    });
    // Create a new alias to the key
    const alias = new aws.kms.Alias("alias", {
    targetKeyId: key.keyId,
    });
    export const aliasArn = alias.arn;
    Example: Create a KMS Key
    

    View Slide

36. # In CLI
    pulumi new ...
    --secrets-provider="awskms://alias/${KEY_ALIAS}?region=us-west-2"
    // In code
    const superSecret = config.requireSecret("supersecret");
    const anotherSecret = pulumi.secret("a secret value");
    Example: Use the Key
    

    View Slide

37. Stack References
    Org: acme-corp
    vpc
    Stack: dev
    env: dev
    region: us-east-1
    k8s-cluster
    Stack: dev
    env: dev
    region: us-east-1
    svc-userprofile
    Stack: dev
    env: dev
    region: us-east-1
    svc-email
    Stack: dev
    env: dev
    region: us-east-1
    

    View Slide

38. Kubernetes layers
    Managed Kubernetes cluster
    Infrastructure Resources (networking, storage, identity)
    Managed Service Managed Service
    Application Application Application
    

    View Slide

39. Kubernetes &
    Multi-stack
    Solutions
    Demo
    

    View Slide

40. Automation API
    Orchestrate deployments from code
    

    View Slide

41. What if I want to…
    ● Drive deployment workflows within CI/CD
    ● Test on ephemeral environments
    ● Multi-stage deployments (blue-green)
    ● Deploy application code and database migrations
    ● Build higher level tools, custom CLIs, application frameworks
    ● Use Pulumi behind a REST or gRPC API
    ● Debug programs as they execute
    

    View Slide

42. Automation API Demo
    

    View Slide

43. Conclusions
    

    View Slide

44. PROVISIONING
    Developer-friendly
    Familiar language experience, toolchain,
    packages – applied to cloud infrastructure.
    Developers and operators working in a
    team.
    Cloud Engineering Transformed
    TESTING
    Confidence and quality
    Unit testing and TDD with battle-tested
    tools to ensure correctness.
    Policy as Code for compliance, cost
    control, and company-wide best
    practices.
    ARCHITECTURE
    Logic and abstractions
    Conditionals, loops, functions, classes,
    and packages out of the box.
    Reusable components that encapsulate
    complex logic and provider the right
    level of abstraction.
    Modern Infrastructure as Code Capabilities
    to ship faster and with confidence
    

    View Slide

45. Inspiring Use Cases
    45
    Patterns
    Codified best practices
    shared as libraries
    Platforms
    Central team managing
    building blocks for other
    teams
    SaaS
    Provision infrastructure
    on-demand for every
    tenant
    

    View Slide

46. Useful Links
    http://bit.ly/pulumilinks
    

    View Slide

47. Q&A
    

    View Slide