$30 off During Our Annual Pro Sale. View Details »

Cloud Management Superpowers with Pulumi

Cloud Management Superpowers with Pulumi

n this talk, Mikhail will introduce Pulumi, an open-source tool that leverages programming languages like C#, TypeScript, and Python to manage cloud resources yet in a declarative manner! Using developer tools that you know and love, you can now apply unit testing, reusable architecture, deployment policies, and continuous delivery while provisioning resources in the cloud.

Mikhail Shilkov

September 08, 2020
Tweet

More Decks by Mikhail Shilkov

Other Decks in Technology

Transcript

  1. About me • Mikhail Shilkov • Software engineer at Pulumi

    Azure, .NET SDK, Core platform • Microsoft Azure MVP @MikhailShilkov https://mikhail.io [email protected]
  2. Intro • Cloud Engineering • Modern Infrastructure as Code Cloud

    Superpowers • Provisioning • Architecture • Testing • Policy • Automation Agenda
  3. Infrastructure Landscape Foundation Security IAM KMS Networking VPC Subnets Firewalls

    Load Balancing DNS Compute VMs Containers Clusters Registries APM Monitoring Logging Alerting Serverless Functions API Gateways Data Object Stores Databases SQL NoSQL MQ Queues Pub/Sub Applications Images Container Images Code Packaging CI/CD
  4. Azure API in numbers • 140+ resource providers • 900+

    resource types • 13.000+ properties to manage • 10.000+ PRs and issues in the specifications repo
  5. Kubernetes automation • Open specifications • Desired state configuration •

    Reconciliation loop • Operators • Rolling updates • Automation in DNA
  6. PAST Lift-and-shift Virtual machines on demand. Rapid procurement cycles. Manual

    provisioning. Cloud Evolution FUTURE Cloud engineering Convergence with application development and software engineering. Abstractions for “the most powerful computer ever”. Architecture as code. PRESENT Cloud native Hundreds of managed services. Specialized solutions for broad set of problems. Infrastructure-as-code, desired state configuration. Cloud has been changing the world and it’s not done yet
  7. Modern Applications • Designed for the cloud • Core business

    differenciator • Fast time to market • Broad footprint of resource types • Resources under management grow fast
  8. Modern Teams & Workflows • Collaboration between Dev, IT, SRE,

    Security • Cloud as a first-class target for developers • Frequent delivery of value • Automation from commit to production
  9. Providers • AWS • Azure • GCP • Digital Ocean

    • Cloudflare … and more • Docker • Kubernetes • OpenStack • PostgreSQL • New Relic
  10. var resourceGroup = new ResourceGroup("rg"); var storageAccount = new Account("storage",

    new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", }); C# Example
  11. Desired State! var resourceGroup = new ResourceGroup("rg"); var storageAccount =

    new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", });
  12. How Pulumi Works CLI & engine Last deployed state index.ts

    Language host AWS Azure GCP Kubernetes new Resource() CRUD
  13. Pulumi relies on existing tools • Compiler: tsc, dotnet, python3,

    go • Language: TS, JS, C#, Python, Go, F#, VB.NET • Editor and IDE: Visual Studio, Code, Rider, … • IntelliSense, ReSharper, StyleCop, DocFX • Package Manager: npm, NuGet, PyPi, Paket, … • Unit Testing: mocha, NUnit, xUnit.net, Moq, …
  14. [Test] public async Task ResourceGroupHasEnvironmentTag() { var resources = await

    Deployment.TestAsync<MyStack>(); var resourceGroup = resources.OfType<ResourceGroup>().First(); var tags = await resourceGroup.Tags.GetValueAsync(); tags.Should().NotBeNull("Tags must be defined"); tags.Should().ContainKey("Environment"); } Unit Testing
  15. it("Max distance between regions is at least 500 km", (done)

    => { sut.cosmosdbAccount.id.apply(id => { let max = 0; // Iterate through all pairs of regions and calculate locations. for (const regionA of accountLocations) { for (const regionB of accountLocations) { const distance = distanceBetweenRegions(regionA, regionB); if (distance > 500) { done(); return; } max = Math.max(max, distance); } } done(new Error(`No regions are at least 500 km apart: max is ${max} km`)); }); }); Unit Testing
  16. Policy as Code const policies = new PolicyPack("azure", { policies:

    [ { name: "prohibited-public-internet", description: "Inbound rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType( azure.network.NetworkSecurityRule, (securityRule, args, reportViolation) => { if (securityRule.sourceAddressPrefix === "*") { reportViolation("Inbound public internet access rules are prohibited."); } }), }], });
  17. Transformations • Apply consistent changes across resources in your stack

    • The full power of general-purpose languages
  18. const autoTags = { "user:Project": pulumi.getProject(), "user:Stack": pulumi.getStack(), "user:Cost Center":

    config.require("costCenter"), }; pulumi.runtime.registerStackTransformation((args) => { if (isTaggable(args.type)) { args.props["tags"] = { ...args.props["tags"], ...autoTags }; return { props: args.props, opts: args.opts }; } return undefined; }); Example: Auto tagging resources
  19. const autoTags = { "user:Project": pulumi.getProject(), "user:Stack": pulumi.getStack(), "user:Cost Center":

    config.require("costCenter"), }; pulumi.runtime.registerStackTransformation((args) => { if (isTaggable(args.type)) { args.props["tags"] = { ...args.props["tags"], ...autoTags }; return { props: args.props, opts: args.opts }; } return undefined; }); Example: Auto tagging resources
  20. const autoTags = { "user:Project": pulumi.getProject(), "user:Stack": pulumi.getStack(), "user:Cost Center":

    config.require("costCenter"), }; pulumi.runtime.registerStackTransformation((args) => { if (isTaggable(args.type)) { args.props["tags"] = { ...args.props["tags"], ...autoTags }; return { props: args.props, opts: args.opts }; } return undefined; }); Example: Auto tagging resources
  21. Secret management • Mark any input, output, or internal value

    as secret • Encrypt with AWS KMS, Azure KeyVault, Google Cloud KMS, HashiCorp Vault, Pulumi Service, or self-managed key • Automatic secret flow
  22. // Create a new KMS key const key = new

    aws.kms.Key("stack-encryption-key", { deletionWindowInDays: 10, description: "KMS key for encrypting secret values", }); // Create a new alias to the key const alias = new aws.kms.Alias("alias", { targetKeyId: key.keyId, }); export const aliasArn = alias.arn; Example: Create a KMS Key
  23. # In CLI pulumi new ... --secrets-provider="awskms://alias/${KEY_ALIAS}?region=us-west-2" // In code

    const superSecret = config.requireSecret("supersecret"); const anotherSecret = pulumi.secret("a secret value"); Example: Use the Key
  24. Stack References Org: acme-corp vpc Stack: dev env: dev region:

    us-east-1 k8s-cluster Stack: dev env: dev region: us-east-1 svc-userprofile Stack: dev env: dev region: us-east-1 svc-email Stack: dev env: dev region: us-east-1
  25. Kubernetes layers Managed Kubernetes cluster Infrastructure Resources (networking, storage, identity)

    Managed Service Managed Service Application Application Application
  26. What if I want to… • Drive deployment workflows within

    CI/CD • Test on ephemeral environments • Multi-stage deployments (blue-green) • Deploy application code and database migrations • Build higher level tools, custom CLIs, application frameworks • Use Pulumi behind a REST or gRPC API • Debug programs as they execute
  27. PROVISIONING Developer-friendly Familiar language experience, toolchain, packages – applied to

    cloud infrastructure. Developers and operators working in a team. Cloud Engineering Transformed TESTING Confidence and quality Unit testing and TDD with battle-tested tools to ensure correctness. Policy as Code for compliance, cost control, and company-wide best practices. ARCHITECTURE Logic and abstractions Conditionals, loops, functions, classes, and packages out of the box. Reusable components that encapsulate complex logic and provider the right level of abstraction. Modern Infrastructure as Code Capabilities to ship faster and with confidence
  28. Inspiring Use Cases 45 Patterns Codified best practices shared as

    libraries Platforms Central team managing building blocks for other teams SaaS Provision infrastructure on-demand for every tenant
  29. Q&A