Cloud Management Superpowers with Pulumi

n this talk, Mikhail will introduce Pulumi, an open-source tool that leverages programming languages like C#, TypeScript, and Python to manage cloud resources yet in a declarative manner! Using developer tools that you know and love, you can now apply unit testing, reusable architecture, deployment policies, and continuous delivery while provisioning resources in the cloud.

Transcript

1. Cloud Management Superpowers with Pulumi Mikhail Shilkov
2. About me • Mikhail Shilkov • Software engineer at Pulumi Azure, .NET SDK, Core platform • Microsoft Azure MVP @MikhailShilkov https://mikhail.io mikhail@pulumi.com
3. Intro • Cloud Engineering • Modern Infrastructure as Code Cloud Superpowers • Provisioning • Architecture • Testing • Policy • Automation Agenda
4. Cloud Engineering Provision cloud infrastructure using C#, TypeScript, Python, Go
5. Infrastructure Landscape Foundation Security IAM KMS Networking VPC Subnets Firewalls Load Balancing DNS Compute VMs Containers Clusters Registries APM Monitoring Logging Alerting Serverless Functions API Gateways Data Object Stores Databases SQL NoSQL MQ Queues Pub/Sub Applications Images Container Images Code Packaging CI/CD
6. Azure API in numbers • 140+ resource providers • 900+ resource types • 13.000+ properties to manage • 10.000+ PRs and issues in the specifications repo
7. Kubernetes automation • Open specifications • Desired state configuration • Reconciliation loop • Operators • Rolling updates • Automation in DNA
8. PAST Lift-and-shift Virtual machines on demand. Rapid procurement cycles. Manual provisioning. Cloud Evolution FUTURE Cloud engineering Convergence with application development and software engineering. Abstractions for “the most powerful computer ever”. Architecture as code. PRESENT Cloud native Hundreds of managed services. Specialized solutions for broad set of problems. Infrastructure-as-code, desired state configuration. Cloud has been changing the world and it’s not done yet
9. Modern Applications • Designed for the cloud • Core business differenciator • Fast time to market • Broad footprint of resource types • Resources under management grow fast
10. Modern Teams & Workflows • Collaboration between Dev, IT, SRE, Security • Cloud as a first-class target for developers • Frequent delivery of value • Automation from commit to production
11. Provisioning Cloud infrastructure using C#, TypeScript, Python, Go 10
12. Desired State Configuration Target Current Tool
13. Managing Resource Graphs Target Current Tool
14. Managing Resource Graphs Target Current Tool
15. General-purpose Programming Languages
16. Providers • AWS • Azure • GCP • Digital Ocean • Cloudflare … and more • Docker • Kubernetes • OpenStack • PostgreSQL • New Relic
17. var resourceGroup = new ResourceGroup("rg"); var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", }); C# Example
18. Desired State! var resourceGroup = new ResourceGroup("rg"); var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", });
19. Sample Pulumi Application Demo
20. How Pulumi Works CLI & engine Last deployed state index.ts Language host AWS Azure GCP Kubernetes new Resource() CRUD
21. Tools That You Love Developers can apply their existing skills to infrastructure
22. Pulumi relies on existing tools • Compiler: tsc, dotnet, python3, go • Language: TS, JS, C#, Python, Go, F#, VB.NET • Editor and IDE: Visual Studio, Code, Rider, … • IntelliSense, ReSharper, StyleCop, DocFX • Package Manager: npm, NuGet, PyPi, Paket, … • Unit Testing: mocha, NUnit, xUnit.net, Moq, …
23. Architecture Reusable Abstractions
24. Components Demo
25. Testing and Policy Validate deployments
26. [Test] public async Task ResourceGroupHasEnvironmentTag() { var resources = await Deployment.TestAsync<MyStack>(); var resourceGroup = resources.OfType<ResourceGroup>().First(); var tags = await resourceGroup.Tags.GetValueAsync(); tags.Should().NotBeNull("Tags must be defined"); tags.Should().ContainKey("Environment"); } Unit Testing
27. it("Max distance between regions is at least 500 km", (done) => { sut.cosmosdbAccount.id.apply(id => { let max = 0; // Iterate through all pairs of regions and calculate locations. for (const regionA of accountLocations) { for (const regionB of accountLocations) { const distance = distanceBetweenRegions(regionA, regionB); if (distance > 500) { done(); return; } max = Math.max(max, distance); } } done(new Error(`No regions are at least 500 km apart: max is ${max} km`)); }); }); Unit Testing
28. Policy as Code const policies = new PolicyPack("azure", { policies: [ { name: "prohibited-public-internet", description: "Inbound rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType( azure.network.NetworkSecurityRule, (securityRule, args, reportViolation) => { if (securityRule.sourceAddressPrefix === "*") { reportViolation("Inbound public internet access rules are prohibited."); } }), }], });
29. Management Multi-cloud cross-stack automation
30. Transformations • Apply consistent changes across resources in your stack • The full power of general-purpose languages
31. const autoTags = { "user:Project": pulumi.getProject(), "user:Stack": pulumi.getStack(), "user:Cost Center": config.require("costCenter"), }; pulumi.runtime.registerStackTransformation((args) => { if (isTaggable(args.type)) { args.props["tags"] = { ...args.props["tags"], ...autoTags }; return { props: args.props, opts: args.opts }; } return undefined; }); Example: Auto tagging resources
32. const autoTags = { "user:Project": pulumi.getProject(), "user:Stack": pulumi.getStack(), "user:Cost Center": config.require("costCenter"), }; pulumi.runtime.registerStackTransformation((args) => { if (isTaggable(args.type)) { args.props["tags"] = { ...args.props["tags"], ...autoTags }; return { props: args.props, opts: args.opts }; } return undefined; }); Example: Auto tagging resources
33. const autoTags = { "user:Project": pulumi.getProject(), "user:Stack": pulumi.getStack(), "user:Cost Center": config.require("costCenter"), }; pulumi.runtime.registerStackTransformation((args) => { if (isTaggable(args.type)) { args.props["tags"] = { ...args.props["tags"], ...autoTags }; return { props: args.props, opts: args.opts }; } return undefined; }); Example: Auto tagging resources
34. Secret management • Mark any input, output, or internal value as secret • Encrypt with AWS KMS, Azure KeyVault, Google Cloud KMS, HashiCorp Vault, Pulumi Service, or self-managed key • Automatic secret flow
35. // Create a new KMS key const key = new aws.kms.Key("stack-encryption-key", { deletionWindowInDays: 10, description: "KMS key for encrypting secret values", }); // Create a new alias to the key const alias = new aws.kms.Alias("alias", { targetKeyId: key.keyId, }); export const aliasArn = alias.arn; Example: Create a KMS Key
36. # In CLI pulumi new ... --secrets-provider="awskms://alias/${KEY_ALIAS}?region=us-west-2" // In code const superSecret = config.requireSecret("supersecret"); const anotherSecret = pulumi.secret("a secret value"); Example: Use the Key
37. Stack References Org: acme-corp vpc Stack: dev env: dev region: us-east-1 k8s-cluster Stack: dev env: dev region: us-east-1 svc-userprofile Stack: dev env: dev region: us-east-1 svc-email Stack: dev env: dev region: us-east-1
38. Kubernetes layers Managed Kubernetes cluster Infrastructure Resources (networking, storage, identity) Managed Service Managed Service Application Application Application
39. Kubernetes & Multi-stack Solutions Demo
40. Automation API Orchestrate deployments from code
41. What if I want to… • Drive deployment workflows within CI/CD • Test on ephemeral environments • Multi-stage deployments (blue-green) • Deploy application code and database migrations • Build higher level tools, custom CLIs, application frameworks • Use Pulumi behind a REST or gRPC API • Debug programs as they execute
42. Automation API Demo
43. Conclusions
44. PROVISIONING Developer-friendly Familiar language experience, toolchain, packages – applied to cloud infrastructure. Developers and operators working in a team. Cloud Engineering Transformed TESTING Confidence and quality Unit testing and TDD with battle-tested tools to ensure correctness. Policy as Code for compliance, cost control, and company-wide best practices. ARCHITECTURE Logic and abstractions Conditionals, loops, functions, classes, and packages out of the box. Reusable components that encapsulate complex logic and provider the right level of abstraction. Modern Infrastructure as Code Capabilities to ship faster and with confidence
45. Inspiring Use Cases 45 Patterns Codified best practices shared as libraries Platforms Central team managing building blocks for other teams SaaS Provision infrastructure on-demand for every tenant
46. Useful Links http://bit.ly/pulumilinks
47. Q&A