Cloud Management Superpowers with Pulumi Mikhail Shilkov

About me • Mikhail Shilkov • Software engineer at Pulumi Azure, .NET SDK, Core platform • Microsoft Azure MVP @MikhailShilkov https://mikhail.io [email protected]

Intro ● Cloud Engineering ● Modern Infrastructure as Code Cloud Superpowers ● Provisioning ● Architecture ● Testing ● Policy ● Automation Agenda

Cloud Engineering Provision cloud infrastructure using C#, TypeScript, Python, Go

Infrastructure Landscape Foundation Security IAM KMS Networking VPC Subnets Firewalls Load Balancing DNS Compute VMs Containers Clusters Registries APM Monitoring Logging Alerting Serverless Functions API Gateways Data Object Stores Databases SQL NoSQL MQ Queues Pub/Sub Applications Images Container Images Code Packaging CI/CD

Azure API in numbers ● 140+ resource providers ● 900+ resource types ● 13.000+ properties to manage ● 10.000+ PRs and issues in the specifications repo

Kubernetes automation ● Open specifications ● Desired state configuration ● Reconciliation loop ● Operators ● Rolling updates ● Automation in DNA

PAST Lift-and-shift Virtual machines on demand. Rapid procurement cycles. Manual provisioning. Cloud Evolution FUTURE Cloud engineering Convergence with application development and software engineering. Abstractions for “the most powerful computer ever”. Architecture as code. PRESENT Cloud native Hundreds of managed services. Specialized solutions for broad set of problems. Infrastructure-as-code, desired state configuration. Cloud has been changing the world and it’s not done yet

Modern Applications ● Designed for the cloud ● Core business differenciator ● Fast time to market ● Broad footprint of resource types ● Resources under management grow fast

Modern Teams & Workflows ● Collaboration between Dev, IT, SRE, Security ● Cloud as a first-class target for developers ● Frequent delivery of value ● Automation from commit to production

Provisioning Cloud infrastructure using C#, TypeScript, Python, Go 10

Desired State Configuration Target Current Tool

Managing Resource Graphs Target Current Tool

Managing Resource Graphs Target Current Tool

General-purpose Programming Languages

Providers ● AWS ● Azure ● GCP ● Digital Ocean ● Cloudflare … and more ● Docker ● Kubernetes ● OpenStack ● PostgreSQL ● New Relic

var resourceGroup = new ResourceGroup("rg"); var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", }); C# Example

Desired State! var resourceGroup = new ResourceGroup("rg"); var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", });

Sample Pulumi Application Demo

How Pulumi Works CLI & engine Last deployed state index.ts Language host AWS Azure GCP Kubernetes new Resource() CRUD

Tools That You Love Developers can apply their existing skills to infrastructure

Pulumi relies on existing tools ● Compiler: tsc, dotnet, python3, go ● Language: TS, JS, C#, Python, Go, F#, VB.NET ● Editor and IDE: Visual Studio, Code, Rider, … ● IntelliSense, ReSharper, StyleCop, DocFX ● Package Manager: npm, NuGet, PyPi, Paket, … ● Unit Testing: mocha, NUnit, xUnit.net, Moq, …

Architecture Reusable Abstractions

Components Demo

Testing and Policy Validate deployments

[Test] public async Task ResourceGroupHasEnvironmentTag() { var resources = await Deployment.TestAsync(); var resourceGroup = resources.OfType().First(); var tags = await resourceGroup.Tags.GetValueAsync(); tags.Should().NotBeNull("Tags must be defined"); tags.Should().ContainKey("Environment"); } Unit Testing

it("Max distance between regions is at least 500 km", (done) => { sut.cosmosdbAccount.id.apply(id => { let max = 0; // Iterate through all pairs of regions and calculate locations. for (const regionA of accountLocations) { for (const regionB of accountLocations) { const distance = distanceBetweenRegions(regionA, regionB); if (distance > 500) { done(); return; } max = Math.max(max, distance); } } done(new Error(`No regions are at least 500 km apart: max is ${max} km`)); }); }); Unit Testing

Policy as Code const policies = new PolicyPack("azure", { policies: [ { name: "prohibited-public-internet", description: "Inbound rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType( azure.network.NetworkSecurityRule, (securityRule, args, reportViolation) => { if (securityRule.sourceAddressPrefix === "*") { reportViolation("Inbound public internet access rules are prohibited."); } }), }], });

Management Multi-cloud cross-stack automation

Transformations ● Apply consistent changes across resources in your stack ● The full power of general-purpose languages

const autoTags = { "user:Project": pulumi.getProject(), "user:Stack": pulumi.getStack(), "user:Cost Center": config.require("costCenter"), }; pulumi.runtime.registerStackTransformation((args) => { if (isTaggable(args.type)) { args.props["tags"] = { ...args.props["tags"], ...autoTags }; return { props: args.props, opts: args.opts }; } return undefined; }); Example: Auto tagging resources

const autoTags = { "user:Project": pulumi.getProject(), "user:Stack": pulumi.getStack(), "user:Cost Center": config.require("costCenter"), }; pulumi.runtime.registerStackTransformation((args) => { if (isTaggable(args.type)) { args.props["tags"] = { ...args.props["tags"], ...autoTags }; return { props: args.props, opts: args.opts }; } return undefined; }); Example: Auto tagging resources

const autoTags = { "user:Project": pulumi.getProject(), "user:Stack": pulumi.getStack(), "user:Cost Center": config.require("costCenter"), }; pulumi.runtime.registerStackTransformation((args) => { if (isTaggable(args.type)) { args.props["tags"] = { ...args.props["tags"], ...autoTags }; return { props: args.props, opts: args.opts }; } return undefined; }); Example: Auto tagging resources

Secret management ● Mark any input, output, or internal value as secret ● Encrypt with AWS KMS, Azure KeyVault, Google Cloud KMS, HashiCorp Vault, Pulumi Service, or self-managed key ● Automatic secret flow

// Create a new KMS key const key = new aws.kms.Key("stack-encryption-key", { deletionWindowInDays: 10, description: "KMS key for encrypting secret values", }); // Create a new alias to the key const alias = new aws.kms.Alias("alias", { targetKeyId: key.keyId, }); export const aliasArn = alias.arn; Example: Create a KMS Key

# In CLI pulumi new ... --secrets-provider="awskms://alias/${KEY_ALIAS}?region=us-west-2" // In code const superSecret = config.requireSecret("supersecret"); const anotherSecret = pulumi.secret("a secret value"); Example: Use the Key

Stack References Org: acme-corp vpc Stack: dev env: dev region: us-east-1 k8s-cluster Stack: dev env: dev region: us-east-1 svc-userprofile Stack: dev env: dev region: us-east-1 svc-email Stack: dev env: dev region: us-east-1

Kubernetes layers Managed Kubernetes cluster Infrastructure Resources (networking, storage, identity) Managed Service Managed Service Application Application Application

Kubernetes & Multi-stack Solutions Demo

Automation API Orchestrate deployments from code

What if I want to… ● Drive deployment workflows within CI/CD ● Test on ephemeral environments ● Multi-stage deployments (blue-green) ● Deploy application code and database migrations ● Build higher level tools, custom CLIs, application frameworks ● Use Pulumi behind a REST or gRPC API ● Debug programs as they execute

Automation API Demo

Conclusions

PROVISIONING Developer-friendly Familiar language experience, toolchain, packages – applied to cloud infrastructure. Developers and operators working in a team. Cloud Engineering Transformed TESTING Confidence and quality Unit testing and TDD with battle-tested tools to ensure correctness. Policy as Code for compliance, cost control, and company-wide best practices. ARCHITECTURE Logic and abstractions Conditionals, loops, functions, classes, and packages out of the box. Reusable components that encapsulate complex logic and provider the right level of abstraction. Modern Infrastructure as Code Capabilities to ship faster and with confidence

Inspiring Use Cases 45 Patterns Codified best practices shared as libraries Platforms Central team managing building blocks for other teams SaaS Provision infrastructure on-demand for every tenant

Useful Links http://bit.ly/pulumilinks

Q&A