Slide 1

Slide 1 text

Cloud Management Superpowers with Pulumi Mikhail Shilkov

Slide 2

Slide 2 text

About me • Mikhail Shilkov • Software engineer at Pulumi Azure, .NET SDK, Core platform • Microsoft Azure MVP @MikhailShilkov https://mikhail.io [email protected]

Slide 3

Slide 3 text

Intro ● Cloud Engineering ● Modern Infrastructure as Code Cloud Superpowers ● Provisioning ● Architecture ● Testing ● Policy ● Automation Agenda

Slide 4

Slide 4 text

Cloud Engineering Provision cloud infrastructure using C#, TypeScript, Python, Go

Slide 5

Slide 5 text

Infrastructure Landscape Foundation Security IAM KMS Networking VPC Subnets Firewalls Load Balancing DNS Compute VMs Containers Clusters Registries APM Monitoring Logging Alerting Serverless Functions API Gateways Data Object Stores Databases SQL NoSQL MQ Queues Pub/Sub Applications Images Container Images Code Packaging CI/CD

Slide 6

Slide 6 text

Azure API in numbers ● 140+ resource providers ● 900+ resource types ● 13.000+ properties to manage ● 10.000+ PRs and issues in the specifications repo

Slide 7

Slide 7 text

Kubernetes automation ● Open specifications ● Desired state configuration ● Reconciliation loop ● Operators ● Rolling updates ● Automation in DNA

Slide 8

Slide 8 text

PAST Lift-and-shift Virtual machines on demand. Rapid procurement cycles. Manual provisioning. Cloud Evolution FUTURE Cloud engineering Convergence with application development and software engineering. Abstractions for “the most powerful computer ever”. Architecture as code. PRESENT Cloud native Hundreds of managed services. Specialized solutions for broad set of problems. Infrastructure-as-code, desired state configuration. Cloud has been changing the world and it’s not done yet

Slide 9

Slide 9 text

Modern Applications ● Designed for the cloud ● Core business differenciator ● Fast time to market ● Broad footprint of resource types ● Resources under management grow fast

Slide 10

Slide 10 text

Modern Teams & Workflows ● Collaboration between Dev, IT, SRE, Security ● Cloud as a first-class target for developers ● Frequent delivery of value ● Automation from commit to production

Slide 11

Slide 11 text

Provisioning Cloud infrastructure using C#, TypeScript, Python, Go 10

Slide 12

Slide 12 text

Desired State Configuration Target Current Tool

Slide 13

Slide 13 text

Managing Resource Graphs Target Current Tool

Slide 14

Slide 14 text

Managing Resource Graphs Target Current Tool

Slide 15

Slide 15 text

General-purpose Programming Languages

Slide 16

Slide 16 text

Providers ● AWS ● Azure ● GCP ● Digital Ocean ● Cloudflare … and more ● Docker ● Kubernetes ● OpenStack ● PostgreSQL ● New Relic

Slide 17

Slide 17 text

var resourceGroup = new ResourceGroup("rg"); var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", }); C# Example

Slide 18

Slide 18 text

Desired State! var resourceGroup = new ResourceGroup("rg"); var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", });

Slide 19

Slide 19 text

Sample Pulumi Application Demo

Slide 20

Slide 20 text

How Pulumi Works CLI & engine Last deployed state index.ts Language host AWS Azure GCP Kubernetes new Resource() CRUD

Slide 21

Slide 21 text

Tools That You Love Developers can apply their existing skills to infrastructure

Slide 22

Slide 22 text

Pulumi relies on existing tools ● Compiler: tsc, dotnet, python3, go ● Language: TS, JS, C#, Python, Go, F#, VB.NET ● Editor and IDE: Visual Studio, Code, Rider, … ● IntelliSense, ReSharper, StyleCop, DocFX ● Package Manager: npm, NuGet, PyPi, Paket, … ● Unit Testing: mocha, NUnit, xUnit.net, Moq, …

Slide 23

Slide 23 text

Architecture Reusable Abstractions

Slide 24

Slide 24 text

Components Demo

Slide 25

Slide 25 text

Testing and Policy Validate deployments

Slide 26

Slide 26 text

[Test] public async Task ResourceGroupHasEnvironmentTag() { var resources = await Deployment.TestAsync(); var resourceGroup = resources.OfType().First(); var tags = await resourceGroup.Tags.GetValueAsync(); tags.Should().NotBeNull("Tags must be defined"); tags.Should().ContainKey("Environment"); } Unit Testing

Slide 27

Slide 27 text

it("Max distance between regions is at least 500 km", (done) => { sut.cosmosdbAccount.id.apply(id => { let max = 0; // Iterate through all pairs of regions and calculate locations. for (const regionA of accountLocations) { for (const regionB of accountLocations) { const distance = distanceBetweenRegions(regionA, regionB); if (distance > 500) { done(); return; } max = Math.max(max, distance); } } done(new Error(`No regions are at least 500 km apart: max is ${max} km`)); }); }); Unit Testing

Slide 28

Slide 28 text

Policy as Code const policies = new PolicyPack("azure", { policies: [ { name: "prohibited-public-internet", description: "Inbound rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateResourceOfType( azure.network.NetworkSecurityRule, (securityRule, args, reportViolation) => { if (securityRule.sourceAddressPrefix === "*") { reportViolation("Inbound public internet access rules are prohibited."); } }), }], });

Slide 29

Slide 29 text

Management Multi-cloud cross-stack automation

Slide 30

Slide 30 text

Transformations ● Apply consistent changes across resources in your stack ● The full power of general-purpose languages

Slide 31

Slide 31 text

const autoTags = { "user:Project": pulumi.getProject(), "user:Stack": pulumi.getStack(), "user:Cost Center": config.require("costCenter"), }; pulumi.runtime.registerStackTransformation((args) => { if (isTaggable(args.type)) { args.props["tags"] = { ...args.props["tags"], ...autoTags }; return { props: args.props, opts: args.opts }; } return undefined; }); Example: Auto tagging resources

Slide 32

Slide 32 text

const autoTags = { "user:Project": pulumi.getProject(), "user:Stack": pulumi.getStack(), "user:Cost Center": config.require("costCenter"), }; pulumi.runtime.registerStackTransformation((args) => { if (isTaggable(args.type)) { args.props["tags"] = { ...args.props["tags"], ...autoTags }; return { props: args.props, opts: args.opts }; } return undefined; }); Example: Auto tagging resources

Slide 33

Slide 33 text

const autoTags = { "user:Project": pulumi.getProject(), "user:Stack": pulumi.getStack(), "user:Cost Center": config.require("costCenter"), }; pulumi.runtime.registerStackTransformation((args) => { if (isTaggable(args.type)) { args.props["tags"] = { ...args.props["tags"], ...autoTags }; return { props: args.props, opts: args.opts }; } return undefined; }); Example: Auto tagging resources

Slide 34

Slide 34 text

Secret management ● Mark any input, output, or internal value as secret ● Encrypt with AWS KMS, Azure KeyVault, Google Cloud KMS, HashiCorp Vault, Pulumi Service, or self-managed key ● Automatic secret flow

Slide 35

Slide 35 text

// Create a new KMS key const key = new aws.kms.Key("stack-encryption-key", { deletionWindowInDays: 10, description: "KMS key for encrypting secret values", }); // Create a new alias to the key const alias = new aws.kms.Alias("alias", { targetKeyId: key.keyId, }); export const aliasArn = alias.arn; Example: Create a KMS Key

Slide 36

Slide 36 text

# In CLI pulumi new ... --secrets-provider="awskms://alias/${KEY_ALIAS}?region=us-west-2" // In code const superSecret = config.requireSecret("supersecret"); const anotherSecret = pulumi.secret("a secret value"); Example: Use the Key

Slide 37

Slide 37 text

Stack References Org: acme-corp vpc Stack: dev env: dev region: us-east-1 k8s-cluster Stack: dev env: dev region: us-east-1 svc-userprofile Stack: dev env: dev region: us-east-1 svc-email Stack: dev env: dev region: us-east-1

Slide 38

Slide 38 text

Kubernetes layers Managed Kubernetes cluster Infrastructure Resources (networking, storage, identity) Managed Service Managed Service Application Application Application

Slide 39

Slide 39 text

Kubernetes & Multi-stack Solutions Demo

Slide 40

Slide 40 text

Automation API Orchestrate deployments from code

Slide 41

Slide 41 text

What if I want to… ● Drive deployment workflows within CI/CD ● Test on ephemeral environments ● Multi-stage deployments (blue-green) ● Deploy application code and database migrations ● Build higher level tools, custom CLIs, application frameworks ● Use Pulumi behind a REST or gRPC API ● Debug programs as they execute

Slide 42

Slide 42 text

Automation API Demo

Slide 43

Slide 43 text

Conclusions

Slide 44

Slide 44 text

PROVISIONING Developer-friendly Familiar language experience, toolchain, packages – applied to cloud infrastructure. Developers and operators working in a team. Cloud Engineering Transformed TESTING Confidence and quality Unit testing and TDD with battle-tested tools to ensure correctness. Policy as Code for compliance, cost control, and company-wide best practices. ARCHITECTURE Logic and abstractions Conditionals, loops, functions, classes, and packages out of the box. Reusable components that encapsulate complex logic and provider the right level of abstraction. Modern Infrastructure as Code Capabilities to ship faster and with confidence

Slide 45

Slide 45 text

Inspiring Use Cases 45 Patterns Codified best practices shared as libraries Platforms Central team managing building blocks for other teams SaaS Provision infrastructure on-demand for every tenant

Slide 46

Slide 46 text

Useful Links http://bit.ly/pulumilinks

Slide 47

Slide 47 text

Q&A