Slide 1

Slide 1 text

Cloud Management Superpowers with .NET Mikhail Shilkov

Slide 2

Slide 2 text

DevOps Developer IT Infrastructure User

Slide 3

Slide 3 text

DevOps in Cloud age Developer Cloud Infrastructure User IT

Slide 4

Slide 4 text

Cloud tooling Developer

Slide 5

Slide 5 text

About me • Mikhail Shilkov • Software engineer at Pulumi .NET SDK, Azure, Core platform • Serverless • Functional programming, F# • Microsoft Azure MVP @MikhailShilkov [email protected]

Slide 6

Slide 6 text

Intro ● Pulumi: Modern Infrastructure as Code ● .NET languages to manage cloud resources Cloud Superpowers ● Provisioning ● Architecture ● Testing ● Policy ● Delivery Agenda

Slide 7

Slide 7 text

.NET for All the Clouds Provision infrastructure anywhere using C#, F#, or VB.NET

Slide 8

Slide 8 text

General-purpose Programming Languages

Slide 9

Slide 9 text

Providers ● AWS ● Azure ● GCP ● Digital Ocean ● Cloudflare … and more ● Docker ● Kubernetes ● OpenStack ● PostgreSQL ● New Relic

Slide 10

Slide 10 text

Static Website on AWS Demo

Slide 11

Slide 11 text

Provisioning with Desired State Configuration 10

Slide 12

Slide 12 text

Cloud Management API Management API (e.g., Azure Resource Manager) Cloud Service Cloud Service Cloud Service Web Portal CLI SDK 3rd Party

Slide 13

Slide 13 text

var launchRequest = new RunInstancesRequest { ImageId = amiID, InstanceType = "t1.micro", MinCount = 1, MaxCount = 1, KeyName = keyPairName, SecurityGroupIds = groups }; var launchResponse = ec2Client.RunInstances(launchRequest); var instances = launchResponse.Reservation.Instances; SDKs to call those REST endpoints

Slide 14

Slide 14 text

aws ec2 run-instances \ --image-id ami-5ec1673e \ --key-name MyKey \ --security-groups EC2SecurityGroup \ --instance-type t2.micro \ --placement AvailabilityZone=us-west-2b Command-line Tools

Slide 15

Slide 15 text

SDK and Scripting: Challenges 15 Imperative Step-by-step provisioning process Complex Depend on both the current and the target state Error-prone How to handle failures?

Slide 16

Slide 16 text

Desired State Configuration Target Current Tool

Slide 17

Slide 17 text

Managing Resource Graphs Target Current Tool

Slide 18

Slide 18 text

var resourceGroup = new ResourceGroup("rg"); var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", }); C# Example

Slide 19

Slide 19 text

Desired State! var resourceGroup = new ResourceGroup("rg"); var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", });

Slide 20

Slide 20 text

How Pulumi Works CLI & engine Last deployed state Program.cs Language host AWS Azure GCP Kubernetes new Resource() CRUD

Slide 21

Slide 21 text

Markup-based Languages 21 AWS CloudFormation YAML Azure RM Templates JSON Terraform HCL "resources": [ { "apiVersion": "2016-01-01", "type": "Microsoft.Storage/storageAccounts", "name": "mystorageaccount", "location": "westus", "sku": { "name": "Standard_LRS" }, "kind": "Storage", "properties": { } } ] Resources: S3BucketForURLs: Type: "AWS::S3::Bucket" DeletionPolicy: Delete Properties: BucketName: !If [ "CreateNewBucket", !Ref WebsiteConfiguration: IndexDocument: "index.html" LifecycleConfiguration: Rules: - Id: DisposeShortUrls ExpirationInDays: !Ref URLExpiration Prefix: "u" Status: Enabled resource "azurerm_storage_account" "sa" { name = "mysa" location = azurerm_resource_group .rg.location resource_group_name = azurerm_resource_group .rg.name account_tier = "Standard" account_replication_type = "LRS" } Every alternative has its own language, own tools, own conventions

Slide 22

Slide 22 text

Markup Authoring Experience 22 Tooling Lack of tooling on day 1. No fast feedback from editors. Discovery Which properties can be set, and which values are valid? Size Thousands of lines of markup are hard to write, read, and maintain

Slide 23

Slide 23 text

Example App: Serverless URL Shortener KV Store Table “URLs” Function “Add URL” Function “Open URL” Tool Lines of code Function Code ~20 CloudFormation ~240 ARM Templates ~160 Terraform ~220 Estimates based on public examples

Slide 24

Slide 24 text

Tools That You Love Developers can apply their existing skills to infrastructure

Slide 25

Slide 25 text

Pulumi ❤ .NET ● .NET Core 3.1 ● OS: Windows, macOS, Linux ● Language: C#, F#, VB.NET ● Editor and IDE: Visual Studio, Code, Rider ● IntelliSense, ReSharper, StyleCop, DocFX ● Package Manager: NuGet, MyGet, Paket ● NUnit, xUnit.net, Moq

Slide 26

Slide 26 text

var storageAccount = new Account("sa", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard" }); var appServicePlan = new Plan("asp", new PlanArgs { ResourceGroupName = resourceGroup.Name, Kind = "App", Sku = new PlanSkuArgs { Tier = "Basic", Size = "B1” } }); Cloud Resources with C#

Slide 27

Slide 27 text

let storageAccount = Account("sa", AccountArgs (ResourceGroupName = io resourceGroup.Name, AccountReplicationType = input "LRS", AccountTier = input "Standard")) let sku = PlanSkuArgs(Tier = input "Basic", Size = input "B1") let appServicePlan = Plan("asp", PlanArgs (ResourceGroupName = io resourceGroup.Name, Kind = input "App", Sku = input sku)) Cloud Resources with F#

Slide 28

Slide 28 text

' Create an Azure Resource Group Dim resourceGroup = New ResourceGroup("resourceGroup") ' Create an Azure Storage Account Dim storageAccount = New Account("storage", New AccountArgs With { .ResourceGroupName = resourceGroup.Name, .AccountReplicationType = "LRS", .AccountTier = "Standard" }) Cloud Resources with VB.NET

Slide 29

Slide 29 text

Architecture Reusable Abstractions

Slide 30

Slide 30 text

Resources to Provision an Azure Function Blob App Plan Container Storage Account Function App

Slide 31

Slide 31 text

// A resource group to contain our Azure Functions var resourceGroup = new ResourceGroup("functions-rg"); // A .NET Azure Function var dotnet = new ArchiveFunctionApp("http", new ArchiveFunctionAppArgs { ResourceGroup = resourceGroup, Archive = new FileArchive("./dotnet/bin/Debug/netcoreapp3.1/publish") }); .NET Azure Function

Slide 32

Slide 32 text

Let’s run pulumi up … Type Name Plan + pulumi:pulumi:Stack functions-dev create + ├─ azure:appservice:ArchiveFunctionApp http create + │ ├─ azure:storage:Account http create + │ ├─ azure:appservice:Plan http create + │ ├─ azure:storage:Container http create + │ ├─ azure:storage:ZipBlob http create + │ └─ azure:appservice:FunctionApp http create + └─ azure:core:ResourceGroup rg create

Slide 33

Slide 33 text

// A Node.js Azure Function var js = new ArchiveFunctionApp("js", new ArchiveFunctionAppArgs { ResourceGroup = resourceGroup, Archive = new FileArchive("./javascript"), Runtime = "node" }); Node.js Azure Function

Slide 34

Slide 34 text

var linuxResourceGroup = new ResourceGroup("linux-rg"); var linuxPlan = new Plan("linux-asp", new PlanArgs { ResourceGroupName = linuxResourceGroup.Name, Kind = "Linux", Reserved = true, Sku = new PlanSkuArgs { Tier = "Dynamic", Size = "Y1" } }); var python = new ArchiveFunctionApp(...); Python Azure Function on Linux

Slide 35

Slide 35 text

var linuxResourceGroup = new ResourceGroup("linux-rg"); var linuxPlan = new Plan("linux-asp", new PlanArgs { ... }); var python = new ArchiveFunctionApp("py", new ArchiveFunctionAppArgs { ResourceGroup = linuxResourceGroup, Archive = new FileArchive("./python"), Plan = linuxPlan, Runtime = "python" }); Python Azure Function on Linux

Slide 36

Slide 36 text

Building a Cosmos App Create custom abstractions

Slide 37

Slide 37 text

Cosmos App (1)

Slide 38

Slide 38 text

Cosmos App (2)

Slide 39

Slide 39 text

Cosmos App (3)

Slide 40

Slide 40 text

Cosmos App (4)

Slide 41

Slide 41 text

var app = new CosmosApp("functions", new CosmosAppArgs { ResourceGroup = resourceGroup, Locations = ["WestUS", "EastUS", "WestEurope"], Factory = global => region => { var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs { ResourceGroupName = resourceGroup.Name, Location = region.Location, Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"), AppSettings = { { "CosmosDBConnection", global.ConnectionString } } }); return new AzureEndpoint(func.AppId); } }); Global Applications

Slide 42

Slide 42 text

var app = new CosmosApp("functions", new CosmosAppArgs { ResourceGroup = resourceGroup, Locations = ["WestUS", "EastUS", "WestEurope"], Factory = global => region => { var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs { ResourceGroupName = resourceGroup.Name, Location = region.Location, Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"), AppSettings = { { "CosmosDBConnection", global.ConnectionString } } }); return new AzureEndpoint(func.AppId); } }); Global Applications

Slide 43

Slide 43 text

Testing and Policy Validate deployments

Slide 44

Slide 44 text

[Fact] public async Task InstancesMustNotHaveSshPortOpenToInternet() { var resources = await Deployment.TestAsync(); var resource = resources.OfType().Single(); var ingress = await resource.GetAsync(sg => sg.Ingress); foreach (var rule in ingress) foreach (var cidrBlock in rule.CidrBlocks) { Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0", "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)"); } } Unit Testing

Slide 45

Slide 45 text

[Fact] public async Task InstancesMustNotHaveSshPortOpenToInternet() { var resources = await Deployment.TestAsync(); var resource = resources.OfType().Single(); var ingress = await resource.GetAsync(sg => sg.Ingress); foreach (var rule in ingress) foreach (var cidrBlock in rule.CidrBlocks) { Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0", "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)"); } } Unit Testing

Slide 46

Slide 46 text

[Fact] public async Task InstancesMustNotHaveSshPortOpenToInternet() { var resources = await Deployment.TestAsync(); var resource = resources.OfType().Single(); var ingress = await resource.GetAsync(sg => sg.Ingress); foreach (var rule in ingress) foreach (var cidrBlock in rule.CidrBlocks) { Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0", "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)"); } } Unit Testing

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

Policy as Code (.NET coming later) const policies = new PolicyPack("aws", { policies: [ { name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateTypedResource(aws.ec2.SecurityGroup, (sg, args, report) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { report("Ingress rules with public internet access are prohibited."); } }), }, ], });

Slide 49

Slide 49 text

Delivery Multi-cloud cross-stack automation

Slide 50

Slide 50 text

State Management CLI and Engine Last deployed state Program.cs Language host AWS Azure GCP Kubernetes new Resource() CRUD Pulumi Service S3 / Blob Storage Local disk

Slide 51

Slide 51 text

Pulumi Components 51 CLI Open-source Free for any use Cross-platform Written in Go Providers Open-source Free for any use Can be created by 1st or 3rd party SaaS backend Web API + portal “GitHub for infra” Free for individuals Paid plans for teams Useful but not required

Slide 52

Slide 52 text

Deployment process Allow developers to iterate quickly on dev environments. Gain visibility and change management with Continuous Delivery.

Slide 53

Slide 53 text

Cloud Transition 53 Kubernetes Azure Functions AWS S3 Azure Analytics Google ML MySQL DataDog App Docker DataDog New Relic App MySQL EARLY CLOUD Mostly Static N-Tier Apps VMs Private Cloud CURRENT CLOUD Partly Dynamic Less Monolithic VMs and Containers* Hybrid - Public/Private *ExperimentaDon FUTURE CLOUD Fully Dynamic Hyper-Connected Services Containers and Serverless Mostly Public Cloud

Slide 54

Slide 54 text

Infrastructure Landscape 54 Foundation Security IAM KMS Networking VPC Subnets Firewalls Load Balancing DNS Compute VMs Containers Clusters Registries APM Monitoring Logging Alerting Serverless Functions API Gateways Data Object Stores Databases SQL NoSQL MQ Queues Pub/Sub Applications Images Container Images Code Packaging CI/CD

Slide 55

Slide 55 text

Kubernetes layers Managed Kubernetes cluster Infrastructure Resources (networking, storage, identity) Managed Service Managed Service Application Application Application

Slide 56

Slide 56 text

Stack References Org: acme-corp vpc Stack: dev env: dev region: us-east-1 k8s-cluster Stack: dev env: dev region: us-east-1 svc-userprofile Stack: dev env: dev region: us-east-1 svc-email Stack: dev env: dev region: us-east-1

Slide 57

Slide 57 text

Kubernetes & Multi-stack Solutions Demo

Slide 58

Slide 58 text

Conclusions

Slide 59

Slide 59 text

PROVISIONING Developer-friendly Familiar language experience, toolchain, packages – applied to cloud infrastructure. Developers and operators working in a team. Cloud Engineering Transformed TESTING Confidence and quality Unit testing and TDD with battle-tested tools to ensure correctness. Policy as Code for compliance, cost control, and company-wide best practices. ARCHITECTURE Logic and abstractions Conditionals, loops, functions, classes, and packages out of the box. Reusable components that encapsulate complex logic and provider the right level of abstraction. Modern Infrastructure as Code Capabilities to ship faster and with confidence

Slide 60

Slide 60 text

Useful Links http://bit.ly/pulumilinks

Slide 61

Slide 61 text

Q&A