Tweet
Tweet

Slide 1

Slide 1 text

Cloud Infrastructure as C# and F# with Pulumi Mikhail Shilkov

Slide 2

Slide 2 text

• Software engineer at Pulumi: .NET SDK, Azure, Core platform • Cloud, Serverless • Functional programming, F# • Microsoft Azure MVP https://mikhail.io @MikhailShilkov About me

Slide 3

Slide 3 text

Agenda

Slide 4

Slide 4 text

Agenda

Slide 5

Slide 5 text

Agenda

Slide 6

Slide 6 text

Cloud Automation 5

Slide 7

Slide 7 text

Unified Management API Management API (e.g., Azure Resource Manager) Cloud Service Cloud Service Cloud Service Web Portal CLI SDK 3rd Party

Slide 8

Slide 8 text

REST calls Content-Type: application/x-www-form-urlencoded; charset=UTF-8 X-Amz-Date: 20130813T150211Z Host: ec2.amazonaws.com Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20130813/us-east- 1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz- date, Signature=ced6826de92d2bdeed8f846f0bf508e8559e98e4b0194b84example54174deb456c http://ec2.amazonaws.com/?Action=RunInstances ImageId=ami-2bb65342 &MaxCount=3 &MinCount=1 &Monitoring.Enabled=true &Placement.AvailabilityZone=us-east-1a &Version=2016-11-15

Slide 9

Slide 9 text

var launchRequest = new RunInstancesRequest { ImageId = amiID, InstanceType = "t1.micro", MinCount = 1, MaxCount = 1, KeyName = keyPairName, SecurityGroupIds = groups }; var launchResponse = ec2Client.RunInstances(launchRequest); var instances = launchResponse.Reservation.Instances; SDKs to call those REST endpoints

Slide 10

Slide 10 text

aws ec2 run-instances \ --image-id ami-5ec1673e \ --key-name MyKey \ --security-groups EC2SecurityGroup \ --instance-type t2.micro \ --placement AvailabilityZone=us-west-2b Command-line Tools

Slide 11

Slide 11 text

Step-by-step provisioning process Depend on both the current and the target state How to handle failures? Imperative Complex Error-prone SDK and Scripting: Challenges

Slide 12

Slide 12 text

Desired State Configuration 10

Slide 13

Slide 13 text

Desired State Configuration Target Current Tool

Slide 14

Slide 14 text

Managing Resource Graphs Target Current Tool

Slide 15

Slide 15 text

Be able to describe the desired state in full, concisely, and correctly Determine the correct order of creation and destruction Preview the changes before running the deployment Expressiveness Correctness Transparency Challenges

Slide 16

Slide 16 text

AWS CloudFormation Azure Resource Manager Google Cloud Deployment Manager Vendor Tools

Slide 17

Slide 17 text

Markup Languages 14 YET ANOTHER MARKUP LANGUAGE

Slide 18

Slide 18 text

Azure ARM Templates: JSON "resources": [ { "apiVersion": "2016-01-01", "type": "Microsoft.Storage/storageAccounts", "name": "mystorageaccount", "location": "westus", "sku": { "name": "Standard_LRS" }, "kind": "Storage", "properties": { } } ]

Slide 19

Slide 19 text

AWS CloudFormation: YAML Resources: S3BucketForURLs: Type: "AWS::S3::Bucket" DeletionPolicy: Delete Properties: BucketName: !If [ "CreateNewBucket", !Ref "AWS::NoValue", !Ref S3BucketName ] WebsiteConfiguration: IndexDocument: "index.html" LifecycleConfiguration: Rules: - Id: DisposeShortUrls ExpirationInDays: !Ref URLExpiration Prefix: "u" Status: Enabled

Slide 20

Slide 20 text

resource "azurerm_storage_account" "sa" { name = "mysa" location = azurerm_resource_group.rg.location resource_group_name = azurerm_resource_group.rg.name account_tier = "Standard" account_replication_type = "LRS" } Terraform: DSL (HashiCorp Configuration Language)

Slide 21

Slide 21 text

Lack of fast feedback about errors Which options need to be filled and which values are possible Thousands of lines of markup are hard to write, read, and navigate Tooling Discovery Size Markup Authoring Experience

Slide 22

Slide 22 text

Example: Code to infrastructure ratio KV Store Table “URLs” Function “Add URL” Function “Open URL” Lines of code Function Code ~20 CloudFormation ~240 ARM Templates ~160 Terraform ~220 Estimates based on public examples

Slide 23

Slide 23 text

{ "Resources": { "InstanceSecurityGroup": { "Type": "AWS::EC2::SecurityGroup", "Properties": { "SecurityGroupIngress": [ ... ] } }, "EC2Instance": { "Type": "AWS::EC2::Instance", "Properties": { "InstanceType": "t2.micro", "SecurityGroups": [ { "Ref": "InstanceSecurityGroup" } ], "ImageId": "ami-0080e4c5bc078760e" } } } } References Example for AWS CloudFormation

Slide 24

Slide 24 text

{ "condition": "[equals(parameters('production'), 'Yes')]", "type": "Microsoft.Compute/availabilitySets", "apiVersion": "2017-03-30", "name": "[variables('availabilitySetName')]", "location": "[resourceGroup().location]", "properties": { "platformFaultDomainCount": 2, "platformUpdateDomainCount": 3 } } Conditionals Example for Azure RM Templates

Slide 25

Slide 25 text

locals { files = ["index.php","main.php"] } resource "aws_s3_bucket_object" "files" { count = "${length(local.files)}" key = "${local.files[count.index]}" bucket = "${aws_s3_bucket.examplebucket.id}" source = "./src/${local.files[count.index]}" } Loops Example for Terraform

Slide 26

Slide 26 text

Different shape of infrastructure for different environments Bundling several related resources together for a higher-level component Sharing libraries of components within the company or in community Configuration Abstraction Packages Reusability

Slide 27

Slide 27 text

Inspect the resulting resource tree before deployment (against mocks) Deploy a part of your graph to a test environment, inspect the result Test the result of a complete deployment Unit Integration End-to-end Testability

Slide 28

Slide 28 text

I think we solved these problems before?

Slide 29

Slide 29 text

IDEs Code completion Unit tests Types Functions Package managers Versioning Classes Code reviews Contracts SDKs Workflows Refactoring

Slide 30

Slide 30 text

Pulumi Cloud infrastructure using real languages Developers and operators working together 21

Slide 31

Slide 31 text

General-Purpose Programming Languages Node.js Go Python .NET

Slide 32

Slide 32 text

• AWS • Azure • GCP • Digital Ocean • Cloudflare … and more Providers • Docker • Kubernetes • OpenStack • PostgreSQL • New Relic

Slide 33

Slide 33 text

var resourceGroup = new ResourceGroup("rg"); var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", }); C# example

Slide 34

Slide 34 text

Still, Desired State! var resourceGroup = new ResourceGroup("rg"); var storageAccount = new Account("storage", new AccountArgs { ResourceGroupName = resourceGroup.Name, AccountReplicationType = "LRS", AccountTier = "Standard", });

Slide 35

Slide 35 text

CLI drives the deployment How Pulumi works CLI & engine Last deployed state Program.cs Language host AWS Azure GCP Kubernetes new Resource() CRUD

Slide 36

Slide 36 text

Static Website on AWS Demo 26

Slide 37

Slide 37 text

Serverless Functions on Azure Infrastructure Components 33

Slide 38

Slide 38 text

Resources to Provision an Azure Function Blob Plan Container Storage Account Function App

Slide 39

Slide 39 text

• Encapsulate low-level resources into a higher-level abstraction • Reuse, shared abstractions, versioning, package management • Grouping and structure Component Resources

Slide 40

Slide 40 text

// A resource group to contain our Azure Functions var resourceGroup = new ResourceGroup("functions-rg"); // A .NET Azure Function var dotnet = new ArchiveFunctionApp("http", new ArchiveFunctionAppArgs { ResourceGroup = resourceGroup, Archive = new FileArchive("./dotnet/bin/Debug/netcoreapp3.1/publish") }); .NET Azure Function

Slide 41

Slide 41 text

Let’s run pulumi up … Type Name Plan + pulumi:pulumi:Stack functions-dev create + ├─ azure:appservice:ArchiveFunctionApp http create + │ ├─ azure:storage:Account http create + │ ├─ azure:appservice:Plan http create + │ ├─ azure:storage:Container http create + │ ├─ azure:storage:ZipBlob http create + │ └─ azure:appservice:FunctionApp http create + └─ azure:core:ResourceGroup rg create

Slide 42

Slide 42 text

// A Node.js Azure Function var js = new ArchiveFunctionApp("js", new ArchiveFunctionAppArgs { ResourceGroup = resourceGroup, Archive = new FileArchive("./javascript"), Runtime = "node" }); Node.js Azure Function

Slide 43

Slide 43 text

var linuxResourceGroup = new ResourceGroup("linux-rg"); var linuxPlan = new Plan("linux-asp", new PlanArgs { ResourceGroupName = linuxResourceGroup.Name, Kind = "Linux", Reserved = true, Sku = new PlanSkuArgs { Tier = "Dynamic", Size = "Y1" } }); var python = new ArchiveFunctionApp(...); Python Azure Function on Linux

Slide 44

Slide 44 text

var linuxResourceGroup = new ResourceGroup("linux-rg"); var linuxPlan = new Plan("linux-asp", new PlanArgs { ... }); var python = new ArchiveFunctionApp("py", new ArchiveFunctionAppArgs { ResourceGroup = linuxResourceGroup, Archive = new FileArchive("./python"), Plan = linuxPlan, Runtime = "python" }); Python Azure Function on Linux

Slide 45

Slide 45 text

Cosmos App Creating custom abstractions 37

Slide 46

Slide 46 text

Cosmos App (1)

Slide 47

Slide 47 text

Cosmos App (2)

Slide 48

Slide 48 text

Cosmos App (3)

Slide 49

Slide 49 text

Cosmos App (4)

Slide 50

Slide 50 text

var app = new CosmosApp("functions", new CosmosAppArgs { ResourceGroup = resourceGroup, Locations = ["WestUS", "EastUS", "WestEurope"], Factory = global => region => { var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs { ResourceGroupName = resourceGroup.Name, Location = region.Location, Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"), AppSettings = { { "CosmosDBConnection", global.ConnectionString } } }); return new AzureEndpoint(func.AppId); } }); Global Applications

Slide 51

Slide 51 text

var app = new CosmosApp("functions", new CosmosAppArgs { ResourceGroup = resourceGroup, Locations = ["WestUS", "EastUS", "WestEurope"], Factory = global => region => { var func = new ArchiveFunctionApp("app", new ArchiveFunctionAppArgs { ResourceGroupName = resourceGroup.Name, Location = region.Location, Archive = new FileArchive("./app/bin/Debug/netcoreapp2.2/publish"), AppSettings = { { "CosmosDBConnection", global.ConnectionString } } }); return new AzureEndpoint(func.AppId); } }); Global Applications

Slide 52

Slide 52 text

How it works Behind the scenes 41

Slide 53

Slide 53 text

N providers, M runtimes

Slide 54

Slide 54 text

N providers, M runtimes

Slide 55

Slide 55 text

O(N*M) complexity

Slide 56

Slide 56 text

O(N+M) complexity Intermediate Representation

Slide 57

Slide 57 text

O(N+M) complexity

Slide 58

Slide 58 text

State Management and Backends 41

Slide 59

Slide 59 text

State is a persisted graph State Management CLI and Engine Last deployed state Program.cs Language host AWS Azure GCP Kubernetes new Resource() CRUD Pulumi Service S3 / Blob Storage Local disk

Slide 60

Slide 60 text

Open-source Free for any use Cross-platform Written in Go Open-source Free for any use Can be created by 1st or 3rd party Web API + portal “GitHub for infra” Free for individuals Paid plans for teams Useful but not required CLI Providers SaaS backend Pulumi components

Slide 61

Slide 61 text

Cloud credentials must be configured for CLI to run AWS profiles az login Tokens can be set via environment variables CLI Local logins Environment Cloud Authentication

Slide 62

Slide 62 text

Deployment process Allow developers to iterate quickly on dev environments. Gain visibility and change management with Continuous Delivery.

Slide 63

Slide 63 text

Track Infrastructure Changes from Commit to Deployment End to End Visibility

Slide 64

Slide 64 text

Dependency Visualization Understand Dependencies Between Resources

Slide 65

Slide 65 text

APIs and Webhooks Integrate Infrastructure Changes via Webhooks

Slide 66

Slide 66 text

Role-Based Access Controls Manage Access Based on Stacks and Teams

Slide 67

Slide 67 text

Kubernetes Deploy to multiple clouds 45

Slide 68

Slide 68 text

Kubernetes layers Managed Kubernetes cluster Infrastructure Resources (networking, storage, identity) Managed Service Managed Service Application Application Application

Slide 69

Slide 69 text

Organizations, Projects, and Stacks Org: acme-corp Project: vpc Stack: dev env: dev region: us-east-1 Stack: prod env: prod region: us-west-2 Project: k8s-cluster Stack: dev env: dev region: us-east-1 Stack: prod env: prod region: us-west-2 Project: svc-userprofile Stack: dev env: dev region: us-east-1 Stack: prod env: prod region: us-west-2 Project: svc-email Stack: dev env: dev region: us-east-1 Stack: prod env: prod region: us-west-2

Slide 70

Slide 70 text

Stack References Org: acme-corp vpc Stack: dev env: dev region: us-east-1 k8s-cluster Stack: dev env: dev region: us-east-1 svc-userprofile Stack: dev env: dev region: us-east-1 svc-email Stack: dev env: dev region: us-east-1

Slide 71

Slide 71 text

Advanced scenarios Demo 46

Slide 72

Slide 72 text

F# 48

Slide 73

Slide 73 text

let storageAccount = Account("sa", AccountArgs (ResourceGroupName = io resourceGroup.Name, AccountReplicationType = input "LRS", AccountTier = input "Standard")) let sku = PlanSkuArgs(Tier = input "Basic", Size = input "B1") let appServicePlan = Plan("asp", PlanArgs (ResourceGroupName = io resourceGroup.Name, Kind = input "App", Sku = input sku)) Resources in F#

Slide 74

Slide 74 text

let d = deployment "mydep" { deploymentSpec { replicas 3 selector { matchLabels appLabels } podTemplateSpec { metadata { labels appLabels } podSpec { container { name "nginx" image "nginx" containerPort 80 } } } } } Computation Expression DSL? https://github.com/pulumi/pulumi/issues/3644

Slide 75

Slide 75 text

Quality Control 50

Slide 76

Slide 76 text

Programming Languages for Infrastructure? You sure?

Slide 77

Slide 77 text

[Fact] public async Task InstancesMustNotHaveSshPortOpenToInternet() { var result = await Deployment.TestAsync(); var resource = result.Resources.OfType().Single(); var ingress = await resource.GetAsync(sg => sg.Ingress); foreach (var rule in ingress) foreach (var cidrBlock in rule.CidrBlocks) { Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0", "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)"); } } Unit Testing

Slide 78

Slide 78 text

[Fact] public async Task InstancesMustNotHaveSshPortOpenToInternet() { var result = await Deployment.TestAsync(); var resource = result.Resources.OfType().Single(); var ingress = await resource.GetAsync(sg => sg.Ingress); foreach (var rule in ingress) foreach (var cidrBlock in rule.CidrBlocks) { Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0", "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)"); } } Unit Testing

Slide 79

Slide 79 text

[Fact] public async Task InstancesMustNotHaveSshPortOpenToInternet() { var result = await Deployment.TestAsync(); var resource = result.Resources.OfType().Single(); var ingress = await resource.GetAsync(sg => sg.Ingress); foreach (var rule in ingress) foreach (var cidrBlock in rule.CidrBlocks) { Assert.False(rule.FromPort == 22 && cidrBlock == "0.0.0.0/0", "Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0)"); } } Unit Testing

Slide 80

Slide 80 text

Policy as Code (.NET coming soon) const policies = new PolicyPack("aws", { policies: [ { name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateTypedResource(aws.ec2.SecurityGroup, (sg, args, report) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { report("Ingress rules with public internet access are prohibited."); } }), }, ], });

Slide 81

Slide 81 text

Alternatives 53

Slide 82

Slide 82 text

TypeScript Python Java .NET Transpiled to and deployed by AWS CloudFormation AWS only Languages CloudFormation AWS Cloud Development Kit (CDK)

Slide 83

Slide 83 text

var queue = new Queue(this, "MyFirstQueue", new QueueProps { VisibilityTimeoutSec = 300 }); var topic = new Topic(this, "MyFirstTopic", new TopicProps { DisplayName = "My First Topic Yeah" }); AWS CDK example

Slide 84

Slide 84 text

// This is F# let myStorage = storageAccount { name "mystorage" // set account name sku Storage.Sku.PremiumLRS // use Premium LRS } let myWebApp = webApp { name "mysuperwebapp" // set web app name sku WebApp.Sku.S1 // use S1 size setting "storage_key" myStorage.Key // app setting depends_on myStorage // webapp is dependent on storage } Azure? ARM Generators (ex: Farmer) https://github.com/CompositionalIT/farmer

Slide 85

Slide 85 text

Conclusions 55

Slide 86

Slide 86 text

Define Infrastructure as Code Making Cloud Apps?

Slide 87

Slide 87 text

Fine-grained components Better Automation

Slide 88

Slide 88 text

Familiar language, experience, toolchain, packages Conditionals, loops, functions, classes, components, modules Reusable components that encapsulate complex logic and provide the right level of abstraction Developer-friendly Logic Abstractions Benefits of Real Code

Slide 89

Slide 89 text

м Infrastructure as Code Architecture as Code

Slide 90

Slide 90 text

• Get Started with Pulumi: https://pulumi.com/dotnet • Pulumi + .NET blog post: “Building Modern Cloud Applications using Pulumi and .NET Core” https://devblogs.microsoft.com/dotnet/building-modern-cloud- applications-using-pulumi-and-net-core/ • Pulumi + Cosmos blog post: “How to build globally distributed applications with Azure Cosmos DB and Pulumi” https://azure.microsoft.com/en-gb/blog/how-to-build-globally- distributed-applications-with-azure-cosmos-db-and-pulumi/ Useful Links

Slide 91

Slide 91 text

Meetups, Talks and Workshops with Pulumi Upcoming Events @PulumiCorp pulumi http://bit.ly/2OQ00Tt 18 Feb Copenhagen, DK 18 Feb Kaunas, LT 19 Feb Malmo, SE 20 Feb Amsterdam, NL 21 Feb Madrid, ES 25 Feb Prague, CZ 26 Feb Frankfurt, DE 30 Mar Amsterdam, NL

Slide 92

Slide 92 text

Amsterdam – 30 March The Pulumi team will provide hands-on instruction in a 4-hour workshop showing you how to setup your infrastructure for Kubernetes and other useful configurations. Includes lunch, workshop, labs and happy hour. Coming to a city near you Pulumi IaC Workshops @PulumiCorp pulumi http://bit.ly/2HgD9wd You don’t have to attend KubeCon to join this workshop

Slide 93

Slide 93 text

@MikhailShilkov https://mikhail.io