Tweet
Tweet

Slide 1

Slide 1 text

Infrastructure as Software DevOps Pro Europe | 25 March 2020 Mikhail Shilkov

Slide 2

Slide 2 text

My first DevOps team Developer Cloud Infrastructure User IT

Slide 3

Slide 3 text

• Software engineer at Pulumi: .NET SDK, Azure, Core platform • Cloud, Serverless • Functional programming, F# • Microsoft Azure MVP https://mikhail.io @MikhailShilkov About me

Slide 4

Slide 4 text

Agenda

Slide 5

Slide 5 text

Agenda

Slide 6

Slide 6 text

Agenda

Slide 7

Slide 7 text

Cloud Automation 3

Slide 8

Slide 8 text

Unified Management API Management API (e.g., Azure Resource Manager) Cloud Service Cloud Service Cloud Service Web Portal CLI SDK 3rd Party

Slide 9

Slide 9 text

REST calls Content-Type: application/x-www-form-urlencoded; charset=UTF-8 X-Amz-Date: 20130813T150211Z Host: ec2.amazonaws.com Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20130813/us-east- 1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz- date, Signature=ced6826de92d2bdeed8f846f0bf508e8559e98e4b0194b84example54174deb456c http://ec2.amazonaws.com/?Action=RunInstances ImageId=ami-2bb65342 &MaxCount=3 &MinCount=1 &Monitoring.Enabled=true &Placement.AvailabilityZone=us-east-1a &Version=2016-11-15

Slide 10

Slide 10 text

var launchRequest = new RunInstancesRequest() { ImageId = amiID, InstanceType = "t1.micro", MinCount = 1, MaxCount = 1, KeyName = keyPairName, SecurityGroupIds = groups }; var launchResponse = ec2Client.RunInstances(launchRequest); var instances = launchResponse.Reservation.Instances; SDKs to call those REST endpoints

Slide 11

Slide 11 text

aws ec2 run-instances \ --image-id ami-5ec1673e \ --key-name MyKey \ --security-groups EC2SecurityGroup \ --instance-type t2.micro \ --placement AvailabilityZone=us-west-2b Command-line Tools

Slide 12

Slide 12 text

Step-by-step provisioning process Depend on both the current and the target state How to handle failures? Imperative Complex Error-prone Scripting: Challenges

Slide 13

Slide 13 text

Desired State Configuration 5

Slide 14

Slide 14 text

Desired State Configuration Target Current Tool

Slide 15

Slide 15 text

Managing Resource Graphs Target Current Tool

Slide 16

Slide 16 text

Be able to describe the desired state in full, concisely, and correctly Determine the correct order of creation and destruction Preview the changes before running the deployment Expressiveness Dependencies Transparency Challenges

Slide 17

Slide 17 text

AWS CloudFormation Azure Resource Manager Google Cloud Deployment Manager Vendor Tools

Slide 18

Slide 18 text

Markup Languages 8 YET ANOTHER MARKUP LANGUAGE

Slide 19

Slide 19 text

Azure ARM Templates: JSON "resources": [ { "apiVersion": "2016-01-01", "type": "Microsoft.Storage/storageAccounts", "name": "mystorageaccount", "location": "westus", "sku": { "name": "Standard_LRS" }, "kind": "Storage", "properties": { } } ]

Slide 20

Slide 20 text

AWS CloudFormation: YAML Resources: S3BucketForURLs: Type: "AWS::S3::Bucket" DeletionPolicy: Delete Properties: BucketName: !If [ "CreateNewBucket", !Ref "AWS::NoValue", !Ref S3BucketName ] WebsiteConfiguration: IndexDocument: "index.html" LifecycleConfiguration: Rules: - Id: DisposeShortUrls ExpirationInDays: !Ref URLExpiration Prefix: "u" Status: Enabled

Slide 21

Slide 21 text

resource "azurerm_storage_account" "sa" { name = "mysa" location = azurerm_resource_group.rg.location resource_group_name = azurerm_resource_group.rg.name account_tier = "Standard" account_replication_type = "LRS" } Terraform: DSL (HashiCorp Configuration Language)

Slide 22

Slide 22 text

Lack of fast feedback about errors Which options need to be filled and which values are possible Thousands of lines of markup are hard to write, read, and navigate Tooling Discovery Size Markup Authoring Experience

Slide 23

Slide 23 text

Example: Code to infrastructure ratio KV Store Table “URLs” Function “Add URL” Function “Open URL” Lines of code Function Code ~20 CloudFormation ~240 ARM Templates ~160 Terraform ~220 Estimates based on public examples

Slide 24

Slide 24 text

{ "Resources": { "InstanceSecurityGroup": { "Type": "AWS::EC2::SecurityGroup", "Properties": { "SecurityGroupIngress": [ ... ] } }, "EC2Instance": { "Type": "AWS::EC2::Instance", "Properties": { "InstanceType": "t2.micro", "SecurityGroups": [ { "Ref": "InstanceSecurityGroup" } ], "ImageId": "ami-0080e4c5bc078760e" } } } } References Example for AWS CloudFormation

Slide 25

Slide 25 text

{ "condition": "[equals(parameters('production'), 'Yes')]", "type": "Microsoft.Compute/availabilitySets", "apiVersion": "2017-03-30", "name": "[variables('availabilitySetName')]", "location": "[resourceGroup().location]", "properties": { "platformFaultDomainCount": 2, "platformUpdateDomainCount": 3 } } Conditionals Example for Azure RM Templates

Slide 26

Slide 26 text

locals { files = ["index.php","main.php"] } resource "aws_s3_bucket_object" "files" { count = "${length(local.files)}" key = "${local.files[count.index]}" bucket = "${aws_s3_bucket.examplebucket.id}" source = "./src/${local.files[count.index]}" } Loops Example for Terraform

Slide 27

Slide 27 text

Different shape of infrastructure for different environments Bundling several related resources together for a higher-level component Sharing libraries of components within the company or in community Configuration Abstraction Packages Reusability

Slide 28

Slide 28 text

Inspect the resulting resource tree before deployment (against mocks) Deploy a part of your graph to a test environment, inspect the result Test the result of a complete deployment Unit Integration End-to-end Testability

Slide 29

Slide 29 text

I think we solved these problems before?

Slide 30

Slide 30 text

IDEs Code completion Unit tests Types Functions Package managers Versioning Classes Code reviews Contracts SDKs Workflows Refactoring

Slide 31

Slide 31 text

Pulumi Developers and operators working on infrastructure together 12

Slide 32

Slide 32 text

General-Purpose Programming Languages Node.js Go Python .NET

Slide 33

Slide 33 text

• AWS • Azure • GCP • Digital Ocean • Cloudflare … and more Providers • Docker • Kubernetes • OpenStack • PostgreSQL • New Relic

Slide 34

Slide 34 text

const resourceGroup = new azure.core.ResourceGroup("rg"); const storageAccount = new azure.storage.Account("storage", { resourceGroupName: resourceGroup.name, accountReplicationType: "LRS", accountTier: "Standard", }); TypeScript example

Slide 35

Slide 35 text

const resourceGroup = new azure.core.ResourceGroup("rg"); const storageAccount = new azure.storage.Account("storage", { resourceGroupName: resourceGroup.name, accountReplicationType: "LRS", accountTier: "Standard", }); Still, Desired State!

Slide 36

Slide 36 text

Command-Line Interface (CLI) drives the deployment How Pulumi works CLI & engine Last deployed state Program.cs Language host AWS Azure GCP Kubernetes new Resource() CRUD

Slide 37

Slide 37 text

Demo! An S3 website with Pulumi 16

Slide 38

Slide 38 text

Serverless Functions on Azure a.k.a. Azure Functions 21

Slide 39

Slide 39 text

Resources to Provision an Azure Function Blob Plan Container Storage Account Function App

Slide 40

Slide 40 text

• Encapsulate low-level resources into a higher-level abstraction • Reuse, shared abstractions, versioning, package management • Grouping and structure Component Resources

Slide 41

Slide 41 text

// A resource group to contain our Azure Functions const resourceGroup = new azure.core.ResourceGroup("rg"); // A Node.js Azure Function const nodeApp = new azure.appservice.ArchiveFunctionApp("http", { resourceGroup, archive: new pulumi.asset.FileArchive("./javascript"), }); export const nodeEndpoint = nodeApp.endpoint; Node.js Azure Function

Slide 42

Slide 42 text

Let’s run pulumi up … Type Name Plan + pulumi:pulumi:Stack functions-dev create + ├─ azure:appservice:ArchiveFunctionApp http create + │ ├─ azure:storage:Account http create + │ ├─ azure:appservice:Plan http create + │ ├─ azure:storage:Container http create + │ ├─ azure:storage:ZipBlob http create + │ └─ azure:appservice:FunctionApp http create + └─ azure:core:ResourceGroup rg create

Slide 43

Slide 43 text

// A .NET (C#) Azure Function const dotnetApp = new azure.appservice.ArchiveFunctionApp("dotnet", { resourceGroup, archive: new pulumi.asset.FileArchive("./dotnet/bin/Debug/"), appSettings: { runtime: "dotnet", }, }); .NET Azure Function

Slide 44

Slide 44 text

const linuxResourceGroup = new azure.core.ResourceGroup("linuxrg"); const linuxPlan = new azure.appservice.Plan("linux-asp", { resourceGroupName: linuxResourceGroup.name, kind: "Linux", sku: { tier: "Dynamic", size: "Y1" }, reserved: true, }); const pythonApp = new azure.appservice.ArchiveFunctionApp(...); Azure Function in Python on Linux

Slide 45

Slide 45 text

const linuxResourceGroup = new azure.core.ResourceGroup("linuxrg"); const linuxPlan = new azure.appservice.Plan("linux-asp", {...}); const pythonApp = new azure.appservice.ArchiveFunctionApp("http", { resourceGroup: linuxResourceGroup, plan: linuxPlan, archive: new pulumi.asset.FileArchive("./python"), appSettings: { runtime: "python", }, }); Azure Function in Python on Linux

Slide 46

Slide 46 text

const f = new azure.appservice.HttpEventSubscription("greeting", { resourceGroup, callback: async (context, req) => { return { status: 200, body: `Hello ${req.params.name}!`, }; }, }); Serverless Functions as Callbacks

Slide 47

Slide 47 text

const queue = new azure.storage.Queue("myqueue", { storageAccountName: storageAccount.name, }); queue.onEvent("MyMessageHandler", { callback: async (context, msg) => { doWork(msg); }, }); Serverless Functions as Callbacks

Slide 48

Slide 48 text

Containers Simple AWS Fargate deployment 25

Slide 49

Slide 49 text

Crosswalk for AWS

Slide 50

Slide 50 text

Load- balanced container instances Load Balancer Container Container

Slide 51

Slide 51 text

const listener = new awsx.elasticloadbalancingv2.NetworkListener("lb",{ port: 80 }); const service = new awsx.ecs.FargateService("nginx", { desiredCount: 2, taskDefinitionArgs: { containers: { nginx: { image: "jonashackt/spring-boot-vuejs", memory: 512, portMappings: [listener], }}}}); Nginx on Fargate with a Load Balancer

Slide 52

Slide 52 text

Let’s run pulumi up … Type Name Status + pulumi:pulumi:Stack pulumi-typescript-aws-fargate-dev created + ├─ awsx:x:ec2:SecurityGroup nginx created + │ └─ aws:ec2:SecurityGroup nginx created + ├─ awsx:x:ec2:Vpc default-vpc-accd9bc5 created + │ ├─ awsx:x:ec2:Subnet default-vpc-accd9bc5-public-1 created + │ └─ awsx:x:ec2:Subnet default-vpc-accd9bc5-public-0 created + ├─ awsx:x:ecs:FargateService spring-boot-vuejs created + │ └─ aws:ecs:Service spring-boot-vuejs created + ├─ awsx:x:ecs:FargateTaskDefinition spring-boot-vuejs created + │ ├─ aws:cloudwatch:LogGroup spring-boot-vuejs created + │ ├─ aws:iam:Role spring-boot-vuejs-execution created + │ ├─ aws:iam:Role spring-boot-vuejs-task created + │ ├─ aws:iam:RolePolicyAttachment spring-boot-vuejs-execution-9a42f520 created + │ ├─ aws:iam:RolePolicyAttachment spring-boot-vuejs-task-32be53a2 created + │ ├─ aws:iam:RolePolicyAttachment spring-boot-vuejs-task-fd1a00e5 created + │ └─ aws:ecs:TaskDefinition spring-boot-vuejs created + ├─ awsx:x:ecs:Cluster default-cluster created + │ ├─ awsx:x:ec2:SecurityGroup default-cluster created + │ │ ├─ awsx:x:ec2:IngressSecurityGroupRule default-cluster-ssh created + │ │ │ └─ aws:ec2:SecurityGroupRule default-cluster-ssh created + │ │ ├─ awsx:x:ec2:IngressSecurityGroupRule default-cluster-containers created + │ │ │ └─ aws:ec2:SecurityGroupRule default-cluster-containers created + │ │ ├─ awsx:x:ec2:EgressSecurityGroupRule default-cluster-egress created + │ │ │ └─ aws:ec2:SecurityGroupRule default-cluster-egress created + │ │ └─ aws:ec2:SecurityGroup default-cluster created + │ └─ aws:ecs:Cluster default-cluster created + └─ aws:lb:ApplicationLoadBalancer nginx created + ├─ awsx:lb:ApplicationTargetGroup nginx created + │ └─ aws:lb:TargetGroup nginx created + ├─ awsx:lb:ApplicationListener nginx created + │ ├─ awsx:x:ec2:IngressSecurityGroupRule nginx-external-0-ingress created + │ │ └─ aws:ec2:SecurityGroupRule nginx-external-0-ingress created + │ ├─ awsx:x:ec2:EgressSecurityGroupRule nginx-external-0-egress created + │ │ └─ aws:ec2:SecurityGroupRule nginx-external-0-egress created + │ └─ aws:lb:Listener nginx created + └─ aws:lb:LoadBalancer nginx created

Slide 53

Slide 53 text

Cosmos App Creating custom abstractions 28

Slide 54

Slide 54 text

Cosmos App (1)

Slide 55

Slide 55 text

Cosmos App (2)

Slide 56

Slide 56 text

Cosmos App (3)

Slide 57

Slide 57 text

Cosmos App (4)

Slide 58

Slide 58 text

export const functions = new CosmosApp("urls", { resourceGroup, locations: ["WestEurope", "WestUS", "SouthEastAsia"], factory: ({location, cosmosdb}) => { const app = new azure.appservice.ArchiveFunctionApp("app", { location, archive: new pulumi.asset.FileArchive("./app"), appSettings: { COSMOSDB_ENDPOINT: cosmosdb.endpoint, }, }); return { id: app.functionApp.id }; }}); Global Applications

Slide 59

Slide 59 text

export const functions = new CosmosApp("urls", { resourceGroup, locations: ["WestEurope", "WestUS", "SouthEastAsia"], factory: ({location, cosmosdb}) => { const app = new azure.appservice.ArchiveFunctionApp("app", { location, archive: new pulumi.asset.FileArchive("./app"), appSettings: { COSMOSDB_ENDPOINT: cosmosdb.endpoint, }, }); return { id: app.functionApp.id }; }}); Global Applications

Slide 60

Slide 60 text

Kubernetes Deploy to multiple clouds 32

Slide 61

Slide 61 text

Kubernetes layers Managed Kubernetes cluster Infrastructure Resources (networking, storage, identity) Managed Service Managed Service Application Application Application

Slide 62

Slide 62 text

Crosswalk for Kubernetes

Slide 63

Slide 63 text

// Define the application Pod. const appPod = new kx.PodBuilder({ containers: [{ image: "app:1.0.0", env: { "APP_CONFIG_PATH": "/app/config" }, volumeMounts: [configs.mount("/app/config")], }], }); // Create a Kubernetes Deployment using the previous Pod definition. const deployment = new kx.Deployment("nginx", { spec: appPod.asDeploymentSpec({ replicas: 3 }), }); // Expose the Deployment with a load balancer using a Kubernetes Service. const service = deployment.createService({ type: kx.types.ServiceType.LoadBalancer, }); Kubernetesx Library

Slide 64

Slide 64 text

Quality Control 34

Slide 65

Slide 65 text

Programming Languages for Infrastructure? You sure?

Slide 66

Slide 66 text

const group = require("./index").group; // check: Instances must not have SSH open to the Internet. it("must not open port 22 (SSH) to the Internet", done => { pulumi.all([ group.urn, group.ingress ]).apply(([ urn, ingress ]) => { if (ingress.find(rule => rule.fromPort == 22 && rule.cidrBlocks.find(block => block === "0.0.0.0/0"))) { done(new Error(`Illegal SSH port 22 open to the Internet on group ${urn}`)); } else { done(); } }); }); Unit Testing

Slide 67

Slide 67 text

Policy as Code const policies = new PolicyPack("aws", { policies: [ { name: "prohibited-public-internet", description: "Ingress rules with public internet access are prohibited.", enforcementLevel: "mandatory", validateResource: validateTypedResource(aws.ec2.SecurityGroup, (sg, args, report) => { const publicInternetRules = (sg.ingress || []).find(ingressRule => (ingressRule.cidrBlocks || []).find(cidr => cidr === "0.0.0.0/0")); if (publicInternetRules) { report("Ingress rules with public internet access are prohibited."); } }), }, ], });

Slide 68

Slide 68 text

Conclusions 37

Slide 69

Slide 69 text

Define Infrastructure as Code Making Cloud Apps?

Slide 70

Slide 70 text

Familiar language, experience, toolchain, packages Conditionals, loops, functions, classes, components, modules Reusable components that encapsulate complex logic and provide the right level of abstraction Developer-friendly Logic Abstractions Benefits of Real Code

Slide 71

Slide 71 text

м Infrastructure as Code Architecture as Code

Slide 72

Slide 72 text

http://bit.ly/pulumilinks Useful Links

Slide 73

Slide 73 text

@MikhailShilkov https://mikhail.io