Serverless Infrastructure as Code

Serverless applications utilize multiple cloud services and consist of many moving parts, so they are hard to manage without employing Infrastructure as Code (IaC) approach.

Get a 360-degree overview of the IaC tools from bash scripts and CloudFormation to Terraform and Pulumi. Learn the pros and cons of different approaches and see how coding and infrastructure are blending together.

Transcript

1. Serverless Infrastructure as Code Serverless Computing London | November 6 | 2019
2. Serverless Functions
3. Serverless Functions
4. Serverless Functions
5. Agenda
6. Agenda
7. URL Shortener 4
8. Serverless URL Shortener (AWS) AWS DynamoDB Table “URLs” AWS Lambda “Add URL” AWS Lambda “Open URL”
9. AWS Resources Lambda “Add URL” Lambda “Open URL” DynamoDB “URLs” API Gateway S3 Bucket Static site
10. AWS Resources Lambda “Add URL” Lambda “Open URL” DynamoDB “URLs” API Gateway S3 Bucket Static site Stage Deployment REST endpoint IAM IAM Role IAM Policy Bucket Objects CloudFront Route53
11. Global URL Shortener (Azure)
12. How do we create such an application?
13. Web Portals:
14. Goals of Infrastructure as Code
15. Cloud Automation 10
16. Unified Management API
17. REST calls Content-Type: application/x-www-form-urlencoded; charset=UTF-8 X-Amz-Date: 20130813T150211Z Host: ec2.amazonaws.com Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20130813/us-east- 1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz- date, Signature=ced6826de92d2bdeed8f846f0bf508e8559e98e4b0194b84example54174deb456c http://ec2.amazonaws.com/?Action=RunInstances ImageId=ami-2bb65342 &MaxCount=3 &MinCount=1 &Monitoring.Enabled=true &Placement.AvailabilityZone=us-east-1a &Version=2016-11-15
18. var launchRequest = new RunInstancesRequest() { ImageId = amiID, InstanceType = "t1.micro", MinCount = 1, MaxCount = 1, KeyName = keyPairName, SecurityGroupIds = groups }; var launchResponse = ec2Client.RunInstances(launchRequest); var instances = launchResponse.Reservation.Instances; SDKs to call those REST endpoints
19. az storage account create \ --location westus \ --name samplesa \ --resource-group myrg \ --sku LRS Unified CLI
20. Scripting: Challenges
21. Desired State Configuration 14
22. Desired State Configuration
23. Managing Resource Graphs
24. Challenges
25. More Challenges
26. Desired State Provisioning
27. ARM Templates: JSON "resources": [ { "apiVersion": "2016-01-01", "type": "Microsoft.Storage/storageAccounts", "name": "mystorageaccount", "location": "westus", "sku": { "name": "Standard_LRS" }, "kind": "Storage", "properties": { } } ]
28. Converted to REST calls PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/ {resourceGroupName}/providers/Microsoft.Storage/storageAccounts/mystorageaccount?api- version=2016-01-01 { "location": "westus", "properties": { } "sku": { "name": "Standard_LRS" }, "kind": "Storage" }
29. Tracked Deployment
30. AWS CloudFormation: YAML Resources: S3BucketForURLs: Type: "AWS::S3::Bucket" DeletionPolicy: Delete Properties: BucketName: !If [ "CreateNewBucket", !Ref "AWS::NoValue", !Ref S3BucketName ] WebsiteConfiguration: IndexDocument: "index.html" LifecycleConfiguration: Rules: - Id: DisposeShortUrls ExpirationInDays: !Ref URLExpiration Prefix: "u" Status: Enabled
31. AWS CloudFormation Features
32. Challenges
33. Terraform 22
34. resource "azurerm_storage_account" "sa" { name = "mysa" location = azurerm_resource_group.rg.location resource_group_name = azurerm_resource_group.rg.name account_tier = "Standard" account_replication_type = "LRS" } Custom DSL: HCL (HashiCorp Configuration Language)
35. • • • • • Providers • • • • •
36. Client-driven Engine Azure Resource Manager Azure Go SDK Azure TF Provider TF CLI HCL
37. terraform plan # Creates a draft of changes, does nothing to the infra terraform apply # Rolls the infra from its current state to the desired one terraform destroy # Deletes all resources in the current workspace Workflow: Plan, Apply, Destroy
38. > terraform plan Terraform will perform the following actions: # azurerm_storage_account.sa must be replaced -/+ resource "azurerm_storage_account" "sa" { ... ~ location = "westus" -> "westus2" # forces replacement ... Preview Changes
39. More Terraform today
40. Need for higher abstraction and reusable components KV Store Table “URLs” Function “Add URL” Function “Open URL”
41. How?
42. Serverless Framework 28
43. service: serverless-simple-http-endpoint provider: name: aws runtime: nodejs8.10 functions: currentTime: handler: handler.endpoint events: - http: path: ping method: get YAML Definition Files
44. Focus on target scenarios
45. Pulumi 30
46. General-Purpose Programming Languages
47. • • • • • Providers • • • • •
48. const storageAccount = new azure.storage.Account("storage", { resourceGroupName: resourceGroup.name, accountReplicationType: "LRS", accountTier: "Standard", }); TypeScript example
49. const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); URL Shortener
50. const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); URL Shortener
51. const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); URL Shortener
52. const counterTable = new aws.dynamodb.Table("url-table", { attributes: [{ name: "id", type: "S" }], hashKey: "id", readCapacity: 5, }); const lambda = new aws.lambda.Function("mylambda", { code: new pulumi.asset.FileArchive("./app"), handler: "index.handler", environment: { variables: { "COUNTER_TABLE": counterTable.name, }}, }); const endpoint = new awsx.apigateway.API("short-url", { routes: [{ path: "/{route+}", method: "GET", eventHandler: lambda, }], }); Still, Desired State!
53. How Pulumi works CLI and Engine Last deployed state index.ts Language host AWS Azure GCP Kubernetes new Resource() Create, update, delete
54. Auto- completion
55. Compile Errors 55
56. for (const location of locations) { // Azure Function - one per region const fn = new appservice.FunctionApp(`GetUrl-${location}`, { resourceGroup, location, route: "{key}", archive: zip, }); } Conditionals, loops
57. class AksCluster extends pulumi.ComponentResource { constructor(name: string, args: AksClusterArgs, opts: pulumi.ComponentResourceOptions = {}) { super("examples:aks:AksCluster", name, args, opts); const password = new random.RandomPassword("password"/*, {...}*/); const sshPublicKey = new tls.PrivateKey("key"/*, {...}*/); const adApp = new azuread.Application("aks"/*, {...}*/); const adSp = new azuread.ServicePrincipal("aksSp"/*, {...}*/); const vnet = new azure.network.VirtualNetwork("vnet"/*, {...}*/); const subnet = new azure.network.Subnet("subnet"/*, {...}*/); const cluster = new azure.containerservice.KubernetesCluster("aksCluster"/*, {...}*/) } } Components
58. AWS Cloud Development Kit (CDK) 36
59. м TypeScript Python Java .NET Compiles to CloudFormation
60. const hello = new lambda.Function(this, 'HelloHandler', { runtime: lambda.Runtime.NODEJS_8_10, code: lambda.Code.asset('lambda'), handler: 'hello.handler' }); new apigw.LambdaRestApi(this, 'Endpoint', { handler: hello }); AWS CDK example
61. Want to talk CDK? Find this guy:
62. Programming Languages for Infrastructure? You sure?
63. Architecture as Code 36 38
64. Scheduled Tasks
65. const ecsScheduledTask = new ScheduledEc2Task(stack, 'ScheduledTask’, { schedule: events.Schedule.expression('rate(1 minute)’) scheduledEc2TaskImageOptions: { image: ecs.ContainerImage.fromRegistry('amazon/amazon-ecs-sample'), memoryLimitMiB: 256, environment: { name: 'TRIGGER', value: 'CloudWatch Events' }, }, }); Scheduled Tasks
66. Zip Incoming Reports
67. const tpsReports = new aws.s3.Bucket("tpsReports"); const tpsZips = new aws.s3.Bucket("tpsZips"); tpsReports.onObjectCreated("zipTpsReports", (e) => { for (const rec of e.Records || []) { const [ buck, key ] = [ rec.s3.bucket.name, rec.s3.object.key ]; const data = await s3.getObject({ Bucket: buck, Key: key })); zip.addFile(key, data.Body); await s3.putObject({ Bucket: tpsZips.bucket.get(), Key: `${key}.zip`, Body: zip.toBuffer(), }); } }); Functions = Callbacks
68. const tpsReports = new aws.s3.Bucket("tpsReports"); const tpsZips = new aws.s3.Bucket("tpsZips"); tpsReports.onObjectCreated("zipTpsReports", (e) => { for (const rec of e.Records || []) { const [ buck, key ] = [ rec.s3.bucket.name, rec.s3.object.key ]; const data = await s3.getObject({ Bucket: buck, Key: key })); zip.addFile(key, data.Body); await s3.putObject({ Bucket: tpsZips.bucket.get(), Key: `${key}.zip`, Body: zip.toBuffer(), }); } }); Functions = Callbacks
69. Global Apps
70. Global Apps
71. export const functions = new CosmosApp("urls", { resourceGroup, locations: ["WestEurope", "WestUS", "SouthEastAsia"], factory: ({location, cosmosdb}) => { const app = new azure.appservice.ArchiveFunctionApp("app", { location, archive: new pulumi.asset.FileArchive("./app"), appSettings: { COSMOSDB_ENDPOINT: cosmosdb.endpoint, }, }); return { id: app.functionApp.id }; }}); Global Apps
72. export const functions = new CosmosApp("urls", { resourceGroup, locations: ["WestEurope", "WestUS", "SouthEastAsia"], factory: ({location, cosmosdb}) => { const app = new azure.appservice.ArchiveFunctionApp("app", { location, archive: new pulumi.asset.FileArchive("./app"), appSettings: { COSMOSDB_ENDPOINT: cosmosdb.endpoint, }, }); return { id: app.functionApp.id }; }}); Global Apps
73. Conclusions 42
74. Define Infrastructure as Code Making Cloud Apps?
75. Infrastructure as Code Tools: Tradeoffs
76. Fine-grained components Better Automation