Do more with less code in a serverless world

There are many software engineering practices that can and should be applied to Lambda functions: Single Responsibility Principle (from SOLID), You Ain't Gonna Need It (YAGNI), Keep It Simply Stupid (KISS). In this presentation, I'll go through the different ways to apply those principles in the AWS serverless world and even to avoid the usage of Lambda functions sometimes.

Transcript

1. Less is more Do more with less code in a serverless world Jerome Van Der Linden Geneva Serverless Meetup - 26/05/2020
2. About me • In a previous life, ”Mr Cut Cut” (M. Coupe coupe) • Developer & software craftsman • And now, Solutions Architect @ AWS 2 linkedin.com/in/jeromevdl/ Jerome Van Der Linden
3. Few principles Clean Code
4. SOLID 4 S Single Responsibility Principle O Open / closed Principle L Liskov substitution Principle I Interface segregation Principle D Dependency Inversion Principle
5. Single Responsibility Principle 5 “A class or module should have one, and only one, reason to be changed” - Robert C. Martin, aka Uncle Bob
6. YAGNI (You Ain’t Gonna Need It) 6 Just because you can, doesn’t mean you should…
7. KISS 7
8. 8 And many more… Clean Code DRY Don’t Repeat Yourself Law of Demeter Broken window theory Boy scout rule Not invented here
9. How to apply those principles? In the serverless world
10. Serverless World? 10 Serverless == FAAS Function As A Service
11. Serverless World? 11 Serverless ⊃ FAAS Function As A Service
12. AWS Serverless world 12 AWS Lambda AWS Fargate Amazon API Gateway Amazon SNS Amazon SQS COMPUTE DATA STORES INTEGRATION Amazon Aurora Serverless Amazon S3 Amazon DynamoDB Amazon EventBridge FAAS AWS Step Functions AWS AppSync
13. Single Responsibility Principle ü Do’s ✘ Don’ts * - Input validation - Business logic /!\ - Transform data - Return result - Event/Input Filtering - Transport data - Orchestration & long transactions - Retry/Failure handling * Most of the time
14. Event Filtering 15 SNS Topic Publisher if event_type == 'order_created’: lambda_client.invoke('OrderCreation', ...) elif event_type == 'order_placed’: lambda_client.invoke('OrderPlacement', ...) elif event_type == 'order_cancelled’: lambda_client.invoke('OrderCancelation', ...) OrderCreation OrderPlacement OrderCancellation { "event_type": "order_placed", "order": { "id": "232134", "amount": "2341,45", "stock_ref": "AMZN” } } the wrong way function code Input event OrderFiltering
15. Event Filtering with SNS 16 OrderCreation OrderPlacement OrderCancellation Publisher OrderCreationEvent: Type: AWS::SNS::Subscription Properties: TopicArn: 'arn:aws:sns:eu-central-1:123456789:OrdersTopic’ Protocol: lambda Endpoint: 'arn:aws:lambda:eu-central-1:123456789:function:OrderCreation’ FilterPolicy: event_type: - order_created SNS Topic https://docs.aws.amazon.com/sns/latest/dg/sns-subscription-filter-policies.html Cloudformation template { "event_type": [ "order_created" ] } Filter policy
16. Event Filtering with SNS 17 OrderCreation OrderPlacement OrderCancellation Publisher OrderPlacementEvent: Type: AWS::SNS::Subscription Properties: TopicArn: 'arn:aws:sns:eu-central-1:123456789:OrdersTopic’ Protocol: lambda Endpoint: 'arn:aws:lambda:eu-central-1:123456789:function:OrderPlacement’ FilterPolicy: event_type: - order_placed SNS Topic Cloudformation template
17. Event Filtering with SNS 18 OrderCreation OrderPlacement OrderCancellation Publisher OrderCancellationEvent: Type: AWS::SNS::Subscription Properties: TopicArn: 'arn:aws:sns:eu-central-1:123456789:OrdersTopic’ Protocol: lambda Endpoint: 'arn:aws:lambda:eu-central-1:123456789:function:OrderCancellation’ FilterPolicy: event_type: - order_cancelled SNS Topic Cloudformation template
18. Event Filtering with EventBridge 19 https://aws.amazon.com/blogs/compute/reducing-custom-code-by-using-advanced-rules-in-amazon-eventbridge/ { "Source": "custom.myATMapp", "EventBusName": "default", "DetailType": "transaction", "Time": "Wed Jan 29 2020 08:03:18 GMT-0500", "Detail":{ "action": "withdrawal", "location": "NY-NYC-001", "amount": 300, "result": "approved", "transactionId": "123456", "cardPresent": true, "partnerBank": "Example Bank", "remainingFunds": 722.34 } } { "source": [ "custom.myATMapp" ], "detail-type": [ "transaction" ], "detail": { "amount": [ { "numeric": [ ">", 300 ] } ] } } { "source": [ "custom.myATMapp" ], "detail-type": [ "transaction" ], "detail": { "location": [ { "prefix": "NY-NYC-" } ] } } { "source": [ "custom.myATMapp" ], "detail-type": [ "transaction" ], "detail": { "partnerBank": [ { "exists": true } ] } } { "source": [ "custom.myATMapp" ], "detail-type": [ "transaction" ], "detail": { "result": [ "approved" ], "partnerBank": [ { "exists": false } ], "location": [ { "anything-but": "NY-NYC-002" }] } }
19. Orchestration 20 the wrong way invoke invoke if (… ) invoke // else if (success) Notify with SNS n * invoke if (failure) Enqueue error notify with SNS need approval App…
20. Simple orchestration with Lambda destinations 21 invoke invoke if (… ) invoke // else if (success) Notify with SNS n * invoke if (failure) Enqueue error notify with SNS need approval App… Lambda Destinations
21. Simple orchestration with Lambda destinations 22 Amazon SNS Amazon EventBridge Amazon Cloudwatch Logs Amazon S3 Amazon SES AWS Config Amazon CloudFormation AWS CodeCommit A S Y N C "DestinationConfig": { "onSuccess": { "Destination": "arn:aws:lambda:..." }, "onFailure": { "Destination": "arn:aws:sqs:..." } } Cloudformation template Amazon SNS Amazon EventBridge Amazon SQS AWS Lambda if success: return {...} else: raise Exception(‘Failure', {...}) function code Lambda function A S Y N C
22. Advanced orchestration with Step Functions 23 invoke invoke if (… ) invoke // else if (success) Notify with SNS n * invoke if (failure) Enqueue error notify with SNS need approval App…
23. Advanced orchestration with Step Functions 24 invoke invoke if (… ) invoke // else if (success) Notify with SNS n * invoke if (failure) Enqueue error notify with SNS need approval App… { "StartAt": "SimpleInvocation", "States": { "SimpleInvocation": { "Type": "Task", "Resource": "arn:aws:lambda:eu-central- 1:123456789012:function:HelloFunction", "Next": "Choose1or2" },
24. Advanced orchestration with Step Functions 25 invoke invoke if (… ) invoke // else if (success) Notify with SNS n * invoke if (failure) Enqueue error notify with SNS need approval App… "Choose1or2": { "Type": "Choice", "Choices": [ { "Variable": "$.foo”, "NumericEquals": 1, "Next": "Lambda1" }, { "Variable": "$.foo", "NumericEquals": 2, "Next": "ParallelInvocation" } ], "Default": "Unmatched" },
25. Advanced orchestration with Step Functions 26 invoke invoke if (… ) invoke // else if (success) Notify with SNS n * invoke if (failure) Enqueue error notify with SNS need approval App… "Lambda1": { "Type": "Task", "Resource": "arn:aws:lambda:eu- central-1:123456789012:function:Lambda1", "Next": "SuccessOrFailure" },
26. Advanced orchestration with Step Functions 27 invoke invoke if (… ) invoke // else if (success) Notify with SNS n * invoke if (failure) Enqueue error notify with SNS need approval App… "SuccessOrFailure": { "Type": "Choice", "Choices": [ { "Variable": "$.status", "StringEquals": "SUCCESS", "Next": "SendNotification" }, { "Variable": "$.status", "StringEquals": "FAILURE", "Next": "QueueError" } ], "Default": "Unmatched" }
27. Advanced orchestration with Step Functions 28 invoke invoke if (… ) invoke // else if (success) Notify with SNS n * invoke if (failure) Enqueue error notify with SNS need approval App… "SendNotification": { "Type": "Succeed" }, "QueueError": { "Type": "Fail" },
28. Advanced orchestration with Step Functions 29 invoke invoke if (… ) invoke // else if (success) Notify with SNS n * invoke if (failure) Enqueue error notify with SNS need approval App… "ParallelInvocation": { "Type": "Parallel", "Branches": [ { "StartAt": "SendApprovalRequest", "States": { "SendApprovalRequest": { // ... } }, { "StartAt": "Loop", "States": { "Loop": { // ... } } }
29. Advanced orchestration with Step Functions 30 invoke invoke if (… ) invoke // else if (success) Notify with SNS n * invoke if (failure) Enqueue error notify with SNS need approval App… "SendApprovalRequest": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken", "Parameters": { "FunctionName": "sendMailForApprovalFunction", "Payload": { "step.$": "$$.State.Name", "model.$": "$.data", "token.$": "$$.Task.Token" } }, "ResultPath": "$.output", "Next": "Approved", "Catch": [ { "ErrorEquals": [ "rejected" ], "ResultPath": "$.reason", "Next": "Rejected" } ] } SendTaskSuccess SendTaskFailure
30. Advanced orchestration with Step Functions 31 invoke invoke if (… ) invoke // else if (success) Notify with SNS n * invoke if (failure) Enqueue error notify with SNS need approval App… "Loop": { "Type": "Map", "ItemsPath": "$.loopItems", "Iterator": { "StartAt": "LoopLambda", "States": { "LoopLambda": { "Type": "Task", "Resource": "arn:aws:lambda:us-east- 1:123456789012:function:LoopFunction", "End": true } } }, "End": true }
31. Advanced orchestration with Step Functions 32 invoke invoke if (… ) invoke // else if (success) Notify with SNS n * invoke if (failure) Enqueue error notify with SNS need approval App…
32. TL;DR Single Responsibility Principle 33 èKeep your code focused on the business This is not the responsibility of a to do orchestration
33. You ain’t gonna need it 34 Welcome in the “functionless” world !
34. API Gateway Service Proxy 35
35. Ex: insert data in DynamoDB 36 Resource: /comments HTTP Method: POST HTTP Request Body: { "pageId": "example-page-id", "userName": "ExampleUserName", "message": "Example comment to be added." } Table: Comments commentId: String userName: String message: String pageId: String PK: commentId API Gateway DynamoDB Lambda var AWS = require('aws-sdk'); var ddb = new AWS.DynamoDB({apiVersion: '2012-08-10'}); exports.handler = async function(event, context) { var params = { TableName: 'Comments’, Item: { 'commentId' : { S: 'randomid’ }, 'pageId' : { S: event.pageId }, 'userName' : { S: event.userName }, 'message' : { S: event.message } } }; ddb.putItem(params, function(err, data) { if (err) { console.log("Error", err); } else { console.log("Success", data); } }); }
36. Ex: insert data in DynamoDB 37 Resource: /comments HTTP Method: POST HTTP Request Body: { "pageId": "example-page-id", "userName": "ExampleUserName", "message": "Example comment to be added." } Table: Comments commentId: String userName: String message: String pageId: String PK: commentId API Gateway DynamoDB
37. Ex: insert data in DynamoDB 38 Resource: /comments HTTP Method: POST HTTP Request Body: { "pageId": "example-page-id", "userName": "ExampleUserName", "message": "Example comment to be added." } Table: Comments commentId: String userName: String message: String pageId: String PK: commentId API Gateway DynamoDB
38. Ex: insert data in DynamoDB 39 Resource: /comments HTTP Method: POST HTTP Request Body: { "pageId": "example-page-id", "userName": "ExampleUserName", "message": "Example comment to be added." } Table: Comments commentId: String userName: String message: String pageId: String PK: commentId API Gateway DynamoDB
39. Ex: insert data in DynamoDB 40 Resource: /comments HTTP Method: POST HTTP Request Body: { "pageId": "example-page-id", "userName": "ExampleUserName", "message": "Example comment to be added." } Table: Comments commentId: String userName: String message: String pageId: String PK: commentId API Gateway DynamoDB
40. Ex: insert data in DynamoDB 41 Resource: /comments HTTP Method: POST HTTP Request Body: { "pageId": "example-page-id", "userName": "ExampleUserName", "message": "Example comment to be added." } Table: Comments commentId: String userName: String message: String pageId: String PK: commentId API Gateway DynamoDB
41. Use with caution 42 • Not a generic design pattern, not always applicable • Apply when the lambda is a passthrough (no business code, just mapping) • Mapping can sometimes be complex (Velocity Template Language)
42. Step Function integrations 43 "QueueError": { "Type": "Task", "Resource": "arn:aws:states:::sqs:sendMessage", "Parameters": { "QueueUrl": "https://sqs.eu-central-1.amazonaws.com/123456789012/myQueue", "MessageBody.$": "$.input.message" }, "End": true }, "SendNotification": { "Type": "Task", "Resource": "arn:aws:states:::sns:publish", "Parameters": { "TopicARN": "arn:aws:sns:eu-central- 1:123456789012:myTopic", "Subject": "Lambda1 has successfully finish its job", "Message.$" : "$.input.message" }, "End": true }, SQS SNS
43. Step Function integrations 44 AWS Batch Amazon DynamoDB Amazon ECS / Fargate Amazon EMR AWS Glue Amazon SageMaker Amazon SNS Amazon SQS AWS Step Functions AWS Lambda AWS CodeBuild
44. TL;DR You Ain’t Gonna Need It 45 è is not always needed You can save costs and optimize performance when functions are just passthrough/mapping
45. Keep it simple stupid 46 Lighten your functions
46. Keep your functions simple & stupid nano functions API GW /res1 /res2 /res3 = 1 handler (1 function) = 1 file Not that stupid so they need sisters to do the job !
47. Keep your functions simple & stupid nano functions API GW /res1 /res2 /res3 = 1 handler (1 function) = 1 file function-lith API GW /{proxy} /res1 /res2 /res3 That’s really not simple & stupid! = 1 handler (1 function) = 1 file
48. Keep your functions simple & stupid nano functions API GW /res1 /res2 /res3 function-lith API GW /{proxy} /res1 /res2 /res3 = 1 handler (1 function) = 1 file fat functions API GW /res1 /res2 /res3 = 3 handlers (3 functions) = 1 file = 1 handler (1 function) = 1 file
49. Keep your functions simple & stupid 50 nano functions API GW /res1 /res2 /res3 function-lith API GW /{proxy} /res1 /res2 /res3 fat functions = 1 handler (1 function) = 1 file API GW = 3 handler (3 functions) = 1 file micro functions API GW /res1 /res2 /res3 /res1 /res2 /res3 = 1 handler (1 function) = 1 file = 1 handler (1 function) = 1 file
50. Keep your functions simple & stupid https://github.com/cdk-patterns/serverless/tree/master/the-lambda-trilogy Potential duplicated code ++ Cognitive burden ++ Coupled functions Coupled deployments nano functions API GW /res1 /res2 /res3 function-lith API GW /{proxy} /res1 /res2 /res3 fat functions = 1 handler (1 function) = 1 file API GW = 3 handler (3 functions) = 1 file micro functions /res1 /res2 /res3 = 1 handler (1 function) = 1 file API GW /res1 /res2 /res3 = 1 handler (1 function) = 1 file Stupidity Complexity FAAS? Longer cold starts ++ Risk of blast radius Framework dependent Longer cold starts Risk of blast radius Potential duplicated code Cognitive burden
51. TL;DR Keep it simple stupid 52 Lighter functions = Easier to test and maintain Faster to start: less cold start More scalable: higher throughput More secure: less permission needed
52. Conclusion No function is easier to manage than ‘no function’ – me è If you can do better without a function, just do it è Else, apply software craftsmanship principles (clean code, tests, code reviews…) “No server is easier to manage than ‘no server’” – Werner Vogels
53. Thank You @jeromevdl https://tinyurl.com/ydeowzut 2-minutes survey:
54. Appendix
55. AWSTemplateFormatVersion: 2010-09-09 Resources: API: Type: AWS::ApiGateway::RestApi Properties: Name: !Sub ${AWS::StackName}-api- ${AWS::AccountId} APIDeployment: Type: AWS::ApiGateway::Deployment Properties: RestApiId: !Ref API StageName: prod DependsOn: - ListBucketAPI APIBucketResource: Type: AWS::ApiGateway::Resource Properties: ParentId: !GetAtt API.RootResourceId PathPart: "{bucket}" RestApiId: !Ref API ListBucketAPI: Type: AWS::ApiGateway::Method Properties: HttpMethod: GET ResourceId: !Ref APIBucketResource RestApiId: !Ref API AuthorizationType: NONE Integration: Credentials: 'arn:aws:iam::1234567890:role/role-demo-api-gw-s3-integration’ IntegrationHttpMethod: GET IntegrationResponses: - StatusCode: "200" PassthroughBehavior: WHEN_NO_MATCH RequestParameters: integration.request.header.Content-Type: method.request.header.Content-Type integration.request.header.x-amz-acl: "'authenticated-read’” integration.request.path.bucket: method.request.path.bucket Type: AWS Uri: !Sub arn:aws:apigateway:${AWS::Region}:s3:path/{bucket} MethodResponses: - StatusCode: "200" RequestParameters: method.request.path.bucket: true method.request.header.Content-Type: false API Gateway & S3 sample https://docs.aws.amazon.com/apigateway/latest/developerguide/integrating-api-with-aws-services-s3.html
56. 57 Advanced orchestration with Step Functions { "StartAt": "SimpleInvocation", "States": { "SimpleInvocation": { "Type": "Task", "Resource": "arn:aws:lambda:eu-central-1:123456789012:function:HelloFunction", "Next": "Choose1or2" }, "Choose1or2": { "Type": "Choice", "Choices": [ { "Variable": "$.foo", "NumericEquals": 1, "Next": "Lambda1" }, { "Variable": "$.foo", "NumericEquals": 2, "Next": "ParallelInvocation" } ], "Default": "Unmatched" }, "Lambda1": { "Type": "Task", "Resource": "arn:aws:lambda:eu-central-1:123456789012:function:Lambda1", "Next": "SuccessOrFailure" }, "SuccessOrFailure": { "Type": "Choice", "Choices": [ { "Variable": "$.status", "StringEquals": "SUCCESS", "Next": "SendNotification" }, { "Variable": "$.status", "StringEquals": "FAILURE", "Next": "QueueError" } ], "Default": "Unmatched" }, "SendNotification": { "Type": "Succeed" }, "QueueError": { "Type": "Fail" }, "ParallelInvocation": { "Type": "Parallel", "Branches": [ { "StartAt": "SendApprovalRequest", "States": { "SendApprovalRequest": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken", "Parameters": { "FunctionName": "sendMailForApprovalFunction", "Payload": { "step.$": "$$.State.Name", "model.$": "$.data", "token.$": "$$.Task.Token" } }, "ResultPath": "$.output", "Next": "Approved", "Catch": [ { "ErrorEquals": [ "rejected" ], "ResultPath": "$.reason", "Next": "Rejected" } ] }, "Approved": { "Type": "Task", "Resource": "arn:aws:lambda:eu-central-1:123456789012:function:Lambda1", "End": true }, "Rejected": { "Type": "Fail" } } }, { "StartAt": "Loop", "States": { "Loop": { "Type": "Map", "ItemsPath": "$.loopItems", "Iterator": { "StartAt": "LoopLambda", "States": { "LoopLambda": { "Type": "Task", "Resource": "arn:aws:lambda:us-east-1:123456789012:function:LoopFunction", "End": true } } }, "End": true } } } ], "End": true }, "Unmatched": { "Type": "Fail", "Error": "DefaultStateError", "Cause": "No Matches!" } } }