Think Big, Start Small, Scale Fast: Develop a Marketplace app using AWS Lambda

Are you planning to develop a new Cloud app? Serverless might be right for you! In this talk, we'll show you what it takes to run an app using a serverless architecture based on AWS Lambda, from design and deployment, to testing, debugging, and most importantly – how to scale. We'll go beyond the basics to demonstrate real use cases, best practices, and explain why serverless might be a good fit for you.

Talk is available online: https://www.youtube.com/watch?v=XfpskCjfBBw

Transcript

1. Think Big, Start Small, Scale Fast Develop a Marketplace app using AWS Lambda SEBASTIAN HESSE | SOFTWARE ENGINEER | K15T SOFTWARE | @SEEEBIII
2. Start Setup Develop Scale Goal
3. Start Setup Develop Scale Goal
4. Start Setup Develop Scale Goal
5. Backbone Issue Sync for Jira Sync issue data across company borders
6. After Implemented native cloud to cloud sync. Before Sync initiated from on-prem to another on-prem or cloud.
7. TYPICAL CLOUD ARCHITECTURE SERVERLESS Lambda Function Lambda Function Lambda Function
8. Our Cloud Setup + Amazon Web Services AWS Lambda
9. module.exports = function(event, context, cb) { cb(null, "Hello World."); }
10. Long Running Tasks Perform bigger tasks, e.g. to initialize data. Our Use Cases For Serverless Scheduled Tasks Schedule functions for cleaning up data or other tasks. Webhooks Receive webhooks from Jira, Confluence, etc. and process them.
11. Start Setup Develop Scale Goal
12. Start Setup Develop Scale Goal DEVELOPMENT
13. Local Machine Execute and debug + test your code on your local machine. Cloud Upload your code to the cloud and test it there. Testing & Debugging
14. No pain, no gain Log as much as necessary and test it in the cloud.
15. Quick code updates Wrote our own library lambda-updater to quickly update the Lambda code. Write tests Writing unit and integration tests can already identify lots of problems. Real environment No service mocks, latencies don't explode and a better feeling about the execution. Why?!
16. Quick code updates Wrote our own library lambda-updater to quickly update the Lambda code. Write tests Writing unit and integration tests can already identify lots of problems. Real environment No service mocks, latencies don't explode and a better feeling about the execution. Why?!
17. Quick code updates Wrote our own library lambda-updater to quickly update the Lambda code. Write tests Writing unit and integration tests can already identify lots of problems. Real environment No service mocks, latencies don't explode and a better feeling about the execution. Why?!
18. Start Setup Develop Scale Goal DEPLOYMENT
19. CloudFormation Infrastructure as Code
20. "webhooks": [ { "event": "jira:issue_updated", "url": "/webhooks?appVersion={{version}}" } ] Different Versions atlassian-connect.json:
21. + </> DevOps
22. Start Setup Develop Scale Goal
23. Start Setup Develop Scale Goal FUNCTION SCOPE
24. Task #2 Check content. Function Scope Task #3 Do something with it. Task #1 Receive webhook.
25. Task #2 Check content. Function Scope Task #3 Do something with it. Task #1 Receive webhook.
26. Keep it simple, stupid. KELLY JOHNSON
27. Reusability Reuse your functions from multiple contexts. Performance Single purpose functions execute faster. Multiple Scopes
28. Lambda Response Times 0 500 1000 1500 2000 1 2 3 4 5
29. Lambda Response Times 0 500 1000 1500 2000 1 2 3 4 5
30. Programming Language Java for background functions, NodeJS for functions which are performance relevant. Function Scope One scope per function. For example, one webhook receiver function, one email sending function, etc. Consider
31. Can we use AWS Lambda as a REST API? SOME DEVELOPERS
32. Lambda Response Times 0 500 1000 1500 2000 1 2 3 4 5
33. Start Setup Develop Scale Goal COMMUNICATION
34. Function Payload Communicate Function 1 Function 2
35. Function Payload Communicate Function 1 Function 2
36. Solution #1 Function 1 Function 2 Function 2
37. Solution #2 KEY Function 1 Function 2 Function Payload S3
38. Start Setup Develop Scale Goal
39. Start Setup Develop Scale Goal WEBHOOKS
40. 1000 webhooks at the same time
41. Scaling for AWS Lambda is done automatically... right? MANY DEVELOPERS
42. Other services are not necessarily scaled. Lambda functions are scaled by AWS. YES NO
43. DynamoDB S3
44. DynamoDB S3
45. Buffer Webhooks Function 1 AWS Kinesis Webhook
46. Buffer Webhooks Function 1 Function 2 AWS Kinesis Webhook
47. Start Setup Develop Scale Goal LONG RUNNING TASKS
48. 5 minute execution time limit
49. 5 minutes are enough for the task my function is performing. A FEW DEVELOPERS
50. Collect tenant information from database. < 1 sec 350ms / 100 issues Retrieve all issues from a project in Jira. Find issues matching your configuration. < 1 sec Example: Retrieve 800 Issues For Sync Perform further steps and update issue data. 350ms / 1 issue 10 sec Store relevant information in database.
51. Solution #1 Call yourself before function reaches execution limit.
52. Solution #2 Use other services like an EC2 server to execute your tasks.
53. Solution #3 Use AWS Step Functions to define an execution order of your functions. AWS Step Functions
54. Example Step Function "Retrieve issues": { "Type": "Task", "Resource": "arn:aws:lambda:...", "Next": "All issues?", "Retry": [ { ... } ] } Retrieve issues All issues? Process issues Failed START END
55. Start Setup Develop Scale Goal SCHEDULED TASKS
56. Examples Webhooks Webhooks are not 100% reliable. Catch up missed webhooks every day. Weekly Updates Send a status update every week and let your customers know about some numbers. Clean Up Clean up data in your environment, e.g. in your database.
57. Trigger Function AWS CloudWatch Lambda Function
58. Monitoring Enable monitoring and notification about failures. Learn from it and adapt your architecture. Failures Functions can fail quite easily and you might never know, because you don't check the logs or any metrics.
59. Start Setup Develop Scale Goal
60. Trade Offs Potentially less operational costs, but more complex architecture. Please Remember Scalability Consider scalability for all of your services. Function Scope Focus on one scope per function. Think about the use case of a function.
61. Start Setup Develop Scale Goal
62. Start Setup Develop Scale Goal
63. Start now! SEBASTIAN HESSE | SOFTWARE ENGINEER | K15T SOFTWARE | @SEEEBIII