Serverless Architectures with AWS Lambda and MongoDB Atlas

Talk from Big Data Day LA 2017 on Serverless Architectures with AWS Lambda and MongoDB Atlas

Transcript

1. @SigNarvaez | @MongoDB | @BigDataDayLA On-Prem, IaaS, etc. Serverless
2. Serverless Architectures with AWS Lambda and MongoDB Atlas Sig Narváez Sr. Solutions Architect sig@mongodb.com @SigNarvaez
3. Serverless
4. Big Iron Commodity Hardware Virtualized Containers Functions Where will my code run? @SigNarvaez | @MongoDB | @BigDataDayLA
5. Thoughtworks Technology Radar
6. AWS re:Invent 2016: The State of Serverless Computing (SVR311)
7. AWS re:Invent 2016: The State of Serverless Computing (SVR311)
8. https://www.slideshare.net/AmazonWebServices/aws-reinvent-2016-how-thermo- fisher-is-reducing-mass-spectrometry-experiment-times-from-days-to-minutes-with- mongodb-aws-dat204 @SigNarvaez | @MongoDB | @BigDataDayLA
9. Microservices
10. Serverless Microservice @SigNarvaez | @MongoDB | @BigDataDayLA
11. Shape • Person • Insurance Policies • Shape changes per policy type • Addresses Operations via API • GET Customers with soon-to-expire policies, within a geo radius • GET Customers / by SSN, id, etc. • PATCH Update basic contact info (cell, email, …) Customer Single View - Insurance Industry (hypothetical) High-level architecture of a single view platform @SigNarvaez | @MongoDB | @BigDataDayLA
12. CQRS on Serverless Microservices COMMAND Update basic info QUERY Soon to Expire GEO By SSN, ID, … CUD API Key Read API Key Lambda Function(s) Majority Writes Secondary Reads Lambda Function(s) VPC Peering @SigNarvaez | @MongoDB | @BigDataDayLA
13. MongoDB Atlas & AWS Build it!
14. Automated Service On-demand DBaaS Secure Highly Available Disaster Recovery Elastic Scalability MongoDB Atlas
15. Setup MongoDB Atlas - M10+ Need assigned AWS region @SigNarvaez | @MongoDB | @BigDataDayLA
16. mgeneratejs • https://github.com/rueckstiess/mgeneratejs • npm install -g mgeneratejs • Create template – generate data • Upload to Atlas via mongoimport • Hint: get connection string from Atlas UI! • Browse with Compass Generate dataset (optional) Template (CustomerSingleView.json) mgeneratejs -n 100 CustomerSingleView.json | mongoimport --host ”YOUR ATLAS CLUSTER" --numInsertionWorkers 4 --db SingleView--collection Customers --authenticationDatabase admin -- ssl --username YOURUSER --password YOURPASSWORD @SigNarvaez | @MongoDB | @BigDataDayLA
17. IAM • Role with Lambda execute policies VPC • VPC Peering Connection • Security Group Required AWS Services Lambda • Set VPC, Security Group and IAM role • Upload deployment package (.zip) API Gateway • API definition (Resources & HTTP Methods) • Map Routes to Lambda functions • API Keys & Usage Plans @SigNarvaez | @MongoDB | @BigDataDayLA
18. VPC Peering
19. VPC Peering Atlas
20. VPC Peering Atlas AWS
21. VPC Peering Atlas AWS
22. VPC Peering Atlas AWS
23. VPC Peering Atlas AWS
24. VPC Peering Atlas AWS
25. Lambda
26. Connections & Containers http://docs.aws.amazon.com/lambda/latest/dg/lambda-introduction.html … AWS Lambda maintains the container for some time in anticipation of another Lambda function invocation. … the service freezes the container after a function completes, and thaws the container for reuse. If AWS Lambda chooses to reuse the container, this has the following implications: - Any declarations in your Lambda function code (outside the handler code, see Programming Model) remains initialized, providing additional optimization when the function is invoked again. For example, if your Lambda function establishes a database connection, instead of reestablishing the connection, the original connection is used in subsequent invocations. You can add logic in your code to check if a connection already exists before creating one. @SigNarvaez | @MongoDB | @BigDataDayLA
27. Python lambda function skeleton @SigNarvaez | @MongoDB | @BigDataDayLA
28. Node.js lambda function skeleton @SigNarvaez | @MongoDB | @BigDataDayLA
29. Local Emulators : Test on EC2 instance against Atlas python-lambda-local at https://pypi.python.org/pypi/python-lambda-local lambda-local (node.js) at https://www.npmjs.com/package/lambda-local
30. Code packaging http://docs.aws.amazon.com/lambd a/latest/dg/deployment-package- v2.html @SigNarvaez | @MongoDB | @BigDataDayLA
31. Lambda functions @SigNarvaez | @MongoDB | @BigDataDayLA
32. Upload & configure function The handler function The role with lambda permissions The VPC (peered with Atlas) The security group that allows traffic At least 2 subnets
33. API Gateway
34. Read API – GET /api/v1/customers
35. CUD API - PATCH /api/v1/customers
36. Deploying the API
37. Access and throttling via API Keys
38. Test!
39. Test with Postman @SigNarvaez | @MongoDB | @BigDataDayLA
40. Load test too!
41. AWS CloudWatch @SigNarvaez | @MongoDB | @BigDataDayLA
42. MongoDB Atlas Monitoring and Alerts @SigNarvaez | @MongoDB | @BigDataDayLA
43. MongoDB Compass @SigNarvaez | @MongoDB | @BigDataDayLA
44. Serverless Web too!!
45. RetroPie – Old-school games, killer tiny hardware @SigNarvaez | @MongoDB | @BigDataDayLA
46. Retro Web App based on aws-serverless-express https://github.com/snarvaez/bddla17/t ree/master/singleViewWebApp https://github.com/awslabs/aws- serverless-express @SigNarvaez | @MongoDB | @BigDataDayLA
47. Done! … But what about?
48. Scaling? Scaling Lambda No user intervention required - Default safety throttle of 100 concurrent executions per account per region. Functions invoked synchronously throw 429 error code. Functions invoked asynchronously can absorb reasonable bursts for approx. 15-30 minutes. If exhausted, consider using Simple Queue Service (SQS) or Simple Notification Service (SNS) as the Dead Letter Queue (DLQ). Read more at https://aws.amazon.com/lambda/faqs/ Scaling MongoDB Atlas On-Demand Zero downtime Upscale/Downscale: • Instance size • Storage size • IOPS • Replication factor. @SigNarvaez | @MongoDB | @BigDataDayLA
49. Serverless Architectures with AWS Lambda and MongoDB Atlas Sig Narváez Sr. Solutions Architect sig@mongodb.com @SigNarvaez Q & A https://github.com/snarvaez/bddla17 https://resources.mongodb.com/serverles s-architectures @SigNarvaez | @MongoDB | @BigDataDayLA