AWS Lambda @ MindTouch.

In this talk we talk, presented at the AWS San Diego User Group, about how MindTouch uses AWS Lambda, and describe the design goals and implementation of lambda-ops, the MindTouch deployment tool of AWS Lambda functions.

Transcript

1. AWS LAMBDA @ . Juan Manuel Torres César López-Natarén Jan. 2016
2. JUAN AND CÉSAR MindTouch Engineers
3. JUAN MANUEL TORRES ➤ Born and raised in Bogotá Colombia ➤ Software Engineer @ ➤ Love ➤ Programming ➤ Photography ➤ Lizards ➤ Traveling ➤ twitter.com/onema ➤ juant@mindtouch.com
4. CÉSAR LÓPEZ-NATARÉN ➤ Senior Software Engineer @ ➤ Love to work on: ➤ APIs ➤ Data engineering ➤ Making code fast ➤ Infrastructure automation ➤ twitter.com/nataren ➤ cesarn@mindtouch.com
5. Customer Success SaaS
6. ➤ Authoring and publishing platform for Product Help ➤ We use your product’s knowledge to: ➤ Accelerate User Adoption ➤ Build Customer Loyalty ➤ We help you: ➤ Understand your Customers ➤ Accelerate your Sales Cycle
7. SOME MINDTOUCH CUSTOMERS
8. MINDTOUCH’S SCALE ➤ Up to 2M req/day ➤ Up to 5M events/day
9. AWS LAMBDA Serverless computing
10. WHAT IS AWS LAMBDA? ➤ Compute service that hosts an execution environment for your code on high availability compute infrastructure. ➤ The service: ➤ Administers the compute resources (CPU, memory, network and OS maintenance). ➤ Guarantees the selected capacity. ➤ Scales your software automatically. ➤ Provides monitoring and logging services.
11. WHY DOES IT MATTER? ➤ Because all the tasks described before consume programmer’s time ➤ And instead of focusing on your problem domain you are dealing with those other things ➤ It is actually complicated to get all those pieces right
12. HOW DO I USE IT? ➤ You provide your Node.js, Java, or Python code. ➤ Configure: ➤ Event sources ➤ Memory ➤ Execution role ➤ Handler function
13. LET’S GET ONE PYTHON LAMBDA FUNCTION DEPLOYED photo by Kool Cats Photography
14. DEMO Lambda S3 to Tweet
15. LAMBDA FUNCTION LIFE CYCLE ➤ Develop ➤ Test ➤ Deploy ➤ Repeat
16. LAMBDA FUNCTION DEVELOPMENT photo by Lachlan Hardy
17. PROGRAMMING MODEL / ENTRY POINT Handler 1 def lambda_handler(event, context): 2 s3_info = event['Records'][0]['s3'] 3 ... 4 tweet_images.send_image( 5 s3_info['bucket']['name'], 6 s3_info['object']['key'], 7 cleanup=True 8 )
18. PROGRAMMING MODEL / LOGGING Logging 1 import logging 2 3 logger = logging.getLogger() 4 logger.setLevel(logging.INFO) 5 6 def lambda_handler(event, context): 7 logger.info("Invoked with {0} events".format( 8 len(event['Records'])))
19. LAMBDA FUNCTION TESTING
20. TESTING OPTIONS ➤ Local testing through unit testing ➤ Make sure you can swap out implementations of components that perform I/O ➤ Mock testing through the AWS Console in the Lambda service itself ➤ Create your own simulation environment for Lambda
21. photo by NASA's Marshall Space Flight Center LAMBDA FUNCTION DEPLOYMENT
22. DEPLOYMENT ➤ Creating the lambda function on the service ➤ Granting permissions to event sources and any other components that the functions needs to interact with ➤ Sending the packaged code to the service ➤ Configuring the application
23. AWS LAMBDA USAGE @ MINDTOUCH ➤ Mirror data streams across accounts and regions ➤ Archive events to S3 ➤ Generate SQS messages from S3 object creation ➤ Publish Auto Scaling Group’s notifications to DataDog
24. CHALLENGES FACED while using AWS Lambda
25. CHALLENGES WHILE USING AWS LAMBDA ➤ Consistent function configuration ➤ Automated function deployment on different environments (dev, test, production)
26. MINDTOUCH LAMBDA OPS MindTouch deployment tool for Lambda functions
27. MINDTOUCH LAMBDA OPS DESIGN OBJECTIVE To enable a consistent development/test/ deployment lifecycle for AWS Lambda functions.
28. TOOL REQUIREMENTS ➤ Developers + DevOps ➤ Configuration must be simple • Single file • Checked-in to version control • Many deployments ➤ Permission management ➤ Managing Event Source Mappings
29. IMPLEMENTATION DETAILS ➤ Single YAML Configuration file ➤ Automatic generation of config.json including names for all sources and resources ➤ Prepares lambda function for deployment ➤ Creates a CloudFormation stack
30. DEMO Kinesis to S3
31. KINESIS2S3XMLARCHIVER CONFIGURATION 1 app: 2 3 resources: 4 source: 5 type: kinesis 6 name: events-stream 7 event-source: 8 batch-size: 10000 9 starting-position: latest 10 11 s3bucket: 12 type: s3 13 name: events-archive 14 allow: 15 - s3:PutObject 16 - s3:GetObject
32. photo by Andreas Levers WARNINGS
33. LIMITS ➤ Execution time is limited to 300s (5 minutes) ➤ 1.5GB memory limits photo by Navalarp Teratanatorn
34. SECURITY ➤ As Jan 2016 there is no support for accessing resources on a VPC (Coming soon)
35. DEPLOYMENT ➤ No straight forward deployment other than the console or using a third party tool photo by NASA's Marshall Space Flight Center
36. CONCLUSIONS ➤ Huge developer productivity gains ➤ Reduces complexity in large infrastructure systems ➤ They do what they claim to do ➤ We will choose AWS Lambda for suitable projects in the future
37. MINDTOUCH IS HIRING! www.mindtouch.com/jobs Join the Nerds