Monitoring, Hold the Infrastructure - Getting the most out of AWS Lambda

Just as we got a hang of monitoring our server-based applications, they take away the server. How do you monitor something that doesn’t exist? What metrics matter most in a serverless world? In this session, we will look at how applications are different in a AWS Lambda-based world and how to monitor them. Join us as we work our way through the stack and demonstrate how to capture the health and performance of your services.

The focus of this session is not tool specific. Attendees will learn production tested lessons and leave with frameworks they can implement with their serverless workloads regardless of the platforms and tools they use.

Transcript

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Matt Williams, Evangelist @ Datadog matt.williams@datadoghq.com, @technovangelist Monitoring, Hold the Infrastructure Getting the most out of AWS Lambda
2. The Plan Datadog overview The what and the why of AWS Lambda Adding AWS Lambda to your infrastructure Monitoring 101 (the tl;dr version) Adding AWS Lambda to your monitoring
3. The Plan Datadog overview The what and the why of AWS Lambda Adding AWS Lambda to your infrastructure Monitoring 101 (the tl;dr version) Adding AWS Lambda to your monitoring
4. Datadog Overview SaaS-based Infrastructure Monitoring Focus on Modern Infrastructure (cloud, containers, micro-services) Processing about a trillion metrics per day Intelligent alerting
5. Monitor Everything
6. The Plan Datadog overview The what and the why of AWS Lambda Adding AWS Lambda to your infrastructure Monitoring 101 (the tl;dr version) Adding AWS Lambda to your monitoring
7. What is AWS Lambda The next stage in a long progression... physical boxes virtual machines on physical boxes containers lambda AWS instances
8. Physical Boxes • Pre-defined resources & capacity • Variable consumption but paying for 100% • Scalability relies on getting new hardware • Other costs (power, cooling, maintenance, etc) physical boxes
9. VM on Physical Boxes • Scalability: VM level; Abstraction: HW level • Similar physical resources, capacity, scalability • Flexible VM resources & capacity • Paying for 100% of hardware • Variable consumption on each VM plus overhead physical box vm vm vm vm vm vm
10. AWS Instances • Scalability: VM level, abstraction: HW level, no physical restrictions • Paying for 100% of instance • Variable consumption • Much easier to scale each VM virtual machines on physical boxes AWS instances
11. Containers • Much more flexible, variable consumption • Paying for 100% of instance • No VM overhead • Scalability at the app level, abstraction at OS level virtual machines on physical boxes AWS/VM/ Physical ctr ctr ctr ctr ctr ctr ctr ctr ctr ctr ctr ctr ctr ctr ctr ctr ctr
12. AWS Lambda • Much more flexible, variable consumption • Paying time and memory used by the function • Scalability at the function level, abstraction at runtime level • Very early days lambda lambda lambda lambda lambda lambda lambda lambda lambda lambda lambda lambda lambda lambda lambda lambda
13. Important AWS Lambda Concepts TRIGGER BASED CC BY-SA 2.0: https://commons.wikimedia.org/wiki/File:Roy_Rogers_Prints_at_the_Chinese_Theatre.jpg
14. Important AWS Lambda Concepts STATELESS STATELESS
15. Important Lambda Concepts SERVERLESS AS FAR AS YOU ARE CONCERNED CC BY 2.0: https://commons.wikimedia.org/wiki/File:Chapa_traffic_jam_in_Maputo_(9717216968).jpg
16. What is a AWS Lambda Function Each AWS Lambda Function is: • Executable code in a container on LXC on AMZN Linux • Container implementation hidden from user • No restrictions on what it can do • Built-in runtimes for Java, Node, Python • Other languages supported through exec/spawn • Priced based on execution time, quantity, and memory
17. What can trigger AWS Lambda? Files (S3) Database Updates (DynamoDB) Streams (Kinesis) Messages (SNS, SES) User Data & Authentication (Cognito) Infrastructure Changes (CloudFormation) Logs, Events, Metrics (CloudWatch) IoT (Echo, API Gateway)
18. Concurrency Each event will trigger a new Lambda function There is a safety limit per region of 100 (contact support for changes)
19. So What Are Some Good Use Cases Bootstrapping a startup (first 1M requests/month free, up to 3.2M free seconds / month) Cron File manipulation (transcode, compress, etc) API translation Stream processing (social media analysis) Scaling infrastructure Mobile/IoT Backends
20. https://goad.io/
21. Benchling CRISPR Searches Genome searching Few 100k searches per month 90 GB of genome data 1. A web server receives a request to do CRISPR search 2. The web server splits up the genome into smaller regions. 3. The web server invokes the Lambda function for each region. 4. Results are combined and returned to the user. Old infrastructure cost $1000’s per month; New scales better and costs $60 per month http://benchling.engineering/crispr-aws-lambda/
22. Best Practices from AWS • Write stateless style code; ensure no affinity between code and underlying compute infrastructure • Avoid declaring variables outside scope of handler • Set +rx permissions in uploaded ZIP • Minimize use of startup code not processing current event • Monitor your Lambdas to view and optimize req latencies • Delete old Lambda functions https://docs.aws.amazon.com/lambda/lates t/dg/best-practices.html
23. The Plan Datadog overview The what and the why of AWS Lambda Adding AWS Lambda to your infrastructure Monitoring 101 (the tl;dr version) Adding AWS Lambda to your monitoring
24. How to Create a AWS Lambda Function 1. Write the code (Node, Java, Python, anything via exec/spawn) 2. Setup on AWS (memory, timeout, role, handler, invocation) • Web interface • node-lambda, apex, serverless, etc 3. Debug • logs
25. AWS Lambda Demo
26. Other AWS Lambda Tools • Apex (http://apex.run) • Serverless framework (http://serverless.com) • AWS Chalice (https://github.com/awslabs/chalice)
27. The Plan Datadog overview The what and the why of AWS Lambda Adding AWS Lambda to your infrastructure Monitoring 101 (the tl;dr version) Adding AWS Lambda to your monitoring
30. Collecting data is cheap
31. Collecting data is cheap; Not having it when you need it can be expensive
39. The Plan Datadog overview The what and the why of AWS Lambda Adding AWS Lambda to your infrastructure Monitoring 101 (the tl;dr version) Adding AWS Lambda to your monitoring
40. What are the AWS Lambda Metrics aws.lambda.duration, min, max, sum aws.lambda.errors aws.lambda.invocations aws.lambda.throttles
41. Three Options for Monitoring AWS Lambda 1. Add a custom metric to CloudWatch 2. Add metric directly to monitoring application 3. Add a line to the CloudWatch logs
42. The Downsides of the First Two • CloudWatch Metrics • CloudWatch metrics are updated every minute • Send more than one per minute and you get a histogram • Directly to Monitoring App • Adds complexity and size to your lambda (should be kept small) • But no lag time
43. How Datadog Does It Add line to CloudWatch logs: MONITORING|unix_epoch_timestamp|value|type| my.metric.name|#tag1:value,tag2 metric type can be count or gauge In Python: print(monitoringstring) In NodeJS: console.log(monitoringstring)
44. Monitoring example
45. Monitor All the AWS Lambda Things Monitor the triggers Monitor the function Monitor the output
46. Examples • Stream processing pipeline • each message gets a new lambda invocation • monitor duration of each step • tag based on data in the message, other metadata • dashboards that look at the data in new ways
47. Examples • Image processing app • different size images are uploaded • resized to standard sizes • monitor performance per size, other metadata
50. Summary AWS Lambda can be an interesting part of your infrastructure Very low cost to start Serverless doesn’t mean monitorless Matt Williams - matt.williams@datadoghq.com @technovangelist https://github.com/awslabs?query=lambda Twitter: @technovangelist