End-to-End Observability for Your Serverless Apps

AWS re:Invent 2019 Recap Day Serverless Meetup, London, February 18th, 2020

Transcript

1. © 2020, Amazon Web Services, Inc. or its Affiliates. Danilo Poccia Principal Evangelist, Serverless @danilop End-to-End Observability for Your Serverless Apps © 2020, Amazon Web Services, Inc. or its Affiliates. Danilo Poccia Principal Evangelist, Serverless @danilop End-to-End Observability for Your Serverless Apps
2. © 2020, Amazon Web Services, Inc. or its Affiliates. Sample Order Processing App Users API Gateway Endpoint ReadOrder Lambda Function Orders Queue ProcessOrder Lambda Function Orders DynamoDB Table CreateOrder Lambda Function POST /order GET /order
3. © 2020, Amazon Web Services, Inc. or its Affiliates. $ sam validate $ sam build $ sam deploy Let’s deploy using AWS SAM… Demo
4. © 2020, Amazon Web Services, Inc. or its Affiliates. Don’t build a network of connected “black boxes” Observability is a developer responsibility
5. © 2020, Amazon Web Services, Inc. or its Affiliates. Levels of Observability Machine (HW, OS) Application Network
6. © 2020, Amazon Web Services, Inc. or its Affiliates. Machine (HW, OS) Application Network The Three Pillars of Observability Distributed Systems Observability by Cindy Sridharan
7. © 2020, Amazon Web Services, Inc. or its Affiliates. Machine (HW, OS) Application Network The Three Pillars of Observability Logs Metrics Tracing Distributed Systems Observability by Cindy Sridharan
8. © 2020, Amazon Web Services, Inc. or its Affiliates. Metric Filters & Correlations IDs Logs Tracing Metric Filter Correlation ID Metrics
9. © 2020, Amazon Web Services, Inc. or its Affiliates. Using Observability Logs Tracing Log aggregation & analytics Visualizations Alerting Metric Filter Correlation ID Metrics
10. © 2020, Amazon Web Services, Inc. or its Affiliates. Using Observability on AWS CloudWatch Logs AWS X-Ray Traces CloudWatch Insights CloudWatch Dashboard CloudWatch Alarms CloudWatch ServiceLens Metric Filter CloudWatch Metrics
11. © 2020, Amazon Web Services, Inc. or its Affiliates. Understand performance… Systems Performance by Brendan Gregg
12. © 2020, Amazon Web Services, Inc. or its Affiliates. Understand performance… and latency… Systems Performance by Brendan Gregg
13. © 2020, Amazon Web Services, Inc. or its Affiliates. Understand performance… and latency… and percentiles! P50 P90 P99 P100
14. © 2020, Amazon Web Services, Inc. or its Affiliates.
15. © 2020, Amazon Web Services, Inc. or its Affiliates. Let’s go back to our app…
16. © 2020, Amazon Web Services, Inc. or its Affiliates. Sample Order Processing App Users API Gateway Endpoint ReadOrder Lambda Function Orders Queue ProcessOrder Lambda Function Orders DynamoDB Table CreateOrder Lambda Function POST /order GET /order
17. © 2020, Amazon Web Services, Inc. or its Affiliates. HTTP POST /order { "order": { "accountId": "Danilo", "items": [ { "sku": "123", "price": 5 }, { "sku": "321", "price": 10 } ] } } Demo
18. © 2020, Amazon Web Services, Inc. or its Affiliates. HTTP GET /order/{id} { "order": { "accountId": "Danilo", "id": "4b0e8231-4eca-4356-b6d7-b626499b61e0", "items": [ { "sku": "123", "price": 5 }, { "sku": "321", "price": 10 } ] } } Demo
19. © 2020, Amazon Web Services, Inc. or its Affiliates. Adding Custom Metrics with PutMetricData const metricData = await cloudWatch.putMetricData({ MetricData: [ { MetricName: 'My Business Metric', Dimensions: [ { Name: 'Location', Value: 'Paris' } ], Timestamp: new Date, Value: 123.4 } ], Namespace: METRIC_NAMESPACE }).promise(); • Metric name • Dimensions • Timestamp • Value • Namespace
20. © 2020, Amazon Web Services, Inc. or its Affiliates. CloudWatch Embedded Metric Format Demo
21. © 2020, Amazon Web Services, Inc. or its Affiliates. Using CloudWatch Anomaly Detection Alarms
22. © 2020, Amazon Web Services, Inc. or its Affiliates. Editing the Anomaly Detection Model Demo
23. © 2020, Amazon Web Services, Inc. or its Affiliates. Safe deployments in SAM Resources: GetFunction: Type: AWS::Serverless::Function Properties: AutoPublishAlias: live DeploymentPreference: Type: Canary10Percent5Minutes Alarms: - !Ref ApiErrorsAlarm - !Ref ApiLatencyAlarm Hooks: PreTraffic: !Ref PreTrafficLambdaFunction PostTraffic: !Ref PreTrafficLambdaFunction Canary10Percent30Minutes Canary10Percent5Minutes Canary10Percent10Minutes Canary10Percent15Minutes Linear10PercentEvery10Minutes Linear10PercentEvery1Minute Linear10PercentEvery2Minutes Linear10PercentEvery3Minutes AllAtOnce + Custom Deployment Configurations
24. © 2020, Amazon Web Services, Inc. or its Affiliates. Safe deployments in SAM Resources: GetFunction: Type: AWS::Serverless::Function Properties: AutoPublishAlias: live DeploymentPreference: Type: Canary10Percent5Minutes Alarms: - !Ref ApiErrorsAlarm - !Ref ApiLatencyAlarm Hooks: PreTraffic: !Ref PreTrafficLambdaFunction PostTraffic: !Ref PreTrafficLambdaFunction These can be CloudWatch anomaly detection alarms
25. © 2020, Amazon Web Services, Inc. or its Affiliates. Understanding Data – CloudWatch Logs + Insights Demo
26. © 2020, Amazon Web Services, Inc. or its Affiliates. End-to-end tracing – AWS X-Ray Traces
27. © 2020, Amazon Web Services, Inc. or its Affiliates. AWS X-Ray Key Concepts Segments Subsegments
28. © 2020, Amazon Web Services, Inc. or its Affiliates. End-to-end tracing – AWS X-Ray Service Map
29. © 2020, Amazon Web Services, Inc. or its Affiliates. Enabling X-Ray tracing in the AWS Console AWS Lambda Console (per function) Amazon API Gateway Console (per stage)
30. © 2020, Amazon Web Services, Inc. or its Affiliates. Enabling X-Ray tracing using AWS SAM Globals: Function: Tracing: Active Api: TracingEnabled: True Global section All Lambda functions All API Gateway REST API
31. © 2020, Amazon Web Services, Inc. or its Affiliates. Enabling X-Ray tracing in your code (Node.js) const AWS = require('aws-sdk'); const AWSXRay = require('aws-xray-sdk'); const AWS = AWSXRay.captureAWS(require('aws-sdk’)); const https = AWSXRay.captureHTTPs(require('https'));
32. © 2020, Amazon Web Services, Inc. or its Affiliates. Using X-Ray tracing in your code (Express.js) const AWSXRay = require('aws-xray-sdk’); const app = express(); app.use(AWSXRay.express.openSegment('my-segment')); app.get('/send', function (req, res) { res.setHeader('Content-Type', 'application/json’); res.send('{"hello": "world"}'); }); app.use(AWSXRay.express.closeSegment());
33. © 2020, Amazon Web Services, Inc. or its Affiliates. Using X-Ray tracing in your code (Express.js) app.get('/', function (req, res) { var host = 'api.example.com’; AWSXRay.captureAsyncFunc('send', function(subsegment) { sendRequest(host, function() { console.log('rendering!’); res.render('index’); subsegment.close(); }); }); });
34. © 2020, Amazon Web Services, Inc. or its Affiliates. Using X-Ray tracing in your code (Lambda function) AWSXRay.captureFunc('annotations', function(subsegment) { subsegment.addAnnotation('Name', name); subsegment.addAnnotation('UserID', event.userid); }); Demo
35. © 2020, Amazon Web Services, Inc. or its Affiliates. CloudWatch ServiceLens CloudWatch metrics and logs + AWS X-Ray traces Demo
36. © 2020, Amazon Web Services, Inc. or its Affiliates. CloudWatch Synthetics – Monitor Endpoints Demo
37. © 2020, Amazon Web Services, Inc. or its Affiliates. How does Serverless work? Storage Databases Analytics Machine Learning . . . Your unique business logic User uploads a picture Customer data updated Anomaly detected API call . . . Fully-managed services Events Functions
38. © 2020, Amazon Web Services, Inc. or its Affiliates. What is an “event” ? “something that happens” Events tell us a fact Immutable time series Time What 2019 06 21 08 07 06 CustomerCreated 2019 06 21 08 07 09 OrderCreated 2019 06 21 08 07 13 PaymentSuccessful 2019 06 21 08 07 17 CustomerUpdated . . . . . .
39. © 2020, Amazon Web Services, Inc. or its Affiliates. Time is important “Modelling events forces you to have a temporal focus on what’s going on in the system. Time becomes a crucial factor of the system.” – Greg Young, A Decade of DDD, CQRS, Event Sourcing, 2016
40. © 2020, Amazon Web Services, Inc. or its Affiliates. More tooling available
41. © 2020, Amazon Web Services, Inc. or its Affiliates. Takeaways 1. Logs – Use Leverage Embedded Metric Format to create custom metrics 2. Metrics – Add business metrics to understand how your application is doing 3. Anomaly Detection – To find the needle in the haystack 4. Tracing – Check the overall flow and drill down to find the root cause 5. Endpoints – Are they working as expected? 6. Events – Store them, they can tell the “story” of your application
42. © 2020, Amazon Web Services, Inc. or its Affiliates. Thank you! @danilop Please give me your feedback!