Share
Embed
End-to-End Observability for Your Serverless Apps
AWS re:Invent 2019 Recap Day Serverless Meetup, London, February 18th, 2020
Danilo Poccia
February 18, 2020
Transcript
42 slides
-
© 2020, Amazon Web Services, Inc. or its Affiliates. Danilo
Poccia Principal Evangelist, Serverless @danilop End-to-End Observability for Your Serverless Apps © 2020, Amazon Web Services, Inc. or its Affiliates. Danilo Poccia Principal Evangelist, Serverless @danilop End-to-End Observability for Your Serverless Apps -
© 2020, Amazon Web Services, Inc. or its Affiliates. Sample
Order Processing App Users API Gateway Endpoint ReadOrder Lambda Function Orders Queue ProcessOrder Lambda Function Orders DynamoDB Table CreateOrder Lambda Function POST /order GET /order -
© 2020, Amazon Web Services, Inc. or its Affiliates. $
sam validate $ sam build $ sam deploy Let’s deploy using AWS SAM… Demo -
© 2020, Amazon Web Services, Inc. or its Affiliates. Don’t
build a network of connected “black boxes” Observability is a developer responsibility -
© 2020, Amazon Web Services, Inc. or its Affiliates. Levels
of Observability Machine (HW, OS) Application Network -
© 2020, Amazon Web Services, Inc. or its Affiliates. Machine
(HW, OS) Application Network The Three Pillars of Observability Distributed Systems Observability by Cindy Sridharan -
© 2020, Amazon Web Services, Inc. or its Affiliates. Machine
(HW, OS) Application Network The Three Pillars of Observability Logs Metrics Tracing Distributed Systems Observability by Cindy Sridharan -
© 2020, Amazon Web Services, Inc. or its Affiliates. Metric
Filters & Correlations IDs Logs Tracing Metric Filter Correlation ID Metrics -
© 2020, Amazon Web Services, Inc. or its Affiliates. Using
Observability Logs Tracing Log aggregation & analytics Visualizations Alerting Metric Filter Correlation ID Metrics -
© 2020, Amazon Web Services, Inc. or its Affiliates. Using
Observability on AWS CloudWatch Logs AWS X-Ray Traces CloudWatch Insights CloudWatch Dashboard CloudWatch Alarms CloudWatch ServiceLens Metric Filter CloudWatch Metrics -
© 2020, Amazon Web Services, Inc. or its Affiliates. Understand
performance… Systems Performance by Brendan Gregg -
© 2020, Amazon Web Services, Inc. or its Affiliates. Understand
performance… and latency… Systems Performance by Brendan Gregg -
© 2020, Amazon Web Services, Inc. or its Affiliates. Understand
performance… and latency… and percentiles! P50 P90 P99 P100 -
© 2020, Amazon Web Services, Inc. or its Affiliates. Let’s
go back to our app… -
© 2020, Amazon Web Services, Inc. or its Affiliates. Sample
Order Processing App Users API Gateway Endpoint ReadOrder Lambda Function Orders Queue ProcessOrder Lambda Function Orders DynamoDB Table CreateOrder Lambda Function POST /order GET /order -
© 2020, Amazon Web Services, Inc. or its Affiliates. HTTP
POST /order { "order": { "accountId": "Danilo", "items": [ { "sku": "123", "price": 5 }, { "sku": "321", "price": 10 } ] } } Demo -
© 2020, Amazon Web Services, Inc. or its Affiliates. HTTP
GET /order/{id} { "order": { "accountId": "Danilo", "id": "4b0e8231-4eca-4356-b6d7-b626499b61e0", "items": [ { "sku": "123", "price": 5 }, { "sku": "321", "price": 10 } ] } } Demo -
© 2020, Amazon Web Services, Inc. or its Affiliates. Adding
Custom Metrics with PutMetricData const metricData = await cloudWatch.putMetricData({ MetricData: [ { MetricName: 'My Business Metric', Dimensions: [ { Name: 'Location', Value: 'Paris' } ], Timestamp: new Date, Value: 123.4 } ], Namespace: METRIC_NAMESPACE }).promise(); • Metric name • Dimensions • Timestamp • Value • Namespace -
© 2020, Amazon Web Services, Inc. or its Affiliates. CloudWatch
Embedded Metric Format Demo -
© 2020, Amazon Web Services, Inc. or its Affiliates. Using
CloudWatch Anomaly Detection Alarms -
© 2020, Amazon Web Services, Inc. or its Affiliates. Editing
the Anomaly Detection Model Demo -
© 2020, Amazon Web Services, Inc. or its Affiliates. Safe
deployments in SAM Resources: GetFunction: Type: AWS::Serverless::Function Properties: AutoPublishAlias: live DeploymentPreference: Type: Canary10Percent5Minutes Alarms: - !Ref ApiErrorsAlarm - !Ref ApiLatencyAlarm Hooks: PreTraffic: !Ref PreTrafficLambdaFunction PostTraffic: !Ref PreTrafficLambdaFunction Canary10Percent30Minutes Canary10Percent5Minutes Canary10Percent10Minutes Canary10Percent15Minutes Linear10PercentEvery10Minutes Linear10PercentEvery1Minute Linear10PercentEvery2Minutes Linear10PercentEvery3Minutes AllAtOnce + Custom Deployment Configurations -
© 2020, Amazon Web Services, Inc. or its Affiliates. Safe
deployments in SAM Resources: GetFunction: Type: AWS::Serverless::Function Properties: AutoPublishAlias: live DeploymentPreference: Type: Canary10Percent5Minutes Alarms: - !Ref ApiErrorsAlarm - !Ref ApiLatencyAlarm Hooks: PreTraffic: !Ref PreTrafficLambdaFunction PostTraffic: !Ref PreTrafficLambdaFunction These can be CloudWatch anomaly detection alarms -
© 2020, Amazon Web Services, Inc. or its Affiliates. Understanding
Data – CloudWatch Logs + Insights Demo -
© 2020, Amazon Web Services, Inc. or its Affiliates. End-to-end
tracing – AWS X-Ray Traces -
© 2020, Amazon Web Services, Inc. or its Affiliates. AWS
X-Ray Key Concepts Segments Subsegments -
© 2020, Amazon Web Services, Inc. or its Affiliates. End-to-end
tracing – AWS X-Ray Service Map -
© 2020, Amazon Web Services, Inc. or its Affiliates. Enabling
X-Ray tracing in the AWS Console AWS Lambda Console (per function) Amazon API Gateway Console (per stage) -
© 2020, Amazon Web Services, Inc. or its Affiliates. Enabling
X-Ray tracing using AWS SAM Globals: Function: Tracing: Active Api: TracingEnabled: True Global section All Lambda functions All API Gateway REST API -
© 2020, Amazon Web Services, Inc. or its Affiliates. Enabling
X-Ray tracing in your code (Node.js) const AWS = require('aws-sdk'); const AWSXRay = require('aws-xray-sdk'); const AWS = AWSXRay.captureAWS(require('aws-sdk’)); const https = AWSXRay.captureHTTPs(require('https')); -
© 2020, Amazon Web Services, Inc. or its Affiliates. Using
X-Ray tracing in your code (Express.js) const AWSXRay = require('aws-xray-sdk’); const app = express(); app.use(AWSXRay.express.openSegment('my-segment')); app.get('/send', function (req, res) { res.setHeader('Content-Type', 'application/json’); res.send('{"hello": "world"}'); }); app.use(AWSXRay.express.closeSegment()); -
© 2020, Amazon Web Services, Inc. or its Affiliates. Using
X-Ray tracing in your code (Express.js) app.get('/', function (req, res) { var host = 'api.example.com’; AWSXRay.captureAsyncFunc('send', function(subsegment) { sendRequest(host, function() { console.log('rendering!’); res.render('index’); subsegment.close(); }); }); }); -
© 2020, Amazon Web Services, Inc. or its Affiliates. Using
X-Ray tracing in your code (Lambda function) AWSXRay.captureFunc('annotations', function(subsegment) { subsegment.addAnnotation('Name', name); subsegment.addAnnotation('UserID', event.userid); }); Demo -
© 2020, Amazon Web Services, Inc. or its Affiliates. CloudWatch
ServiceLens CloudWatch metrics and logs + AWS X-Ray traces Demo -
© 2020, Amazon Web Services, Inc. or its Affiliates. CloudWatch
Synthetics – Monitor Endpoints Demo -
© 2020, Amazon Web Services, Inc. or its Affiliates. How
does Serverless work? Storage Databases Analytics Machine Learning . . . Your unique business logic User uploads a picture Customer data updated Anomaly detected API call . . . Fully-managed services Events Functions -
© 2020, Amazon Web Services, Inc. or its Affiliates. What
is an “event” ? “something that happens” Events tell us a fact Immutable time series Time What 2019 06 21 08 07 06 CustomerCreated 2019 06 21 08 07 09 OrderCreated 2019 06 21 08 07 13 PaymentSuccessful 2019 06 21 08 07 17 CustomerUpdated . . . . . . -
© 2020, Amazon Web Services, Inc. or its Affiliates. Time
is important “Modelling events forces you to have a temporal focus on what’s going on in the system. Time becomes a crucial factor of the system.” – Greg Young, A Decade of DDD, CQRS, Event Sourcing, 2016 -
© 2020, Amazon Web Services, Inc. or its Affiliates. More
tooling available -
© 2020, Amazon Web Services, Inc. or its Affiliates. Takeaways
1. Logs – Use Leverage Embedded Metric Format to create custom metrics 2. Metrics – Add business metrics to understand how your application is doing 3. Anomaly Detection – To find the needle in the haystack 4. Tracing – Check the overall flow and drill down to find the root cause 5. Endpoints – Are they working as expected? 6. Events – Store them, they can tell the “story” of your application -
© 2020, Amazon Web Services, Inc. or its Affiliates. Thank
you! @danilop Please give me your feedback!