End-to-End Observability for Your Serverless Apps

End-to-End Observability for Your Serverless Apps

AWS re:Invent 2019 Recap Day Serverless Meetup, London, February 18th, 2020

7c9b8b368924556d8642bdaed3ded1f5?s=128

Danilo Poccia

February 18, 2020
Tweet

Transcript

  1. © 2020, Amazon Web Services, Inc. or its Affiliates. Danilo

    Poccia Principal Evangelist, Serverless @danilop End-to-End Observability for Your Serverless Apps © 2020, Amazon Web Services, Inc. or its Affiliates. Danilo Poccia Principal Evangelist, Serverless @danilop End-to-End Observability for Your Serverless Apps
  2. © 2020, Amazon Web Services, Inc. or its Affiliates. Sample

    Order Processing App Users API Gateway Endpoint ReadOrder Lambda Function Orders Queue ProcessOrder Lambda Function Orders DynamoDB Table CreateOrder Lambda Function POST /order GET /order
  3. © 2020, Amazon Web Services, Inc. or its Affiliates. $

    sam validate $ sam build $ sam deploy Let’s deploy using AWS SAM… Demo
  4. © 2020, Amazon Web Services, Inc. or its Affiliates. Don’t

    build a network of connected “black boxes” Observability is a developer responsibility
  5. © 2020, Amazon Web Services, Inc. or its Affiliates. Levels

    of Observability Machine (HW, OS) Application Network
  6. © 2020, Amazon Web Services, Inc. or its Affiliates. Machine

    (HW, OS) Application Network The Three Pillars of Observability Distributed Systems Observability by Cindy Sridharan
  7. © 2020, Amazon Web Services, Inc. or its Affiliates. Machine

    (HW, OS) Application Network The Three Pillars of Observability Logs Metrics Tracing Distributed Systems Observability by Cindy Sridharan
  8. © 2020, Amazon Web Services, Inc. or its Affiliates. Metric

    Filters & Correlations IDs Logs Tracing Metric Filter Correlation ID Metrics
  9. © 2020, Amazon Web Services, Inc. or its Affiliates. Using

    Observability Logs Tracing Log aggregation & analytics Visualizations Alerting Metric Filter Correlation ID Metrics
  10. © 2020, Amazon Web Services, Inc. or its Affiliates. Using

    Observability on AWS CloudWatch Logs AWS X-Ray Traces CloudWatch Insights CloudWatch Dashboard CloudWatch Alarms CloudWatch ServiceLens Metric Filter CloudWatch Metrics
  11. © 2020, Amazon Web Services, Inc. or its Affiliates. Understand

    performance… Systems Performance by Brendan Gregg
  12. © 2020, Amazon Web Services, Inc. or its Affiliates. Understand

    performance… and latency… Systems Performance by Brendan Gregg
  13. © 2020, Amazon Web Services, Inc. or its Affiliates. Understand

    performance… and latency… and percentiles! P50 P90 P99 P100
  14. © 2020, Amazon Web Services, Inc. or its Affiliates.

  15. © 2020, Amazon Web Services, Inc. or its Affiliates. Let’s

    go back to our app…
  16. © 2020, Amazon Web Services, Inc. or its Affiliates. Sample

    Order Processing App Users API Gateway Endpoint ReadOrder Lambda Function Orders Queue ProcessOrder Lambda Function Orders DynamoDB Table CreateOrder Lambda Function POST /order GET /order
  17. © 2020, Amazon Web Services, Inc. or its Affiliates. HTTP

    POST /order { "order": { "accountId": "Danilo", "items": [ { "sku": "123", "price": 5 }, { "sku": "321", "price": 10 } ] } } Demo
  18. © 2020, Amazon Web Services, Inc. or its Affiliates. HTTP

    GET /order/{id} { "order": { "accountId": "Danilo", "id": "4b0e8231-4eca-4356-b6d7-b626499b61e0", "items": [ { "sku": "123", "price": 5 }, { "sku": "321", "price": 10 } ] } } Demo
  19. © 2020, Amazon Web Services, Inc. or its Affiliates. Adding

    Custom Metrics with PutMetricData const metricData = await cloudWatch.putMetricData({ MetricData: [ { MetricName: 'My Business Metric', Dimensions: [ { Name: 'Location', Value: 'Paris' } ], Timestamp: new Date, Value: 123.4 } ], Namespace: METRIC_NAMESPACE }).promise(); • Metric name • Dimensions • Timestamp • Value • Namespace
  20. © 2020, Amazon Web Services, Inc. or its Affiliates. CloudWatch

    Embedded Metric Format Demo
  21. © 2020, Amazon Web Services, Inc. or its Affiliates. Using

    CloudWatch Anomaly Detection Alarms
  22. © 2020, Amazon Web Services, Inc. or its Affiliates. Editing

    the Anomaly Detection Model Demo
  23. © 2020, Amazon Web Services, Inc. or its Affiliates. Safe

    deployments in SAM Resources: GetFunction: Type: AWS::Serverless::Function Properties: AutoPublishAlias: live DeploymentPreference: Type: Canary10Percent5Minutes Alarms: - !Ref ApiErrorsAlarm - !Ref ApiLatencyAlarm Hooks: PreTraffic: !Ref PreTrafficLambdaFunction PostTraffic: !Ref PreTrafficLambdaFunction Canary10Percent30Minutes Canary10Percent5Minutes Canary10Percent10Minutes Canary10Percent15Minutes Linear10PercentEvery10Minutes Linear10PercentEvery1Minute Linear10PercentEvery2Minutes Linear10PercentEvery3Minutes AllAtOnce + Custom Deployment Configurations
  24. © 2020, Amazon Web Services, Inc. or its Affiliates. Safe

    deployments in SAM Resources: GetFunction: Type: AWS::Serverless::Function Properties: AutoPublishAlias: live DeploymentPreference: Type: Canary10Percent5Minutes Alarms: - !Ref ApiErrorsAlarm - !Ref ApiLatencyAlarm Hooks: PreTraffic: !Ref PreTrafficLambdaFunction PostTraffic: !Ref PreTrafficLambdaFunction These can be CloudWatch anomaly detection alarms
  25. © 2020, Amazon Web Services, Inc. or its Affiliates. Understanding

    Data – CloudWatch Logs + Insights Demo
  26. © 2020, Amazon Web Services, Inc. or its Affiliates. End-to-end

    tracing – AWS X-Ray Traces
  27. © 2020, Amazon Web Services, Inc. or its Affiliates. AWS

    X-Ray Key Concepts Segments Subsegments
  28. © 2020, Amazon Web Services, Inc. or its Affiliates. End-to-end

    tracing – AWS X-Ray Service Map
  29. © 2020, Amazon Web Services, Inc. or its Affiliates. Enabling

    X-Ray tracing in the AWS Console AWS Lambda Console (per function) Amazon API Gateway Console (per stage)
  30. © 2020, Amazon Web Services, Inc. or its Affiliates. Enabling

    X-Ray tracing using AWS SAM Globals: Function: Tracing: Active Api: TracingEnabled: True Global section All Lambda functions All API Gateway REST API
  31. © 2020, Amazon Web Services, Inc. or its Affiliates. Enabling

    X-Ray tracing in your code (Node.js) const AWS = require('aws-sdk'); const AWSXRay = require('aws-xray-sdk'); const AWS = AWSXRay.captureAWS(require('aws-sdk’)); const https = AWSXRay.captureHTTPs(require('https'));
  32. © 2020, Amazon Web Services, Inc. or its Affiliates. Using

    X-Ray tracing in your code (Express.js) const AWSXRay = require('aws-xray-sdk’); const app = express(); app.use(AWSXRay.express.openSegment('my-segment')); app.get('/send', function (req, res) { res.setHeader('Content-Type', 'application/json’); res.send('{"hello": "world"}'); }); app.use(AWSXRay.express.closeSegment());
  33. © 2020, Amazon Web Services, Inc. or its Affiliates. Using

    X-Ray tracing in your code (Express.js) app.get('/', function (req, res) { var host = 'api.example.com’; AWSXRay.captureAsyncFunc('send', function(subsegment) { sendRequest(host, function() { console.log('rendering!’); res.render('index’); subsegment.close(); }); }); });
  34. © 2020, Amazon Web Services, Inc. or its Affiliates. Using

    X-Ray tracing in your code (Lambda function) AWSXRay.captureFunc('annotations', function(subsegment) { subsegment.addAnnotation('Name', name); subsegment.addAnnotation('UserID', event.userid); }); Demo
  35. © 2020, Amazon Web Services, Inc. or its Affiliates. CloudWatch

    ServiceLens CloudWatch metrics and logs + AWS X-Ray traces Demo
  36. © 2020, Amazon Web Services, Inc. or its Affiliates. CloudWatch

    Synthetics – Monitor Endpoints Demo
  37. © 2020, Amazon Web Services, Inc. or its Affiliates. How

    does Serverless work? Storage Databases Analytics Machine Learning . . . Your unique business logic User uploads a picture Customer data updated Anomaly detected API call . . . Fully-managed services Events Functions
  38. © 2020, Amazon Web Services, Inc. or its Affiliates. What

    is an “event” ? “something that happens” Events tell us a fact Immutable time series Time What 2019 06 21 08 07 06 CustomerCreated 2019 06 21 08 07 09 OrderCreated 2019 06 21 08 07 13 PaymentSuccessful 2019 06 21 08 07 17 CustomerUpdated . . . . . .
  39. © 2020, Amazon Web Services, Inc. or its Affiliates. Time

    is important “Modelling events forces you to have a temporal focus on what’s going on in the system. Time becomes a crucial factor of the system.” – Greg Young, A Decade of DDD, CQRS, Event Sourcing, 2016
  40. © 2020, Amazon Web Services, Inc. or its Affiliates. More

    tooling available
  41. © 2020, Amazon Web Services, Inc. or its Affiliates. Takeaways

    1. Logs – Use Leverage Embedded Metric Format to create custom metrics 2. Metrics – Add business metrics to understand how your application is doing 3. Anomaly Detection – To find the needle in the haystack 4. Tracing – Check the overall flow and drill down to find the root cause 5. Endpoints – Are they working as expected? 6. Events – Store them, they can tell the “story” of your application
  42. © 2020, Amazon Web Services, Inc. or its Affiliates. Thank

    you! @danilop Please give me your feedback!