End-to-End Observability for Your Serverless Apps

AWS re:Invent 2019 Recap Day Serverless Meetup, London, February 18th, 2020

Transcript

1. © 2020, Amazon Web Services, Inc. or its Affiliates.
   Danilo Poccia
   Principal Evangelist, Serverless
   @danilop
   End-to-End Observability
   for Your Serverless Apps
   © 2020, Amazon Web Services, Inc. or its Affiliates.
   Danilo Poccia
   Principal Evangelist, Serverless
   @danilop
   End-to-End Observability
   for Your Serverless Apps
   

   View Slide

2. © 2020, Amazon Web Services, Inc. or its Affiliates.
   Sample Order Processing App
   Users
   API Gateway
   Endpoint
   ReadOrder
   Lambda Function
   Orders
   Queue
   ProcessOrder
   Lambda Function
   Orders
   DynamoDB Table
   CreateOrder
   Lambda Function
   POST /order
   GET /order
   

   View Slide

3. © 2020, Amazon Web Services, Inc. or its Affiliates.
   $ sam validate
   $ sam build
   $ sam deploy
   Let’s deploy using AWS SAM…
   Demo
   

   View Slide

4. © 2020, Amazon Web Services, Inc. or its Affiliates.
   Don’t build a network of connected “black boxes”
   Observability is a developer responsibility
   

   View Slide

5. © 2020, Amazon Web Services, Inc. or its Affiliates.
   Levels of Observability
   Machine (HW, OS)
   Application
   Network
   

   View Slide

6. © 2020, Amazon Web Services, Inc. or its Affiliates.
   Machine (HW, OS)
   Application
   Network
   The Three Pillars of Observability
   Distributed Systems Observability by Cindy Sridharan
   

   View Slide

7. © 2020, Amazon Web Services, Inc. or its Affiliates.
   Machine (HW, OS)
   Application
   Network
   The Three Pillars of Observability
   Logs Metrics Tracing
   Distributed Systems Observability by Cindy Sridharan
   

   View Slide

8. © 2020, Amazon Web Services, Inc. or its Affiliates.
   Metric Filters & Correlations IDs
   Logs Tracing
   Metric
   Filter
   Correlation
   ID
   Metrics
   

   View Slide

9. © 2020, Amazon Web Services, Inc. or its Affiliates.
   Using Observability
   Logs Tracing
   Log aggregation
   & analytics
   Visualizations
   Alerting
   Metric
   Filter
   Correlation
   ID
   Metrics
   

   View Slide

10. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Using Observability on AWS
    CloudWatch
    Logs
    AWS X-Ray
    Traces
    CloudWatch
    Insights
    CloudWatch
    Dashboard
    CloudWatch
    Alarms
    CloudWatch
    ServiceLens
    Metric
    Filter
    CloudWatch
    Metrics
    

    View Slide

11. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Understand performance…
    Systems Performance by Brendan Gregg
    

    View Slide

12. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Understand performance… and latency…
    Systems Performance by Brendan Gregg
    

    View Slide

13. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Understand performance… and latency… and
    percentiles!
    P50
    P90
    P99
    P100
    

    View Slide

14. © 2020, Amazon Web Services, Inc. or its Affiliates.
    

    View Slide

15. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Let’s go back
    to our app…
    

    View Slide

16. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Sample Order Processing App
    Users
    API Gateway
    Endpoint
    ReadOrder
    Lambda Function
    Orders
    Queue
    ProcessOrder
    Lambda Function
    Orders
    DynamoDB Table
    CreateOrder
    Lambda Function
    POST /order
    GET /order
    

    View Slide

17. © 2020, Amazon Web Services, Inc. or its Affiliates.
    HTTP POST /order
    {
    "order": {
    "accountId": "Danilo",
    "items": [
    {
    "sku": "123",
    "price": 5
    },
    {
    "sku": "321",
    "price": 10
    }
    ]
    }
    }
    Demo
    

    View Slide

18. © 2020, Amazon Web Services, Inc. or its Affiliates.
    HTTP GET /order/{id}
    {
    "order": {
    "accountId": "Danilo",
    "id": "4b0e8231-4eca-4356-b6d7-b626499b61e0",
    "items": [
    {
    "sku": "123",
    "price": 5
    },
    {
    "sku": "321",
    "price": 10
    }
    ]
    }
    }
    Demo
    

    View Slide

19. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Adding Custom Metrics with PutMetricData
    const metricData = await cloudWatch.putMetricData({
    MetricData: [
    {
    MetricName: 'My Business Metric',
    Dimensions: [
    {
    Name: 'Location',
    Value: 'Paris'
    }
    ],
    Timestamp: new Date,
    Value: 123.4
    }
    ],
    Namespace: METRIC_NAMESPACE
    }).promise();
    • Metric name
    • Dimensions
    • Timestamp
    • Value
    • Namespace
    

    View Slide

20. © 2020, Amazon Web Services, Inc. or its Affiliates.
    CloudWatch Embedded Metric Format
    Demo
    

    View Slide

21. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Using CloudWatch Anomaly Detection Alarms
    

    View Slide

22. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Editing the Anomaly Detection Model
    Demo
    

    View Slide

23. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Safe deployments in SAM
    Resources:
    GetFunction:
    Type: AWS::Serverless::Function
    Properties:
    AutoPublishAlias: live
    DeploymentPreference:
    Type: Canary10Percent5Minutes
    Alarms:
    - !Ref ApiErrorsAlarm
    - !Ref ApiLatencyAlarm
    Hooks:
    PreTraffic: !Ref PreTrafficLambdaFunction
    PostTraffic: !Ref PreTrafficLambdaFunction
    Canary10Percent30Minutes
    Canary10Percent5Minutes
    Canary10Percent10Minutes
    Canary10Percent15Minutes
    Linear10PercentEvery10Minutes
    Linear10PercentEvery1Minute
    Linear10PercentEvery2Minutes
    Linear10PercentEvery3Minutes
    AllAtOnce
    +
    Custom Deployment Configurations
    

    View Slide

24. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Safe deployments in SAM
    Resources:
    GetFunction:
    Type: AWS::Serverless::Function
    Properties:
    AutoPublishAlias: live
    DeploymentPreference:
    Type: Canary10Percent5Minutes
    Alarms:
    - !Ref ApiErrorsAlarm
    - !Ref ApiLatencyAlarm
    Hooks:
    PreTraffic: !Ref PreTrafficLambdaFunction
    PostTraffic: !Ref PreTrafficLambdaFunction
    These can be
    CloudWatch
    anomaly detection
    alarms
    

    View Slide

25. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Understanding Data – CloudWatch Logs + Insights
    Demo
    

    View Slide

26. © 2020, Amazon Web Services, Inc. or its Affiliates.
    End-to-end tracing – AWS X-Ray Traces
    

    View Slide

27. © 2020, Amazon Web Services, Inc. or its Affiliates.
    AWS X-Ray Key Concepts
    Segments
    Subsegments
    

    View Slide

28. © 2020, Amazon Web Services, Inc. or its Affiliates.
    End-to-end tracing – AWS X-Ray Service Map
    

    View Slide

29. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Enabling X-Ray tracing in the AWS Console
    AWS Lambda
    Console
    (per function)
    Amazon
    API Gateway
    Console
    (per stage)
    

    View Slide

30. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Enabling X-Ray tracing using AWS SAM
    Globals:
    Function:
    Tracing: Active
    Api:
    TracingEnabled: True
    Global
    section
    All Lambda
    functions
    All API Gateway
    REST API
    

    View Slide

31. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Enabling X-Ray tracing in your code (Node.js)
    const AWS = require('aws-sdk');
    const AWSXRay = require('aws-xray-sdk');
    const AWS = AWSXRay.captureAWS(require('aws-sdk’));
    const https = AWSXRay.captureHTTPs(require('https'));
    

    View Slide

32. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Using X-Ray tracing in your code (Express.js)
    const AWSXRay = require('aws-xray-sdk’);
    const app = express();
    app.use(AWSXRay.express.openSegment('my-segment'));
    app.get('/send', function (req, res) {
    res.setHeader('Content-Type', 'application/json’);
    res.send('{"hello": "world"}');
    });
    app.use(AWSXRay.express.closeSegment());
    

    View Slide

33. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Using X-Ray tracing in your code (Express.js)
    app.get('/', function (req, res) {
    var host = 'api.example.com’;
    AWSXRay.captureAsyncFunc('send', function(subsegment) {
    sendRequest(host, function() {
    console.log('rendering!’);
    res.render('index’);
    subsegment.close();
    });
    });
    });
    

    View Slide

34. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Using X-Ray tracing in your code (Lambda function)
    AWSXRay.captureFunc('annotations', function(subsegment) {
    subsegment.addAnnotation('Name', name);
    subsegment.addAnnotation('UserID', event.userid);
    });
    Demo
    

    View Slide

35. © 2020, Amazon Web Services, Inc. or its Affiliates.
    CloudWatch ServiceLens
    CloudWatch metrics and logs + AWS X-Ray traces
    Demo
    

    View Slide

36. © 2020, Amazon Web Services, Inc. or its Affiliates.
    CloudWatch Synthetics – Monitor Endpoints
    Demo
    

    View Slide

37. © 2020, Amazon Web Services, Inc. or its Affiliates.
    How does Serverless work?
    Storage
    Databases
    Analytics
    Machine Learning
    . . .
    Your
    unique
    business
    logic
    User uploads a picture
    Customer data updated
    Anomaly detected
    API call
    . . .
    Fully-managed
    services
    Events
    Functions
    

    View Slide

38. © 2020, Amazon Web Services, Inc. or its Affiliates.
    What is an “event” ?
    “something that happens”
    Events tell us a fact
    Immutable time series
    Time What
    2019 06 21 08 07 06 CustomerCreated
    2019 06 21 08 07 09 OrderCreated
    2019 06 21 08 07 13 PaymentSuccessful
    2019 06 21 08 07 17 CustomerUpdated
    . . . . . .
    

    View Slide

39. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Time is important
    “Modelling events forces you to have a temporal
    focus on what’s going on in the system.
    Time becomes a crucial factor of the system.”
    – Greg Young, A Decade of DDD, CQRS, Event Sourcing, 2016
    

    View Slide

40. © 2020, Amazon Web Services, Inc. or its Affiliates.
    More tooling available
    

    View Slide

41. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Takeaways
    1. Logs – Use Leverage Embedded Metric Format to create custom metrics
    2. Metrics – Add business metrics to understand how your application is doing
    3. Anomaly Detection – To find the needle in the haystack
    4. Tracing – Check the overall flow and drill down to find the root cause
    5. Endpoints – Are they working as expected?
    6. Events – Store them, they can tell the “story” of your application
    

    View Slide

42. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Thank you!
    @danilop Please give me your feedback!
    

    View Slide