$30 off During Our Annual Pro Sale. View Details »

End-to-End Observability for Your Serverless Apps

End-to-End Observability for Your Serverless Apps

AWS re:Invent 2019 Recap Day Serverless Meetup, London, February 18th, 2020

Danilo Poccia

February 18, 2020
Tweet

More Decks by Danilo Poccia

Other Decks in Programming

Transcript

  1. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Danilo Poccia
    Principal Evangelist, Serverless
    @danilop
    End-to-End Observability
    for Your Serverless Apps
    © 2020, Amazon Web Services, Inc. or its Affiliates.
    Danilo Poccia
    Principal Evangelist, Serverless
    @danilop
    End-to-End Observability
    for Your Serverless Apps

    View Slide

  2. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Sample Order Processing App
    Users
    API Gateway
    Endpoint
    ReadOrder
    Lambda Function
    Orders
    Queue
    ProcessOrder
    Lambda Function
    Orders
    DynamoDB Table
    CreateOrder
    Lambda Function
    POST /order
    GET /order

    View Slide

  3. © 2020, Amazon Web Services, Inc. or its Affiliates.
    $ sam validate
    $ sam build
    $ sam deploy
    Let’s deploy using AWS SAM…
    Demo

    View Slide

  4. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Don’t build a network of connected “black boxes”
    Observability is a developer responsibility

    View Slide

  5. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Levels of Observability
    Machine (HW, OS)
    Application
    Network

    View Slide

  6. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Machine (HW, OS)
    Application
    Network
    The Three Pillars of Observability
    Distributed Systems Observability by Cindy Sridharan

    View Slide

  7. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Machine (HW, OS)
    Application
    Network
    The Three Pillars of Observability
    Logs Metrics Tracing
    Distributed Systems Observability by Cindy Sridharan

    View Slide

  8. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Metric Filters & Correlations IDs
    Logs Tracing
    Metric
    Filter
    Correlation
    ID
    Metrics

    View Slide

  9. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Using Observability
    Logs Tracing
    Log aggregation
    & analytics
    Visualizations
    Alerting
    Metric
    Filter
    Correlation
    ID
    Metrics

    View Slide

  10. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Using Observability on AWS
    CloudWatch
    Logs
    AWS X-Ray
    Traces
    CloudWatch
    Insights
    CloudWatch
    Dashboard
    CloudWatch
    Alarms
    CloudWatch
    ServiceLens
    Metric
    Filter
    CloudWatch
    Metrics

    View Slide

  11. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Understand performance…
    Systems Performance by Brendan Gregg

    View Slide

  12. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Understand performance… and latency…
    Systems Performance by Brendan Gregg

    View Slide

  13. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Understand performance… and latency… and
    percentiles!
    P50
    P90
    P99
    P100

    View Slide

  14. © 2020, Amazon Web Services, Inc. or its Affiliates.

    View Slide

  15. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Let’s go back
    to our app…

    View Slide

  16. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Sample Order Processing App
    Users
    API Gateway
    Endpoint
    ReadOrder
    Lambda Function
    Orders
    Queue
    ProcessOrder
    Lambda Function
    Orders
    DynamoDB Table
    CreateOrder
    Lambda Function
    POST /order
    GET /order

    View Slide

  17. © 2020, Amazon Web Services, Inc. or its Affiliates.
    HTTP POST /order
    {
    "order": {
    "accountId": "Danilo",
    "items": [
    {
    "sku": "123",
    "price": 5
    },
    {
    "sku": "321",
    "price": 10
    }
    ]
    }
    }
    Demo

    View Slide

  18. © 2020, Amazon Web Services, Inc. or its Affiliates.
    HTTP GET /order/{id}
    {
    "order": {
    "accountId": "Danilo",
    "id": "4b0e8231-4eca-4356-b6d7-b626499b61e0",
    "items": [
    {
    "sku": "123",
    "price": 5
    },
    {
    "sku": "321",
    "price": 10
    }
    ]
    }
    }
    Demo

    View Slide

  19. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Adding Custom Metrics with PutMetricData
    const metricData = await cloudWatch.putMetricData({
    MetricData: [
    {
    MetricName: 'My Business Metric',
    Dimensions: [
    {
    Name: 'Location',
    Value: 'Paris'
    }
    ],
    Timestamp: new Date,
    Value: 123.4
    }
    ],
    Namespace: METRIC_NAMESPACE
    }).promise();
    • Metric name
    • Dimensions
    • Timestamp
    • Value
    • Namespace

    View Slide

  20. © 2020, Amazon Web Services, Inc. or its Affiliates.
    CloudWatch Embedded Metric Format
    Demo

    View Slide

  21. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Using CloudWatch Anomaly Detection Alarms

    View Slide

  22. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Editing the Anomaly Detection Model
    Demo

    View Slide

  23. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Safe deployments in SAM
    Resources:
    GetFunction:
    Type: AWS::Serverless::Function
    Properties:
    AutoPublishAlias: live
    DeploymentPreference:
    Type: Canary10Percent5Minutes
    Alarms:
    - !Ref ApiErrorsAlarm
    - !Ref ApiLatencyAlarm
    Hooks:
    PreTraffic: !Ref PreTrafficLambdaFunction
    PostTraffic: !Ref PreTrafficLambdaFunction
    Canary10Percent30Minutes
    Canary10Percent5Minutes
    Canary10Percent10Minutes
    Canary10Percent15Minutes
    Linear10PercentEvery10Minutes
    Linear10PercentEvery1Minute
    Linear10PercentEvery2Minutes
    Linear10PercentEvery3Minutes
    AllAtOnce
    +
    Custom Deployment Configurations

    View Slide

  24. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Safe deployments in SAM
    Resources:
    GetFunction:
    Type: AWS::Serverless::Function
    Properties:
    AutoPublishAlias: live
    DeploymentPreference:
    Type: Canary10Percent5Minutes
    Alarms:
    - !Ref ApiErrorsAlarm
    - !Ref ApiLatencyAlarm
    Hooks:
    PreTraffic: !Ref PreTrafficLambdaFunction
    PostTraffic: !Ref PreTrafficLambdaFunction
    These can be
    CloudWatch
    anomaly detection
    alarms

    View Slide

  25. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Understanding Data – CloudWatch Logs + Insights
    Demo

    View Slide

  26. © 2020, Amazon Web Services, Inc. or its Affiliates.
    End-to-end tracing – AWS X-Ray Traces

    View Slide

  27. © 2020, Amazon Web Services, Inc. or its Affiliates.
    AWS X-Ray Key Concepts
    Segments
    Subsegments

    View Slide

  28. © 2020, Amazon Web Services, Inc. or its Affiliates.
    End-to-end tracing – AWS X-Ray Service Map

    View Slide

  29. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Enabling X-Ray tracing in the AWS Console
    AWS Lambda
    Console
    (per function)
    Amazon
    API Gateway
    Console
    (per stage)

    View Slide

  30. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Enabling X-Ray tracing using AWS SAM
    Globals:
    Function:
    Tracing: Active
    Api:
    TracingEnabled: True
    Global
    section
    All Lambda
    functions
    All API Gateway
    REST API

    View Slide

  31. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Enabling X-Ray tracing in your code (Node.js)
    const AWS = require('aws-sdk');
    const AWSXRay = require('aws-xray-sdk');
    const AWS = AWSXRay.captureAWS(require('aws-sdk’));
    const https = AWSXRay.captureHTTPs(require('https'));

    View Slide

  32. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Using X-Ray tracing in your code (Express.js)
    const AWSXRay = require('aws-xray-sdk’);
    const app = express();
    app.use(AWSXRay.express.openSegment('my-segment'));
    app.get('/send', function (req, res) {
    res.setHeader('Content-Type', 'application/json’);
    res.send('{"hello": "world"}');
    });
    app.use(AWSXRay.express.closeSegment());

    View Slide

  33. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Using X-Ray tracing in your code (Express.js)
    app.get('/', function (req, res) {
    var host = 'api.example.com’;
    AWSXRay.captureAsyncFunc('send', function(subsegment) {
    sendRequest(host, function() {
    console.log('rendering!’);
    res.render('index’);
    subsegment.close();
    });
    });
    });

    View Slide

  34. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Using X-Ray tracing in your code (Lambda function)
    AWSXRay.captureFunc('annotations', function(subsegment) {
    subsegment.addAnnotation('Name', name);
    subsegment.addAnnotation('UserID', event.userid);
    });
    Demo

    View Slide

  35. © 2020, Amazon Web Services, Inc. or its Affiliates.
    CloudWatch ServiceLens
    CloudWatch metrics and logs + AWS X-Ray traces
    Demo

    View Slide

  36. © 2020, Amazon Web Services, Inc. or its Affiliates.
    CloudWatch Synthetics – Monitor Endpoints
    Demo

    View Slide

  37. © 2020, Amazon Web Services, Inc. or its Affiliates.
    How does Serverless work?
    Storage
    Databases
    Analytics
    Machine Learning
    . . .
    Your
    unique
    business
    logic
    User uploads a picture
    Customer data updated
    Anomaly detected
    API call
    . . .
    Fully-managed
    services
    Events
    Functions

    View Slide

  38. © 2020, Amazon Web Services, Inc. or its Affiliates.
    What is an “event” ?
    “something that happens”
    Events tell us a fact
    Immutable time series
    Time What
    2019 06 21 08 07 06 CustomerCreated
    2019 06 21 08 07 09 OrderCreated
    2019 06 21 08 07 13 PaymentSuccessful
    2019 06 21 08 07 17 CustomerUpdated
    . . . . . .

    View Slide

  39. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Time is important
    “Modelling events forces you to have a temporal
    focus on what’s going on in the system.
    Time becomes a crucial factor of the system.”
    – Greg Young, A Decade of DDD, CQRS, Event Sourcing, 2016

    View Slide

  40. © 2020, Amazon Web Services, Inc. or its Affiliates.
    More tooling available

    View Slide

  41. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Takeaways
    1. Logs – Use Leverage Embedded Metric Format to create custom metrics
    2. Metrics – Add business metrics to understand how your application is doing
    3. Anomaly Detection – To find the needle in the haystack
    4. Tracing – Check the overall flow and drill down to find the root cause
    5. Endpoints – Are they working as expected?
    6. Events – Store them, they can tell the “story” of your application

    View Slide

  42. © 2020, Amazon Web Services, Inc. or its Affiliates.
    Thank you!
    @danilop Please give me your feedback!

    View Slide