AWS re:Invent 2019 Recap Day Serverless Meetup, London, February 18th, 2020
© 2020, Amazon Web Services, Inc. or its Affiliates.Danilo PocciaPrincipal Evangelist, Serverless@danilopEnd-to-End Observabilityfor Your Serverless Apps© 2020, Amazon Web Services, Inc. or its Affiliates.Danilo PocciaPrincipal Evangelist, Serverless@danilopEnd-to-End Observabilityfor Your Serverless Apps
View Slide
© 2020, Amazon Web Services, Inc. or its Affiliates.Sample Order Processing AppUsersAPI GatewayEndpointReadOrderLambda FunctionOrdersQueueProcessOrderLambda FunctionOrdersDynamoDB TableCreateOrderLambda FunctionPOST /orderGET /order
© 2020, Amazon Web Services, Inc. or its Affiliates.$ sam validate$ sam build$ sam deployLet’s deploy using AWS SAM…Demo
© 2020, Amazon Web Services, Inc. or its Affiliates.Don’t build a network of connected “black boxes”Observability is a developer responsibility
© 2020, Amazon Web Services, Inc. or its Affiliates.Levels of ObservabilityMachine (HW, OS)ApplicationNetwork
© 2020, Amazon Web Services, Inc. or its Affiliates.Machine (HW, OS)ApplicationNetworkThe Three Pillars of ObservabilityDistributed Systems Observability by Cindy Sridharan
© 2020, Amazon Web Services, Inc. or its Affiliates.Machine (HW, OS)ApplicationNetworkThe Three Pillars of ObservabilityLogs Metrics TracingDistributed Systems Observability by Cindy Sridharan
© 2020, Amazon Web Services, Inc. or its Affiliates.Metric Filters & Correlations IDsLogs TracingMetricFilterCorrelationIDMetrics
© 2020, Amazon Web Services, Inc. or its Affiliates.Using ObservabilityLogs TracingLog aggregation& analyticsVisualizationsAlertingMetricFilterCorrelationIDMetrics
© 2020, Amazon Web Services, Inc. or its Affiliates.Using Observability on AWSCloudWatchLogsAWS X-RayTracesCloudWatchInsightsCloudWatchDashboardCloudWatchAlarmsCloudWatchServiceLensMetricFilterCloudWatchMetrics
© 2020, Amazon Web Services, Inc. or its Affiliates.Understand performance…Systems Performance by Brendan Gregg
© 2020, Amazon Web Services, Inc. or its Affiliates.Understand performance… and latency…Systems Performance by Brendan Gregg
© 2020, Amazon Web Services, Inc. or its Affiliates.Understand performance… and latency… andpercentiles!P50P90P99P100
© 2020, Amazon Web Services, Inc. or its Affiliates.
© 2020, Amazon Web Services, Inc. or its Affiliates.Let’s go backto our app…
© 2020, Amazon Web Services, Inc. or its Affiliates.HTTP POST /order{"order": {"accountId": "Danilo","items": [{"sku": "123","price": 5},{"sku": "321","price": 10}]}}Demo
© 2020, Amazon Web Services, Inc. or its Affiliates.HTTP GET /order/{id}{"order": {"accountId": "Danilo","id": "4b0e8231-4eca-4356-b6d7-b626499b61e0","items": [{"sku": "123","price": 5},{"sku": "321","price": 10}]}}Demo
© 2020, Amazon Web Services, Inc. or its Affiliates.Adding Custom Metrics with PutMetricDataconst metricData = await cloudWatch.putMetricData({MetricData: [{MetricName: 'My Business Metric',Dimensions: [{Name: 'Location',Value: 'Paris'}],Timestamp: new Date,Value: 123.4}],Namespace: METRIC_NAMESPACE}).promise();• Metric name• Dimensions• Timestamp• Value• Namespace
© 2020, Amazon Web Services, Inc. or its Affiliates.CloudWatch Embedded Metric FormatDemo
© 2020, Amazon Web Services, Inc. or its Affiliates.Using CloudWatch Anomaly Detection Alarms
© 2020, Amazon Web Services, Inc. or its Affiliates.Editing the Anomaly Detection ModelDemo
© 2020, Amazon Web Services, Inc. or its Affiliates.Safe deployments in SAMResources:GetFunction:Type: AWS::Serverless::FunctionProperties:AutoPublishAlias: liveDeploymentPreference:Type: Canary10Percent5MinutesAlarms:- !Ref ApiErrorsAlarm- !Ref ApiLatencyAlarmHooks:PreTraffic: !Ref PreTrafficLambdaFunctionPostTraffic: !Ref PreTrafficLambdaFunctionCanary10Percent30MinutesCanary10Percent5MinutesCanary10Percent10MinutesCanary10Percent15MinutesLinear10PercentEvery10MinutesLinear10PercentEvery1MinuteLinear10PercentEvery2MinutesLinear10PercentEvery3MinutesAllAtOnce+Custom Deployment Configurations
© 2020, Amazon Web Services, Inc. or its Affiliates.Safe deployments in SAMResources:GetFunction:Type: AWS::Serverless::FunctionProperties:AutoPublishAlias: liveDeploymentPreference:Type: Canary10Percent5MinutesAlarms:- !Ref ApiErrorsAlarm- !Ref ApiLatencyAlarmHooks:PreTraffic: !Ref PreTrafficLambdaFunctionPostTraffic: !Ref PreTrafficLambdaFunctionThese can beCloudWatchanomaly detectionalarms
© 2020, Amazon Web Services, Inc. or its Affiliates.Understanding Data – CloudWatch Logs + InsightsDemo
© 2020, Amazon Web Services, Inc. or its Affiliates.End-to-end tracing – AWS X-Ray Traces
© 2020, Amazon Web Services, Inc. or its Affiliates.AWS X-Ray Key ConceptsSegmentsSubsegments
© 2020, Amazon Web Services, Inc. or its Affiliates.End-to-end tracing – AWS X-Ray Service Map
© 2020, Amazon Web Services, Inc. or its Affiliates.Enabling X-Ray tracing in the AWS ConsoleAWS LambdaConsole(per function)AmazonAPI GatewayConsole(per stage)
© 2020, Amazon Web Services, Inc. or its Affiliates.Enabling X-Ray tracing using AWS SAMGlobals:Function:Tracing: ActiveApi:TracingEnabled: TrueGlobalsectionAll LambdafunctionsAll API GatewayREST API
© 2020, Amazon Web Services, Inc. or its Affiliates.Enabling X-Ray tracing in your code (Node.js)const AWS = require('aws-sdk');const AWSXRay = require('aws-xray-sdk');const AWS = AWSXRay.captureAWS(require('aws-sdk’));const https = AWSXRay.captureHTTPs(require('https'));
© 2020, Amazon Web Services, Inc. or its Affiliates.Using X-Ray tracing in your code (Express.js)const AWSXRay = require('aws-xray-sdk’);const app = express();app.use(AWSXRay.express.openSegment('my-segment'));app.get('/send', function (req, res) {res.setHeader('Content-Type', 'application/json’);res.send('{"hello": "world"}');});app.use(AWSXRay.express.closeSegment());
© 2020, Amazon Web Services, Inc. or its Affiliates.Using X-Ray tracing in your code (Express.js)app.get('/', function (req, res) {var host = 'api.example.com’;AWSXRay.captureAsyncFunc('send', function(subsegment) {sendRequest(host, function() {console.log('rendering!’);res.render('index’);subsegment.close();});});});
© 2020, Amazon Web Services, Inc. or its Affiliates.Using X-Ray tracing in your code (Lambda function)AWSXRay.captureFunc('annotations', function(subsegment) {subsegment.addAnnotation('Name', name);subsegment.addAnnotation('UserID', event.userid);});Demo
© 2020, Amazon Web Services, Inc. or its Affiliates.CloudWatch ServiceLensCloudWatch metrics and logs + AWS X-Ray tracesDemo
© 2020, Amazon Web Services, Inc. or its Affiliates.CloudWatch Synthetics – Monitor EndpointsDemo
© 2020, Amazon Web Services, Inc. or its Affiliates.How does Serverless work?StorageDatabasesAnalyticsMachine Learning. . .YouruniquebusinesslogicUser uploads a pictureCustomer data updatedAnomaly detectedAPI call. . .Fully-managedservicesEventsFunctions
© 2020, Amazon Web Services, Inc. or its Affiliates.What is an “event” ?“something that happens”Events tell us a factImmutable time seriesTime What2019 06 21 08 07 06 CustomerCreated2019 06 21 08 07 09 OrderCreated2019 06 21 08 07 13 PaymentSuccessful2019 06 21 08 07 17 CustomerUpdated. . . . . .
© 2020, Amazon Web Services, Inc. or its Affiliates.Time is important“Modelling events forces you to have a temporalfocus on what’s going on in the system.Time becomes a crucial factor of the system.”– Greg Young, A Decade of DDD, CQRS, Event Sourcing, 2016
© 2020, Amazon Web Services, Inc. or its Affiliates.More tooling available
© 2020, Amazon Web Services, Inc. or its Affiliates.Takeaways1. Logs – Use Leverage Embedded Metric Format to create custom metrics2. Metrics – Add business metrics to understand how your application is doing3. Anomaly Detection – To find the needle in the haystack4. Tracing – Check the overall flow and drill down to find the root cause5. Endpoints – Are they working as expected?6. Events – Store them, they can tell the “story” of your application
© 2020, Amazon Web Services, Inc. or its Affiliates.Thank you!@danilop Please give me your feedback!