Instrumenting Kubernetes for Observability Using AWS X-Ray and Amazon CloudWatch (DEV303)

re:Invent 2018 workshop. See https://github.com/aws-samples/reinvent2018-dev303-code to access the code.

Transcript

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Instrumenting Kubernetes for Observability Using AWS X-Ray and Amazon CloudWatch Christoph Kassen Senior Solutions Architect D E V 3 0 3
3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What to expect? Modern applications Building modern applications Deploying to Amazon Elastic Container Service for Kubernetes (Amazon EKS) Monitoring Amazon CloudWatch Distributed tracing Hands-on labs
4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What are modern applications? Built on containers and serverless Microservices architecture and distributed
6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Benefits of modern applications Improves fault isolation Enables continuous delivery and deployment Each component is autonomous and independent
7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Public API POST /cart GET /cart … Application / Logic Data store Anatomy of a microservice
8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Recommender Catalog Cart Order Frontend Ecosystem of microservices
9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Challenges with microservices Complexity of deploying a distributed system Service discovery Build and deployment automation Application configuration Monitoring and operations Security Evolvability of the system
10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Kubernetes with modern applications Open source Container and microservices platform Hybrid & portable
11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Introducing Amazon EKS Managed Kubernetes Control Plane Highly Available Master and etcd Bring-your-own worker nodes (like Amazon Elastic Container Service [Amazon ECS]) Core Tenets Platform for enterprises to run production-grade workloads Provides a native and upstream experience – Certified Not forced to use additional AWS services, but offer seamless integration Actively contribute to upstream project
12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Remaining challenges Monitoring Varying log formats across services Collecting, aggregating, and collating logs from services Metrics and alarms Distributed systems Cross-service interactions Cascading errors Request flow End-user impact
13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gain insights across resources and applications by enabling observability
14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Getting to answers quickly means collecting and aggregating as much data as possible Observability Metrics Logs Traces
15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Web Services (AWS) observability portfolio Complete visibility of cloud resources and applications • Monitor applications • Respond to performance changes • Optimize resource utilization • Get a unified view of operational health Analyze and debug production, distributed applications • Identify performance bottlenecks • Troubleshoot root cause • Trace user requests • For simple & complex applications
16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon CloudWatch Use AWS generated metrics, logs, and events over time to understand the behavior of your system Publish custom metrics, logs, and events for your application specific telemetry
18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon CloudWatch Trigger automatic notifications based on your own rules and metric thresholds AnyCompany.com
19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Automatically correct issues using common actions that you control Define your own custom actions based on AWS Lambda functions for more fine-grained control Amazon CloudWatch
20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tracing connects the dots Get insights into individual operations Discover multiple services See issues isolated within a service Perform root cause analysis for specific issue
22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Identify performance bottlenecks How does X-Ray help? Pinpoint specific service issues Identify errors Identify impact to users
23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. X-Ray concepts user request response Trace Segment Sub-segment Frontend API Amazon DynamoDB table Amazon Simple Queue Service (Amazon SQS)
24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. X-Ray concepts Trace End-to-end data related a single request across services Segments Portions of the trace that correspond to a single service Sub-segments Remote call or local compute sections within a service Annotations Business data that can be used to filter traces Metadata Business data that can be added to the trace but not used for filtering traces Errors Normalized error message and stack trace Sampling Percentage of requests to your application to capture as traces
25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. X-Ray SDK Enables you to get started quickly without having to manually instrument your application code to log metadata about requests Source on GitHub at https://www.github.com/aws/ Available for Java, .NET, Python, Go, Ruby, and Node.js Adds filters to automatically capture metadata for calls to: AWS services using the AWS SDK Non-AWS services over HTTP and HTTPS Databases (MySQL, PostgreSQL, and Amazon DynamoDB) Queues (Amazon SQS)
26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. X-Ray daemon Receives data from the SDK over UDP and acts as a local buffer; data is flushed to the backend every second or when the local buffer fills Available for Amazon Linux AMI, RHEL, Ubuntu, OS X, and Windows Pre-installed on AWS Lambda Can be run anywhere as long as AWS credentials are provided (for example, Amazon Elastic Compute Cloud [Amazon EC2], Amazon ECS, on- premises, developer machine, and others)
27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. App & X-Ray SDK Pod X-Ray daemon UDP X-Ray API HTTPS HTTPS X-Ray console App & X-Ray SDK X-Ray daemon UDP DevOps Team HTTPS X-Ray workflow Pod Pod Pod
28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Sampling configuration
29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. X-Ray API X-Ray provides a set of APIs to enable you to send, filter, and retrieve trace data You can send trace data directly to the service without having to use our SDKs (that is, you can write your own SDKs for languages not currently supported) Raw trace data is available using batch get APIs You can build your own data analysis applications on top of the data collected by X-Ray
30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. X-Ray API PutTraceSegments Uploads segment documents to AWS X-Ray BatchGetTraces Retrieves a list of traces specified by ID GetServiceGraph Retrieves a document that describes services in your application and their connections GetTraceSummaries Retrieves IDs and metadata for traces available for a specified time frame using an optional filter
31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. X-Ray pricing Free tier The first 100,000 traces recorded are free The first 1,000,000 traces retrieved or scanned are free https://aws.amazon.com/free/ Additional charges Beyond the free tier, traces recorded cost $5.00 per million traces Beyond the free tier, traces retrieved or scanned cost $0.50 per million traces
32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Key takeaways Serverless and containers are building blocks of modern applications AWS X-Ray is powerful tool to visualize and troubleshoot issues Take a user-centric approach to monitor modern applications Leverage X-Ray to debug and quantify customer impact X-Ray works with Amazon EKS and any other Kubernetes cluster and supports multiple languages and use cases
34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Service diagram
36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What are we going to deploy? Tooling: AWS Command Line Interface (AWS CLI), kubectl, aws-iam- authenticator EKS cluster Sample application “AnyCompany Shop“ Microservices architecture Node.JS + Express web frontend Python + Flask based microservices Amazon DynamoDB, Amazon SQS GitHub Repo https://github.com/aws-samples/reinvent2018-dev303-code See README.md for detailed instructions
37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What are we going to deploy? CloudWatch Logs AWS Identity and Access Management (IAM) permissions to send logs FluentD configuration & DaemonSet Prometheus + Grafana Prometheus - Metrics storage - Automated Kubernetes metrics collection Grafana dashboards
40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What you should see
41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What are we going to deploy? Distributed tracing X-Ray daemon Deployed to every EKS node Enable our applications to send traces to AWS X-Ray Incoming requests Calls to supported AWS services Outgoing HTTP requests
45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Deploying X-Ray to Amazon EKS X-Ray daemon Recieves traces from applications Forwards traces to AWS X-Ray service Listening on Port 2000 (TCP and UDP) Should be present on every node Deploy as DaemonSet Configure listening address Configure IAM policies AWSXRayDaemonWriteAccess Configure Pod (or Node)
46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Instrumenting appliation code Add the X-Ray SDKs as dependency Configure the SDK Get IP address of X-Ray daemon via environment variable Set up plugins to get Amazon EC2 and container information Add tracing middleware Allows tracing incoming requests Middlewares for Express, Flask, ... Add tracing to AWS SDKs Patch libraries to trace outgoing requests
47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. App instrumentation (Node.js) //Add aws-xray-sdk package to package.json const AWSXRay = require('aws-xray-sdk'); AWSXRay.config([AWSXRay.plugins.EC2Plugin,AWSXRay.plugins.ECSPlugin]); const xrayExpress = require('aws-xray-sdk-express’); app.use(xrayExpress.openSegment('Frontend’)); app.get('/', function(req, res) … app.get(‘/static', function(req, res) app.use(xrayExpress.closeSegment());
48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Adding business data (Node.js) //Example showing how to add business data to traces app.use(function(req, res, next){ if (req.session !== undefined) { let segment = AWSXRay.getSegment() // User sessionID as userID segment.addAnnotation(‘userID', req.sessionID); } next(); })
49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Node.JS – HTTP client instrumentation const axios = require("axios"); const AWSXRay = require('aws-xray-sdk‘); // Instrument http client globally AWSXRay.captureHTTPsGlobal(require('http')); const http = require('http'); AWSXRay.capturePromise(); // Make sure we capture promises // Get axios instance const instance = axios.create({ httpAgent: new http.Agent() }); X-Ray Branch src/frontend/lib/ws_client.js#L15
50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. App instrumentation (Python) from aws_xray_sdk.core import xray_recorder, patch_all from aws_xray_sdk.ext.flask.middleware import XRayMiddleware plugins = ('EC2Plugin', 'ECSPlugin') # Plugins # Configure recorder xray_recorder.configure(service='recommenderservice',plugins=plugins) # Add Xray middleware to Flask app XRayMiddleware(app, xray_recorder) # Patch clients, SDK‘s such as boto3, requests, ... patch_all()
51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Adding tracing to “AnyCompany Shop” Try to add the instrumentation to the code yourself If you need a little help: Check out the xray branch in the GitHub repository Pre-built containers are also available Just replace the latest tag with xray to update the image
52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What it should look like
53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. • Trace List
54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. • Trace Overview
55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. • Trace Detail
56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Links Amazon EKS: https://aws.amazon.com/eks/ AWS X-Ray: https://aws.amazon.com/xray/ Amazon CloudWatch: https://aws.amazon.com/cloudwatch/ Blog: Application tracing on Kubernetes with AWS X-Ray
58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Links Documentation: http://docs.aws.amazon.com/xray/latest/devguide/ Samples: .NET: https://github.com/awslabs/aws-xray-dotnet-webapp Java: https://github.com/awslabs/eb-java-scorekeep/tree/xray Node.js: https://github.com/awslabs/eb-node-express-sample/tree/xray Python: https://github.com/awslabs/eb-py-flask-signup/tree/xray Lambda: https://github.com/awslabs/aws-xray-rekognition-lambda-sample Alarms & Alerts: https://github.com/aws-samples/aws-xray-cloudwatch-event Heatmap & Trends: https://github.com/aws-samples/aws-xray-scatter-sample
59. Please do not forget to clean up the resources you created See instructions for more details!
60. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Christoph Kassen @christoph_k
61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.