Tracing, Fast & Slow: Digging into and improving your web service's performance

* PyLadies in St Petersburg, Nov 2018
* EuroPython 2017
* PyCon 2017

Transcript

1. Lynn Root | SRE | @roguelynn Tracing: Fast & Slow Digging into and improving your web service’s performance
2. $ whoami
3. —
4. agenda —
5. agenda • Overview and problem space —
6. agenda • Overview and problem space • Approaches to tracing —
7. agenda • Overview and problem space • Approaches to tracing • Tracing at scale —
8. agenda • Overview and problem space • Approaches to tracing • Tracing at scale • Diagnosing performance issues —
9. agenda • Overview and problem space • Approaches to tracing • Tracing at scale • Diagnosing performance issues • Tracing services & systems —
10. Tracing Overview —
11. machine-centric • Focus on a single machine —
12. machine-centric • Focus on a single machine • No view into a service’s dependencies —
13. workflow-centric • Understand causal relationships —
14. workflow-centric • Understand causal relationships • End-to-end tracing —
16. 100k’s client connections 100’s access point hosts 1,000’s unique services running on 10k’s hosts
17. why trace? —
18. why trace? • Performance analysis —
19. why trace? • Performance analysis • Anomaly detection —
20. why trace? • Performance analysis • Anomaly detection • Profiling —
21. why trace? • Performance analysis • Anomaly detection • Profiling • Resource attribution —
22. why trace? • Performance analysis • Anomaly detection • Profiling • Resource attribution • Workload modeling —
23. Tracing Approaches —
24. manual
25. def request_id(f): @wraps(f) def decorated(*args, **kwargs): req_id = request.headers.get( "X-Request-Id", uuid.uuid4()) return f(req_id, *args, **kwargs) return decorated @app.route("/") @request_id def list_services(req_id): # log w/ ID for wherever you want to trace # app logic
26. upstream appserver { 10.0.0.0:80; } server { listen 80; # Return to client add_header X-Request-ID $request_id; location / { proxy_pass http://appserver; # Pass to app server proxy_set_header X-Request-ID $request_id; } }
27. log_format trace '$remote_addr … $request_id'; server { listen 80; add_header X-Request-ID $request_id; location / { proxy_pass http://app_server; proxy_set_header X-Request-ID $request_id; # Log $request_id access_log /var/log/nginx/access_trace.log trace; } }
28. blackbox
29. metadata propagation
31. Tracing at Scale —
32. four things to think about —
33. four things to think about • What relationships to track —
34. four things to think about • What relationships to track • How to track them —
35. four things to think about • What relationships to track • How to track them • Which sampling approach to take —
36. four things to think about • What relationships to track • How to track them • Which sampling approach to take • How to visualize —
37. what to track
38. Request One Request Two Submitter Flow PoV
39. Request One Request Two Trigger Flow PoV
40. how to track
41. request ID
42. request ID + logical clock
43. request ID + logical clock + previous trace points
44. tradeoffs —
45. tradeoffs • Payload size —
46. tradeoffs • Payload size • Explicit relationships —
47. tradeoffs • Payload size • Explicit relationships • Collate despite lost data —
48. tradeoffs • Payload size • Explicit relationships • Collate despite lost data • Immediate availability —
49. how to sample
50. sampling approaches • Head-based —
51. sampling approaches • Head-based • Tail-based —
52. sampling approaches • Head-based • Tail-based • Unitary —
53. what to visualize
54. gantt chart — GET /home GET /feed GET /profile GET /messages GET /friends Trace ID: de4db33f
55. — request flow graph A call B call C call C call D call E call E reply D reply B reply C reply C reply A reply 2200µs 1500µs 500µs 300µs 400µs 600µs 800µs 500µs 500µs 700µs 500µs 400µs 600µs 100µs
56. — context calling tree A B C C D E
57. keep in mind • What do I want to know? —
58. keep in mind • What do I want to know? • How much can I instrument? —
59. keep in mind • What do I want to know? • How much can I instrument? • How much do I want to know? —
60. suggested for performance —
61. suggested for performance — • Trigger PoV
62. suggested for performance — • Trigger PoV • Head-based sampling
63. suggested for performance — • Trigger PoV • Head-based sampling • Flow graphs
64. Diagnosing —
65. questions to ask — • Batch requests?
66. questions to ask • Batch requests? • Any parallelization opportunities? —
67. questions to ask • Batch requests? • Any parallelization opportunities? • Useful to add/fix caching? —
68. questions to ask • Batch requests? • Any parallelization opportunities? • Useful to add/fix caching? • Frontend resource loading? —
69. questions to ask • Batch requests? • Any parallelization opportunities? • Useful to add/fix caching? • Frontend resource loading? • Chunked or JIT responses? —
70. Frameworks, Systems & Services —
71. OpenTracing
72. OpenCensus
73. self-hosted
74. Zipkin (Twitter) —
75. Zipkin (Twitter) • Out-of-band reporting to remote collector —
76. Zipkin (Twitter) • Out-of-band reporting to remote collector • Report via HTTP, Kafka, and Scribe —
77. Zipkin (Twitter) • Out-of-band reporting to remote collector • Report via HTTP, Kafka, and Scribe • Python libs only support propagation via HTTP —
78. Zipkin (Twitter) • Out-of-band reporting to remote collector • Report via HTTP, Kafka, and Scribe • Python libs only support propagation via HTTP • Limited web UI —
79. def http_transport(span_data): requests.post( "http://zipkinserver:9411/api/v1/spans", data=span_data, headers={"Content-type": "application/x-thrift"}) @app.route("/") def index(): with zipkin_span(service_name="myawesomeapp", span_name="index", # need to write own transport func transport_handler=http_transport, port=app_port, # 0-100 percent sample_rate=100): # do something
80. Jaeger (Uber) —
81. Jaeger (Uber) • Local daemon to collect & report —
82. Jaeger (Uber) • Local daemon to collect & report • Storage support for only Cassandra —
83. Jaeger (Uber) • Local daemon to collect & report • Storage support for only Cassandra • Lacking in documentation —
84. Jaeger (Uber) • Local daemon to collect & report • Storage support for only Cassandra • Lacking in documentation • Cringe-worthy client library —
85. import opentracing as ot config = Config(…) tracer = config.initialize_tracer() @app.route("/") def index(): with ot.tracer.start_span("ASpan") as span: span.log_event("test message", payload={"life": 42}) with ot.tracer.start_span("AChildSpan", child_of=span) as cspan: span.log_event("another test message") # wat time.sleep(2) # yield to IOLoop to flush the spans tracer.close() # flush any buffered spans
86. honorable mentions • AppDash —
87. services
88. Stackdriver Trace (Google) —
89. Stackdriver Trace (Google) • OpenCensus Python library with gRPC support —
90. Stackdriver Trace (Google) • OpenCensus Python library with gRPC support • Forward traces from Zipkin —
91. Stackdriver Trace (Google) • OpenCensus Python library with gRPC support • Forward traces from Zipkin • Storage limitation of 30 days —
92. Stackdriver Trace (Google) • OpenCensus Python library with gRPC support • Forward traces from Zipkin • Storage limitation of 30 days • Recreate graphs per time period —
93. X-Ray (AWS) —
94. X-Ray (AWS) • Supports OpenCensus, not OpenTracing —
95. X-Ray (AWS) • Supports OpenCensus, not OpenTracing • SDK has Python support —
96. X-Ray (AWS) • Supports OpenCensus, not OpenTracing • SDK has Python support • Lots of flexibility with configuring sampling —
97. X-Ray (AWS) • Supports OpenCensus, not OpenTracing • SDK has Python support • Lots of flexibility with configuring sampling • Send metrics from outside AWS environment —
98. X-Ray (AWS) • Supports OpenCensus, not OpenTracing • SDK has Python support • Lots of flexibility with configuring sampling • Send metrics from outside AWS environment • Flow graphs with latency, response %, sample % —
99. honorable mentions • Datadog • New Relic • LightStep • Azure Monitor —
100. TL;DR —
101. tl;dr — • You need this
102. tl;dr — • You need this • Docs are lacking
103. tl;dr — • You need this • Docs are lacking • Language support is improving
104. tl;dr — • You need this • Docs are lacking • Language support is improving • One size fits all approaches
105. tl;dr — • You need this • Docs are lacking • Language support is improving • One size fits all approaches • But there are open specs!
106. Thanks! — Write up: rogue.ly/tracing Lynn Root | SRE | @roguelynn