Instrumenting Async Runtime

This is a presentation I gave in the Rust.Tokyo conference 2024.

Transcript

1. Instrumenting Async Runtime Rust.Tokyo 2024 Motoyuki Kimura 1
2. • Rust Developer 🦀 • Used to use Scala • A organization member of tokio-rs Hi, I’m Motoyuki(@mox692) 👋 2
3. Agenda • Share my attempt for async runtime tracing ◦ Recap monitoring ecosystem in Async Rust ◦ What is runtime tracing? ◦ How does it works? ◦ Lessons, challenges 3
4. Monitoring is important 4
5. How to achieve a monitoring? • Logging (println!, log! etc…) • Capture a backtrace • System metrics etc … 5
6. What about async Rust? 6
7. … Sometimes, a bit tricky 7
8. 8 Async Task Task1 Task2 Execution
9. 9 Async Task Task1 Task2 Execution .await .await .await
10. 10 Async Task Task1 Task2 .await .await .await Task1 Task1 Task2 Task2
14. Are there any async-aware tools available today? 14
15. • Metrics • Tokio-console • Taskdump 15
16. 16 Runtime Metrics
20. 20 Task dump
21. 21 Task dump
22. Would this be enough? 22
23. 23 Case1: Tasks are not distributed Thread Core
24. 24 Case2: Long running tasks
25. 25 Case2: Long running tasks
26. 26 Case3: Not sure what happens behind the scene 😇
27. 27 • Async runtime tend to be a blackbox • It would be cool if we have a tool that focuses on the event occurred in the runtime
28. Prior Art? 28
31. 31 Goroutine Backtrace
34. 34 It is integrated into the Java Virtual Machine (JVM) and causes almost no performance overhead, so it can be used even in heavily loaded production environments.
35. Can we do something similar with Rust? 35
37. Let’s do a demo to see how it goes 37
38. Demo time 38
39. How does it work? 39
40. High level archtecture 40 Gathering Runtime Event Tokio Runtime Visualize Log File Runtime thread Runtime thread Runtime thread
41. 41 Runtime Event
42. 42 Runtime Event Runtime Parking Run Task
43. 43 Runtime Event Backtrace Task Name Duration
44. Visualize 44
45. Visualize 45
47. Tracing × Perfetto 47
49. 49 Hacking Runtime Internal !
52. 52 Trace Store backtrace
53. Challenges 53
54. Backtrace 54
55. 55 Invoked for every task run!!
56. 56 Backtrace • It is crucial to be able to get backtraces efficiently, • Because the locations where traces are obtained are often hot paths in the runtime 🔥
57. 57 🤔
63. Other Subtleties • Symbolize • When to capture a backtrace? ◦ For now, capture backtraces when a future returns Pending ◦ … but this means that we cannot capture a backtrace if that future complete at first poll 63
64. Trace Log 64
66. 66 Trace related stuff
67. Can we do better? 67
68. 68 Continuous Monitoring with JDK Flight Recorder (JFR)
69. 69 Execution tracer overhaul
70. 70 Trace Log • Should be optimized by e.g., ◦ Using thread local buffer ◦ Using a dedicated trace format
72. 72 worker-trace.md
73. 73 Thank you!