β Rust Developer π¦
β Used to use Scala
β A organization member of tokio-rs
Hi, Iβm Motoyuki(@mox692) π
2
Slide 3
Slide 3 text
Agenda
β Share my attempt for async runtime tracing
β Recap monitoring ecosystem in Async Rust
β What is runtime tracing?
β How does it works?
β Lessons, challenges
3
Slide 4
Slide 4 text
Monitoring is important
4
Slide 5
Slide 5 text
How to achieve a monitoring?
β Logging (println!, log! etcβ¦)
β Capture a backtrace
β System metrics
etc β¦
5
Are there any async-aware tools available today?
14
Slide 15
Slide 15 text
β Metrics
β Tokio-console
β Taskdump
15
Slide 16
Slide 16 text
16
Runtime Metrics
Slide 17
Slide 17 text
17
Slide 18
Slide 18 text
18
Slide 19
Slide 19 text
19
Slide 20
Slide 20 text
20
Task dump
Slide 21
Slide 21 text
21
Task dump
Slide 22
Slide 22 text
Would this be enough?
22
Slide 23
Slide 23 text
23
Case1: Tasks are not distributed
Thread
Core
Slide 24
Slide 24 text
24
Case2: Long running tasks
Slide 25
Slide 25 text
25
Case2: Long running tasks
Slide 26
Slide 26 text
26
Case3: Not sure what happens behind the scene π
Slide 27
Slide 27 text
27
β Async runtime tend to be a blackbox
β It would be cool if we have a tool that focuses on the event
occurred in the runtime
Slide 28
Slide 28 text
Prior Art?
28
Slide 29
Slide 29 text
29
Slide 30
Slide 30 text
30
Slide 31
Slide 31 text
31
Goroutine
Backtrace
Slide 32
Slide 32 text
32
Slide 33
Slide 33 text
33
Slide 34
Slide 34 text
34
It is integrated into the Java Virtual Machine
(JVM) and causes almost no performance
overhead, so it can be used even in heavily
loaded production environments.
Slide 35
Slide 35 text
Can we do something similar with Rust?
35
Slide 36
Slide 36 text
36
Slide 37
Slide 37 text
Letβs do a demo to see how it goes
37
Slide 38
Slide 38 text
Demo time
38
Slide 39
Slide 39 text
How does it work?
39
Slide 40
Slide 40 text
High level archtecture
40
Gathering Runtime Event
Tokio Runtime
Visualize
Log File
Runtime thread
Runtime thread
Runtime thread
Slide 41
Slide 41 text
41
Runtime Event
Slide 42
Slide 42 text
42
Runtime Event
Runtime Parking
Run Task
Slide 43
Slide 43 text
43
Runtime Event
Backtrace
Task Name
Duration
Slide 44
Slide 44 text
Visualize
44
Slide 45
Slide 45 text
Visualize
45
Slide 46
Slide 46 text
46
Slide 47
Slide 47 text
Tracing Γ Perfetto
47
Slide 48
Slide 48 text
48
Slide 49
Slide 49 text
49
Hacking Runtime Internal !
Slide 50
Slide 50 text
50
Slide 51
Slide 51 text
51
Slide 52
Slide 52 text
52
Trace
Store backtrace
Slide 53
Slide 53 text
Challenges
53
Slide 54
Slide 54 text
Backtrace
54
Slide 55
Slide 55 text
55
Invoked for
every task run!!
Slide 56
Slide 56 text
56
Backtrace
β It is crucial to be able to get backtraces efficiently,
β Because the locations where traces are obtained are often
hot paths in the runtime π₯
Slide 57
Slide 57 text
57
π€
Slide 58
Slide 58 text
58
Slide 59
Slide 59 text
59
Slide 60
Slide 60 text
60
Slide 61
Slide 61 text
61
Slide 62
Slide 62 text
62
Slide 63
Slide 63 text
Other Subtleties
β Symbolize
β When to capture a backtrace?
β For now, capture backtraces when a future returns
Pending
β β¦ but this means that we cannot capture a backtrace if
that future complete at first poll
63
Slide 64
Slide 64 text
Trace Log
64
Slide 65
Slide 65 text
65
Slide 66
Slide 66 text
66
Trace related stuff
Slide 67
Slide 67 text
Can we do better?
67
Slide 68
Slide 68 text
68
Continuous Monitoring with JDK Flight Recorder (JFR)
Slide 69
Slide 69 text
69
Execution tracer overhaul
Slide 70
Slide 70 text
70
Trace Log
β Should be optimized by e.g.,
β Using thread local buffer
β Using a dedicated trace format