DTrace USDT for Rust

Remember DTrace? It's back! In Rust form.

Transcript

1. USDT for Rust OxCon 2023 Benjamin Naecker Adam Leventhal
2. OXIDE DTrace… in case you haven’t heard of it 2 • Dynamic instrumentation, systemic in scope, safe in production • “Concise answers to arbitrary questions” • Separates probe logic from instrumentation methodology: ◦ Code written in the “D” language (the good one) ◦ Providers for various aspect of the system e.g.: fbt Kernel function boundary tracing (entry, return) sched Scheduling events (on-cpu, off-cpu, wakeup) io Disk I/O events (start, done) proc Process lifecycle events (create, fork, exec) pid User-land tracing (entry, return, any instruction)
3. OXIDE How to DTrace 3 • On our Helios systems (illumos, macOS*, FreeBSD, available on Linux & Windows) • Do it now (as root or under pfexec): * google: SIP macOS DTrace List all probes # dtrace -l List some probes # dtrace -l -n ‘sched:::on-cpu’ Enable a probe # dtrace -n ‘sched:::on-cpu’ Trace some info # dtrace -n ‘sched:::on-cpu{ trace(execname) }’ Aggregate data # dtrace -n ‘sched:::on-cpu{ @[execname] = count() }’
4. OXIDE Tracing in userland 4 • Can trace arbitrary userland code with the pid provider • dtrace -p $$ -n 'pid$target:::entry' -l • Flexible and powerful, but unstructured • Developers add userland statically-defined tracing (USDT) to parts of the program they know are important or significant to users • Easily add to Rust code with the usdt crate!
5. OXIDE Aside: DTrace in async code 5 • We love async without reservation or caveat** • self->x creates a thread-local variable • Commonly used in DTrace programs for correlating different probes • Async executors like tokio move tasks between threads • Even single-threaded executors multiplex tasks on a single thread • usdt::UniqueId to the rescue! Thread-local Global array indexed by UniqueId
6. OXIDE Let’s go! git clone git@github.com:oxidecomputer/system-simulator …or bring your own 6
7. OXIDE But wait! There’s more! 7 • DTrace Guide • DTrace Bootcamp (2005) • Tips, tricks, and gotchas (2005) • usdt crate • Lots of great blogs by rm, dap, jmc, and many others • OxF on the history of DTrace (~2 hours)