USDT for Rust OxCon 2023 Benjamin Naecker Adam Leventhal

OXIDE DTrace… in case you haven’t heard of it 2 ● Dynamic instrumentation, systemic in scope, safe in production ● “Concise answers to arbitrary questions” ● Separates probe logic from instrumentation methodology: ○ Code written in the “D” language (the good one) ○ Providers for various aspect of the system e.g.: fbt Kernel function boundary tracing (entry, return) sched Scheduling events (on-cpu, off-cpu, wakeup) io Disk I/O events (start, done) proc Process lifecycle events (create, fork, exec) pid User-land tracing (entry, return, any instruction)

OXIDE How to DTrace 3 ● On our Helios systems (illumos, macOS*, FreeBSD, available on Linux & Windows) ● Do it now (as root or under pfexec): * google: SIP macOS DTrace List all probes # dtrace -l List some probes # dtrace -l -n ‘sched:::on-cpu’ Enable a probe # dtrace -n ‘sched:::on-cpu’ Trace some info # dtrace -n ‘sched:::on-cpu{ trace(execname) }’ Aggregate data # dtrace -n ‘sched:::on-cpu{ @[execname] = count() }’

OXIDE Tracing in userland 4 ● Can trace arbitrary userland code with the pid provider ● dtrace -p $$ -n 'pid$target:::entry' -l ● Flexible and powerful, but unstructured ● Developers add userland statically-defined tracing (USDT) to parts of the program they know are important or significant to users ● Easily add to Rust code with the usdt crate!

OXIDE Aside: DTrace in async code 5 ● We love async without reservation or caveat** ● self->x creates a thread-local variable ● Commonly used in DTrace programs for correlating different probes ● Async executors like tokio move tasks between threads ● Even single-threaded executors multiplex tasks on a single thread ● usdt::UniqueId to the rescue! Thread-local Global array indexed by UniqueId

OXIDE Let’s go! git clone [email protected]:oxidecomputer/system-simulator …or bring your own 6

OXIDE But wait! There’s more! 7 ● DTrace Guide ● DTrace Bootcamp (2005) ● Tips, tricks, and gotchas (2005) ● usdt crate ● Lots of great blogs by rm, dap, jmc, and many others ● OxF on the history of DTrace (~2 hours)