DTrace dtrace.conf(12) SVP, Engineering bryan@joyent.com Bryan Cantrill @bcantrill

Welcome to dtrace.conf! • ~80 participants, representing: • ~35 companies, ~7 operating systems, ~3 language runtimes, ~3 databases, ~2 networking infrastructure companies, ~2 universities • …and 1 industry analyst • Twitter: #dtraceconf • Thank you to our sponsors: Stone Cobra, DEY for illumos, Nexenta, Basho and the Joyent Cloud • Thanks to @substack for the T-shirt design • Huge thank you to @deirdres for organizing dtrace.conf!

DTrace Version: 2005-2008 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2005 2006 2007 2008 2009 2010 2011

DTrace Version: 2005-2010 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2005 2006 2007 2008 2009 2010 2011

DTrace Version: 2005-present 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2005 2006 2007 2008 2009 2010 2011

DTrace in the past year • Added a log-linear quantization action (llquantize()) • Added KVM support with vmregs[] variable • Added the CTF-aware print action (ht: @ericschrock) • A bunch of little stuff: • Allowed tracemem() to take a variable-size • Added toupper()/tolower() • Allowed lltostr() to take an optional base • A bunch of bug fixes: • Fixed USDT in the non-global zone from the global zone • Added USDT provider reaping (a long-time issue)

DTrace: Some open problems • Implementing (finishing) dynamic translators • Instrumenting hardware-virtualized guests from the host • Support dynamic USDT providers • Integrating with user-level CTF • Supporting user-level postmortem DTrace • Transparent instrumentation of multiple processes • Unifying theme: better instrumentation up-stack

Dynamic translators • DTrace uses translators to translate arguments from their implementation to a stable (synthetic) structure • These translators are all static — they are found (and compiled) at compile time by libdtrace • Implementation-specific .d files must therefore be delivered into the DTrace library path • If it must be said, this is a mess: different versions of implementations canʼt co-exist • Static translation also means that having DTrace privs in the non-global zone is insufficient to get to useful (translated) in-kernel structures

Dynamic translators, cont. • For as long as we have had translators, we have talked of making them dynamic • Idea: translation definition lives in the binary (either the application or the kernel), and the DIF use of the translator is symbolic — DTrace figures it out at run-time • Difficult in that it requires changes at many levels of the DTrace stack (compiler, in-kernel, app build chain) • Good news: some compiler work has already been done (see use of DIF_OP_XLATE and DIF_OP_XLARG — only generated when -xlate=dynamic!) • Bad news: no in-kernel consumer (yet) — and the compiler work is almost certainly incomplete...

Instrumenting VM guests • DTrace is present on several OSs that also support first class hardware virtualization (e.g., KVM) • We have taken a first stab with the vmregs[] variable and exit instrumentation — but there is much more to do • The guest is just executing on host-accessible physical; what guest cooperation do we need to be able to instrument that? • What would the mechanism for guest cooperation be? • Can we meaningfully instrument guests without any guest cooperation at all?

Supporting dynamic USDT providers • Chris Andrews (@chrisandrews) developed a very useful technique for defining providers in dynamic languages: the provider DOF is generated dynamically • Chris has done this for node (node-dtrace-provider), Ruby (ruby-dtrace) and Perl (Devel::DTrace::Provider) • This has proven itself in other software, e.g. Joyentʼs Mark Cavage (@mcavage) has used this to great effect in node-restify and ldap.js • Is there something that we can or should do to further generalize this technique or better support it? • What is the overlap with dynamic translators?

Integrating with user-level CTF • Currently, DTrace does not consume user-level CTF • When $target is specified, it should in theory be possible to consume these definitions in lieu of or in addition to kernel CTF • How to know the types of arguments to pid-provider probes? (Could a first cut be made without this?) • How should user-level types be scoped? (Double back- tick as in symbols for helpers?) • If done implicitly, should we take a swing at implicit copyin() at the same time?

User-level postmortem support • Postmortem support in DTrace — like anonymous tracing (and speculative tracing!) — is one of those features that you donʼt need until you need it… • ...at which point it pretty much saves your butt • Is there interest in bringing this to user-level? • Would involve extending “bufpolicy=ring” when $target is set and $target dumps core to write ring buffer to core file in new section • New section would be consumed via mdbʼs ::dtrace

Instrumenting multiple processes • Currently, instrumenting multiple processes requires either explicit enumeration of pid probes or USDT • Might be useful to follow-on-fork with the pid provider and/or specify multiple processes to instrument • There are lots of weird corner cases (e.g., what does $target evaluate to if used in a predicate?) • We have used USDT to largely skirt this issue; have we merely bought ourselves time, or does USDT largely obviate the need for this?

On with dtrace.conf!