dtrace.conf(12): DTrace state of the union

Opening remarks at dtrace.conf(12) in 2012. This talk was videoed, but the video has since been made unavailable. If it helps anyone recover it, the video was at: https://www.youtube.com/watch?v=l_7v7Fn7uMQ

Transcript

1. DTrace dtrace.conf(12) SVP, Engineering bryan@joyent.com Bryan Cantrill @bcantrill
2. Welcome to dtrace.conf! • ~80 participants, representing: • ~35 companies, ~7 operating systems, ~3 language runtimes, ~3 databases, ~2 networking infrastructure companies, ~2 universities • …and 1 industry analyst • Twitter: #dtraceconf • Thank you to our sponsors: Stone Cobra, DEY for illumos, Nexenta, Basho and the Joyent Cloud • Thanks to @substack for the T-shirt design • Huge thank you to @deirdres for organizing dtrace.conf!
3. DTrace Version: 2005-2008 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2005 2006 2007 2008 2009 2010 2011
4. DTrace Version: 2005-2010 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2005 2006 2007 2008 2009 2010 2011
5. DTrace Version: 2005-present 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2005 2006 2007 2008 2009 2010 2011
6. DTrace in the past year • Added a log-linear quantization action (llquantize()) • Added KVM support with vmregs[] variable • Added the CTF-aware print action (ht: @ericschrock) • A bunch of little stuff: • Allowed tracemem() to take a variable-size • Added toupper()/tolower() • Allowed lltostr() to take an optional base • A bunch of bug fixes: • Fixed USDT in the non-global zone from the global zone • Added USDT provider reaping (a long-time issue)
7. DTrace: Some open problems • Implementing (finishing) dynamic translators • Instrumenting hardware-virtualized guests from the host • Support dynamic USDT providers • Integrating with user-level CTF • Supporting user-level postmortem DTrace • Transparent instrumentation of multiple processes • Unifying theme: better instrumentation up-stack
8. Dynamic translators • DTrace uses translators to translate arguments from their implementation to a stable (synthetic) structure • These translators are all static — they are found (and compiled) at compile time by libdtrace • Implementation-specific .d files must therefore be delivered into the DTrace library path • If it must be said, this is a mess: different versions of implementations canʼt co-exist • Static translation also means that having DTrace privs in the non-global zone is insufficient to get to useful (translated) in-kernel structures
9. Dynamic translators, cont. • For as long as we have had translators, we have talked of making them dynamic • Idea: translation definition lives in the binary (either the application or the kernel), and the DIF use of the translator is symbolic — DTrace figures it out at run-time • Difficult in that it requires changes at many levels of the DTrace stack (compiler, in-kernel, app build chain) • Good news: some compiler work has already been done (see use of DIF_OP_XLATE and DIF_OP_XLARG — only generated when -xlate=dynamic!) • Bad news: no in-kernel consumer (yet) — and the compiler work is almost certainly incomplete...
10. Instrumenting VM guests • DTrace is present on several OSs that also support first class hardware virtualization (e.g., KVM) • We have taken a first stab with the vmregs[] variable and exit instrumentation — but there is much more to do • The guest is just executing on host-accessible physical; what guest cooperation do we need to be able to instrument that? • What would the mechanism for guest cooperation be? • Can we meaningfully instrument guests without any guest cooperation at all?
11. Supporting dynamic USDT providers • Chris Andrews (@chrisandrews) developed a very useful technique for defining providers in dynamic languages: the provider DOF is generated dynamically • Chris has done this for node (node-dtrace-provider), Ruby (ruby-dtrace) and Perl (Devel::DTrace::Provider) • This has proven itself in other software, e.g. Joyentʼs Mark Cavage (@mcavage) has used this to great effect in node-restify and ldap.js • Is there something that we can or should do to further generalize this technique or better support it? • What is the overlap with dynamic translators?
12. Integrating with user-level CTF • Currently, DTrace does not consume user-level CTF • When $target is specified, it should in theory be possible to consume these definitions in lieu of or in addition to kernel CTF • How to know the types of arguments to pid-provider probes? (Could a first cut be made without this?) • How should user-level types be scoped? (Double back- tick as in symbols for helpers?) • If done implicitly, should we take a swing at implicit copyin() at the same time?
13. User-level postmortem support • Postmortem support in DTrace — like anonymous tracing (and speculative tracing!) — is one of those features that you donʼt need until you need it… • ...at which point it pretty much saves your butt • Is there interest in bringing this to user-level? • Would involve extending “bufpolicy=ring” when $target is set and $target dumps core to write ring buffer to core file in new section • New section would be consumed via mdbʼs ::dtrace
14. Instrumenting multiple processes • Currently, instrumenting multiple processes requires either explicit enumeration of pid probes or USDT • Might be useful to follow-on-fork with the pid provider and/or specify multiple processes to instrument • There are lots of weird corner cases (e.g., what does $target evaluate to if used in a predicate?) • We have used USDT to largely skirt this issue; have we merely bought ourselves time, or does USDT largely obviate the need for this?
15. On with dtrace.conf!