Upgrade to Pro — share decks privately, control downloads, hide ads and more …

dtrace.conf(12): DTrace state of the union

Bryan Cantrill
April 07, 2012
24

dtrace.conf(12): DTrace state of the union

Opening remarks at dtrace.conf(12) in 2012. This talk was videoed, but the video has since been made unavailable. If it helps anyone recover it, the video was at: https://www.youtube.com/watch?v=l_7v7Fn7uMQ

Bryan Cantrill

April 07, 2012
Tweet

Transcript

  1. DTrace
    dtrace.conf(12)
    SVP, Engineering
    [email protected]
    Bryan Cantrill
    @bcantrill

    View full-size slide

  2. Welcome to dtrace.conf!
    • ~80 participants, representing:
    • ~35 companies, ~7 operating systems, ~3 language
    runtimes, ~3 databases, ~2 networking infrastructure
    companies, ~2 universities
    • …and 1 industry analyst
    • Twitter: #dtraceconf
    • Thank you to our sponsors: Stone Cobra, DEY for
    illumos, Nexenta, Basho and the Joyent Cloud
    • Thanks to @substack for the T-shirt design
    • Huge thank you to @deirdres for organizing dtrace.conf!

    View full-size slide

  3. DTrace Version: 2005-2008
    1
    1.1
    1.2
    1.3
    1.4
    1.5
    1.6
    1.7
    1.8
    1.9
    2005 2006 2007 2008 2009 2010 2011

    View full-size slide

  4. DTrace Version: 2005-2010
    1
    1.1
    1.2
    1.3
    1.4
    1.5
    1.6
    1.7
    1.8
    1.9
    2005 2006 2007 2008 2009 2010 2011

    View full-size slide

  5. DTrace Version: 2005-present
    1
    1.1
    1.2
    1.3
    1.4
    1.5
    1.6
    1.7
    1.8
    1.9
    2005 2006 2007 2008 2009 2010 2011

    View full-size slide

  6. DTrace in the past year
    • Added a log-linear quantization action (llquantize())
    • Added KVM support with vmregs[] variable
    • Added the CTF-aware print action (ht: @ericschrock)
    • A bunch of little stuff:
    • Allowed tracemem() to take a variable-size
    • Added toupper()/tolower()
    • Allowed lltostr() to take an optional base
    • A bunch of bug fixes:
    • Fixed USDT in the non-global zone from the global zone
    • Added USDT provider reaping (a long-time issue)

    View full-size slide

  7. DTrace: Some open problems
    • Implementing (finishing) dynamic translators
    • Instrumenting hardware-virtualized guests from the host
    • Support dynamic USDT providers
    • Integrating with user-level CTF
    • Supporting user-level postmortem DTrace
    • Transparent instrumentation of multiple processes
    • Unifying theme: better instrumentation up-stack

    View full-size slide

  8. Dynamic translators
    • DTrace uses translators to translate arguments from
    their implementation to a stable (synthetic) structure
    • These translators are all static — they are found (and
    compiled) at compile time by libdtrace
    • Implementation-specific .d files must therefore be
    delivered into the DTrace library path
    • If it must be said, this is a mess: different versions of
    implementations canʼt co-exist
    • Static translation also means that having DTrace privs in
    the non-global zone is insufficient to get to useful
    (translated) in-kernel structures

    View full-size slide

  9. Dynamic translators, cont.
    • For as long as we have had translators, we have talked
    of making them dynamic
    • Idea: translation definition lives in the binary (either the
    application or the kernel), and the DIF use of the
    translator is symbolic — DTrace figures it out at run-time
    • Difficult in that it requires changes at many levels of the
    DTrace stack (compiler, in-kernel, app build chain)
    • Good news: some compiler work has already been done
    (see use of DIF_OP_XLATE and DIF_OP_XLARG —
    only generated when -xlate=dynamic!)
    • Bad news: no in-kernel consumer (yet) — and the
    compiler work is almost certainly incomplete...

    View full-size slide

  10. Instrumenting VM guests
    • DTrace is present on several OSs that also support first
    class hardware virtualization (e.g., KVM)
    • We have taken a first stab with the vmregs[] variable
    and exit instrumentation — but there is much more to do
    • The guest is just executing on host-accessible physical;
    what guest cooperation do we need to be able to
    instrument that?
    • What would the mechanism for guest cooperation be?
    • Can we meaningfully instrument guests without any
    guest cooperation at all?

    View full-size slide

  11. Supporting dynamic USDT providers
    • Chris Andrews (@chrisandrews) developed a very
    useful technique for defining providers in dynamic
    languages: the provider DOF is generated dynamically
    • Chris has done this for node (node-dtrace-provider),
    Ruby (ruby-dtrace) and Perl (Devel::DTrace::Provider)
    • This has proven itself in other software, e.g. Joyentʼs
    Mark Cavage (@mcavage) has used this to great effect
    in node-restify and ldap.js
    • Is there something that we can or should do to further
    generalize this technique or better support it?
    • What is the overlap with dynamic translators?

    View full-size slide

  12. Integrating with user-level CTF
    • Currently, DTrace does not consume user-level CTF
    • When $target is specified, it should in theory be possible
    to consume these definitions in lieu of or in addition to
    kernel CTF
    • How to know the types of arguments to pid-provider
    probes? (Could a first cut be made without this?)
    • How should user-level types be scoped? (Double back-
    tick as in symbols for helpers?)
    • If done implicitly, should we take a swing at implicit
    copyin() at the same time?

    View full-size slide

  13. User-level postmortem support
    • Postmortem support in DTrace — like anonymous
    tracing (and speculative tracing!) — is one of those
    features that you donʼt need until you need it…
    • ...at which point it pretty much saves your butt
    • Is there interest in bringing this to user-level?
    • Would involve extending “bufpolicy=ring” when $target is
    set and $target dumps core to write ring buffer to core
    file in new section
    • New section would be consumed via mdbʼs ::dtrace

    View full-size slide

  14. Instrumenting multiple processes
    • Currently, instrumenting multiple processes requires
    either explicit enumeration of pid probes or USDT
    • Might be useful to follow-on-fork with the pid provider
    and/or specify multiple processes to instrument
    • There are lots of weird corner cases (e.g., what does
    $target evaluate to if used in a predicate?)
    • We have used USDT to largely skirt this issue; have we
    merely bought ourselves time, or does USDT largely
    obviate the need for this?

    View full-size slide

  15. On with dtrace.conf!

    View full-size slide