DTrace Your Application

Many know that DTrace is a great tool for profiling and understanding operating system level behavior. Less know that many popular programming languages provide DTrace probes which enable full stack tracing and probing.

Transcript

1. DTrace Your Application! (Not Your Operating System)! ! ! Mark Allen! mrallen1@yahoo.com! @bytemeorg! http://byte-me.org! https://github.com/mrallen1! https://speakerdeck.com/mrallen1!
2. DTrace Your Application! (Not AND Your Operating System)! ! ! Mark Allen! mrallen1@yahoo.com! @bytemeorg! http://byte-me.org! https://github.com/mrallen1! https://speakerdeck.com/mrallen1!
3. DTrace Basics What is DTrace?!
4. DTrace Basics Dynamic Tracing!
5. DTrace Basics "Observability technology"1! 1  DTrace:  Dynamic  Tracing  in  Oracle  Solaris,  Pren(ce  Hall,  2nd  ed,  2012.  
6. DTrace Basics Dynamic versus! print "got here!" time_it(Module, Func, Args) -> S = now(), Result = Module:Func(Args), log_elapsed_time(S, now()), Result.
7. I work at Alert Logic*.! ! I write a lot of Erlang for work, but I have open source projects in Golang, Clojure, bash, Python and Perl.! * We're hiring.!
8. DTrace Basics DTrace vs. profiler!
9. DTrace Basics DTrace vs.and profiler!
10. DTrace Basics DTrace vs. debugger!
11. DTrace Basics DTrace vs.and debugger!
12. DTrace Basics Applica(on   Libraries   System   Calls   Kernel  
13. DTrace Basics It's turtles all the way down...! h?p://flic.kr/p/aQDPrZ  
14. DTrace Basics DTrace enabled OSes:! •  Solaris 10+! •  OpenSolarisIllumos! •  SmartOS (Joyent)! •  OmniOS (OmniTI)! •  FreeBSD! •  Mac OS X!
15. DTrace Basics What about Linux?!
16. DTrace Basics DTrace Terms!
17. DTrace Basics Provider! ! Manages probes in a subsystem!
18. DTrace Basics Provider! ! In our case, this is the application language itself. ! ! (More on this soon)!
19. DTrace Basics Probes! ! DTrace ! instrumentation or! "observables"!
20. DTrace Basics Consumer! ! A user-mode program! that calls into DTrace!
21. D Language Basics D Language Basics!
22. D Language Basics D Language Overview! •  awk-like! •  Define a probe, optional predicate and optional actions in a braced clause! •  Supports BEGIN and END blocks! •  Local variables (this->foo = 42)! •  Aggregate/associative variables (prefixed with @)! •  One liner support in the form! dtrace -n 'probe /predicate/ {action}'
23. D Language Basics Python Example!
24. D Language Basics #!/usr/sbin/dtrace -qFZs function-entry, function-return { /* arg1 is function name */ trace(copyinstr(arg1)) }
25. D Language Basics #!/usr/sbin/dtrace -qFZs function-entry, function-return { /* arg1 is subroutine name */ trace(copyinstr(arg1)) }
26. D Language Basics #!/usr/sbin/dtrace -qFZs function-entry, function-return { /* arg1 is subroutine name */ trace(copyinstr(arg1)) }
27. D Language Basics Perl Example!
28. D Language Basics #!/usr/sbin/dtrace -qZs sub-entry { /* Count sub entries by package and sub name * arg3 = package name (Foo::Bar) * arg0 = subroutine name (do_something) */ @[strjoin(strjoin(copyinstr(arg3),"::"),copyinstr(arg0))] = count() } END { /* Give me top 10 highest counts; throw away rest */ trunc(@, 10) }
29. D Language Basics #!/usr/sbin/dtrace -qZs sub-entry { /* Count sub entries by package and sub name * arg3 = package name (Foo::Bar) * arg0 = subroutine name (do_something) */ @[strjoin(strjoin(copyinstr(arg3),"::"),copyinstr(arg0))] = count() } END { /* Give me top 10 highest counts; throw away rest */ trunc(@, 10) }
30. D Language Basics #!/usr/sbin/dtrace -qZs sub-entry { /* Count sub entries by package and sub name * arg3 = package name (Foo::Bar) * arg0 = subroutine name (do_something) */ @[strjoin(strjoin(copyinstr(arg3),"::"),copyinstr(arg0))] = count() } END { /* Give me top 10 highest counts; throw away rest */ trunc(@, 10) }
31. D Language Basics #!/usr/sbin/dtrace -qZs sub-entry { /* Count sub entries by package and sub name * arg3 = package name (Foo::Bar) * arg0 = subroutine name (do_something) */ @[strjoin(strjoin(copyinstr(arg3),"::"),copyinstr(arg0))] = count() } END { /* Give me top 10 highest counts; throw away rest */ trunc(@, 10) }
32. D Language Basics subcount.d  for  Pod::Perldoc  3.20  
33. D Language Basics Erlang Example!
34. D Language Basics #!/usr/sbin/dtrace -qs erlang*:::process-spawn { printf("pid %s mfa %s\n", copyinstr(arg0), copyinstr(arg1)); } erlang*:::process-exit { printf("pid %s reason %s\n", copyinstr(arg0), copyinstr(arg1)); } erlang*:::process-exit_signal { printf("sender %s -> pid %s reason %s\n", copyinstr(arg0), copyinstr(arg1), copyinstr(arg2)); }
35. D Language Basics erl  shell  input  
36. D Language Basics spawn-­‐exit.d  output  
37. D Language Basics Common aggregation functions! •  avg! •  count! •  lquantize (linear)! •  quantize (log - power of 2)! •  sum! •  min! •  max!
38. DTrace and Perl DTrace and Perl!
39. DTrace and Perl Mac OS X vendor perl and OmniOS vendor perl ! ship with DTrace.!
40. DTrace and Perl Build your own with perlbrew perlbrew install \ perl-5.18.0 \ --as 5.18-dtrace \ -Dusedtrace
41. DTrace and Perl DTrace support by Perl release! Perl release! DTrace probes! 5.10.1! sub-entry, sub-return! 5.14.x! sub-entry and return get package name as an argument! 5.16.x! phase-change! 5.18.0! op-entry, loading-file, loaded- file!
42. DTrace and Perl sub-entry! sub-return! (SUBNAME, FILE, LINE, PACKAGE)
43. DTrace and Perl phase-change! (NEWPHASE, OLDPHASE)
44. DTrace and Perl op-entry! (OPNAME) Fires before op executed. If the debugger is active, fires after debug hooks but still before the op is executed.!
45. DTrace and Perl loading-file (fires before load)! loaded-file (fires after load)! ! (FILENAME) The filename is a path! (e.g., Foo/Bar not Foo::Bar)!
46. DTrace and Perl Make your own DTrace probes in Perl by using ! ! Devel::DTrace::Provider! !
47. DTrace and Python DTrace and Python!
48. DTrace and Python Build your own with brew brew install python \ --with-dtrace
49. DTrace and Python Build your own with pythonz* pythonz install \ --with-dtrace 2.7.5 (* apply https://github.com/saghul/pythonz/pull/43)
50. DTrace and Python
51. DTrace and Python function-entry! function-return! line! (FILE, FUNCTION, LINE)
52. DTrace and Python gc-start! (GENERATION) Similar to gc.collect()!
53. DTrace and Python gc-done! (OBJECTS_COLLECTED)
54. DTrace and Python instance-new-start! instance-new-done! instance-delete-start! instance-delete-done! (CLASS_NAME, FILE)
55. DTrace and Python Make your own DTrace probes in Python by using ! ! http://tmetsch.github.io/python-dtrace/! ! Available on PyPI too!
56. DTrace and Erlang DTrace and Erlang!
57. DTrace and Erlang Build your own with kerl $ KERL_CONFIGURE_OPTIONS="--with- dynamic-trace=dtrace" \ kerl build R16B01 \ r16b01-dtrace
58. DTrace and Erlang Build your own with kerl $ kerl install \ r16b01-dtrace $ ~`whoami`/activate
59. DTrace and Erlang message-send (local)! (SEND_PID, RECV_PID, SIZE, LABEL, PREV_TOKEN_CNT, CURRENT_TOKEN_CNT) PIDs are strings to DTrace, not ints!
60. DTrace and Erlang message-send_remote! (SEND_PID, RECV_PID, SIZE, LABEL, PREV_TOKEN_CNT, CURRENT_TOKEN_CNT) ! PIDs are strings to DTrace, not ints!
61. DTrace and Erlang message-send_remote! (SEND_PID, NODE_NAME, RECV_PID, SIZE, LABEL, PREV_TOKEN_CNT, CURRENT_TOKEN_CNT) ! PIDs are strings to DTrace, not ints!
62. DTrace and Erlang message-queued! message-receive! (RECV_PID, SIZE, Q_LEN, TOKEN, PREV_TOKEN_CNT, CURRENT_TOKEN_CNT) ! PIDs are strings to DTrace, not ints!
63. DTrace and Erlang copy-struct! (SIZE)
64. DTrace and Erlang copy-object! (RECV_PID, SIZE)
65. DTrace and Erlang copy-object! (RECV_PID, SIZE)
66. DTrace and Erlang local-function_entry! global-function_entry! function-return! (PID, MFA, DEPTH)
67. DTrace and Erlang bif-entry! bif-return! nif-entry! nif-return! (PID, MFA)
68. DTrace and Erlang gc_major-start! gc_minor-start! (PID, NEEDED_HEAP_WORDS)
69. DTrace and Erlang gc_major-end! gc_minor-end! (PID,RECLAIMED_SPACE)
70. DTrace and Erlang process-spawn! (PID, MFA)
71. DTrace and Erlang process-exit! (PID, REASON)
72. DTrace and Erlang process-exit! (PID, REASON)
73. DTrace and Erlang process-exit_signal! (SEND_PID, RECV_PID, REASON)
74. DTrace and Erlang process-exit_signal_remote! (SEND_PID, NODE_NAME, RECV_PID, REASON, TOKEN, PREV_TOKEN_CNT, CURR_TOKEN_CNT)
75. DTrace and Erlang process-scheduled! process-hibernate! (PID, MFA)
76. DTrace and Erlang process-unscheduled! (PID)
77. DTrace and Erlang process-port_unblocked! (PID, PORT)
78. DTrace and Erlang process-heap_grow! process-heap_shrink! (PID, OLD_SIZE, NEW_SIZE)
79. DTrace and Erlang dist-monitor! (NODE_NAME, EVENT_TYPE, MONITORED_NODE_NAME, NODE_TYPE, REASON)
80. DTrace and Erlang dist-port_busy! (NODE_NAME, PORT, REMOTE_NODE_NAME, BLOCKED_PID)
81. DTrace and Erlang dist-output! dist-outputv! (NODE_NAME, PORT, REMOTE_NODE, BYTE_CNT)
82. DTrace and Erlang dist-port_not_busy! (NODE_NAME, PORT, REMOTE_NODE)
83. DTrace and Erlang port-open! (PID, PORT_NAME, PORT)
84. DTrace and Erlang port-command! (PID, PORT, PORT_NAME, CMD_TYPE)
85. DTrace and Erlang port-control! (PID, PORT, PORT_NAME, CMD_NUM)
86. DTrace and Erlang port-exit! (PID, PORT, PORT_NAME, REASON)
87. DTrace and Erlang port-connect! (PID, PORT, PORT_NAME, NEW_PID)
88. DTrace and Erlang port-busy! port-not_busy! (PORT)
89. DTrace and Erlang driver-init! (NAME, MAJOR, MINOR, FLAGS)
90. DTrace and Erlang driver-finish! (NAME)
91. DTrace and Erlang driver-start! driver-stop! driver-flush! (PID, NAME, PORT)
92. DTrace and Erlang driver-stop! (PID, NAME, PORT)
93. DTrace and Erlang driver-output! driver-outputv! (PID, PORT, PORT_NAME, BYTES)
94. DTrace and Erlang driver-control! driver-call! (PID, PORT, PORT_NAME, CMD, BYTES)
95. DTrace and Erlang driver-event! driver-ready_input! driver-ready_output! driver-timeout! driver-ready_async! driver-process_exit! driver-call! (PID, PORT, PORT_NAME)
96. DTrace and Erlang driver-stop_select! (DRVR_NAME)
97. DTrace and Erlang aio-pool_add! aio-pool_get! (PID, Q_LEN)
98. DTrace and Erlang efile_drv-entry! (THREAD_ID, DRV_TAG, USER_CMD, CMD_NUM, STR_ARG0, STR_ARG1, INT_ARG0, INT_ARG1, INT_ARG2, INT_ARG3, PORT_ID)
99. DTrace and Erlang efile_dvr-int_entry! efile_dvr-int_return! (THREAD_ID, DRV_TAG, CMD_NUM)
100. DTrace and Erlang efile_dvr-return! (THREAD_ID, DRV_TAG, USER_TAG, CMD_NUM, SUCCESS_BOOL, ERRNO)
101. DTrace and Perl Make your own DTrace probes in Erlang by using ! ! dyntrace:p()! ! Comes with the standard install.!
102. DTrace and Erlang user_trace-n0 .. user_trace-n950! (PID, USER_TAG, INT_ARG0, INT_ARG1, INT_ARG2, INT_ARG3, STR_ARG0, STR_ARG1, STR_ARG2, STR_ARG3)
103. DTrace and Erlang
104. DTrace and Perl Erlang DTrace resources:! •  $ERL_INSTALL/ runtime_tools-*/examples! •  http://www.erlang.org/ doc/apps/runtime_tools/ DTRACE.html! •  http://vimeo.com/33999876!
105. Resources DTrace resources: •  http://dtracehol.com/#Intro •  http://dtrace.org/guide/preface.html •  http://dtracebook.com/index.php/ Languages •  http://www.amazon.com/dp/0132091518 •  Brendan Gregg's talk Friday: http:// www.oscon.com/oscon2013/public/ schedule/detail/29398
106. Thanks! Thank you!!