Profiling Erlang Applications using DTrace

Slide 1

Slide 1 text

Proﬁling Applications ! using DTrace! ! ! Mark Allen! mrallen1@yahoo.com! @bytemeorg! https://github.com/mrallen1! https://speakerdeck.com/mrallen1!

Slide 2

Slide 2 text

DTrace Basics What is DTrace?!

Slide 3

Slide 3 text

DTrace Basics Dynamic Tracing!

Slide 4

Slide 4 text

DTrace Basics "Observability technology"1! 1 DTrace: Dynamic Tracing in Oracle Solaris, Pren(ce Hall, 2nd ed, 2012.

Slide 5

Slide 5 text

DTrace Basics Dynamic versus! time_it(Module, Func, Args) -> S = now(), Result = Module:Func(Args), log_elapsed_time(S, now()), Result.

Slide 6

Slide 6 text

h2p://tmblr.co/ZqJL4s18SkeBK

Slide 7

Slide 7 text

I work at Alert Logic*.! ! I write a lot of Erlang for work and sometimes its nice to know why an application is doing that one thing that seems to talk a long time.! * We're hiring.!

Slide 8

Slide 8 text

DTrace Basics DTrace vs. proﬁler!

Slide 9

Slide 9 text

DTrace Basics DTrace vs.and proﬁler!

Slide 10

Slide 10 text

DTrace Basics DTrace vs. debugger!

Slide 11

Slide 11 text

DTrace Basics DTrace vs.and debugger!

Slide 12

Slide 12 text

DTrace Basics Applica(on Libraries System Calls Kernel

Slide 13

Slide 13 text

DTrace Basics It's turtles all the way down...! h2p://ﬂic.kr/p/aQDPrZ

Slide 14

Slide 14 text

DTrace Basics DTrace enabled OSes:! •  Solaris 10+! •  OpenSolarisIllumos! •  SmartOS (Joyent)! •  OmniOS (OmniTI)! •  FreeBSD! •  Mac OS X!

Slide 15

Slide 15 text

DTrace Basics What about Linux?!

Slide 16

Slide 16 text

DTrace Basics It's Complicated.! Sorry.!

Slide 17

Slide 17 text

DTrace Basics DTrace Terms!

Slide 18

Slide 18 text

DTrace Basics Provider! ! Manages probes in a subsystem!

Slide 19

Slide 19 text

DTrace Basics Provider! ! In our case, this is the application language itself. ! ! (More on this soon)!

Slide 20

Slide 20 text

DTrace Basics Probes! ! DTrace ! instrumentation or! "observables"!

Slide 21

Slide 21 text

DTrace Basics Consumer! ! A user-mode program! that calls into DTrace!

Slide 22

Slide 22 text

D Language Basics D Language Basics!

Slide 23

Slide 23 text

D Language Basics D Language Overview! •  awk-like! •  Deﬁne a probe, optional predicate and optional actions in a braced clause! •  Supports BEGIN and END blocks! •  Local variables (this->foo = 42)! •  Aggregate/associative variables (preﬁxed with @)! •  One liner support in the form! dtrace -n 'probe /predicate/ {action}'

Slide 24

Slide 24 text

D Language Basics Example!

Slide 25

Slide 25 text

D Language Basics #!/usr/sbin/dtrace -qs erlang*:::process-spawn { printf("pid %s mfa %s\n", copyinstr(arg0), copyinstr(arg1)); } erlang*:::process-exit { printf("pid %s reason %s\n", copyinstr(arg0), copyinstr(arg1)); } erlang*:::process-exit_signal { printf("sender %s -> pid %s reason %s\n", copyinstr(arg0), copyinstr(arg1), copyinstr(arg2)); }

Slide 26

Slide 26 text

Thanks! Perilous! Live Demo!

Slide 27

Slide 27 text

D Language Basics erl shell input

Slide 28

Slide 28 text

D Language Basics spawn-‐exit.d output

Slide 29

Slide 29 text

D Language Basics Common aggregation functions! •  avg! •  count! •  lquantize (linear)! •  quantize (log - power of 2)! •  sum! •  min! •  max!

Slide 30

Slide 30 text

DTrace and Erlang Enabling DTrace! in Erlang!

Slide 31

Slide 31 text

DTrace and Erlang Build your own with kerl $ KERL_CONFIGURE_OPTIONS="--with- dynamic-trace=dtrace" \ kerl build R16B03 \ r16b03-dtrace

Slide 32

Slide 32 text

DTrace and Erlang Build your own with kerl $ kerl install \ r16b03-dtrace ~ $ ~`whoami`/activate

Slide 33

Slide 33 text

DTrace and Erlang Build your own with erlbrew* $ ERLBREW_CONFIGURE_OPTIONS="-- with-dynamic-trace=dtrace" \ erlbrew install R16B03 * I wrote erlbrew because reasons

Slide 34

Slide 34 text

DTrace and Erlang erltrace: ! DTrace from inside Erlang! ! https://github.com/project-ﬁfo/erltrace !

Slide 35

Slide 35 text

erltrace Write DTrace scripts and process DTrace output from inside Erlang. ! ! Why?! ! •  Push into folsom or ! •  Send an alert to a human that "the thing is happening again"! Might be too expensive depending on environment/load.!

Slide 36

Slide 36 text

Intrepreting DTrace Output Interpreting DTrace Output!

Slide 37

Slide 37 text

D Language Basics erl shell input

Slide 38

Slide 38 text

spawn-‐exit.d output Intrepreting DTrace Output

Slide 39

Slide 39 text

#!/usr/sbin/dtrace -qs erlang*:::process-spawn { printf("pid %s mfa %s\n", copyinstr(arg0), copyinstr(arg1)); } erlang*:::process-exit { printf("pid %s reason %s\n", copyinstr(arg0), copyinstr(arg1)); } erlang*:::process-exit_signal { printf("sender %s -> pid %s reason %s\n", copyinstr(arg0), copyinstr(arg1), copyinstr(arg2)); } Intrepreting DTrace Output

Slide 40

Slide 40 text

DTrace and Erlang process-spawn! (PID, MFA)

Slide 41

Slide 41 text

DTrace and Erlang process-exit! (PID, REASON)

Slide 42

Slide 42 text

DTrace and Erlang process-exit_signal! (SEND_PID, RECV_PID, REASON)

Slide 43

Slide 43 text

spawn-‐exit.d output Intrepreting DTrace Output

Slide 44

Slide 44 text

Thanks! (More) Perilous! Live Demos!

Slide 45

Slide 45 text

DTrace and Erlang Erlang DTrace resources:! •  $ERL_INSTALL/ runtime_tools-*/examples! •  http://www.erlang.org/ doc/apps/runtime_tools/ DTRACE.html! •  http://vimeo.com/33999876!

Slide 46

Slide 46 text

Resources DTrace resources: •  http://dtracehol.com/#Intro •  http://dtrace.org/guide/preface.html •  http://dtracebook.com/index.php/ Languages •  http://www.amazon.com/dp/0132091518 •  European Union RELEASE project whitepaper: http://release- project.softlab.ntua.gr/documents/ D2.2.pdf

Slide 47

Slide 47 text

Thanks! Thank you!!

Slide 48

Slide 48 text

DTrace and Erlang Appendix:! Erlang DTrace Probes!

Slide 49

Slide 49 text

DTrace and Erlang message-send (local)! (SEND_PID, RECV_PID, SIZE, LABEL, PREV_TOKEN_CNT, CURRENT_TOKEN_CNT) PIDs are strings to DTrace, not ints!

Slide 50

Slide 50 text

DTrace and Erlang message-send_remote! (SEND_PID, RECV_PID, SIZE, LABEL, PREV_TOKEN_CNT, CURRENT_TOKEN_CNT) ! PIDs are strings to DTrace, not ints!

Slide 51

Slide 51 text

DTrace and Erlang message-send_remote! (SEND_PID, NODE_NAME, RECV_PID, SIZE, LABEL, PREV_TOKEN_CNT, CURRENT_TOKEN_CNT) ! PIDs are strings to DTrace, not ints!

Slide 52

Slide 52 text

DTrace and Erlang message-queued! message-receive! (RECV_PID, SIZE, Q_LEN, TOKEN, PREV_TOKEN_CNT, CURRENT_TOKEN_CNT) ! PIDs are strings to DTrace, not ints!

Slide 53

Slide 53 text

DTrace and Erlang copy-struct! (SIZE)

Slide 54

Slide 54 text

DTrace and Erlang copy-object! (RECV_PID, SIZE)

Slide 55

Slide 55 text

DTrace and Erlang local-function_entry! global-function_entry! function-return! (PID, MFA, DEPTH)

Slide 56

Slide 56 text

DTrace and Erlang bif-entry! bif-return! nif-entry! nif-return! (PID, MFA)

Slide 57

Slide 57 text

DTrace and Erlang gc_major-start! gc_minor-start! (PID, NEEDED_HEAP_WORDS)

Slide 58

Slide 58 text

DTrace and Erlang gc_major-end! gc_minor-end! (PID,RECLAIMED_SPACE)

Slide 59

Slide 59 text

DTrace and Erlang process-spawn! (PID, MFA)

Slide 60

Slide 60 text

DTrace and Erlang process-exit! (PID, REASON)

Slide 61

Slide 61 text

DTrace and Erlang process-exit_signal! (SEND_PID, RECV_PID, REASON)

Slide 62

Slide 62 text

DTrace and Erlang process-exit_signal_remote! (SEND_PID, NODE_NAME, RECV_PID, REASON, TOKEN, PREV_TOKEN_CNT, CURR_TOKEN_CNT)

Slide 63

Slide 63 text

DTrace and Erlang process-scheduled! process-hibernate! (PID, MFA)

Slide 64

Slide 64 text

DTrace and Erlang process-unscheduled! (PID)

Slide 65

Slide 65 text

DTrace and Erlang process-port_unblocked! (PID, PORT)

Slide 66

Slide 66 text

DTrace and Erlang process-heap_grow! process-heap_shrink! (PID, OLD_SIZE, NEW_SIZE)

Slide 67

Slide 67 text

DTrace and Erlang dist-monitor! (NODE_NAME, EVENT_TYPE, MONITORED_NODE_NAME, NODE_TYPE, REASON)

Slide 68

Slide 68 text

DTrace and Erlang dist-port_busy! (NODE_NAME, PORT, REMOTE_NODE_NAME, BLOCKED_PID)

Slide 69

Slide 69 text

DTrace and Erlang dist-output! dist-outputv! (NODE_NAME, PORT, REMOTE_NODE, BYTE_CNT)

Slide 70

Slide 70 text

DTrace and Erlang dist-port_not_busy! (NODE_NAME, PORT, REMOTE_NODE)

Slide 71

Slide 71 text

DTrace and Erlang port-open! (PID, PORT_NAME, PORT)

Slide 72

Slide 72 text

DTrace and Erlang port-command! (PID, PORT, PORT_NAME, CMD_TYPE)

Slide 73

Slide 73 text

DTrace and Erlang port-control! (PID, PORT, PORT_NAME, CMD_NUM)

Slide 74

Slide 74 text

DTrace and Erlang port-exit! (PID, PORT, PORT_NAME, REASON)

Slide 75

Slide 75 text

DTrace and Erlang port-connect! (PID, PORT, PORT_NAME, NEW_PID)

Slide 76

Slide 76 text

DTrace and Erlang port-busy! port-not_busy! (PORT)

Slide 77

Slide 77 text

DTrace and Erlang driver-init! (NAME, MAJOR, MINOR, FLAGS)

Slide 78

Slide 78 text

DTrace and Erlang driver-ﬁnish! (NAME)

Slide 79

Slide 79 text

DTrace and Erlang driver-start! driver-stop! driver-ﬂush! (PID, NAME, PORT)

Slide 80

Slide 80 text

DTrace and Erlang driver-stop! (PID, NAME, PORT)

Slide 81

Slide 81 text

DTrace and Erlang driver-output! driver-outputv! (PID, PORT, PORT_NAME, BYTES)

Slide 82

Slide 82 text

DTrace and Erlang driver-control! driver-call! (PID, PORT, PORT_NAME, CMD, BYTES)

Slide 83

Slide 83 text

DTrace and Erlang driver-event! driver-ready_input! driver-ready_output! driver-timeout! driver-ready_async! driver-process_exit! driver-call! (PID, PORT, PORT_NAME)

Slide 84

Slide 84 text

DTrace and Erlang driver-stop_select! (DRVR_NAME)

Slide 85

Slide 85 text

DTrace and Erlang aio-pool_add! aio-pool_get! (PID, Q_LEN)

Slide 86

Slide 86 text

DTrace and Erlang eﬁle_drv-entry! (THREAD_ID, DRV_TAG, USER_CMD, CMD_NUM, STR_ARG0, STR_ARG1, INT_ARG0, INT_ARG1, INT_ARG2, INT_ARG3, PORT_ID)

Slide 87

Slide 87 text

DTrace and Erlang eﬁle_dvr-int_entry! eﬁle_dvr-int_return! (THREAD_ID, DRV_TAG, CMD_NUM)

Slide 88

Slide 88 text

DTrace and Erlang eﬁle_dvr-return! (THREAD_ID, DRV_TAG, USER_TAG, CMD_NUM, SUCCESS_BOOL, ERRNO)

Slide 89

Slide 89 text

DTrace and Perl Make your own DTrace probes in Erlang by using ! ! dyntrace:p()! ! Comes with the standard install.!

Slide 90

Slide 90 text

DTrace and Erlang user_trace-n0 .. user_trace-n950! (PID, USER_TAG, INT_ARG0, INT_ARG1, INT_ARG2, INT_ARG3, STR_ARG0, STR_ARG1, STR_ARG2, STR_ARG3)

Slide 91

Slide 91 text

DTrace and Erlang