Debugging Python in Production

Due to dynamic and interpreted nature of Python applications it's not hard to employ a limited set of debugging techniques in development and use a variety of debugging tools. However all nasty bugs tend to happen only in production, the very constrained environment without possibilities to edit code, restart, freeze or reconfigure in runtime to suspend services or scare the bug away. This talk is going to cover a set of techniques of debugging, tracing and profiling production Python code in Linux and Solaris environments (with focus on Linux). I'm going to show how to take advantage of GDB, strace, kernel memory, tcpdump, DTrace, SystemTap and all the related software tools to fully introspect what's going on in the Python-powered system.

Video (in Russian): http://www.youtube.com/watch?v=F9FHIghn_Vk

Transcript

1. Debugging Python in Production Vladimir Kirillov @darkproger
2. Debugging Python in Production
3. Typical Production System Audience Test
5. Goals suffer
7. Goals suffer introspection abilities in common Linux production systems are fairly limited (and may vary) miss you, Illumos
8. Goals analyze the runtime environment localize the issue move on to development
9. Constraints • can not scare the bug away • process is unusually stuck • rare race condition • hard to reproduce • only happens in production • limited set of tools
10. diff --git a/dispatch.py b/dispatch.py index 9f2b556..cec4514 100644 --- a/dispatch.py +++ b/dispatch.py @@ -11,9 +11,11 @@ from django.dispatch import receiver from django.core.cache import cache def cachehandler(sender, **kwargs): + log.debug('cachehandler from %s' % sender) obj = kwargs['instance'] for o in (obj._meta.get_all_related_obj_instances(obj) + [obj]): + log.debug('cache invalidation: %s %s' % (o, o.__class__)) cache.delete(o.cachekey()) for key in o.cachekeys(): cache.delete(key) no print
11. @@ -220,7 +220,7 @@ +def safe(f, message='safe action failed'): + try: + return f() + except Exception as e: + import pdb; pdb.set_trace() no breakpoints
12. % time python prog.py % python -mtimeit 'import prog' kernprof line_profiler memory_profiler objgraph? no $REGULAR_PROFILING_STUFF
13. no other desperate practices
15. Stuck Process
16. # date Thu Oct 18 14:58:03 UTC 2012 # ps -ef ... root 26817 26808 0 Jul28 ? 00:00:00 python2.6 periodic_job.py ...
17. # apt-get install gdb python-dbg # gcore 26817 [Thread debugging using libthread_db enabled] warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff2f16b000 0x00007f9b0cbbd130 in __read_nocancel () from /lib/ libpthread.so.0 Saved corefile core.26817 # (printf "set height 0\nt a a bt\nquit\n"; cat /dev/zero) \ | gdb -q /usr/bin/python2.6 core.26817
18. Thread 1 (Thread 26817): #0 0x00007f9b0cbbd130 in __read_nocancel () from /lib/libpthread.so.0 #1 0x00007f9b0c284101 in ?? () from /usr/lib/libcrypto.so.0.9.8 #2 0x00007f9b0c2823a9 in BIO_read () from /usr/lib/libcrypto.so.0.9.8 #3 0x00007f9b0c57a04d in ssl3_read_n () from /usr/lib/libssl.so.0.9.8 #4 0x00007f9b0c57a493 in ssl3_read_bytes () from /usr/lib/libssl.so. 0.9.8 #5 0x00007f9b0c57b562 in ssl3_get_message () from /usr/lib/libssl.so. 0.9.8 #6 0x00007f9b0c57b710 in ssl3_get_finished () from /usr/lib/libssl.so. 0.9.8 #7 0x00007f9b0c5761d0 in ssl3_connect () from /usr/lib/libssl.so.0.9.8 #8 0x00007f9b0c57d183 in ssl23_connect () from /usr/lib/libssl.so.0.9.8 #9 0x000000000050031b in PySSL_SSLdo_handshake (self=0xa12040) at ../ Modules/_ssl.c:460 #10 0x00000000004a7cdf in call_function (f= Frame 0xa12ba0, for file /usr/lib/python2.6/ssl.py, line 279, in do_handshake (self=<SSLSocket(_sslobj=<ssl.SSLContext at remote 0xa12040>) at remote 0x7f9b0cf0b230>), throwflag=<value optimized out>) at ../Python/ceval.c:3734 #11 PyEval_EvalFrameEx (f=
19. Thread 1 (Thread 26817): #0 0x00007f9b0cbbd130 in __read_nocancel () from /lib/libpthread.so.0 #1 0x00007f9b0c284101 in ?? () from /usr/lib/libcrypto.so.0.9.8 #2 0x00007f9b0c2823a9 in BIO_read () from /usr/lib/libcrypto.so.0.9.8 #3 0x00007f9b0c57a04d in ssl3_read_n () from /usr/lib/libssl.so.0.9.8 #4 0x00007f9b0c57a493 in ssl3_read_bytes () from /usr/lib/libssl.so. 0.9.8 #5 0x00007f9b0c57b562 in ssl3_get_message () from /usr/lib/libssl.so. 0.9.8 #6 0x00007f9b0c57b710 in ssl3_get_finished () from /usr/lib/libssl.so. 0.9.8 #7 0x00007f9b0c5761d0 in ssl3_connect () from /usr/lib/libssl.so.0.9.8 #8 0x00007f9b0c57d183 in ssl23_connect () from /usr/lib/libssl.so.0.9.8 #9 0x000000000050031b in PySSL_SSLdo_handshake (self=0xa12040) at ../ Modules/_ssl.c:460 #10 0x00000000004a7cdf in call_function (f= Frame 0xa12ba0, for file /usr/lib/python2.6/ssl.py, line 279, in do_handshake (self=<SSLSocket(_sslobj=<ssl.SSLContext at remote 0xa12040>) at remote 0x7f9b0cf0b230>), throwflag=<value optimized out>) at ../Python/ceval.c:3734 #11 PyEval_EvalFrameEx (f= ^- decodes all python frames!
20. TCP keepalive with SO_KEEPALIVE socket.settimeout ... Solutions
22. CPU Utilization
23. USER time or SYS time? # pid=30684 # ps -p $pid PID TTY TIME CMD 30684 pts/5 26-10:40:14 python2.6 # man proc | awk ' follow && /^\s+\/proc/ {follow=0} /\/stat$/ {follow=1} follow && /[a-z_] %/ {printf "%s ", $1} ' | awk -v RS=" " ' FILENAME==ARGV[2] {print k[FNR], $1} {k[FNR]=$1} ' - /proc/$pid/stat | egrep '^[us]time' # or just use htop :) utime 225060686 # seconds*HZ (x86: =100) stime 3402746
24. Profile! SYS time ptrace consumer •strace -c • count syscall times/duration • helps sometimes
25. Incorrect interpretation of syscall return values example commit c001babe Author: Vladimir Kirillov <proger@wilab.org.ua> $component: correctly handle EOF case in sftp file downloader - avoids spinning workers doing useless i/o and constantly getting EOFs diff --git a/sftp_client.py b/sftp_client.py index deafbeef..0badf00d 100644 --- a/sftp_client.py +++ b/sftp_client.py @@ -66,7 +66,7 @@ def file_sender(sftp_args, path): try: while True: b = f.read(1024 * 300) - if b is None: + if not b: return else: yield b
26. Profile! (not for production) python profilers • line_profiler • cProfile • hotshot generic ad-hoc profilers • gprof • gperftools http://code.google.com/p/gperftools
27. Profile! system-level profilers • oProfile — quite old • perf tracing frameworks • DTrace • not so production in Linux (yet, hopefully) • SystemTap (kprobes/uprobes) • not so production ever
28. perf(1) / CONFIG_PERF_EVENTS # apt-get install linux-tools ## version must match ## the running kernel
29. perf(1) # perf record -p $pid -g sleep 10 # perf script | grep -v '^#' python $pid cpu-clock: ffffffff813356dc retint_careful ([kernel.kallsyms]) 4a5312 update_code_filenames (/usr/bin/python2.6) python $pid cpu-clock: ! 4a6255 PyImport_ReloadModule (/usr/bin/python2.6) python $pid cpu-clock: ffffffff8106068f clocksource_watchdog ([kernel.kallsyms]) ffffffff8103a3a3 run_timer_softirq ([kernel.kallsyms]) ffffffff81035d64 __do_softirq ([kernel.kallsyms]) ffffffff81336eac call_softirq ([kernel.kallsyms]) ffffffff81003e6f do_softirq ([kernel.kallsyms]) ffffffff81035af0 irq_exit ([kernel.kallsyms]) ffffffff8101dec7 smp_apic_timer_interrupt ([kernel.kallsyms]) ffffffff813367c7 apic_timer_interrupt ([kernel.kallsyms]) 450725 tupleiter_next (/usr/bin/python2.6)
30. perf(1) # perf record -p $pid -g sleep 10 # perf script | grep -v '^#' python $pid cpu-clock: ffffffff813356dc retint_careful ([kernel.kallsyms]) 4a5312 update_code_filenames (/usr/bin/python2.6) python $pid cpu-clock: ! 4a6255 PyImport_ReloadModule (/usr/bin/python2.6) python $pid cpu-clock: ffffffff8106068f clocksource_watchdog ([kernel.kallsyms]) ffffffff8103a3a3 run_timer_softirq ([kernel.kallsyms]) ffffffff81035d64 __do_softirq ([kernel.kallsyms]) ffffffff81336eac call_softirq ([kernel.kallsyms]) ffffffff81003e6f do_softirq ([kernel.kallsyms]) ffffffff81035af0 irq_exit ([kernel.kallsyms]) ffffffff8101dec7 smp_apic_timer_interrupt ([kernel.kallsyms]) ffffffff813367c7 apic_timer_interrupt ([kernel.kallsyms]) 450725 tupleiter_next (/usr/bin/python2.6) Where is the user stack trace?
31. -fomit-frame-pointer
32. # gcc -v ... gcc version 4.4.5 (Debian 4.4.5-8) # gcc -Q -O2 --help=optimizers | grep omit -fomit-frame-pointer ! [enabled]
33. -fomit-frame-pointer -fno-omit-frame-pointer 00000000004004d0 <f1>: 0: 83 c7 01 add $0x1,%edi 3: 31 c0 xor %eax,%eax 5: e9 e6 ff ff ff jmpq 4004c0 <f2> 00000000004004d0 <f1>: 0: 55 push %rbp 1: 83 c7 01 add $0x1,%edi 4: 31 c0 xor %eax,%eax 6: 48 89 e5 mov %rsp,%rbp 9: c9 leaveq a: e9 e1 ff ff ff jmpq 4004c0 <f2> int f1(int arg) { ! return f2(arg+1); }
34. -fomit-frame-pointer avoid at all costs
35. # export OPTDEBUGSETTINGS=OPT=-fno-omit-frame-pointer # export CFLAGS=-fno-omit-frame-pointer ... # gunzip -c /usr/share/doc/python2.6/changelog.Debian.gz|head -3 python2.6 (2.6.6+ss1) unstable; urgency=low * -fno-omit-frame-pointer bitch!
36. perf(1) # apt-get install linux-tools # must match the kernel # perf record -p $pid -g sleep 10 # perf script | grep -v '^#' | head -11 python $pid cpu-clock: ! 4a6255 PyImport_ReloadModule (/usr/bin/python2.6) ! 4a95dc w_more (/usr/bin/python2.6) ! 4a95dc w_more (/usr/bin/python2.6) ! 4a95dc w_more (/usr/bin/python2.6) ! 4aa508 w_object (/usr/bin/python2.6) ! 4aa5e6 w_object (/usr/bin/python2.6) ! 4ca88c lp_ulonglong (/usr/bin/python2.6) ! 4caa7c pack_into (/usr/bin/python2.6) ! 41a728 file_repr (/usr/bin/python2.6) ! 7f35a6b64c8d __libc_start_main (/lib/libc-2.11.3.so)
37. perf(1) # apt-get install linux-tools # must match the kernel # perf record -p $pid -g sleep 10 # perf script | grep -v '^#' | head -11 python $pid cpu-clock: ! 4a6255 PyImport_ReloadModule (/usr/bin/python2.6) ! 4a95dc w_more (/usr/bin/python2.6) ! 4a95dc w_more (/usr/bin/python2.6) ! 4a95dc w_more (/usr/bin/python2.6) ! 4aa508 w_object (/usr/bin/python2.6) ! 4aa5e6 w_object (/usr/bin/python2.6) ! 4ca88c lp_ulonglong (/usr/bin/python2.6) ! 4caa7c pack_into (/usr/bin/python2.6) ! 41a728 file_repr (/usr/bin/python2.6) ! 7f35a6b64c8d __libc_start_main (/lib/libc-2.11.3.so) WTF?
38. Verifying the stack trace with GDB # (printf 'set height 0\n t a a bt\nquit\n'; cat /dev/zero) \ | gdb -q /usr/bin/python2.6 $pid Thread 1 (Thread 0x7f15ca79e700 (LWP 30684)): ... #10 0x000000000041a728 in Py_Main (argc=-898936704, argv=<value optimized out>) at ../Modules/main.c:577 #11 0x00007f35a6b64c8d in __libc_start_main () from /lib/ libc.so.6 ...
39. Verifying the stack trace with GDB # (printf 'set height 0\n t a a bt\nquit\n'; cat /dev/zero) \ | gdb -q /usr/bin/python2.6 $pid Thread 1 (Thread 0x7f15ca79e700 (LWP 30684)): ... #10 0x000000000041a728 in Py_Main (argc=-898936704, argv=<value optimized out>) at ../Modules/main.c:577 #11 0x00007f35a6b64c8d in __libc_start_main () from /lib/ libc.so.6 ... 41a728 file_repr (/usr/bin/python2.6) ! 7f35a6b64c8d __libc_start_main (/lib/libc-2.11.3.so)
40. addr2line(1) # perf script | awk ' /^python/ {follow=1; next} /^$/ {exit(0)} follow {gsub("[\(\)]", "", $3); print $3, $1} ' | xargs -n2 addr2line -fe | grep '^\w' PyEval_EvalFrameEx call_function call_function call_function PyEval_EvalCodeEx PyEval_EvalCode run_mod PyRun_SimpleFileExFlags Py_Main
41. addr2line(1) # perf script | awk ' /^python/ {follow=1; next} /^$/ {exit(0)} follow {gsub("[\(\)]", "", $3); print $3, $1} ' | xargs -n2 addr2line -fe | grep '^\w' PyEval_EvalFrameEx call_function call_function call_function PyEval_EvalCodeEx PyEval_EvalCode run_mod PyRun_SimpleFileExFlags Py_Main
42. Flame Graphs! https://github.com/proger/FlameGraph kudos to Brendan Gregg @ Joyent
43. Flame Graphs! CPython is a VM Need to get VM context
44. Getting VM context •Hacking SystemTap? •Hacking perf to look at stack frames? •Good luck on amd64! http://dtrace.org/blogs/eschrock/2004/11/20/ debugging-on-amd64-part-two/
45. Other ways •poor man’s profiler •http://poormansprofiler.org/ •attach GDB to the process at a regular interval •problem: •running backtrace on a python image takes ~7 seconds on a fairly fast machine :) •tons of workarounds possible anyway
46. Other ways •Determine workload by I/O with the world •IPC •Network!
47. tcpdump(1) / LSF (SO_ATTACH_FILTER) # tcpdump -i lo tcp port 8000 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes 15:04:30.853839 IP localhost.34814 > localhost.8000: Flags [S], seq 1999589484, win 32792, options [mss 16396,sackOK,TS val 501850893 ecr 0,nop,wscale 5], length 0 15:04:30.853848 IP localhost.8000 > localhost.34814: Flags [S.], seq 2619299683, ack 1999589485, win 32768, options [mss 16396,sackOK,TS val 501850893 ecr 501850893,nop,wscale 5], length 0 15:04:30.853855 IP localhost.34814 > localhost.8000: Flags [.], ack 1, win 1025, options [nop,nop,TS val 501850893 ecr 501850893], length 0 15:04:30.854268 IP localhost.34814 > localhost.8000: Flags [P.], seq 1:303, ack 1, win 1025, options [nop,nop,TS val 501850893 ecr 501850893], length 302 15:04:30.854291 IP localhost.8000 > localhost.34814: Flags [.], ack 303, win 1058, options [nop,nop,TS val 501850893 ecr 501850893], length 0 15:04:31.445265 IP localhost.8000 > localhost.34814: Flags [P.], seq 1:169, ack 303, win 1058, options [nop,nop,TS val 501850952 ecr 501850893], length 168 15:04:31.445285 IP localhost.34814 > localhost.8000: Flags [.], ack 169, win 1059, options [nop,nop,TS val 501850952 ecr 501850952], length 0 15:04:31.445443 IP localhost.8000 > localhost.34814: Flags [.], seq 169:16553, ack 303, win 1058, options [nop,nop,TS val 501850952 ecr 501850952], length 16384 15:04:31.445460 IP localhost.34814 > localhost.8000: Flags [.], ack 16553, win 1548, options [nop,nop,TS val 501850952 ecr 501850952], length 0 15:04:31.445480 IP localhost.8000 > localhost.34814: Flags [P.], seq 16553:32937, ack 303, win 1058, options [nop,nop,TS val 501850952 ecr 501850952], length 16384
48. tcpdump(1) / LSF (SO_ATTACH_FILTER) # tcpdump -i lo tcp port 8000 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes 15:04:30.853839 IP localhost.34814 > localhost.8000: Flags [S], seq 1999589484, win 32792, options [mss 16396,sackOK,TS val 501850893 ecr 0,nop,wscale 5], length 0 15:04:30.853848 IP localhost.8000 > localhost.34814: Flags [S.], seq 2619299683, ack 1999589485, win 32768, options [mss 16396,sackOK,TS val 501850893 ecr 501850893,nop,wscale 5], length 0 15:04:30.853855 IP localhost.34814 > localhost.8000: Flags [.], ack 1, win 1025, options [nop,nop,TS val 501850893 ecr 501850893], length 0 15:04:30.854268 IP localhost.34814 > localhost.8000: Flags [P.], seq 1:303, ack 1, win 1025, options [nop,nop,TS val 501850893 ecr 501850893], length 302 15:04:30.854291 IP localhost.8000 > localhost.34814: Flags [.], ack 303, win 1058, options [nop,nop,TS val 501850893 ecr 501850893], length 0 15:04:31.445265 IP localhost.8000 > localhost.34814: Flags [P.], seq 1:169, ack 303, win 1058, options [nop,nop,TS val 501850952 ecr 501850893], length 168 15:04:31.445285 IP localhost.34814 > localhost.8000: Flags [.], ack 169, win 1059, options [nop,nop,TS val 501850952 ecr 501850952], length 0 15:04:31.445443 IP localhost.8000 > localhost.34814: Flags [.], seq 169:16553, ack 303, win 1058, options [nop,nop,TS val 501850952 ecr 501850952], length 16384 15:04:31.445460 IP localhost.34814 > localhost.8000: Flags [.], ack 16553, win 1548, options [nop,nop,TS val 501850952 ecr 501850952], length 0 15:04:31.445480 IP localhost.8000 > localhost.34814: Flags [P.], seq 16553:32937, ack 303, win 1058, options [nop,nop,TS val 501850952 ecr 501850952], length 16384 Need to attribute packets to sockets/procs
49. tcpdump(1) / bpf(4) on OS X # tcpdump -k tcp port 80 17:11:43.022867 pid firefox.452 svc BE IP 192.168.1.10.52978 > fa-in-f132.1e100.net.http: Flags [.], ack 15387, win 8109, options [nop,nop,TS val 878633526 ecr 1830256527], length 0
50. lsof(1) # pgrep -f gunicorn \ | xargs -n1 lsof -a -nPi -Rp | grep -v '^COMMAND' python2.6 997 1 root 6u IPv4 47851460 0t0 TCP 127.0.0.1:8000 (LISTEN) python2.6 1005 997 root 6u IPv4 47851460 0t0 TCP 127.0.0.1:8000 (LISTEN) python2.6 1005 997 root 11u IPv4 48003820 0t0 TCP 127.0.0.1:8000->127.0.0.1:55786 (ESTABLISHED) python2.6 1005 997 root 12u IPv4 48003823 0t0 TCP 127.0.0.1:33052->127.0.0.1:5432 (ESTABLISHED) python2.6 1006 997 root 6u IPv4 47851460 0t0 TCP 127.0.0.1:8000 (LISTEN) python2.6 1007 997 root 6u IPv4 47851460 0t0 TCP 127.0.0.1:8000 (LISTEN) python2.6 1008 997 root 6u IPv4 47851460 0t0 TCP 127.0.0.1:8000 (LISTEN) python2.6 1009 997 root 6u IPv4 47851460 0t0 TCP 127.0.0.1:8000 (LISTEN) python2.6 1010 997 root 6u IPv4 47851460 0t0 TCP 127.0.0.1:8000 (LISTEN) python2.6 1011 997 root 6u IPv4 47851460 0t0 TCP 127.0.0.1:8000 (LISTEN) python2.6 1012 997 root 6u IPv4 47851460 0t0 TCP 127.0.0.1:8000 (LISTEN) <- master process ^- worker
51. ss(1) / netlink(7) # ss -npa | grep python2.6 LISTEN 0 128 127.0.0.1:8000 *:* users: (("python2.6",997,6),("python2.6",1005,6),("python2.6", 1006,6),("python2.6",1007,6),("python2.6",1008,6), ("python2.6",1009,6),("python2.6",1010,6),("python2.6", 1011,6),("python2.6",1012,6)) ESTAB 0 0 127.0.0.1:8000 127.0.0.1:57983 users: (("python2.6",1011,17)) ESTAB 0 0 127.0.0.1:35249 127.0.0.1:5432 users: (("python2.6",1011,19)) ^- TCP source port
52. tcpdump(1) + strace(1) / ptrace(2) # tcpdump -w gunicorn.pcap -s0 -i lo tcp port 8000 & # pgrep -f gunicorn \ | xargs -P$(pgrep -cf gunicorn) -tn1 timeout 10 \ strace -eaccept,close -tttTff -o workers -v -p # kill %1
53. tcpdump(1) + strace(1) / ptrace(2) # grep 'accept.*sin_port' workers.$pid 1350569881.935014 accept(6, {sa_family=AF_INET, sin_port=htons(38138), sin_addr=inet_addr("127.0.0.1")}, [16]) = 16 <0.000051> # tcpdump -X -r gunicorn.pcap port 38138 | grep GET !0x0030: 1dea df38 4745 5420 2f61 7069 2f6e 6f64 ... 8GET./api/nod
54. tcpdump(1) # grep 'accept.*sin_port' workers.$pid 1350569881.935014 accept(6, {sa_family=AF_INET, sin_port=htons(38138), sin_addr=inet_addr("127.0.0.1")}, [16]) = 16 <0.000051> # tcpdump -X -r gunicorn.pcap port 38138 | grep GET ! 0x0030: 1dea df38 4745 5420 2f61 7069 2f6e 6f64 ...8GET./api/nod # tcpdump -ttttt -r gunicorn.pcap port 38138 and 'tcp[tcpflags] & (tcp-syn|tcp-fin) != 0' 00:00:00.000000 IP localhost.38138 > localhost.8000: Flags [S], seq 426428352, win 32792, options [mss 16396,sackOK,TS val 501931832 ecr 0,nop,wscale 5], length 0 ... 00:00:01.663210 IP localhost.38138 > localhost.8000: Flags [F.], seq 281, ack 261541, win 1548, options [nop,nop,TS val 501931998 ecr 501931997], length 0 <- request duration (assuming no pipelining/keepalive)
55. more fun with tcpdump(1) poor man’s connection monitor # tcpdump -kttt 'tcp[tcpflags] == tcp-syn' judging application performance using pt-tcp-model (kudos to Baron Schwartz @ Percona) http://www.percona.com/doc/percona-toolkit/2.1/pt-tcp- model.html http://www.percona.com/files/presentations/percona-live/ london-2011/PLUK2011-measuring-scalability-and- performance-with-tcp.pdf
56. A single tool for everything • Want to set a breakpoint • at any step in code • at any high-frequency time interval • Collect some data, repeat • Like a scripted debugger • But faster • But for the whole system
57. It’s been open source and production-ready since 2005
58. DTrace http://dtrace.org
59. DTrace Secret
61. DTrace •Dynamic Tracing Framework •Solaris 10 in 2004 •open source in 2005 •kernel modifications to hot-patch program text at run time •+ D programming language •+ user-space libdtrace •provides observability across the entire software stack •tons of success stories at http://dtrace.org
62. This talk was about suffering without DTrace
63. It deserves another talk at least •Just know it can do Python • https://blogs.oracle.com/binujp/entry/dtrace_provider_for_python • https://blogs.oracle.com/levon/entry/python_and_dtrace_in_build • http://cvs.opensolaris.org/source/xref//jds/spec-files/trunk/patches/ Python-07-dtrace.diff
64. It still is not production on Linux •Paul Fox has been porting it alone for several years https://github.com/dtrace4linux/linux •Oracle picked up recently •quite slow for such a big company though •they’ve taken another route •google site:oracle.com dtrace linux •We have already solved several issues on staging environments with it
65. Alternatives • SystemTap • it just makes you suffer again • and crash your system • sometimes • Use Illumos :) • if you can :/
66. kthxbai! @darkproger more stuff at http://kirillov.im