Statistical Thread Profiler

@amjithr WRITE YOUR OWN WRITE YOUR OWN STATISTICAL THREAD PROFILER
STATISTICAL THREAD PROFILER PDX PYTHON ▪ Tuesday, February 12, 13

@amjithr Hi  Tuesday, February 12, 13

@amjithr ♥ Tuesday, February 12, 13

@amjithr Tuesday, February 12, 13

@amjithr PERFORMANCE! Tuesday, February 12, 13

@amjithr WHAT’S PROFILING? Tuesday, February 12, 13

@amjithr WHAT’S PROFILING? Measuring Performance Tuesday, February 12, 13

@amjithr cPROFILE Tuesday, February 12, 13

@amjithr $ python -m cProfile lcm.py 7780242 function calls in
3.822 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 3.822 3.822 lcm.py:12(<module>) 1 2.383 2.383 3.822 3.822 lcm.py:12(lcm) 3890120 0.724 0.000 0.724 0.000 {max} 1 0.000 0.000 0.000 0.000 {method 'disable' of 3890119 0.716 0.000 0.716 0.000 {min} Tuesday, February 12, 13

@amjithr  RunSnakeRun Tuesday, February 12, 13

@amjithr $ time python fib.py python fib.py 5.43s user 0.01s
system 99% cpu 5.451 total $ time python -m cProfile fib.py python -m cProfile fib.py 9.63s user 0.04s system 99% cpu 9.670 total Tuesday, February 12, 13

@amjithr You can’t run it in production Tuesday, February 12,
13

@amjithr SLOW PRODUCTION NOT REALISTIC cProﬁle Tuesday, February 12, 13

@amjithr Cool Story, Bro! let’s talk about statistical proﬁlers Tuesday,
February 12, 13

@amjithr STATISTICAL THREAD PROFILER Tuesday, February 12, 13

@amjithr WAT? Tuesday, February 12, 13

@amjithr OVERLY ATTACHED Tuesday, February 12, 13

@amjithr OVERLY ATTACHED Interrupt Inquire Collate Tuesday, February 12, 13

@amjithr OVERLY ATTACHED  Interrupt Inquire Collate Tuesday, February 12,
13

@amjithr OVERLY ATTACHED  ◴ Interrupt Inquire Collate Tuesday, February
12, 13

@amjithr  ◴ Interrupt Inquire Collate STATISTICAL PROFILER Tuesday, February
12, 13

@amjithr SIGNALS THREADS INTERRUPT Tuesday, February 12, 13

@amjithr HTTP://FLIC.KR/P/bE NXH SIGNALS Tuesday, February 12, 13

@amjithr import signal # Use the virtual timer sig =
signal.ITIMER_VIRTUAL # register a handler signal.signal(sig, handler) # 100ms repeating timer signal.setitimer(sig, 0.1, 0.1) SIGNAL Tuesday, February 12, 13

@amjithr PROs CONs Accurate timer Unix only - Ignored in
mod_wsgi - Can’t interrupt C-extensions SIGNAL Tuesday, February 12, 13

@amjithr HTTP://FLIC.KR/P/db ZS THREADS Tuesday, February 12, 13

@amjithr import threading # Create a background thread t =
threading.Thread(target=handler) t.setDaemon(True) # Start the thread t.start() # Sleep for 100ms in handler time.sleep(0.1) THREAD Tuesday, February 12, 13

@amjithr PROs CONs Cross Platform Inaccurate for CPU tasks mod_wsgi
compatible Can’t interrupt C-extensions THREAD Tuesday, February 12, 13

@amjithr INQUIRE Stack frame of every thread Tuesday, February 12,
13

@amjithr >>> import sys # Get stack frames of every
thread >>> frames = sys._current_frames() # Dictionary of { thread_id: frame_obj } >>> frames {4420: <frame object at 0x7fd96c04a650>, 1407: <frame object at 0x7fd96c301730>} INQUIRE Tuesday, February 12, 13

@amjithr function3 function2 function1 FRAMES Tuesday, February 12, 13

@amjithr function3 function2 function1 Fn Name File Name Line No
Arguments Globals Locals ... FRAMES Tuesday, February 12, 13

@amjithr >>> import traceback # Extract useful info from frame
>>> traceback.extract_stack(frame) [('code.py', 88, 'runsource', 'return'), ('code.py', 110, 'runcode', 'print')] EXTRACT Tuesday, February 12, 13

@amjithr COLLATE Tuesday, February 12, 13

@amjithr Default Dictionary ATTEMPT Tuesday, February 12, 13

@amjithr Default Dictionary Uniquely identify a function fdata = (f_name,
file, line) Build a dictionary calls = defaultdict(int) Increment the value when found in frames calls[fn_a] += 1 ATTEMPT Tuesday, February 12, 13

@amjithr Results fn_a=3 fn_b=4 fn_c=5 fn_d=1 { } ATTEMPT Tuesday,
February 12, 13

@amjithr Simple to implement Identiﬁes hot functions No context ATTEMPT
Tuesday, February 12, 13

@amjithr Call Tree ATTEMPT Tuesday, February 12, 13

@amjithr Example T1 Tuesday, February 12, 13

@amjithr Example fn_a fn_b fn_c T1 Tuesday, February 12, 13

@amjithr Example fn_a fn_b fn_c T1 fn_a=1 Tuesday, February 12,
13

@amjithr Example fn_a fn_b fn_c T1 fn_a=1 fn_b=1 Tuesday, February
12, 13

@amjithr Example fn_a fn_b fn_c T1 fn_a=1 fn_b=1 fn_c=1 Tuesday,
February 12, 13

@amjithr T2 fn_a=1 fn_b=1 fn_c=1 Example Tuesday, February 12, 13

@amjithr fn_a fn_b fn_d T2 fn_a=1 fn_b=1 fn_c=1 Example Tuesday,
February 12, 13

@amjithr fn_a fn_b fn_d T2 fn_a=1 fn_b=1 fn_c=1 fn_a=2 Example

@amjithr fn_a fn_b fn_d T2 fn_a=1 fn_b=1 fn_c=1 fn_a=2 fn_b=2
Example Tuesday, February 12, 13

@amjithr fn_a fn_b fn_d T2 fn_a=1 fn_b=1 fn_a=2 fn_b=2 fn_c=1

@amjithr fn_a fn_b fn_d T2 fn_a=1 fn_b=1 fn_a=2 fn_b=2 fn_c=1
fn_d=1 Example Tuesday, February 12, 13

@amjithr fn_a=1 fn_a=2 fn_b=2 fn_c=1 fn_d=1 T3 Example Tuesday, February
12, 13

@amjithr fn_a fn_d fn_a=1 fn_a=2 fn_b=2 fn_c=1 fn_d=1 T3 Example

@amjithr fn_a fn_d fn_a=1 fn_a=2 fn_b=2 fn_c=1 fn_d=1 fn_a=3 T3

@amjithr fn_a fn_d fn_a=1 fn_a=2 fn_a=3 fn_b=2 fn_c=1 fn_d=1 T3

@amjithr fn_a fn_d fn_a=1 fn_a=2 fn_a=3 fn_b=2 fn_c=1 fn_d=1 fn_d=1
T3 Example Tuesday, February 12, 13

@amjithr EXISTING TOOLS Tuesday, February 12, 13

@amjithr NewRelic Threads GUI Tuesday, February 12, 13

@amjithr PLOP Signals D Call Graph https://github.com/bdarnell/plop Tuesday, February 12,
13

@amjithr StatProf Signals CLI http://pypi.python.org/pypi/statprof Tuesday, February 12, 13

@amjithr DEMO Tuesday, February 12, 13

@amjithr SUMMARY Features Limitations Low Overhead No green threads CPython
& PyPy No Jython support Tuesday, February 12, 13

@amjithr PROFILER in PRODUCTION Tuesday, February 12, 13

@amjithr THANK YOU! @amjithr Tuesday, February 12, 13

Statistical Thread Profiler

Statistical Thread Profiler

More Decks by Amjith

Other Decks in Programming

Featured

Transcript