Tracing python applications

As hackers we love to understand how stuff works and how to optimize it. A very good tool to do both is software tracing. During the talk we'll see how tracing tools work and we'll zoom on one particular project called pyflame.

Transcript

1. Tracing python applications @stoitsev
3. “Bottlenecks occur in surprising places, so don't try to second guess and put in a speed hack until you have proven that's where the bottleneck is.” Rob Pike
4. Measure Change
5. Software tracing
6. Deterministic Profilers
8. Example https://pymotw.com/2/sys/tracing.html
9. cProfile https://docs.python.org/2/library/profile.html
11. https://pymotw.com/2/profile/
12. https://pymotw.com/2/profile/
13. High overhead
14. No full call stack information
15. Services not written for profiling
16. Sampling Profilers
17. Python
18. Pyflame https://github.com/uber/pyflame
19. Collect the full Python stack
20. Have low overhead
21. Just works for existing code
22. Can do flame graphs
23. Flame graph
24. http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html
25. http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html
26. main() print() compute_seq() send() compute() get() find_solution() slow_multiplication()
27. main() print() compute_seq() send() compute() get() find_solution() slow_multiplication()
28. main() print() compute_seq() send() compute() get() find_solution() slow_multiplication()
29. https://pymotw.com/2/profile/
30. Demo
32. $ pyflame <PID>
33. script.py:<module>:27;script.py:main:23 122 script.py:<module>:27;script.py:main:22 34 script.py:<module>:27;script.py:main:23;script.py:run:13 11 script.py:<module>:27;script.py:main:23;script.py:run:15 114 script.py:<module>:27;script.py:main:23;script.py:run:17 141 script.py:<module>:27;script.py:main:23;script.py:run:16 95 script.py:<module>:27;script.py:main:23;script.py:run:14 4 script.py:<module>:27;script.py:main:23;script.py:run:16;script.py:square:9;script.py:multiply:5 44 script.py:<module>:27;script.py:main:23;script.py:run:15;script.py:square:9;script.py:multiply:5 41 script.py:<module>:27;script.py:main:23;script.py:run:17;script.py:multiply:5 39 script.py:<module>:27;script.py:main:23;script.py:run:16;script.py:square:9 100 script.py:<module>:27;script.py:main:23;script.py:run:15;script.py:square:9 119
34. pyflame <PID> | ./flamegraph.pl profile.svg From https://github.com/brendangregg/FlameGraph
37. https://github.com/uber/pyflame
38. Thanks!
39. eng.uber.com