Python Debugging with PuDB, Charles, and cProfile Christopher Beacham / Lady Red, Senior Engineer @ Hipmunk Hipmunk: [email protected] Personal: [email protected]

I write code, and this is a little bit embarrassing, but…

I write code, and this is a little bit embarrassing, but… My Code Has Bugs

No content

The faster we can find bugs, the faster we can fix them

The faster we can find bugs, the faster we can fix them It’s hard to see what a program is doing!

Debugging tools can make the invisible visible.

PuDB - Interactive visual debugger. Charles Proxy - Web debugging proxy cProfile + others - python profilers and visualization tools Tools we’ll cover:

Stop using Print Statements One is never enough You’ll never find a problem you aren’t specifically looking for You have to remember to delete them

Debugging Process Systematically check your assumptions in a binary search First Test

Debugging Process Systematically check your assumptions in a binary search First Test Second Test

Debugging Process Systematically check your assumptions in a binary search First Test Second Test Third Test

Debugging Process Systematically check your assumptions in a binary search First Test Second Test Third Test FOUND THE BUG

How many of you have used a debugger before?

PuDB PuDB is my favorite debugger. It’s very similar to pdb and ipdb, but it’s VISUAL. It show you everything in scope, your code, and a terminal.

No content

No content

No content

No content

No content

No content

No content

No content

No content

No content

Let’s USE it!

We’re going to use PuDB to debug a fortune webserver written in python (flask)

PuDB: Extra Credit - You can use PuDB for unit tests - both in the test, - the code under test, - and you can step smoothly between them - If you use nose testing, you can drop into Pudb on failures with the nose-pudb package - PuDB will catch exceptions, giving you a chance to inspect the entire scope.

No content

How many of you have used one of these tools? (Dev tools) nettop curl

What is Charles? ● Charles is a web debugging proxy ● It shows the content and statistics about any HTTP/HTTPS traffic that passes through it. ● You can repeat requests, modify, intercept, all the forms of beautiful meddling.

No content

Breaking SSL Charles is the Man in the Middle

Fortune - now with distributed architecture?

No content

No content

No content

No content

No content

No content

No content

No content

(make slide showing editing a request)

No content

No content

No content

No content

No content

No content

No content

No content

No content

Let’s USE it!

Charles - Extra Credit ● It can publish a gist to github of a request/response ● You can view xml/json/protocol buffers in a structured way ● You can get the curl command for a given request ● Use breakpoints or rewrite tool to modify any request in realtime

Charles extra credit - continued ● You can throttle to mimic low-speed connections ● You can blacklist hosts to block connections to them ● It can serve local files from a local folder in response to requests to a server ● You can use Charles “Mirror” tool to record a mirror of a site to disk, and then use Map Local to serve it back up ● … there’s a ton more

Charles is nagware After you install it, it’ll keep bugging you until you buy a license. I think it’s just one developer building it, so yeah, if you find it useful, it’s a good cause to buy a license and support his work. (I’m not being compensated for this talk - I just like this software)

No content

How many of you have had code that was slow?

How many of you have had code that was slow? How many have used profiling to identify why?

Is anyone not sure what a profiler is or how it works?

Profiling Profiling is a different hippopotamus from debugging.

Profiling - Interpreting a profiler output is almost an art.

Profiling - Interpreting a profiler output is almost an art. - You can’t recognize wrong unless you would know what right looks like.

Profiling - Interpreting a profiler output is almost an art. - You can’t recognize wrong unless you would know what right looks like. - A lot of profiling is reading between the lines

Profiling - Always profile before making performance-related improvements.

Profiling - Always profile before making performance-related improvements. - Profiling before making performance-related improvements can keep you from wasting your time

Profiling - Always profile before making performance-related improvements. - Profiling before making performance-related improvements can keep you from wasting your time - Your assumptions about what is taking the most time are often wrong.

Profiling - Always profile before making performance-related improvements. - Profiling before making performance-related improvements can keep you from wasting your time - Your assumptions about what is taking the most time are often wrong. - Speeding up code that is already fast is useless!

Profiling - Always profile before making performance-related improvements. - Profiling before making performance-related improvements can keep you from wasting your time - Your assumptions about what is taking the most time are often wrong. - Speeding up code that is already fast is useless! - You must profile with realistic load / realistic data

The Sad State of Python Profilers I wanted to tell you that there was an awesome profiler tool you should use...

The Sad State of Python Profilers I wanted to tell you that there was an awesome profiler tool you should use...

cProfile! Python has 3 built-in profilers, but cProfile is the most commonly used one. It’s in the standard library. python -m cProfile [-o output_file] [-s sort_order] myscript.py Two output formats - binary and human readable.

“Human readable” output I find it’s best to sort by cumtime (cumulative time) Difficult to distinguish which times are additive, and which times are nested. Gets confusing for anything more than the simplest program 47371645 function calls in 20.013 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.001 0.001 20.013 20.013 simulation.py:3() 2 9.524 4.762 20.006 10.003 simulation.py:24(run_simulation) 2441124 1.842 0.000 2.324 0.000 queue.py:82(enqueue) 2297290 1.804 0.000 2.320 0.000 queue.py:39(enqueue) 2294714 1.369 0.000 1.369 0.000 queue.py:52(dequeue) 18940222 1.260 0.000 1.260 0.000 {method 'random' of '_random.Random' objects} 2438880 1.160 0.000 1.160 0.000 queue.py:95(dequeue) 4738410 0.750 0.000 0.750 0.000 {max} 2297290 0.517 0.000 0.517 0.000 queue.py:28(__init__) 2441124 0.482 0.000 0.482 0.000 queue.py:69(__init__) 2439596 0.347 0.000 0.347 0.000 queue.py:105(is_empty) 2295021 0.342 0.000 0.342 0.000 queue.py:62(is_empty) 2441122 0.300 0.000 0.300 0.000 queue.py:102(size) 2297288 0.282 0.000 0.282 0.000 queue.py:59(size) 9471 0.028 0.000 0.028 0.000 {built-in method now} 1 0.001 0.001 0.004 0.004 random.py:40() 1 0.002 0.002 0.002 0.002 hashlib.py:56() 1 0.001 0.001 0.002 0.002 queue.py:1()

47371645 function calls in 20.013 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.001 0.001 20.013 20.013 simulation.py:3() 2 9.524 4.762 20.006 10.003 simulation.py:24(run_simulation) 2441124 1.842 0.000 2.324 0.000 queue.py:82(enqueue) 2297290 1.804 0.000 2.320 0.000 queue.py:39(enqueue) 2294714 1.369 0.000 1.369 0.000 queue.py:52(dequeue) 18940222 1.260 0.000 1.260 0.000 {method 'random' of '_random.Random' objects} 2438880 1.160 0.000 1.160 0.000 queue.py:95(dequeue) 4738410 0.750 0.000 0.750 0.000 {max}

cProfile Visualizers cProfile can make binary output that can be read by several tools.

pyprof2calltree + qcachegrind

No content

while True: now = datetime.now() ...

No content

After - datetime.now is no more! while True: now = datetime.now() ... while True: if count % 1000 == 0: now = datetime.now() …

Simulation complete, ran 287,768.6 operations per second Simulation complete, ran 419,200.0 operations per second

No content

No content

PuDB Install: pip install pudb Invoke import pudb; pudb.set_trace() PUDB https://documen.tician.de/pudb/ Nose-PUDB (for nose testing) - https://pypi.python.org/pypi/nose-pudb

Charles https://www.charlesproxy.com

Profiling cProfile https://docs.python.org/3/library/profile.html Using cProfile w/ pyprof2calltree and kcachegrind/qcachegrind: https://julien.danjou.info/blog/2015/guide-to-python-profiling-cprofile-concrete-case -carbonara

Happy Hunting!!