In Depth PDB

Presented at PyCon 2014 in Montreal, Canada.

Original slides available at http://presentotron.com/nyergler/pdb. Video from the talk available at http://pyvideo.org/video/2673/in-depth-pdb

Transcript

1. In Depth PDB Nathan Yergler CC BY-NC-SA // www.flickr.com/photos/boobook48/5041751802/
2. http://presentotron.com/nyergler/pdb http://github.com/nyergler/in-depth-pdb 2/67
3. PDB: The Python Debugger
4. Look around running code > pfcalc.py(28)push() -> value = int(value_or_operator) (Pdb) p value_or_operator 'abc' 4/67
5. Look around Python's code >>> os.path.join('/Users', 'nathan') > /.../python2.7/posixpath.py(73)join() -> path = a (Pdb) list 68 B def join(a, *p): 69 """Join two or more pathname components, inserting '/' as needed. 70 If any component is an absolute path, all previous path components 71 will be discarded. An empty last part will result in a path that 72 ends with a separator.""" 73 -> path = a 74 for b in p: 75 if b.startswith('/'): 76 path = b 77 elif path == '' or path.endswith('/'): 78 path += b (Pdb) !a '/Users' (Pdb) !p ('nathan',) 5/67
6. Return to the scene of the crime >>> s = make_server('', 8000, pfcalc.rpn_app) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.7/wsgiref/simple_server.py", line 144, in make_server server = server_class((host, port), handler_class) File "/usr/lib/python2.7/SocketServer.py", line 419, in __init__ self.server_bind() File "/usr/lib/python2.7/wsgiref/simple_server.py", line 48, in server_bind HTTPServer.server_bind(self) File "/usr/lib/python2.7/SocketServer.py", line 430, in server_bind self.socket.bind(self.server_address) socket.error: [Errno 98] Address already in use >>> import pdb >>> pdb.pm() > /usr/lib/python2.7/socket.py(224)meth() -> return getattr(self._sock,name)(*args) (Pdb) 6/67
7. Everybody Prints 7/67
8. PDB is Better Explore the state of a running program. Or a dead one. Repeatedly run a program to debug it Build the tools you need to extend it · · · 8/67
9. PDB 101 9/67
10. Explicit Trace Points PDB stops the program immediately after the trace point. import sys def fib(n): """Return the n'th fibonacci number.""" n = int(n) if n <= 1: return 1 return fib(n-1) + fib(n-2) if __name__ == '__main__': import pdb; pdb.set_trace() print (fib(sys.argv[-1])) $ python fibonacci_trace.py 5 > fibonacci_trace.py(12)<module>() -> print (fib(int(sys.argv[-1]))) (Pdb) 10/67
11. next next will execute the current line (including any calls it makes) import sys def fib(n): """Return the n'th fibonacci number.""" n = int(n) if n <= 1: return 1 return fib(n-1) + fib(n-2) if __name__ == '__main__': import pdb; pdb.set_trace() print (fib(sys.argv[-1])) $ python fibonacci_trace.py 5 > fibonacci_trace.py(14)<module>() -> print (fib(sys.argv[-1])) (Pdb) next 8 --Return-- > fibonacci_trace.py(14)<module>()->None -> print (fib(sys.argv[-1])) (Pdb) 11/67
12. step step will stop at the next statement (which may be inside a function call) import sys def fib(n): """Return the n'th fibonacci number.""" n = int(n) if n <= 1: return 1 return fib(n-1) + fib(n-2) if __name__ == '__main__': import pdb; pdb.set_trace() print (fib(sys.argv[-1])) $ python fibonacci_trace.py 5 > fibonacci_trace.py(14)<module>() -> print (fib(sys.argv[-1])) (Pdb) step --Call-- > fibonacci_trace.py(3)fib() -> def fib(n): (Pdb) 12/67
13. cont cont will leave the debugger and let your program continue executing import sys def fib(n): """Return the n'th fibonacci number.""" n = int(n) if n <= 1: return 1 return fib(n-1) + fib(n-2) if __name__ == '__main__': import pdb; pdb.set_trace() print (fib(sys.argv[-1])) $ python fibonacci_trace.py 5 > fibonacci_trace.py(14)<module>() -> print (fib(sys.argv[-1])) (Pdb) step --Call-- > fibonacci_trace.py(3)fib() -> def fib(n): (Pdb) cont 8 13/67
14. PDB 101 Review next executes the current line step executes the current line, stop ASAP cont continue execution Pressing Enter repeats previous command · · · · 14/67
15. Asking for Help (Pdb) help Documented commands (type help <topic>): ======================================== EOF bt cont enable jump pp run unt a c continue exit l q s until alias cl d h list quit step up args clear debug help n r tbreak w b commands disable ignore next restart u whatis break condition down j p return unalias where Miscellaneous help topics: ========================== exec pdb Undocumented commands: ====================== retval rv 15/67
16. Executing Code Under PDB 16/67
17. Running PDB as a Script $ python -m pdb samples/fibonacci.py 5 > fibonacci.py(1)<module>() -> import sys (Pdb) 17/67
18. Running Django under PDB It's entirely possible to run your Django app under PDB: If you want to set a low level breakpoint in your application, that's a good way to get the PDB console before anything happens. $ python -m pdb django runserver --settings=pdbdemo.settings 18/67
19. pdb.run Execute a statement or expression under the debugger. >>> import pdb >>> pdb.run("fib('25')") > <string>(1)<module>() (Pdb) >>> import pdb >>> pdb.runcall(fib, 25) > fibonacci.py(7)fib() -> if n <= 1: (Pdb) 19/67
20. Debugging with PDB 20/67
21. HTTP Maths Consider a small web application that provides a postfix notation calculator. You pass your arguments as path elements, and it applies them to the stack and returns the result. $ curl "http://localhost:8000/2/1/+" The answer is 3 $ curl "http://localhost:8000/2/10/*" The answer is 20 curl "http://localhost:8000/2/10/+/2/*" The answer is 24 21/67
22. HTTP Maths Bugs It's cool, but not great with unexpected input. $ curl http://localhost:8000/2/abc/+/ Traceback (most recent call last): ... File "pfcalc_wsgi.py", line 39, in handle handler.run(self.server.get_app()) File "pfcalc_wsgi.py", line 17, in run self.result = application(self.environ, self.start_response) File "pfcalc.py", line 46, in rpn_app c.push(element) File "pfcalc.py", line 28, in push value = int(value_or_operator) ValueError: invalid literal for int() with base 10: 'abc' 22/67
23. HTTP Maths Bugs We can run it under pdb to see what's actually happening when it blows up. python -m pdb pfcalc.py > pfcalc.py(1)<module>() -> from wsgiref.simple_server import make_server (Pdb) cont Serving on port 8000... $ curl http://localhost:8000/2/abc/+ 23/67
24. HTTP Maths Bugs Now when we hit the bad URL with curl, Python drops into PDB. $ curl http://localhost:8000/2/abc/+/ Traceback (most recent call last): ... File "pfcalc_wsgi.py", line 17, in run self.result = application(self.environ, self.start_response) File "pfcalc.py", line 46, in rpn_app c.push(element) File "pfcalc.py", line 28, in push value = int(value_or_operator) ValueError: invalid literal for int() with base 10: 'abc' Uncaught exception. Entering post mortem debugging Running 'cont' or 'step' will restart the program > pfcalc.py(28)push() -> value = int(value_or_operator) (Pdb) 24/67
25. Inspecting State You can print the value of a variable. You can also look at what arguments were passed to the current function using args. > pfcalc.py(28)push() -> value = int(value_or_operator) (Pdb) p value_or_operator 'abc' (Pdb) args self = <__main__.Calculator object at 0x1047bff10> value_or_operator = abc 25/67
26. Listing Code (Pdb) list 23 value = self.OPERATORS[value_or_operator]( 24 self.state.pop(), 25 self.state.pop(), 26 ) 27 else: 28 -> value = int(value_or_operator) 29 30 self.state.append(value) 31 32 def result(self): 33 26/67
27. Listing Code Python 3.2 added the long list (ll) command, which lets you see the entire function. (Pdb) list 22 value = self.OPERATORS[value_or_operator]( 23 self.state.pop(), 24 self.state.pop(), 25 ) 26 else: 27 -> value = int(value_or_operator) 28 29 self.state.append(value) 30 31 def result(self): 32 (Pdb) ll 19 def push(self, value_or_operator): 20 21 if value_or_operator in self.OPERATORS 22 value = self.OPERATORS[value_or_op 23 self.state.pop(), 24 self.state.pop(), 25 ) 26 else: 27 -> value = int(value_or_operator) 28 29 self.state.append(value) 27/67
28. Pretty Print You can pretty print the value of a variable using the pp command. (Pdb) p self.OPERATORS {'*': <slot wrapper '__mul__' of 'int' objects>, '+': <slot wrapper '__add__' of 'int' objects> (Pdb) pp self.OPERATORS {'*': <slot wrapper '__mul__' of 'int' objects>, '+': <slot wrapper '__add__' of 'int' objects>, '/': <slot wrapper '__div__' of 'int' objects>} 28/67
29. Evaluating Expressions You can also evaluate expressions using the ! command. def add(a, b, c): import pdb; pdb.set_trace() return a + b + c > add.py(5)add() -> return a + b + c (Pdb) b+c *** The specified object '+c' is not a function or was not found along sys.path. (Pdb) !b+c 5 29/67
30. Evaluating Expressions Python 3.2 added the interact command (Pdb) interact *interactive* >>> locals().keys() dict_keys(['make_server', 'self', 'httpd', '__name__', '__builtins__', 'CalculatorWSGIHandler', >>> 30/67
31. Navigating Execution 31/67
32. Navigating Execution Let's consider another example where our calculator isn't so hot. $ curl http://localhost:8000/2/3/+/5 Traceback (most recent call last): ... File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py" self.handle() File "pfcalc_wsgi.py", line 17, in run self.result = application(self.environ, self.start_response) File "pfcalc.py", line 54, in rpn_app "The answer is %d" % (c.result(),), File "pfcalc.py", line 36, in result raise SyntaxError("Invalid expression.") SyntaxError: Invalid expression. > pfcalc.py(36)result() -> raise SyntaxError("Invalid expression.") (Pdb) 32/67
33. Where am I? The where command shows the call stack that got us into this mess. (Pdb) where /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pdb.py(1314)main() -> pdb._runscript(mainpyfile) /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/bdb.py(400)run() -> exec cmd in globals, locals /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py(238)se -> self._handle_request_noblock() pfcalc_wsgi.py(39)handle() -> handler.run(self.server.get_app()) pfcalc_wsgi.py(17)run() -> self.result = application(self.environ, self.start_response) pfcalc.py(54)rpn_app() -> "The answer is %d" % (c.result(),), > pfcalc.py(36)result() -> raise SyntaxError("Invalid expression.") (Pdb) 33/67
34. Navigating the Stack You can use the up command to move one frame up the stack. PDB shows that the current position is now the call to result. (Pdb) up > pfcalc.py(54)rpn_app() -> "The answer is %d" % (c.result(),), 34/67
35. Navigating the Stack Other PDB commands operate in the context of the current position. (Pdb) list 49 headers = [('Content-type', 'text/plain')] 50 51 start_response(status, headers) 52 53 return [ 54 -> "The answer is %d" % (c.result(),), 55 ] 56 57 (Pdb) 35/67
36. Post-Mortem Debugging 36/67
37. Post-Mortem Debugging We've been running the server using the PDB module: $ python -m pdb pfcalc.py 37/67
38. Post-Mortem Debugging Our server starts with the following lines: httpd = make_server('', 8000, rpn_app, server_class=CalculatorServer, handler_class=CalculatorWSGIHandler, ) print "Serving on port 8000..." httpd.serve_forever() 38/67
39. Post-Mortem Debugging The post-mortem debugger acts as a top-level exception handler. httpd = make_server('', 8000, rpn_app, server_class=CalculatorServer, handler_class=CalculatorWSGIHandler, ) print "Serving on port 8000..." httpd.serve_forever() try: httpd = make_server('', 8000, rpn_app, server_class=CalculatorServer, handler_class=CalculatorWSGIHandler, ) print "Serving on port 8000..." httpd.serve_forever() except: import pdb # debug the exception being handled pdb.post_mortem() 39/67
40. Post-Mortem Debugging You can imagine writing this using an explicit set_trace() instead. try: httpd = make_server('', 8000, rpn_app, server_class=CalculatorServer, handler_class=CalculatorWSGIHandler, ) print "Serving on port 8000..." httpd.serve_forever() except: import pdb pdb.set_trace() 40/67
41. Post-Mortem Debugging If we run this version of the server and trigger an error, the difference will be obvious. $ curl http://localhost:8000/2/3/+/5 $ python pf_settrace.py --Return-- > pf_settrace.py(15)<module>()->None -> pdb.set_trace() (Pdb) 41/67
42. Breakpoints 42/67
43. Setting Breakpoints def rpn_app(environ, start_response): c = Calculator() for element in environ['PATH_INFO'][1:].split('/'): c.push(element) status = '200 OK' headers = [('Content-type', 'text/plain')] start_response(status, headers) return [ "The answer is %d" % (c.result(),), ] $ python -m pdb pfcalc.py > pfcalc.py(1)<module>() -> from wsgiref.simple_server import make_server (Pdb) break pfcalc.rpn_app Breakpoint 1 at pfcalc.py:41 43/67
44. Setting Breakpoints You can also give it a filename and a line number. Note that we then exit the debugger by telling it to continue. def rpn_app(environ, start_response): c = Calculator() for element in environ['PATH_INFO'][1:].split('/'): c.push(element) status = '200 OK' headers = [('Content-type', 'text/plain')] start_response(status, headers) return [ "The answer is %d" % (c.result(),), ] $ python -m pdb pfcalc.py > pfcalc.py(1)<module>() -> from wsgiref.simple_server import make_server (Pdb) break pfcalc.py:41 Breakpoint 1 at pfcalc.py:41 (Pdb) cont 44/67
45. Setting Breakpoints (Pdb) help break b(reak) ([file:]lineno | function) [, condition] 45/67
46. Setting Breakpoints If we make a request to our application, we'll see it drop into PDB. $ curl http://localhost:8000/2/3/+ > pfcalc.py(43)rpn_app() -> c = Calculator() (Pdb) n > pfcalc.py(45)rpn_app() (Pdb) !environ['PATH_INFO'] '/2/3/+' 46/67
47. Setting Breakpoints Issuing the break command without any arguments will report on the defined breakpoints: (Pdb) break Num Type Disp Enb Where 1 breakpoint keep yes at pfcalc.py:41 breakpoint already hit 1 times 47/67
48. Setting Breakpoints If you press Ctrl-C, the program will restart since we're running under the PDB module. The breakpoint is still active. (Pdb) break Num Type Disp Enb Where 1 breakpoint keep yes at pfcalc.py:41 breakpoint already hit 4 times 48/67
49. Manipulating Breakpoints The breakpoint number (1) above can be used to control its behavior with the following commands: disable [bpnum] Disables the given breakpoint. The breakpoint remains set, but will not be triggered when the line is encountered. enable [bpnum] Enables the given breakpoint. ignore bpnum [count] Ignore a breakpoint for [count] hits. clear [bpnum] Clears the breakpoints specified. If no breakpoints are specified, prompt to clear all breakpoints. · · · · 49/67
50. Breakpoint Conditions (Pdb) help break b(reak) ([file:]lineno | function) [, condition] 50/67
51. Breakpoint Conditions $ python -m pdb pfcalc.py > pfcalc.py(1)<module>() -> from wsgiref.simple_server import make_server (Pdb) break pfcalc.rpn_app, environ['REQUEST_METHOD'] != 'GET' Breakpoint 1 at pfcalc.py:40 (Pdb) 51/67
52. Breakpoint Conditions $ python -m pdb pfcalc.py > pfcalc.py(1)<module>() -> from wsgiref.simple_server import make_server (Pdb) break pfcalc.rpn_app, environ['REQUEST_METHOD'] != 'GET' Breakpoint 1 at pfcalc.py:40 (Pdb) break Num Type Disp Enb Where 1 breakpoint keep yes at pfcalc.py:40 stop only if environ['REQUEST_METHOD'] != 'GET' (Pdb) cont Serving on port 8000... 52/67
53. Breakpoint Conditions $ python -m pdb pfcalc.py ... Serving on port 8000... 127.0.0.1 - - [12/Mar/2014 12:52:24] "GET /2/3/* HTTP/1.1" 200 15 127.0.0.1 - - [12/Mar/2014 12:52:30] "GET /2/3/+ HTTP/1.1" 200 15 > pfcalc.py(42)rpn_app() -> c = Calculator() (Pdb) environ['REQUEST_METHOD'] 'POST' (Pdb) cont 127.0.0.1 - - [12/Mar/2014 12:52:54] "POST /2/3/+ HTTP/1.1" 200 15 $ curl http://localhost:8000/2/3/\* The answer is 6 $ curl http://localhost:8000/2/3/+ The answer is 5 $ curl http://localhost:8000/2/3/+ -d foo The answer is 5 53/67
54. Extending PDB 54/67
55. Aliases Define aliases for frequently used commands. alias dr pp dir(%1) (Pdb) p self <__main__.Calculator object at 0x7eff1e054790> (Pdb) dr self ['OPERATORS', '__class__', '__dict__', '__hash__', '__init__', '__new__', '__repr__', '__str__', 'push', 'result', 'state'] 55/67
56. Aliases alias loc locals().keys() (Pdb) loc ['status', 'c', 'start_response', 'element', 'headers', 'environ'] 56/67
57. Aliases alias printdict for key, value in %1.items(): print "%s: %s" % (key, value) alias pd printdict Aliases can refer to other aliases Arguments are always passed on · · 57/67
58. Combining Aliases Aliases can reference other aliased commands, allowing you to compose more powerful commands. alias cookies printdict getattr(locals().get('request', %1), 'COOKIES') 58/67
59. Breakpoint Commands PDB can execute debugger commands when a breakpoint is encountered These commands can be anything you normally enter at the (Pdb) prompt The command list ends with either the end command, or any command that resumes execution (step, next, cont) · · · 59/67
60. Breakpoint Commands $ python -m pdb pfcalc.py > pfcalc.py(1)<module>() -> from wsgiref.simple_server import make_server (Pdb) break pfcalc.py:21 Breakpoint 1 at pfcalc.py:21 60/67
61. Breakpoint Commands $ python -m pdb pfcalc.py > pfcalc.py(1)<module>() -> from wsgiref.simple_server import make_server (Pdb) break pfcalc.py:21 Breakpoint 1 at pfcalc.py:21 (Pdb) commands 1 (com) pp self.state (com) pp value_or_operator (com) cont (Pdb) cont Serving on port 8000... 61/67
62. Breakpoint Commands $ python -m pdb pfcalc.py ... Serving on port 8000... [] '2' > pfcalc.py(21)push() -> if value_or_operator in self.OPERATORS: [2] '3' > pfcalc.py(21)push() -> if value_or_operator in self.OPERATORS: [2, 3] '*' > pfcalc.py(21)push() -> if value_or_operator in self.OPERATORS: 127.0.0.1 - - [12/Mar/2014 12:38:42] "GET /2/3/* HTTP/1.1" 200 15 $ curl http://localhost:8000/2/3/\* The answer is 6 62/67
63. .pdbrc # Aliases for looking around alias d pp dir(%1) alias loc locals().keys() alias printdict for key, value in %1.items(): print "%s: %s" % (key, value) alias pd printdict .pdbrc will be loaded from your home directory and current directory Executed line by line in PDB Define aliases, common breakpoints, etc at startup Comments can be included with # · · · · 63/67
64. step-watch Python expressions + PDB commands Credit: http://stackoverflow.com/questions/7668979/how-do-you-watch- a-variable-in-pdb # stepwatch, nextwatch !global __currentframe, __stack; from inspect import currentframe as __currentframe, stack as _ !global __copy; from copy import copy as __copy !global __Pdb; from pdb import Pdb as __Pdb !global __pdb; __pdb = [__framerec[0].f_locals.get("pdb") or __framerec[0].f_locals.get("self") alias _setup_watchpoint !global __key, __dict, __val; __key = '%1'; __dict = __currentframe().f alias _nextwatch_internal next;; !if __dict[__key] == __val: __pdb.cmdqueue.append("_nextwatch_ alias _stepwatch_internal step;; !if __dict[__key] == __val: __pdb.cmdqueue.append("_stepwatch_ alias nextwatch __pdb.cmdqueue.extend(["_setup_watchpoint %1", "_nextwatch_internal"]) alias stepwatch __pdb.cmdqueue.extend(["_setup_watchpoint %1", "_stepwatch_internal"]) 64/67
65. See Also ipdb: Drop-in replacement for PDB that provides syntax highlighting and tab completion rdb: PDB over a socket pudb: Full screen, console debugger pdb++: Overrides PDB with some advanced functionality like watches wdb: PDB over WebSockets pdbtrack is included with modern distributions of python-mode, and allows Emacs to open files as they're debugged by PDB. · · · · · · 65/67
66. So remember... PDB lets you explore your program You can stop in code you can't edit PDB is an extensible tool · · · 66/67
67. Thank You! twitter @nyergler www presentotron.com/nyergler/pdb github github.com/nyergler/in-depth-pdb