DIY Python Debugger

Debuggers are indispensable tools for all Python developers, empowering them to conquer bugs and unravel complex systems. But have you ever wondered how they work? Curious about the implementation of features like conditional breakpoints and single stepping? Join me for a talk in which we create our own debugger with conditional breakpoints, single stepping and a Python based debugging shell and learn a lot on debuggers along the way.

Recording at https://youtu.be/zCWjj98Wvg0?t=735
Blog series at https://mostlynerdless.de/blog/tag/lets-create-a-debugger-together/

Transcript

1. DIY Python Debugger Johannes Bechberger mostlynerdless.de
3. If debugging is the process of removing software bugs, then programming must be the process of putting them in. — Edsger Dijkstra “
4. ➜ python3 counter.py \ lines counter.py 0
5. ➜ python3 counter.py \ lines counter.py 26
6. Let’s look at the code
7. def main(): match cmd := sys.argv[1]: case "lines": count = count_code_lines(Path(sys.argv[2])) print(count) case "help": print_help() case _: raise ValueError(f"Unknown operation {cmd}")
8. def is_code_line(line: str) -> bool: return line.isspace() and line.strip().startswith("#") def count_code_lines(file: Path) -> int: count = 0 with file.open('r') as f: for line in f: if is_code_line(line): count += 1 return count
9. Any ideas?
10. Debuggers are your friend
11. Who of you used a debugger before?
12. Debugging Testing Profiling Toolbox
14. We could use an existing debugger...
15. Debuggers are no rocket science, so ...
18. Does the interpreter "know" breakpoints?
19. No.
20. Any ideas?
21. def is_code_line(line: str) -> bool: return line.isspace() and line.strip().startswith("# def count_code_lines(file: Path) -> int: count = 0 with file.open('r') as f: for line in f: if is_code_line(line): count += 1 return count dbg(); dbg(); dbg(); dbg(); dbg(); dbg(); dbg();
22. def dbg(): if at_breakpoint(file, line): dbg_shell() ? dbg(); line
23. sys._getframe
24. sys._getframe
25. sys._getframe CPython implementation detail locals(), globals(), sys._getframe(), sys.exc_info(), and sys.settrace work in PyPy, but they incur a performance penalty that can be huge by disabling the JIT over the enclosing JIT scope. “ – https://www.pypy.org/performance.html
26. main count_code_lines is_code_line dbg sys._getframe(0) sys._getframe(1) …
27. main count_code_lines is_code_line dbg sys._getframe(0) sys._getframe(1) f_back f_lineno 6 f_globals ... f_locals {'line': 'import sys\n'} f_code. co_filename counter.py
28. def dbg(): frame = sys._getframe(1) line = frame.f_lineno file = Path(frame.f_code.co_filename).stem if at_breakpoint(file, line): dbg_shell() dbg(); line
29. def dbg(): frame = sys._getframe(1) line = frame.f_lineno file = Path(frame.f_code.co_filename).stem if at_breakpoint(file, line): dbg_shell(frame) dbg(); line
30. def dbg(): frame = sys._getframe(1) line = frame.f_lineno file = Path(frame.f_code.co_filename).stem if at_breakpoint(file, line): dbg_shell(frame) dbg(); line
31. def dbg(): frame = sys._getframe(1) line = frame.f_lineno file = Path(frame.f_code.co_filename).stem if at_breakpoint(file, line): dbg_shell(frame) def at_breakpoint(file: str, line: int) -> bool: return file == "counter" and line == 6 dbg(); line
32. Demo: counter0.py
33. But how do we automate this?
34. sys.settrace
35. sys.settrace(handler) Event = Union['call', 'line', 'return', 'exception', 'opcode'] def handler(frame: FrameType, event: Event, arg): pass
36. def is_code_line(line: str) -> bool: return line.isspace() and line.strip().startswith("#") def count_code_lines(file: Path) -> int: count = 0 with file.open('r') as f: for line in f: if is_code_line(line): count += 1 return count handler(frame, 'call', None) handler(frame, 'call', None)
37. Demo settrace1.py event: call main event: call count_code_lines event: call is_code_line event: call is_code_line event: call is_code_line event: call is_code_line ...
38. sys.settrace(handler) def inner_handler(frame: FrameType, event: str, arg): pass def handler(frame: FrameType, event: Event, arg) \ -> Optional[Callable[[FrameType, Event, Any], None]]: return inner_handler
39. sys.settrace(handler) def inner_handler(frame: FrameType, event: Event, arg): pass def handler(frame: FrameType, event: Event, arg) \ -> Optional[Callable[[FrameType, Event, Any], None]]: return inner_handler
40. Demo settrace2.py event: call is_code_line inner: line 6 inner: return 6 inner: line 12 inner: line 13 ...
41. def dbg(): frame = sys._getframe(1) line = frame.f_lineno file = Path(frame.f_code.co_filename).stem if at_breakpoint(file, line): dbg_shell(frame) def at_breakpoint(file: str, line: int) -> bool: return file == "counter" and line == 6 dbg(); line
42. def inner_handler(frame: FrameType, event: str, arg): if event != 'line': return line = frame.f_lineno file = Path(frame.f_code.co_filename).stem if at_breakpoint(file, line): dbg_shell(frame) def at_breakpoint(file: str, line: int) -> bool: return file == "counter" and line == 6 dbg(); line
43. def inner_handler(frame: FrameType, event: str, arg): if event != 'line': return line = frame.f_lineno file = Path(frame.f_code.co_filename).stem if at_breakpoint(file, line): dbg_shell(frame) def at_breakpoint(file: str, line: int) -> bool: return file == "counter" and line == 6 dbg(); line
44. Demo settrace3.py event: call main event: call count_code_lines event: call is_code_line in break point at line 6 >>> line 'import sys\n'
45. def inner_handler(frame: FrameType, event: str, arg): if event != 'line': return line = frame.f_lineno file = Path(frame.f_code.co_filename).stem if at_breakpoint(file, line): dbg_shell(frame) def at_breakpoint(file: str, line: int) -> bool: return file == "counter" and line == 6 dbg(); line make configurable first_line or Breakpoint(file, line) in current_breakpoints
46. Demo settrace4.py event: call main in break point at line 23 >>> br('counter', 6) >>> event: call count_code_lines event: call is_code_line in break point at line 6 >>> line 'import sys\n'
47. Single Stepping
48. main:25 count_code_lines:13 is_code_line:6 main:25 count_code_lines:12 step out Just extend at_breakpoint
49. main:25 count_code_lines:12 main:25 count_code_lines:13 step Just extend at_breakpoint
50. main:25 count_code_lines:12 main:25 count_code_lines:13 step into is_code_line:6 Just extend at_breakpoint
51. Do we get line events for every function?
52. def is_code_line(line: str) -> bool: return line.isspace() and line.strip().startswith("#") def count_code_lines(file: Path) -> int: count = 0 with file.open('r') as f: for line in f: if is_code_line(line): count += 1 return count handler(frame, …, None) add breakpoint handler(frame, 'call', None)
53. Problems?
54. Performance?
55. Any ideas to improve it?
56. Make at_breakpoint faster
57. Add a new API Python 3.12 and PEP 669
59. # some aliases and constants mon = sys.monitoring E = mon.events TOOL_ID = mon.DEBUGGER_ID # register the tool mon.use_tool_id(TOOL_ID, "dbg")
60. # some aliases and constants mon = sys.monitoring E = mon.events TOOL_ID = mon.DEBUGGER_ID # register the tool mon.use_tool_id(TOOL_ID, "dbg") # register callbacks for the events we are interested in mon.register_callback(TOOL_ID, E.LINE, line_handler) mon.register_callback(TOOL_ID, E.PY_START, start_handler) def start_handler(code: CodeType, offset: int): pass def line_handler(code: CodeType, line: int): pass
61. # some aliases and constants mon = sys.monitoring E = mon.events TOOL_ID = mon.DEBUGGER_ID # register the tool mon.use_tool_id(TOOL_ID, "dbg") # register callbacks for the events we are interested in mon.register_callback(TOOL_ID, E.LINE, line_handler) mon.register_callback(TOOL_ID, E.PY_START, start_handler) # enable PY_START event globally mon.set_events(TOOL_ID, E.PY_START) # Later mon.set_local_events(TOOL_ID, code, E.LINE)
62. Globally set events Locally set events Enabled events per function
63. What's fast?
64. register_callback get_tool set_local_events use_tool_id set_events Fast Rather fast Slow The earlier the faster
65. Demo sys_mon1.py start main start count_code_lines start is_code_line hit line 6 start is_code_line hit line 6 ...
66. Putting it all together
67. Debugging Testing Profiling Toolbox
68. @parttimen3rd on Twitter parttimenerd on GitHub mostlynerdless.de @SweetSapMachine sapmachine.io