Slide 1
Slide 1 text
DIY Python Debugger
Johannes Bechberger
mostlynerdless.de
Slide 2
Slide 2 text
No content
Slide 3
Slide 3 text
If debugging is the
process of removing
software bugs, then
programming must be
the process of putting
them in.
— Edsger Dijkstra
“
Slide 4
Slide 4 text
➜ python3 counter.py \
lines counter.py
0
Slide 5
Slide 5 text
➜ python3 counter.py \
lines counter.py
26
Slide 6
Slide 6 text
Let’s look at the code
Slide 7
Slide 7 text
def main():
match cmd := sys.argv[1]:
case "lines":
count = count_code_lines(Path(sys.argv[2]))
print(count)
case "help":
print_help()
case _:
raise ValueError(f"Unknown operation {cmd}")
Slide 8
Slide 8 text
def is_code_line(line: str) -> bool:
return line.isspace() and line.strip().startswith("#")
def count_code_lines(file: Path) -> int:
count = 0
with file.open('r') as f:
for line in f:
if is_code_line(line):
count += 1
return count
Slide 9
Slide 9 text
Any ideas?
Slide 10
Slide 10 text
Debuggers are your friend
Slide 11
Slide 11 text
Who of you used
a debugger before?
Slide 12
Slide 12 text
Debugging
Testing
Profiling
Toolbox
Slide 13
Slide 13 text
No content
Slide 14
Slide 14 text
We could use an existing
debugger...
Slide 15
Slide 15 text
Debuggers are no rocket
science, so ...
Slide 16
Slide 16 text
No content
Slide 17
Slide 17 text
No content
Slide 18
Slide 18 text
Does the interpreter
"know" breakpoints?
Slide 19
Slide 19 text
No.
Slide 20
Slide 20 text
Any ideas?
Slide 21
Slide 21 text
def is_code_line(line: str) -> bool:
return line.isspace() and line.strip().startswith("#
def count_code_lines(file: Path) -> int:
count = 0
with file.open('r') as f:
for line in f:
if is_code_line(line):
count += 1
return count
dbg();
dbg();
dbg();
dbg();
dbg();
dbg();
dbg();
Slide 22
Slide 22 text
def dbg():
if at_breakpoint(file, line):
dbg_shell()
?
dbg(); line
Slide 23
Slide 23 text
sys._getframe
Slide 24
Slide 24 text
sys._getframe
Slide 25
Slide 25 text
sys._getframe
CPython implementation detail
locals(), globals(), sys._getframe(), sys.exc_info(), and
sys.settrace work in PyPy, but they incur a performance
penalty that can be huge by disabling the JIT over the
enclosing JIT scope.
“
– https://www.pypy.org/performance.html
Slide 26
Slide 26 text
main
count_code_lines
is_code_line
dbg sys._getframe(0)
sys._getframe(1)
…
Slide 27
Slide 27 text
main
count_code_lines
is_code_line
dbg sys._getframe(0)
sys._getframe(1)
f_back
f_lineno 6
f_globals ...
f_locals {'line':
'import sys\n'}
f_code.
co_filename
counter.py
Slide 28
Slide 28 text
def dbg():
frame = sys._getframe(1)
line = frame.f_lineno
file = Path(frame.f_code.co_filename).stem
if at_breakpoint(file, line):
dbg_shell()
dbg(); line
Slide 29
Slide 29 text
def dbg():
frame = sys._getframe(1)
line = frame.f_lineno
file = Path(frame.f_code.co_filename).stem
if at_breakpoint(file, line):
dbg_shell(frame)
dbg(); line
Slide 30
Slide 30 text
def dbg():
frame = sys._getframe(1)
line = frame.f_lineno
file = Path(frame.f_code.co_filename).stem
if at_breakpoint(file, line):
dbg_shell(frame)
dbg(); line
Slide 31
Slide 31 text
def dbg():
frame = sys._getframe(1)
line = frame.f_lineno
file = Path(frame.f_code.co_filename).stem
if at_breakpoint(file, line):
dbg_shell(frame)
def at_breakpoint(file: str, line: int) -> bool:
return file == "counter" and line == 6
dbg(); line
Slide 32
Slide 32 text
Demo: counter0.py
Slide 33
Slide 33 text
But how do we
automate this?
Slide 34
Slide 34 text
sys.settrace
Slide 35
Slide 35 text
sys.settrace(handler)
Event = Union['call', 'line', 'return', 'exception', 'opcode']
def handler(frame: FrameType, event: Event, arg):
pass
Slide 36
Slide 36 text
def is_code_line(line: str) -> bool:
return line.isspace() and line.strip().startswith("#")
def count_code_lines(file: Path) -> int:
count = 0
with file.open('r') as f:
for line in f:
if is_code_line(line):
count += 1
return count
handler(frame, 'call', None)
handler(frame, 'call', None)
Slide 37
Slide 37 text
Demo settrace1.py
event: call main
event: call count_code_lines
event: call is_code_line
event: call is_code_line
event: call is_code_line
event: call is_code_line
...
Slide 38
Slide 38 text
sys.settrace(handler)
def inner_handler(frame: FrameType, event: str, arg):
pass
def handler(frame: FrameType, event: Event, arg) \
-> Optional[Callable[[FrameType, Event, Any], None]]:
return inner_handler
Slide 39
Slide 39 text
sys.settrace(handler)
def inner_handler(frame: FrameType, event: Event, arg):
pass
def handler(frame: FrameType, event: Event, arg) \
-> Optional[Callable[[FrameType, Event, Any], None]]:
return inner_handler
Slide 40
Slide 40 text
Demo settrace2.py
event: call is_code_line
inner: line 6
inner: return 6
inner: line 12
inner: line 13
...
Slide 41
Slide 41 text
def dbg():
frame = sys._getframe(1)
line = frame.f_lineno
file = Path(frame.f_code.co_filename).stem
if at_breakpoint(file, line):
dbg_shell(frame)
def at_breakpoint(file: str, line: int) -> bool:
return file == "counter" and line == 6
dbg(); line
Slide 42
Slide 42 text
def inner_handler(frame: FrameType, event: str, arg):
if event != 'line':
return
line = frame.f_lineno
file = Path(frame.f_code.co_filename).stem
if at_breakpoint(file, line):
dbg_shell(frame)
def at_breakpoint(file: str, line: int) -> bool:
return file == "counter" and line == 6
dbg(); line
Slide 43
Slide 43 text
def inner_handler(frame: FrameType, event: str, arg):
if event != 'line':
return
line = frame.f_lineno
file = Path(frame.f_code.co_filename).stem
if at_breakpoint(file, line):
dbg_shell(frame)
def at_breakpoint(file: str, line: int) -> bool:
return file == "counter" and line == 6
dbg(); line
Slide 44
Slide 44 text
Demo settrace3.py
event: call main
event: call count_code_lines
event: call is_code_line
in break point at line 6
>>> line
'import sys\n'
Slide 45
Slide 45 text
def inner_handler(frame: FrameType, event: str, arg):
if event != 'line':
return
line = frame.f_lineno
file = Path(frame.f_code.co_filename).stem
if at_breakpoint(file, line):
dbg_shell(frame)
def at_breakpoint(file: str, line: int) -> bool:
return file == "counter" and line == 6
dbg(); line
make configurable first_line or Breakpoint(file,
line) in current_breakpoints
Slide 46
Slide 46 text
Demo settrace4.py
event: call main
in break point at line 23
>>> br('counter', 6)
>>>
event: call count_code_lines
event: call is_code_line
in break point at line 6
>>> line
'import sys\n'
Slide 47
Slide 47 text
Single Stepping
Slide 48
Slide 48 text
main:25
count_code_lines:13
is_code_line:6
main:25
count_code_lines:12
step out
Just extend at_breakpoint
Slide 49
Slide 49 text
main:25
count_code_lines:12
main:25
count_code_lines:13
step
Just extend at_breakpoint
Slide 50
Slide 50 text
main:25
count_code_lines:12
main:25
count_code_lines:13
step into is_code_line:6
Just extend at_breakpoint
Slide 51
Slide 51 text
Do we get line events
for every function?
Slide 52
Slide 52 text
def is_code_line(line: str) -> bool:
return line.isspace() and line.strip().startswith("#")
def count_code_lines(file: Path) -> int:
count = 0
with file.open('r') as f:
for line in f:
if is_code_line(line):
count += 1
return count
handler(frame, …, None)
add breakpoint
handler(frame, 'call', None)
Slide 53
Slide 53 text
Problems?
Slide 54
Slide 54 text
Performance?
Slide 55
Slide 55 text
Any ideas to improve it?
Slide 56
Slide 56 text
Make at_breakpoint faster
Slide 57
Slide 57 text
Add a new API
Python 3.12
and PEP 669
Slide 58
Slide 58 text
No content
Slide 59
Slide 59 text
# some aliases and constants
mon = sys.monitoring
E = mon.events
TOOL_ID = mon.DEBUGGER_ID
# register the tool
mon.use_tool_id(TOOL_ID, "dbg")
Slide 60
Slide 60 text
# some aliases and constants
mon = sys.monitoring
E = mon.events
TOOL_ID = mon.DEBUGGER_ID
# register the tool
mon.use_tool_id(TOOL_ID, "dbg")
# register callbacks for the events we are interested in
mon.register_callback(TOOL_ID, E.LINE, line_handler)
mon.register_callback(TOOL_ID, E.PY_START, start_handler)
def start_handler(code: CodeType, offset: int):
pass
def line_handler(code: CodeType, line: int):
pass
Slide 61
Slide 61 text
# some aliases and constants
mon = sys.monitoring
E = mon.events
TOOL_ID = mon.DEBUGGER_ID
# register the tool
mon.use_tool_id(TOOL_ID, "dbg")
# register callbacks for the events we are interested in
mon.register_callback(TOOL_ID, E.LINE, line_handler)
mon.register_callback(TOOL_ID, E.PY_START, start_handler)
# enable PY_START event globally
mon.set_events(TOOL_ID, E.PY_START)
# Later
mon.set_local_events(TOOL_ID, code, E.LINE)
Slide 62
Slide 62 text
Globally
set events
Locally
set events
Enabled events per function
Slide 63
Slide 63 text
What's fast?
Slide 64
Slide 64 text
register_callback
get_tool
set_local_events
use_tool_id
set_events
Fast
Rather fast
Slow
The earlier the faster
Slide 65
Slide 65 text
Demo sys_mon1.py
start main
start count_code_lines
start is_code_line
hit line 6
start is_code_line
hit line 6
...
Slide 66
Slide 66 text
Putting it all together
Slide 67
Slide 67 text
Debugging
Testing
Profiling
Toolbox
Slide 68
Slide 68 text
@parttimen3rd on Twitter
parttimenerd on GitHub
mostlynerdless.de
@SweetSapMachine
sapmachine.io