Debugging in Python 3.6: Better, Faster, Stronger

The new frame evaluation API was introduced to CPython in PEP 523 and it allows to specify a per-interpreter function pointer to handle the evaluation of frames. It might not be evident how to use this new feature in everyday life, but it’s quite easy to understand how to build a fast debugger based on it.

In this talk we are going to explain how standard way of debugging in Python works and how a new frame evaluation API may be useful for creating the fast debugger. Also we will consider why such fast debugging was not possible in the previous versions of Python. If someone hasn’t made a final decision to move to Python 3.6 this talk will provide some new reasons to do it.

Transcript

1. Debugging in Python 3.6: Better, Faster, Stronger Elizaveta Shashkova JetBrains
 
 EuroPython 2017
2. Bio • Software developer of PyCharm IDE at JetBrains • Debugger • Saint Petersburg, Russia 2
3. Debugging 3 • Adding print statements • Logging
4. Debugging • Adding print statements • Logging • Special tools: debuggers 4
5. Debugger’s Performance 5 Run Debug Debuggers are 30 times slower
6. Contents • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results 6
7. Contents 7 • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results
8. 8 def tracefunc(frame, event, arg): print(frame.f_lineno, event) return tracefunc sys.settrace(tracefunc) 1 2 3 4 5 6 sys.settrace(tracefunc) - set system tracing function Tracing Function
9. def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() 1 2 3 4 5 6 7 8 9 Tracing Function 9
10. def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() 1 2 3 4 5 6 7 8 9 Tracing Function 10 1 call
11. def foo(): friends = ["Bob", “Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() 1 2 3 4 5 6 7 8 9 Tracing Function 11 1 call 2 line
12. def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() 1 2 3 4 5 6 7 8 9 Tracing Function 12 1 call 2 line 3 line 4 line Hi Bob!
13. def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() 1 2 3 4 5 6 7 8 9 Tracing Function 13 1 call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom!
14. def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() 1 2 3 4 5 6 7 8 9 Tracing Function 14 1 call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom! 5 line 5 return
15. Build Python Debugger 15 • Breakpoints • Stepping
16. Tracing Debugger 16 • Suspend program if breakpoint’s line equals frame.f_lineno • Handle events for stepping
17. Performance def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() 17 1 2 3 4 5 6 7 8 9 1 call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom! 5 line 5 return
18. Example 1 18 1 2 3 4 5 6 7 8 9 def calculate(): sum = 0 for i in range(10 ** 7): sum += i return sum 18
19. Example 1 19 1 2 3 4 5 6 7 8 9 def calculate(): sum = 0 for i in range(10 ** 7): sum += i return sum def tracefunc(frame, event, arg): return tracefunc 19 19
20. Performance 20 6,85 sec 19,81 sec Run Tracing Breakpoints 0,80 sec
21. Performance 21 0,80 sec 6,85 sec 19,81 sec Run Tracing Breakpoints
22. Performance 22 Run Tracing Breakpoints 0,80 sec 6,85 sec 19,81 sec
23. Performance 23 ~ 25 times slower! Run Tracing Breakpoints 0,80 sec 6,85 sec 19,81 sec
24. Problem 24 • Tracing call on every line
25. Contents 25 • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results
26. Contents 26 • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results
27. Python 3.6 27
28. Python 3.6 28 • New frame evaluation API • PEP 523
29. PEP 523 29 • Handle evaluation of frames • Add a new field to code objects
30. Frame Evaluation 30 def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9
31. Frame Evaluation 31 def frame_eval(frame, exc): func_name = frame.f_code.co_name f line_number = frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9
32. Frame Evaluation 32 def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno f print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9
33. Frame Evaluation 33 def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno print(line_number, func_name) f return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9
34. Frame Evaluation 34 def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) f 1 2 3 4 5 6 7 8 9
35. def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) def set_frame_eval(): state = PyThreadState_Get() state.interp.eval_frame = frame_eval 1 2 3 4 5 6 7 8 9 Frame Evaluation 35
36. Example 36 1 2 3 4 5 6 7 8 9 10 11 36 def first(): second() def second(): third() def third(): pass set_frame_eval() first()
37. Example 1 first 4 second 7 third 37 1 2 3 4 5 6 7 8 9 10 11 def first(): second() def second(): third() def third(): pass set_frame_eval() first()
38. Custom Frame Evaluation 38 • It works! • Executed while entering a frame • Access to frame and code object
39. Contents 39 • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results
40. Problem 40 • Tracing call on every line
41. Problem 41 • Tracing call on every line • Remove the tracing function!
42. Replace tracing function with frame evaluation function 42
43. Contents 43 • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results
44. Build Python Debugger 44 • Breakpoints • Stepping
45. Breakpoints 45 • Access to the whole code object
46. Breakpoints 46 • Access to the whole code object • Insert breakpoint’s code into frame’s code
47. Breakpoints 47 1 2 3 4 5 6 7 8 9 def maximum(a, b): if a > b: return a else: return b
48. Breakpoints 48 1 2 3 4 5 6 7 8 9 def maximum(a, b): if a > b: return a # breakpoint else: return b
49. def maximum(a, b): if a > b: return a # breakpoint else: return b Breakpoints breakpoint() 49 1 2 3 4 5 6 7 8 9
50. Breakpoints 50 1 2 3 4 5 6 7 8 9 def maximum(a, b): if a > b: breakpoint() return a # breakpoint else: return b
51. Python Bytecode 51 1 2 3 4 5 6 7 8 9 def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum) 51
52. Python Bytecode 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 52 1 2 3 4 5 6 7 8 9 def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum)
53. Python Bytecode 53 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 1 2 3 4 5 6 7 8 9 def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum) 53
54. Python Bytecode 54 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE f 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 1 2 3 4 5 6 7 8 9 def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum) 54
55. Python Bytecode 55 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 1 2 3 4 5 6 7 8 9 def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum)
56. Python Bytecode 56 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 1 2 3 4 5 6 7 8 9 def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum)
57. Python Bytecode 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 57
58. 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE Python Bytecode 58
59. 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE Python Bytecode 59
60. 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE Python Bytecode 60
61. Bytecode Modification 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE breakpoint() 61
62. 62 • Insert breakpoint’s code • Update arguments and offsets Bytecode Modification
63. 63 • Insert breakpoint’s code • Update arguments and offsets • 200 lines in Python Bytecode Modification
64. 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE Bytecode Modification ?! 64 breakpoint()
65. Breakpoint Bytecode 65 1 2 3 4 5 6 7 8 9 def _stop_at_break(): # a lot of code here def breakpoint(): _stop_at_break() 0 LOAD_GLOBAL 0 2 CALL_FUNCTION 0 4 POP_TOP 6 LOAD_CONST 0 8 RETURN_VALUE
66. Build Python Debugger 66 • Breakpoints • Stepping
67. Stepping 67 • Inserting temporary breakpoint on every line • Use old tracing function
68. Frame evaluation debugger is ready! 68
69. Example 1 1 2 3 4 5 6 7 8 9 def calculate(): sum = 0 for i in range(10 ** 7): sum += i return sum
70. Run Tracing Frame 
 evaluation Example 1 70 0,80 sec 19,81 sec 70
71. Example 1 71 Run Tracing Frame 
 evaluation 0,80 sec 19,81 sec 0,81 sec
72. Example 2 72 1 2 3 4 5 6 7 8 9 def foo(): pass def calculate(): sum = 0 for i in range(10 ** 7): foo() sum += i return sum
73. Example 2 73 Run Tracing Frame 
 evaluation 43,58 sec 37,41 sec 1,73 sec
74. PEP 523 74 • Handle evaluation of frames • Add a new field to code objects
75. PEP 523 75 • Expand PyCodeObject struct • co_extra - “scratch space” for the code object • Mark frames without breakpoints
76. Mark Frames 76 1 2 3 4 5 6 7 8 9 def frame_eval(frame, exc): flag = _PyCode_GetExtra(frame.f_code, index) if flag == NO_BREAKS_IN_FRAME: return _PyEval_EvalFrameDefault(frame, exc)
 
 # check for breakpoints ...
77. Example 2 77 Run Tracing Frame 
 evaluation 1,73 sec 43,58 sec 1,91 sec
78. PEP 523 78 • Handle evaluation of frames • Add a new field to code objects
79. Contents 79 • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results
80. Contents 80 • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results
81. Real Life Example 81
82. Real Life Example 82 • Included into PyCharm 2017.1 • Works in production
83. PyCharm 83 Tracing
 w/o Cython Tracing
 with Cython Frame 
 evaluation 11,59 sec 5,66 sec 0,28 sec
84. Frame evaluation rocks! 84
85. Disadvantages 85 • More complicated • Only with CPython • Only with Python 3.6
86. Frame Evaluation 86 • Let’s move to Python 3.6!
87. Frame Evaluation 87 • Let’s move to Python 3.6! • Let’s find another use cases!
88. Use cases 88 def maximum(a, b): if a > b: return a # breakpoint else: return b breakpoint() 1 2 3 4 5 6 7 8 9 88
89. PEP 523 89 • Pyjion project • JIT for Python
90. Frame Evaluation 90 • Let’s move to Python 3.6! • Let’s find another use cases!
91. Links 91 • Prototype: https://github.com/Elizaveta239/ frame-eval • PyCharm Community Edition source code • bytesinsert on PyPi
92. Questions? 92 @lisa_shashkova • Prototype: https://github.com/Elizaveta239/ frame-eval • PyCharm Community Edition source code • bytesinsert on PyPi