Debugging in Python 3.6: Better, Faster, Stronger (PyCon US 2017)

Python 3.6 was released in December of 2016 and it has a lot of new cool features. Some of them are quite easy for using: a developer can read, for example, about f-strings and they can start using them in their programs as soon as possible. But sometimes features are not so evident, and a new frame evaluation API is one of them.
The new frame evaluation API was introduced to CPython in PEP 523 and it allows to specify a per-interpreter function pointer to handle the evaluation of frames. It might not be evident how to use this new feature in everyday life, but it’s quite easy to understand how to build a fast debugger based on it.
In this talk we are going to explain how standard way of debugging in Python works and how a new frame evaluation API may be useful for creating the fast debugger. Also we will consider why such fast debugging was not possible in the previous versions of Python. If someone hasn’t made a final decision to move to Python 3.6 this talk will provide some new reasons to do it.

Transcript

1. Debugging in Python 3.6: Better, Faster, Stronger Elizaveta Shashkova JetBrains
 
 PyCon US 2017
2. Bio • Software developer of PyCharm IDE at JetBrains • Debugger • Saint Petersburg, Russia 2
3. Debugging • Adding print statements • Logging 3
4. Debugging • Adding print statements • Logging • Special tools: debuggers 4
5. Debugger’s Performance 5 Run Debug Debuggers are 30 times slower
6. Contents • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results 6
7. Contents • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results 7
8. 8 def tracefunc(frame, event, arg): print(frame.f_lineno, event) return tracefunc sys.settrace(tracefunc) 1 2 3 4 5 6 sys.settrace(tracefunc) - set the system tracing function Tracing Function
9. def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() 1 2 3 4 5 6 7 8 9 Tracing Function 9
10. 1 2 3 4 5 6 7 8 9 1 call def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 10
11. 1 2 3 4 5 6 7 8 9 1 call 2 line def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 11
12. 1 2 3 4 5 6 7 8 9 1 call 2 line 3 line 4 line Hi Bob! def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 12
13. 1 2 3 4 5 6 7 8 9 1 call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom! def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 13
14. 1 2 3 4 5 6 7 8 9 1 call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom! 5 line 5 return def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 14
15. Build Python Debugger • Breakpoints • Stepping 15
16. Tracing Debugger • Suspend program if breakpoint’s line equals frame.f_lineno • Handle events for stepping 16
17. Performance def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() 1 2 3 4 5 6 7 8 9 1 call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom! 5 line 5 return 17
18. Example 1 def calculate(): sum = 0 for i in range(10 ** 7): sum += i return sum 1 2 3 4 5 6 7 8 9 18
19. Example 1 def calculate(): sum = 0 for i in range(10 ** 7): sum += i return sum def tracefunc(frame, event, arg): return tracefunc 1 2 3 4 5 6 7 8 9 19
20. Performance 20 Run Tracing Breakpoints 0,80 sec 6,85 sec 19,81 sec
21. Performance 21 Run Tracing Breakpoints 0,80 sec 6,85 sec 19,81 sec
22. Performance 22 Run Tracing Breakpoints 0,80 sec 6,85 sec 19,81 sec
23. Performance 23 Run Tracing Breakpoints 0,80 sec 6,85 sec 19,81 sec ~ 25 times slower!
24. Problem • Tracing call on every line 24
25. Contents • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results 25
26. Contents • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results 26
27. Python 3.6 27
28. Python 3.6 • New frame evaluation API • PEP 523 28
29. PEP 523 • Handle evaluation of frames • Add a new field to code objects 29
30. Frame Evaluation def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 30
31. def frame_eval(frame, exc): func_name = frame.f_code.co_name a line_number = frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 Frame Evaluation 31
32. def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 Frame Evaluation 32
33. def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 Frame Evaluation 33
34. def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 Frame Evaluation 34
35. def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) def set_frame_eval(): state = PyThreadState_Get() state.interp.eval_frame = frame_eval 1 2 3 4 5 6 7 8 9 Frame Evaluation 35
36. Example 1 2 3 4 5 6 7 8 9 10 11 def first(): second() def second(): third() def third(): pass set_frame_eval() first() 36
37. Example def first(): second() def second(): third() def third(): pass set_frame_eval() first() 1 2 3 4 5 6 7 8 9 10 11 1 first 4 second 7 third 37
38. Custom Frame Evaluation • It works! • Executed while entering a frame • Access to frame and code object 38
39. Contents • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results 39
40. Problem • Tracing call on every line 40
41. Problem • Tracing call on every line • Remove the tracing function! 41
42. Replace tracing function with frame evaluation function 42
43. Contents • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results 43
44. Build Python Debugger • Breakpoints • Stepping 44
45. Breakpoints • Access to the whole code object 45
46. Breakpoints • Access to the whole code object • Insert breakpoint’s code into frame’s code 46
47. Breakpoints def maximum(a, b): if a > b: return a else: return b 1 2 3 4 5 6 7 8 47
48. Breakpoints def maximum(a, b): if a > b: return a # breakpoint else: return b 1 2 3 4 5 6 7 8 48
49. Breakpoints def maximum(a, b): if a > b: return a # breakpoint else: return b 1 2 3 4 5 6 7 8 breakpoint() 49
50. Breakpoints def maximum(a, b): if a > b: breakpoint() return a # breakpoint else: return b 1 2 3 4 5 6 7 8 50
51. Python Bytecode def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 51
52. Python Bytecode def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 52
53. Python Bytecode def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 53
54. Python Bytecode def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 54
55. Python Bytecode def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 55
56. Python Bytecode def maximum(a, b): if a > b: return a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 56
57. Python Bytecode 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 57
58. 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE Python Bytecode 58
59. 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE Python Bytecode 59
60. 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE Python Bytecode 60
61. Bytecode Modification 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE breakpoint() 61
62. Bytecode Modification • Insert breakpoint’s code • Update arguments and offsets 62
63. Bytecode Modification • Insert breakpoint’s code • Update arguments and offsets • 200 lines in Python 63
64. Bytecode Modification 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE breakpoint() ?! 64
65. Breakpoint Bytecode def _stop_at_break(): # a lot of code here def breakpoint(): _stop_at_break() 1 2 3 4 5 6 7 8 0 LOAD_GLOBAL 0 2 CALL_FUNCTION 0 4 POP_TOP 6 LOAD_CONST 0 8 RETURN_VALUE 65
66. Build Python Debugger • Breakpoints • Stepping 66
67. Stepping • Inserting temporary breakpoint on every line • Use old tracing function 67
68. Frame evaluation debugger is ready! 68
69. Example 1 def calculate(): sum = 0 for i in range(10 ** 7): sum += i return sum 1 2 3 4 5 69
70. Example 1 70 Run Tracing Frame 
 evaluation 0,80 sec 19,81 sec 0,81 sec
71. Example 1 71 Run Tracing Frame 
 evaluation 0,80 sec 19,81 sec 0,81 sec
72. Example 2 def foo(): pass def calculate(): sum = 0 for i in range(10 ** 7): foo() sum += i return sum 1 2 3 4 5 6 7 8 9 72
73. Example 2 73 Run Tracing Frame 
 evaluation 1,73 sec 43,58 sec 37,41 sec
74. PEP 523 • Handle evaluation of frames • Add a new field to code objects 74
75. PEP 523 • Expand PyCodeObject struct • co_extra - “scratch space” for the code object • Mark frames without breakpoints 75
76. Mark Frames def frame_eval(frame, exc): flag = _PyCode_GetExtra(frame.f_code, index) if flag == NO_BREAKS_IN_FRAME: return _PyEval_EvalFrameDefault(frame, exc)
 
 # check for breakpoints ... 1 2 3 4 5 6 7 8 9 76
77. Example 2 77 Run Tracing Frame 
 evaluation 1,73 sec 43,58 sec 1,91 sec
78. PEP 523 • Handle evaluation of frames • Add a new field to code objects 78
79. Contents • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results 79
80. Contents • Tracing debugger • Python 3.6 • Frame evaluation debugger • Results 80
81. Real Life Example 81
82. Real Life Example • Included into PyCharm 2017.1 • Works in production 82
83. PyCharm 83 Tracing
 w/o Cython Tracing
 with Cython Frame 
 evaluation 11,59 sec 5,66 sec 0,28 sec
84. Frame evaluation rocks! 84
85. Disadvantages • More complicated • Only with CPython • Only with Python 3.6 85
86. Frame Evaluation • Let’s move to Python 3.6! 86
87. Frame Evaluation • Let’s move to Python 3.6! • Let’s find another use cases! 87
88. Use cases def maximum(a, b): if a > b: return a # breakpoint else: return b 1 2 3 4 5 6 7 8 breakpoint() 88
89. PEP 523 • Pyjion project • JIT for Python 89
90. Frame Evaluation • Let’s move to Python 3.6! • Let’s find another use cases! 90
91. Links • Prototype: https://github.com/ Elizaveta239/frame-eval • PyCharm Community Edition source code 91
92. Questions? • Prototype: https://github.com/ Elizaveta239/frame-eval • PyCharm Community Edition source code 92 @lisa_shashkova