Elizaveta Shashkova - Debugging in Python 3.6: Better, Faster, Stronger

Elizaveta Shashkova - Debugging in Python 3.6: Better, Faster, Stronger

Python 3.6 was released in December of 2016 and it has a lot of new cool features. Some of them are quite easy for using: a developer can read, for example, about f-strings and they can start using them in their programs as soon as possible. But sometimes features are not so evident, and a new frame evaluation API is one of them.
The new frame evaluation API was introduced to CPython in PEP 523 and it allows to specify a per-interpreter function pointer to handle the evaluation of frames. It might not be evident how to use this new feature in everyday life, but it’s quite easy to understand how to build a fast debugger based on it.
In this talk we are going to explain how standard way of debugging in Python works and how a new frame evaluation API may be useful for creating the fast debugger. Also we will consider why such fast debugging was not possible in the previous versions of Python. If someone hasn’t made a final decision to move to Python 3.6 this talk will provide some new reasons to do it.

https://us.pycon.org/2017/schedule/presentation/526/

Bde70c0ba031a765ff25c19e6b7d6d23?s=128

PyCon 2017

May 21, 2017
Tweet

Transcript

  1. Debugging in Python 3.6: Better, Faster, Stronger Elizaveta Shashkova JetBrains


    
 PyCon US 2017
  2. Bio • Software developer of PyCharm IDE at JetBrains •

    Debugger • Saint Petersburg, Russia 2
  3. Debugging • Adding print statements • Logging 3

  4. Debugging • Adding print statements • Logging • Special tools:

    debuggers 4
  5. Debugger’s Performance 5 Run Debug Debuggers are 30 times slower

  6. Contents • Tracing debugger • Python 3.6 • Frame evaluation

    debugger • Results 6
  7. Contents • Tracing debugger • Python 3.6 • Frame evaluation

    debugger • Results 7
  8. 8 def tracefunc(frame, event, arg): print(frame.f_lineno, event) return tracefunc sys.settrace(tracefunc)

    1 2 3 4 5 6 sys.settrace(tracefunc) - set the system tracing function Tracing Function
  9. def foo(): friends = ["Bob", "Tom"] for f in friends:

    print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() 1 2 3 4 5 6 7 8 9 Tracing Function 9
  10. 1 2 3 4 5 6 7 8 9 1

    call def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 10
  11. 1 2 3 4 5 6 7 8 9 1

    call 2 line def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 11
  12. 1 2 3 4 5 6 7 8 9 1

    call 2 line 3 line 4 line Hi Bob! def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 12
  13. 1 2 3 4 5 6 7 8 9 1

    call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom! def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 13
  14. 1 2 3 4 5 6 7 8 9 1

    call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom! 5 line 5 return def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 14
  15. Build Python Debugger • Breakpoints • Stepping 15

  16. Tracing Debugger • Suspend program if breakpoint’s line equals frame.f_lineno

    • Handle events for stepping 16
  17. Performance def foo(): friends = ["Bob", "Tom"] for f in

    friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() 1 2 3 4 5 6 7 8 9 1 call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom! 5 line 5 return 17
  18. Example 1 def calculate(): sum = 0 for i in

    range(10 ** 7): sum += i return sum 1 2 3 4 5 6 7 8 9 18
  19. Example 1 def calculate(): sum = 0 for i in

    range(10 ** 7): sum += i return sum def tracefunc(frame, event, arg): return tracefunc 1 2 3 4 5 6 7 8 9 19
  20. Performance 20 Run Tracing Breakpoints 0,80 sec 6,85 sec 19,81

    sec
  21. Performance 21 Run Tracing Breakpoints 0,80 sec 6,85 sec 19,81

    sec
  22. Performance 22 Run Tracing Breakpoints 0,80 sec 6,85 sec 19,81

    sec
  23. Performance 23 Run Tracing Breakpoints 0,80 sec 6,85 sec 19,81

    sec ~ 25 times slower!
  24. Problem • Tracing call on every line 24

  25. Contents • Tracing debugger • Python 3.6 • Frame evaluation

    debugger • Results 25
  26. Contents • Tracing debugger • Python 3.6 • Frame evaluation

    debugger • Results 26
  27. Python 3.6 27

  28. Python 3.6 • New frame evaluation API • PEP 523

    28
  29. PEP 523 • Handle evaluation of frames • Add a

    new field to code objects 29
  30. Frame Evaluation def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number =

    frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 30
  31. def frame_eval(frame, exc): func_name = frame.f_code.co_name a line_number = frame.f_lineno

    print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 Frame Evaluation 31
  32. def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno print(line_number,

    func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 Frame Evaluation 32
  33. def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno print(line_number,

    func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 Frame Evaluation 33
  34. def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno print(line_number,

    func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 Frame Evaluation 34
  35. def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno print(line_number,

    func_name) return _PyEval_EvalFrameDefault(frame, exc) def set_frame_eval(): state = PyThreadState_Get() state.interp.eval_frame = frame_eval 1 2 3 4 5 6 7 8 9 Frame Evaluation 35
  36. Example 1 2 3 4 5 6 7 8 9

    10 11 def first(): second() def second(): third() def third(): pass set_frame_eval() first() 36
  37. Example def first(): second() def second(): third() def third(): pass

    set_frame_eval() first() 1 2 3 4 5 6 7 8 9 10 11 1 first 4 second 7 third 37
  38. Custom Frame Evaluation • It works! • Executed while entering

    a frame • Access to frame and code object 38
  39. Contents • Tracing debugger • Python 3.6 • Frame evaluation

    debugger • Results 39
  40. Problem • Tracing call on every line 40

  41. Problem • Tracing call on every line • Remove the

    tracing function! 41
  42. Replace tracing function with frame evaluation function 42

  43. Contents • Tracing debugger • Python 3.6 • Frame evaluation

    debugger • Results 43
  44. Build Python Debugger • Breakpoints • Stepping 44

  45. Breakpoints • Access to the whole code object 45

  46. Breakpoints • Access to the whole code object • Insert

    breakpoint’s code into frame’s code 46
  47. Breakpoints def maximum(a, b): if a > b: return a

    else: return b 1 2 3 4 5 6 7 8 47
  48. Breakpoints def maximum(a, b): if a > b: return a

    # breakpoint else: return b 1 2 3 4 5 6 7 8 48
  49. Breakpoints def maximum(a, b): if a > b: return a

    # breakpoint else: return b 1 2 3 4 5 6 7 8 breakpoint() 49
  50. Breakpoints def maximum(a, b): if a > b: breakpoint() return

    a # breakpoint else: return b 1 2 3 4 5 6 7 8 50
  51. Python Bytecode def maximum(a, b): if a > b: return

    a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 51
  52. Python Bytecode def maximum(a, b): if a > b: return

    a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 52
  53. Python Bytecode def maximum(a, b): if a > b: return

    a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 53
  54. Python Bytecode def maximum(a, b): if a > b: return

    a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 54
  55. Python Bytecode def maximum(a, b): if a > b: return

    a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 55
  56. Python Bytecode def maximum(a, b): if a > b: return

    a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 56
  57. Python Bytecode 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1

    (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 57
  58. 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4

    COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE Python Bytecode 58
  59. 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4

    COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE Python Bytecode 59
  60. 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4

    COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE Python Bytecode 60
  61. Bytecode Modification 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1

    (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE breakpoint() 61
  62. Bytecode Modification • Insert breakpoint’s code • Update arguments and

    offsets 62
  63. Bytecode Modification • Insert breakpoint’s code • Update arguments and

    offsets • 200 lines in Python 63
  64. Bytecode Modification 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1

    (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE breakpoint() ?! 64
  65. Breakpoint Bytecode def _stop_at_break(): # a lot of code here

    def breakpoint(): _stop_at_break() 1 2 3 4 5 6 7 8 0 LOAD_GLOBAL 0 2 CALL_FUNCTION 0 4 POP_TOP 6 LOAD_CONST 0 8 RETURN_VALUE 65
  66. Build Python Debugger • Breakpoints • Stepping 66

  67. Stepping • Inserting temporary breakpoint on every line • Use

    old tracing function 67
  68. Frame evaluation debugger is ready! 68

  69. Example 1 def calculate(): sum = 0 for i in

    range(10 ** 7): sum += i return sum 1 2 3 4 5 69
  70. Example 1 70 Run Tracing Frame 
 evaluation 0,80 sec

    19,81 sec 0,81 sec
  71. Example 1 71 Run Tracing Frame 
 evaluation 0,80 sec

    19,81 sec 0,81 sec
  72. Example 2 def foo(): pass def calculate(): sum = 0

    for i in range(10 ** 7): foo() sum += i return sum 1 2 3 4 5 6 7 8 9 72
  73. Example 2 73 Run Tracing Frame 
 evaluation 1,73 sec

    43,58 sec 37,41 sec
  74. PEP 523 • Handle evaluation of frames • Add a

    new field to code objects 74
  75. PEP 523 • Expand PyCodeObject struct • co_extra - “scratch

    space” for the code object • Mark frames without breakpoints 75
  76. Mark Frames def frame_eval(frame, exc): flag = _PyCode_GetExtra(frame.f_code, index) if

    flag == NO_BREAKS_IN_FRAME: return _PyEval_EvalFrameDefault(frame, exc)
 
 # check for breakpoints ... 1 2 3 4 5 6 7 8 9 76
  77. Example 2 77 Run Tracing Frame 
 evaluation 1,73 sec

    43,58 sec 1,91 sec
  78. PEP 523 • Handle evaluation of frames • Add a

    new field to code objects 78
  79. Contents • Tracing debugger • Python 3.6 • Frame evaluation

    debugger • Results 79
  80. Contents • Tracing debugger • Python 3.6 • Frame evaluation

    debugger • Results 80
  81. Real Life Example 81

  82. Real Life Example • Included into PyCharm 2017.1 • Works

    in production 82
  83. PyCharm 83 Tracing
 w/o Cython Tracing
 with Cython Frame 


    evaluation 11,59 sec 5,66 sec 0,28 sec
  84. Frame evaluation rocks! 84

  85. Disadvantages • More complicated • Only with CPython • Only

    with Python 3.6 85
  86. Frame Evaluation • Let’s move to Python 3.6! 86

  87. Frame Evaluation • Let’s move to Python 3.6! • Let’s

    find another use cases! 87
  88. Use cases def maximum(a, b): if a > b: return

    a # breakpoint else: return b 1 2 3 4 5 6 7 8 breakpoint() 88
  89. PEP 523 • Pyjion project • JIT for Python 89

  90. Frame Evaluation • Let’s move to Python 3.6! • Let’s

    find another use cases! 90
  91. Links • Prototype: https://github.com/ Elizaveta239/frame-eval • PyCharm Community Edition source

    code 91
  92. Questions? • Prototype: https://github.com/ Elizaveta239/frame-eval • PyCharm Community Edition source

    code 92 @lisa_shashkova