Visual Debugger for Jupyter Notebooks: Myth or Reality?

Many Python developers like Jupyter Notebooks for their flexibility: they are very useful for interactive prototyping, scientific experiments, visualizations and many other tasks. There are different development tools which make working with Jupyter Notebooks easier and smoother, but all of them lack very important feature: visual debugger. Since Jupyter Kernel is a usual Python process, it looks reasonably to use one of existing Python debuggers with it. But is it really possible?

In this talk we’ll try to understand how Python debugger should be changed to work with Jupyter cells and how these changes are already implemented in the PyCharm IDE. After that we’ll look into the whole Jupyter architecture and try to understand which bottlenecks in it prevent creation of universal Jupyter debugger at the moment.

This talk requires a basic knowledge of Jupyter Notebooks and understanding of Python functions and objects. It will be interesting for people who want to learn internals of the tools they use every day. Also it might be an inspiration for people who want to implement a visual debugger in their favourite IDE.

Transcript

1. Visual Debugger for Jupyter Notebooks: Myth or Reality? Elizaveta Shashkova
 EuroPython 2019
2. About Me •Software Developer at JetBrains, PyCharm IDE •Debugger and Data Science tools • @lisa_shashkova 2
3. Visual Debugger 3
4. Jupyter Notebooks •Popular scientific tool •File is a sequence of cells 4
5. Jupyter Notebooks Debug 5 •Logging with print statements •Command-line debugger ipdb
6. Jupyter Notebooks Debug 6
7. Myth or Reality?
8. Myth or Reality?
9. Contents •Python files debugging •Jupyter breakpoints •Debugger communication •Jupyter visual debugger 9
10. Tracing Function 10 def tracefunc(frame, event, arg): print(frame.f_lineno, event) return tracefunc sys.settrace(tracefunc) 1 2 3 4 5 6
11. Tracing Function 11 def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets) sys.settrace(tracefunc) greet_neighbors() 1 2 3 4 5 6 7 8 9
12. Tracing Function 12 1 2 3 4 5 6 7 8 9 1 call def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets) sys.settrace(tracefunc) greet_neighbors()
13. Tracing Function 13 1 2 3 4 5 6 7 8 9 1 call 2 line def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets) sys.settrace(tracefunc) greet_neighbors()
14. Tracing Function 14 1 2 3 4 5 6 7 8 9 1 call 2 line 3 line 4 line Hi Mars! def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets) sys.settrace(tracefunc) greet_neighbors()
15. Tracing Function 15 1 2 3 4 5 6 7 8 9 1 call 2 line 3 line 4 line Hi Mars! 3 line 4 line Hi Venus! def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets) sys.settrace(tracefunc) greet_neighbors()
16. Tracing Function 16 1 2 3 4 5 6 7 8 9 def greet_neighbors(): planets = ["Mars", "Venus"] for p in planets: print(f"Hi {p}!") return len(planets) sys.settrace(tracefunc) greet_neighbors() 1 call 2 line 3 line 4 line Hi Mars! 3 line 4 line Hi Venus! 5 line 5 return
17. Breakpoint •frame.f_lineno - current line number •frame.f_code.co_filename - current file name 17
18. Breakpoint •frame.f_lineno - current line number •frame.f_code.co_filename - current file name •Equals to breakpoint’s file and line -> suspend program! 18
19. Contents •Python files debugging •Jupyter breakpoints •Debugger communication •Jupyter visual debugger 19
20. Contents •Python files debugging •Jupyter breakpoints •Debugger communication •Jupyter visual debugger 20
21. Cells Execution 21 IPython kernel Front-end IDE
22. Cells Execution 22 code IPython kernel Front-end IDE
23. IPython kernel Cells Execution 23 code execution Front-end IDE
24. Cells Execution 24 result IPython kernel Front-end IDE
25. Cells Execution •Kernel generates a unique name for each cell •<ipython-input-5-11faed10a894> •File name of a generated 
 code object 25 IPython kernel code execution
26. Jupyter Breakpoints •Python files: (filename, line number) -> unique location 26
27. Jupyter Breakpoints •Python files: (filename, line number) -> unique location •Jupyter Notebooks? 27
28. Jupyter Breakpoints •Python files: (filename, line number) -> unique location •Jupyter Notebooks: •generated cell name •line inside code object 28
29. Source Mapping 29 IPython kernel IDE cell source code MyNotebook.ipynb generated <code object>
30. Source Mapping 30 IPython kernel IDE cell source code MyNotebook.ipynb cell id generated <code object>
31. IPython kernel IDE Source Mapping 31 ? cell source code MyNotebook.ipynb cell id generated <code object>
32. Source Mapping 32 •Tracking cells execution in the IDE
33. Source Mapping 33 •Tracking cells execution in the IDE •Silent cell execution in IPython kernel
34. Debug Cell Execution 34 <cell source code>
35. Debug Cell Execution 35 <cell source code> patch name generation - silent mode
36. Debug Cell Execution 36 <cell source code> patch name generation cell id - silent mode
37. Jupyter Tracing Function •frame.f_code.co_filename - generated name 37
38. Jupyter Tracing Function •frame.f_code.co_filename - generated name •Map: generated name -> cell id 38
39. Jupyter Tracing Function •frame.f_code.co_filename - generated name •Map: generated name -> cell id •Send message to the IDE 39
40. Jupyter Tracing Function •frame.f_code.co_filename - generated name •Map: generated name -> cell id •Send message to the IDE 40
41. Contents •Python files debugging •Jupyter breakpoints •Debugger communication •Jupyter visual debugger 41
42. Contents •Python files debugging •Jupyter breakpoints •Debugger communication •Jupyter visual debugger 42
43. Debug Communication 43 “Add breakpoint in a cell 3, line 2” IPython kernel Front-end IDE
44. Debug Communication 44 “Add breakpoint in a cell 3, line 2” IPython kernel Front-end IDE
45. Debug Communication 45 •Additional connection •Reuse Jupyter channels
46. Jupyter Messaging 46 Front-end IPython kernel Kernel 
 proxy
47. Jupyter Messaging 47 Front-end IPython kernel Kernel 
 proxy
48. Debug Communication 48 •Additional connection •Reuse Jupyter channels
49. Debug Communication 49 •Additional connection •Reuse Jupyter channels
50. Jupyter Messaging 50 Front-end Kernel 
 proxy IPython kernel Shell IOPub stdin
51. Jupyter Architecture 51 •Event loop in a main thread for execution events •Event loop for output events
52. Jupyter Architecture 52 code debug x Blocked IPython kernel Front-end IDE
53. Debug Communication 53 •Additional connection •Reuse Jupyter channels
54. Debug Communication 54 •Additional connection •Reuse Jupyter channels But ipdb works!
55. But ipdb Works! 55
56. But ipdb Works! 56 •Based on input() •Reuses user input channel
57. Debug Communication 57 •Additional connection •Reuse Jupyter channels
58. Contents •Python files debugging •Jupyter breakpoints •Debugger communication •Jupyter visual debugger 58
59. Contents •Python files debugging •Jupyter breakpoints •Debugger communication •Jupyter visual debugger 59
60. Jupyter Visual Debugger •Jupyter tracing function 60
61. Jupyter Visual Debugger •Jupyter tracing function •Mapping between editor and generated code 61
62. Jupyter Visual Debugger •Jupyter tracing function •Mapping between editor and generated code •Debugger connection 62
63. Live Demo 63
64. Live Demo 64 •PyCharm doesn’t convert Jupyter Notebooks to Python files! •On disk it’s still the same JSON file with .ipynb extension
65. Jupyter Visual Debugger •Jupyter tracing function •Mapping between editor and generated code •Debugger connection 65
66. Jupyter Visual Debugger •Implement in your favourite IDE 66
67. Jupyter Visual Debugger •Implement in your favourite IDE •Try it in PyCharm Pro! 67
68. Jupyter Visual Debugger •Implement in your favourite IDE •Try it in PyCharm Pro! •Questions? 68 @lisa_shashkova