Ruby Debugger: Upside Down - Inside Out

Agenda
- How does Ruby execute your code?
- How do Ruby debuggers work?
- Build your down debugger
- A better experience while debugging Ruby

Transcript

1. Ruby Debugger Upside Down - Inside Out
2. About me - Rubyist for 5 years. - Motorcyclist, cat lover - Core contributor for Ruby Kafka client. - Author of Ruby Jard, a debugger for Ruby. - (Ex) Engineering Manager at Employment Hero. - Contact me at https://github.com/nguyenquangminh0711/
3. Agenda - How does Ruby execute your code? - How do Ruby debuggers work? - Build your down debugger - A better experience while debugging Ruby
4. How does Ruby execute your code?
5. Ruby Interpretation Pipeline Source Code Tokens AST Nodes YARV Instructions RubyVM Control Frames Stack Tokenized Parsed Compiled Intercepted
6. Example
7. Tokenize (or lexical analyze or lex) - Input: Any ruby source code - Output: Meaningful Identified Tokens
8. Tokenize
9. Parse into an Abstract Syntax Tree - Input: Tokens - Output: AST Nodes
10. Parse into Abstract Syntax Tree
11. Compile into YARV Instructions - Input: AST Nodes - Output: Yarv Instructions - Note: YARV stands for “Yet another RubyVM”
12. Compile into YARV Instructions - Yarv Instruction is a nested structure. - Each instruction contains low-level execution used by RubyVM + Discrete values + Local variables + Arguments + Catch table (for raising, throwing) - Each instruction also contains reflection information: + File, line, code range that creates that instruction + Trace events
13. Ruby Virtual Machine RubyVM Control Frames Stack - Input: YARV Instructions - Ruby VM is a double-stack engine - All of the instructions are to manipulate these two stacks.
14. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx>
15. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> <Object#xxx>
16. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> <Object#xxx> 10
17. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx>
18. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> a = nil, b = nil, n = 10
19. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> 1 a = nil, b = nil, n = 10
20. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> a = 1, b = nil, n = 10
21. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> 1 a = 1, b = nil, n = 10
22. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> a = 1, b = 1, n = 10
23. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> a = 1, b = 1, n = 10 3
24. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> a = 1, b = 1, n = 10 3 10
25. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> a = 1, b = 1, n = 10 3..10
26. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> a = 1, b = 1, n = 10
27. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> c = nil Range in <each>
28. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> c = nil Range in <each> Object in <fibonacci block>
29. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> c = nil Range in <each> Object in <fibonacci block> 1
30. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> c = nil Range in <each> Object in <fibonacci block> 1 1
31. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> c = nil Range in <each> Object in <fibonacci block> 2
32. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> c = 2 Range in <each> Object in <fibonacci block>
33. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> c = 2 Range in <each> Object in <fibonacci block> 1
34. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> c = 2 Range in <each> Object in <fibonacci block>
35. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> c = 2 Range in <each> Object in <fibonacci block> 2
36. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> Object in <fibonacci> c = 2 Range in <each>
38. Ruby Virtual Machine RubyVM Control Frames Stack Object in <main> <Object#xxx> 55
39. Ruby Virtual Machine RubyVM Control Frames Stack
40. Some takeaways - RubyVM executes instruction by instruction, not line by line. - Each line consists of one or more than 1 instructions. - Control frames are persisted during program execution. - A control frame’s context is persisted too.
41. How do Ruby debuggers work?
42. Ruby is generous - It provide a hook system in the VM, called TracePoint. - Each instruction is masked with some flags called trace events. - *Before* Ruby executes any instruction, it checks the flags, and call registered hooks.
43. Ruby is generous Ruby TracePoint supports different types of events: - Line event (Li) - Class definition (Cl) - End definition (En) - Call (Ca) - Return (Re) - ...
44. How do Ruby debuggers use this feature? - All debuggers have their own breakpoints and control flow managements. - They register the tracepoints with Ruby - When the tracepoint stops at an event - Start a REPL session to receive user input if necessary - Read, evaluate, and print result - Continue if flow control commands received
45. Build your down debugger
46. Getting started Register a new tracepoint. We care about line event only
47. Getting started Enter a REPL loop, and start fetching user input
48. Getting started Check control flow command. - Break the loop if next - Disable tracepoint if continue
49. Getting started Otherwise, evaluate input at the current context, and print result
50. Getting started Inject into source code
51. Demo
52. Print surrounding lines Print lines before starting REPL
53. Print surrounding lines Print location information
54. Print surrounding lines Read current source file, and extract display window
55. Print surrounding lines Print line by line, with “=>” mark to indicate current line
56. Demo 2
57. What’s next? - Breakpoint management - Advanced flow control - Advanced context capture - Stability and reliability - etc. - Ruby’s built-in debugger: https://github.com/ruby/ruby/blob/master/lib/debug.rb - Byebug debugger: https://github.com/deivid-rodriguez/byebug/blob/master/ext/byebug/byebug.c# L479
58. A better experience while debugging Ruby
59. Debugging is painful - Your bug is painful enough, it costs a lot of mental effort to think about - All the current debuggers help you debug, but not with pleasure: - You have to fit the current context, remember the current lines, current variables - You have to type a lot, go back and forth to examine your variables, and source code - Jumping between control frames is available, but extremely unfriendly - Sometimes, you have have no idea what your debugger is leading you. - Sometimes, you struggle with inspecting a nested of nested of nested array, and forgot where are you now. - Or, well, you have a visualized mindset, and annoyed that information keeps hiding from you
60. Is there any way out? - I search over and over and over again, but found no matches for me. - RubyMine, VSCode provide some better experiences, but still, a lot of features are missing, and no way it can compare with debugging in terminal. - So, I build a debugger myself to solve this.
61. Introducing Ruby Jard - Provide a better experience while debugging Ruby - It provides modular visual interfaces to show relevant information about your debugging program, usability, and highly friendly to developers, especially new-comers. - They help you reduce the commands you need to type, the mental efforts wasted trying to navigate and grab the information you need. - As a result, you can now focus more on the debug flow.
62. Demo 3: Ruby Jard
63. Demo 3: Ruby Jard - Visualize all information you need on the screen. - Advanced flow control - Support keybinding - Support rich customization - Ignore unwanted jumps (go deep into Gem) - Support catching exception - Support watch variable - Support mouse and scrolling - Tools to inspect and explore variables - Multi-thread debugging https://asciinema.org/a/UpcIkky3FQnZPuKMa6TqPlM9G
64. Getting started with Jard - `gem install ruby_jard` - Visit Jard’s home page at rubyjard.org - Visit source code page at github.com/nguyenquangminh0711/ruby_jard
65. Takeaways
66. Takeaways - Ruby process your code into a pipeline: tokenize, parse into AST, compile into yarv instructions, and feed instructions into RubyVM. - RubyVM executes instruction by instruction, push/pop control frames when open/close a scope, and manipulate environment stack. - RubyVM provides a hook system called TracePoint. - Ruby debuggers implement its magic based on TracePoint and REPL. - Ruby Jard looks cool. Let’s download it at rubyjard.org
67. The end