error_highlight: User-friendly Error Diagnostics

RubyKaigi 2022
https://rubykaigi.org/2022/presentations/mametter.html

Transcript

1. error_highlight: User-friendly Error Diagnostics Yusuke Endoh (@mametter) RubyKaigi 2022
3. Yusuke Endoh / @mametter • A Ruby committer working at Cookpad w/ @ko1 • My major Ruby contributions: • Keyword arguments • coverage.so • TypeProf • error_highlight  Today's topic 3 https://ruby-puzzles-2022.cookpad.tech/
4. error_highlight Ruby 3.1 reports the fine-grained error location 4 $ ruby test.rb test.rb:1:in `<main>': undefined method `time' for 42:Integer (NoMethodError) 42.time { print "Hello" } ^^^^^ Did you mean? times error_highlight
5. json is nil ? Or json[:article] is nil ? undefined method `[]' for nil:NilClass (NoMethodError) How useful? To tell which of the calls failed in a line Typical case: 5 json[:article][:author] ^^^^^^^^^ undefined method `[]' for nil:NilClass (NoMethodError) json[:article][:author] ^^^^^^^^^^ json[:article][:author] json is nil json[:article] is nil
6. More useful than theory Developer experience is more improved than expected • Suggests "how to fix" • Helps you reach the wrong code in the editor 6 $ ruby test.rb test.rb:123:in `<main>': undefined method `gsuub' for ... str.gsuub(/....../, "foobar") ^^^^^^ Did you mean? gsub! gsub
7. What will change in Ruby 3.2? • ArgumentError / TypeError • Ruby 3.1 supported only NameError / NoMethodError 7 $ ruby test.rb test.rb:1:in `+': nil can't be coerced into Integer (TypeError) 1 + nil ^^^ Today's topic: the road to achieve them • Rails' error page
8. Agenda • ➔Background • How error_highlight implemented in Ruby 3.1 • Problems and solutions • Conclusion 8
9. Implementation overview • 1. Record the code range in bytecode • 2. Determine where to underline • 3. Prints an error message 9 3.1
10. 1. Record code ranges in byte code 10 compile foo.bar(1) 0..2 0..9 8..8 send "foo" put 1 send "bar" byte code source code code range 0..2 8..8 0..9
11. 2. Determine where to underline 11 send "foo" put 1 send "bar" foo.bar(42) 0..9 foo.bar(42) 0..2 error error 0..2 8..8 0..9 This approach is too naïve! Why?
12. ary.map do … end.selct do … end Naïve code range is too wide • error_highlight uses AST analysis & Regexps (!) • AST does not retain a location of punctuation (e.g., period) • Kevin Newton's parser work may improve the situation (related to the next talk in this session!) 12 ary.map do … end.selct do … end expected ary.map do … end.selct do … end naïve Typo
13. Support various method calls 13 obj.foo += 1 obj.foo is not defined obj.foo += 1 obj.foo returned nil obj.foo += 1 obj.foo= is not defined prinnt(str) nil + 1 nil[1] obj&.foo obj. foo obj .foo Suggestion for improvement is welcome
14. 3. Prints code with the underline • Overrides NameError#message • Ruby's error printer writes #message to stderr • Ruby 3.1 released this version, but 14 class NameError def message super + "¥n" + code + "¥n" + underline end end This approach had many problems!
15. Agenda • Background • ➔Problems and solutions • Less expandable • Wrong underline location • Poor ecosystem support • Conclusion 15
16. Problem 1: Less expandable • Only NameError / NoMethodError were supported • Why? • Because an error message has some usages • A. To show an error trace • B. To test error-handling code • C. To log an error • error_highlight breaks B and C 16
17. Problem 1-B: Test compatibility • Many tests in the wild checks an error message • Changing Exception#message is incompatible • Acceptable to change NameError#message since few test cases check its result 17 expect { … }.to raise_error("foobar")
18. Problem 1-C: Logging compatibility Many expect #message to return a one-line string •"error_highlight makes it difficult to parse a log file!" •"error_highlight makes so confusing!" 18 E, [2022-09-10T10:00:00.000000 #12345] ERROR -- : undefined method `time' for 42:Integer 42.time ^^^^^ Did you mean? times #<NoMethodError: undefined method `time' for 42:Integer 42.time ^^^^^ Did you mean? times> p $!
19. Solution for Problem 1 Exception#detailed_message is introduced 19 $!.message undefined method `time' for 1:Integer $!.detailed_message undefined method `time' for 1:Integer (NoMethodError) 1.time ^^^^^ Did you mean? times
20. Request to framework developers • Frameworks may want to use #detailed_message • Instead of #message • Example: https://github.com/rack/rack/pull/1926 • Sentry and DataDog said they would support this • https://bugs.ruby-lang.org/issues/18438 • thanks to Stan Lo (Sentry) and Ivo Anjo (DataDog) 20
21. Problem 1 is solved ArgumentError / TypeError are now supported Without significant incompatibility 21 test.rb:1:in `+': nil can't be coerced into Integer (TypeError) 1 + nil ^^^
22. Problem 2: Wrong underline Ruby's error printer "escapes" the message 22 test.rb:1:in `<main>': undefined method `gsuub' for " ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥":String (NoMethodError) " ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥".gsuub("", "") ^^^^^^ " ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥".gsuub("", "") Wrong! Wrong too!
23. Solution to Problem 2 Stopped Ruby's error printer from escaping 23 This simple solution took about a month! Why?
24. Why did it escape the message? • Because of security consideration! • "TERMINAL EMULATOR SECURITY ISSUES" [1] • Some terminals had insecure escape sequences • Create a file • Input as if a user typed it 24 [1] https://marc.info/?l=bugtraq&m=104612710031920&w=2
25. Is this concern still valid? • [1] is very old (in 2003) and also says: • Surveyed the current situation • Rxvt / Eterm / XTerm disabled the dangerous features The terminals mentioned in [1] • Gnome Terminal, Windows Terminal and iTerm2 don't support the dangerous features 25 The responsibility should rest on the actual terminal emulator
26. Problem 2 is solved • We agreed for Ruby to stop escaping • Correct underline in Ruby 3.2 26 test.rb:1:in `<main>': undefined method `gsuub' for " ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥":String (NoMethodError) " ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥".gsuub("", "") ^^^^^^ " ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥ ⃥".gsuub("", "")
27. Problem 3: Poor ecosystem support Rack error page 27 broken
28. Solution to Problem 3 Write a patch! https://github.com/rack/rack/pull/1925 28 <pre>
29. Support Rails error page • Rails provides a more dedicated error page • error_highlight provides only Exception#message Parse #message …? 29
30. Export error_highlight API 30 ErrorHighlight.spot( $!, backtrace_location: $!.backtrace_locations[0] ) { :first_lineno=>2, :first_column=>3, :last_lineno=>2, :last_column=>8, :snippet=>" 1.time¥n", :script_lines=>[…] }
31. Problem 3 is solved Rails error page now shows error_highlight The patch is merged https://github.com/rails/rails/pull/45818 31
32. Agenda • Background • Problems and solutions • ➔Conclusion 32
33. Ruby 3.2's error_highlight • ArgumentError / TypeError Ruby 3.1 supported only NameError / NoMethodError 33 $ ruby test.rb test.rb:1:in `+': nil can't be coerced into Integer (TypeError) 1 + nil ^^^ • Rails' error page
34. Acknowledgments • @yui-knk made the early prototype of error_highlight • @ioquatix reported a lot of issues • Those who joined in the discussion on tickets, PRs, or the dev meeting 34
36. Memory overhead of bytecode • Approx. 3% (on scaffold Rails app) • Record AST node_id instead of raw linenos/columns • Simple compression 36
37. Future work / Known problems • Support CJK full-width characters • I don't want to do that! • Support more exception classes • User-defined exceptions • Show column number • test.rb:2:2 37
38. Does error_highlight use escape sequence? • It can, but I have no plan • Why? • Terminal messages are often copy/pasted as a text • Critical use of escape sequence will make people want to use screenshots • ➔ Searching by error message will be difficult 38
39. Lots of considerations and tasks • ANSI escape code (= terminal font style) • Gems • did_you_mean gem (thanks to Yuki Nishijima) • syntax_suggest gem (thanks to Richard Schneeman) 39 $!.detailed_message(highlight: true) undefined method `time' for 1:Integer (NoMethodError) 1.time ^^^^^ Did you mean? times