[RubyConf] Beware the Dreaded Dead End

Nothing stops a program from executing quite as fast as a syntax error. After years of “unexpected end” in my dev life, I decided to “do” something about it. In this talk we'll cover lexing, parsing, and indentation informed syntax tree search that power that dead_end Ruby library.

I also gave this talk at RubyKaigi. I recommend this version, it is more up-to-date and covers specific details and "gotchas" on the problem space.

Transcript

1. Beware the Dreaded Dead End!!! By @schneems Hello everyone. My name is Richard Schneeman and I want to talk to you about the scariest thing a Ruby programmer can face. You hear that?
2. Everyone run
3. Oh no, there are more of them. AHHHH
4. Beware the Dreaded Dead End!!! By @schneems Wow, I can hardly believe that the those dinosaurs from RubyKaigi found me all the way over here, I thought I escaped them already. If you've seen that talk, know that this one has some new content.
5. Beware the Dreaded Dead End!!! By @schneems While dinosaurs are scary. There's something scarier. Its..
6. > # <= routes.rb:121: syntax error, unexpected end-of-input, expecting `end’ Rails.application.routes.draw do constraints -> { Rails.application.config.non_production } do namespace :foo do resource :bar end end constraints -> { Rails.application.config.non_production } do namespace :bar do resource :baz end end namespace :admin do resource :session match "/foobar(*path)", via: :all, to: redirect { |_params, req| uri = URI(req.path.gsub("foobar", "foobaz")) uri.query = req.query_string.presence uri.to_s } end Syntax errors <click: laughter>
7. > Rails.application.routes.draw do constraints -> { Rails.application.config.non_production } do namespace :foo do resource :bar end end constraints -> { Rails.application.config.non_production } do namespace :bar do resource :baz end end namespace :admin do resource :session match "/foobar(*path)", via: :all, to: redirect { |_params, req| uri = URI(req.path.gsub("foobar", "foobaz")) uri.query = req.query_string.presence uri.to_s } end # <= routes.rb:121: syntax error, unexpected end-of-input, expecting `end’ Just look at this unexpected syntax error. It's horrifying. Where's the problem?
8. > RSpec.describe Cutlass::BashResult do it "preserves stdout, stderr, and status" stdout = SecureRandom.hex(16) stderr = SecureRandom.hex(16) status = 0 result = BashResult.new( stdout: stdout, stderr: stderr, status: status ) expect(result.stdout).to eq(stdout) expect(result.stderr).to eq(stderr) expect(result.status).to eq(status) end end # <= result_spec:19 syntax error, unexpected end-of-input, expecting `end’ Here's some di ff erent code. Wanna guess where Ruby think the problem is? <click> On the last line, nope. This is frustrating
9. 🤔 What if we had something better
10. > RSpec.describe Cutlass::BashResult do it "preserves stdout, stderr, and status" stdout = SecureRandom.hex(16) stderr = SecureRandom.hex(16) status = 0 result = BashResult.new( stdout: stdout, stderr: stderr, status: status ) expect(result.stdout).to eq(stdout) expect(result.stderr).to eq(stderr) expect(result.status).to eq(status) end end # <= result_spec:19 syntax error, unexpected end-of-input, expecting `end’ Instead of this
11. > 3 module Cutlass 4 RSpec.describe Cutlass::BashResult do ❯ 5 it "preserves stdout, stderr, and status" ❯ 19 end 21 it "success?" do 52 end 53 end 54 end This code has an unmatched `end`. Ensure that all `end` lines in your code have a matching syntax keyword (`def`, `do`, etc.) What if we got this? Do you see the error now? Doesn't line 5 look suspicious, like maybe it's missing a do.
12. 🎂 Have you ever seen a cooking show where they show o ff the fi nal product? I want to show o ff dead end before we get too far.
13. demo Here is a demo of how dead_end fi nds syntax errors
14. module SyntaxErrorSearch # Used for formatting invalid blocks class DisplayInvalidBlocks attr_reader :filename def initialize(block_array, io: $stderr, filename: nil) @filename = filename @io = io @blocks = block_array @lines = @blocks.map(&:lines).flatten @digit_count = @lines.last.line_number.to_s.length @code_lines = @blocks.first.code_lines @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} end def call @io.puts <<~EOM 64: syntax error, unexpected end-of-input, expecting `end' 📕 ❌ This is real ruby source code with a syntax error in it. It cannot parse
15. 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s 58 string << "\e[0m" 59 string 60 end 61 end.join 62 end 63 end 64 end <click>
16. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end
17. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end
18. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end
19. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end
20. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end
21. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else ❯ 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end When a valid code block is found dead_end safely removes it
22. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 60 end 61 end.join 62 end 63 end 64 end 📕 ❌ After each step in the search, dead end re-evaluates the whole document to see if it's parsable yet. <click> parsing failed, we need to keep looking
23. 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? ❯ 52 "#{number.to_s}#{line}" 53 else 60 end 61 end.join 62 end 63 end 64 end
24. 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 53 else 60 end 61 end.join 62 end 63 end 64 end 📕 ❌
25. 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) ❯ 51 if line.empty? ❯ 53 else ❯ 60 end 61 end.join 62 end 63 end 64 end
26. 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 61 end.join 62 end 63 end 64 end 📕 ❌
27. 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| ❯ 49 next if line.hidden? ❯ 50 number = line.line_number.to_s.rjust(@digit_count) 61 end.join 62 end 63 end 64 end
28. 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 61 end.join 62 end 63 end 64 end 📕 ❌
29. 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines ❯ 48 @code_lines.map do |line| ❯ 61 end.join 62 end 63 end 64 end
30. 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 62 end 63 end 64 end 📕 ❌
31. 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 ❯ 47 def code_with_lines ❯ 62 end 63 end 64 end
32. 40 string << "```\n" 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 63 end 64 end 📕 ❌
33. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 63 end 64 end
34. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
35. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
36. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename ❯ 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
37. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" ❯ 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename ❯ 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
38. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") ❯ 40 string << "```\n" ❯ 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename ❯ 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
39. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename ❯ 39 string = String.new("") ❯ 40 string << "```\n" ❯ 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename ❯ 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
40. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename 45 end 46 63 end 64 end 📕 ❌
41. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array 10 @lines = @blocks.map(&:lines).flatten 11 @digit_count = @lines.last.line_number.to_s.length 12 @code_lines = @blocks.first.code_lines 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
42. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array 10 @lines = @blocks.map(&:lines).flatten 11 @digit_count = @lines.last.line_number.to_s.length 12 @code_lines = @blocks.first.code_lines 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
43. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array 10 @lines = @blocks.map(&:lines).flatten 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
44. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
45. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array ❯ 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
46. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io ❯ 9 @blocks = block_array ❯ 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
47. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename ❯ 8 @io = io ❯ 9 @blocks = block_array ❯ 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
48. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) ❯ 7 @filename = filename ❯ 8 @io = io ❯ 9 @blocks = block_array ❯ 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
49. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 15 end 16 35 36 def filename 37 38 def code_with_filename 45 end 46 63 end 64 end 📕 ❌
50. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 15 end 16 35 36 def filename 37 ❯ 38 def code_with_filename ❯ 45 end 46 63 end 64 end
51. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 15 end 16 35 36 def filename 37 46 63 end 64 end 📕 ❌
52. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 ❯ 6 def initialize(block_array, io: $stderr, filename: nil) ❯ 15 end 16 35 36 def filename 37 46 63 end 64 end
53. 📕 ❌ 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 16 35 36 def filename 37 46 63 end 64 end
54. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename ❯ 5 ❯ 16 ❯ 35 ❯ 36 def filename ❯ 37 ❯ 46 63 end 64 end
55. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 63 end 64 end 📕 ✅ The document is checked. <click> The parser reports that this very minimal document is now parsable.
56. Syntax O 🥰🥰🥰 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 63 end 64 end 📕 ✅ Once this happens dead_end has found a way to transform the document to be valid. It can stop searching the document and instead search the invalid code blocks that it has stored
57. DeadEnd: Missing `end` detected This code has a missing `end`. Ensure that all syntax keywords (`def`, `do`, etc.) have a matching `end`. file: /Users/rschneeman/Documents/projects/dead_end/spec/fixtures/this_project_extra_def.rb.txt simplified: 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename ❯ 36 def filename ❯ 38 def code_with_filename ❯ 45 end 63 end 64 end This is the actual output the dead end gem returns on that fi le. The issue is on line 36 there's a missing end statement
58. $ gem install dead_end The dead_end gem is released and on ruby gems.
59. By the time you see this slide dead_end will have surpassed half a million downloads
60. Ruby 3.2? I am also talking with Ruby core to get dead_end into Ruby directly. We are targeting that integration for 2022 release of Ruby 3.2
61. 🤔 What all can dead_end do?
62. > 3 module Cutlass 4 RSpec.describe Cutlass::BashResult do ❯ 5 it "preserves stdout, stderr, and status" ❯ 19 end 21 it "success?" do 52 end 53 end 54 end DeadEnd: Unmatched `end` detected When you miss a keyword like "if", "do" or "def". <click> Dead end fi nds the problem.
63. > 1 module Cutlass 2 RSpec.describe Cutlass::BashResult do ❯ 3 it "preserves stdout, stderr, and status" do 5 it "success?" do 6 end 7 end 8 end DeadEnd: Missing `end` detected Even when you miss an end keyword, <click> dead end fi nds the problem
64. > 1 class Cat 2 def to_json ❯ 3 hash = { ❯ 4 name: name, ❯ 5 fur_color: fur_color, ❯ 6 type: 'Cat', ❯ 7 better_than_dog: false ❯ 8 hash.to_json 9 end 10 end DeadEnd: Unmatched `}` character detected When you miss a curly bracket <click> dead end fi nds the problem
65. > 1 class Animal 2 def self.ones_my_three_year_old_likes ❯ 3 array = [ ❯ 4 Dog.new, ❯ 5 Lion.new, ❯ 6 Tiger.new, ❯ 7 Aligator.new, 10 return array 11 end 12 end DeadEnd: Unmatched `[` detected When you miss a square bracket <click> dead end fi nds the problem
66. > 1 class Cat 2 def meow ❯ 3 Animal.call do |a ❯ 5 end 6 end 7 end DeadEnd: Unmatched `|` character detected Have a problem with a missing pipe character? Dead end. Finds. <click> The. <click> Problem <click> <Pause> Have a problem with a missing family member in a Korean thriller drama?
67. > Unfortunately. Dead end cannot fi nd that problem. <click: laughter>. Okay, wow that current event reference got really dark. Let's rewind a try again.
68. > Have a problem with a missing marvel universe character?
69. > 1 class Cat 2 def meow ❯ 3 Animal.call do |a ❯ 5 end 6 end 7 end DeadEnd: Unmatched `|` character detected ? ? ? Unfortunately. <click: laughter> Dead end cannot fi nd that problem. Much better. Remember crocodile Loki? That was 2021, this year is lasting forever!
70. $ gem install dead_end Today we'll dig into
71. $ dead_end bad.rb --record tmp/ You can follow the algorithm yourself by running the CLI with the record fl ag
72. You'll see each step
73. Along with annotated source code
74. Syntax errors Today we will talk about
75. Lexing & Parsing How to parse and lex source code with ripper
76. AI How AI and path fi nding algorithms work
77. dead_end We will put it all together to see how dead_end works
78. Internals Finally we'll crack open the cover and see how it works at a low level. Warning: We will get technical. We will cover a lot of code.
79. 👋 But wait, who am I?
80. @schneems I go by schneems on the internet. If you forget how to pronounce my name you can go to
81. @schneems the about page on my blog on Schneems dot com
82. I Created CodeTriage which is a platform for learning how to contribute to open source. To date I've got over 60 thousand developers signed up for CodeTriage
83. 📕 HowTo Open Source .dev I'm writing a book on how to contribute to open source as well. It's for developers who want to start contributing. Developers who got stuck, and developers looking for ways to sustain contributions. I'm not ready to release it yet, but you can sign up for a pre-order on HowToOpenSource dot dev Also you can sign up for Code Triage dot com and I will email members when the book is ready.
84. Heroku When I'm not working on open source or teaching people to open source with CodeTriage. I like to get paid. I work at <click> Heroku
85. Heroku Right now I'm working on salesforce functions. It's an easy way to work with the data you've got in salesforce using the languages you love. Right now we support Java and JavaScript languages. We'll be rolling out Ruby support later. If you use love Ruby and use Salesforce I would love to hear from you. My DMs on Twitter are open. Also people tell me
86. NoMemoryError ScriptError LoadError SyntaxError SecurityError SignalException KeyError StandardError ArgumentError EncodingError IOError EOFError I am Exceptional i'm an exceptional programmer... <click> My programs generate a lot of exceptions <click: laugh>
87. Syntax errors Syntax errors. Let's start from the beginning what is a syntax error? Why are syntax errors so hard to understand?
88. > while b != 0 if a > b a = a - b else b = b - a end end return a This code worked wonderfully comparing a, and b all day.
89. > while b != 0 if a > b a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a When Ruby parses this code it will convert it into an abstract syntax tree. <click> <point> this code generates <point> that tree. The tree was beautiful and Ruby's parser looked upon it with happiness
90. > while b != 0 if a > b a = a - b else b = b - a while b != 0 if a > b a = a - b else b = b - a end end return a 😈 While the code was good a stranger <click> came upon the land. This stranger had a secret power, behold
91. # The octothorpe, This tiny character gave the stranger great power to create, or destroy.
92. > while b != 0 if a > b a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a 😈 With one key the stranger used their power <click>
93. > while b != 0 # if a > b a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a 😈 to transform the code <click>
94. > while b != 0 # if a > b a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a 😈 A critical line had been commented out. Without that line, the tree was no longer whole
95. > while b != 0 # if a > b a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a 😈 Huge sections of code are no longer reachable. The code no longer parses. And our parser is sad.
96. > while b != 0 # if a > b a = a - b else b = b - a end end return a while b != 0 a = a - b else b = b - a 😈 With that the stranger left <click> and behind them stood
97. > while b != 0 # if a > b a = a - b else b = b - a end end return a while b != 0 a = a - b else b = b - a Syntax error A syntax error
98. Syntax errors A syntax error occurs when the parser cannot build a valid syntax tree. That explains what they are, but why are they di ffi cult to understand?
99. > module Cutlass def call 1, 2, 3, 4, 5] end end syntax error, unexpected ',', expecting end-of-input (SyntaxError) When the parser tries to parse this code it fi nds a <click> Syntax Error. Here our developer forgot a bracket. As the parser is building the tree, it hit an error because it wasn't expecting the comma
100. > syntax error, unexpected ',', expecting end-of-input (SyntaxError) module Cutlass def call 1, 2, 3, 4, 5] end end Ruby's parser has rules. It knows that after a method de fi nition it should look for something such as
101. > syntax error, unexpected ',', expecting end-of-input (SyntaxError) module Cutlass def call 1, 2, 3, 4, 5] end end Like a return or an end. Or a variable assignment. Instead what did the parser fi nd fi nd? It found a number and a comma.
102. > module Cutlass def call 1, 2, 3, 4, 5] end end syntax error, unexpected ',', expecting end-of-input (SyntaxError) 🚫 That comma violated ruby's parser rules. Do not pass go, do not collect 200 <click>
103. 🤷 What's so bad about this example?
104. > module Cutlass def call 1, 2, 3, 4, 5] end end syntax error, unexpected ',', expecting end-of-input (SyntaxError) The error <click> isn't caused by a comma, it's caused by the developer missing a bracket. Essentially, the location the parse error occurs isn't always where developer made a mistake
105. 🙇 Let's look at some more complicated cases
106. > module Cutlass defcall end end Here's code with a syntax error. The developer forgot a space after the def
107. > module Cutlass defcall end end syntax error, unexpected `end', expecting end-of-input Ruby thinks the error is at the last line. <click> right here. Why?
108. > module Cutlass defcall end end syntax error, unexpected `end', expecting end-of-input Welllll... When we start parsing Ruby sees a module de fi nition, A module is a keyword that requires an end so Ruby starts looking for a matching end.
109. > module Cutlass defcall end end syntax error, unexpected `end', expecting end-of-input On the next line it sees this combination of characters and thinks it's a method call
110. > module Cutlass defcall end end syntax error, unexpected `end', expecting end-of-input Then it sees this end and thinks it matches with the module. Our parser hasn't raised a syntax error because it doesn't see a problem yet.
111. > module Cutlass defcall end end syntax error, unexpected `end', expecting end-of-input Finally it gets to our second end, but there's no thing to match with. A syntax error is raised. <click>
112. > end syntax error, unexpected `end', expecting end-of-input Right now this is the error you get from ruby on this code. It's not helpful to a human because deleting that end won't solve anything.
113. parser problems != human problems Another way to put this, is that parse errors are di ff erent from human errors.
114. $ gem install dead_end dead_end's goal is to turn that parser problem into something a human can see and instantly recognize as an issue.
115. $ gem install dead_end How does dead end work?
116. Dead end works by using a library called Ripper
117. require "ripper" Ripper.new(source).tap(&:parse).error? It's not a band. Ripper is Ruby's parser that ships with Ruby. How cool is that?
118. require "ripper" source = <<~EOM module Cutlass defcall end end EOM Ripper.new(source).tap(&:parse).error? Ripper can evaluate code and tell us if there's a syntax error or not. We saw this code before
119. require "ripper" source = <<~EOM module Cutlass defcall end end EOM Ripper.new(source).tap(&:parse).error? , there's a syntax error here
120. require "ripper" source = <<~EOM module Cutlass defcall end end EOM Ripper.new(source).tap(&:parse).error? # => true When we run ripper it says there's an error
121. require "ripper" source = <<~EOM module Cutlass def call end end EOM Ripper.new(source).tap(&:parse).error? # => false When we fi x the error, ripper tells us it's fi xed too. <click>
122. 🙇 If we could fi x the source errors then ripper would tell us, but that's too hard to automate. Can we do something simpler?
123. require "ripper" source = <<~EOM module Cutlass defcall end end EOM Ripper.new(source).tap(&:parse).error? # => true Back to our code with a syntax error. Instead of fi xing it. What if we...commented it out?
124. require "ripper" source = <<~EOM module Cutlass # defcall end end EOM Ripper.new(source).tap(&:parse).error? # => true That didn't work, let's keep going.
125. require "ripper" source = <<~EOM module Cutlass # defcall # end end EOM Ripper.new(source).tap(&:parse).error? # => false Oooh, nice <click> ripper can tell us when we've commented out the source code causing the problem
126. # defcall # end Ripper.new(source).tap(&:parse).error? # => false Now if we only look at the part that was commented out, dead_end can show you where the issue occured.
127. $ gem install dead_end Remember our stranger and the mess they left behind?
128. > while b != 0 a = a - b else b = b - a end end return a while b != 0 a = a - b else b = b - a 😈 Ruby knows there's a syntax error here, but it doesn't know where.
129. > while b != 0 a = a - b else b = b - a end end return a while b != 0 a = a - b else b = b - a 😈 It will tell you
130. > while b != 0 a = a - b else b = b - a end end return a syntax error, unexpected `end', expecting end-of-input That the problem is this line. We know that's not helpful. What does dead end do?
131. > while b != 0 a = a - b else b = b - a end end return a while b != 0 a = a - b else b = b - a Dead end uses indentation and lexical parsing to deconstruct the source code from the outside in <click>
132. > while b != 0 a = a - b else # b = b - a end end return a while b != 0 a = a - b else b = b - a Commenting out this code removes a node from our tree
133. > while b != 0 # a = a - b else # b = b - a end end return a while b != 0 a = a - b else Commenting out this code removes a node from our tree, but it's still invalid
134. > while b != 0 # a = a - b # else # b = b - a # end end return a while b != 0 else Finally here we have a valid tree. All of the orphaned syntax nodes are gone
135. > 1 while b != 0 ❯ 4 else ❯ 6 end 7 end This code has an unmatched `end`. Ensure that all `end` lines in your code have a matching syntax keyword (`def`, `do`, etc.) Here's the output for that code. It shows that the else and end are missing an if statement.
136. I bet you're thinking wow, that was easy. So all he did was comment out code based on indentation?
137. Gotchas There are some major gotchas when executing this recursive comment approach.
138. Gotcha #1 We already know the developer has a syntax error. We can't assume they've done everything else perfectly. 
 We cannot assume perfect indentation
139. > describe "things" do it "valid" do expect(the).to eq(unexpected) end it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything Our syntax error is on this line. It is missing keyword do
140. > describe "things" do it "valid" do expect(the).to eq(unexpected) end it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything Without it, the parser has an extra end here
141. > describe "things" do it "valid" do expect(the).to eq(unexpected) end it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything If we only look at indentation then we would remove this line
142. > describe "things" do it "valid" do end it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything Then this mis-indented end
143. > describe "things" do it "valid" do it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything 📕 ✅ With the end gone, the document parses<click>. Why?
144. > describe "things" do it "valid" do it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything 📕 ✅ Because this keyword now matches with
145. > describe "things" do it "valid" do it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything 🚫 This end It's not what we wanted, and didn't capture the source of the syntax error <click>
146. 🙇 How can we fi x it? Instead of only looking at indentation we also look at lex output
147. source = <<~EOM it "valid" do expect(the).to eq(unexpected) end EOM Ripper.lex(source) Ruby ships with a lexer
148. source = <<~EOM it "valid" do expect(the).to eq(unexpected) end EOM Ripper.lex(source) # => [[[1, 0], :on_ident, "it", CMDARG], [[1, 2], :on_sp, " ", CMDARG], [[1, 3], :on_tstring_beg, "\"", CMDARG], [[1, 4], :on_tstring_content, "valid", CMDARG], [[1, 9], :on_tstring_end, "\"", END], [[1, 10], :on_sp, " ", END], [[1, 11], :on_kw, "do", BEG], [[1, 13], :on_ignored_nl, "\n", BEG], Here's what it looks like
149. # => [[[1, 0], :on_ident, "it", CMDARG], [[1, 2], :on_sp, " ", CMDARG], [[1, 3], :on_tstring_beg, "\"", CMDARG], [[1, 4], :on_tstring_content, "valid", CMDARG], [[1, 9], :on_tstring_end, "\"", END], [[1, 10], :on_sp, " ", END], [[1, 11], :on_kw, "do", BEG], [[1, 13], :on_ignored_nl, "\n", BEG], [[2, 0], :on_sp, " ", BEG], [[2, 6], :on_ident, "expect", CMDARG], [[2, 12], :on_lparen, "(", BEG|LABEL], [[2, 13], :on_ident, "the", ARG], [[2, 16], :on_rparen, ")", ENDFN], [[2, 17], :on_period, ".", DOT], [[2, 18], :on_ident, "to", ARG], DO NOT CLICK, auto advance
150. # => [[[1, 0], :on_ident, "it", CMDARG], [[1, 2], :on_sp, " ", CMDARG], [[1, 3], :on_tstring_beg, "\"", CMDARG], [[1, 4], :on_tstring_content, "valid", CMDARG], [[1, 9], :on_tstring_end, "\"", END], [[1, 10], :on_sp, " ", END], [[1, 11], :on_kw, "do", BEG], [[1, 13], :on_ignored_nl, "\n", BEG], [[2, 0], :on_sp, " ", BEG], [[2, 6], :on_ident, "expect", CMDARG], [[2, 12], :on_lparen, "(", BEG|LABEL], [[2, 13], :on_ident, "the", ARG], [[2, 16], :on_rparen, ")", ENDFN], [[2, 17], :on_period, ".", DOT], [[2, 18], :on_ident, "to", ARG], It tells you the code contains a do keyword
151. [[2, 12], :on_lparen, "(", BEG|LABEL], [[2, 13], :on_ident, "the", ARG], [[2, 16], :on_rparen, ")", ENDFN], [[2, 17], :on_period, ".", DOT], [[2, 18], :on_ident, "to", ARG], [[2, 20], :on_sp, " ", ARG], [[2, 21], :on_ident, "eq", ARG], [[2, 23], :on_lparen, "(", BEG|LABEL], [[2, 24], :on_ident, "unexpected", ARG], [[2, 34], :on_rparen, ")", ENDFN], [[2, 35], :on_sp, " ", ENDFN], [[2, 36], :on_nl, "\n", BEG], [[3, 0], :on_sp, " ", BEG], [[3, 2], :on_kw, "end", END], [[3, 5], :on_nl, "\n", BEG]] DO NOT CLICK, auto advance
152. [[2, 12], :on_lparen, "(", BEG|LABEL], [[2, 13], :on_ident, "the", ARG], [[2, 16], :on_rparen, ")", ENDFN], [[2, 17], :on_period, ".", DOT], [[2, 18], :on_ident, "to", ARG], [[2, 20], :on_sp, " ", ARG], [[2, 21], :on_ident, "eq", ARG], [[2, 23], :on_lparen, "(", BEG|LABEL], [[2, 24], :on_ident, "unexpected", ARG], [[2, 34], :on_rparen, ")", ENDFN], [[2, 35], :on_sp, " ", ENDFN], [[2, 36], :on_nl, "\n", BEG], [[3, 0], :on_sp, " ", BEG], [[3, 2], :on_kw, "end", END], [[3, 5], :on_nl, "\n", BEG]] Also an end
153. $ gem install dead_end With this info dead_end can fi nd the problem
154. > describe "things" do it "valid" do expect(the).to eq(unexpected) end it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything It starts at the same place
155. > describe "things" do it "valid" do end it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything Now it needs to decide where to expand, it looks down and sees an `end`
156. > describe "things" do it "valid" do end it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything dead_end knows if it removes this end, it might trigger a false positive, so it searches up for keywords. It fi nds...
157. > describe "things" do it "valid" do end it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything This do keyword
158. > describe "things" do it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything They are both safely removed
159. > file: /private/tmp/scratch.rb simplified: 1 describe "things" do ❯ 6 it "forgot a keyword -> " ❯ 8 end 9 end Gotcha: Indentation isn't everything The algorithm continues until the problem is found
160. Gotcha #2 Even with correct indentation, removing the wrong line can show false positives
161. > class User puts "hello" end def schneems User. where(name: 'schneems'). first end end Gotcha: Removing the wrong line Our syntax error is on these lines.
162. > class User puts "hello" end def schneems User. where(name: 'schneems'). first end end Gotcha: Removing the wrong line We are missing a whole method de fi nition line
163. > class User puts "hello" end def schneems User. where(name: 'schneems'). first end end Gotcha: Removing the wrong line If we tried our indentation comment out strategy we would start at the largest indentation
164. > class User puts "hello" end def schneems User. where(name: 'schneems'). first end end Gotcha: Removing the wrong line here
165. > class User puts "hello" end def schneems User. where(name: 'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ And then dead_end would stop looking because the document is now parsable <click>
166. WAT WAT on earth? Let's look at it a little di ff erently.
167. > class User puts "hello" end def schneems User. where(name: 'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ It will help if we reformat it to show what Ruby thinks you're trying to do. It thinks
168. > class User puts "hello" end def schneems User. where(name: 'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ That this user class
169. > class User puts "hello" end def schneems User. where(name: 'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ Is matched with the fi rst end.
170. > class User puts "hello" end def schneems User. where(name: 'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ Is matched with the fi rst end.
171. > class User puts "hello" end def schneems User. where(name: 'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ It thinks that this end, is actually a method due to the trailing dot
172. > class User puts "hello" end def schneems User. where(name: 'schneems').end end Gotcha: Removing the wrong line 📕 ✅ Finally it thinks the last end matches with our method de fi nition
173. > class User puts "hello" end def schneems User. where(name: 'schneems').end end Gotcha: Removing the wrong line 📕 ✅ Finally it thinks the this method de fi nition
174. > class User puts "hello" end def schneems User. where(name: 'schneems').end end Gotcha: Removing the wrong line 📕 ✅ 🚫 Matches with this last end. <click> It's a problem because none of this is what the user intended and we didn't highlight the source of the problems
175. WAT But wait there's more
176. > source = <<~EOM Hello World From A heredoc EOM Gotcha: Removing the wrong line Heredocs cause problems
177. > source = <<~EOM Hello World From A heredoc EOM Gotcha: Removing the wrong line If you evaluate this line in isolation, it looks like a constant. But if you remove it
178. > source = <<~EOM Hello World From A heredoc Gotcha: Removing the wrong line 🚫 Then you've introduced a syntax error<click>
179. WAT But wait there's more
180. > it "should" \ "handle trailing slash" do expect(the).to eq(unexpected) end Gotcha: Removing the wrong line Trailing slashes cause problems.
181. > it "should" \ "handle trailing slash" do expect(the).to eq(unexpected) end Gotcha: Removing the wrong line If you start in the middle
182. > it "should" \ "handle trailing slash" do end Gotcha: Removing the wrong line Now expand out using lex detection
183. > it "should" \ "handle trailing slash" do end Gotcha: Removing the wrong line Find the end
184. > it "should" \ "handle trailing slash" do end Gotcha: Removing the wrong line Find the matching keyword.
185. > it "should" \ Gotcha: Removing the wrong line 🚫 Now remove them.<click> This is not valid ruby
186. 🙇 What
187. 🙇 🙇 To
188. 🙇 🙇 🙇 do
189. source = <<~EOM User. where(name: 'schneems'). first EOM Ripper.lex(source) # => [[[1, 0], :on_const, "User", CMDARG], [[1, 4], :on_period, ".", DOT], [[1, 5], :on_ignored_nl, "\n", DOT], [[2, 0], :on_sp, " ", DOT], [[2, 2], :on_ident, "where", ARG], [[2, 7], :on_lparen, "(", BEG|LABEL], [[2, 8], :on_label, "name:", ARG|LABELED], [[2, 13], :on_sp, " ", ARG|LABELED], We can use the lexer to detect these speci fi c edge cases. <click> Then we can use that information to combine all logically consecutive lines together so there's no way to remove one without removing all of them
190. > it "should" \ "handle trailing slash" do expect(the).to eq(unexpected) end Gotcha: Removing the wrong line It's hard to show visually, but it essentially would looks like re-writing source code. This trailing slash would become
191. > it "should" \ "handle trailing slash" do expect(the).to eq(unexpected) end Gotcha: Removing the wrong line
192. > class User puts "hello" end def schneems User. where(name: 'schneems'). first end end Gotcha: Removing the wrong line Our method chain would become
193. > class User puts "hello" end def schneems User.where(name: 'schneems').first end end Gotcha: Removing the wrong line Our method chain would become
194. > source = <<~EOM Hello World From A heredoc EOM Gotcha: Removing the wrong line Heredocs cause problems
195. > source = <<~EOM Hello\nWorld\nFrom\nA\n\nheredoc\nEOM Gotcha: Removing the wrong line
196. 🎉 With all the problem lines joined together, they can now be safely evaluated and removed without accidentally introducing new syntax errors <click>
197. Gotcha #3 Even with joining lines, even with lexically aware search, even with the perfect algorithm there is one problem we cant' avoid
198. ambiguity Ambiguity. Yes, ambiguity is the comic sans of source code parsing.<click> What do I mean by ambiguity?
199. > class Dog def sit puts "no" end def eat puts "munch" end def bark puts "woof" end Gotcha: ambiguity This code has a syntax error
200. > class Dog def sit puts "no" end def eat puts "munch" end def bark puts "woof" end Gotcha: ambiguity It is ambiguous what the coder was trying to do with our current search algorithm. Let's zoom in
201. > class Dog def bark puts "woof" end Gotcha: ambiguity <click>
202. > class Dog def bark puts "woof" end Gotcha: ambiguity Starting from the middle, if we use lexical expansion we get
203. > class Dog def bark puts "woof" end Gotcha: ambiguity This match
204. > class Dog Gotcha: ambiguity When it's removed there's still a syntax error
205. > class Dog Gotcha: ambiguity Highlight
206. > Gotcha: ambiguity 📕 ✅ Remove<click> It parses, yay
207. > ❯ 1 class Dog Gotcha: ambiguity 🚫 📕 ✅ But this clearly isn't our problem <click>
208. 🙇 To produce good results with invalid indentation it means we must make "bad" suggestions. There's a logical inverse of this problem as well.
209. > class Dog puts "no" end def eat puts "munch" end def bark puts "woof" end end Gotcha: ambiguity This code has a syntax error, but this time it's at the top.
210. > class Dog puts "no" end def eat puts "munch" end def bark puts "woof" end end Gotcha: ambiguity Let's focus
211. > class Dog puts "no" end end Gotcha: ambiguity
212. > class Dog puts "no" end end Gotcha: ambiguity Does this problem look familiar? Here's the code from before
213. > class Dog puts "no" end end Gotcha: ambiguity class Dog def bark puts "woof" end Before we were missing an end
214. > class Dog puts "no" end end Gotcha: ambiguity class Dog def bark puts "woof" end Before we were missing an end
215. > class Dog puts "no" end end Gotcha: ambiguity class Dog def bark puts "woof" end This new case is missing a keyword
216. > class Dog puts "no" end end Gotcha: ambiguity class Dog def bark puts "woof" end This new case is missing a keyword
217. > class Dog puts "no" end end Gotcha: ambiguity class Dog def bark puts "woof" end Where does dead end tell us the problem is?
218. > class Dog puts "no" end end class Dog def bark puts "woof" end Gotcha: ambiguity ❯ 10 end ❯ 1 class Dog 🚫 Here's the result. Neither make good suggestion<click>
219. 🙇 If we re-write our rules to let indentation take precedence over lexical keywords then it would make other examples fail. It's impossible to satisfy all search criteria. What if we...
220. 😇 Don't. Because we know this ambiguity exists, we can compensate for it after the search is done
221. > ❯ 1 class Dog Gotcha: ambiguity Even though this is all the user might see, `dead_end` still has all the contents that are hidden. It didn't just match a single line, it matched many lines, but hid most of them
222. > ❯ 1 class Dog ❯ 2 def sit ❯ 3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity If we un-hid the rest of them, this is what it would look like. We can detect this case when our match is only one line and contains an end, then we can go back through our results
223. > ❯ 1 class Dog ❯ 2 def sit ❯ 3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity un-hide the matched end
224. > ❯ 1 class Dog ❯ 2 def sit ❯ 3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity Work backwards and fi nd an unmatched keyword
225. > ❯ 1 class Dog ❯ 10 def bark ❯ 12 end Gotcha: ambiguity Then we show the user all extra context to the user and they can resolve the problem visually
226. Gotcha #4 Syntax errors aren't always alone, sometimes they have friends. A document can contain multiple syntax errors
227. > end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors Here' there's multiple problems
228. > end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors This end matches nothing
229. > end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors We are missing a square bracket
230. > end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors We're missing all sorts of stu ff here
231. > end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors Right here, that's code is fi ne
232. > end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors That code is fi ne too
233. > end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors What we don't want to happen is for our search to start at one side
234. > end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors And continue until it captures everything, even the good code
235. > end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors 🚫 This isn't what we want <click>
236. 🙇 What can we do about it? We can modify our algorithm to hold and fi nd multiple errors.
237. > end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors Instead of searching from one side to the other we start in the outside and search in
238. > end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1
239. > end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2
240. > end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2
241. > end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2
242. > end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2
243. > end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2 #3
244. 🙇 🙇 🙇 🙇 Those are our main gotchas. Let's move on
245. AI Now you know the gotchas. I want to start moving a little closer to the code. Who knows what AI is? Raise your hand, or in chat type "me"
246. Artificial Intelligence Algorithm - Arti fi cial Intelligence is a fancy way of saying <click> "Algorithm" [wait for dialog] "Come on no you're not" <click>
247. Artificial Intelligence Algorithm <click: laughter> More speci fi cally, AI is a goal seeking algorithm
248. Pathfinding Credit: Factorio 'New pathfinding algorithm' - A common example of AI is path fi nding. You want to get from point A to point B
249. $ gem install dead_end Dead end uses a search algorithm to fi nd the problem code. Which uses a variation on uniform cost search
250. Credit @redblobgames: Introduction to the A-star Algorithm - You need a way to break down the problem into discrete actions that either get you closer or further from the goal. In path fi nding this would be deciding what turns to make and what roads to drive on.
251. Credit @redblobgames: Introduction to the A-star Algorithm - The algorithm dead_end uses is also sometimes called dijkstra's algorithm because he invented uniform cost search.
252. Credit @redblobgames: Introduction to the A-star Algorithm I highly recommend this interactive page from redblob games as an introduction to search
253. Credit @redblobgames: Introduction to the A-star Algorithm A search algorithm Is good for when you know what goal you want but you're not quite sure how to get there.
254. > while b != 0 if a > b a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a Before we saw how code is represented by abstract syntax trees.
255. Tree
 Search
 BOTVINNIK, M.M. (1984). "Computers in Chess Solving Inexact Search Problems" That was an important realization for me, as most search problems are represented as a tree or a graph. Here's a quick example of a famous chess problem
256. Tree
 Search
 BOTVINNIK, M.M. (1984). "Computers in Chess Solving Inexact Search Problems" If you have an exact tree you can walk the tree
257. require "ripper" source = <<~EOM module Cutlass defcall end end EOM Ripper.new(source).tap(&:parse).error? # => true In our case, we have bits of source code
258. require "ripper" source = <<~EOM module Cutlass # defcall end end EOM Ripper.new(source).tap(&:parse).error? # => true I found that if I could give my program rules about how to create logical code blocks
259. require "ripper" source = <<~EOM module Cutlass # defcall # end end EOM Ripper.new(source).tap(&:parse).error? # => false Then using ripper we can check a document without their presence
260. # defcall # end Ripper.new(source).tap(&:parse).error? # => false And complete our search
261. Internals We'll now take a closer look at some dead_end algorithms
262. 🎂 If dead end was a cake here's the main steps to make it
263. We start out with messy code
264. Dead end then transforms that code into a tidier input
265. "Time to cook" by Robbert van der Steeg is licensed under CC BY-SA 2.0 It gives that input to our search algorithm to fi nd the syntax errors.
266. Once the syntax errors are found, the output is decorated with more context and given back to the user
267. 🎂 Let's start at the beginning
268. Your app has a syntax error
269. module Kernel module_function def require(file) dead_end_original_require(file) rescue SyntaxError => e DeadEnd.handle_error(e) end end dead_end/auto.rb Like all good Ruby libraries dead end uses monkey patching <click: laughter>. It hooks into require and friends. When a syntax error is raised, it's passed to dead_end
270. # ... Timeout.timeout(timeout) do record_dir ||= ENV["DEBUG"] ? "tmp" : nil search = CodeSearch.new(source, record_dir: record_dir).call end # ... dead_end/internals.rb The source code that caused the syntax error is read to disk and then passed to a search object.
271. First we need to clean it up
272. clean_document.rb # frozen_string_literal: true module DeadEnd # Parses and sanitizes source into a lexically aware document # # Internally the document is represented by an array with each # index containing a CodeLine correlating to a line from the source code. # # There are three main phases in the algorithm: # # 1. Sanitize/format input source # 2. Search for invalid blocks # 3. Format invalid blocks into something meaninful # # This class handles the first part. # # The reason this class exists is to format input source # for better/easier/cleaner exploration. Here's the clean_document class
273. clean_document.rb @source = source @document = CodeLine.from_source(@source) end # Call all of the document "cleaners" # and return self def call clean_sweep .join_trailing_slash! .join_consecutive! .join_heredoc! self end # Return an array of CodeLines in the # document def lines It handles several of the gotcha cases from before by transforming the source code
274. clean_document.rb @source = source @document = CodeLine.from_source(@source) end # Call all of the document "cleaners" # and return self def call clean_sweep .join_trailing_slash! .join_consecutive! .join_heredoc! self end # Return an array of CodeLines in the # document def lines It clears out comments and stray whitespace
275. clean_document.rb @source = source @document = CodeLine.from_source(@source) end # Call all of the document "cleaners" # and return self def call clean_sweep .join_trailing_slash! .join_consecutive! .join_heredoc! self end # Return an array of CodeLines in the # document def lines @document Joins lines with trailing slashes
276. clean_document.rb @source = source @document = CodeLine.from_source(@source) end # Call all of the document "cleaners" # and return self def call clean_sweep .join_trailing_slash! .join_consecutive! .join_heredoc! self end # Return an array of CodeLines in the # document def lines @document Lines with chained methods
277. clean_document.rb @source = source @document = CodeLine.from_source(@source) end # Call all of the document "cleaners" # and return self def call clean_sweep .join_trailing_slash! .join_consecutive! .join_heredoc! self end # Return an array of CodeLines in the # document def lines @document And heredocs
278. "Time to cook" by Robbert van der Steeg is licensed under CC BY-SA 2.0 Next up we feed that to our search class
279. code_search.rb # Searches code for a syntax error # # The bulk of the heavy lifting is done in: # # - CodeFrontier (Holds information for generating blocks and determining if we can stop searching) # - ParseBlocksFromLine (Creates blocks into the frontier) # - BlockExpand (Expands existing blocks to search more code # # ## Syntax error detection # # When the frontier holds the syntax error, we can stop searching # # search = CodeSearch.new(<<~EOM) # def dog # def lol # end # EOM Here's the CodeSearch class
280. end # Main search loop def call until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self end code_search.rb Here's call
281. end # Main search loop def call until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self code_search.rb Under every great AI algorithm is a while loop. There's a class called the frontier. It holds all the code blocks that we've generated from the source code.
282. end # Main search loop def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } code_search.rb 📕 ✅ 📕 ❌ The frontier is responsible for checking the exit condition, here's that logic
283. > 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename ❯ 5 ❯ 16 ❯ 35 ❯ 36 def filename ❯ 37 ❯ 46 63 end 64 end 📕 ❌ If the highlighted lines are on the frontier
284. > 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 63 end 64 end 📕 ✅ Then the entire document without those lines are checked against the parser
285. # ... end module DeadEnd # ... def self.invalid?(source) source = source.join if source.is_a?(Array) source = source.to_s Ripper.new(source).tap(&:parse).error? end # ... end dead_end.rb And here's where the parser is called. <click> Remember Ripper from before?
286. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s 58 string << "\e[0m" 59 string 60 end 61 end.join 62 end 63 end 64 end Let's walk through the logic of searching this same document we saw before. From the beginning
287. end # Main search loop def call until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self end code_search.rb Since we start with an invalid source code, the frontier is empty so this is false
288. end # Main search loop def call until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self end code_search.rb There's two modes of exploration. We can add new things onto the frontier, or we can expand an existing block o ff of the frontier. When do we expand?
289. def expand? return false if @frontier.empty? # ... # Expand all blocks before moving to unvisited lines frontier_indent >= unvisited_indent end code_frontier.rb We can't expand a block if there's nothing there.
290. def expand? return false if @frontier.empty? # ... # Expand all blocks before moving to unvisited lines frontier_indent >= unvisited_indent end code_frontier.rb We're also using indentation to decide when to expand. Essentially we want to explore the largest indentations fi rst.
291. end # Main search loop def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } code_search.rb There's nothing to expand yet so this is false
292. end # Main search loop def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } code_search.rb So we must fi rst add lines to our frontier
293. # Parses the most indented lines into blocks that are marked # and added to the frontier def visit_new_blocks max_indent = frontier.next_indent_line&.indent while (line = frontier.next_indent_line) && (line.indent == max_indent) @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block| record(block: block, name: "add") block.mark_invisible if block.valid? push(block, name: "add") end end end # Given an already existing block in the frontier, expand it to see # if it contains our invalid syntax code_search.rb The code here is looking at the largest indentation, starting at the bottom and iterating up.
294. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
295. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
296. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
297. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
298. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
299. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else ❯ 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
300. # Parses the most indented lines into blocks that are marked # and added to the frontier def visit_new_blocks max_indent = frontier.next_indent_line&.indent while (line = frontier.next_indent_line) && (line.indent == max_indent) @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block| record(block: block, name: "add") block.mark_invisible if block.valid? push(block, name: "add") end end end # Given an already existing block in the frontier, expand it to see # if it contains our invalid syntax code_search.rb This step only uses indentation.
301. # Parses the most indented lines into blocks that are marked # and added to the frontier def visit_new_blocks max_indent = frontier.next_indent_line&.indent while (line = frontier.next_indent_line) && (line.indent == max_indent) @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block| record(block: block, name: "add") block.mark_invisible if block.valid? push(block, name: "add") end end end # Given an already existing block in the frontier, expand it to see code_search.rb If the block that it found is valid ruby code, it's essentially commented out.
302. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else ❯ 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
303. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 60 end 61 end.join 62 end 63 end 64 end
304. # Parses the most indented lines into blocks that are marked # and added to the frontier def visit_new_blocks max_indent = frontier.next_indent_line&.indent while (line = frontier.next_indent_line) && (line.indent == max_indent) @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block| record(block: block, name: "add") block.mark_invisible if block.valid? push(block, name: "add") end end end # Given an already existing block in the frontier, expand it to see code_search.rb The code block is then pushed onto the frontier and the loop continues
305. sweep(block: block, name: "comments") end # Main search loop def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb We've still not found the problem we will continue
306. sweep(block: block, name: "comments") end # Main search loop def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb There's still lines with more indentation
307. sweep(block: block, name: "comments") end # Main search loop def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb We need to add them
308. > 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? ❯ 52 "#{number.to_s}#{line}" 53 else 60 end 61 end.join 62 end 63 end 64 end
309. > 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 53 else 60 end 61 end.join 62 end 63 end 64 end Boop
310. sweep(block: block, name: "comments") end # Main search loop def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb We've added all lines at this indent level. It's time to expand the frontier
311. code_search.rb sweep(block: block, name: "comments") end # Main search loop def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) We call our expansion phase
312. end end end # Given an already existing block in the frontier, expand it to see # if it contains our invalid syntax def expand_invalid_block block = frontier.pop return unless block record(block: block, name: "pop") block = @block_expand.call(block) push(block, name: "expand") end def sweep_heredocs HeredocBlockParse.new( source: @source, code_search.rb Here's our expand code
313. end end end # Given an already existing block in the frontier, expand it to see # if it contains our invalid syntax def expand_invalid_block block = frontier.pop return unless block record(block: block, name: "pop") block = @block_expand.call(block) push(block, name: "expand") end def sweep_heredocs HeredocBlockParse.new( source: @source, code_search.rb The frontier is stored in indentation order. The highest indentation block is popped o f
314. end end end # Given an already existing block in the frontier, expand it to see # if it contains our invalid syntax def expand_invalid_block block = frontier.pop return unless block record(block: block, name: "pop") block = @block_expand.call(block) push(block, name: "expand") end def sweep_heredocs HeredocBlockParse.new( source: @source, code_search.rb The block is expanded using an object and pushed back onto the frontier. How does block expansion work?
315. # puts "wow" # end # class BlockExpand def initialize(code_lines: ) @code_lines = code_lines end def call(block) if (next_block = expand_neighbors(block, grab_empty: true)) return next_block end expand_indent(block) end def expand_indent(block) block = AroundBlockScan.new(code_lines: @code_lines, block: block) block_expand.rb It can either expand to capture more lines at the same indentation, aka neighbors. Or when those are exhausted it can expand out an indentation.
316. def expand_indent(block) block = AroundBlockScan.new(code_lines: @code_lines, block: block) .skip(:hidden?) .stop_after_kw .scan_adjacent_indent .code_block end def expand_neighbors(block, grab_empty: true) scan = AroundBlockScan.new(code_lines: @code_lines, block: block) .skip(:hidden?) .stop_after_kw .scan_neighbors # Slurp up empties if grab_empty scan = AroundBlockScan.new(code_lines: @code_lines, block: scan.code_block) block_expand.rb Both methods use this class AroundBlockScan which acts as a DSL for scanning up and down. This is where we give our algorithm rules about how to build blocks using indentation and lexical data
317. > 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? ❯ 52 "#{number.to_s}#{line}" 53 else 60 end 61 end.join 62 end 63 end 64 end Previously we captured this line, and put it on the frontier.
318. > 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) ❯ 51 if line.empty? ❯ 53 else ❯ 60 end 61 end.join 62 end 63 end 64 end When it's popped and expanded it looks around and fi nds a matched set of keywords using the scanner.
319. > 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 61 end.join 62 end 63 end 64 end The code is valid so it's hidden. The expanded block is put back on the frontier.
320. sweep(block: block, name: "comments") end # Main search loop def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb That's pretty much how the algorithm for search works. It keeps looping, expanding and adding until all syntax errors are found
321. sweep(block: block, name: "comments") end # Main search loop def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb Once we've removed all syntax errors, we have to fi nd our problematic code.
322. def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self end end code_search.rb The frontier can hold more than one syntax error.
323. # Example: # # combination([:a, :b, :c, :d]) # # => [[:a], [:b], [:c], [:d], # [:a, :b], [:a, :c], [:a, :d], # [:b, :c], [:b, :d], [:c, :d], # [:a, :b, :c], [:a, :b, :d], [:a, :c, :d], # [:b, :c, :d], # [:a, :b, :c, :d]] def self.combination(array) guesses = [] 1.upto(array.length).each do |size| guesses.concat(array.combination(size).to_a) end guesses end # Given that we know our syntax error exists somewhere in our frontier, we want to find code_frontier.rb We want to iterate over all of the invalid blocks stored in the frontier to fi nd the shortest possible order that gives us a valid document.
324. # Example: # # combination([:a, :b, :c, :d]) # # => [[:a], [:b], [:c], [:d], # [:a, :b], [:a, :c], [:a, :d], # [:b, :c], [:b, :d], [:c, :d], # [:a, :b, :c], [:a, :b, :d], [:a, :c, :d], # [:b, :c, :d], # [:a, :b, :c, :d]] def self.combination(array) guesses = [] 1.upto(array.length).each do |size| guesses.concat(array.combination(size).to_a) end guesses end # Given that we know our syntax error exists somewhere in our frontier, we want to find code_frontier.rb To do that we create a combination of all possible blocks
325. end # Given that we know our syntax error exists somewhere in our frontier, we want to find # the smallest possible set of blocks that contain all the syntax errors def detect_invalid_blocks self.class.combination(@frontier.select(&:invalid?)).detect do |block_array| holds_all_syntax_errors?(block_array) end || [] end end end code_frontier.rb Then one at a time we iterate through all the combinations
326. guesses end # Given that we know our syntax error exists somewhere in our frontier, we want to find # the smallest possible set of blocks that contain all the syntax errors def detect_invalid_blocks self.class.combination(@frontier.select(&:invalid?)).detect do |block_array| holds_all_syntax_errors?(block_array) end || [] end end end code_frontier.rb And check each one to see if we hold all the syntax errors. The smallest successful combination is returned
327. Timeout.timeout(timeout) do record_dir ||= ENV["DEBUG"] ? "tmp" : nil search = CodeSearch.new(source, record_dir: record_dir).call end blocks = search.invalid_blocks DisplayInvalidBlocks.new( blocks: blocks, filename: filename, terminal: terminal, code_lines: search.code_lines, invalid_obj: invalid_type(source), io: io ).call dead_end/internals.rb Here's where our search started.
328. Timeout.timeout(timeout) do record_dir ||= ENV["DEBUG"] ? "tmp" : nil search = CodeSearch.new(source, record_dir: record_dir).call end blocks = search.invalid_blocks DisplayInvalidBlocks.new( blocks: blocks, filename: filename, terminal: terminal, code_lines: search.code_lines, invalid_obj: invalid_type(source), io: io ).call dead_end/internals.rb Then the invalid blocks were found
329. Now we want to decorate the output
330. Timeout.timeout(timeout) do record_dir ||= ENV["DEBUG"] ? "tmp" : nil search = CodeSearch.new(source, record_dir: record_dir).call end blocks = search.invalid_blocks DisplayInvalidBlocks.new( blocks: blocks, filename: filename, terminal: terminal, code_lines: search.code_lines, invalid_obj: invalid_type(source), io: io ).call dead_end/internals.rb We pass it to a pretty printer display
331. lines = CaptureCodeContext.new( blocks: @blocks, code_lines: @code_lines ).call DisplayCodeWithLineNumbers.new( lines: lines, terminal: @terminal, highlight_lines: @invalid_lines ).call display_invalid_blocks.rb Here's where we take care of the last of our gotchas
332. lines = CaptureCodeContext.new( blocks: @blocks, code_lines: @code_lines ).call DisplayCodeWithLineNumbers.new( lines: lines, terminal: @terminal, highlight_lines: @invalid_lines ).call display_invalid_blocks.rb This capture code context handles the ambiguities we talked about before
333. > ❯ 1 class Dog Gotcha: ambiguity 🚫 It knows about our ambiguous <click> edge cases that we talked about before
334. > ❯ 1 class Dog ❯ 2 def sit ❯ 3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity It goes back through the hidden code
335. > ❯ 1 class Dog ❯ 2 def sit ❯ 3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity
336. > ❯ 1 class Dog ❯ 2 def sit ❯ 3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity Work backwards and fi nd an unmatched keyword
337. > ❯ 1 class Dog ❯ 10 def bark ❯ 12 end Gotcha: ambiguity And transforms it
338. 🎂 Piece of cake
339. > DeadEnd: Missing `end` detected This code has a missing `end`. Ensure that all syntax keywords (`def`, `do`, etc.) have a matching `end`. file: /Users/rschneeman/Documents/projects/dead_end/spec/fixtures/this_project_extra_def.rb.txt simplified: 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename ❯ 36 def filename ❯ 38 def code_with_filename ❯ 45 end 63 end 64 end After all three steps, cleaning, searching, formatting here's the output
340. dead_end That is dead end <click> in a nutshell *laugh*
341. Gem `error_highlight` in Ruby 3.1 Beyond dead_end there's an amazing gem being added to Ruby 3.1 called error_highlight. It shows you which method got a no method error. I love this introduction
342. Gem `error_highlight` in Ruby 3.1 The error highlight gem was created by you-ssssss-kay. He goes by mam-eh on GitHub and gave an amazing talk at a conference I used to run called keep ruby weird.
343. Rust v. Ruby With error highlight and dead end I also want to touch on community values.
344. Rust v. Ruby I've been writing Rust code and their community takes a very aggressive stance towards error messages. It's not enough for them to say there's a problem. They also want to accurately suggest how to fi x the problem
345. Rust v. Ruby I opened this issue against rustlang and in under a month it was addressed to improve the error output.
346. Rust v. Ruby In general the Rust community treats a user experience problem as a bug and error output is a fi rst class feature. If our community wanted we could invest more in errors
347. Dead end isn't that solution, but it's a start. You can add the dead_end gem to your project to try it today and give me feedback on what works and what doesn't.
348. 📕 HowTo Open Source .dev You can pre-order my book on open source contribution by visiting how to open source dot dev
349. You can sign up to triage open source issues at code triage dot com
350. Lexing Parsing Syntax SyntaxErrors AI Pathfinding and goal seeking Today we talked about lexing, parsing, syntax syntax errors, AI, path fi nding and goal seeking
351. But, the technical pieces are not what I want you to remember.
352. - The important part is that everyone sitting in this room is the future of developer tooling. Programming is inherently di ffi cult, but it doesn't mean our tools can't help us.
353. One of the best ways to judge a system is to see how it fails. When we can give care and grace and beauty to our failure modes then we can create experiences that delight us. Experiences that teach us. Experiences that elevate our code and our consciousness.
354. - A syntax error doesn't have to be an ending...it can be a beginning. To a beautiful programming story
355. How To Open Source .dev - My name is Richard Schneeman - I'm writing a book at how to open source dot dev <click> - I created <click> CodeTriage.com the easiest way to get started contributing to OpenSource - <click> - Wait for "bye bye" - Zoom wave everyone
357. Questions?