Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

[RubyConf] Beware the Dreaded Dead End

[RubyConf] Beware the Dreaded Dead End

Nothing stops a program from executing quite as fast as a syntax error. After years of “unexpected end” in my dev life, I decided to “do” something about it. In this talk we'll cover lexing, parsing, and indentation informed syntax tree search that power that dead_end Ruby library.

I also gave this talk at RubyKaigi. I recommend this version, it is more up-to-date and covers specific details and "gotchas" on the problem space.

Richard Schneeman

October 28, 2021
Tweet

More Decks by Richard Schneeman

Other Decks in Technology

Transcript

  1. Beware the Dreaded Dead End!!! By @schneems Hello everyone. My

    name is Richard Schneeman and I want to talk to you about the scariest thing a Ruby programmer can face. You hear that?
  2. Beware the Dreaded Dead End!!! By @schneems Wow, I can

    hardly believe that the those dinosaurs from RubyKaigi found me all the way over here, I thought I escaped them already. If you've seen that talk, know that this one has some new content.
  3. Beware the Dreaded Dead End!!! By @schneems While dinosaurs are

    scary. There's something scarier. Its..
  4. > # <= routes.rb:121: syntax error, unexpected end-of-input, expecting `end’

    Rails.application.routes.draw do constraints -> { Rails.application.config.non_production } do namespace :foo do resource :bar end end constraints -> { Rails.application.config.non_production } do namespace :bar do resource :baz end end namespace :admin do resource :session match "/foobar(*path)", via: :all, to: redirect { |_params, req| uri = URI(req.path.gsub("foobar", "foobaz")) uri.query = req.query_string.presence uri.to_s } end Syntax errors <click: laughter>
  5. > Rails.application.routes.draw do constraints -> { Rails.application.config.non_production } do namespace

    :foo do resource :bar end end constraints -> { Rails.application.config.non_production } do namespace :bar do resource :baz end end namespace :admin do resource :session match "/foobar(*path)", via: :all, to: redirect { |_params, req| uri = URI(req.path.gsub("foobar", "foobaz")) uri.query = req.query_string.presence uri.to_s } end # <= routes.rb:121: syntax error, unexpected end-of-input, expecting `end’ Just look at this unexpected syntax error. It's horrifying. Where's the problem?
  6. > RSpec.describe Cutlass::BashResult do it "preserves stdout, stderr, and status"

    stdout = SecureRandom.hex(16) stderr = SecureRandom.hex(16) status = 0 result = BashResult.new( stdout: stdout, stderr: stderr, status: status ) expect(result.stdout).to eq(stdout) expect(result.stderr).to eq(stderr) expect(result.status).to eq(status) end end # <= result_spec:19 syntax error, unexpected end-of-input, expecting `end’ Here's some di ff erent code. Wanna guess where Ruby think the problem is? <click> On the last line, nope. This is frustrating
  7. > RSpec.describe Cutlass::BashResult do it "preserves stdout, stderr, and status"

    stdout = SecureRandom.hex(16) stderr = SecureRandom.hex(16) status = 0 result = BashResult.new( stdout: stdout, stderr: stderr, status: status ) expect(result.stdout).to eq(stdout) expect(result.stderr).to eq(stderr) expect(result.status).to eq(status) end end # <= result_spec:19 syntax error, unexpected end-of-input, expecting `end’ Instead of this
  8. > 3 module Cutlass 4 RSpec.describe Cutlass::BashResult do ❯ 5

    it "preserves stdout, stderr, and status" ❯ 19 end 21 it "success?" do 52 end 53 end 54 end This code has an unmatched `end`. Ensure that all `end` lines in your code have a matching syntax keyword (`def`, `do`, etc.) What if we got this? Do you see the error now? Doesn't line 5 look suspicious, like maybe it's missing a do.
  9. 🎂 Have you ever seen a cooking show where they

    show o ff the fi nal product? I want to show o ff dead end before we get too far.
  10. module SyntaxErrorSearch # Used for formatting invalid blocks class DisplayInvalidBlocks

    attr_reader :filename def initialize(block_array, io: $stderr, filename: nil) @filename = filename @io = io @blocks = block_array @lines = @blocks.map(&:lines).flatten @digit_count = @lines.last.line_number.to_s.length @code_lines = @blocks.first.code_lines @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} end def call @io.puts <<~EOM 64: syntax error, unexpected end-of-input, expecting `end' 📕 ❌ This is real ruby source code with a syntax error in it. It cannot parse
  11. 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string =

    String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s 58 string << "\e[0m" 59 string 60 end 61 end.join 62 end 63 end 64 end <click>
  12. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53

    else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end
  13. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53

    else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end
  14. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53

    else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end
  15. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53

    else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end
  16. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53

    else 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end
  17. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53

    else ❯ 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end When a valid code block is found dead_end safely removes it
  18. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53

    else 60 end 61 end.join 62 end 63 end 64 end 📕 ❌ After each step in the search, dead end re-evaluates the whole document to see if it's parsable yet. <click> parsing failed, we need to keep looking
  19. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? ❯ 52 "#{number.to_s}#{line}" 53 else 60 end 61 end.join 62 end 63 end 64 end
  20. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 53 else 60 end 61 end.join 62 end 63 end 64 end 📕 ❌
  21. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) ❯ 51 if line.empty? ❯ 53 else ❯ 60 end 61 end.join 62 end 63 end 64 end
  22. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 61 end.join 62 end 63 end 64 end 📕 ❌
  23. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 47 def code_with_lines 48 @code_lines.map do |line| ❯ 49 next if line.hidden? ❯ 50 number = line.line_number.to_s.rjust(@digit_count) 61 end.join 62 end 63 end 64 end
  24. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 61 end.join 62 end 63 end 64 end 📕 ❌
  25. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 47 def code_with_lines ❯ 48 @code_lines.map do |line| ❯ 61 end.join 62 end 63 end 64 end
  26. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 47 def code_with_lines 62 end 63 end 64 end 📕 ❌
  27. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 ❯ 47 def code_with_lines ❯ 62 end 63 end 64 end
  28. 40 string << "```\n" 41 string << "#".rjust(@digit_count) + "

    filename: #{filename}\n\n" if filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 63 end 64 end 📕 ❌
  29. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

    15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 63 end 64 end
  30. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

    15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
  31. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

    15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
  32. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

    15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename ❯ 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
  33. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

    15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" ❯ 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename ❯ 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
  34. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

    15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") ❯ 40 string << "```\n" ❯ 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename ❯ 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
  35. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

    15 end 16 35 36 def filename 37 38 def code_with_filename ❯ 39 string = String.new("") ❯ 40 string << "```\n" ❯ 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename ❯ 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
  36. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

    15 end 16 35 36 def filename 37 38 def code_with_filename 45 end 46 63 end 64 end 📕 ❌
  37. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array 10 @lines = @blocks.map(&:lines).flatten 11 @digit_count = @lines.last.line_number.to_s.length 12 @code_lines = @blocks.first.code_lines 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
  38. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array 10 @lines = @blocks.map(&:lines).flatten 11 @digit_count = @lines.last.line_number.to_s.length 12 @code_lines = @blocks.first.code_lines 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
  39. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array 10 @lines = @blocks.map(&:lines).flatten 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
  40. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
  41. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array ❯ 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
  42. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io ❯ 9 @blocks = block_array ❯ 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
  43. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename ❯ 8 @io = io ❯ 9 @blocks = block_array ❯ 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
  44. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) ❯ 7 @filename = filename ❯ 8 @io = io ❯ 9 @blocks = block_array ❯ 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
  45. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 15 end 16 35 36 def filename 37 38 def code_with_filename 45 end 46 63 end 64 end 📕 ❌
  46. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 15 end 16 35 36 def filename 37 ❯ 38 def code_with_filename ❯ 45 end 46 63 end 64 end
  47. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 15 end 16 35 36 def filename 37 46 63 end 64 end 📕 ❌
  48. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    ❯ 6 def initialize(block_array, io: $stderr, filename: nil) ❯ 15 end 16 35 36 def filename 37 46 63 end 64 end
  49. 📕 ❌ 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader

    :filename 5 16 35 36 def filename 37 46 63 end 64 end
  50. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename ❯

    5 ❯ 16 ❯ 35 ❯ 36 def filename ❯ 37 ❯ 46 63 end 64 end
  51. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 63

    end 64 end 📕 ✅ The document is checked. <click> The parser reports that this very minimal document is now parsable.
  52. Syntax O 🥰🥰🥰 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4

    attr_reader :filename 63 end 64 end 📕 ✅ Once this happens dead_end has found a way to transform the document to be valid. It can stop searching the document and instead search the invalid code blocks that it has stored
  53. DeadEnd: Missing `end` detected This code has a missing `end`.

    Ensure that all syntax keywords (`def`, `do`, etc.) have a matching `end`. file: /Users/rschneeman/Documents/projects/dead_end/spec/fixtures/this_project_extra_def.rb.txt simplified: 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename ❯ 36 def filename ❯ 38 def code_with_filename ❯ 45 end 63 end 64 end This is the actual output the dead end gem returns on that fi le. The issue is on line 36 there's a missing end statement
  54. By the time you see this slide dead_end will have

    surpassed half a million downloads
  55. Ruby 3.2? I am also talking with Ruby core to

    get dead_end into Ruby directly. We are targeting that integration for 2022 release of Ruby 3.2
  56. > 3 module Cutlass 4 RSpec.describe Cutlass::BashResult do ❯ 5

    it "preserves stdout, stderr, and status" ❯ 19 end 21 it "success?" do 52 end 53 end 54 end DeadEnd: Unmatched `end` detected When you miss a keyword like "if", "do" or "def". <click> Dead end fi nds the problem.
  57. > 1 module Cutlass 2 RSpec.describe Cutlass::BashResult do ❯ 3

    it "preserves stdout, stderr, and status" do 5 it "success?" do 6 end 7 end 8 end DeadEnd: Missing `end` detected Even when you miss an end keyword, <click> dead end fi nds the problem
  58. > 1 class Cat 2 def to_json ❯ 3 hash

    = { ❯ 4 name: name, ❯ 5 fur_color: fur_color, ❯ 6 type: 'Cat', ❯ 7 better_than_dog: false ❯ 8 hash.to_json 9 end 10 end DeadEnd: Unmatched `}` character detected When you miss a curly bracket <click> dead end fi nds the problem
  59. > 1 class Animal 2 def self.ones_my_three_year_old_likes ❯ 3 array

    = [ ❯ 4 Dog.new, ❯ 5 Lion.new, ❯ 6 Tiger.new, ❯ 7 Aligator.new, 10 return array 11 end 12 end DeadEnd: Unmatched `[` detected When you miss a square bracket <click> dead end fi nds the problem
  60. > 1 class Cat 2 def meow ❯ 3 Animal.call

    do |a ❯ 5 end 6 end 7 end DeadEnd: Unmatched `|` character detected Have a problem with a missing pipe character? Dead end. Finds. <click> The. <click> Problem <click> <Pause> Have a problem with a missing family member in a Korean thriller drama?
  61. > Unfortunately. Dead end cannot fi nd that problem. <click:

    laughter>. Okay, wow that current event reference got really dark. Let's rewind a try again.
  62. > 1 class Cat 2 def meow ❯ 3 Animal.call

    do |a ❯ 5 end 6 end 7 end DeadEnd: Unmatched `|` character detected ? ? ? Unfortunately. <click: laughter> Dead end cannot fi nd that problem. Much better. Remember crocodile Loki? That was 2021, this year is lasting forever!
  63. $ dead_end bad.rb --record tmp/ You can follow the algorithm

    yourself by running the CLI with the record fl ag
  64. Internals Finally we'll crack open the cover and see how

    it works at a low level. Warning: We will get technical. We will cover a lot of code.
  65. @schneems I go by schneems on the internet. If you

    forget how to pronounce my name you can go to
  66. I Created CodeTriage which is a platform for learning how

    to contribute to open source. To date I've got over 60 thousand developers signed up for CodeTriage
  67. 📕 HowTo Open Source .dev I'm writing a book on

    how to contribute to open source as well. It's for developers who want to start contributing. Developers who got stuck, and developers looking for ways to sustain contributions. I'm not ready to release it yet, but you can sign up for a pre-order on HowToOpenSource dot dev Also you can sign up for Code Triage dot com and I will email members when the book is ready.
  68. Heroku When I'm not working on open source or teaching

    people to open source with CodeTriage. I like to get paid. I work at <click> Heroku
  69. Heroku Right now I'm working on salesforce functions. It's an

    easy way to work with the data you've got in salesforce using the languages you love. Right now we support Java and JavaScript languages. We'll be rolling out Ruby support later. If you use love Ruby and use Salesforce I would love to hear from you. My DMs on Twitter are open. Also people tell me
  70. NoMemoryError ScriptError LoadError SyntaxError SecurityError SignalException KeyError StandardError ArgumentError EncodingError

    IOError EOFError I am Exceptional i'm an exceptional programmer... <click> My programs generate a lot of exceptions <click: laugh>
  71. Syntax errors Syntax errors. Let's start from the beginning what

    is a syntax error? Why are syntax errors so hard to understand?
  72. > while b != 0 if a > b a

    = a - b else b = b - a end end return a This code worked wonderfully comparing a, and b all day.
  73. > while b != 0 if a > b a

    = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a When Ruby parses this code it will convert it into an abstract syntax tree. <click> <point> this code generates <point> that tree. The tree was beautiful and Ruby's parser looked upon it with happiness
  74. > while b != 0 if a > b a

    = a - b else b = b - a while b != 0 if a > b a = a - b else b = b - a end end return a 😈 While the code was good a stranger <click> came upon the land. This stranger had a secret power, behold
  75. > while b != 0 if a > b a

    = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a 😈 With one key the stranger used their power <click>
  76. > while b != 0 # if a > b

    a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a 😈 to transform the code <click>
  77. > while b != 0 # if a > b

    a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a 😈 A critical line had been commented out. Without that line, the tree was no longer whole
  78. > while b != 0 # if a > b

    a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a 😈 Huge sections of code are no longer reachable. The code no longer parses. And our parser is sad.
  79. > while b != 0 # if a > b

    a = a - b else b = b - a end end return a while b != 0 a = a - b else b = b - a 😈 With that the stranger left <click> and behind them stood
  80. > while b != 0 # if a > b

    a = a - b else b = b - a end end return a while b != 0 a = a - b else b = b - a Syntax error A syntax error
  81. Syntax errors A syntax error occurs when the parser cannot

    build a valid syntax tree. That explains what they are, but why are they di ffi cult to understand?
  82. > module Cutlass def call 1, 2, 3, 4, 5]

    end end syntax error, unexpected ',', expecting end-of-input (SyntaxError) When the parser tries to parse this code it fi nds a <click> Syntax Error. Here our developer forgot a bracket. As the parser is building the tree, it hit an error because it wasn't expecting the comma
  83. > syntax error, unexpected ',', expecting end-of-input (SyntaxError) module Cutlass

    def call 1, 2, 3, 4, 5] end end Ruby's parser has rules. It knows that after a method de fi nition it should look for something such as
  84. > syntax error, unexpected ',', expecting end-of-input (SyntaxError) module Cutlass

    def call 1, 2, 3, 4, 5] end end Like a return or an end. Or a variable assignment. Instead what did the parser fi nd fi nd? It found a number and a comma.
  85. > module Cutlass def call 1, 2, 3, 4, 5]

    end end syntax error, unexpected ',', expecting end-of-input (SyntaxError) 🚫 That comma violated ruby's parser rules. Do not pass go, do not collect 200 <click>
  86. > module Cutlass def call 1, 2, 3, 4, 5]

    end end syntax error, unexpected ',', expecting end-of-input (SyntaxError) The error <click> isn't caused by a comma, it's caused by the developer missing a bracket. Essentially, the location the parse error occurs isn't always where developer made a mistake
  87. > module Cutlass defcall end end Here's code with a

    syntax error. The developer forgot a space after the def
  88. > module Cutlass defcall end end syntax error, unexpected `end',

    expecting end-of-input Ruby thinks the error is at the last line. <click> right here. Why?
  89. > module Cutlass defcall end end syntax error, unexpected `end',

    expecting end-of-input Welllll... When we start parsing Ruby sees a module de fi nition, A module is a keyword that requires an end so Ruby starts looking for a matching end.
  90. > module Cutlass defcall end end syntax error, unexpected `end',

    expecting end-of-input On the next line it sees this combination of characters and thinks it's a method call
  91. > module Cutlass defcall end end syntax error, unexpected `end',

    expecting end-of-input Then it sees this end and thinks it matches with the module. Our parser hasn't raised a syntax error because it doesn't see a problem yet.
  92. > module Cutlass defcall end end syntax error, unexpected `end',

    expecting end-of-input Finally it gets to our second end, but there's no thing to match with. A syntax error is raised. <click>
  93. > end syntax error, unexpected `end', expecting end-of-input Right now

    this is the error you get from ruby on this code. It's not helpful to a human because deleting that end won't solve anything.
  94. parser problems != human problems Another way to put this,

    is that parse errors are di ff erent from human errors.
  95. $ gem install dead_end dead_end's goal is to turn that

    parser problem into something a human can see and instantly recognize as an issue.
  96. require "ripper" source = <<~EOM module Cutlass defcall end end

    EOM Ripper.new(source).tap(&:parse).error? Ripper can evaluate code and tell us if there's a syntax error or not. We saw this code before
  97. require "ripper" source = <<~EOM module Cutlass defcall end end

    EOM Ripper.new(source).tap(&:parse).error? , there's a syntax error here
  98. require "ripper" source = <<~EOM module Cutlass defcall end end

    EOM Ripper.new(source).tap(&:parse).error? # => true When we run ripper it says there's an error
  99. require "ripper" source = <<~EOM module Cutlass def call end

    end EOM Ripper.new(source).tap(&:parse).error? # => false When we fi x the error, ripper tells us it's fi xed too. <click>
  100. 🙇 If we could fi x the source errors then

    ripper would tell us, but that's too hard to automate. Can we do something simpler?
  101. require "ripper" source = <<~EOM module Cutlass defcall end end

    EOM Ripper.new(source).tap(&:parse).error? # => true Back to our code with a syntax error. Instead of fi xing it. What if we...commented it out?
  102. require "ripper" source = <<~EOM module Cutlass # defcall end

    end EOM Ripper.new(source).tap(&:parse).error? # => true That didn't work, let's keep going.
  103. require "ripper" source = <<~EOM module Cutlass # defcall #

    end end EOM Ripper.new(source).tap(&:parse).error? # => false Oooh, nice <click> ripper can tell us when we've commented out the source code causing the problem
  104. # defcall # end Ripper.new(source).tap(&:parse).error? # => false Now if

    we only look at the part that was commented out, dead_end can show you where the issue occured.
  105. > while b != 0 a = a - b

    else b = b - a end end return a while b != 0 a = a - b else b = b - a 😈 Ruby knows there's a syntax error here, but it doesn't know where.
  106. > while b != 0 a = a - b

    else b = b - a end end return a while b != 0 a = a - b else b = b - a 😈 It will tell you
  107. > while b != 0 a = a - b

    else b = b - a end end return a syntax error, unexpected `end', expecting end-of-input That the problem is this line. We know that's not helpful. What does dead end do?
  108. > while b != 0 a = a - b

    else b = b - a end end return a while b != 0 a = a - b else b = b - a Dead end uses indentation and lexical parsing to deconstruct the source code from the outside in <click>
  109. > while b != 0 a = a - b

    else # b = b - a end end return a while b != 0 a = a - b else b = b - a Commenting out this code removes a node from our tree
  110. > while b != 0 # a = a -

    b else # b = b - a end end return a while b != 0 a = a - b else Commenting out this code removes a node from our tree, but it's still invalid
  111. > while b != 0 # a = a -

    b # else # b = b - a # end end return a while b != 0 else Finally here we have a valid tree. All of the orphaned syntax nodes are gone
  112. > 1 while b != 0 ❯ 4 else ❯

    6 end 7 end This code has an unmatched `end`. Ensure that all `end` lines in your code have a matching syntax keyword (`def`, `do`, etc.) Here's the output for that code. It shows that the else and end are missing an if statement.
  113. I bet you're thinking wow, that was easy. So all

    he did was comment out code based on indentation?
  114. Gotcha #1 We already know the developer has a syntax

    error. We can't assume they've done everything else perfectly. 
 We cannot assume perfect indentation
  115. > describe "things" do it "valid" do expect(the).to eq(unexpected) end

    it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything Our syntax error is on this line. It is missing keyword do
  116. > describe "things" do it "valid" do expect(the).to eq(unexpected) end

    it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything Without it, the parser has an extra end here
  117. > describe "things" do it "valid" do expect(the).to eq(unexpected) end

    it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything If we only look at indentation then we would remove this line
  118. > describe "things" do it "valid" do end it "forgot

    a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything Then this mis-indented end
  119. > describe "things" do it "valid" do it "forgot a

    keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything 📕 ✅ With the end gone, the document parses<click>. Why?
  120. > describe "things" do it "valid" do it "forgot a

    keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything 📕 ✅ Because this keyword now matches with
  121. > describe "things" do it "valid" do it "forgot a

    keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything 🚫 This end It's not what we wanted, and didn't capture the source of the syntax error <click>
  122. 🙇 How can we fi x it? Instead of only

    looking at indentation we also look at lex output
  123. source = <<~EOM it "valid" do expect(the).to eq(unexpected) end EOM

    Ripper.lex(source) Ruby ships with a lexer
  124. source = <<~EOM it "valid" do expect(the).to eq(unexpected) end EOM

    Ripper.lex(source) # => [[[1, 0], :on_ident, "it", CMDARG], [[1, 2], :on_sp, " ", CMDARG], [[1, 3], :on_tstring_beg, "\"", CMDARG], [[1, 4], :on_tstring_content, "valid", CMDARG], [[1, 9], :on_tstring_end, "\"", END], [[1, 10], :on_sp, " ", END], [[1, 11], :on_kw, "do", BEG], [[1, 13], :on_ignored_nl, "\n", BEG], Here's what it looks like
  125. # => [[[1, 0], :on_ident, "it", CMDARG], [[1, 2], :on_sp,

    " ", CMDARG], [[1, 3], :on_tstring_beg, "\"", CMDARG], [[1, 4], :on_tstring_content, "valid", CMDARG], [[1, 9], :on_tstring_end, "\"", END], [[1, 10], :on_sp, " ", END], [[1, 11], :on_kw, "do", BEG], [[1, 13], :on_ignored_nl, "\n", BEG], [[2, 0], :on_sp, " ", BEG], [[2, 6], :on_ident, "expect", CMDARG], [[2, 12], :on_lparen, "(", BEG|LABEL], [[2, 13], :on_ident, "the", ARG], [[2, 16], :on_rparen, ")", ENDFN], [[2, 17], :on_period, ".", DOT], [[2, 18], :on_ident, "to", ARG], DO NOT CLICK, auto advance
  126. # => [[[1, 0], :on_ident, "it", CMDARG], [[1, 2], :on_sp,

    " ", CMDARG], [[1, 3], :on_tstring_beg, "\"", CMDARG], [[1, 4], :on_tstring_content, "valid", CMDARG], [[1, 9], :on_tstring_end, "\"", END], [[1, 10], :on_sp, " ", END], [[1, 11], :on_kw, "do", BEG], [[1, 13], :on_ignored_nl, "\n", BEG], [[2, 0], :on_sp, " ", BEG], [[2, 6], :on_ident, "expect", CMDARG], [[2, 12], :on_lparen, "(", BEG|LABEL], [[2, 13], :on_ident, "the", ARG], [[2, 16], :on_rparen, ")", ENDFN], [[2, 17], :on_period, ".", DOT], [[2, 18], :on_ident, "to", ARG], It tells you the code contains a do keyword
  127. [[2, 12], :on_lparen, "(", BEG|LABEL], [[2, 13], :on_ident, "the", ARG],

    [[2, 16], :on_rparen, ")", ENDFN], [[2, 17], :on_period, ".", DOT], [[2, 18], :on_ident, "to", ARG], [[2, 20], :on_sp, " ", ARG], [[2, 21], :on_ident, "eq", ARG], [[2, 23], :on_lparen, "(", BEG|LABEL], [[2, 24], :on_ident, "unexpected", ARG], [[2, 34], :on_rparen, ")", ENDFN], [[2, 35], :on_sp, " ", ENDFN], [[2, 36], :on_nl, "\n", BEG], [[3, 0], :on_sp, " ", BEG], [[3, 2], :on_kw, "end", END], [[3, 5], :on_nl, "\n", BEG]] DO NOT CLICK, auto advance
  128. [[2, 12], :on_lparen, "(", BEG|LABEL], [[2, 13], :on_ident, "the", ARG],

    [[2, 16], :on_rparen, ")", ENDFN], [[2, 17], :on_period, ".", DOT], [[2, 18], :on_ident, "to", ARG], [[2, 20], :on_sp, " ", ARG], [[2, 21], :on_ident, "eq", ARG], [[2, 23], :on_lparen, "(", BEG|LABEL], [[2, 24], :on_ident, "unexpected", ARG], [[2, 34], :on_rparen, ")", ENDFN], [[2, 35], :on_sp, " ", ENDFN], [[2, 36], :on_nl, "\n", BEG], [[3, 0], :on_sp, " ", BEG], [[3, 2], :on_kw, "end", END], [[3, 5], :on_nl, "\n", BEG]] Also an end
  129. > describe "things" do it "valid" do expect(the).to eq(unexpected) end

    it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything It starts at the same place
  130. > describe "things" do it "valid" do end it "forgot

    a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything Now it needs to decide where to expand, it looks down and sees an `end`
  131. > describe "things" do it "valid" do end it "forgot

    a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything dead_end knows if it removes this end, it might trigger a false positive, so it searches up for keywords. It fi nds...
  132. > describe "things" do it "valid" do end it "forgot

    a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything This do keyword
  133. > describe "things" do it "forgot a keyword -> "

    expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything They are both safely removed
  134. > file: /private/tmp/scratch.rb simplified: 1 describe "things" do ❯ 6

    it "forgot a keyword -> " ❯ 8 end 9 end Gotcha: Indentation isn't everything The algorithm continues until the problem is found
  135. > class User puts "hello" end def schneems User. where(name:

    'schneems'). first end end Gotcha: Removing the wrong line Our syntax error is on these lines.
  136. > class User puts "hello" end def schneems User. where(name:

    'schneems'). first end end Gotcha: Removing the wrong line We are missing a whole method de fi nition line
  137. > class User puts "hello" end def schneems User. where(name:

    'schneems'). first end end Gotcha: Removing the wrong line If we tried our indentation comment out strategy we would start at the largest indentation
  138. > class User puts "hello" end def schneems User. where(name:

    'schneems'). first end end Gotcha: Removing the wrong line here
  139. > class User puts "hello" end def schneems User. where(name:

    'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ And then dead_end would stop looking because the document is now parsable <click>
  140. > class User puts "hello" end def schneems User. where(name:

    'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ It will help if we reformat it to show what Ruby thinks you're trying to do. It thinks
  141. > class User puts "hello" end def schneems User. where(name:

    'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ That this user class
  142. > class User puts "hello" end def schneems User. where(name:

    'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ Is matched with the fi rst end.
  143. > class User puts "hello" end def schneems User. where(name:

    'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ Is matched with the fi rst end.
  144. > class User puts "hello" end def schneems User. where(name:

    'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ It thinks that this end, is actually a method due to the trailing dot
  145. > class User puts "hello" end def schneems User. where(name:

    'schneems').end end Gotcha: Removing the wrong line 📕 ✅ Finally it thinks the last end matches with our method de fi nition
  146. > class User puts "hello" end def schneems User. where(name:

    'schneems').end end Gotcha: Removing the wrong line 📕 ✅ Finally it thinks the this method de fi nition
  147. > class User puts "hello" end def schneems User. where(name:

    'schneems').end end Gotcha: Removing the wrong line 📕 ✅ 🚫 Matches with this last end. <click> It's a problem because none of this is what the user intended and we didn't highlight the source of the problems
  148. > source = <<~EOM Hello World From A heredoc EOM

    Gotcha: Removing the wrong line Heredocs cause problems
  149. > source = <<~EOM Hello World From A heredoc EOM

    Gotcha: Removing the wrong line If you evaluate this line in isolation, it looks like a constant. But if you remove it
  150. > source = <<~EOM Hello World From A heredoc Gotcha:

    Removing the wrong line 🚫 Then you've introduced a syntax error<click>
  151. > it "should" \ "handle trailing slash" do expect(the).to eq(unexpected)

    end Gotcha: Removing the wrong line Trailing slashes cause problems.
  152. > it "should" \ "handle trailing slash" do expect(the).to eq(unexpected)

    end Gotcha: Removing the wrong line If you start in the middle
  153. > it "should" \ "handle trailing slash" do end Gotcha:

    Removing the wrong line Now expand out using lex detection
  154. > it "should" \ "handle trailing slash" do end Gotcha:

    Removing the wrong line Find the end
  155. > it "should" \ "handle trailing slash" do end Gotcha:

    Removing the wrong line Find the matching keyword.
  156. > it "should" \ Gotcha: Removing the wrong line 🚫

    Now remove them.<click> This is not valid ruby
  157. source = <<~EOM User. where(name: 'schneems'). first EOM Ripper.lex(source) #

    => [[[1, 0], :on_const, "User", CMDARG], [[1, 4], :on_period, ".", DOT], [[1, 5], :on_ignored_nl, "\n", DOT], [[2, 0], :on_sp, " ", DOT], [[2, 2], :on_ident, "where", ARG], [[2, 7], :on_lparen, "(", BEG|LABEL], [[2, 8], :on_label, "name:", ARG|LABELED], [[2, 13], :on_sp, " ", ARG|LABELED], We can use the lexer to detect these speci fi c edge cases. <click> Then we can use that information to combine all logically consecutive lines together so there's no way to remove one without removing all of them
  158. > it "should" \ "handle trailing slash" do expect(the).to eq(unexpected)

    end Gotcha: Removing the wrong line It's hard to show visually, but it essentially would looks like re-writing source code. This trailing slash would become
  159. > class User puts "hello" end def schneems User. where(name:

    'schneems'). first end end Gotcha: Removing the wrong line Our method chain would become
  160. > class User puts "hello" end def schneems User.where(name: 'schneems').first

    end end Gotcha: Removing the wrong line Our method chain would become
  161. > source = <<~EOM Hello World From A heredoc EOM

    Gotcha: Removing the wrong line Heredocs cause problems
  162. 🎉 With all the problem lines joined together, they can

    now be safely evaluated and removed without accidentally introducing new syntax errors <click>
  163. Gotcha #3 Even with joining lines, even with lexically aware

    search, even with the perfect algorithm there is one problem we cant' avoid
  164. ambiguity Ambiguity. Yes, ambiguity is the comic sans of source

    code parsing.<click> What do I mean by ambiguity?
  165. > class Dog def sit puts "no" end def eat

    puts "munch" end def bark puts "woof" end Gotcha: ambiguity This code has a syntax error
  166. > class Dog def sit puts "no" end def eat

    puts "munch" end def bark puts "woof" end Gotcha: ambiguity It is ambiguous what the coder was trying to do with our current search algorithm. Let's zoom in
  167. > class Dog def bark puts "woof" end Gotcha: ambiguity

    Starting from the middle, if we use lexical expansion we get
  168. > ❯ 1 class Dog Gotcha: ambiguity 🚫 📕 ✅

    But this clearly isn't our problem <click>
  169. 🙇 To produce good results with invalid indentation it means

    we must make "bad" suggestions. There's a logical inverse of this problem as well.
  170. > class Dog puts "no" end def eat puts "munch"

    end def bark puts "woof" end end Gotcha: ambiguity This code has a syntax error, but this time it's at the top.
  171. > class Dog puts "no" end def eat puts "munch"

    end def bark puts "woof" end end Gotcha: ambiguity Let's focus
  172. > class Dog puts "no" end end Gotcha: ambiguity Does

    this problem look familiar? Here's the code from before
  173. > class Dog puts "no" end end Gotcha: ambiguity class

    Dog def bark puts "woof" end Before we were missing an end
  174. > class Dog puts "no" end end Gotcha: ambiguity class

    Dog def bark puts "woof" end Before we were missing an end
  175. > class Dog puts "no" end end Gotcha: ambiguity class

    Dog def bark puts "woof" end This new case is missing a keyword
  176. > class Dog puts "no" end end Gotcha: ambiguity class

    Dog def bark puts "woof" end This new case is missing a keyword
  177. > class Dog puts "no" end end Gotcha: ambiguity class

    Dog def bark puts "woof" end Where does dead end tell us the problem is?
  178. > class Dog puts "no" end end class Dog def

    bark puts "woof" end Gotcha: ambiguity ❯ 10 end ❯ 1 class Dog 🚫 Here's the result. Neither make good suggestion<click>
  179. 🙇 If we re-write our rules to let indentation take

    precedence over lexical keywords then it would make other examples fail. It's impossible to satisfy all search criteria. What if we...
  180. 😇 Don't. Because we know this ambiguity exists, we can

    compensate for it after the search is done
  181. > ❯ 1 class Dog Gotcha: ambiguity Even though this

    is all the user might see, `dead_end` still has all the contents that are hidden. It didn't just match a single line, it matched many lines, but hid most of them
  182. > ❯ 1 class Dog ❯ 2 def sit ❯

    3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity If we un-hid the rest of them, this is what it would look like. We can detect this case when our match is only one line and contains an end, then we can go back through our results
  183. > ❯ 1 class Dog ❯ 2 def sit ❯

    3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity un-hide the matched end
  184. > ❯ 1 class Dog ❯ 2 def sit ❯

    3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity Work backwards and fi nd an unmatched keyword
  185. > ❯ 1 class Dog ❯ 10 def bark ❯

    12 end Gotcha: ambiguity Then we show the user all extra context to the user and they can resolve the problem visually
  186. Gotcha #4 Syntax errors aren't always alone, sometimes they have

    friends. A document can contain multiple syntax errors
  187. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors Here' there's multiple problems
  188. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors This end matches nothing
  189. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors We are missing a square bracket
  190. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors We're missing all sorts of stu ff here
  191. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors Right here, that's code is fi ne
  192. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors That code is fi ne too
  193. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors What we don't want to happen is for our search to start at one side
  194. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors And continue until it captures everything, even the good code
  195. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors 🚫 This isn't what we want <click>
  196. 🙇 What can we do about it? We can modify

    our algorithm to hold and fi nd multiple errors.
  197. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors Instead of searching from one side to the other we start in the outside and search in
  198. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1
  199. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2
  200. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2
  201. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2
  202. > end def world puts "hello" end def foo foo

    = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2
  203. > end def world puts "hello" end def foo foo

    = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2 #3
  204. AI Now you know the gotchas. I want to start

    moving a little closer to the code. Who knows what AI is? Raise your hand, or in chat type "me"
  205. Artificial Intelligence Algorithm - Arti fi cial Intelligence is a

    fancy way of saying <click> "Algorithm" [wait for dialog] "Come on no you're not" <click>
  206. Pathfinding Credit: Factorio 'New pathfinding algorithm' - A common example

    of AI is path fi nding. You want to get from point A to point B
  207. $ gem install dead_end Dead end uses a search algorithm

    to fi nd the problem code. Which uses a variation on uniform cost search
  208. Credit @redblobgames: Introduction to the A-star Algorithm - You need

    a way to break down the problem into discrete actions that either get you closer or further from the goal. In path fi nding this would be deciding what turns to make and what roads to drive on.
  209. Credit @redblobgames: Introduction to the A-star Algorithm - The algorithm

    dead_end uses is also sometimes called dijkstra's algorithm because he invented uniform cost search.
  210. Credit @redblobgames: Introduction to the A-star Algorithm I highly recommend

    this interactive page from redblob games as an introduction to search
  211. Credit @redblobgames: Introduction to the A-star Algorithm A search algorithm

    Is good for when you know what goal you want but you're not quite sure how to get there.
  212. > while b != 0 if a > b a

    = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a Before we saw how code is represented by abstract syntax trees.
  213. Tree
 Search
 BOTVINNIK, M.M. (1984). "Computers in Chess Solving Inexact

    Search Problems" That was an important realization for me, as most search problems are represented as a tree or a graph. Here's a quick example of a famous chess problem
  214. Tree
 Search
 BOTVINNIK, M.M. (1984). "Computers in Chess Solving Inexact

    Search Problems" If you have an exact tree you can walk the tree
  215. require "ripper" source = <<~EOM module Cutlass defcall end end

    EOM Ripper.new(source).tap(&:parse).error? # => true In our case, we have bits of source code
  216. require "ripper" source = <<~EOM module Cutlass # defcall end

    end EOM Ripper.new(source).tap(&:parse).error? # => true I found that if I could give my program rules about how to create logical code blocks
  217. require "ripper" source = <<~EOM module Cutlass # defcall #

    end end EOM Ripper.new(source).tap(&:parse).error? # => false Then using ripper we can check a document without their presence
  218. "Time to cook" by Robbert van der Steeg is licensed

    under CC BY-SA 2.0 It gives that input to our search algorithm to fi nd the syntax errors.
  219. Once the syntax errors are found, the output is decorated

    with more context and given back to the user
  220. module Kernel module_function def require(file) dead_end_original_require(file) rescue SyntaxError => e

    DeadEnd.handle_error(e) end end dead_end/auto.rb Like all good Ruby libraries dead end uses monkey patching <click: laughter>. It hooks into require and friends. When a syntax error is raised, it's passed to dead_end
  221. # ... Timeout.timeout(timeout) do record_dir ||= ENV["DEBUG"] ? "tmp" :

    nil search = CodeSearch.new(source, record_dir: record_dir).call end # ... dead_end/internals.rb The source code that caused the syntax error is read to disk and then passed to a search object.
  222. clean_document.rb # frozen_string_literal: true module DeadEnd # Parses and sanitizes

    source into a lexically aware document # # Internally the document is represented by an array with each # index containing a CodeLine correlating to a line from the source code. # # There are three main phases in the algorithm: # # 1. Sanitize/format input source # 2. Search for invalid blocks # 3. Format invalid blocks into something meaninful # # This class handles the first part. # # The reason this class exists is to format input source # for better/easier/cleaner exploration. Here's the clean_document class
  223. clean_document.rb @source = source @document = CodeLine.from_source(@source) end # Call

    all of the document "cleaners" # and return self def call clean_sweep .join_trailing_slash! .join_consecutive! .join_heredoc! self end # Return an array of CodeLines in the # document def lines It handles several of the gotcha cases from before by transforming the source code
  224. clean_document.rb @source = source @document = CodeLine.from_source(@source) end # Call

    all of the document "cleaners" # and return self def call clean_sweep .join_trailing_slash! .join_consecutive! .join_heredoc! self end # Return an array of CodeLines in the # document def lines It clears out comments and stray whitespace
  225. clean_document.rb @source = source @document = CodeLine.from_source(@source) end # Call

    all of the document "cleaners" # and return self def call clean_sweep .join_trailing_slash! .join_consecutive! .join_heredoc! self end # Return an array of CodeLines in the # document def lines @document Joins lines with trailing slashes
  226. clean_document.rb @source = source @document = CodeLine.from_source(@source) end # Call

    all of the document "cleaners" # and return self def call clean_sweep .join_trailing_slash! .join_consecutive! .join_heredoc! self end # Return an array of CodeLines in the # document def lines @document Lines with chained methods
  227. clean_document.rb @source = source @document = CodeLine.from_source(@source) end # Call

    all of the document "cleaners" # and return self def call clean_sweep .join_trailing_slash! .join_consecutive! .join_heredoc! self end # Return an array of CodeLines in the # document def lines @document And heredocs
  228. "Time to cook" by Robbert van der Steeg is licensed

    under CC BY-SA 2.0 Next up we feed that to our search class
  229. code_search.rb # Searches code for a syntax error # #

    The bulk of the heavy lifting is done in: # # - CodeFrontier (Holds information for generating blocks and determining if we can stop searching) # - ParseBlocksFromLine (Creates blocks into the frontier) # - BlockExpand (Expands existing blocks to search more code # # ## Syntax error detection # # When the frontier holds the syntax error, we can stop searching # # search = CodeSearch.new(<<~EOM) # def dog # def lol # end # EOM Here's the CodeSearch class
  230. end # Main search loop def call until frontier.holds_all_syntax_errors? @tick

    += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self end code_search.rb Here's call
  231. end # Main search loop def call until frontier.holds_all_syntax_errors? @tick

    += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self code_search.rb Under every great AI algorithm is a while loop. There's a class called the frontier. It holds all the code blocks that we've generated from the source code.
  232. end # Main search loop def call sweep_heredocs sweep_comments until

    frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } code_search.rb 📕 ✅ 📕 ❌ The frontier is responsible for checking the exit condition, here's that logic
  233. > 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename

    ❯ 5 ❯ 16 ❯ 35 ❯ 36 def filename ❯ 37 ❯ 46 63 end 64 end 📕 ❌ If the highlighted lines are on the frontier
  234. > 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename

    63 end 64 end 📕 ✅ Then the entire document without those lines are checked against the parser
  235. # ... end module DeadEnd # ... def self.invalid?(source) source

    = source.join if source.is_a?(Array) source = source.to_s Ripper.new(source).tap(&:parse).error? end # ... end dead_end.rb And here's where the parser is called. <click> Remember Ripper from before?
  236. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s 58 string << "\e[0m" 59 string 60 end 61 end.join 62 end 63 end 64 end Let's walk through the logic of searching this same document we saw before. From the beginning
  237. end # Main search loop def call until frontier.holds_all_syntax_errors? @tick

    += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self end code_search.rb Since we start with an invalid source code, the frontier is empty so this is false
  238. end # Main search loop def call until frontier.holds_all_syntax_errors? @tick

    += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self end code_search.rb There's two modes of exploration. We can add new things onto the frontier, or we can expand an existing block o ff of the frontier. When do we expand?
  239. def expand? return false if @frontier.empty? # ... # Expand

    all blocks before moving to unvisited lines frontier_indent >= unvisited_indent end code_frontier.rb We can't expand a block if there's nothing there.
  240. def expand? return false if @frontier.empty? # ... # Expand

    all blocks before moving to unvisited lines frontier_indent >= unvisited_indent end code_frontier.rb We're also using indentation to decide when to expand. Essentially we want to explore the largest indentations fi rst.
  241. end # Main search loop def call sweep_heredocs sweep_comments until

    frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } code_search.rb There's nothing to expand yet so this is false
  242. end # Main search loop def call sweep_heredocs sweep_comments until

    frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } code_search.rb So we must fi rst add lines to our frontier
  243. # Parses the most indented lines into blocks that are

    marked # and added to the frontier def visit_new_blocks max_indent = frontier.next_indent_line&.indent while (line = frontier.next_indent_line) && (line.indent == max_indent) @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block| record(block: block, name: "add") block.mark_invisible if block.valid? push(block, name: "add") end end end # Given an already existing block in the frontier, expand it to see # if it contains our invalid syntax code_search.rb The code here is looking at the largest indentation, starting at the bottom and iterating up.
  244. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
  245. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
  246. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
  247. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
  248. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
  249. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else ❯ 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
  250. # Parses the most indented lines into blocks that are

    marked # and added to the frontier def visit_new_blocks max_indent = frontier.next_indent_line&.indent while (line = frontier.next_indent_line) && (line.indent == max_indent) @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block| record(block: block, name: "add") block.mark_invisible if block.valid? push(block, name: "add") end end end # Given an already existing block in the frontier, expand it to see # if it contains our invalid syntax code_search.rb This step only uses indentation.
  251. # Parses the most indented lines into blocks that are

    marked # and added to the frontier def visit_new_blocks max_indent = frontier.next_indent_line&.indent while (line = frontier.next_indent_line) && (line.indent == max_indent) @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block| record(block: block, name: "add") block.mark_invisible if block.valid? push(block, name: "add") end end end # Given an already existing block in the frontier, expand it to see code_search.rb If the block that it found is valid ruby code, it's essentially commented out.
  252. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else ❯ 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
  253. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 60 end 61 end.join 62 end 63 end 64 end
  254. # Parses the most indented lines into blocks that are

    marked # and added to the frontier def visit_new_blocks max_indent = frontier.next_indent_line&.indent while (line = frontier.next_indent_line) && (line.indent == max_indent) @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block| record(block: block, name: "add") block.mark_invisible if block.valid? push(block, name: "add") end end end # Given an already existing block in the frontier, expand it to see code_search.rb The code block is then pushed onto the frontier and the loop continues
  255. sweep(block: block, name: "comments") end # Main search loop def

    call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb We've still not found the problem we will continue
  256. sweep(block: block, name: "comments") end # Main search loop def

    call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb There's still lines with more indentation
  257. sweep(block: block, name: "comments") end # Main search loop def

    call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb We need to add them
  258. > 43 string << "```\n" 44 string 45 end 46

    47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? ❯ 52 "#{number.to_s}#{line}" 53 else 60 end 61 end.join 62 end 63 end 64 end
  259. > 43 string << "```\n" 44 string 45 end 46

    47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 53 else 60 end 61 end.join 62 end 63 end 64 end Boop
  260. sweep(block: block, name: "comments") end # Main search loop def

    call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb We've added all lines at this indent level. It's time to expand the frontier
  261. code_search.rb sweep(block: block, name: "comments") end # Main search loop

    def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) We call our expansion phase
  262. end end end # Given an already existing block in

    the frontier, expand it to see # if it contains our invalid syntax def expand_invalid_block block = frontier.pop return unless block record(block: block, name: "pop") block = @block_expand.call(block) push(block, name: "expand") end def sweep_heredocs HeredocBlockParse.new( source: @source, code_search.rb Here's our expand code
  263. end end end # Given an already existing block in

    the frontier, expand it to see # if it contains our invalid syntax def expand_invalid_block block = frontier.pop return unless block record(block: block, name: "pop") block = @block_expand.call(block) push(block, name: "expand") end def sweep_heredocs HeredocBlockParse.new( source: @source, code_search.rb The frontier is stored in indentation order. The highest indentation block is popped o f
  264. end end end # Given an already existing block in

    the frontier, expand it to see # if it contains our invalid syntax def expand_invalid_block block = frontier.pop return unless block record(block: block, name: "pop") block = @block_expand.call(block) push(block, name: "expand") end def sweep_heredocs HeredocBlockParse.new( source: @source, code_search.rb The block is expanded using an object and pushed back onto the frontier. How does block expansion work?
  265. # puts "wow" # end # class BlockExpand def initialize(code_lines:

    ) @code_lines = code_lines end def call(block) if (next_block = expand_neighbors(block, grab_empty: true)) return next_block end expand_indent(block) end def expand_indent(block) block = AroundBlockScan.new(code_lines: @code_lines, block: block) block_expand.rb It can either expand to capture more lines at the same indentation, aka neighbors. Or when those are exhausted it can expand out an indentation.
  266. def expand_indent(block) block = AroundBlockScan.new(code_lines: @code_lines, block: block) .skip(:hidden?) .stop_after_kw

    .scan_adjacent_indent .code_block end def expand_neighbors(block, grab_empty: true) scan = AroundBlockScan.new(code_lines: @code_lines, block: block) .skip(:hidden?) .stop_after_kw .scan_neighbors # Slurp up empties if grab_empty scan = AroundBlockScan.new(code_lines: @code_lines, block: scan.code_block) block_expand.rb Both methods use this class AroundBlockScan which acts as a DSL for scanning up and down. This is where we give our algorithm rules about how to build blocks using indentation and lexical data
  267. > 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if

    filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? ❯ 52 "#{number.to_s}#{line}" 53 else 60 end 61 end.join 62 end 63 end 64 end Previously we captured this line, and put it on the frontier.
  268. > 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if

    filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) ❯ 51 if line.empty? ❯ 53 else ❯ 60 end 61 end.join 62 end 63 end 64 end When it's popped and expanded it looks around and fi nds a matched set of keywords using the scanner.
  269. > 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if

    filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 61 end.join 62 end 63 end 64 end The code is valid so it's hidden. The expanded block is put back on the frontier.
  270. sweep(block: block, name: "comments") end # Main search loop def

    call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb That's pretty much how the algorithm for search works. It keeps looping, expanding and adding until all syntax errors are found
  271. sweep(block: block, name: "comments") end # Main search loop def

    call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb Once we've removed all syntax errors, we have to fi nd our problematic code.
  272. def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if

    frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self end end code_search.rb The frontier can hold more than one syntax error.
  273. # Example: # # combination([:a, :b, :c, :d]) # #

    => [[:a], [:b], [:c], [:d], # [:a, :b], [:a, :c], [:a, :d], # [:b, :c], [:b, :d], [:c, :d], # [:a, :b, :c], [:a, :b, :d], [:a, :c, :d], # [:b, :c, :d], # [:a, :b, :c, :d]] def self.combination(array) guesses = [] 1.upto(array.length).each do |size| guesses.concat(array.combination(size).to_a) end guesses end # Given that we know our syntax error exists somewhere in our frontier, we want to find code_frontier.rb We want to iterate over all of the invalid blocks stored in the frontier to fi nd the shortest possible order that gives us a valid document.
  274. # Example: # # combination([:a, :b, :c, :d]) # #

    => [[:a], [:b], [:c], [:d], # [:a, :b], [:a, :c], [:a, :d], # [:b, :c], [:b, :d], [:c, :d], # [:a, :b, :c], [:a, :b, :d], [:a, :c, :d], # [:b, :c, :d], # [:a, :b, :c, :d]] def self.combination(array) guesses = [] 1.upto(array.length).each do |size| guesses.concat(array.combination(size).to_a) end guesses end # Given that we know our syntax error exists somewhere in our frontier, we want to find code_frontier.rb To do that we create a combination of all possible blocks
  275. end # Given that we know our syntax error exists

    somewhere in our frontier, we want to find # the smallest possible set of blocks that contain all the syntax errors def detect_invalid_blocks self.class.combination(@frontier.select(&:invalid?)).detect do |block_array| holds_all_syntax_errors?(block_array) end || [] end end end code_frontier.rb Then one at a time we iterate through all the combinations
  276. guesses end # Given that we know our syntax error

    exists somewhere in our frontier, we want to find # the smallest possible set of blocks that contain all the syntax errors def detect_invalid_blocks self.class.combination(@frontier.select(&:invalid?)).detect do |block_array| holds_all_syntax_errors?(block_array) end || [] end end end code_frontier.rb And check each one to see if we hold all the syntax errors. The smallest successful combination is returned
  277. Timeout.timeout(timeout) do record_dir ||= ENV["DEBUG"] ? "tmp" : nil search

    = CodeSearch.new(source, record_dir: record_dir).call end blocks = search.invalid_blocks DisplayInvalidBlocks.new( blocks: blocks, filename: filename, terminal: terminal, code_lines: search.code_lines, invalid_obj: invalid_type(source), io: io ).call dead_end/internals.rb Here's where our search started.
  278. Timeout.timeout(timeout) do record_dir ||= ENV["DEBUG"] ? "tmp" : nil search

    = CodeSearch.new(source, record_dir: record_dir).call end blocks = search.invalid_blocks DisplayInvalidBlocks.new( blocks: blocks, filename: filename, terminal: terminal, code_lines: search.code_lines, invalid_obj: invalid_type(source), io: io ).call dead_end/internals.rb Then the invalid blocks were found
  279. Timeout.timeout(timeout) do record_dir ||= ENV["DEBUG"] ? "tmp" : nil search

    = CodeSearch.new(source, record_dir: record_dir).call end blocks = search.invalid_blocks DisplayInvalidBlocks.new( blocks: blocks, filename: filename, terminal: terminal, code_lines: search.code_lines, invalid_obj: invalid_type(source), io: io ).call dead_end/internals.rb We pass it to a pretty printer display
  280. lines = CaptureCodeContext.new( blocks: @blocks, code_lines: @code_lines ).call DisplayCodeWithLineNumbers.new( lines:

    lines, terminal: @terminal, highlight_lines: @invalid_lines ).call display_invalid_blocks.rb Here's where we take care of the last of our gotchas
  281. lines = CaptureCodeContext.new( blocks: @blocks, code_lines: @code_lines ).call DisplayCodeWithLineNumbers.new( lines:

    lines, terminal: @terminal, highlight_lines: @invalid_lines ).call display_invalid_blocks.rb This capture code context handles the ambiguities we talked about before
  282. > ❯ 1 class Dog Gotcha: ambiguity 🚫 It knows

    about our ambiguous <click> edge cases that we talked about before
  283. > ❯ 1 class Dog ❯ 2 def sit ❯

    3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity It goes back through the hidden code
  284. > ❯ 1 class Dog ❯ 2 def sit ❯

    3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity
  285. > ❯ 1 class Dog ❯ 2 def sit ❯

    3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity Work backwards and fi nd an unmatched keyword
  286. > ❯ 1 class Dog ❯ 10 def bark ❯

    12 end Gotcha: ambiguity And transforms it
  287. > DeadEnd: Missing `end` detected This code has a missing

    `end`. Ensure that all syntax keywords (`def`, `do`, etc.) have a matching `end`. file: /Users/rschneeman/Documents/projects/dead_end/spec/fixtures/this_project_extra_def.rb.txt simplified: 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename ❯ 36 def filename ❯ 38 def code_with_filename ❯ 45 end 63 end 64 end After all three steps, cleaning, searching, formatting here's the output
  288. Gem `error_highlight` in Ruby 3.1 Beyond dead_end there's an amazing

    gem being added to Ruby 3.1 called error_highlight. It shows you which method got a no method error. I love this introduction
  289. Gem `error_highlight` in Ruby 3.1 The error highlight gem was

    created by you-ssssss-kay. He goes by mam-eh on GitHub and gave an amazing talk at a conference I used to run called keep ruby weird.
  290. Rust v. Ruby With error highlight and dead end I

    also want to touch on community values.
  291. Rust v. Ruby I've been writing Rust code and their

    community takes a very aggressive stance towards error messages. It's not enough for them to say there's a problem. They also want to accurately suggest how to fi x the problem
  292. Rust v. Ruby I opened this issue against rustlang and

    in under a month it was addressed to improve the error output.
  293. Rust v. Ruby In general the Rust community treats a

    user experience problem as a bug and error output is a fi rst class feature. If our community wanted we could invest more in errors
  294. Dead end isn't that solution, but it's a start. You

    can add the dead_end gem to your project to try it today and give me feedback on what works and what doesn't.
  295. 📕 HowTo Open Source .dev You can pre-order my book

    on open source contribution by visiting how to open source dot dev
  296. Lexing Parsing Syntax SyntaxErrors AI Pathfinding and goal seeking Today

    we talked about lexing, parsing, syntax syntax errors, AI, path fi nding and goal seeking
  297. - The important part is that everyone sitting in this

    room is the future of developer tooling. Programming is inherently di ffi cult, but it doesn't mean our tools can't help us.
  298. One of the best ways to judge a system is

    to see how it fails. When we can give care and grace and beauty to our failure modes then we can create experiences that delight us. Experiences that teach us. Experiences that elevate our code and our consciousness.
  299. - A syntax error doesn't have to be an ending...it

    can be a beginning. To a beautiful programming story
  300. How To Open Source .dev - My name is Richard

    Schneeman - I'm writing a book at how to open source dot dev <click> - I created <click> CodeTriage.com the easiest way to get started contributing to OpenSource - <click> - Wait for "bye bye" - Zoom wave everyone