$30 off During Our Annual Pro Sale. View Details »

[RubyConf] Beware the Dreaded Dead End

[RubyConf] Beware the Dreaded Dead End

Nothing stops a program from executing quite as fast as a syntax error. After years of “unexpected end” in my dev life, I decided to “do” something about it. In this talk we'll cover lexing, parsing, and indentation informed syntax tree search that power that dead_end Ruby library.

I also gave this talk at RubyKaigi. I recommend this version, it is more up-to-date and covers specific details and "gotchas" on the problem space.

Richard Schneeman

October 28, 2021
Tweet

More Decks by Richard Schneeman

Other Decks in Technology

Transcript

  1. Beware the Dreaded Dead End!!! By @schneems Hello everyone. My

    name is Richard Schneeman and I want to talk to you about the scariest thing a Ruby programmer can face. You hear that?
  2. Everyone run

  3. Oh no, there are more of them. AHHHH

  4. Beware the Dreaded Dead End!!! By @schneems Wow, I can

    hardly believe that the those dinosaurs from RubyKaigi found me all the way over here, I thought I escaped them already. If you've seen that talk, know that this one has some new content.
  5. Beware the Dreaded Dead End!!! By @schneems While dinosaurs are

    scary. There's something scarier. Its..
  6. > # <= routes.rb:121: syntax error, unexpected end-of-input, expecting `end’

    Rails.application.routes.draw do constraints -> { Rails.application.config.non_production } do namespace :foo do resource :bar end end constraints -> { Rails.application.config.non_production } do namespace :bar do resource :baz end end namespace :admin do resource :session match "/foobar(*path)", via: :all, to: redirect { |_params, req| uri = URI(req.path.gsub("foobar", "foobaz")) uri.query = req.query_string.presence uri.to_s } end Syntax errors <click: laughter>
  7. > Rails.application.routes.draw do constraints -> { Rails.application.config.non_production } do namespace

    :foo do resource :bar end end constraints -> { Rails.application.config.non_production } do namespace :bar do resource :baz end end namespace :admin do resource :session match "/foobar(*path)", via: :all, to: redirect { |_params, req| uri = URI(req.path.gsub("foobar", "foobaz")) uri.query = req.query_string.presence uri.to_s } end # <= routes.rb:121: syntax error, unexpected end-of-input, expecting `end’ Just look at this unexpected syntax error. It's horrifying. Where's the problem?
  8. > RSpec.describe Cutlass::BashResult do it "preserves stdout, stderr, and status"

    stdout = SecureRandom.hex(16) stderr = SecureRandom.hex(16) status = 0 result = BashResult.new( stdout: stdout, stderr: stderr, status: status ) expect(result.stdout).to eq(stdout) expect(result.stderr).to eq(stderr) expect(result.status).to eq(status) end end # <= result_spec:19 syntax error, unexpected end-of-input, expecting `end’ Here's some di ff erent code. Wanna guess where Ruby think the problem is? <click> On the last line, nope. This is frustrating
  9. 🤔 What if we had something better

  10. > RSpec.describe Cutlass::BashResult do it "preserves stdout, stderr, and status"

    stdout = SecureRandom.hex(16) stderr = SecureRandom.hex(16) status = 0 result = BashResult.new( stdout: stdout, stderr: stderr, status: status ) expect(result.stdout).to eq(stdout) expect(result.stderr).to eq(stderr) expect(result.status).to eq(status) end end # <= result_spec:19 syntax error, unexpected end-of-input, expecting `end’ Instead of this
  11. > 3 module Cutlass 4 RSpec.describe Cutlass::BashResult do ❯ 5

    it "preserves stdout, stderr, and status" ❯ 19 end 21 it "success?" do 52 end 53 end 54 end This code has an unmatched `end`. Ensure that all `end` lines in your code have a matching syntax keyword (`def`, `do`, etc.) What if we got this? Do you see the error now? Doesn't line 5 look suspicious, like maybe it's missing a do.
  12. 🎂 Have you ever seen a cooking show where they

    show o ff the fi nal product? I want to show o ff dead end before we get too far.
  13. demo Here is a demo of how dead_end fi nds

    syntax errors
  14. module SyntaxErrorSearch # Used for formatting invalid blocks class DisplayInvalidBlocks

    attr_reader :filename def initialize(block_array, io: $stderr, filename: nil) @filename = filename @io = io @blocks = block_array @lines = @blocks.map(&:lines).flatten @digit_count = @lines.last.line_number.to_s.length @code_lines = @blocks.first.code_lines @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} end def call @io.puts <<~EOM 64: syntax error, unexpected end-of-input, expecting `end' 📕 ❌ This is real ruby source code with a syntax error in it. It cannot parse
  15. 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string =

    String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s 58 string << "\e[0m" 59 string 60 end 61 end.join 62 end 63 end 64 end <click>
  16. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53

    else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end
  17. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53

    else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end
  18. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53

    else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end
  19. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53

    else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end
  20. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53

    else 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end
  21. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53

    else ❯ 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end When a valid code block is found dead_end safely removes it
  22. 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53

    else 60 end 61 end.join 62 end 63 end 64 end 📕 ❌ After each step in the search, dead end re-evaluates the whole document to see if it's parsable yet. <click> parsing failed, we need to keep looking
  23. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? ❯ 52 "#{number.to_s}#{line}" 53 else 60 end 61 end.join 62 end 63 end 64 end
  24. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 53 else 60 end 61 end.join 62 end 63 end 64 end 📕 ❌
  25. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) ❯ 51 if line.empty? ❯ 53 else ❯ 60 end 61 end.join 62 end 63 end 64 end
  26. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 61 end.join 62 end 63 end 64 end 📕 ❌
  27. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 47 def code_with_lines 48 @code_lines.map do |line| ❯ 49 next if line.hidden? ❯ 50 number = line.line_number.to_s.rjust(@digit_count) 61 end.join 62 end 63 end 64 end
  28. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 61 end.join 62 end 63 end 64 end 📕 ❌
  29. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 47 def code_with_lines ❯ 48 @code_lines.map do |line| ❯ 61 end.join 62 end 63 end 64 end
  30. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 47 def code_with_lines 62 end 63 end 64 end 📕 ❌
  31. 42 string << code_with_lines 43 string << "```\n" 44 string

    45 end 46 ❯ 47 def code_with_lines ❯ 62 end 63 end 64 end
  32. 40 string << "```\n" 41 string << "#".rjust(@digit_count) + "

    filename: #{filename}\n\n" if filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 63 end 64 end 📕 ❌
  33. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

    15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 63 end 64 end
  34. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

    15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
  35. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

    15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
  36. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

    15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename ❯ 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
  37. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

    15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" ❯ 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename ❯ 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
  38. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

    15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") ❯ 40 string << "```\n" ❯ 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename ❯ 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
  39. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

    15 end 16 35 36 def filename 37 38 def code_with_filename ❯ 39 string = String.new("") ❯ 40 string << "```\n" ❯ 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename ❯ 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end
  40. 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

    15 end 16 35 36 def filename 37 38 def code_with_filename 45 end 46 63 end 64 end 📕 ❌
  41. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array 10 @lines = @blocks.map(&:lines).flatten 11 @digit_count = @lines.last.line_number.to_s.length 12 @code_lines = @blocks.first.code_lines 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
  42. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array 10 @lines = @blocks.map(&:lines).flatten 11 @digit_count = @lines.last.line_number.to_s.length 12 @code_lines = @blocks.first.code_lines 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
  43. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array 10 @lines = @blocks.map(&:lines).flatten 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
  44. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
  45. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array ❯ 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
  46. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io ❯ 9 @blocks = block_array ❯ 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
  47. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename ❯ 8 @io = io ❯ 9 @blocks = block_array ❯ 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
  48. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) ❯ 7 @filename = filename ❯ 8 @io = io ❯ 9 @blocks = block_array ❯ 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename
  49. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 15 end 16 35 36 def filename 37 38 def code_with_filename 45 end 46 63 end 64 end 📕 ❌
  50. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 15 end 16 35 36 def filename 37 ❯ 38 def code_with_filename ❯ 45 end 46 63 end 64 end
  51. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    6 def initialize(block_array, io: $stderr, filename: nil) 15 end 16 35 36 def filename 37 46 63 end 64 end 📕 ❌
  52. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5

    ❯ 6 def initialize(block_array, io: $stderr, filename: nil) ❯ 15 end 16 35 36 def filename 37 46 63 end 64 end
  53. 📕 ❌ 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader

    :filename 5 16 35 36 def filename 37 46 63 end 64 end
  54. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename ❯

    5 ❯ 16 ❯ 35 ❯ 36 def filename ❯ 37 ❯ 46 63 end 64 end
  55. 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 63

    end 64 end 📕 ✅ The document is checked. <click> The parser reports that this very minimal document is now parsable.
  56. Syntax O 🥰🥰🥰 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4

    attr_reader :filename 63 end 64 end 📕 ✅ Once this happens dead_end has found a way to transform the document to be valid. It can stop searching the document and instead search the invalid code blocks that it has stored
  57. DeadEnd: Missing `end` detected This code has a missing `end`.

    Ensure that all syntax keywords (`def`, `do`, etc.) have a matching `end`. file: /Users/rschneeman/Documents/projects/dead_end/spec/fixtures/this_project_extra_def.rb.txt simplified: 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename ❯ 36 def filename ❯ 38 def code_with_filename ❯ 45 end 63 end 64 end This is the actual output the dead end gem returns on that fi le. The issue is on line 36 there's a missing end statement
  58. $ gem install dead_end The dead_end gem is released and

    on ruby gems.
  59. By the time you see this slide dead_end will have

    surpassed half a million downloads
  60. Ruby 3.2? I am also talking with Ruby core to

    get dead_end into Ruby directly. We are targeting that integration for 2022 release of Ruby 3.2
  61. 🤔 What all can dead_end do?

  62. > 3 module Cutlass 4 RSpec.describe Cutlass::BashResult do ❯ 5

    it "preserves stdout, stderr, and status" ❯ 19 end 21 it "success?" do 52 end 53 end 54 end DeadEnd: Unmatched `end` detected When you miss a keyword like "if", "do" or "def". <click> Dead end fi nds the problem.
  63. > 1 module Cutlass 2 RSpec.describe Cutlass::BashResult do ❯ 3

    it "preserves stdout, stderr, and status" do 5 it "success?" do 6 end 7 end 8 end DeadEnd: Missing `end` detected Even when you miss an end keyword, <click> dead end fi nds the problem
  64. > 1 class Cat 2 def to_json ❯ 3 hash

    = { ❯ 4 name: name, ❯ 5 fur_color: fur_color, ❯ 6 type: 'Cat', ❯ 7 better_than_dog: false ❯ 8 hash.to_json 9 end 10 end DeadEnd: Unmatched `}` character detected When you miss a curly bracket <click> dead end fi nds the problem
  65. > 1 class Animal 2 def self.ones_my_three_year_old_likes ❯ 3 array

    = [ ❯ 4 Dog.new, ❯ 5 Lion.new, ❯ 6 Tiger.new, ❯ 7 Aligator.new, 10 return array 11 end 12 end DeadEnd: Unmatched `[` detected When you miss a square bracket <click> dead end fi nds the problem
  66. > 1 class Cat 2 def meow ❯ 3 Animal.call

    do |a ❯ 5 end 6 end 7 end DeadEnd: Unmatched `|` character detected Have a problem with a missing pipe character? Dead end. Finds. <click> The. <click> Problem <click> <Pause> Have a problem with a missing family member in a Korean thriller drama?
  67. > Unfortunately. Dead end cannot fi nd that problem. <click:

    laughter>. Okay, wow that current event reference got really dark. Let's rewind a try again.
  68. > Have a problem with a missing marvel universe character?

  69. > 1 class Cat 2 def meow ❯ 3 Animal.call

    do |a ❯ 5 end 6 end 7 end DeadEnd: Unmatched `|` character detected ? ? ? Unfortunately. <click: laughter> Dead end cannot fi nd that problem. Much better. Remember crocodile Loki? That was 2021, this year is lasting forever!
  70. $ gem install dead_end Today we'll dig into

  71. $ dead_end bad.rb --record tmp/ You can follow the algorithm

    yourself by running the CLI with the record fl ag
  72. You'll see each step

  73. Along with annotated source code

  74. Syntax errors Today we will talk about

  75. Lexing & Parsing How to parse and lex source code

    with ripper
  76. AI How AI and path fi nding algorithms work

  77. dead_end We will put it all together to see how

    dead_end works
  78. Internals Finally we'll crack open the cover and see how

    it works at a low level. Warning: We will get technical. We will cover a lot of code.
  79. 👋 But wait, who am I?

  80. @schneems I go by schneems on the internet. If you

    forget how to pronounce my name you can go to
  81. @schneems the about page on my blog on Schneems dot

    com
  82. I Created CodeTriage which is a platform for learning how

    to contribute to open source. To date I've got over 60 thousand developers signed up for CodeTriage
  83. 📕 HowTo Open Source .dev I'm writing a book on

    how to contribute to open source as well. It's for developers who want to start contributing. Developers who got stuck, and developers looking for ways to sustain contributions. I'm not ready to release it yet, but you can sign up for a pre-order on HowToOpenSource dot dev Also you can sign up for Code Triage dot com and I will email members when the book is ready.
  84. Heroku When I'm not working on open source or teaching

    people to open source with CodeTriage. I like to get paid. I work at <click> Heroku
  85. Heroku Right now I'm working on salesforce functions. It's an

    easy way to work with the data you've got in salesforce using the languages you love. Right now we support Java and JavaScript languages. We'll be rolling out Ruby support later. If you use love Ruby and use Salesforce I would love to hear from you. My DMs on Twitter are open. Also people tell me
  86. NoMemoryError ScriptError LoadError SyntaxError SecurityError SignalException KeyError StandardError ArgumentError EncodingError

    IOError EOFError I am Exceptional i'm an exceptional programmer... <click> My programs generate a lot of exceptions <click: laugh>
  87. Syntax errors Syntax errors. Let's start from the beginning what

    is a syntax error? Why are syntax errors so hard to understand?
  88. > while b != 0 if a > b a

    = a - b else b = b - a end end return a This code worked wonderfully comparing a, and b all day.
  89. > while b != 0 if a > b a

    = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a When Ruby parses this code it will convert it into an abstract syntax tree. <click> <point> this code generates <point> that tree. The tree was beautiful and Ruby's parser looked upon it with happiness
  90. > while b != 0 if a > b a

    = a - b else b = b - a while b != 0 if a > b a = a - b else b = b - a end end return a 😈 While the code was good a stranger <click> came upon the land. This stranger had a secret power, behold
  91. # The octothorpe, This tiny character gave the stranger great

    power to create, or destroy.
  92. > while b != 0 if a > b a

    = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a 😈 With one key the stranger used their power <click>
  93. > while b != 0 # if a > b

    a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a 😈 to transform the code <click>
  94. > while b != 0 # if a > b

    a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a 😈 A critical line had been commented out. Without that line, the tree was no longer whole
  95. > while b != 0 # if a > b

    a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a 😈 Huge sections of code are no longer reachable. The code no longer parses. And our parser is sad.
  96. > while b != 0 # if a > b

    a = a - b else b = b - a end end return a while b != 0 a = a - b else b = b - a 😈 With that the stranger left <click> and behind them stood
  97. > while b != 0 # if a > b

    a = a - b else b = b - a end end return a while b != 0 a = a - b else b = b - a Syntax error A syntax error
  98. Syntax errors A syntax error occurs when the parser cannot

    build a valid syntax tree. That explains what they are, but why are they di ffi cult to understand?
  99. > module Cutlass def call 1, 2, 3, 4, 5]

    end end syntax error, unexpected ',', expecting end-of-input (SyntaxError) When the parser tries to parse this code it fi nds a <click> Syntax Error. Here our developer forgot a bracket. As the parser is building the tree, it hit an error because it wasn't expecting the comma
  100. > syntax error, unexpected ',', expecting end-of-input (SyntaxError) module Cutlass

    def call 1, 2, 3, 4, 5] end end Ruby's parser has rules. It knows that after a method de fi nition it should look for something such as
  101. > syntax error, unexpected ',', expecting end-of-input (SyntaxError) module Cutlass

    def call 1, 2, 3, 4, 5] end end Like a return or an end. Or a variable assignment. Instead what did the parser fi nd fi nd? It found a number and a comma.
  102. > module Cutlass def call 1, 2, 3, 4, 5]

    end end syntax error, unexpected ',', expecting end-of-input (SyntaxError) 🚫 That comma violated ruby's parser rules. Do not pass go, do not collect 200 <click>
  103. 🤷 What's so bad about this example?

  104. > module Cutlass def call 1, 2, 3, 4, 5]

    end end syntax error, unexpected ',', expecting end-of-input (SyntaxError) The error <click> isn't caused by a comma, it's caused by the developer missing a bracket. Essentially, the location the parse error occurs isn't always where developer made a mistake
  105. 🙇 Let's look at some more complicated cases

  106. > module Cutlass defcall end end Here's code with a

    syntax error. The developer forgot a space after the def
  107. > module Cutlass defcall end end syntax error, unexpected `end',

    expecting end-of-input Ruby thinks the error is at the last line. <click> right here. Why?
  108. > module Cutlass defcall end end syntax error, unexpected `end',

    expecting end-of-input Welllll... When we start parsing Ruby sees a module de fi nition, A module is a keyword that requires an end so Ruby starts looking for a matching end.
  109. > module Cutlass defcall end end syntax error, unexpected `end',

    expecting end-of-input On the next line it sees this combination of characters and thinks it's a method call
  110. > module Cutlass defcall end end syntax error, unexpected `end',

    expecting end-of-input Then it sees this end and thinks it matches with the module. Our parser hasn't raised a syntax error because it doesn't see a problem yet.
  111. > module Cutlass defcall end end syntax error, unexpected `end',

    expecting end-of-input Finally it gets to our second end, but there's no thing to match with. A syntax error is raised. <click>
  112. > end syntax error, unexpected `end', expecting end-of-input Right now

    this is the error you get from ruby on this code. It's not helpful to a human because deleting that end won't solve anything.
  113. parser problems != human problems Another way to put this,

    is that parse errors are di ff erent from human errors.
  114. $ gem install dead_end dead_end's goal is to turn that

    parser problem into something a human can see and instantly recognize as an issue.
  115. $ gem install dead_end How does dead end work?

  116. Dead end works by using a library called Ripper

  117. require "ripper" Ripper.new(source).tap(&:parse).error? It's not a band. Ripper is Ruby's

    parser that ships with Ruby. How cool is that?
  118. require "ripper" source = <<~EOM module Cutlass defcall end end

    EOM Ripper.new(source).tap(&:parse).error? Ripper can evaluate code and tell us if there's a syntax error or not. We saw this code before
  119. require "ripper" source = <<~EOM module Cutlass defcall end end

    EOM Ripper.new(source).tap(&:parse).error? , there's a syntax error here
  120. require "ripper" source = <<~EOM module Cutlass defcall end end

    EOM Ripper.new(source).tap(&:parse).error? # => true When we run ripper it says there's an error
  121. require "ripper" source = <<~EOM module Cutlass def call end

    end EOM Ripper.new(source).tap(&:parse).error? # => false When we fi x the error, ripper tells us it's fi xed too. <click>
  122. 🙇 If we could fi x the source errors then

    ripper would tell us, but that's too hard to automate. Can we do something simpler?
  123. require "ripper" source = <<~EOM module Cutlass defcall end end

    EOM Ripper.new(source).tap(&:parse).error? # => true Back to our code with a syntax error. Instead of fi xing it. What if we...commented it out?
  124. require "ripper" source = <<~EOM module Cutlass # defcall end

    end EOM Ripper.new(source).tap(&:parse).error? # => true That didn't work, let's keep going.
  125. require "ripper" source = <<~EOM module Cutlass # defcall #

    end end EOM Ripper.new(source).tap(&:parse).error? # => false Oooh, nice <click> ripper can tell us when we've commented out the source code causing the problem
  126. # defcall # end Ripper.new(source).tap(&:parse).error? # => false Now if

    we only look at the part that was commented out, dead_end can show you where the issue occured.
  127. $ gem install dead_end Remember our stranger and the mess

    they left behind?
  128. > while b != 0 a = a - b

    else b = b - a end end return a while b != 0 a = a - b else b = b - a 😈 Ruby knows there's a syntax error here, but it doesn't know where.
  129. > while b != 0 a = a - b

    else b = b - a end end return a while b != 0 a = a - b else b = b - a 😈 It will tell you
  130. > while b != 0 a = a - b

    else b = b - a end end return a syntax error, unexpected `end', expecting end-of-input That the problem is this line. We know that's not helpful. What does dead end do?
  131. > while b != 0 a = a - b

    else b = b - a end end return a while b != 0 a = a - b else b = b - a Dead end uses indentation and lexical parsing to deconstruct the source code from the outside in <click>
  132. > while b != 0 a = a - b

    else # b = b - a end end return a while b != 0 a = a - b else b = b - a Commenting out this code removes a node from our tree
  133. > while b != 0 # a = a -

    b else # b = b - a end end return a while b != 0 a = a - b else Commenting out this code removes a node from our tree, but it's still invalid
  134. > while b != 0 # a = a -

    b # else # b = b - a # end end return a while b != 0 else Finally here we have a valid tree. All of the orphaned syntax nodes are gone
  135. > 1 while b != 0 ❯ 4 else ❯

    6 end 7 end This code has an unmatched `end`. Ensure that all `end` lines in your code have a matching syntax keyword (`def`, `do`, etc.) Here's the output for that code. It shows that the else and end are missing an if statement.
  136. I bet you're thinking wow, that was easy. So all

    he did was comment out code based on indentation?
  137. Gotchas There are some major gotchas when executing this recursive

    comment approach.
  138. Gotcha #1 We already know the developer has a syntax

    error. We can't assume they've done everything else perfectly. 
 We cannot assume perfect indentation
  139. > describe "things" do it "valid" do expect(the).to eq(unexpected) end

    it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything Our syntax error is on this line. It is missing keyword do
  140. > describe "things" do it "valid" do expect(the).to eq(unexpected) end

    it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything Without it, the parser has an extra end here
  141. > describe "things" do it "valid" do expect(the).to eq(unexpected) end

    it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything If we only look at indentation then we would remove this line
  142. > describe "things" do it "valid" do end it "forgot

    a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything Then this mis-indented end
  143. > describe "things" do it "valid" do it "forgot a

    keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything 📕 ✅ With the end gone, the document parses<click>. Why?
  144. > describe "things" do it "valid" do it "forgot a

    keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything 📕 ✅ Because this keyword now matches with
  145. > describe "things" do it "valid" do it "forgot a

    keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything 🚫 This end It's not what we wanted, and didn't capture the source of the syntax error <click>
  146. 🙇 How can we fi x it? Instead of only

    looking at indentation we also look at lex output
  147. source = <<~EOM it "valid" do expect(the).to eq(unexpected) end EOM

    Ripper.lex(source) Ruby ships with a lexer
  148. source = <<~EOM it "valid" do expect(the).to eq(unexpected) end EOM

    Ripper.lex(source) # => [[[1, 0], :on_ident, "it", CMDARG], [[1, 2], :on_sp, " ", CMDARG], [[1, 3], :on_tstring_beg, "\"", CMDARG], [[1, 4], :on_tstring_content, "valid", CMDARG], [[1, 9], :on_tstring_end, "\"", END], [[1, 10], :on_sp, " ", END], [[1, 11], :on_kw, "do", BEG], [[1, 13], :on_ignored_nl, "\n", BEG], Here's what it looks like
  149. # => [[[1, 0], :on_ident, "it", CMDARG], [[1, 2], :on_sp,

    " ", CMDARG], [[1, 3], :on_tstring_beg, "\"", CMDARG], [[1, 4], :on_tstring_content, "valid", CMDARG], [[1, 9], :on_tstring_end, "\"", END], [[1, 10], :on_sp, " ", END], [[1, 11], :on_kw, "do", BEG], [[1, 13], :on_ignored_nl, "\n", BEG], [[2, 0], :on_sp, " ", BEG], [[2, 6], :on_ident, "expect", CMDARG], [[2, 12], :on_lparen, "(", BEG|LABEL], [[2, 13], :on_ident, "the", ARG], [[2, 16], :on_rparen, ")", ENDFN], [[2, 17], :on_period, ".", DOT], [[2, 18], :on_ident, "to", ARG], DO NOT CLICK, auto advance
  150. # => [[[1, 0], :on_ident, "it", CMDARG], [[1, 2], :on_sp,

    " ", CMDARG], [[1, 3], :on_tstring_beg, "\"", CMDARG], [[1, 4], :on_tstring_content, "valid", CMDARG], [[1, 9], :on_tstring_end, "\"", END], [[1, 10], :on_sp, " ", END], [[1, 11], :on_kw, "do", BEG], [[1, 13], :on_ignored_nl, "\n", BEG], [[2, 0], :on_sp, " ", BEG], [[2, 6], :on_ident, "expect", CMDARG], [[2, 12], :on_lparen, "(", BEG|LABEL], [[2, 13], :on_ident, "the", ARG], [[2, 16], :on_rparen, ")", ENDFN], [[2, 17], :on_period, ".", DOT], [[2, 18], :on_ident, "to", ARG], It tells you the code contains a do keyword
  151. [[2, 12], :on_lparen, "(", BEG|LABEL], [[2, 13], :on_ident, "the", ARG],

    [[2, 16], :on_rparen, ")", ENDFN], [[2, 17], :on_period, ".", DOT], [[2, 18], :on_ident, "to", ARG], [[2, 20], :on_sp, " ", ARG], [[2, 21], :on_ident, "eq", ARG], [[2, 23], :on_lparen, "(", BEG|LABEL], [[2, 24], :on_ident, "unexpected", ARG], [[2, 34], :on_rparen, ")", ENDFN], [[2, 35], :on_sp, " ", ENDFN], [[2, 36], :on_nl, "\n", BEG], [[3, 0], :on_sp, " ", BEG], [[3, 2], :on_kw, "end", END], [[3, 5], :on_nl, "\n", BEG]] DO NOT CLICK, auto advance
  152. [[2, 12], :on_lparen, "(", BEG|LABEL], [[2, 13], :on_ident, "the", ARG],

    [[2, 16], :on_rparen, ")", ENDFN], [[2, 17], :on_period, ".", DOT], [[2, 18], :on_ident, "to", ARG], [[2, 20], :on_sp, " ", ARG], [[2, 21], :on_ident, "eq", ARG], [[2, 23], :on_lparen, "(", BEG|LABEL], [[2, 24], :on_ident, "unexpected", ARG], [[2, 34], :on_rparen, ")", ENDFN], [[2, 35], :on_sp, " ", ENDFN], [[2, 36], :on_nl, "\n", BEG], [[3, 0], :on_sp, " ", BEG], [[3, 2], :on_kw, "end", END], [[3, 5], :on_nl, "\n", BEG]] Also an end
  153. $ gem install dead_end With this info dead_end can fi

    nd the problem
  154. > describe "things" do it "valid" do expect(the).to eq(unexpected) end

    it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything It starts at the same place
  155. > describe "things" do it "valid" do end it "forgot

    a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything Now it needs to decide where to expand, it looks down and sees an `end`
  156. > describe "things" do it "valid" do end it "forgot

    a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything dead_end knows if it removes this end, it might trigger a false positive, so it searches up for keywords. It fi nds...
  157. > describe "things" do it "valid" do end it "forgot

    a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything This do keyword
  158. > describe "things" do it "forgot a keyword -> "

    expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything They are both safely removed
  159. > file: /private/tmp/scratch.rb simplified: 1 describe "things" do ❯ 6

    it "forgot a keyword -> " ❯ 8 end 9 end Gotcha: Indentation isn't everything The algorithm continues until the problem is found
  160. Gotcha #2 Even with correct indentation, removing the wrong line

    can show false positives
  161. > class User puts "hello" end def schneems User. where(name:

    'schneems'). first end end Gotcha: Removing the wrong line Our syntax error is on these lines.
  162. > class User puts "hello" end def schneems User. where(name:

    'schneems'). first end end Gotcha: Removing the wrong line We are missing a whole method de fi nition line
  163. > class User puts "hello" end def schneems User. where(name:

    'schneems'). first end end Gotcha: Removing the wrong line If we tried our indentation comment out strategy we would start at the largest indentation
  164. > class User puts "hello" end def schneems User. where(name:

    'schneems'). first end end Gotcha: Removing the wrong line here
  165. > class User puts "hello" end def schneems User. where(name:

    'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ And then dead_end would stop looking because the document is now parsable <click>
  166. WAT WAT on earth? Let's look at it a little

    di ff erently.
  167. > class User puts "hello" end def schneems User. where(name:

    'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ It will help if we reformat it to show what Ruby thinks you're trying to do. It thinks
  168. > class User puts "hello" end def schneems User. where(name:

    'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ That this user class
  169. > class User puts "hello" end def schneems User. where(name:

    'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ Is matched with the fi rst end.
  170. > class User puts "hello" end def schneems User. where(name:

    'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ Is matched with the fi rst end.
  171. > class User puts "hello" end def schneems User. where(name:

    'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ It thinks that this end, is actually a method due to the trailing dot
  172. > class User puts "hello" end def schneems User. where(name:

    'schneems').end end Gotcha: Removing the wrong line 📕 ✅ Finally it thinks the last end matches with our method de fi nition
  173. > class User puts "hello" end def schneems User. where(name:

    'schneems').end end Gotcha: Removing the wrong line 📕 ✅ Finally it thinks the this method de fi nition
  174. > class User puts "hello" end def schneems User. where(name:

    'schneems').end end Gotcha: Removing the wrong line 📕 ✅ 🚫 Matches with this last end. <click> It's a problem because none of this is what the user intended and we didn't highlight the source of the problems
  175. WAT But wait there's more

  176. > source = <<~EOM Hello World From A heredoc EOM

    Gotcha: Removing the wrong line Heredocs cause problems
  177. > source = <<~EOM Hello World From A heredoc EOM

    Gotcha: Removing the wrong line If you evaluate this line in isolation, it looks like a constant. But if you remove it
  178. > source = <<~EOM Hello World From A heredoc Gotcha:

    Removing the wrong line 🚫 Then you've introduced a syntax error<click>
  179. WAT But wait there's more

  180. > it "should" \ "handle trailing slash" do expect(the).to eq(unexpected)

    end Gotcha: Removing the wrong line Trailing slashes cause problems.
  181. > it "should" \ "handle trailing slash" do expect(the).to eq(unexpected)

    end Gotcha: Removing the wrong line If you start in the middle
  182. > it "should" \ "handle trailing slash" do end Gotcha:

    Removing the wrong line Now expand out using lex detection
  183. > it "should" \ "handle trailing slash" do end Gotcha:

    Removing the wrong line Find the end
  184. > it "should" \ "handle trailing slash" do end Gotcha:

    Removing the wrong line Find the matching keyword.
  185. > it "should" \ Gotcha: Removing the wrong line 🚫

    Now remove them.<click> This is not valid ruby
  186. 🙇 What

  187. 🙇 🙇 To

  188. 🙇 🙇 🙇 do

  189. source = <<~EOM User. where(name: 'schneems'). first EOM Ripper.lex(source) #

    => [[[1, 0], :on_const, "User", CMDARG], [[1, 4], :on_period, ".", DOT], [[1, 5], :on_ignored_nl, "\n", DOT], [[2, 0], :on_sp, " ", DOT], [[2, 2], :on_ident, "where", ARG], [[2, 7], :on_lparen, "(", BEG|LABEL], [[2, 8], :on_label, "name:", ARG|LABELED], [[2, 13], :on_sp, " ", ARG|LABELED], We can use the lexer to detect these speci fi c edge cases. <click> Then we can use that information to combine all logically consecutive lines together so there's no way to remove one without removing all of them
  190. > it "should" \ "handle trailing slash" do expect(the).to eq(unexpected)

    end Gotcha: Removing the wrong line It's hard to show visually, but it essentially would looks like re-writing source code. This trailing slash would become
  191. > it "should" \ "handle trailing slash" do expect(the).to eq(unexpected)

    end Gotcha: Removing the wrong line
  192. > class User puts "hello" end def schneems User. where(name:

    'schneems'). first end end Gotcha: Removing the wrong line Our method chain would become
  193. > class User puts "hello" end def schneems User.where(name: 'schneems').first

    end end Gotcha: Removing the wrong line Our method chain would become
  194. > source = <<~EOM Hello World From A heredoc EOM

    Gotcha: Removing the wrong line Heredocs cause problems
  195. > source = <<~EOM Hello\nWorld\nFrom\nA\n\nheredoc\nEOM Gotcha: Removing the wrong line

  196. 🎉 With all the problem lines joined together, they can

    now be safely evaluated and removed without accidentally introducing new syntax errors <click>
  197. Gotcha #3 Even with joining lines, even with lexically aware

    search, even with the perfect algorithm there is one problem we cant' avoid
  198. ambiguity Ambiguity. Yes, ambiguity is the comic sans of source

    code parsing.<click> What do I mean by ambiguity?
  199. > class Dog def sit puts "no" end def eat

    puts "munch" end def bark puts "woof" end Gotcha: ambiguity This code has a syntax error
  200. > class Dog def sit puts "no" end def eat

    puts "munch" end def bark puts "woof" end Gotcha: ambiguity It is ambiguous what the coder was trying to do with our current search algorithm. Let's zoom in
  201. > class Dog def bark puts "woof" end Gotcha: ambiguity

    <click>
  202. > class Dog def bark puts "woof" end Gotcha: ambiguity

    Starting from the middle, if we use lexical expansion we get
  203. > class Dog def bark puts "woof" end Gotcha: ambiguity

    This match
  204. > class Dog Gotcha: ambiguity When it's removed there's still

    a syntax error
  205. > class Dog Gotcha: ambiguity Highlight

  206. > Gotcha: ambiguity 📕 ✅ Remove<click> It parses, yay

  207. > ❯ 1 class Dog Gotcha: ambiguity 🚫 📕 ✅

    But this clearly isn't our problem <click>
  208. 🙇 To produce good results with invalid indentation it means

    we must make "bad" suggestions. There's a logical inverse of this problem as well.
  209. > class Dog puts "no" end def eat puts "munch"

    end def bark puts "woof" end end Gotcha: ambiguity This code has a syntax error, but this time it's at the top.
  210. > class Dog puts "no" end def eat puts "munch"

    end def bark puts "woof" end end Gotcha: ambiguity Let's focus
  211. > class Dog puts "no" end end Gotcha: ambiguity

  212. > class Dog puts "no" end end Gotcha: ambiguity Does

    this problem look familiar? Here's the code from before
  213. > class Dog puts "no" end end Gotcha: ambiguity class

    Dog def bark puts "woof" end Before we were missing an end
  214. > class Dog puts "no" end end Gotcha: ambiguity class

    Dog def bark puts "woof" end Before we were missing an end
  215. > class Dog puts "no" end end Gotcha: ambiguity class

    Dog def bark puts "woof" end This new case is missing a keyword
  216. > class Dog puts "no" end end Gotcha: ambiguity class

    Dog def bark puts "woof" end This new case is missing a keyword
  217. > class Dog puts "no" end end Gotcha: ambiguity class

    Dog def bark puts "woof" end Where does dead end tell us the problem is?
  218. > class Dog puts "no" end end class Dog def

    bark puts "woof" end Gotcha: ambiguity ❯ 10 end ❯ 1 class Dog 🚫 Here's the result. Neither make good suggestion<click>
  219. 🙇 If we re-write our rules to let indentation take

    precedence over lexical keywords then it would make other examples fail. It's impossible to satisfy all search criteria. What if we...
  220. 😇 Don't. Because we know this ambiguity exists, we can

    compensate for it after the search is done
  221. > ❯ 1 class Dog Gotcha: ambiguity Even though this

    is all the user might see, `dead_end` still has all the contents that are hidden. It didn't just match a single line, it matched many lines, but hid most of them
  222. > ❯ 1 class Dog ❯ 2 def sit ❯

    3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity If we un-hid the rest of them, this is what it would look like. We can detect this case when our match is only one line and contains an end, then we can go back through our results
  223. > ❯ 1 class Dog ❯ 2 def sit ❯

    3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity un-hide the matched end
  224. > ❯ 1 class Dog ❯ 2 def sit ❯

    3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity Work backwards and fi nd an unmatched keyword
  225. > ❯ 1 class Dog ❯ 10 def bark ❯

    12 end Gotcha: ambiguity Then we show the user all extra context to the user and they can resolve the problem visually
  226. Gotcha #4 Syntax errors aren't always alone, sometimes they have

    friends. A document can contain multiple syntax errors
  227. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors Here' there's multiple problems
  228. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors This end matches nothing
  229. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors We are missing a square bracket
  230. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors We're missing all sorts of stu ff here
  231. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors Right here, that's code is fi ne
  232. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors That code is fi ne too
  233. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors What we don't want to happen is for our search to start at one side
  234. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors And continue until it captures everything, even the good code
  235. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors 🚫 This isn't what we want <click>
  236. 🙇 What can we do about it? We can modify

    our algorithm to hold and fi nd multiple errors.
  237. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors Instead of searching from one side to the other we start in the outside and search in
  238. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1
  239. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2
  240. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2
  241. > end def hello puts "world" end def world puts

    "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2
  242. > end def world puts "hello" end def foo foo

    = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2
  243. > end def world puts "hello" end def foo foo

    = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2 #3
  244. 🙇 🙇 🙇 🙇 Those are our main gotchas. Let's

    move on
  245. AI Now you know the gotchas. I want to start

    moving a little closer to the code. Who knows what AI is? Raise your hand, or in chat type "me"
  246. Artificial Intelligence Algorithm - Arti fi cial Intelligence is a

    fancy way of saying <click> "Algorithm" [wait for dialog] "Come on no you're not" <click>
  247. Artificial Intelligence Algorithm <click: laughter> More speci fi cally, AI

    is a goal seeking algorithm
  248. Pathfinding Credit: Factorio 'New pathfinding algorithm' - A common example

    of AI is path fi nding. You want to get from point A to point B
  249. $ gem install dead_end Dead end uses a search algorithm

    to fi nd the problem code. Which uses a variation on uniform cost search
  250. Credit @redblobgames: Introduction to the A-star Algorithm - You need

    a way to break down the problem into discrete actions that either get you closer or further from the goal. In path fi nding this would be deciding what turns to make and what roads to drive on.
  251. Credit @redblobgames: Introduction to the A-star Algorithm - The algorithm

    dead_end uses is also sometimes called dijkstra's algorithm because he invented uniform cost search.
  252. Credit @redblobgames: Introduction to the A-star Algorithm I highly recommend

    this interactive page from redblob games as an introduction to search
  253. Credit @redblobgames: Introduction to the A-star Algorithm A search algorithm

    Is good for when you know what goal you want but you're not quite sure how to get there.
  254. > while b != 0 if a > b a

    = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a Before we saw how code is represented by abstract syntax trees.
  255. Tree
 Search
 BOTVINNIK, M.M. (1984). "Computers in Chess Solving Inexact

    Search Problems" That was an important realization for me, as most search problems are represented as a tree or a graph. Here's a quick example of a famous chess problem
  256. Tree
 Search
 BOTVINNIK, M.M. (1984). "Computers in Chess Solving Inexact

    Search Problems" If you have an exact tree you can walk the tree
  257. require "ripper" source = <<~EOM module Cutlass defcall end end

    EOM Ripper.new(source).tap(&:parse).error? # => true In our case, we have bits of source code
  258. require "ripper" source = <<~EOM module Cutlass # defcall end

    end EOM Ripper.new(source).tap(&:parse).error? # => true I found that if I could give my program rules about how to create logical code blocks
  259. require "ripper" source = <<~EOM module Cutlass # defcall #

    end end EOM Ripper.new(source).tap(&:parse).error? # => false Then using ripper we can check a document without their presence
  260. # defcall # end Ripper.new(source).tap(&:parse).error? # => false And complete

    our search
  261. Internals We'll now take a closer look at some dead_end

    algorithms
  262. 🎂 If dead end was a cake here's the main

    steps to make it
  263. We start out with messy code

  264. Dead end then transforms that code into a tidier input

  265. "Time to cook" by Robbert van der Steeg is licensed

    under CC BY-SA 2.0 It gives that input to our search algorithm to fi nd the syntax errors.
  266. Once the syntax errors are found, the output is decorated

    with more context and given back to the user
  267. 🎂 Let's start at the beginning

  268. Your app has a syntax error

  269. module Kernel module_function def require(file) dead_end_original_require(file) rescue SyntaxError => e

    DeadEnd.handle_error(e) end end dead_end/auto.rb Like all good Ruby libraries dead end uses monkey patching <click: laughter>. It hooks into require and friends. When a syntax error is raised, it's passed to dead_end
  270. # ... Timeout.timeout(timeout) do record_dir ||= ENV["DEBUG"] ? "tmp" :

    nil search = CodeSearch.new(source, record_dir: record_dir).call end # ... dead_end/internals.rb The source code that caused the syntax error is read to disk and then passed to a search object.
  271. First we need to clean it up

  272. clean_document.rb # frozen_string_literal: true module DeadEnd # Parses and sanitizes

    source into a lexically aware document # # Internally the document is represented by an array with each # index containing a CodeLine correlating to a line from the source code. # # There are three main phases in the algorithm: # # 1. Sanitize/format input source # 2. Search for invalid blocks # 3. Format invalid blocks into something meaninful # # This class handles the first part. # # The reason this class exists is to format input source # for better/easier/cleaner exploration. Here's the clean_document class
  273. clean_document.rb @source = source @document = CodeLine.from_source(@source) end # Call

    all of the document "cleaners" # and return self def call clean_sweep .join_trailing_slash! .join_consecutive! .join_heredoc! self end # Return an array of CodeLines in the # document def lines It handles several of the gotcha cases from before by transforming the source code
  274. clean_document.rb @source = source @document = CodeLine.from_source(@source) end # Call

    all of the document "cleaners" # and return self def call clean_sweep .join_trailing_slash! .join_consecutive! .join_heredoc! self end # Return an array of CodeLines in the # document def lines It clears out comments and stray whitespace
  275. clean_document.rb @source = source @document = CodeLine.from_source(@source) end # Call

    all of the document "cleaners" # and return self def call clean_sweep .join_trailing_slash! .join_consecutive! .join_heredoc! self end # Return an array of CodeLines in the # document def lines @document Joins lines with trailing slashes
  276. clean_document.rb @source = source @document = CodeLine.from_source(@source) end # Call

    all of the document "cleaners" # and return self def call clean_sweep .join_trailing_slash! .join_consecutive! .join_heredoc! self end # Return an array of CodeLines in the # document def lines @document Lines with chained methods
  277. clean_document.rb @source = source @document = CodeLine.from_source(@source) end # Call

    all of the document "cleaners" # and return self def call clean_sweep .join_trailing_slash! .join_consecutive! .join_heredoc! self end # Return an array of CodeLines in the # document def lines @document And heredocs
  278. "Time to cook" by Robbert van der Steeg is licensed

    under CC BY-SA 2.0 Next up we feed that to our search class
  279. code_search.rb # Searches code for a syntax error # #

    The bulk of the heavy lifting is done in: # # - CodeFrontier (Holds information for generating blocks and determining if we can stop searching) # - ParseBlocksFromLine (Creates blocks into the frontier) # - BlockExpand (Expands existing blocks to search more code # # ## Syntax error detection # # When the frontier holds the syntax error, we can stop searching # # search = CodeSearch.new(<<~EOM) # def dog # def lol # end # EOM Here's the CodeSearch class
  280. end # Main search loop def call until frontier.holds_all_syntax_errors? @tick

    += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self end code_search.rb Here's call
  281. end # Main search loop def call until frontier.holds_all_syntax_errors? @tick

    += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self code_search.rb Under every great AI algorithm is a while loop. There's a class called the frontier. It holds all the code blocks that we've generated from the source code.
  282. end # Main search loop def call sweep_heredocs sweep_comments until

    frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } code_search.rb 📕 ✅ 📕 ❌ The frontier is responsible for checking the exit condition, here's that logic
  283. > 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename

    ❯ 5 ❯ 16 ❯ 35 ❯ 36 def filename ❯ 37 ❯ 46 63 end 64 end 📕 ❌ If the highlighted lines are on the frontier
  284. > 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename

    63 end 64 end 📕 ✅ Then the entire document without those lines are checked against the parser
  285. # ... end module DeadEnd # ... def self.invalid?(source) source

    = source.join if source.is_a?(Array) source = source.to_s Ripper.new(source).tap(&:parse).error? end # ... end dead_end.rb And here's where the parser is called. <click> Remember Ripper from before?
  286. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s 58 string << "\e[0m" 59 string 60 end 61 end.join 62 end 63 end 64 end Let's walk through the logic of searching this same document we saw before. From the beginning
  287. end # Main search loop def call until frontier.holds_all_syntax_errors? @tick

    += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self end code_search.rb Since we start with an invalid source code, the frontier is empty so this is false
  288. end # Main search loop def call until frontier.holds_all_syntax_errors? @tick

    += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self end code_search.rb There's two modes of exploration. We can add new things onto the frontier, or we can expand an existing block o ff of the frontier. When do we expand?
  289. def expand? return false if @frontier.empty? # ... # Expand

    all blocks before moving to unvisited lines frontier_indent >= unvisited_indent end code_frontier.rb We can't expand a block if there's nothing there.
  290. def expand? return false if @frontier.empty? # ... # Expand

    all blocks before moving to unvisited lines frontier_indent >= unvisited_indent end code_frontier.rb We're also using indentation to decide when to expand. Essentially we want to explore the largest indentations fi rst.
  291. end # Main search loop def call sweep_heredocs sweep_comments until

    frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } code_search.rb There's nothing to expand yet so this is false
  292. end # Main search loop def call sweep_heredocs sweep_comments until

    frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } code_search.rb So we must fi rst add lines to our frontier
  293. # Parses the most indented lines into blocks that are

    marked # and added to the frontier def visit_new_blocks max_indent = frontier.next_indent_line&.indent while (line = frontier.next_indent_line) && (line.indent == max_indent) @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block| record(block: block, name: "add") block.mark_invisible if block.valid? push(block, name: "add") end end end # Given an already existing block in the frontier, expand it to see # if it contains our invalid syntax code_search.rb The code here is looking at the largest indentation, starting at the bottom and iterating up.
  294. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
  295. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
  296. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
  297. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
  298. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
  299. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else ❯ 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
  300. # Parses the most indented lines into blocks that are

    marked # and added to the frontier def visit_new_blocks max_indent = frontier.next_indent_line&.indent while (line = frontier.next_indent_line) && (line.indent == max_indent) @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block| record(block: block, name: "add") block.mark_invisible if block.valid? push(block, name: "add") end end end # Given an already existing block in the frontier, expand it to see # if it contains our invalid syntax code_search.rb This step only uses indentation.
  301. # Parses the most indented lines into blocks that are

    marked # and added to the frontier def visit_new_blocks max_indent = frontier.next_indent_line&.indent while (line = frontier.next_indent_line) && (line.indent == max_indent) @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block| record(block: block, name: "add") block.mark_invisible if block.valid? push(block, name: "add") end end end # Given an already existing block in the frontier, expand it to see code_search.rb If the block that it found is valid ruby code, it's essentially commented out.
  302. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else ❯ 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before
  303. > 47 def code_with_lines 48 @code_lines.map do |line| 49 next

    if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 60 end 61 end.join 62 end 63 end 64 end
  304. # Parses the most indented lines into blocks that are

    marked # and added to the frontier def visit_new_blocks max_indent = frontier.next_indent_line&.indent while (line = frontier.next_indent_line) && (line.indent == max_indent) @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block| record(block: block, name: "add") block.mark_invisible if block.valid? push(block, name: "add") end end end # Given an already existing block in the frontier, expand it to see code_search.rb The code block is then pushed onto the frontier and the loop continues
  305. sweep(block: block, name: "comments") end # Main search loop def

    call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb We've still not found the problem we will continue
  306. sweep(block: block, name: "comments") end # Main search loop def

    call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb There's still lines with more indentation
  307. sweep(block: block, name: "comments") end # Main search loop def

    call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb We need to add them
  308. > 43 string << "```\n" 44 string 45 end 46

    47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? ❯ 52 "#{number.to_s}#{line}" 53 else 60 end 61 end.join 62 end 63 end 64 end
  309. > 43 string << "```\n" 44 string 45 end 46

    47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 53 else 60 end 61 end.join 62 end 63 end 64 end Boop
  310. sweep(block: block, name: "comments") end # Main search loop def

    call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb We've added all lines at this indent level. It's time to expand the frontier
  311. code_search.rb sweep(block: block, name: "comments") end # Main search loop

    def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) We call our expansion phase
  312. end end end # Given an already existing block in

    the frontier, expand it to see # if it contains our invalid syntax def expand_invalid_block block = frontier.pop return unless block record(block: block, name: "pop") block = @block_expand.call(block) push(block, name: "expand") end def sweep_heredocs HeredocBlockParse.new( source: @source, code_search.rb Here's our expand code
  313. end end end # Given an already existing block in

    the frontier, expand it to see # if it contains our invalid syntax def expand_invalid_block block = frontier.pop return unless block record(block: block, name: "pop") block = @block_expand.call(block) push(block, name: "expand") end def sweep_heredocs HeredocBlockParse.new( source: @source, code_search.rb The frontier is stored in indentation order. The highest indentation block is popped o f
  314. end end end # Given an already existing block in

    the frontier, expand it to see # if it contains our invalid syntax def expand_invalid_block block = frontier.pop return unless block record(block: block, name: "pop") block = @block_expand.call(block) push(block, name: "expand") end def sweep_heredocs HeredocBlockParse.new( source: @source, code_search.rb The block is expanded using an object and pushed back onto the frontier. How does block expansion work?
  315. # puts "wow" # end # class BlockExpand def initialize(code_lines:

    ) @code_lines = code_lines end def call(block) if (next_block = expand_neighbors(block, grab_empty: true)) return next_block end expand_indent(block) end def expand_indent(block) block = AroundBlockScan.new(code_lines: @code_lines, block: block) block_expand.rb It can either expand to capture more lines at the same indentation, aka neighbors. Or when those are exhausted it can expand out an indentation.
  316. def expand_indent(block) block = AroundBlockScan.new(code_lines: @code_lines, block: block) .skip(:hidden?) .stop_after_kw

    .scan_adjacent_indent .code_block end def expand_neighbors(block, grab_empty: true) scan = AroundBlockScan.new(code_lines: @code_lines, block: block) .skip(:hidden?) .stop_after_kw .scan_neighbors # Slurp up empties if grab_empty scan = AroundBlockScan.new(code_lines: @code_lines, block: scan.code_block) block_expand.rb Both methods use this class AroundBlockScan which acts as a DSL for scanning up and down. This is where we give our algorithm rules about how to build blocks using indentation and lexical data
  317. > 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if

    filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? ❯ 52 "#{number.to_s}#{line}" 53 else 60 end 61 end.join 62 end 63 end 64 end Previously we captured this line, and put it on the frontier.
  318. > 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if

    filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) ❯ 51 if line.empty? ❯ 53 else ❯ 60 end 61 end.join 62 end 63 end 64 end When it's popped and expanded it looks around and fi nds a matched set of keywords using the scanner.
  319. > 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if

    filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 61 end.join 62 end 63 end 64 end The code is valid so it's hidden. The expanded block is put back on the frontier.
  320. sweep(block: block, name: "comments") end # Main search loop def

    call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb That's pretty much how the algorithm for search works. It keeps looping, expanding and adding until all syntax errors are found
  321. sweep(block: block, name: "comments") end # Main search loop def

    call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb Once we've removed all syntax errors, we have to fi nd our problematic code.
  322. def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if

    frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self end end code_search.rb The frontier can hold more than one syntax error.
  323. # Example: # # combination([:a, :b, :c, :d]) # #

    => [[:a], [:b], [:c], [:d], # [:a, :b], [:a, :c], [:a, :d], # [:b, :c], [:b, :d], [:c, :d], # [:a, :b, :c], [:a, :b, :d], [:a, :c, :d], # [:b, :c, :d], # [:a, :b, :c, :d]] def self.combination(array) guesses = [] 1.upto(array.length).each do |size| guesses.concat(array.combination(size).to_a) end guesses end # Given that we know our syntax error exists somewhere in our frontier, we want to find code_frontier.rb We want to iterate over all of the invalid blocks stored in the frontier to fi nd the shortest possible order that gives us a valid document.
  324. # Example: # # combination([:a, :b, :c, :d]) # #

    => [[:a], [:b], [:c], [:d], # [:a, :b], [:a, :c], [:a, :d], # [:b, :c], [:b, :d], [:c, :d], # [:a, :b, :c], [:a, :b, :d], [:a, :c, :d], # [:b, :c, :d], # [:a, :b, :c, :d]] def self.combination(array) guesses = [] 1.upto(array.length).each do |size| guesses.concat(array.combination(size).to_a) end guesses end # Given that we know our syntax error exists somewhere in our frontier, we want to find code_frontier.rb To do that we create a combination of all possible blocks
  325. end # Given that we know our syntax error exists

    somewhere in our frontier, we want to find # the smallest possible set of blocks that contain all the syntax errors def detect_invalid_blocks self.class.combination(@frontier.select(&:invalid?)).detect do |block_array| holds_all_syntax_errors?(block_array) end || [] end end end code_frontier.rb Then one at a time we iterate through all the combinations
  326. guesses end # Given that we know our syntax error

    exists somewhere in our frontier, we want to find # the smallest possible set of blocks that contain all the syntax errors def detect_invalid_blocks self.class.combination(@frontier.select(&:invalid?)).detect do |block_array| holds_all_syntax_errors?(block_array) end || [] end end end code_frontier.rb And check each one to see if we hold all the syntax errors. The smallest successful combination is returned
  327. Timeout.timeout(timeout) do record_dir ||= ENV["DEBUG"] ? "tmp" : nil search

    = CodeSearch.new(source, record_dir: record_dir).call end blocks = search.invalid_blocks DisplayInvalidBlocks.new( blocks: blocks, filename: filename, terminal: terminal, code_lines: search.code_lines, invalid_obj: invalid_type(source), io: io ).call dead_end/internals.rb Here's where our search started.
  328. Timeout.timeout(timeout) do record_dir ||= ENV["DEBUG"] ? "tmp" : nil search

    = CodeSearch.new(source, record_dir: record_dir).call end blocks = search.invalid_blocks DisplayInvalidBlocks.new( blocks: blocks, filename: filename, terminal: terminal, code_lines: search.code_lines, invalid_obj: invalid_type(source), io: io ).call dead_end/internals.rb Then the invalid blocks were found
  329. Now we want to decorate the output

  330. Timeout.timeout(timeout) do record_dir ||= ENV["DEBUG"] ? "tmp" : nil search

    = CodeSearch.new(source, record_dir: record_dir).call end blocks = search.invalid_blocks DisplayInvalidBlocks.new( blocks: blocks, filename: filename, terminal: terminal, code_lines: search.code_lines, invalid_obj: invalid_type(source), io: io ).call dead_end/internals.rb We pass it to a pretty printer display
  331. lines = CaptureCodeContext.new( blocks: @blocks, code_lines: @code_lines ).call DisplayCodeWithLineNumbers.new( lines:

    lines, terminal: @terminal, highlight_lines: @invalid_lines ).call display_invalid_blocks.rb Here's where we take care of the last of our gotchas
  332. lines = CaptureCodeContext.new( blocks: @blocks, code_lines: @code_lines ).call DisplayCodeWithLineNumbers.new( lines:

    lines, terminal: @terminal, highlight_lines: @invalid_lines ).call display_invalid_blocks.rb This capture code context handles the ambiguities we talked about before
  333. > ❯ 1 class Dog Gotcha: ambiguity 🚫 It knows

    about our ambiguous <click> edge cases that we talked about before
  334. > ❯ 1 class Dog ❯ 2 def sit ❯

    3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity It goes back through the hidden code
  335. > ❯ 1 class Dog ❯ 2 def sit ❯

    3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity
  336. > ❯ 1 class Dog ❯ 2 def sit ❯

    3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity Work backwards and fi nd an unmatched keyword
  337. > ❯ 1 class Dog ❯ 10 def bark ❯

    12 end Gotcha: ambiguity And transforms it
  338. 🎂 Piece of cake

  339. > DeadEnd: Missing `end` detected This code has a missing

    `end`. Ensure that all syntax keywords (`def`, `do`, etc.) have a matching `end`. file: /Users/rschneeman/Documents/projects/dead_end/spec/fixtures/this_project_extra_def.rb.txt simplified: 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename ❯ 36 def filename ❯ 38 def code_with_filename ❯ 45 end 63 end 64 end After all three steps, cleaning, searching, formatting here's the output
  340. dead_end That is dead end <click> in a nutshell *laugh*

  341. Gem `error_highlight` in Ruby 3.1 Beyond dead_end there's an amazing

    gem being added to Ruby 3.1 called error_highlight. It shows you which method got a no method error. I love this introduction
  342. Gem `error_highlight` in Ruby 3.1 The error highlight gem was

    created by you-ssssss-kay. He goes by mam-eh on GitHub and gave an amazing talk at a conference I used to run called keep ruby weird.
  343. Rust v. Ruby With error highlight and dead end I

    also want to touch on community values.
  344. Rust v. Ruby I've been writing Rust code and their

    community takes a very aggressive stance towards error messages. It's not enough for them to say there's a problem. They also want to accurately suggest how to fi x the problem
  345. Rust v. Ruby I opened this issue against rustlang and

    in under a month it was addressed to improve the error output.
  346. Rust v. Ruby In general the Rust community treats a

    user experience problem as a bug and error output is a fi rst class feature. If our community wanted we could invest more in errors
  347. Dead end isn't that solution, but it's a start. You

    can add the dead_end gem to your project to try it today and give me feedback on what works and what doesn't.
  348. 📕 HowTo Open Source .dev You can pre-order my book

    on open source contribution by visiting how to open source dot dev
  349. You can sign up to triage open source issues at

    code triage dot com
  350. Lexing Parsing Syntax SyntaxErrors AI Pathfinding and goal seeking Today

    we talked about lexing, parsing, syntax syntax errors, AI, path fi nding and goal seeking
  351. But, the technical pieces are not what I want you

    to remember.
  352. - The important part is that everyone sitting in this

    room is the future of developer tooling. Programming is inherently di ffi cult, but it doesn't mean our tools can't help us.
  353. One of the best ways to judge a system is

    to see how it fails. When we can give care and grace and beauty to our failure modes then we can create experiences that delight us. Experiences that teach us. Experiences that elevate our code and our consciousness.
  354. - A syntax error doesn't have to be an ending...it

    can be a beginning. To a beautiful programming story
  355. How To Open Source .dev - My name is Richard

    Schneeman - I'm writing a book at how to open source dot dev <click> - I created <click> CodeTriage.com the easiest way to get started contributing to OpenSource - <click> - Wait for "bye bye" - Zoom wave everyone
  356. None
  357. Questions?