Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[RubyConf] Beware the Dreaded Dead End

[RubyConf] Beware the Dreaded Dead End

Nothing stops a program from executing quite as fast as a syntax error. After years of “unexpected end” in my dev life, I decided to “do” something about it. In this talk we'll cover lexing, parsing, and indentation informed syntax tree search that power that dead_end Ruby library.

I also gave this talk at RubyKaigi. I recommend this version, it is more up-to-date and covers specific details and "gotchas" on the problem space.

Richard Schneeman

October 28, 2021
Tweet

More Decks by Richard Schneeman

Other Decks in Technology

Transcript

  1. Beware
    the Dreaded
    Dead End!!!
    By @schneems
    Hello everyone. My name is Richard Schneeman and I want to talk to you about the scariest thing a Ruby programmer can face.

    You hear that?

    View Slide

  2. Everyone run

    View Slide

  3. Oh no, there are more of them. AHHHH

    View Slide

  4. Beware
    the Dreaded
    Dead End!!!
    By @schneems
    Wow, I can hardly believe that the those dinosaurs from RubyKaigi found me all the way over here, I thought I escaped them already.

    If you've seen that talk, know that this one has some new content.

    View Slide

  5. Beware
    the Dreaded
    Dead End!!!
    By @schneems
    While dinosaurs are scary. There's something scarier. Its..

    View Slide

  6. >
    # <= routes.rb:121: syntax error, unexpected end-of-input, expecting `end’
    Rails.application.routes.draw do


    constraints -> { Rails.application.config.non_production } do
    namespace :foo do


    resource :bar


    end


    end


    constraints -> { Rails.application.config.non_production } do
    namespace :bar do


    resource :baz


    end


    end


    namespace :admin do


    resource :session


    match "/foobar(*path)", via: :all, to: redirect { |_params, req|
    uri = URI(req.path.gsub("foobar", "foobaz"))


    uri.query = req.query_string.presence


    uri.to_s


    }


    end
    Syntax errors

    View Slide

  7. >
    Rails.application.routes.draw do


    constraints -> { Rails.application.config.non_production } do
    namespace :foo do


    resource :bar


    end


    end


    constraints -> { Rails.application.config.non_production } do
    namespace :bar do


    resource :baz


    end


    end


    namespace :admin do


    resource :session


    match "/foobar(*path)", via: :all, to: redirect { |_params, req|
    uri = URI(req.path.gsub("foobar", "foobaz"))


    uri.query = req.query_string.presence


    uri.to_s


    }


    end # <= routes.rb:121: syntax error, unexpected end-of-input, expecting `end’
    Just look at this unexpected syntax error.

    It's horrifying.

    Where's the problem?

    View Slide

  8. >
    RSpec.describe Cutlass::BashResult do


    it "preserves stdout, stderr, and status"


    stdout = SecureRandom.hex(16)


    stderr = SecureRandom.hex(16)


    status = 0


    result = BashResult.new(


    stdout: stdout,


    stderr: stderr,


    status: status


    )


    expect(result.stdout).to eq(stdout)


    expect(result.stderr).to eq(stderr)


    expect(result.status).to eq(status)


    end


    end # <= result_spec:19 syntax error, unexpected end-of-input, expecting `end’
    Here's some di
    ff
    erent code. Wanna guess where Ruby think the problem is?

    On the last line, nope. This is frustrating

    View Slide

  9. 🤔
    What if we had something better

    View Slide

  10. >
    RSpec.describe Cutlass::BashResult do


    it "preserves stdout, stderr, and status"


    stdout = SecureRandom.hex(16)


    stderr = SecureRandom.hex(16)


    status = 0


    result = BashResult.new(


    stdout: stdout,


    stderr: stderr,


    status: status


    )


    expect(result.stdout).to eq(stdout)


    expect(result.stderr).to eq(stderr)


    expect(result.status).to eq(status)


    end


    end # <= result_spec:19 syntax error, unexpected end-of-input, expecting `end’
    Instead of this

    View Slide

  11. > 3 module Cutlass


    4 RSpec.describe Cutlass::BashResult do
    ❯ 5 it "preserves stdout, stderr, and status"
    ❯ 19 end


    21 it "success?" do


    52 end


    53 end


    54 end
    This code has an unmatched `end`. Ensure that all
    `end` lines in your code have a matching syntax
    keyword (`def`, `do`, etc.)
    What if we got this?

    Do you see the error now? Doesn't line 5 look suspicious, like maybe it's missing a do.

    View Slide

  12. 🎂
    Have you ever seen a cooking show where they show o
    ff
    the
    fi
    nal product? I want to show o
    ff
    dead end before we get too far.

    View Slide

  13. demo
    Here is a demo of how dead_end
    fi
    nds syntax errors

    View Slide

  14. module SyntaxErrorSearch


    # Used for formatting invalid blocks


    class DisplayInvalidBlocks


    attr_reader :filename


    def initialize(block_array, io: $stderr, filename: nil)


    @filename = filename


    @io = io


    @blocks = block_array


    @lines = @blocks.map(&:lines).flatten


    @digit_count = @lines.last.line_number.to_s.length


    @code_lines = @blocks.first.code_lines


    @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}
    end


    def call


    @io.puts <<~EOM


    64: syntax error, unexpected end-of-input,
    expecting `end'
    📕

    This is real ruby source code with a syntax error in it. It cannot parse

    View Slide

  15. 51 if line.empty?


    52 "#{number.to_s}#{line}"


    53 else


    54 string = String.new


    55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics
    56 string << "#{number.to_s} "


    57 string << line.to_s


    58 string << "\e[0m"


    59 string


    60 end


    61 end.join


    62 end


    63 end


    64 end



    View Slide

  16. 50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    52 "#{number.to_s}#{line}"


    53 else


    54 string = String.new


    55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics
    56 string << "#{number.to_s} "


    57 string << line.to_s


    58 string << "\e[0m"


    ❯ 59 string


    60 end


    61 end.join


    62 end


    63 end


    64 end


    View Slide

  17. 50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    52 "#{number.to_s}#{line}"


    53 else


    54 string = String.new


    55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics
    56 string << "#{number.to_s} "


    57 string << line.to_s


    ❯ 58 string << "\e[0m"


    ❯ 59 string


    60 end


    61 end.join


    62 end


    63 end


    64 end


    View Slide

  18. 50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    52 "#{number.to_s}#{line}"


    53 else


    54 string = String.new


    55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics
    56 string << "#{number.to_s} "


    ❯ 57 string << line.to_s


    ❯ 58 string << "\e[0m"


    ❯ 59 string


    60 end


    61 end.join


    62 end


    63 end


    64 end


    View Slide

  19. 50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    52 "#{number.to_s}#{line}"


    53 else


    54 string = String.new


    55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics
    ❯ 56 string << "#{number.to_s} "


    ❯ 57 string << line.to_s


    ❯ 58 string << "\e[0m"


    ❯ 59 string


    60 end


    61 end.join


    62 end


    63 end


    64 end


    View Slide

  20. 50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    52 "#{number.to_s}#{line}"


    53 else


    54 string = String.new


    ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics
    ❯ 56 string << "#{number.to_s} "


    ❯ 57 string << line.to_s


    ❯ 58 string << "\e[0m"


    ❯ 59 string


    60 end


    61 end.join


    62 end


    63 end


    64 end


    View Slide

  21. 50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    52 "#{number.to_s}#{line}"


    53 else


    ❯ 54 string = String.new


    ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics
    ❯ 56 string << "#{number.to_s} "


    ❯ 57 string << line.to_s


    ❯ 58 string << "\e[0m"


    ❯ 59 string


    60 end


    61 end.join


    62 end


    63 end


    64 end


    When a valid code block is found dead_end safely removes it

    View Slide

  22. 50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    52 "#{number.to_s}#{line}"


    53 else


    60 end


    61 end.join


    62 end


    63 end


    64 end


    📕

    After each step in the search, dead end re-evaluates the whole document to see if it's parsable yet. parsing failed, we need to keep looking

    View Slide

  23. 42 string << code_with_lines


    43 string << "```\n"


    44 string


    45 end


    46


    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    ❯ 52 "#{number.to_s}#{line}"


    53 else


    60 end


    61 end.join


    62 end


    63 end


    64 end


    View Slide

  24. 42 string << code_with_lines


    43 string << "```\n"


    44 string


    45 end


    46


    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    53 else


    60 end


    61 end.join


    62 end


    63 end


    64 end


    📕

    View Slide

  25. 42 string << code_with_lines


    43 string << "```\n"


    44 string


    45 end


    46


    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    ❯ 51 if line.empty?


    ❯ 53 else


    ❯ 60 end


    61 end.join


    62 end


    63 end


    64 end


    View Slide

  26. 42 string << code_with_lines


    43 string << "```\n"


    44 string


    45 end


    46


    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    61 end.join


    62 end


    63 end


    64 end


    📕

    View Slide

  27. 42 string << code_with_lines


    43 string << "```\n"


    44 string


    45 end


    46


    47 def code_with_lines


    48 @code_lines.map do |line|


    ❯ 49 next if line.hidden?


    ❯ 50 number = line.line_number.to_s.rjust(@digit_count)


    61 end.join


    62 end


    63 end


    64 end


    View Slide

  28. 42 string << code_with_lines


    43 string << "```\n"


    44 string


    45 end


    46


    47 def code_with_lines


    48 @code_lines.map do |line|


    61 end.join


    62 end


    63 end


    64 end


    📕

    View Slide

  29. 42 string << code_with_lines


    43 string << "```\n"


    44 string


    45 end


    46


    47 def code_with_lines


    ❯ 48 @code_lines.map do |line|


    ❯ 61 end.join


    62 end


    63 end


    64 end


    View Slide

  30. 42 string << code_with_lines


    43 string << "```\n"


    44 string


    45 end


    46


    47 def code_with_lines


    62 end


    63 end


    64 end


    📕

    View Slide

  31. 42 string << code_with_lines


    43 string << "```\n"


    44 string


    45 end


    46


    ❯ 47 def code_with_lines


    ❯ 62 end


    63 end


    64 end


    View Slide

  32. 40 string << "```\n"


    41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename
    42 string << code_with_lines


    43 string << "```\n"


    44 string


    45 end


    46


    63 end


    64 end


    📕

    View Slide

  33. 13


    14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}
    15 end


    16


    35


    36 def filename


    37


    38 def code_with_filename


    39 string = String.new("")


    40 string << "```\n"


    41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename
    42 string << code_with_lines


    43 string << "```\n"


    44 string


    45 end


    46


    63 end


    64 end


    View Slide

  34. 13


    14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}
    15 end


    16


    35


    36 def filename


    37


    38 def code_with_filename


    39 string = String.new("")


    40 string << "```\n"


    41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename
    42 string << code_with_lines


    43 string << "```\n"


    ❯ 44 string


    45 end


    46


    63 end


    64 end


    View Slide

  35. 13


    14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}
    15 end


    16


    35


    36 def filename


    37


    38 def code_with_filename


    39 string = String.new("")


    40 string << "```\n"


    41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename
    42 string << code_with_lines


    ❯ 43 string << "```\n"


    ❯ 44 string


    45 end


    46


    63 end


    64 end


    View Slide

  36. 13


    14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}
    15 end


    16


    35


    36 def filename


    37


    38 def code_with_filename


    39 string = String.new("")


    40 string << "```\n"


    41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename
    ❯ 42 string << code_with_lines


    ❯ 43 string << "```\n"


    ❯ 44 string


    45 end


    46


    63 end


    64 end


    View Slide

  37. 13


    14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}
    15 end


    16


    35


    36 def filename


    37


    38 def code_with_filename


    39 string = String.new("")


    40 string << "```\n"


    ❯ 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename
    ❯ 42 string << code_with_lines


    ❯ 43 string << "```\n"


    ❯ 44 string


    45 end


    46


    63 end


    64 end


    View Slide

  38. 13


    14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}
    15 end


    16


    35


    36 def filename


    37


    38 def code_with_filename


    39 string = String.new("")


    ❯ 40 string << "```\n"


    ❯ 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename
    ❯ 42 string << code_with_lines


    ❯ 43 string << "```\n"


    ❯ 44 string


    45 end


    46


    63 end


    64 end


    View Slide

  39. 13


    14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}
    15 end


    16


    35


    36 def filename


    37


    38 def code_with_filename


    ❯ 39 string = String.new("")


    ❯ 40 string << "```\n"


    ❯ 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename
    ❯ 42 string << code_with_lines


    ❯ 43 string << "```\n"


    ❯ 44 string


    45 end


    46


    63 end


    64 end


    View Slide

  40. 13


    14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}
    15 end


    16


    35


    36 def filename


    37


    38 def code_with_filename


    45 end


    46


    63 end


    64 end


    📕

    View Slide

  41. 1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    5


    6 def initialize(block_array, io: $stderr, filename: nil)
    7 @filename = filename


    8 @io = io


    9 @blocks = block_array


    10 @lines = @blocks.map(&:lines).flatten


    11 @digit_count = @lines.last.line_number.to_s.length


    12 @code_lines = @blocks.first.code_lines


    13


    14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}
    15 end


    16


    35


    36 def filename


    37


    38 def code_with_filename


    View Slide

  42. 1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    5


    6 def initialize(block_array, io: $stderr, filename: nil)
    7 @filename = filename


    8 @io = io


    9 @blocks = block_array


    10 @lines = @blocks.map(&:lines).flatten


    11 @digit_count = @lines.last.line_number.to_s.length


    12 @code_lines = @blocks.first.code_lines


    13


    ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}
    15 end


    16


    35


    36 def filename


    37


    38 def code_with_filename


    View Slide

  43. 1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    5


    6 def initialize(block_array, io: $stderr, filename: nil)
    7 @filename = filename


    8 @io = io


    9 @blocks = block_array


    10 @lines = @blocks.map(&:lines).flatten


    11 @digit_count = @lines.last.line_number.to_s.length


    ❯ 12 @code_lines = @blocks.first.code_lines


    ❯ 13


    ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}
    15 end


    16


    35


    36 def filename


    37


    38 def code_with_filename


    View Slide

  44. 1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    5


    6 def initialize(block_array, io: $stderr, filename: nil)
    7 @filename = filename


    8 @io = io


    9 @blocks = block_array


    10 @lines = @blocks.map(&:lines).flatten


    ❯ 11 @digit_count = @lines.last.line_number.to_s.length


    ❯ 12 @code_lines = @blocks.first.code_lines


    ❯ 13


    ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}
    15 end


    16


    35


    36 def filename


    37


    38 def code_with_filename


    View Slide

  45. 1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    5


    6 def initialize(block_array, io: $stderr, filename: nil)
    7 @filename = filename


    8 @io = io


    9 @blocks = block_array


    ❯ 10 @lines = @blocks.map(&:lines).flatten


    ❯ 11 @digit_count = @lines.last.line_number.to_s.length


    ❯ 12 @code_lines = @blocks.first.code_lines


    ❯ 13


    ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}
    15 end


    16


    35


    36 def filename


    37


    38 def code_with_filename


    View Slide

  46. 1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    5


    6 def initialize(block_array, io: $stderr, filename: nil)
    7 @filename = filename


    8 @io = io


    ❯ 9 @blocks = block_array


    ❯ 10 @lines = @blocks.map(&:lines).flatten


    ❯ 11 @digit_count = @lines.last.line_number.to_s.length


    ❯ 12 @code_lines = @blocks.first.code_lines


    ❯ 13


    ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}
    15 end


    16


    35


    36 def filename


    37


    38 def code_with_filename


    View Slide

  47. 1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    5


    6 def initialize(block_array, io: $stderr, filename: nil)
    7 @filename = filename


    ❯ 8 @io = io


    ❯ 9 @blocks = block_array


    ❯ 10 @lines = @blocks.map(&:lines).flatten


    ❯ 11 @digit_count = @lines.last.line_number.to_s.length


    ❯ 12 @code_lines = @blocks.first.code_lines


    ❯ 13


    ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}
    15 end


    16


    35


    36 def filename


    37


    38 def code_with_filename


    View Slide

  48. 1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    5


    6 def initialize(block_array, io: $stderr, filename: nil)
    ❯ 7 @filename = filename


    ❯ 8 @io = io


    ❯ 9 @blocks = block_array


    ❯ 10 @lines = @blocks.map(&:lines).flatten


    ❯ 11 @digit_count = @lines.last.line_number.to_s.length


    ❯ 12 @code_lines = @blocks.first.code_lines


    ❯ 13


    ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}
    15 end


    16


    35


    36 def filename


    37


    38 def code_with_filename


    View Slide

  49. 1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    5


    6 def initialize(block_array, io: $stderr, filename: nil)
    15 end


    16


    35


    36 def filename


    37


    38 def code_with_filename


    45 end


    46


    63 end


    64 end


    📕

    View Slide

  50. 1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    5


    6 def initialize(block_array, io: $stderr, filename: nil)
    15 end


    16


    35


    36 def filename


    37


    ❯ 38 def code_with_filename


    ❯ 45 end


    46


    63 end


    64 end


    View Slide

  51. 1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    5


    6 def initialize(block_array, io: $stderr, filename: nil)
    15 end


    16


    35


    36 def filename


    37


    46


    63 end


    64 end


    📕

    View Slide

  52. 1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    5


    ❯ 6 def initialize(block_array, io: $stderr, filename: nil)
    ❯ 15 end


    16


    35


    36 def filename


    37


    46


    63 end


    64 end


    View Slide

  53. 📕

    1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    5


    16


    35


    36 def filename


    37


    46


    63 end


    64 end


    View Slide

  54. 1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    ❯ 5


    ❯ 16


    ❯ 35


    ❯ 36 def filename


    ❯ 37


    ❯ 46


    63 end


    64 end


    View Slide

  55. 1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    63 end


    64 end


    📕

    The document is checked. The parser reports that this very minimal document is now parsable.

    View Slide

  56. Syntax O
    🥰🥰🥰
    1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    63 end


    64 end


    📕

    Once this happens dead_end has found a way to transform the document to be valid. It can stop searching the document and instead search the invalid code blocks that
    it has stored

    View Slide

  57. DeadEnd: Missing `end` detected


    This code has a missing `end`. Ensure that all


    syntax keywords (`def`, `do`, etc.) have a matching `end`.
    file: /Users/rschneeman/Documents/projects/dead_end/spec/fixtures/this_project_extra_def.rb.txt
    simplified:


    1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    ❯ 36 def filename


    ❯ 38 def code_with_filename


    ❯ 45 end


    63 end


    64 end
    This is the actual output the dead end gem returns on that
    fi
    le. The issue is on line 36 there's a missing end statement

    View Slide

  58. $ gem install dead_end
    The dead_end gem is released and on ruby gems.

    View Slide

  59. By the time you see this slide dead_end will have surpassed half a million downloads

    View Slide

  60. Ruby 3.2?
    I am also talking with Ruby core to get dead_end into Ruby directly. We are targeting that integration for 2022 release of Ruby 3.2

    View Slide

  61. 🤔
    What all can dead_end do?

    View Slide

  62. >
    3 module Cutlass


    4 RSpec.describe Cutlass::BashResult do
    ❯ 5 it "preserves stdout, stderr, and status"
    ❯ 19 end


    21 it "success?" do


    52 end


    53 end


    54 end
    DeadEnd: Unmatched `end` detected
    When you miss a keyword like "if", "do" or "def". Dead end
    fi
    nds the problem.

    View Slide

  63. > 1 module Cutlass


    2 RSpec.describe Cutlass::BashResult do
    ❯ 3 it "preserves stdout, stderr, and status" do
    5 it "success?" do


    6 end


    7 end


    8 end
    DeadEnd: Missing `end` detected
    Even when you miss an end keyword, dead end
    fi
    nds the problem

    View Slide

  64. >
    1 class Cat


    2 def to_json


    ❯ 3 hash = {


    ❯ 4 name: name,


    ❯ 5 fur_color: fur_color,


    ❯ 6 type: 'Cat',


    ❯ 7 better_than_dog: false


    ❯ 8 hash.to_json


    9 end


    10 end
    DeadEnd: Unmatched `}` character detected
    When you miss a curly bracket dead end
    fi
    nds the problem

    View Slide

  65. >
    1 class Animal


    2 def self.ones_my_three_year_old_likes
    ❯ 3 array = [


    ❯ 4 Dog.new,


    ❯ 5 Lion.new,


    ❯ 6 Tiger.new,


    ❯ 7 Aligator.new,


    10 return array


    11 end


    12 end
    DeadEnd: Unmatched `[` detected
    When you miss a square bracket dead end
    fi
    nds the problem

    View Slide

  66. > 1 class Cat


    2 def meow


    ❯ 3 Animal.call do |a


    ❯ 5 end


    6 end


    7 end
    DeadEnd: Unmatched `|` character detected
    Have a problem with a missing pipe character? Dead end.

    Finds.

    The.

    Problem



    Have a problem with a missing family member in a Korean thriller drama?

    View Slide

  67. >
    Unfortunately. Dead end cannot
    fi
    nd that problem. . Okay, wow that current event reference got really dark. Let's rewind a try again.

    View Slide

  68. >
    Have a problem with a missing marvel universe character?

    View Slide

  69. > 1 class Cat


    2 def meow


    ❯ 3 Animal.call do |a


    ❯ 5 end


    6 end


    7 end
    DeadEnd: Unmatched `|` character detected
    ?
    ?
    ?
    Unfortunately. Dead end cannot
    fi
    nd that problem.

    Much better. Remember crocodile Loki? That was 2021, this year is lasting forever!

    View Slide

  70. $ gem install dead_end
    Today we'll dig into

    View Slide

  71. $ dead_end bad.rb --record tmp/
    You can follow the algorithm yourself by running the CLI with the record
    fl
    ag

    View Slide

  72. You'll see each step

    View Slide

  73. Along with annotated source code

    View Slide

  74. Syntax
    errors
    Today we will talk about

    View Slide

  75. Lexing &
    Parsing
    How to parse and lex source code with ripper

    View Slide

  76. AI
    How AI and path
    fi
    nding algorithms work

    View Slide

  77. dead_end
    We will put it all together to see how dead_end works

    View Slide

  78. Internals
    Finally we'll crack open the cover and see how it works at a low level. Warning: We will get technical. We will cover a lot of code.

    View Slide

  79. 👋
    But wait, who am I?

    View Slide

  80. @schneems
    I go by schneems on the internet. If you forget how to pronounce my name you can go to

    View Slide

  81. @schneems
    the about page on my blog on Schneems dot com

    View Slide

  82. I Created CodeTriage which is a platform for learning how to contribute to open source. To date I've got over 60 thousand developers signed up for CodeTriage

    View Slide

  83. 📕
    HowTo
    Open
    Source
    .dev
    I'm writing a book on how to contribute to open source as well. It's for developers who want to start contributing. Developers who got stuck, and developers looking for
    ways to sustain contributions.

    I'm not ready to release it yet, but you can sign up for a pre-order on HowToOpenSource dot dev

    Also you can sign up for Code Triage dot com and I will email members when the book is ready.

    View Slide

  84. Heroku
    When I'm not working on open source or teaching people to open source with CodeTriage. I like to get paid. I work at Heroku

    View Slide

  85. Heroku
    Right now I'm working on salesforce functions. It's an easy way to work with the data you've got in salesforce using the languages you love. Right now we support Java
    and JavaScript languages. We'll be rolling out Ruby support later. If you use love Ruby and use Salesforce I would love to hear from you. My DMs on Twitter are open.

    Also people tell me

    View Slide

  86. NoMemoryError

    ScriptError

    LoadError

    SyntaxError

    SecurityError

    SignalException

    KeyError

    StandardError

    ArgumentError

    EncodingError

    IOError

    EOFError
    I am Exceptional
    i'm an exceptional programmer...

    My programs generate a lot of exceptions

    View Slide

  87. Syntax
    errors
    Syntax errors. Let's start from the beginning what is a syntax error? Why are syntax errors so hard to understand?

    View Slide

  88. >
    while b != 0


    if a > b


    a = a - b


    else


    b = b - a


    end


    end


    return a
    This code worked wonderfully comparing a, and b all day.

    View Slide

  89. >
    while b != 0


    if a > b


    a = a - b


    else


    b = b - a


    end


    end


    return a
    while b != 0
    if a > b
    a = a - b else
    b = b - a
    When Ruby parses this code it will convert it into an abstract syntax tree.

    this code generates that tree.

    The tree was beautiful and Ruby's parser looked upon it with happiness

    View Slide

  90. >
    while b != 0
    if a > b
    a = a - b else
    b = b - a
    while b != 0


    if a > b


    a = a - b


    else


    b = b - a


    end


    end


    return a
    😈
    While the code was good a stranger came upon the land. This stranger had a secret power, behold

    View Slide

  91. #
    The octothorpe, This tiny character gave the stranger great power to create, or destroy.

    View Slide

  92. >
    while b != 0


    if a > b


    a = a - b


    else


    b = b - a


    end


    end


    return a
    while b != 0
    if a > b
    a = a - b else
    b = b - a
    😈
    With one key the stranger used their power

    View Slide

  93. >
    while b != 0


    # if a > b


    a = a - b


    else


    b = b - a


    end


    end


    return a
    while b != 0
    if a > b
    a = a - b else
    b = b - a
    😈
    to transform the code

    View Slide

  94. >
    while b != 0


    # if a > b


    a = a - b


    else


    b = b - a


    end


    end


    return a
    while b != 0
    if a > b
    a = a - b else
    b = b - a
    😈
    A critical line had been commented out. Without that line, the tree was no longer whole

    View Slide

  95. >
    while b != 0


    # if a > b


    a = a - b


    else


    b = b - a


    end


    end


    return a
    while b != 0
    if a > b
    a = a - b else
    b = b - a
    😈
    Huge sections of code are no longer reachable. The code no longer parses. And our parser is sad.

    View Slide

  96. >
    while b != 0


    # if a > b


    a = a - b


    else


    b = b - a


    end


    end


    return a
    while b != 0
    a = a - b else
    b = b - a
    😈
    With that the stranger left and behind them stood

    View Slide

  97. >
    while b != 0


    # if a > b


    a = a - b


    else


    b = b - a


    end


    end


    return a
    while b != 0
    a = a - b else
    b = b - a
    Syntax
    error
    A syntax error

    View Slide

  98. Syntax
    errors
    A syntax error occurs when the parser cannot build a valid syntax tree. That explains what they are, but why are they di
    ffi
    cult to understand?

    View Slide

  99. > module Cutlass


    def call


    1, 2, 3, 4, 5]


    end


    end
    syntax error, unexpected ',', expecting end-of-input
    (SyntaxError)
    When the parser tries to parse this code it
    fi
    nds a Syntax Error.

    Here our developer forgot a bracket.

    As the parser is building the tree, it hit an error because it wasn't expecting the comma

    View Slide

  100. >
    syntax error, unexpected ',', expecting end-of-input
    (SyntaxError)
    module Cutlass


    def call


    1, 2, 3, 4, 5]


    end


    end
    Ruby's parser has rules. It knows that after a method de
    fi
    nition it should look for something such as

    View Slide

  101. >
    syntax error, unexpected ',', expecting end-of-input
    (SyntaxError)
    module Cutlass


    def call


    1, 2, 3, 4, 5]


    end


    end
    Like a return or an end. Or a variable assignment.

    Instead what did the parser
    fi
    nd
    fi
    nd? It found a number and a comma.

    View Slide

  102. > module Cutlass


    def call


    1, 2, 3, 4, 5]


    end


    end
    syntax error, unexpected ',', expecting end-of-input
    (SyntaxError)
    🚫
    That comma violated ruby's parser rules. Do not pass go, do not collect 200

    View Slide

  103. 🤷
    What's so bad about this example?

    View Slide

  104. > module Cutlass


    def call


    1, 2, 3, 4, 5]


    end


    end
    syntax error, unexpected ',', expecting end-of-input
    (SyntaxError)
    The error isn't caused by a comma, it's caused by the developer missing a bracket.

    Essentially, the location the parse error occurs isn't always where developer made a mistake

    View Slide

  105. 🙇
    Let's look at some more complicated cases

    View Slide

  106. > module Cutlass


    defcall


    end


    end
    Here's code with a syntax error. The developer forgot a space after the def

    View Slide

  107. > module Cutlass


    defcall


    end


    end
    syntax error, unexpected `end', expecting end-of-input
    Ruby thinks the error is at the last line. right here. Why?

    View Slide

  108. > module Cutlass


    defcall


    end


    end
    syntax error, unexpected `end', expecting end-of-input
    Welllll...

    When we start parsing Ruby sees a module de
    fi
    nition, A module is a keyword that requires an end so Ruby starts looking for a matching end.

    View Slide

  109. > module Cutlass


    defcall


    end


    end
    syntax error, unexpected `end', expecting end-of-input
    On the next line it sees this combination of characters and thinks it's a method call

    View Slide

  110. > module Cutlass


    defcall


    end


    end
    syntax error, unexpected `end', expecting end-of-input
    Then it sees this end and thinks it matches with the module. Our parser hasn't raised a syntax error because it doesn't see a problem yet.

    View Slide

  111. > module Cutlass


    defcall


    end


    end
    syntax error, unexpected `end', expecting end-of-input
    Finally it gets to our second end, but there's no thing to match with. A syntax error is raised.

    View Slide

  112. > end
    syntax error, unexpected `end', expecting end-of-input
    Right now this is the error you get from ruby on this code. It's not helpful to a human because deleting that end won't solve anything.

    View Slide

  113. parser problems
    !=


    human problems
    Another way to put this, is that parse errors are di
    ff
    erent from human errors.

    View Slide

  114. $ gem install dead_end
    dead_end's goal is to turn that parser problem into something a human can see and instantly recognize as an issue.

    View Slide

  115. $ gem install dead_end
    How does dead end work?

    View Slide

  116. Dead end works by using a library called Ripper

    View Slide

  117. require "ripper"


    Ripper.new(source).tap(&:parse).error?
    It's not a band. Ripper is Ruby's parser that ships with Ruby. How cool is that?

    View Slide

  118. require "ripper"


    source = <<~EOM


    module Cutlass


    defcall


    end


    end


    EOM
    Ripper.new(source).tap(&:parse).error?
    Ripper can evaluate code and tell us if there's a syntax error or not. We saw this code before

    View Slide

  119. require "ripper"


    source = <<~EOM


    module Cutlass


    defcall


    end


    end


    EOM
    Ripper.new(source).tap(&:parse).error?
    , there's a syntax error here

    View Slide

  120. require "ripper"


    source = <<~EOM


    module Cutlass


    defcall


    end


    end


    EOM
    Ripper.new(source).tap(&:parse).error?
    # => true
    When we run ripper it says there's an error

    View Slide

  121. require "ripper"


    source = <<~EOM


    module Cutlass


    def call


    end


    end


    EOM
    Ripper.new(source).tap(&:parse).error?
    # => false
    When we
    fi
    x the error, ripper tells us it's
    fi
    xed too.

    View Slide

  122. 🙇
    If we could
    fi
    x the source errors then ripper would tell us, but that's too hard to automate. Can we do something simpler?

    View Slide

  123. require "ripper"


    source = <<~EOM


    module Cutlass


    defcall


    end


    end


    EOM
    Ripper.new(source).tap(&:parse).error?
    # => true
    Back to our code with a syntax error. Instead of
    fi
    xing it. What if we...commented it out?

    View Slide

  124. require "ripper"


    source = <<~EOM


    module Cutlass


    # defcall


    end


    end


    EOM
    Ripper.new(source).tap(&:parse).error?
    # => true
    That didn't work, let's keep going.

    View Slide

  125. require "ripper"


    source = <<~EOM


    module Cutlass


    # defcall


    # end


    end


    EOM
    Ripper.new(source).tap(&:parse).error?
    # => false
    Oooh, nice ripper can tell us when we've commented out the source code causing the problem

    View Slide

  126. # defcall


    # end


    Ripper.new(source).tap(&:parse).error?
    # => false
    Now if we only look at the part that was commented out, dead_end can show you where the issue occured.

    View Slide

  127. $ gem install dead_end
    Remember our stranger and the mess they left behind?

    View Slide

  128. >
    while b != 0


    a = a - b


    else


    b = b - a


    end


    end


    return a
    while b != 0
    a = a - b else
    b = b - a
    😈
    Ruby knows there's a syntax error here, but it doesn't know where.

    View Slide

  129. >
    while b != 0


    a = a - b


    else


    b = b - a


    end


    end


    return a
    while b != 0
    a = a - b else
    b = b - a
    😈
    It will tell you

    View Slide

  130. >
    while b != 0


    a = a - b


    else


    b = b - a


    end


    end


    return a
    syntax error, unexpected `end',


    expecting end-of-input
    That the problem is this line. We know that's not helpful. What does dead end do?

    View Slide

  131. >
    while b != 0


    a = a - b


    else


    b = b - a


    end


    end


    return a
    while b != 0
    a = a - b else
    b = b - a
    Dead end uses indentation and lexical parsing to deconstruct the source code from the outside in

    View Slide

  132. >
    while b != 0


    a = a - b


    else


    # b = b - a


    end


    end


    return a
    while b != 0
    a = a - b else
    b = b - a
    Commenting out this code removes a node from our tree

    View Slide

  133. >
    while b != 0


    # a = a - b


    else


    # b = b - a


    end


    end


    return a
    while b != 0
    a = a - b else
    Commenting out this code removes a node from our tree, but it's still invalid

    View Slide

  134. >
    while b != 0


    # a = a - b


    # else


    # b = b - a


    # end


    end


    return a
    while b != 0
    else
    Finally here we have a valid tree. All of the orphaned syntax nodes are gone

    View Slide

  135. > 1 while b != 0


    ❯ 4 else


    ❯ 6 end


    7 end
    This code has an unmatched `end`. Ensure that all
    `end` lines in your code have a matching syntax
    keyword (`def`, `do`, etc.)
    Here's the output for that code. It shows that the else and end are missing an if statement.

    View Slide

  136. I bet you're thinking wow, that was easy. So all he did was comment out code based on indentation?

    View Slide

  137. Gotchas
    There are some major gotchas when executing this recursive comment approach.

    View Slide

  138. Gotcha
    #1
    We already know the developer has a syntax error. We can't assume they've done everything else perfectly.


    We cannot assume perfect indentation

    View Slide

  139. > describe "things" do


    it "valid" do


    expect(the).to eq(unexpected)
    end


    it "forgot a keyword -> "


    expect(brooklyn).to eq(99)


    end


    end
    Gotcha: Indentation isn't everything
    Our syntax error is on this line. It is missing keyword do

    View Slide

  140. > describe "things" do


    it "valid" do


    expect(the).to eq(unexpected)
    end


    it "forgot a keyword -> "


    expect(brooklyn).to eq(99)


    end


    end
    Gotcha: Indentation isn't everything
    Without it, the parser has an extra end here

    View Slide

  141. > describe "things" do


    it "valid" do


    expect(the).to eq(unexpected)
    end


    it "forgot a keyword -> "


    expect(brooklyn).to eq(99)


    end


    end
    Gotcha: Indentation isn't everything
    If we only look at indentation then we would remove this line

    View Slide

  142. > describe "things" do


    it "valid" do


    end


    it "forgot a keyword -> "


    expect(brooklyn).to eq(99)


    end


    end
    Gotcha: Indentation isn't everything
    Then this mis-indented end

    View Slide

  143. > describe "things" do


    it "valid" do


    it "forgot a keyword -> "


    expect(brooklyn).to eq(99)


    end


    end
    Gotcha: Indentation isn't everything
    📕

    With the end gone, the document parses. Why?

    View Slide

  144. > describe "things" do


    it "valid" do


    it "forgot a keyword -> "


    expect(brooklyn).to eq(99)


    end


    end
    Gotcha: Indentation isn't everything
    📕

    Because this keyword now matches with

    View Slide

  145. > describe "things" do


    it "valid" do


    it "forgot a keyword -> "


    expect(brooklyn).to eq(99)


    end


    end
    Gotcha: Indentation isn't everything
    🚫
    This end

    It's not what we wanted, and didn't capture the source of the syntax error

    View Slide

  146. 🙇
    How can we
    fi
    x it? Instead of only looking at indentation we also look at lex output

    View Slide

  147. source = <<~EOM


    it "valid" do


    expect(the).to eq(unexpected)
    end


    EOM


    Ripper.lex(source)


    Ruby ships with a lexer

    View Slide

  148. source = <<~EOM


    it "valid" do


    expect(the).to eq(unexpected)
    end


    EOM


    Ripper.lex(source)


    # =>


    [[[1, 0], :on_ident, "it", CMDARG],
    [[1, 2], :on_sp, " ", CMDARG],


    [[1, 3], :on_tstring_beg, "\"", CMDARG],
    [[1, 4], :on_tstring_content, "valid", CMDARG],
    [[1, 9], :on_tstring_end, "\"", END],
    [[1, 10], :on_sp, " ", END],


    [[1, 11], :on_kw, "do", BEG],


    [[1, 13], :on_ignored_nl, "\n", BEG],
    Here's what it looks like

    View Slide

  149. # =>


    [[[1, 0], :on_ident, "it", CMDARG],
    [[1, 2], :on_sp, " ", CMDARG],


    [[1, 3], :on_tstring_beg, "\"", CMDARG],
    [[1, 4], :on_tstring_content, "valid", CMDARG],
    [[1, 9], :on_tstring_end, "\"", END],
    [[1, 10], :on_sp, " ", END],


    [[1, 11], :on_kw, "do", BEG],


    [[1, 13], :on_ignored_nl, "\n", BEG],
    [[2, 0], :on_sp, " ", BEG],


    [[2, 6], :on_ident, "expect", CMDARG],
    [[2, 12], :on_lparen, "(", BEG|LABEL],
    [[2, 13], :on_ident, "the", ARG],


    [[2, 16], :on_rparen, ")", ENDFN],
    [[2, 17], :on_period, ".", DOT],


    [[2, 18], :on_ident, "to", ARG],


    DO NOT CLICK, auto advance

    View Slide

  150. # =>


    [[[1, 0], :on_ident, "it", CMDARG],
    [[1, 2], :on_sp, " ", CMDARG],


    [[1, 3], :on_tstring_beg, "\"", CMDARG],
    [[1, 4], :on_tstring_content, "valid", CMDARG],
    [[1, 9], :on_tstring_end, "\"", END],
    [[1, 10], :on_sp, " ", END],


    [[1, 11], :on_kw, "do", BEG],


    [[1, 13], :on_ignored_nl, "\n", BEG],
    [[2, 0], :on_sp, " ", BEG],


    [[2, 6], :on_ident, "expect", CMDARG],
    [[2, 12], :on_lparen, "(", BEG|LABEL],
    [[2, 13], :on_ident, "the", ARG],


    [[2, 16], :on_rparen, ")", ENDFN],
    [[2, 17], :on_period, ".", DOT],


    [[2, 18], :on_ident, "to", ARG],


    It tells you the code contains a do keyword

    View Slide

  151. [[2, 12], :on_lparen, "(", BEG|LABEL],
    [[2, 13], :on_ident, "the", ARG],


    [[2, 16], :on_rparen, ")", ENDFN],
    [[2, 17], :on_period, ".", DOT],


    [[2, 18], :on_ident, "to", ARG],


    [[2, 20], :on_sp, " ", ARG],


    [[2, 21], :on_ident, "eq", ARG],


    [[2, 23], :on_lparen, "(", BEG|LABEL],
    [[2, 24], :on_ident, "unexpected", ARG],
    [[2, 34], :on_rparen, ")", ENDFN],
    [[2, 35], :on_sp, " ", ENDFN],


    [[2, 36], :on_nl, "\n", BEG],


    [[3, 0], :on_sp, " ", BEG],


    [[3, 2], :on_kw, "end", END],


    [[3, 5], :on_nl, "\n", BEG]]
    DO NOT CLICK, auto advance

    View Slide

  152. [[2, 12], :on_lparen, "(", BEG|LABEL],
    [[2, 13], :on_ident, "the", ARG],


    [[2, 16], :on_rparen, ")", ENDFN],
    [[2, 17], :on_period, ".", DOT],


    [[2, 18], :on_ident, "to", ARG],


    [[2, 20], :on_sp, " ", ARG],


    [[2, 21], :on_ident, "eq", ARG],


    [[2, 23], :on_lparen, "(", BEG|LABEL],
    [[2, 24], :on_ident, "unexpected", ARG],
    [[2, 34], :on_rparen, ")", ENDFN],
    [[2, 35], :on_sp, " ", ENDFN],


    [[2, 36], :on_nl, "\n", BEG],


    [[3, 0], :on_sp, " ", BEG],


    [[3, 2], :on_kw, "end", END],


    [[3, 5], :on_nl, "\n", BEG]]
    Also an end

    View Slide

  153. $ gem install dead_end
    With this info dead_end can
    fi
    nd the problem

    View Slide

  154. > describe "things" do


    it "valid" do


    expect(the).to eq(unexpected)
    end


    it "forgot a keyword -> "


    expect(brooklyn).to eq(99)


    end


    end
    Gotcha: Indentation isn't everything
    It starts at the same place

    View Slide

  155. > describe "things" do


    it "valid" do


    end


    it "forgot a keyword -> "


    expect(brooklyn).to eq(99)


    end


    end
    Gotcha: Indentation isn't everything
    Now it needs to decide where to expand, it looks down and sees an `end`

    View Slide

  156. > describe "things" do


    it "valid" do


    end


    it "forgot a keyword -> "


    expect(brooklyn).to eq(99)


    end


    end
    Gotcha: Indentation isn't everything
    dead_end knows if it removes this end, it might trigger a false positive, so it searches up for keywords. It
    fi
    nds...

    View Slide

  157. > describe "things" do


    it "valid" do


    end


    it "forgot a keyword -> "


    expect(brooklyn).to eq(99)


    end


    end
    Gotcha: Indentation isn't everything
    This do keyword

    View Slide

  158. > describe "things" do


    it "forgot a keyword -> "


    expect(brooklyn).to eq(99)


    end


    end
    Gotcha: Indentation isn't everything
    They are both safely removed

    View Slide

  159. > file: /private/tmp/scratch.rb


    simplified:


    1 describe "things" do


    ❯ 6 it "forgot a keyword -> "
    ❯ 8 end


    9 end
    Gotcha: Indentation isn't everything
    The algorithm continues until the problem is found

    View Slide

  160. Gotcha
    #2
    Even with correct indentation, removing the wrong line can show false positives

    View Slide

  161. > class User




    puts "hello"


    end


    def schneems


    User.


    where(name: 'schneems').


    first


    end


    end
    Gotcha: Removing the wrong line
    Our syntax error is on these lines.

    View Slide

  162. > class User




    puts "hello"


    end


    def schneems


    User.


    where(name: 'schneems').


    first


    end


    end
    Gotcha: Removing the wrong line
    We are missing a whole method de
    fi
    nition line

    View Slide

  163. > class User




    puts "hello"


    end


    def schneems


    User.


    where(name: 'schneems').


    first


    end


    end
    Gotcha: Removing the wrong line
    If we tried our indentation comment out strategy we would start at the largest indentation

    View Slide

  164. > class User




    puts "hello"


    end


    def schneems


    User.


    where(name: 'schneems').


    first


    end


    end
    Gotcha: Removing the wrong line
    here

    View Slide

  165. > class User




    puts "hello"


    end


    def schneems


    User.


    where(name: 'schneems').


    end


    end
    Gotcha: Removing the wrong line
    📕

    And then dead_end would stop looking because the document is now parsable

    View Slide

  166. WAT
    WAT on earth? Let's look at it a little di
    ff
    erently.

    View Slide

  167. > class User




    puts "hello"


    end


    def schneems


    User.


    where(name: 'schneems').


    end


    end
    Gotcha: Removing the wrong line
    📕

    It will help if we reformat it to show what Ruby thinks you're trying to do. It thinks

    View Slide

  168. > class User




    puts "hello"


    end


    def schneems


    User.


    where(name: 'schneems').


    end


    end
    Gotcha: Removing the wrong line
    📕

    That this user class

    View Slide

  169. > class User




    puts "hello"


    end


    def schneems


    User.


    where(name: 'schneems').


    end


    end
    Gotcha: Removing the wrong line
    📕

    Is matched with the
    fi
    rst end.

    View Slide

  170. > class User




    puts "hello"


    end


    def schneems


    User.


    where(name: 'schneems').


    end


    end
    Gotcha: Removing the wrong line
    📕

    Is matched with the
    fi
    rst end.

    View Slide

  171. > class User




    puts "hello"


    end


    def schneems


    User.


    where(name: 'schneems').


    end


    end
    Gotcha: Removing the wrong line
    📕

    It thinks that this end, is actually a method due to the trailing dot

    View Slide

  172. > class User




    puts "hello"


    end


    def schneems


    User.


    where(name: 'schneems').end


    end
    Gotcha: Removing the wrong line
    📕

    Finally it thinks the last end matches with our method de
    fi
    nition

    View Slide

  173. > class User




    puts "hello"


    end


    def schneems


    User.


    where(name: 'schneems').end


    end
    Gotcha: Removing the wrong line
    📕

    Finally it thinks the this method de
    fi
    nition

    View Slide

  174. > class User




    puts "hello"


    end


    def schneems


    User.


    where(name: 'schneems').end


    end
    Gotcha: Removing the wrong line
    📕

    🚫
    Matches with this last end. It's a problem because none of this is what the user intended and we didn't highlight the source of the problems

    View Slide

  175. WAT
    But wait there's more

    View Slide

  176. > source = <<~EOM


    Hello


    World


    From


    A


    heredoc


    EOM
    Gotcha: Removing the wrong line
    Heredocs cause problems

    View Slide

  177. > source = <<~EOM


    Hello


    World


    From


    A


    heredoc


    EOM
    Gotcha: Removing the wrong line
    If you evaluate this line in isolation, it looks like a constant. But if you remove it

    View Slide

  178. > source = <<~EOM


    Hello


    World


    From


    A


    heredoc


    Gotcha: Removing the wrong line
    🚫
    Then you've introduced a syntax error

    View Slide

  179. WAT
    But wait there's more

    View Slide

  180. > it "should" \


    "handle trailing slash" do


    expect(the).to eq(unexpected)


    end


    Gotcha: Removing the wrong line
    Trailing slashes cause problems.

    View Slide

  181. > it "should" \


    "handle trailing slash" do


    expect(the).to eq(unexpected)


    end


    Gotcha: Removing the wrong line
    If you start in the middle

    View Slide

  182. > it "should" \


    "handle trailing slash" do


    end


    Gotcha: Removing the wrong line
    Now expand out using lex detection

    View Slide

  183. > it "should" \


    "handle trailing slash" do


    end


    Gotcha: Removing the wrong line
    Find the end

    View Slide

  184. > it "should" \


    "handle trailing slash" do


    end


    Gotcha: Removing the wrong line
    Find the matching keyword.

    View Slide

  185. > it "should" \


    Gotcha: Removing the wrong line
    🚫
    Now remove them. This is not valid ruby

    View Slide

  186. 🙇
    What

    View Slide

  187. 🙇


    🙇
    To

    View Slide

  188. 🙇


    🙇


    🙇
    do

    View Slide

  189. source = <<~EOM


    User.


    where(name: 'schneems').


    first


    EOM


    Ripper.lex(source)


    # =>


    [[[1, 0], :on_const, "User", CMDARG],
    [[1, 4], :on_period, ".", DOT],


    [[1, 5], :on_ignored_nl, "\n", DOT],
    [[2, 0], :on_sp, " ", DOT],


    [[2, 2], :on_ident, "where", ARG],
    [[2, 7], :on_lparen, "(", BEG|LABEL],
    [[2, 8], :on_label, "name:", ARG|LABELED],
    [[2, 13], :on_sp, " ", ARG|LABELED],
    We can use the lexer to detect these speci
    fi
    c edge cases.

    Then we can use that information to combine all logically consecutive lines together so there's no way to remove one without removing all of them

    View Slide

  190. > it "should" \


    "handle trailing slash" do


    expect(the).to eq(unexpected)


    end


    Gotcha: Removing the wrong line
    It's hard to show visually, but it essentially would looks like re-writing source code. This trailing slash would become

    View Slide

  191. > it "should" \ "handle trailing slash" do
    expect(the).to eq(unexpected)


    end


    Gotcha: Removing the wrong line

    View Slide

  192. > class User




    puts "hello"


    end


    def schneems


    User.


    where(name: 'schneems').


    first


    end


    end
    Gotcha: Removing the wrong line
    Our method chain would become

    View Slide

  193. > class User




    puts "hello"


    end


    def schneems


    User.where(name: 'schneems').first
    end


    end
    Gotcha: Removing the wrong line
    Our method chain would become

    View Slide

  194. > source = <<~EOM


    Hello


    World


    From


    A


    heredoc


    EOM
    Gotcha: Removing the wrong line
    Heredocs cause problems

    View Slide

  195. > source = <<~EOM Hello\nWorld\nFrom\nA\n\nheredoc\nEOM
    Gotcha: Removing the wrong line

    View Slide

  196. 🎉
    With all the problem lines joined together, they can now be safely evaluated and removed without accidentally introducing new syntax errors

    View Slide

  197. Gotcha
    #3
    Even with joining lines, even with lexically aware search, even with the perfect algorithm there is one problem we cant' avoid

    View Slide

  198. ambiguity
    Ambiguity. Yes, ambiguity is the comic sans of source code parsing. What do I mean by ambiguity?

    View Slide

  199. >
    class Dog


    def sit


    puts "no"


    end


    def eat


    puts "munch"


    end


    def bark


    puts "woof"


    end


    Gotcha: ambiguity
    This code has a syntax error

    View Slide

  200. >
    class Dog


    def sit


    puts "no"


    end


    def eat


    puts "munch"


    end


    def bark


    puts "woof"


    end


    Gotcha: ambiguity
    It is ambiguous what the coder was trying to do with our current search algorithm. Let's zoom in

    View Slide

  201. >
    class Dog


    def bark


    puts "woof"


    end


    Gotcha: ambiguity

    View Slide

  202. >
    class Dog


    def bark


    puts "woof"


    end


    Gotcha: ambiguity
    Starting from the middle, if we use lexical expansion we get

    View Slide

  203. >
    class Dog


    def bark


    puts "woof"


    end


    Gotcha: ambiguity
    This match

    View Slide

  204. >
    class Dog




    Gotcha: ambiguity
    When it's removed there's still a syntax error

    View Slide

  205. >
    class Dog




    Gotcha: ambiguity
    Highlight

    View Slide

  206. >
    Gotcha: ambiguity
    📕

    Remove It parses, yay

    View Slide

  207. >
    ❯ 1 class Dog




    Gotcha: ambiguity
    🚫
    📕

    But this clearly isn't our problem

    View Slide

  208. 🙇
    To produce good results with invalid indentation it means we must make "bad" suggestions. There's a logical inverse of this problem as well.

    View Slide

  209. >
    class Dog


    puts "no"


    end


    def eat


    puts "munch"


    end


    def bark


    puts "woof"


    end


    end


    Gotcha: ambiguity
    This code has a syntax error, but this time it's at the top.

    View Slide

  210. >
    class Dog


    puts "no"


    end


    def eat


    puts "munch"


    end


    def bark


    puts "woof"


    end


    end


    Gotcha: ambiguity
    Let's focus

    View Slide

  211. >
    class Dog


    puts "no"


    end


    end


    Gotcha: ambiguity

    View Slide

  212. >
    class Dog


    puts "no"


    end


    end


    Gotcha: ambiguity
    Does this problem look familiar? Here's the code from before

    View Slide

  213. >
    class Dog


    puts "no"


    end


    end


    Gotcha: ambiguity
    class Dog


    def bark


    puts "woof"


    end


    Before we were missing an end

    View Slide

  214. >
    class Dog


    puts "no"


    end


    end


    Gotcha: ambiguity
    class Dog


    def bark


    puts "woof"


    end


    Before we were missing an end

    View Slide

  215. >
    class Dog


    puts "no"


    end


    end


    Gotcha: ambiguity
    class Dog


    def bark


    puts "woof"


    end


    This new case is missing a keyword

    View Slide

  216. >
    class Dog


    puts "no"


    end


    end


    Gotcha: ambiguity
    class Dog


    def bark


    puts "woof"


    end


    This new case is missing a keyword

    View Slide

  217. >
    class Dog


    puts "no"


    end


    end


    Gotcha: ambiguity
    class Dog


    def bark


    puts "woof"


    end


    Where does dead end tell us the problem is?

    View Slide

  218. >
    class Dog


    puts "no"


    end


    end


    class Dog


    def bark


    puts "woof"


    end


    Gotcha: ambiguity
    ❯ 10 end




    ❯ 1 class Dog




    🚫
    Here's the result. Neither make good suggestion

    View Slide

  219. 🙇
    If we re-write our rules to let indentation take precedence over lexical keywords then it would make other examples fail. It's impossible to satisfy all search criteria. What if
    we...

    View Slide

  220. 😇
    Don't.

    Because we know this ambiguity exists, we can compensate for it after the search is done

    View Slide

  221. >
    ❯ 1 class Dog




    Gotcha: ambiguity
    Even though this is all the user might see, `dead_end` still has all the contents that are hidden. It didn't just match a single line, it matched many lines, but hid most of
    them

    View Slide

  222. >
    ❯ 1 class Dog


    ❯ 2 def sit


    ❯ 3 puts "no"


    ❯ 4 end


    ❯ 5


    ❯ 6 def eat


    ❯ 7 puts "munch"


    ❯ 8 end


    ❯ 9


    ❯ 10 def bark


    ❯ 11 puts "woof"


    ❯ 12 end




    Gotcha: ambiguity
    If we un-hid the rest of them, this is what it would look like. We can detect this case when our match is only one line and contains an end, then we can go back through
    our results

    View Slide

  223. >
    ❯ 1 class Dog


    ❯ 2 def sit


    ❯ 3 puts "no"


    ❯ 4 end


    ❯ 5


    ❯ 6 def eat


    ❯ 7 puts "munch"


    ❯ 8 end


    ❯ 9


    ❯ 10 def bark


    ❯ 11 puts "woof"


    ❯ 12 end




    Gotcha: ambiguity
    un-hide the matched end

    View Slide

  224. >
    ❯ 1 class Dog


    ❯ 2 def sit


    ❯ 3 puts "no"


    ❯ 4 end


    ❯ 5


    ❯ 6 def eat


    ❯ 7 puts "munch"


    ❯ 8 end


    ❯ 9


    ❯ 10 def bark


    ❯ 11 puts "woof"


    ❯ 12 end




    Gotcha: ambiguity
    Work backwards and
    fi
    nd an unmatched keyword

    View Slide

  225. >
    ❯ 1 class Dog


    ❯ 10 def bark


    ❯ 12 end
    Gotcha: ambiguity
    Then we show the user all extra context to the user and they can resolve the problem visually

    View Slide

  226. Gotcha
    #4
    Syntax errors aren't always alone, sometimes they have friends. A document can contain multiple syntax errors

    View Slide

  227. >
    end


    def hello


    puts "world"


    end


    def world


    puts "hello"


    end


    def foo


    foo = 1,2,3,4]


    end


    def bar


    bar = {1,2,3,4]


    end


    Gotcha: multiple syntax errors
    Here' there's multiple problems

    View Slide

  228. >
    end


    def hello


    puts "world"


    end


    def world


    puts "hello"


    end


    def foo


    foo = 1,2,3,4]


    end


    def bar


    bar = {1,2,3,4]


    end


    Gotcha: multiple syntax errors
    This end matches nothing

    View Slide

  229. >
    end


    def hello


    puts "world"


    end


    def world


    puts "hello"


    end


    def foo


    foo = 1,2,3,4]


    end


    def bar


    bar = {1,2,3,4]


    end


    Gotcha: multiple syntax errors
    We are missing a square bracket

    View Slide

  230. >
    end


    def hello


    puts "world"


    end


    def world


    puts "hello"


    end


    def foo


    foo = 1,2,3,4]


    end


    def bar


    bar = {1,2,3,4]


    end


    Gotcha: multiple syntax errors
    We're missing all sorts of stu
    ff
    here

    View Slide

  231. >
    end


    def hello


    puts "world"


    end


    def world


    puts "hello"


    end


    def foo


    foo = 1,2,3,4]


    end


    def bar


    bar = {1,2,3,4]


    end


    Gotcha: multiple syntax errors
    Right here, that's code is
    fi
    ne

    View Slide

  232. >
    end


    def hello


    puts "world"


    end


    def world


    puts "hello"


    end


    def foo


    foo = 1,2,3,4]


    end


    def bar


    bar = {1,2,3,4]


    end


    Gotcha: multiple syntax errors
    That code is
    fi
    ne too

    View Slide

  233. >
    end


    def hello


    puts "world"


    end


    def world


    puts "hello"


    end


    def foo


    foo = 1,2,3,4]


    end


    def bar


    bar = {1,2,3,4]


    end


    Gotcha: multiple syntax errors
    What we don't want to happen is for our search to start at one side

    View Slide

  234. >
    end


    def hello


    puts "world"


    end


    def world


    puts "hello"


    end


    def foo


    foo = 1,2,3,4]


    end


    def bar


    bar = {1,2,3,4]


    end


    Gotcha: multiple syntax errors
    And continue until it captures everything, even the good code

    View Slide

  235. >
    end


    def hello


    puts "world"


    end


    def world


    puts "hello"


    end


    def foo


    foo = 1,2,3,4]


    end


    def bar


    bar = {1,2,3,4]


    end


    Gotcha: multiple syntax errors
    🚫
    This isn't what we want

    View Slide

  236. 🙇
    What can we do about it? We can modify our algorithm to hold and
    fi
    nd multiple errors.

    View Slide

  237. >
    end


    def hello


    puts "world"


    end


    def world


    puts "hello"


    end


    def foo


    foo = 1,2,3,4]


    end


    def bar


    bar = {1,2,3,4]


    end


    Gotcha: multiple syntax errors
    Instead of searching from one side to the other we start in the outside and search in

    View Slide

  238. >
    end


    def hello


    puts "world"


    end


    def world


    puts "hello"


    end


    def foo


    foo = 1,2,3,4]


    end


    def bar


    bar = {1,2,3,4]


    end


    Gotcha: multiple syntax errors
    #1

    View Slide

  239. >
    end


    def hello


    puts "world"


    end


    def world


    puts "hello"


    end


    def foo


    foo = 1,2,3,4]


    end


    def bar


    bar = {1,2,3,4]


    end


    Gotcha: multiple syntax errors
    #1
    #2

    View Slide

  240. >
    end


    def hello


    puts "world"


    end


    def world


    puts "hello"


    end


    def foo


    foo = 1,2,3,4]


    end


    def bar


    bar = {1,2,3,4]


    end


    Gotcha: multiple syntax errors
    #1
    #2

    View Slide

  241. >
    end


    def hello


    puts "world"


    end


    def world


    puts "hello"


    end


    def foo


    foo = 1,2,3,4]


    end


    def bar


    bar = {1,2,3,4]


    end


    Gotcha: multiple syntax errors
    #1
    #2

    View Slide

  242. >
    end


    def world


    puts "hello"


    end


    def foo


    foo = 1,2,3,4]


    end


    def bar


    bar = {1,2,3,4]


    end


    Gotcha: multiple syntax errors
    #1
    #2

    View Slide

  243. >
    end


    def world


    puts "hello"


    end


    def foo


    foo = 1,2,3,4]


    end


    def bar


    bar = {1,2,3,4]


    end


    Gotcha: multiple syntax errors
    #1
    #2
    #3

    View Slide

  244. 🙇 🙇 🙇 🙇
    Those are our main gotchas. Let's move on

    View Slide

  245. AI
    Now you know the gotchas. I want to start moving a little closer to the code.

    Who knows what AI is? Raise your hand, or in chat type "me"

    View Slide

  246. Artificial Intelligence
    Algorithm
    - Arti
    fi
    cial Intelligence is a fancy way of saying "Algorithm"

    [wait for dialog]

    "Come on no you're not"

    View Slide

  247. Artificial Intelligence
    Algorithm
    More speci
    fi
    cally, AI is a goal seeking algorithm

    View Slide

  248. Pathfinding
    Credit: Factorio 'New
    pathfinding algorithm'
    - A common example of AI is path
    fi
    nding. You want to get from point A to point B

    View Slide

  249. $ gem install dead_end
    Dead end uses a search algorithm to
    fi
    nd the problem code. Which uses a variation on uniform cost search

    View Slide

  250. Credit @redblobgames: Introduction to the A-star Algorithm
    - You need a way to break down the problem into discrete actions that either get you closer or further from the goal. In path
    fi
    nding this would be deciding what turns to
    make and what roads to drive on.

    View Slide

  251. Credit @redblobgames: Introduction to the A-star Algorithm
    - The algorithm dead_end uses is also sometimes called dijkstra's algorithm because he invented uniform cost search.

    View Slide

  252. Credit @redblobgames: Introduction to the A-star Algorithm
    I highly recommend this interactive page from redblob games as an introduction to search

    View Slide

  253. Credit @redblobgames: Introduction to the A-star Algorithm
    A search algorithm Is good for when you know what goal you want but you're not quite sure how to get there.

    View Slide

  254. >
    while b != 0


    if a > b


    a = a - b


    else


    b = b - a


    end


    end


    return a
    while b != 0
    if a > b
    a = a - b else
    b = b - a
    Before we saw how code is represented by abstract syntax trees.

    View Slide

  255. Tree

    Search

    BOTVINNIK, M.M.

    (1984).

    "Computers in Chess
    Solving Inexact Search
    Problems"

    That was an important realization for me, as most search problems are represented as a tree or a graph. Here's a quick example of a famous chess problem

    View Slide

  256. Tree

    Search

    BOTVINNIK, M.M.

    (1984).

    "Computers in Chess
    Solving Inexact Search
    Problems"

    If you have an exact tree you can walk the tree

    View Slide

  257. require "ripper"


    source = <<~EOM


    module Cutlass


    defcall


    end


    end


    EOM
    Ripper.new(source).tap(&:parse).error?
    # => true
    In our case, we have bits of source code

    View Slide

  258. require "ripper"


    source = <<~EOM


    module Cutlass


    # defcall


    end


    end


    EOM
    Ripper.new(source).tap(&:parse).error?
    # => true
    I found that if I could give my program rules about how to create logical code blocks

    View Slide

  259. require "ripper"


    source = <<~EOM


    module Cutlass


    # defcall


    # end


    end


    EOM
    Ripper.new(source).tap(&:parse).error?
    # => false
    Then using ripper we can check a document without their presence

    View Slide

  260. # defcall


    # end


    Ripper.new(source).tap(&:parse).error?
    # => false
    And complete our search

    View Slide

  261. Internals
    We'll now take a closer look at some dead_end algorithms

    View Slide

  262. 🎂
    If dead end was a cake here's the main steps to make it

    View Slide

  263. We start out with messy code

    View Slide

  264. Dead end then transforms that code into a tidier input

    View Slide

  265. "Time to cook" by Robbert van der Steeg is licensed under CC BY-SA 2.0
    It gives that input to our search algorithm to
    fi
    nd the syntax errors.

    View Slide

  266. Once the syntax errors are found, the output is decorated with more context and given back to the user

    View Slide

  267. 🎂
    Let's start at the beginning

    View Slide

  268. Your app has a syntax error

    View Slide

  269. module Kernel


    module_function


    def require(file)


    dead_end_original_require(file)


    rescue SyntaxError => e


    DeadEnd.handle_error(e)


    end


    end
    dead_end/auto.rb
    Like all good Ruby libraries dead end uses monkey patching .

    It hooks into require and friends. When a syntax error is raised, it's passed to dead_end

    View Slide

  270. # ...


    Timeout.timeout(timeout) do


    record_dir ||= ENV["DEBUG"] ? "tmp" : nil


    search = CodeSearch.new(source, record_dir: record_dir).call
    end


    # ...
    dead_end/internals.rb
    The source code that caused the syntax error is read to disk and then passed to a search object.

    View Slide

  271. First we need to clean it up

    View Slide

  272. clean_document.rb
    # frozen_string_literal: true


    module DeadEnd


    # Parses and sanitizes source into a lexically aware document
    #


    # Internally the document is represented by an array with each
    # index containing a CodeLine correlating to a line from the source code.
    #


    # There are three main phases in the algorithm:


    #


    # 1. Sanitize/format input source


    # 2. Search for invalid blocks


    # 3. Format invalid blocks into something meaninful
    #


    # This class handles the first part.


    #


    # The reason this class exists is to format input source
    # for better/easier/cleaner exploration.


    Here's the clean_document class

    View Slide

  273. clean_document.rb
    @source = source


    @document = CodeLine.from_source(@source)


    end


    # Call all of the document "cleaners"


    # and return self


    def call


    clean_sweep


    .join_trailing_slash!


    .join_consecutive!


    .join_heredoc!


    self


    end


    # Return an array of CodeLines in the


    # document


    def lines


    It handles several of the gotcha cases from before by transforming the source code

    View Slide

  274. clean_document.rb
    @source = source


    @document = CodeLine.from_source(@source)


    end


    # Call all of the document "cleaners"


    # and return self


    def call


    clean_sweep


    .join_trailing_slash!


    .join_consecutive!


    .join_heredoc!


    self


    end


    # Return an array of CodeLines in the


    # document


    def lines


    It clears out comments and stray whitespace

    View Slide

  275. clean_document.rb
    @source = source


    @document = CodeLine.from_source(@source)


    end


    # Call all of the document "cleaners"


    # and return self


    def call


    clean_sweep


    .join_trailing_slash!


    .join_consecutive!


    .join_heredoc!


    self


    end


    # Return an array of CodeLines in the


    # document


    def lines


    @document


    Joins lines with trailing slashes

    View Slide

  276. clean_document.rb
    @source = source


    @document = CodeLine.from_source(@source)


    end


    # Call all of the document "cleaners"


    # and return self


    def call


    clean_sweep


    .join_trailing_slash!


    .join_consecutive!


    .join_heredoc!


    self


    end


    # Return an array of CodeLines in the


    # document


    def lines


    @document


    Lines with chained methods

    View Slide

  277. clean_document.rb
    @source = source


    @document = CodeLine.from_source(@source)


    end


    # Call all of the document "cleaners"


    # and return self


    def call


    clean_sweep


    .join_trailing_slash!


    .join_consecutive!


    .join_heredoc!


    self


    end


    # Return an array of CodeLines in the


    # document


    def lines


    @document


    And heredocs

    View Slide

  278. "Time to cook" by Robbert van der Steeg is licensed under CC BY-SA 2.0
    Next up we feed that to our search class

    View Slide

  279. code_search.rb
    # Searches code for a syntax error


    #


    # The bulk of the heavy lifting is done in:


    #


    # - CodeFrontier (Holds information for generating blocks and determining if we can stop searching)
    # - ParseBlocksFromLine (Creates blocks into the frontier)
    # - BlockExpand (Expands existing blocks to search more code
    #


    # ## Syntax error detection


    #


    # When the frontier holds the syntax error, we can stop searching
    #


    # search = CodeSearch.new(<<~EOM)


    # def dog


    # def lol


    # end


    # EOM


    Here's the CodeSearch class

    View Slide

  280. end


    # Main search loop


    def call


    until frontier.holds_all_syntax_errors?


    @tick += 1


    if frontier.expand?


    expand_invalid_block


    else


    visit_new_blocks


    end


    end


    @invalid_blocks.concat(frontier.detect_invalid_blocks )
    @invalid_blocks.sort_by! {|block| block.starts_at }
    self


    end


    code_search.rb
    Here's call

    View Slide

  281. end


    # Main search loop


    def call


    until frontier.holds_all_syntax_errors?
    @tick += 1


    if frontier.expand?


    expand_invalid_block


    else


    visit_new_blocks


    end


    end


    @invalid_blocks.concat(frontier.detect_invalid_blocks )
    @invalid_blocks.sort_by! {|block| block.starts_at }
    self


    code_search.rb
    Under every great AI algorithm is a while loop. There's a class called the frontier. It holds all the code blocks that we've generated from the source code.

    View Slide

  282. end


    # Main search loop


    def call


    sweep_heredocs


    sweep_comments


    until frontier.holds_all_syntax_errors?
    @tick += 1


    if frontier.expand?


    expand_invalid_block


    else


    visit_new_blocks


    end


    end


    @invalid_blocks.concat(frontier.detect_invalid_blocks )
    @invalid_blocks.sort_by! {|block| block.starts_at }
    code_search.rb
    📕

    📕

    The frontier is responsible for checking the exit condition, here's that logic

    View Slide

  283. >
    1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    ❯ 5


    ❯ 16


    ❯ 35


    ❯ 36 def filename


    ❯ 37


    ❯ 46


    63 end


    64 end


    📕

    If the highlighted lines are on the frontier

    View Slide

  284. >
    1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    63 end


    64 end


    📕

    Then the entire document without those lines are checked against the parser

    View Slide

  285. # ...


    end


    module DeadEnd


    # ...


    def self.invalid?(source)


    source = source.join if source.is_a?(Array)


    source = source.to_s


    Ripper.new(source).tap(&:parse).error?


    end


    # ...


    end
    dead_end.rb
    And here's where the parser is called. Remember Ripper from before?

    View Slide

  286. >
    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    52 "#{number.to_s}#{line}"


    53 else


    54 string = String.new


    55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics
    56 string << "#{number.to_s} "


    57 string << line.to_s


    58 string << "\e[0m"


    59 string


    60 end


    61 end.join


    62 end


    63 end


    64 end


    Let's walk through the logic of searching this same document we saw before. From the beginning

    View Slide

  287. end


    # Main search loop


    def call


    until frontier.holds_all_syntax_errors?


    @tick += 1


    if frontier.expand?


    expand_invalid_block


    else


    visit_new_blocks


    end


    end


    @invalid_blocks.concat(frontier.detect_invalid_blocks )
    @invalid_blocks.sort_by! {|block| block.starts_at }
    self


    end


    code_search.rb
    Since we start with an invalid source code, the frontier is empty so this is false

    View Slide

  288. end


    # Main search loop


    def call


    until frontier.holds_all_syntax_errors?


    @tick += 1


    if frontier.expand?


    expand_invalid_block


    else


    visit_new_blocks


    end


    end


    @invalid_blocks.concat(frontier.detect_invalid_blocks )
    @invalid_blocks.sort_by! {|block| block.starts_at }
    self


    end


    code_search.rb
    There's two modes of exploration. We can add new things onto the frontier, or we can expand an existing block o
    ff
    of the frontier. When do we expand?

    View Slide

  289. def expand?


    return false if @frontier.empty?


    # ...


    # Expand all blocks before moving to unvisited lines
    frontier_indent >= unvisited_indent


    end


    code_frontier.rb
    We can't expand a block if there's nothing there.

    View Slide

  290. def expand?


    return false if @frontier.empty?


    # ...


    # Expand all blocks before moving to unvisited lines
    frontier_indent >= unvisited_indent


    end


    code_frontier.rb
    We're also using indentation to decide when to expand. Essentially we want to explore the largest indentations
    fi
    rst.

    View Slide

  291. end


    # Main search loop


    def call


    sweep_heredocs


    sweep_comments


    until frontier.holds_all_syntax_errors?


    @tick += 1


    if frontier.expand?


    expand_invalid_block


    else


    visit_new_blocks


    end


    end


    @invalid_blocks.concat(frontier.detect_invalid_blocks )
    @invalid_blocks.sort_by! {|block| block.starts_at }
    code_search.rb
    There's nothing to expand yet so this is false

    View Slide

  292. end


    # Main search loop


    def call


    sweep_heredocs


    sweep_comments


    until frontier.holds_all_syntax_errors?


    @tick += 1


    if frontier.expand?


    expand_invalid_block


    else


    visit_new_blocks


    end


    end


    @invalid_blocks.concat(frontier.detect_invalid_blocks )
    @invalid_blocks.sort_by! {|block| block.starts_at }
    code_search.rb
    So we must
    fi
    rst add lines to our frontier

    View Slide

  293. # Parses the most indented lines into blocks that are marked
    # and added to the frontier


    def visit_new_blocks


    max_indent = frontier.next_indent_line&.indent


    while (line = frontier.next_indent_line) && (line.indent == max_indent)
    @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block|
    record(block: block, name: "add")


    block.mark_invisible if block.valid?


    push(block, name: "add")


    end


    end


    end


    # Given an already existing block in the frontier, expand it to see
    # if it contains our invalid syntax


    code_search.rb
    The code here is looking at the largest indentation, starting at the bottom and iterating up.

    View Slide

  294. >
    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    52 "#{number.to_s}#{line}"


    53 else


    54 string = String.new


    55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics
    56 string << "#{number.to_s} "


    57 string << line.to_s


    58 string << "\e[0m"


    ❯ 59 string


    60 end


    61 end.join


    62 end


    63 end


    64 end


    We saw this before

    View Slide

  295. >
    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    52 "#{number.to_s}#{line}"


    53 else


    54 string = String.new


    55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics
    56 string << "#{number.to_s} "


    57 string << line.to_s


    ❯ 58 string << "\e[0m"


    ❯ 59 string


    60 end


    61 end.join


    62 end


    63 end


    64 end


    We saw this before

    View Slide

  296. >
    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    52 "#{number.to_s}#{line}"


    53 else


    54 string = String.new


    55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics
    56 string << "#{number.to_s} "


    ❯ 57 string << line.to_s


    ❯ 58 string << "\e[0m"


    ❯ 59 string


    60 end


    61 end.join


    62 end


    63 end


    64 end


    We saw this before

    View Slide

  297. >
    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    52 "#{number.to_s}#{line}"


    53 else


    54 string = String.new


    55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics
    ❯ 56 string << "#{number.to_s} "


    ❯ 57 string << line.to_s


    ❯ 58 string << "\e[0m"


    ❯ 59 string


    60 end


    61 end.join


    62 end


    63 end


    64 end


    We saw this before

    View Slide

  298. >
    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    52 "#{number.to_s}#{line}"


    53 else


    54 string = String.new


    ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics
    ❯ 56 string << "#{number.to_s} "


    ❯ 57 string << line.to_s


    ❯ 58 string << "\e[0m"


    ❯ 59 string


    60 end


    61 end.join


    62 end


    63 end


    64 end


    We saw this before

    View Slide

  299. >
    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    52 "#{number.to_s}#{line}"


    53 else


    ❯ 54 string = String.new


    ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics
    ❯ 56 string << "#{number.to_s} "


    ❯ 57 string << line.to_s


    ❯ 58 string << "\e[0m"


    ❯ 59 string


    60 end


    61 end.join


    62 end


    63 end


    64 end


    We saw this before

    View Slide

  300. # Parses the most indented lines into blocks that are marked
    # and added to the frontier


    def visit_new_blocks


    max_indent = frontier.next_indent_line&.indent


    while (line = frontier.next_indent_line) && (line.indent == max_indent)
    @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block|
    record(block: block, name: "add")


    block.mark_invisible if block.valid?


    push(block, name: "add")


    end


    end


    end


    # Given an already existing block in the frontier, expand it to see
    # if it contains our invalid syntax


    code_search.rb
    This step only uses indentation.

    View Slide

  301. # Parses the most indented lines into blocks that are marked
    # and added to the frontier


    def visit_new_blocks


    max_indent = frontier.next_indent_line&.indent


    while (line = frontier.next_indent_line) && (line.indent == max_indent)
    @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block|
    record(block: block, name: "add")


    block.mark_invisible if block.valid?


    push(block, name: "add")


    end


    end


    end


    # Given an already existing block in the frontier, expand it to see
    code_search.rb
    If the block that it found is valid ruby code, it's essentially commented out.

    View Slide

  302. >
    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    52 "#{number.to_s}#{line}"


    53 else


    ❯ 54 string = String.new


    ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics
    ❯ 56 string << "#{number.to_s} "


    ❯ 57 string << line.to_s


    ❯ 58 string << "\e[0m"


    ❯ 59 string


    60 end


    61 end.join


    62 end


    63 end


    64 end


    We saw this before

    View Slide

  303. >
    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    52 "#{number.to_s}#{line}"


    53 else


    60 end


    61 end.join


    62 end


    63 end


    64 end


    View Slide

  304. # Parses the most indented lines into blocks that are marked
    # and added to the frontier


    def visit_new_blocks


    max_indent = frontier.next_indent_line&.indent


    while (line = frontier.next_indent_line) && (line.indent == max_indent)
    @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block|
    record(block: block, name: "add")


    block.mark_invisible if block.valid?


    push(block, name: "add")


    end


    end


    end


    # Given an already existing block in the frontier, expand it to see
    code_search.rb
    The code block is then pushed onto the frontier and the loop continues

    View Slide

  305. sweep(block: block, name: "comments")


    end


    # Main search loop


    def call


    sweep_heredocs


    sweep_comments


    until frontier.holds_all_syntax_errors?


    @tick += 1


    if frontier.expand?


    expand_invalid_block


    else


    visit_new_blocks


    end


    end


    @invalid_blocks.concat(frontier.detect_invalid_blocks )
    code_search.rb
    We've still not found the problem we will continue

    View Slide

  306. sweep(block: block, name: "comments")


    end


    # Main search loop


    def call


    sweep_heredocs


    sweep_comments


    until frontier.holds_all_syntax_errors?


    @tick += 1


    if frontier.expand?


    expand_invalid_block


    else


    visit_new_blocks


    end


    end


    @invalid_blocks.concat(frontier.detect_invalid_blocks )
    code_search.rb
    There's still lines with more indentation

    View Slide

  307. sweep(block: block, name: "comments")


    end


    # Main search loop


    def call


    sweep_heredocs


    sweep_comments


    until frontier.holds_all_syntax_errors?


    @tick += 1


    if frontier.expand?


    expand_invalid_block


    else


    visit_new_blocks


    end


    end


    @invalid_blocks.concat(frontier.detect_invalid_blocks )
    code_search.rb
    We need to add them

    View Slide

  308. >
    43 string << "```\n"


    44 string


    45 end


    46


    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    ❯ 52 "#{number.to_s}#{line}"


    53 else


    60 end


    61 end.join


    62 end


    63 end


    64 end


    View Slide

  309. >
    43 string << "```\n"


    44 string


    45 end


    46


    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    53 else


    60 end


    61 end.join


    62 end


    63 end


    64 end


    Boop

    View Slide

  310. sweep(block: block, name: "comments")


    end


    # Main search loop


    def call


    sweep_heredocs


    sweep_comments


    until frontier.holds_all_syntax_errors?


    @tick += 1


    if frontier.expand?


    expand_invalid_block


    else


    visit_new_blocks


    end


    end


    @invalid_blocks.concat(frontier.detect_invalid_blocks )
    code_search.rb
    We've added all lines at this indent level. It's time to expand the frontier

    View Slide

  311. code_search.rb
    sweep(block: block, name: "comments")


    end


    # Main search loop


    def call


    sweep_heredocs


    sweep_comments


    until frontier.holds_all_syntax_errors?


    @tick += 1


    if frontier.expand?


    expand_invalid_block


    else


    visit_new_blocks


    end


    end


    @invalid_blocks.concat(frontier.detect_invalid_blocks )
    We call our expansion phase

    View Slide

  312. end


    end


    end


    # Given an already existing block in the frontier, expand it to see
    # if it contains our invalid syntax


    def expand_invalid_block


    block = frontier.pop


    return unless block


    record(block: block, name: "pop")


    block = @block_expand.call(block)


    push(block, name: "expand")


    end


    def sweep_heredocs


    HeredocBlockParse.new(


    source: @source,


    code_search.rb
    Here's our expand code

    View Slide

  313. end


    end


    end


    # Given an already existing block in the frontier, expand it to see
    # if it contains our invalid syntax


    def expand_invalid_block


    block = frontier.pop


    return unless block


    record(block: block, name: "pop")


    block = @block_expand.call(block)


    push(block, name: "expand")


    end


    def sweep_heredocs


    HeredocBlockParse.new(


    source: @source,


    code_search.rb
    The frontier is stored in indentation order. The highest indentation block is popped o
    f

    View Slide

  314. end


    end


    end


    # Given an already existing block in the frontier, expand it to see
    # if it contains our invalid syntax


    def expand_invalid_block


    block = frontier.pop


    return unless block


    record(block: block, name: "pop")


    block = @block_expand.call(block)


    push(block, name: "expand")


    end


    def sweep_heredocs


    HeredocBlockParse.new(


    source: @source,


    code_search.rb
    The block is expanded using an object and pushed back onto the frontier. How does block expansion work?

    View Slide

  315. # puts "wow"


    # end


    #


    class BlockExpand


    def initialize(code_lines: )


    @code_lines = code_lines


    end


    def call(block)


    if (next_block = expand_neighbors(block, grab_empty: true))
    return next_block


    end


    expand_indent(block)


    end


    def expand_indent(block)


    block = AroundBlockScan.new(code_lines: @code_lines, block: block)
    block_expand.rb
    It can either expand to capture more lines at the same indentation, aka neighbors. Or when those are exhausted it can expand out an indentation.

    View Slide

  316. def expand_indent(block)


    block = AroundBlockScan.new(code_lines: @code_lines, block: block)
    .skip(:hidden?)


    .stop_after_kw


    .scan_adjacent_indent


    .code_block


    end


    def expand_neighbors(block, grab_empty: true)


    scan = AroundBlockScan.new(code_lines: @code_lines, block: block)
    .skip(:hidden?)


    .stop_after_kw


    .scan_neighbors


    # Slurp up empties


    if grab_empty


    scan = AroundBlockScan.new(code_lines: @code_lines, block: scan.code_block)
    block_expand.rb
    Both methods use this class AroundBlockScan which acts as a DSL for scanning up and down. This is where we give our algorithm rules about how to build blocks using
    indentation and lexical data

    View Slide

  317. >
    41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename
    42 string << code_with_lines


    43 string << "```\n"


    44 string


    45 end


    46


    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    51 if line.empty?


    ❯ 52 "#{number.to_s}#{line}"


    53 else


    60 end


    61 end.join


    62 end


    63 end


    64 end


    Previously we captured this line, and put it on the frontier.

    View Slide

  318. >
    41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename
    42 string << code_with_lines


    43 string << "```\n"


    44 string


    45 end


    46


    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    ❯ 51 if line.empty?


    ❯ 53 else


    ❯ 60 end


    61 end.join


    62 end


    63 end


    64 end


    When it's popped and expanded it looks around and
    fi
    nds a matched set of keywords using the scanner.

    View Slide

  319. >
    41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename
    42 string << code_with_lines


    43 string << "```\n"


    44 string


    45 end


    46


    47 def code_with_lines


    48 @code_lines.map do |line|


    49 next if line.hidden?


    50 number = line.line_number.to_s.rjust(@digit_count)


    61 end.join


    62 end


    63 end


    64 end


    The code is valid so it's hidden. The expanded block is put back on the frontier.

    View Slide

  320. sweep(block: block, name: "comments")


    end


    # Main search loop


    def call


    sweep_heredocs


    sweep_comments


    until frontier.holds_all_syntax_errors?


    @tick += 1


    if frontier.expand?


    expand_invalid_block


    else


    visit_new_blocks


    end


    end


    @invalid_blocks.concat(frontier.detect_invalid_blocks )
    code_search.rb
    That's pretty much how the algorithm for search works. It keeps looping, expanding and adding until all syntax errors are found

    View Slide

  321. sweep(block: block, name: "comments")


    end


    # Main search loop


    def call


    sweep_heredocs


    sweep_comments


    until frontier.holds_all_syntax_errors?


    @tick += 1


    if frontier.expand?


    expand_invalid_block


    else


    visit_new_blocks


    end


    end


    @invalid_blocks.concat(frontier.detect_invalid_blocks )
    code_search.rb
    Once we've removed all syntax errors, we have to
    fi
    nd our problematic code.

    View Slide

  322. def call


    sweep_heredocs


    sweep_comments


    until frontier.holds_all_syntax_errors?


    @tick += 1


    if frontier.expand?


    expand_invalid_block


    else


    visit_new_blocks


    end


    end


    @invalid_blocks.concat(frontier.detect_invalid_blocks )
    @invalid_blocks.sort_by! {|block| block.starts_at }
    self


    end


    end


    code_search.rb
    The frontier can hold more than one syntax error.

    View Slide

  323. # Example:


    #


    # combination([:a, :b, :c, :d])


    # # => [[:a], [:b], [:c], [:d],


    # [:a, :b], [:a, :c], [:a, :d],


    # [:b, :c], [:b, :d], [:c, :d],


    # [:a, :b, :c], [:a, :b, :d], [:a, :c, :d],
    # [:b, :c, :d],


    # [:a, :b, :c, :d]]


    def self.combination(array)


    guesses = []


    1.upto(array.length).each do |size|


    guesses.concat(array.combination(size).to_a)


    end


    guesses


    end


    # Given that we know our syntax error exists somewhere in our frontier, we want to find
    code_frontier.rb
    We want to iterate over all of the invalid blocks stored in the frontier to
    fi
    nd the shortest possible order that gives us a valid document.

    View Slide

  324. # Example:


    #


    # combination([:a, :b, :c, :d])


    # # => [[:a], [:b], [:c], [:d],


    # [:a, :b], [:a, :c], [:a, :d],


    # [:b, :c], [:b, :d], [:c, :d],


    # [:a, :b, :c], [:a, :b, :d], [:a, :c, :d],
    # [:b, :c, :d],


    # [:a, :b, :c, :d]]


    def self.combination(array)


    guesses = []


    1.upto(array.length).each do |size|


    guesses.concat(array.combination(size).to_a)


    end


    guesses


    end


    # Given that we know our syntax error exists somewhere in our frontier, we want to find
    code_frontier.rb
    To do that we create a combination of all possible blocks

    View Slide

  325. end


    # Given that we know our syntax error exists somewhere in our frontier, we want to find
    # the smallest possible set of blocks that contain all the syntax errors
    def detect_invalid_blocks


    self.class.combination(@frontier.select(&:invalid?)).detect do |block_array|
    holds_all_syntax_errors?(block_array)


    end || []


    end


    end


    end


    code_frontier.rb
    Then one at a time we iterate through all the combinations

    View Slide

  326. guesses


    end


    # Given that we know our syntax error exists somewhere in our frontier, we want to find
    # the smallest possible set of blocks that contain all the syntax errors
    def detect_invalid_blocks


    self.class.combination(@frontier.select(&:invalid?)).detect do |block_array|
    holds_all_syntax_errors?(block_array)


    end || []


    end


    end


    end


    code_frontier.rb
    And check each one to see if we hold all the syntax errors. The smallest successful combination is returned

    View Slide

  327. Timeout.timeout(timeout) do


    record_dir ||= ENV["DEBUG"] ? "tmp" : nil


    search = CodeSearch.new(source, record_dir: record_dir).call
    end


    blocks = search.invalid_blocks


    DisplayInvalidBlocks.new(


    blocks: blocks,


    filename: filename,


    terminal: terminal,


    code_lines: search.code_lines,


    invalid_obj: invalid_type(source),


    io: io


    ).call


    dead_end/internals.rb
    Here's where our search started.

    View Slide

  328. Timeout.timeout(timeout) do


    record_dir ||= ENV["DEBUG"] ? "tmp" : nil


    search = CodeSearch.new(source, record_dir: record_dir).call
    end


    blocks = search.invalid_blocks


    DisplayInvalidBlocks.new(


    blocks: blocks,


    filename: filename,


    terminal: terminal,


    code_lines: search.code_lines,


    invalid_obj: invalid_type(source),


    io: io


    ).call


    dead_end/internals.rb
    Then the invalid blocks were found

    View Slide

  329. Now we want to decorate the output

    View Slide

  330. Timeout.timeout(timeout) do


    record_dir ||= ENV["DEBUG"] ? "tmp" : nil


    search = CodeSearch.new(source, record_dir: record_dir).call
    end


    blocks = search.invalid_blocks


    DisplayInvalidBlocks.new(


    blocks: blocks,


    filename: filename,


    terminal: terminal,


    code_lines: search.code_lines,


    invalid_obj: invalid_type(source),


    io: io


    ).call


    dead_end/internals.rb
    We pass it to a pretty printer display

    View Slide

  331. lines = CaptureCodeContext.new(


    blocks: @blocks,


    code_lines: @code_lines


    ).call


    DisplayCodeWithLineNumbers.new(


    lines: lines,


    terminal: @terminal,


    highlight_lines: @invalid_lines


    ).call
    display_invalid_blocks.rb
    Here's where we take care of the last of our gotchas

    View Slide

  332. lines = CaptureCodeContext.new(


    blocks: @blocks,


    code_lines: @code_lines


    ).call


    DisplayCodeWithLineNumbers.new(


    lines: lines,


    terminal: @terminal,


    highlight_lines: @invalid_lines


    ).call
    display_invalid_blocks.rb
    This capture code context handles the ambiguities we talked about before

    View Slide

  333. >
    ❯ 1 class Dog




    Gotcha: ambiguity
    🚫
    It knows about our ambiguous edge cases that we talked about before

    View Slide

  334. >
    ❯ 1 class Dog


    ❯ 2 def sit


    ❯ 3 puts "no"


    ❯ 4 end


    ❯ 5


    ❯ 6 def eat


    ❯ 7 puts "munch"


    ❯ 8 end


    ❯ 9


    ❯ 10 def bark


    ❯ 11 puts "woof"


    ❯ 12 end




    Gotcha: ambiguity
    It goes back through the hidden code

    View Slide

  335. >
    ❯ 1 class Dog


    ❯ 2 def sit


    ❯ 3 puts "no"


    ❯ 4 end


    ❯ 5


    ❯ 6 def eat


    ❯ 7 puts "munch"


    ❯ 8 end


    ❯ 9


    ❯ 10 def bark


    ❯ 11 puts "woof"


    ❯ 12 end




    Gotcha: ambiguity

    View Slide

  336. >
    ❯ 1 class Dog


    ❯ 2 def sit


    ❯ 3 puts "no"


    ❯ 4 end


    ❯ 5


    ❯ 6 def eat


    ❯ 7 puts "munch"


    ❯ 8 end


    ❯ 9


    ❯ 10 def bark


    ❯ 11 puts "woof"


    ❯ 12 end




    Gotcha: ambiguity
    Work backwards and
    fi
    nd an unmatched keyword

    View Slide

  337. >
    ❯ 1 class Dog


    ❯ 10 def bark


    ❯ 12 end
    Gotcha: ambiguity
    And transforms it

    View Slide

  338. 🎂
    Piece of cake

    View Slide

  339. >
    DeadEnd: Missing `end` detected


    This code has a missing `end`. Ensure that all


    syntax keywords (`def`, `do`, etc.) have a matching `end`.
    file: /Users/rschneeman/Documents/projects/dead_end/spec/fixtures/this_project_extra_def.rb.txt
    simplified:


    1 module SyntaxErrorSearch


    3 class DisplayInvalidBlocks


    4 attr_reader :filename


    ❯ 36 def filename


    ❯ 38 def code_with_filename


    ❯ 45 end


    63 end


    64 end
    After all three steps, cleaning, searching, formatting here's the output

    View Slide

  340. dead_end
    That is dead end in a nutshell *laugh*

    View Slide

  341. Gem `error_highlight` in Ruby 3.1
    Beyond dead_end there's an amazing gem being added to Ruby 3.1 called error_highlight. It shows you which method got a no method error. I love this introduction

    View Slide

  342. Gem `error_highlight` in Ruby 3.1
    The error highlight gem was created by you-ssssss-kay. He goes by mam-eh on GitHub and gave an amazing talk at a conference I used to run called keep ruby weird.

    View Slide

  343. Rust v. Ruby
    With error highlight and dead end I also want to touch on community values.

    View Slide

  344. Rust v. Ruby
    I've been writing Rust code and their community takes a very aggressive stance towards error messages. It's not enough for them to say there's a problem. They also
    want to accurately suggest how to
    fi
    x the problem

    View Slide

  345. Rust v. Ruby
    I opened this issue against rustlang and in under a month it was addressed to improve the error output.

    View Slide

  346. Rust v. Ruby
    In general the Rust community treats a user experience problem as a bug and error output is a
    fi
    rst class feature. If our community wanted we could invest more in errors

    View Slide

  347. Dead end isn't that solution, but it's a start. You can add the dead_end gem to your project to try it today and give me feedback on what works and what doesn't.

    View Slide

  348. 📕
    HowTo
    Open
    Source
    .dev
    You can pre-order my book on open source contribution by visiting how to open source dot dev

    View Slide

  349. You can sign up to triage open source issues at code triage dot com

    View Slide

  350. Lexing

    Parsing

    Syntax

    SyntaxErrors

    AI

    Pathfinding and goal seeking
    Today we talked about lexing, parsing, syntax syntax errors, AI, path
    fi
    nding and goal seeking

    View Slide

  351. But, the technical pieces are not what I want you to remember.

    View Slide

  352. - The important part is that everyone sitting in this room is the future of developer tooling. Programming is inherently di
    ffi
    cult, but it doesn't mean our tools can't help us.

    View Slide

  353. One of the best ways to judge a system is to see how it fails. When we can give care and grace and beauty to our failure modes then we can create experiences that
    delight us. Experiences that teach us. Experiences that elevate our code and our consciousness.

    View Slide

  354. - A syntax error doesn't have to be an ending...it can be a beginning. To a beautiful programming story

    View Slide

  355. How

    To

    Open

    Source

    .dev
    - My name is Richard Schneeman

    - I'm writing a book at how to open source dot dev

    - I created CodeTriage.com the easiest way to get started contributing to OpenSource

    -

    - Wait for "bye bye"

    - Zoom wave everyone

    View Slide

  356. View Slide

  357. Questions?

    View Slide