[RubyConf] Beware the Dreaded Dead End

Slide 1

Slide 1 text

Beware the Dreaded Dead End!!! By @schneems Hello everyone. My name is Richard Schneeman and I want to talk to you about the scariest thing a Ruby programmer can face. You hear that?

Slide 2

Slide 2 text

Everyone run

Slide 3

Slide 3 text

Oh no, there are more of them. AHHHH

Slide 4

Slide 4 text

Beware the Dreaded Dead End!!! By @schneems Wow, I can hardly believe that the those dinosaurs from RubyKaigi found me all the way over here, I thought I escaped them already. If you've seen that talk, know that this one has some new content.

Slide 5

Slide 5 text

Beware the Dreaded Dead End!!! By @schneems While dinosaurs are scary. There's something scarier. Its..

Slide 6

Slide 6 text

> # <= routes.rb:121: syntax error, unexpected end-of-input, expecting `end’ Rails.application.routes.draw do constraints -> { Rails.application.config.non_production } do namespace :foo do resource :bar end end constraints -> { Rails.application.config.non_production } do namespace :bar do resource :baz end end namespace :admin do resource :session match "/foobar(*path)", via: :all, to: redirect { |_params, req| uri = URI(req.path.gsub("foobar", "foobaz")) uri.query = req.query_string.presence uri.to_s } end Syntax errors

Slide 7

Slide 7 text

> Rails.application.routes.draw do constraints -> { Rails.application.config.non_production } do namespace :foo do resource :bar end end constraints -> { Rails.application.config.non_production } do namespace :bar do resource :baz end end namespace :admin do resource :session match "/foobar(*path)", via: :all, to: redirect { |_params, req| uri = URI(req.path.gsub("foobar", "foobaz")) uri.query = req.query_string.presence uri.to_s } end # <= routes.rb:121: syntax error, unexpected end-of-input, expecting `end’ Just look at this unexpected syntax error. It's horrifying. Where's the problem?

Slide 8

Slide 8 text

> RSpec.describe Cutlass::BashResult do it "preserves stdout, stderr, and status" stdout = SecureRandom.hex(16) stderr = SecureRandom.hex(16) status = 0 result = BashResult.new( stdout: stdout, stderr: stderr, status: status ) expect(result.stdout).to eq(stdout) expect(result.stderr).to eq(stderr) expect(result.status).to eq(status) end end # <= result_spec:19 syntax error, unexpected end-of-input, expecting `end’ Here's some di ff erent code. Wanna guess where Ruby think the problem is? On the last line, nope. This is frustrating

Slide 9

Slide 9 text

🤔 What if we had something better

Slide 10

Slide 10 text

Slide 11

Slide 11 text

> 3 module Cutlass 4 RSpec.describe Cutlass::BashResult do ❯ 5 it "preserves stdout, stderr, and status" ❯ 19 end 21 it "success?" do 52 end 53 end 54 end This code has an unmatched `end`. Ensure that all `end` lines in your code have a matching syntax keyword (`def`, `do`, etc.) What if we got this? Do you see the error now? Doesn't line 5 look suspicious, like maybe it's missing a do.

Slide 12

Slide 12 text

🎂 Have you ever seen a cooking show where they show o ff the fi nal product? I want to show o ff dead end before we get too far.

Slide 13

Slide 13 text

demo Here is a demo of how dead_end fi nds syntax errors

Slide 14

Slide 14 text

module SyntaxErrorSearch # Used for formatting invalid blocks class DisplayInvalidBlocks attr_reader :filename def initialize(block_array, io: $stderr, filename: nil) @filename = filename @io = io @blocks = block_array @lines = @blocks.map(&:lines).flatten @digit_count = @lines.last.line_number.to_s.length @code_lines = @blocks.first.code_lines @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} end def call @io.puts <<~EOM 64: syntax error, unexpected end-of-input, expecting `end' 📕 ❌ This is real ruby source code with a syntax error in it. It cannot parse

Slide 15

Slide 15 text

51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s 58 string << "\e[0m" 59 string 60 end 61 end.join 62 end 63 end 64 end

Slide 16

Slide 16 text

50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end

Slide 17

Slide 17 text

50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end

Slide 18

Slide 18 text

50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end

Slide 19

Slide 19 text

50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end

Slide 20

Slide 20 text

50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end

Slide 21

Slide 21 text

50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else ❯ 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end When a valid code block is found dead_end safely removes it

Slide 22

Slide 22 text

50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 60 end 61 end.join 62 end 63 end 64 end 📕 ❌ After each step in the search, dead end re-evaluates the whole document to see if it's parsable yet. parsing failed, we need to keep looking

Slide 23

Slide 23 text

42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? ❯ 52 "#{number.to_s}#{line}" 53 else 60 end 61 end.join 62 end 63 end 64 end

Slide 24

Slide 24 text

42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 53 else 60 end 61 end.join 62 end 63 end 64 end 📕 ❌

Slide 25

Slide 25 text

42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) ❯ 51 if line.empty? ❯ 53 else ❯ 60 end 61 end.join 62 end 63 end 64 end

Slide 26

Slide 26 text

Slide 27

Slide 27 text

42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| ❯ 49 next if line.hidden? ❯ 50 number = line.line_number.to_s.rjust(@digit_count) 61 end.join 62 end 63 end 64 end

Slide 28

Slide 28 text

42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 61 end.join 62 end 63 end 64 end 📕 ❌

Slide 29

Slide 29 text

42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines ❯ 48 @code_lines.map do |line| ❯ 61 end.join 62 end 63 end 64 end

Slide 30

Slide 30 text

42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 62 end 63 end 64 end 📕 ❌

Slide 31

Slide 31 text

42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 ❯ 47 def code_with_lines ❯ 62 end 63 end 64 end

Slide 32

Slide 32 text

40 string << "```\n" 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 63 end 64 end 📕 ❌

Slide 33

Slide 33 text

13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 63 end 64 end

Slide 34

Slide 34 text

Slide 35

Slide 35 text

13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end

Slide 36

Slide 36 text

13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename ❯ 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end

Slide 37

Slide 37 text

13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") 40 string << "```\n" ❯ 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename ❯ 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end

Slide 38

Slide 38 text

13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename 39 string = String.new("") ❯ 40 string << "```\n" ❯ 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename ❯ 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end

Slide 39

Slide 39 text

13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename ❯ 39 string = String.new("") ❯ 40 string << "```\n" ❯ 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename ❯ 42 string << code_with_lines ❯ 43 string << "```\n" ❯ 44 string 45 end 46 63 end 64 end

Slide 40

Slide 40 text

13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename 45 end 46 63 end 64 end 📕 ❌

Slide 41

Slide 41 text

1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array 10 @lines = @blocks.map(&:lines).flatten 11 @digit_count = @lines.last.line_number.to_s.length 12 @code_lines = @blocks.first.code_lines 13 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename

Slide 42

Slide 42 text

1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array 10 @lines = @blocks.map(&:lines).flatten 11 @digit_count = @lines.last.line_number.to_s.length 12 @code_lines = @blocks.first.code_lines 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename

Slide 43

Slide 43 text

1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array 10 @lines = @blocks.map(&:lines).flatten 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename

Slide 44

Slide 44 text

1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename

Slide 45

Slide 45 text

1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io 9 @blocks = block_array ❯ 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename

Slide 46

Slide 46 text

1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename 8 @io = io ❯ 9 @blocks = block_array ❯ 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename

Slide 47

Slide 47 text

1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 7 @filename = filename ❯ 8 @io = io ❯ 9 @blocks = block_array ❯ 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename

Slide 48

Slide 48 text

1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) ❯ 7 @filename = filename ❯ 8 @io = io ❯ 9 @blocks = block_array ❯ 10 @lines = @blocks.map(&:lines).flatten ❯ 11 @digit_count = @lines.last.line_number.to_s.length ❯ 12 @code_lines = @blocks.first.code_lines ❯ 13 ❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true} 15 end 16 35 36 def filename 37 38 def code_with_filename

Slide 49

Slide 49 text

1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 15 end 16 35 36 def filename 37 38 def code_with_filename 45 end 46 63 end 64 end 📕 ❌

Slide 50

Slide 50 text

1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 15 end 16 35 36 def filename 37 ❯ 38 def code_with_filename ❯ 45 end 46 63 end 64 end

Slide 51

Slide 51 text

1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 6 def initialize(block_array, io: $stderr, filename: nil) 15 end 16 35 36 def filename 37 46 63 end 64 end 📕 ❌

Slide 52

Slide 52 text

1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 ❯ 6 def initialize(block_array, io: $stderr, filename: nil) ❯ 15 end 16 35 36 def filename 37 46 63 end 64 end

Slide 53

Slide 53 text

📕 ❌ 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 5 16 35 36 def filename 37 46 63 end 64 end

Slide 54

Slide 54 text

1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename ❯ 5 ❯ 16 ❯ 35 ❯ 36 def filename ❯ 37 ❯ 46 63 end 64 end

Slide 55

Slide 55 text

1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 63 end 64 end 📕 ✅ The document is checked. The parser reports that this very minimal document is now parsable.

Slide 56

Slide 56 text

Syntax O 🥰🥰🥰 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 63 end 64 end 📕 ✅ Once this happens dead_end has found a way to transform the document to be valid. It can stop searching the document and instead search the invalid code blocks that it has stored

Slide 57

Slide 57 text

DeadEnd: Missing ènd` detected This code has a missing ènd`. Ensure that all syntax keywords (`def`, `do`, etc.) have a matching ènd`. file: /Users/rschneeman/Documents/projects/dead_end/spec/fixtures/this_project_extra_def.rb.txt simplified: 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename ❯ 36 def filename ❯ 38 def code_with_filename ❯ 45 end 63 end 64 end This is the actual output the dead end gem returns on that fi le. The issue is on line 36 there's a missing end statement

Slide 58

Slide 58 text

$ gem install dead_end The dead_end gem is released and on ruby gems.

Slide 59

Slide 59 text

By the time you see this slide dead_end will have surpassed half a million downloads

Slide 60

Slide 60 text

Ruby 3.2? I am also talking with Ruby core to get dead_end into Ruby directly. We are targeting that integration for 2022 release of Ruby 3.2

Slide 61

Slide 61 text

🤔 What all can dead_end do?

Slide 62

Slide 62 text

> 3 module Cutlass 4 RSpec.describe Cutlass::BashResult do ❯ 5 it "preserves stdout, stderr, and status" ❯ 19 end 21 it "success?" do 52 end 53 end 54 end DeadEnd: Unmatched `end` detected When you miss a keyword like "if", "do" or "def". Dead end fi nds the problem.

Slide 63

Slide 63 text

> 1 module Cutlass 2 RSpec.describe Cutlass::BashResult do ❯ 3 it "preserves stdout, stderr, and status" do 5 it "success?" do 6 end 7 end 8 end DeadEnd: Missing `end` detected Even when you miss an end keyword, dead end fi nds the problem

Slide 64

Slide 64 text

> 1 class Cat 2 def to_json ❯ 3 hash = { ❯ 4 name: name, ❯ 5 fur_color: fur_color, ❯ 6 type: 'Cat', ❯ 7 better_than_dog: false ❯ 8 hash.to_json 9 end 10 end DeadEnd: Unmatched `}` character detected When you miss a curly bracket dead end fi nds the problem

Slide 65

Slide 65 text

> 1 class Animal 2 def self.ones_my_three_year_old_likes ❯ 3 array = [ ❯ 4 Dog.new, ❯ 5 Lion.new, ❯ 6 Tiger.new, ❯ 7 Aligator.new, 10 return array 11 end 12 end DeadEnd: Unmatched `[` detected When you miss a square bracket dead end fi nds the problem

Slide 66

Slide 66 text

> 1 class Cat 2 def meow ❯ 3 Animal.call do |a ❯ 5 end 6 end 7 end DeadEnd: Unmatched `|` character detected Have a problem with a missing pipe character? Dead end. Finds. The. Problem Have a problem with a missing family member in a Korean thriller drama?

Slide 67

Slide 67 text

> Unfortunately. Dead end cannot fi nd that problem. . Okay, wow that current event reference got really dark. Let's rewind a try again.

Slide 68

Slide 68 text

> Have a problem with a missing marvel universe character?

Slide 69

Slide 69 text

> 1 class Cat 2 def meow ❯ 3 Animal.call do |a ❯ 5 end 6 end 7 end DeadEnd: Unmatched `|` character detected ? ? ? Unfortunately. Dead end cannot fi nd that problem. Much better. Remember crocodile Loki? That was 2021, this year is lasting forever!

Slide 70

Slide 70 text

$ gem install dead_end Today we'll dig into

Slide 71

Slide 71 text

$ dead_end bad.rb --record tmp/ You can follow the algorithm yourself by running the CLI with the record fl ag

Slide 72

Slide 72 text

You'll see each step

Slide 73

Slide 73 text

Along with annotated source code

Slide 74

Slide 74 text

Syntax errors Today we will talk about

Slide 75

Slide 75 text

Lexing & Parsing How to parse and lex source code with ripper

Slide 76

Slide 76 text

AI How AI and path fi nding algorithms work

Slide 77

Slide 77 text

dead_end We will put it all together to see how dead_end works

Slide 78

Slide 78 text

Internals Finally we'll crack open the cover and see how it works at a low level. Warning: We will get technical. We will cover a lot of code.

Slide 79

Slide 79 text

👋 But wait, who am I?

Slide 80

Slide 80 text

@schneems I go by schneems on the internet. If you forget how to pronounce my name you can go to

Slide 81

Slide 81 text

@schneems the about page on my blog on Schneems dot com

Slide 82

Slide 82 text

I Created CodeTriage which is a platform for learning how to contribute to open source. To date I've got over 60 thousand developers signed up for CodeTriage

Slide 83

Slide 83 text

📕 HowTo Open Source .dev I'm writing a book on how to contribute to open source as well. It's for developers who want to start contributing. Developers who got stuck, and developers looking for ways to sustain contributions. I'm not ready to release it yet, but you can sign up for a pre-order on HowToOpenSource dot dev Also you can sign up for Code Triage dot com and I will email members when the book is ready.

Slide 84

Slide 84 text

Heroku When I'm not working on open source or teaching people to open source with CodeTriage. I like to get paid. I work at Heroku

Slide 85

Slide 85 text

Heroku Right now I'm working on salesforce functions. It's an easy way to work with the data you've got in salesforce using the languages you love. Right now we support Java and JavaScript languages. We'll be rolling out Ruby support later. If you use love Ruby and use Salesforce I would love to hear from you. My DMs on Twitter are open. Also people tell me

Slide 86

Slide 86 text

NoMemoryError ScriptError LoadError SyntaxError SecurityError SignalException KeyError StandardError ArgumentError EncodingError IOError EOFError I am Exceptional i'm an exceptional programmer... My programs generate a lot of exceptions

Slide 87

Slide 87 text

Syntax errors Syntax errors. Let's start from the beginning what is a syntax error? Why are syntax errors so hard to understand?

Slide 88

Slide 88 text

> while b != 0 if a > b a = a - b else b = b - a end end return a This code worked wonderfully comparing a, and b all day.

Slide 89

Slide 89 text

> while b != 0 if a > b a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a When Ruby parses this code it will convert it into an abstract syntax tree. this code generates that tree. The tree was beautiful and Ruby's parser looked upon it with happiness

Slide 90

Slide 90 text

> while b != 0 if a > b a = a - b else b = b - a while b != 0 if a > b a = a - b else b = b - a end end return a 😈 While the code was good a stranger came upon the land. This stranger had a secret power, behold

Slide 91

Slide 91 text

# The octothorpe, This tiny character gave the stranger great power to create, or destroy.

Slide 92

Slide 92 text

> while b != 0 if a > b a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a 😈 With one key the stranger used their power

Slide 93

Slide 93 text

> while b != 0 # if a > b a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a 😈 to transform the code

Slide 94

Slide 94 text

> while b != 0 # if a > b a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a 😈 A critical line had been commented out. Without that line, the tree was no longer whole

Slide 95

Slide 95 text

> while b != 0 # if a > b a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a 😈 Huge sections of code are no longer reachable. The code no longer parses. And our parser is sad.

Slide 96

Slide 96 text

> while b != 0 # if a > b a = a - b else b = b - a end end return a while b != 0 a = a - b else b = b - a 😈 With that the stranger left and behind them stood

Slide 97

Slide 97 text

> while b != 0 # if a > b a = a - b else b = b - a end end return a while b != 0 a = a - b else b = b - a Syntax error A syntax error

Slide 98

Slide 98 text

Syntax errors A syntax error occurs when the parser cannot build a valid syntax tree. That explains what they are, but why are they di ffi cult to understand?

Slide 99

Slide 99 text

> module Cutlass def call 1, 2, 3, 4, 5] end end syntax error, unexpected ',', expecting end-of-input (SyntaxError) When the parser tries to parse this code it fi nds a Syntax Error. Here our developer forgot a bracket. As the parser is building the tree, it hit an error because it wasn't expecting the comma

Slide 100

Slide 100 text

> syntax error, unexpected ',', expecting end-of-input (SyntaxError) module Cutlass def call 1, 2, 3, 4, 5] end end Ruby's parser has rules. It knows that after a method de fi nition it should look for something such as

Slide 101

Slide 101 text

> syntax error, unexpected ',', expecting end-of-input (SyntaxError) module Cutlass def call 1, 2, 3, 4, 5] end end Like a return or an end. Or a variable assignment. Instead what did the parser fi nd fi nd? It found a number and a comma.

Slide 102

Slide 102 text

> module Cutlass def call 1, 2, 3, 4, 5] end end syntax error, unexpected ',', expecting end-of-input (SyntaxError) 🚫 That comma violated ruby's parser rules. Do not pass go, do not collect 200

Slide 103

Slide 103 text

🤷 What's so bad about this example?

Slide 104

Slide 104 text

> module Cutlass def call 1, 2, 3, 4, 5] end end syntax error, unexpected ',', expecting end-of-input (SyntaxError) The error isn't caused by a comma, it's caused by the developer missing a bracket. Essentially, the location the parse error occurs isn't always where developer made a mistake

Slide 105

Slide 105 text

🙇 Let's look at some more complicated cases

Slide 106

Slide 106 text

> module Cutlass defcall end end Here's code with a syntax error. The developer forgot a space after the def

Slide 107

Slide 107 text

> module Cutlass defcall end end syntax error, unexpected `end', expecting end-of-input Ruby thinks the error is at the last line. right here. Why?

Slide 108

Slide 108 text

> module Cutlass defcall end end syntax error, unexpected `end', expecting end-of-input Welllll... When we start parsing Ruby sees a module de fi nition, A module is a keyword that requires an end so Ruby starts looking for a matching end.

Slide 109

Slide 109 text

> module Cutlass defcall end end syntax error, unexpected `end', expecting end-of-input On the next line it sees this combination of characters and thinks it's a method call

Slide 110

Slide 110 text

> module Cutlass defcall end end syntax error, unexpected `end', expecting end-of-input Then it sees this end and thinks it matches with the module. Our parser hasn't raised a syntax error because it doesn't see a problem yet.

Slide 111

Slide 111 text

> module Cutlass defcall end end syntax error, unexpected `end', expecting end-of-input Finally it gets to our second end, but there's no thing to match with. A syntax error is raised.

Slide 112

Slide 112 text

> end syntax error, unexpected `end', expecting end-of-input Right now this is the error you get from ruby on this code. It's not helpful to a human because deleting that end won't solve anything.

Slide 113

Slide 113 text

parser problems != human problems Another way to put this, is that parse errors are di ff erent from human errors.

Slide 114

Slide 114 text

$ gem install dead_end dead_end's goal is to turn that parser problem into something a human can see and instantly recognize as an issue.

Slide 115

Slide 115 text

$ gem install dead_end How does dead end work?

Slide 116

Slide 116 text

Dead end works by using a library called Ripper

Slide 117

Slide 117 text

require "ripper" Ripper.new(source).tap(&:parse).error? It's not a band. Ripper is Ruby's parser that ships with Ruby. How cool is that?

Slide 118

Slide 118 text

require "ripper" source = <<~EOM module Cutlass defcall end end EOM Ripper.new(source).tap(&:parse).error? Ripper can evaluate code and tell us if there's a syntax error or not. We saw this code before

Slide 119

Slide 119 text

require "ripper" source = <<~EOM module Cutlass defcall end end EOM Ripper.new(source).tap(&:parse).error? , there's a syntax error here

Slide 120

Slide 120 text

require "ripper" source = <<~EOM module Cutlass defcall end end EOM Ripper.new(source).tap(&:parse).error? # => true When we run ripper it says there's an error

Slide 121

Slide 121 text

require "ripper" source = <<~EOM module Cutlass def call end end EOM Ripper.new(source).tap(&:parse).error? # => false When we fi x the error, ripper tells us it's fi xed too.

Slide 122

Slide 122 text

🙇 If we could fi x the source errors then ripper would tell us, but that's too hard to automate. Can we do something simpler?

Slide 123

Slide 123 text

require "ripper" source = <<~EOM module Cutlass defcall end end EOM Ripper.new(source).tap(&:parse).error? # => true Back to our code with a syntax error. Instead of fi xing it. What if we...commented it out?

Slide 124

Slide 124 text

require "ripper" source = <<~EOM module Cutlass # defcall end end EOM Ripper.new(source).tap(&:parse).error? # => true That didn't work, let's keep going.

Slide 125

Slide 125 text

require "ripper" source = <<~EOM module Cutlass # defcall # end end EOM Ripper.new(source).tap(&:parse).error? # => false Oooh, nice ripper can tell us when we've commented out the source code causing the problem

Slide 126

Slide 126 text

# defcall # end Ripper.new(source).tap(&:parse).error? # => false Now if we only look at the part that was commented out, dead_end can show you where the issue occured.

Slide 127

Slide 127 text

$ gem install dead_end Remember our stranger and the mess they left behind?

Slide 128

Slide 128 text

> while b != 0 a = a - b else b = b - a end end return a while b != 0 a = a - b else b = b - a 😈 Ruby knows there's a syntax error here, but it doesn't know where.

Slide 129

Slide 129 text

> while b != 0 a = a - b else b = b - a end end return a while b != 0 a = a - b else b = b - a 😈 It will tell you

Slide 130

Slide 130 text

> while b != 0 a = a - b else b = b - a end end return a syntax error, unexpected `end', expecting end-of-input That the problem is this line. We know that's not helpful. What does dead end do?

Slide 131

Slide 131 text

> while b != 0 a = a - b else b = b - a end end return a while b != 0 a = a - b else b = b - a Dead end uses indentation and lexical parsing to deconstruct the source code from the outside in

Slide 132

Slide 132 text

> while b != 0 a = a - b else # b = b - a end end return a while b != 0 a = a - b else b = b - a Commenting out this code removes a node from our tree

Slide 133

Slide 133 text

> while b != 0 # a = a - b else # b = b - a end end return a while b != 0 a = a - b else Commenting out this code removes a node from our tree, but it's still invalid

Slide 134

Slide 134 text

> while b != 0 # a = a - b # else # b = b - a # end end return a while b != 0 else Finally here we have a valid tree. All of the orphaned syntax nodes are gone

Slide 135

Slide 135 text

> 1 while b != 0 ❯ 4 else ❯ 6 end 7 end This code has an unmatched `end`. Ensure that all `end` lines in your code have a matching syntax keyword (`def`, `do`, etc.) Here's the output for that code. It shows that the else and end are missing an if statement.

Slide 136

Slide 136 text

I bet you're thinking wow, that was easy. So all he did was comment out code based on indentation?

Slide 137

Slide 137 text

Gotchas There are some major gotchas when executing this recursive comment approach.

Slide 138

Slide 138 text

Gotcha #1 We already know the developer has a syntax error. We can't assume they've done everything else perfectly.   We cannot assume perfect indentation

Slide 139

Slide 139 text

> describe "things" do it "valid" do expect(the).to eq(unexpected) end it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything Our syntax error is on this line. It is missing keyword do

Slide 140

Slide 140 text

> describe "things" do it "valid" do expect(the).to eq(unexpected) end it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything Without it, the parser has an extra end here

Slide 141

Slide 141 text

> describe "things" do it "valid" do expect(the).to eq(unexpected) end it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything If we only look at indentation then we would remove this line

Slide 142

Slide 142 text

> describe "things" do it "valid" do end it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything Then this mis-indented end

Slide 143

Slide 143 text

> describe "things" do it "valid" do it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything 📕 ✅ With the end gone, the document parses. Why?

Slide 144

Slide 144 text

> describe "things" do it "valid" do it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything 📕 ✅ Because this keyword now matches with

Slide 145

Slide 145 text

> describe "things" do it "valid" do it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything 🚫 This end It's not what we wanted, and didn't capture the source of the syntax error

Slide 146

Slide 146 text

🙇 How can we fi x it? Instead of only looking at indentation we also look at lex output

Slide 147

Slide 147 text

source = <<~EOM it "valid" do expect(the).to eq(unexpected) end EOM Ripper.lex(source) Ruby ships with a lexer

Slide 148

Slide 148 text

source = <<~EOM it "valid" do expect(the).to eq(unexpected) end EOM Ripper.lex(source) # => [[[1, 0], :on_ident, "it", CMDARG], [[1, 2], :on_sp, " ", CMDARG], [[1, 3], :on_tstring_beg, "\"", CMDARG], [[1, 4], :on_tstring_content, "valid", CMDARG], [[1, 9], :on_tstring_end, "\"", END], [[1, 10], :on_sp, " ", END], [[1, 11], :on_kw, "do", BEG], [[1, 13], :on_ignored_nl, "\n", BEG], Here's what it looks like

Slide 149

Slide 149 text

# => [[[1, 0], :on_ident, "it", CMDARG], [[1, 2], :on_sp, " ", CMDARG], [[1, 3], :on_tstring_beg, "\"", CMDARG], [[1, 4], :on_tstring_content, "valid", CMDARG], [[1, 9], :on_tstring_end, "\"", END], [[1, 10], :on_sp, " ", END], [[1, 11], :on_kw, "do", BEG], [[1, 13], :on_ignored_nl, "\n", BEG], [[2, 0], :on_sp, " ", BEG], [[2, 6], :on_ident, "expect", CMDARG], [[2, 12], :on_lparen, "(", BEG|LABEL], [[2, 13], :on_ident, "the", ARG], [[2, 16], :on_rparen, ")", ENDFN], [[2, 17], :on_period, ".", DOT], [[2, 18], :on_ident, "to", ARG], DO NOT CLICK, auto advance

Slide 150

Slide 150 text

Slide 151

Slide 151 text

[[2, 12], :on_lparen, "(", BEG|LABEL], [[2, 13], :on_ident, "the", ARG], [[2, 16], :on_rparen, ")", ENDFN], [[2, 17], :on_period, ".", DOT], [[2, 18], :on_ident, "to", ARG], [[2, 20], :on_sp, " ", ARG], [[2, 21], :on_ident, "eq", ARG], [[2, 23], :on_lparen, "(", BEG|LABEL], [[2, 24], :on_ident, "unexpected", ARG], [[2, 34], :on_rparen, ")", ENDFN], [[2, 35], :on_sp, " ", ENDFN], [[2, 36], :on_nl, "\n", BEG], [[3, 0], :on_sp, " ", BEG], [[3, 2], :on_kw, "end", END], [[3, 5], :on_nl, "\n", BEG]] DO NOT CLICK, auto advance

Slide 152

Slide 152 text

Slide 153

Slide 153 text

$ gem install dead_end With this info dead_end can fi nd the problem

Slide 154

Slide 154 text

> describe "things" do it "valid" do expect(the).to eq(unexpected) end it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything It starts at the same place

Slide 155

Slide 155 text

> describe "things" do it "valid" do end it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything Now it needs to decide where to expand, it looks down and sees an `end`

Slide 156

Slide 156 text

> describe "things" do it "valid" do end it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything dead_end knows if it removes this end, it might trigger a false positive, so it searches up for keywords. It fi nds...

Slide 157

Slide 157 text

> describe "things" do it "valid" do end it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything This do keyword

Slide 158

Slide 158 text

> describe "things" do it "forgot a keyword -> " expect(brooklyn).to eq(99) end end Gotcha: Indentation isn't everything They are both safely removed

Slide 159

Slide 159 text

> file: /private/tmp/scratch.rb simplified: 1 describe "things" do ❯ 6 it "forgot a keyword -> " ❯ 8 end 9 end Gotcha: Indentation isn't everything The algorithm continues until the problem is found

Slide 160

Slide 160 text

Gotcha #2 Even with correct indentation, removing the wrong line can show false positives

Slide 161

Slide 161 text

> class User puts "hello" end def schneems User. where(name: 'schneems'). first end end Gotcha: Removing the wrong line Our syntax error is on these lines.

Slide 162

Slide 162 text

> class User puts "hello" end def schneems User. where(name: 'schneems'). first end end Gotcha: Removing the wrong line We are missing a whole method de fi nition line

Slide 163

Slide 163 text

> class User puts "hello" end def schneems User. where(name: 'schneems'). first end end Gotcha: Removing the wrong line If we tried our indentation comment out strategy we would start at the largest indentation

Slide 164

Slide 164 text

> class User puts "hello" end def schneems User. where(name: 'schneems'). first end end Gotcha: Removing the wrong line here

Slide 165

Slide 165 text

> class User puts "hello" end def schneems User. where(name: 'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ And then dead_end would stop looking because the document is now parsable

Slide 166

Slide 166 text

WAT WAT on earth? Let's look at it a little di ff erently.

Slide 167

Slide 167 text

> class User puts "hello" end def schneems User. where(name: 'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ It will help if we reformat it to show what Ruby thinks you're trying to do. It thinks

Slide 168

Slide 168 text

> class User puts "hello" end def schneems User. where(name: 'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ That this user class

Slide 169

Slide 169 text

> class User puts "hello" end def schneems User. where(name: 'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ Is matched with the fi rst end.

Slide 170

Slide 170 text

> class User puts "hello" end def schneems User. where(name: 'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ Is matched with the fi rst end.

Slide 171

Slide 171 text

> class User puts "hello" end def schneems User. where(name: 'schneems'). end end Gotcha: Removing the wrong line 📕 ✅ It thinks that this end, is actually a method due to the trailing dot

Slide 172

Slide 172 text

> class User puts "hello" end def schneems User. where(name: 'schneems').end end Gotcha: Removing the wrong line 📕 ✅ Finally it thinks the last end matches with our method de fi nition

Slide 173

Slide 173 text

> class User puts "hello" end def schneems User. where(name: 'schneems').end end Gotcha: Removing the wrong line 📕 ✅ Finally it thinks the this method de fi nition

Slide 174

Slide 174 text

> class User puts "hello" end def schneems User. where(name: 'schneems').end end Gotcha: Removing the wrong line 📕 ✅ 🚫 Matches with this last end. It's a problem because none of this is what the user intended and we didn't highlight the source of the problems

Slide 175

Slide 175 text

WAT But wait there's more

Slide 176

Slide 176 text

> source = <<~EOM Hello World From A heredoc EOM Gotcha: Removing the wrong line Heredocs cause problems

Slide 177

Slide 177 text

> source = <<~EOM Hello World From A heredoc EOM Gotcha: Removing the wrong line If you evaluate this line in isolation, it looks like a constant. But if you remove it

Slide 178

Slide 178 text

> source = <<~EOM Hello World From A heredoc Gotcha: Removing the wrong line 🚫 Then you've introduced a syntax error

Slide 179

Slide 179 text

WAT But wait there's more

Slide 180

Slide 180 text

> it "should" \ "handle trailing slash" do expect(the).to eq(unexpected) end Gotcha: Removing the wrong line Trailing slashes cause problems.

Slide 181

Slide 181 text

> it "should" \ "handle trailing slash" do expect(the).to eq(unexpected) end Gotcha: Removing the wrong line If you start in the middle

Slide 182

Slide 182 text

> it "should" \ "handle trailing slash" do end Gotcha: Removing the wrong line Now expand out using lex detection

Slide 183

Slide 183 text

> it "should" \ "handle trailing slash" do end Gotcha: Removing the wrong line Find the end

Slide 184

Slide 184 text

> it "should" \ "handle trailing slash" do end Gotcha: Removing the wrong line Find the matching keyword.

Slide 185

Slide 185 text

> it "should" \ Gotcha: Removing the wrong line 🚫 Now remove them. This is not valid ruby

Slide 186

Slide 186 text

🙇 What

Slide 187

Slide 187 text

🙇 🙇 To

Slide 188

Slide 188 text

🙇 🙇 🙇 do

Slide 189

Slide 189 text

source = <<~EOM User. where(name: 'schneems'). first EOM Ripper.lex(source) # => [[[1, 0], :on_const, "User", CMDARG], [[1, 4], :on_period, ".", DOT], [[1, 5], :on_ignored_nl, "\n", DOT], [[2, 0], :on_sp, " ", DOT], [[2, 2], :on_ident, "where", ARG], [[2, 7], :on_lparen, "(", BEG|LABEL], [[2, 8], :on_label, "name:", ARG|LABELED], [[2, 13], :on_sp, " ", ARG|LABELED], We can use the lexer to detect these speci fi c edge cases. Then we can use that information to combine all logically consecutive lines together so there's no way to remove one without removing all of them

Slide 190

Slide 190 text

> it "should" \ "handle trailing slash" do expect(the).to eq(unexpected) end Gotcha: Removing the wrong line It's hard to show visually, but it essentially would looks like re-writing source code. This trailing slash would become

Slide 191

Slide 191 text

> it "should" \ "handle trailing slash" do expect(the).to eq(unexpected) end Gotcha: Removing the wrong line

Slide 192

Slide 192 text

> class User puts "hello" end def schneems User. where(name: 'schneems'). first end end Gotcha: Removing the wrong line Our method chain would become

Slide 193

Slide 193 text

> class User puts "hello" end def schneems User.where(name: 'schneems').first end end Gotcha: Removing the wrong line Our method chain would become

Slide 194

Slide 194 text

> source = <<~EOM Hello World From A heredoc EOM Gotcha: Removing the wrong line Heredocs cause problems

Slide 195

Slide 195 text

> source = <<~EOM Hello\nWorld\nFrom\nA\n\nheredoc\nEOM Gotcha: Removing the wrong line

Slide 196

Slide 196 text

🎉 With all the problem lines joined together, they can now be safely evaluated and removed without accidentally introducing new syntax errors

Slide 197

Slide 197 text

Gotcha #3 Even with joining lines, even with lexically aware search, even with the perfect algorithm there is one problem we cant' avoid

Slide 198

Slide 198 text

ambiguity Ambiguity. Yes, ambiguity is the comic sans of source code parsing. What do I mean by ambiguity?

Slide 199

Slide 199 text

> class Dog def sit puts "no" end def eat puts "munch" end def bark puts "woof" end Gotcha: ambiguity This code has a syntax error

Slide 200

Slide 200 text

> class Dog def sit puts "no" end def eat puts "munch" end def bark puts "woof" end Gotcha: ambiguity It is ambiguous what the coder was trying to do with our current search algorithm. Let's zoom in

Slide 201

Slide 201 text

> class Dog def bark puts "woof" end Gotcha: ambiguity

Slide 202

Slide 202 text

> class Dog def bark puts "woof" end Gotcha: ambiguity Starting from the middle, if we use lexical expansion we get

Slide 203

Slide 203 text

> class Dog def bark puts "woof" end Gotcha: ambiguity This match

Slide 204

Slide 204 text

> class Dog Gotcha: ambiguity When it's removed there's still a syntax error

Slide 205

Slide 205 text

> class Dog Gotcha: ambiguity Highlight

Slide 206

Slide 206 text

> Gotcha: ambiguity 📕 ✅ Remove It parses, yay

Slide 207

Slide 207 text

> ❯ 1 class Dog Gotcha: ambiguity 🚫 📕 ✅ But this clearly isn't our problem

Slide 208

Slide 208 text

🙇 To produce good results with invalid indentation it means we must make "bad" suggestions. There's a logical inverse of this problem as well.

Slide 209

Slide 209 text

> class Dog puts "no" end def eat puts "munch" end def bark puts "woof" end end Gotcha: ambiguity This code has a syntax error, but this time it's at the top.

Slide 210

Slide 210 text

> class Dog puts "no" end def eat puts "munch" end def bark puts "woof" end end Gotcha: ambiguity Let's focus

Slide 211

Slide 211 text

> class Dog puts "no" end end Gotcha: ambiguity

Slide 212

Slide 212 text

> class Dog puts "no" end end Gotcha: ambiguity Does this problem look familiar? Here's the code from before

Slide 213

Slide 213 text

> class Dog puts "no" end end Gotcha: ambiguity class Dog def bark puts "woof" end Before we were missing an end

Slide 214

Slide 214 text

> class Dog puts "no" end end Gotcha: ambiguity class Dog def bark puts "woof" end Before we were missing an end

Slide 215

Slide 215 text

> class Dog puts "no" end end Gotcha: ambiguity class Dog def bark puts "woof" end This new case is missing a keyword

Slide 216

Slide 216 text

> class Dog puts "no" end end Gotcha: ambiguity class Dog def bark puts "woof" end This new case is missing a keyword

Slide 217

Slide 217 text

> class Dog puts "no" end end Gotcha: ambiguity class Dog def bark puts "woof" end Where does dead end tell us the problem is?

Slide 218

Slide 218 text

> class Dog puts "no" end end class Dog def bark puts "woof" end Gotcha: ambiguity ❯ 10 end ❯ 1 class Dog 🚫 Here's the result. Neither make good suggestion

Slide 219

Slide 219 text

🙇 If we re-write our rules to let indentation take precedence over lexical keywords then it would make other examples fail. It's impossible to satisfy all search criteria. What if we...

Slide 220

Slide 220 text

😇 Don't. Because we know this ambiguity exists, we can compensate for it after the search is done

Slide 221

Slide 221 text

> ❯ 1 class Dog Gotcha: ambiguity Even though this is all the user might see, `dead_end` still has all the contents that are hidden. It didn't just match a single line, it matched many lines, but hid most of them

Slide 222

Slide 222 text

> ❯ 1 class Dog ❯ 2 def sit ❯ 3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity If we un-hid the rest of them, this is what it would look like. We can detect this case when our match is only one line and contains an end, then we can go back through our results

Slide 223

Slide 223 text

> ❯ 1 class Dog ❯ 2 def sit ❯ 3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity un-hide the matched end

Slide 224

Slide 224 text

> ❯ 1 class Dog ❯ 2 def sit ❯ 3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity Work backwards and fi nd an unmatched keyword

Slide 225

Slide 225 text

> ❯ 1 class Dog ❯ 10 def bark ❯ 12 end Gotcha: ambiguity Then we show the user all extra context to the user and they can resolve the problem visually

Slide 226

Slide 226 text

Gotcha #4 Syntax errors aren't always alone, sometimes they have friends. A document can contain multiple syntax errors

Slide 227

Slide 227 text

> end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors Here' there's multiple problems

Slide 228

Slide 228 text

> end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors This end matches nothing

Slide 229

Slide 229 text

> end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors We are missing a square bracket

Slide 230

Slide 230 text

> end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors We're missing all sorts of stu ff here

Slide 231

Slide 231 text

> end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors Right here, that's code is fi ne

Slide 232

Slide 232 text

> end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors That code is fi ne too

Slide 233

Slide 233 text

> end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors What we don't want to happen is for our search to start at one side

Slide 234

Slide 234 text

> end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors And continue until it captures everything, even the good code

Slide 235

Slide 235 text

> end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors 🚫 This isn't what we want

Slide 236

Slide 236 text

🙇 What can we do about it? We can modify our algorithm to hold and fi nd multiple errors.

Slide 237

Slide 237 text

> end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors Instead of searching from one side to the other we start in the outside and search in

Slide 238

Slide 238 text

> end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1

Slide 239

Slide 239 text

> end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2

Slide 240

Slide 240 text

> end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2

Slide 241

Slide 241 text

> end def hello puts "world" end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2

Slide 242

Slide 242 text

> end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2

Slide 243

Slide 243 text

> end def world puts "hello" end def foo foo = 1,2,3,4] end def bar bar = {1,2,3,4] end Gotcha: multiple syntax errors #1 #2 #3

Slide 244

Slide 244 text

🙇 🙇 🙇 🙇 Those are our main gotchas. Let's move on

Slide 245

Slide 245 text

AI Now you know the gotchas. I want to start moving a little closer to the code. Who knows what AI is? Raise your hand, or in chat type "me"

Slide 246

Slide 246 text

Artificial Intelligence Algorithm - Arti fi cial Intelligence is a fancy way of saying "Algorithm" [wait for dialog] "Come on no you're not"

Slide 247

Slide 247 text

Artificial Intelligence Algorithm More speci fi cally, AI is a goal seeking algorithm

Slide 248

Slide 248 text

Pathfinding Credit: Factorio 'New pathfinding algorithm' - A common example of AI is path fi nding. You want to get from point A to point B

Slide 249

Slide 249 text

$ gem install dead_end Dead end uses a search algorithm to fi nd the problem code. Which uses a variation on uniform cost search

Slide 250

Slide 250 text

Credit @redblobgames: Introduction to the A-star Algorithm - You need a way to break down the problem into discrete actions that either get you closer or further from the goal. In path fi nding this would be deciding what turns to make and what roads to drive on.

Slide 251

Slide 251 text

Credit @redblobgames: Introduction to the A-star Algorithm - The algorithm dead_end uses is also sometimes called dijkstra's algorithm because he invented uniform cost search.

Slide 252

Slide 252 text

Credit @redblobgames: Introduction to the A-star Algorithm I highly recommend this interactive page from redblob games as an introduction to search

Slide 253

Slide 253 text

Credit @redblobgames: Introduction to the A-star Algorithm A search algorithm Is good for when you know what goal you want but you're not quite sure how to get there.

Slide 254

Slide 254 text

> while b != 0 if a > b a = a - b else b = b - a end end return a while b != 0 if a > b a = a - b else b = b - a Before we saw how code is represented by abstract syntax trees.

Slide 255

Slide 255 text

Tree  Search  BOTVINNIK, M.M. (1984). "Computers in Chess Solving Inexact Search Problems" That was an important realization for me, as most search problems are represented as a tree or a graph. Here's a quick example of a famous chess problem

Slide 256

Slide 256 text

Tree  Search  BOTVINNIK, M.M. (1984). "Computers in Chess Solving Inexact Search Problems" If you have an exact tree you can walk the tree

Slide 257

Slide 257 text

require "ripper" source = <<~EOM module Cutlass defcall end end EOM Ripper.new(source).tap(&:parse).error? # => true In our case, we have bits of source code

Slide 258

Slide 258 text

require "ripper" source = <<~EOM module Cutlass # defcall end end EOM Ripper.new(source).tap(&:parse).error? # => true I found that if I could give my program rules about how to create logical code blocks

Slide 259

Slide 259 text

require "ripper" source = <<~EOM module Cutlass # defcall # end end EOM Ripper.new(source).tap(&:parse).error? # => false Then using ripper we can check a document without their presence

Slide 260

Slide 260 text

# defcall # end Ripper.new(source).tap(&:parse).error? # => false And complete our search

Slide 261

Slide 261 text

Internals We'll now take a closer look at some dead_end algorithms

Slide 262

Slide 262 text

🎂 If dead end was a cake here's the main steps to make it

Slide 263

Slide 263 text

We start out with messy code

Slide 264

Slide 264 text

Dead end then transforms that code into a tidier input

Slide 265

Slide 265 text

"Time to cook" by Robbert van der Steeg is licensed under CC BY-SA 2.0 It gives that input to our search algorithm to fi nd the syntax errors.

Slide 266

Slide 266 text

Once the syntax errors are found, the output is decorated with more context and given back to the user

Slide 267

Slide 267 text

🎂 Let's start at the beginning

Slide 268

Slide 268 text

Your app has a syntax error

Slide 269

Slide 269 text

module Kernel module_function def require(file) dead_end_original_require(file) rescue SyntaxError => e DeadEnd.handle_error(e) end end dead_end/auto.rb Like all good Ruby libraries dead end uses monkey patching . It hooks into require and friends. When a syntax error is raised, it's passed to dead_end

Slide 270

Slide 270 text

# ... Timeout.timeout(timeout) do record_dir ||= ENV["DEBUG"] ? "tmp" : nil search = CodeSearch.new(source, record_dir: record_dir).call end # ... dead_end/internals.rb The source code that caused the syntax error is read to disk and then passed to a search object.

Slide 271

Slide 271 text

First we need to clean it up

Slide 272

Slide 272 text

clean_document.rb # frozen_string_literal: true module DeadEnd # Parses and sanitizes source into a lexically aware document # # Internally the document is represented by an array with each # index containing a CodeLine correlating to a line from the source code. # # There are three main phases in the algorithm: # # 1. Sanitize/format input source # 2. Search for invalid blocks # 3. Format invalid blocks into something meaninful # # This class handles the first part. # # The reason this class exists is to format input source # for better/easier/cleaner exploration. Here's the clean_document class

Slide 273

Slide 273 text

clean_document.rb @source = source @document = CodeLine.from_source(@source) end # Call all of the document "cleaners" # and return self def call clean_sweep .join_trailing_slash! .join_consecutive! .join_heredoc! self end # Return an array of CodeLines in the # document def lines It handles several of the gotcha cases from before by transforming the source code

Slide 274

Slide 274 text

Slide 275

Slide 275 text

Slide 276

Slide 276 text

Slide 277

Slide 277 text

Slide 278

Slide 278 text

"Time to cook" by Robbert van der Steeg is licensed under CC BY-SA 2.0 Next up we feed that to our search class

Slide 279

Slide 279 text

code_search.rb # Searches code for a syntax error # # The bulk of the heavy lifting is done in: # # - CodeFrontier (Holds information for generating blocks and determining if we can stop searching) # - ParseBlocksFromLine (Creates blocks into the frontier) # - BlockExpand (Expands existing blocks to search more code # # ## Syntax error detection # # When the frontier holds the syntax error, we can stop searching # # search = CodeSearch.new(<<~EOM) # def dog # def lol # end # EOM Here's the CodeSearch class

Slide 280

Slide 280 text

Slide 281

Slide 281 text

end # Main search loop def call until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self code_search.rb Under every great AI algorithm is a while loop. There's a class called the frontier. It holds all the code blocks that we've generated from the source code.

Slide 282

Slide 282 text

end # Main search loop def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } code_search.rb 📕 ✅ 📕 ❌ The frontier is responsible for checking the exit condition, here's that logic

Slide 283

Slide 283 text

> 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename ❯ 5 ❯ 16 ❯ 35 ❯ 36 def filename ❯ 37 ❯ 46 63 end 64 end 📕 ❌ If the highlighted lines are on the frontier

Slide 284

Slide 284 text

> 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename 63 end 64 end 📕 ✅ Then the entire document without those lines are checked against the parser

Slide 285

Slide 285 text

# ... end module DeadEnd # ... def self.invalid?(source) source = source.join if source.is_a?(Array) source = source.to_s Ripper.new(source).tap(&:parse).error? end # ... end dead_end.rb And here's where the parser is called. Remember Ripper from before?

Slide 286

Slide 286 text

> 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s 58 string << "\e[0m" 59 string 60 end 61 end.join 62 end 63 end 64 end Let's walk through the logic of searching this same document we saw before. From the beginning

Slide 287

Slide 287 text

end # Main search loop def call until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self end code_search.rb Since we start with an invalid source code, the frontier is empty so this is false

Slide 288

Slide 288 text

end # Main search loop def call until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self end code_search.rb There's two modes of exploration. We can add new things onto the frontier, or we can expand an existing block o ff of the frontier. When do we expand?

Slide 289

Slide 289 text

def expand? return false if @frontier.empty? # ... # Expand all blocks before moving to unvisited lines frontier_indent >= unvisited_indent end code_frontier.rb We can't expand a block if there's nothing there.

Slide 290

Slide 290 text

def expand? return false if @frontier.empty? # ... # Expand all blocks before moving to unvisited lines frontier_indent >= unvisited_indent end code_frontier.rb We're also using indentation to decide when to expand. Essentially we want to explore the largest indentations fi rst.

Slide 291

Slide 291 text

Slide 292

Slide 292 text

Slide 293

Slide 293 text

# Parses the most indented lines into blocks that are marked # and added to the frontier def visit_new_blocks max_indent = frontier.next_indent_line&.indent while (line = frontier.next_indent_line) && (line.indent == max_indent) @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block| record(block: block, name: "add") block.mark_invisible if block.valid? push(block, name: "add") end end end # Given an already existing block in the frontier, expand it to see # if it contains our invalid syntax code_search.rb The code here is looking at the largest indentation, starting at the bottom and iterating up.

Slide 294

Slide 294 text

> 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before

Slide 295

Slide 295 text

> 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before

Slide 296

Slide 296 text

> 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before

Slide 297

Slide 297 text

> 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before

Slide 298

Slide 298 text

> 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before

Slide 299

Slide 299 text

> 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else ❯ 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before

Slide 300

Slide 300 text

Slide 301

Slide 301 text

# Parses the most indented lines into blocks that are marked # and added to the frontier def visit_new_blocks max_indent = frontier.next_indent_line&.indent while (line = frontier.next_indent_line) && (line.indent == max_indent) @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block| record(block: block, name: "add") block.mark_invisible if block.valid? push(block, name: "add") end end end # Given an already existing block in the frontier, expand it to see code_search.rb If the block that it found is valid ruby code, it's essentially commented out.

Slide 302

Slide 302 text

> 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 52 "#{number.to_s}#{line}" 53 else ❯ 54 string = String.new ❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics ❯ 56 string << "#{number.to_s} " ❯ 57 string << line.to_s ❯ 58 string << "\e[0m" ❯ 59 string 60 end 61 end.join 62 end 63 end 64 end We saw this before

Slide 303

Slide 303 text

Slide 304

Slide 304 text

# Parses the most indented lines into blocks that are marked # and added to the frontier def visit_new_blocks max_indent = frontier.next_indent_line&.indent while (line = frontier.next_indent_line) && (line.indent == max_indent) @parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block| record(block: block, name: "add") block.mark_invisible if block.valid? push(block, name: "add") end end end # Given an already existing block in the frontier, expand it to see code_search.rb The code block is then pushed onto the frontier and the loop continues

Slide 305

Slide 305 text

sweep(block: block, name: "comments") end # Main search loop def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) code_search.rb We've still not found the problem we will continue

Slide 306

Slide 306 text

Slide 307

Slide 307 text

Slide 308

Slide 308 text

> 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? ❯ 52 "#{number.to_s}#{line}" 53 else 60 end 61 end.join 62 end 63 end 64 end

Slide 309

Slide 309 text

> 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? 53 else 60 end 61 end.join 62 end 63 end 64 end Boop

Slide 310

Slide 310 text

Slide 311

Slide 311 text

code_search.rb sweep(block: block, name: "comments") end # Main search loop def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) We call our expansion phase

Slide 312

Slide 312 text

end end end # Given an already existing block in the frontier, expand it to see # if it contains our invalid syntax def expand_invalid_block block = frontier.pop return unless block record(block: block, name: "pop") block = @block_expand.call(block) push(block, name: "expand") end def sweep_heredocs HeredocBlockParse.new( source: @source, code_search.rb Here's our expand code

Slide 313

Slide 313 text

Slide 314

Slide 314 text

Slide 315

Slide 315 text

# puts "wow" # end # class BlockExpand def initialize(code_lines: ) @code_lines = code_lines end def call(block) if (next_block = expand_neighbors(block, grab_empty: true)) return next_block end expand_indent(block) end def expand_indent(block) block = AroundBlockScan.new(code_lines: @code_lines, block: block) block_expand.rb It can either expand to capture more lines at the same indentation, aka neighbors. Or when those are exhausted it can expand out an indentation.

Slide 316

Slide 316 text

def expand_indent(block) block = AroundBlockScan.new(code_lines: @code_lines, block: block) .skip(:hidden?) .stop_after_kw .scan_adjacent_indent .code_block end def expand_neighbors(block, grab_empty: true) scan = AroundBlockScan.new(code_lines: @code_lines, block: block) .skip(:hidden?) .stop_after_kw .scan_neighbors # Slurp up empties if grab_empty scan = AroundBlockScan.new(code_lines: @code_lines, block: scan.code_block) block_expand.rb Both methods use this class AroundBlockScan which acts as a DSL for scanning up and down. This is where we give our algorithm rules about how to build blocks using indentation and lexical data

Slide 317

Slide 317 text

> 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 51 if line.empty? ❯ 52 "#{number.to_s}#{line}" 53 else 60 end 61 end.join 62 end 63 end 64 end Previously we captured this line, and put it on the frontier.

Slide 318

Slide 318 text

> 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) ❯ 51 if line.empty? ❯ 53 else ❯ 60 end 61 end.join 62 end 63 end 64 end When it's popped and expanded it looks around and fi nds a matched set of keywords using the scanner.

Slide 319

Slide 319 text

> 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename 42 string << code_with_lines 43 string << "```\n" 44 string 45 end 46 47 def code_with_lines 48 @code_lines.map do |line| 49 next if line.hidden? 50 number = line.line_number.to_s.rjust(@digit_count) 61 end.join 62 end 63 end 64 end The code is valid so it's hidden. The expanded block is put back on the frontier.

Slide 320

Slide 320 text

Slide 321

Slide 321 text

Slide 322

Slide 322 text

def call sweep_heredocs sweep_comments until frontier.holds_all_syntax_errors? @tick += 1 if frontier.expand? expand_invalid_block else visit_new_blocks end end @invalid_blocks.concat(frontier.detect_invalid_blocks ) @invalid_blocks.sort_by! {|block| block.starts_at } self end end code_search.rb The frontier can hold more than one syntax error.

Slide 323

Slide 323 text

# Example: # # combination([:a, :b, :c, :d]) # # => [[:a], [:b], [:c], [:d], # [:a, :b], [:a, :c], [:a, :d], # [:b, :c], [:b, :d], [:c, :d], # [:a, :b, :c], [:a, :b, :d], [:a, :c, :d], # [:b, :c, :d], # [:a, :b, :c, :d]] def self.combination(array) guesses = [] 1.upto(array.length).each do |size| guesses.concat(array.combination(size).to_a) end guesses end # Given that we know our syntax error exists somewhere in our frontier, we want to find code_frontier.rb We want to iterate over all of the invalid blocks stored in the frontier to fi nd the shortest possible order that gives us a valid document.

Slide 324

Slide 324 text

# Example: # # combination([:a, :b, :c, :d]) # # => [[:a], [:b], [:c], [:d], # [:a, :b], [:a, :c], [:a, :d], # [:b, :c], [:b, :d], [:c, :d], # [:a, :b, :c], [:a, :b, :d], [:a, :c, :d], # [:b, :c, :d], # [:a, :b, :c, :d]] def self.combination(array) guesses = [] 1.upto(array.length).each do |size| guesses.concat(array.combination(size).to_a) end guesses end # Given that we know our syntax error exists somewhere in our frontier, we want to find code_frontier.rb To do that we create a combination of all possible blocks

Slide 325

Slide 325 text

end # Given that we know our syntax error exists somewhere in our frontier, we want to find # the smallest possible set of blocks that contain all the syntax errors def detect_invalid_blocks self.class.combination(@frontier.select(&:invalid?)).detect do |block_array| holds_all_syntax_errors?(block_array) end || [] end end end code_frontier.rb Then one at a time we iterate through all the combinations

Slide 326

Slide 326 text

guesses end # Given that we know our syntax error exists somewhere in our frontier, we want to find # the smallest possible set of blocks that contain all the syntax errors def detect_invalid_blocks self.class.combination(@frontier.select(&:invalid?)).detect do |block_array| holds_all_syntax_errors?(block_array) end || [] end end end code_frontier.rb And check each one to see if we hold all the syntax errors. The smallest successful combination is returned

Slide 327

Slide 327 text

Timeout.timeout(timeout) do record_dir ||= ENV["DEBUG"] ? "tmp" : nil search = CodeSearch.new(source, record_dir: record_dir).call end blocks = search.invalid_blocks DisplayInvalidBlocks.new( blocks: blocks, filename: filename, terminal: terminal, code_lines: search.code_lines, invalid_obj: invalid_type(source), io: io ).call dead_end/internals.rb Here's where our search started.

Slide 328

Slide 328 text

Slide 329

Slide 329 text

Now we want to decorate the output

Slide 330

Slide 330 text

Slide 331

Slide 331 text

lines = CaptureCodeContext.new( blocks: @blocks, code_lines: @code_lines ).call DisplayCodeWithLineNumbers.new( lines: lines, terminal: @terminal, highlight_lines: @invalid_lines ).call display_invalid_blocks.rb Here's where we take care of the last of our gotchas

Slide 332

Slide 332 text

lines = CaptureCodeContext.new( blocks: @blocks, code_lines: @code_lines ).call DisplayCodeWithLineNumbers.new( lines: lines, terminal: @terminal, highlight_lines: @invalid_lines ).call display_invalid_blocks.rb This capture code context handles the ambiguities we talked about before

Slide 333

Slide 333 text

> ❯ 1 class Dog Gotcha: ambiguity 🚫 It knows about our ambiguous edge cases that we talked about before

Slide 334

Slide 334 text

> ❯ 1 class Dog ❯ 2 def sit ❯ 3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity It goes back through the hidden code

Slide 335

Slide 335 text

> ❯ 1 class Dog ❯ 2 def sit ❯ 3 puts "no" ❯ 4 end ❯ 5 ❯ 6 def eat ❯ 7 puts "munch" ❯ 8 end ❯ 9 ❯ 10 def bark ❯ 11 puts "woof" ❯ 12 end Gotcha: ambiguity

Slide 336

Slide 336 text

Slide 337

Slide 337 text

> ❯ 1 class Dog ❯ 10 def bark ❯ 12 end Gotcha: ambiguity And transforms it

Slide 338

Slide 338 text

🎂 Piece of cake

Slide 339

Slide 339 text

> DeadEnd: Missing ènd` detected This code has a missing ènd`. Ensure that all syntax keywords (`def`, `do`, etc.) have a matching ènd`. file: /Users/rschneeman/Documents/projects/dead_end/spec/fixtures/this_project_extra_def.rb.txt simplified: 1 module SyntaxErrorSearch 3 class DisplayInvalidBlocks 4 attr_reader :filename ❯ 36 def filename ❯ 38 def code_with_filename ❯ 45 end 63 end 64 end After all three steps, cleaning, searching, formatting here's the output

Slide 340

Slide 340 text

dead_end That is dead end in a nutshell *laugh*

Slide 341

Slide 341 text

Gem `error_highlight` in Ruby 3.1 Beyond dead_end there's an amazing gem being added to Ruby 3.1 called error_highlight. It shows you which method got a no method error. I love this introduction

Slide 342

Slide 342 text

Gem `error_highlight` in Ruby 3.1 The error highlight gem was created by you-ssssss-kay. He goes by mam-eh on GitHub and gave an amazing talk at a conference I used to run called keep ruby weird.

Slide 343

Slide 343 text

Rust v. Ruby With error highlight and dead end I also want to touch on community values.

Slide 344

Slide 344 text

Rust v. Ruby I've been writing Rust code and their community takes a very aggressive stance towards error messages. It's not enough for them to say there's a problem. They also want to accurately suggest how to fi x the problem

Slide 345

Slide 345 text

Rust v. Ruby I opened this issue against rustlang and in under a month it was addressed to improve the error output.

Slide 346

Slide 346 text

Rust v. Ruby In general the Rust community treats a user experience problem as a bug and error output is a fi rst class feature. If our community wanted we could invest more in errors

Slide 347

Slide 347 text

Dead end isn't that solution, but it's a start. You can add the dead_end gem to your project to try it today and give me feedback on what works and what doesn't.

Slide 348

Slide 348 text

📕 HowTo Open Source .dev You can pre-order my book on open source contribution by visiting how to open source dot dev

Slide 349

Slide 349 text

You can sign up to triage open source issues at code triage dot com

Slide 350

Slide 350 text

Lexing Parsing Syntax SyntaxErrors AI Pathfinding and goal seeking Today we talked about lexing, parsing, syntax syntax errors, AI, path fi nding and goal seeking

Slide 351

Slide 351 text

But, the technical pieces are not what I want you to remember.

Slide 352

Slide 352 text

- The important part is that everyone sitting in this room is the future of developer tooling. Programming is inherently di ffi cult, but it doesn't mean our tools can't help us.

Slide 353

Slide 353 text

One of the best ways to judge a system is to see how it fails. When we can give care and grace and beauty to our failure modes then we can create experiences that delight us. Experiences that teach us. Experiences that elevate our code and our consciousness.

Slide 354

Slide 354 text

- A syntax error doesn't have to be an ending...it can be a beginning. To a beautiful programming story

Slide 355

Slide 355 text

How To Open Source .dev - My name is Richard Schneeman - I'm writing a book at how to open source dot dev - I created CodeTriage.com the easiest way to get started contributing to OpenSource - - Wait for "bye bye" - Zoom wave everyone