[Kaigi] Beware the Dead End

Beware
the Dreaded
Dead End!!!
By @schneems

View Slide

View Slide

Beware
the Dreaded
Dead End!!!
By @schneems

View Slide

>
# <= routes.rb:121: syntax error, unexpected end-of-input, expecting `end’
Rails.application.routes.draw do

constraints -> { Rails.application.config.non_production } do

namespace :foo do

resource :bar

end

end


namespace :bar do

resource :baz

end

end

namespace :admin do

resource :session

match "/foobar(*path)", via: :all, to: redirect { |_params, req|

uri = URI(req.path.gsub("foobar", "foobaz"))

uri.query = req.query_string.presence

uri.to_s

}

end

View Slide

>
Rails.application.routes.draw do


namespace :foo do

resource :bar

end

end


namespace :bar do

resource :baz

end

end

namespace :admin do

resource :session

match "/foobar(*path)", via: :all, to: redirect { |_params, req|

uri = URI(req.path.gsub("foobar", "foobaz"))

uri.query = req.query_string.presence

uri.to_s

}

end # <= routes.rb:121: syntax error, unexpected end-of-input, expecting `end’

View Slide

>
RSpec.describe Cutlass::BashResult do

it "preserves stdout, stderr, and status"

stdout = SecureRandom.hex(16)

stderr = SecureRandom.hex(16)

status = 0

result = BashResult.new(

stdout: stdout,

stderr: stderr,

status: status

)

expect(result.stdout).to eq(stdout)

expect(result.stderr).to eq(stderr)

expect(result.status).to eq(status)

end

end # <= result_spec:19 syntax error, unexpected end-of-input, expecting `end’

View Slide

🤔

View Slide

> 3 module Cutlass

4 RSpec.describe Cutlass::BashResult do

❯ 5 it "preserves stdout, stderr, and status"

❯ 19 end

21 it "success?" do

52 end

53 end

54 end
This code has an unmatched `end`. Ensure that all
`end` lines in your code have a matching syntax
keyword (`def`, `do`, etc.)

View Slide

>

View Slide

>
# frozen_string_literal: true

module Cutlass

RSpec.describe Cutlass::BashResult do

it "preserves stdout, stderr, and status"

stdout = SecureRandom.hex(16)

stderr = SecureRandom.hex(16)

status = 0

result = BashResult.new(

stdout: stdout,

stderr: stderr,

status: status

)

expect(result.stdout).to eq(stdout)

expect(result.stderr).to eq(stderr)

expect(result.status).to eq(status)

end

end

end # <= result_spec:19 syntax error, unexpected end-of-input, expecting `end’

View Slide

>
3 module Cutlass



❯ 19 end

21 it "success?" do

52 end

53 end

54 end

View Slide

0
25
50
75
100
April May June July
(I did not do a study)

View Slide

(This is a lie)
-30
-15
0
15
30
45
60
April May June July August

View Slide

(Also a lie)
7%
8%
11%
11%
31%
32%

View Slide

🥰

View Slide

$ gem install dead_end

View Slide

>
3 module Cutlass



❯ 19 end

21 it "success?" do

52 end

53 end

54 end
DeadEnd: Unmatched `end` detected

View Slide

> 1 module Cutlass


❯ 3 it "preserves stdout, stderr, and status" do

5 it "success?" do

6 end

7 end

8 end
DeadEnd: Missing `end` detected

View Slide

>
1 class Cat

2 def to_json

❯ 3 hash = {

❯ 4 name: name,

❯ 5 fur_color: fur_color,

❯ 6 type: 'Cat',

❯ 7 better_than_dog: false

❯ 8 hash.to_json

9 end

10 end
DeadEnd: Unmatched `}` character detected

View Slide

>
1 class Animal

2 def self.ones_my_three_year_old_likes

❯ 3 array =

❯ 4 Dog.new,

❯ 5 Lion.new,

❯ 6 Tiger.new,

❯ 7 Aligator.new,

❯ 8 ]

10 return array

11 end

12 end
DeadEnd: Unmatched `[` detected

View Slide

> 1 class Cat

2 def meow

❯ 3 Animal.call do |a

❯ 5 end

6 end

7 end
DeadEnd: Unmatched `|` character detected

View Slide

> 1 class Cat

2 def meow

❯ 3 Animal.call do |a

❯ 5 end

6 end

7 end
DeadEnd: Unmatched `|` character detected
?
?
?

View Slide

$ gem install dead_end

View Slide

Syntax
errors

View Slide

Lexing &

Parsing

View Slide

AI

View Slide

dead_end

View Slide

Internals

View Slide

🤚

View Slide

@schneems

View Slide

View Slide

📕
How to
Open Source

View Slide

Heroku

View Slide

NoMemoryError

ScriptError

LoadError

SyntaxError

SecurityError

SignalException

KeyError

StandardError

ArgumentError

EncodingError

IOError

EOFError
I am Exceptional

View Slide

>
Dir.chhhdir("/tmp") do

puts `ls`

end

View Slide

>
Dir.chhhdir("/tmp") do

puts `ls`

end
# unde
fi
ned method `chhhdir' for

# Dir:Class (NoMethodError
)

# Did you mean? chdir

View Slide

>

View Slide

require 'bundler'

require 'date'

require 'optparse'

require 'ostruct'

require 'shellwords'

class Rexe

VERSION = '1.5.1'

PROJECT_URL = 'https://github.com/keithrbennett/rexe'

module Helpers

# Try executing code. If error raised, print message (but not stack trace) & exit -1.

def try

begin

View Slide

end

def bundler_run(&block)

# This used to be an unconditional call to with_clean_env but that method is now deprecated:

# [DEPRECATED] `Bundler.with_clean_env` has been deprecated in favor of `Bundler.with_unbundled_env`.

# If you instead want the environment before bundler was originally loaded,

# use `Bundler.with_original_env`

if Bundler.respond_to?(:with_unbundled_env)

Bundler.with_unbundled_env { block.call }

else

Bundler.with_clean_env { block.call }

end

end

bundler_run { Rexe::Main.new.call }

View Slide

if / end

unless / end

while / end

until / end

def / end

for / end

begin / end

class / end

module / end

do / end

View Slide

>
Dir.chdir("lol")

puts `ls`

end
☠☠☠

View Slide

>
Dir.chdir("lol")

puts `ls`

end
??

View Slide

>
Dir.chdir("/tmp")

Dir.chdir("lol") do

puts `ls`

end

View Slide

>
def bark

puts "woof"

end

bark()

# => "woof"

View Slide

>
def bark

puts "woof"

end

bark do

puts "not a syntax error"

end

# => "woof"

View Slide

>
expect(WebMock).to

have_requested(

:post,

"localhost:#{port}"

).

end
$ ruby -wc scratch.rb

Syntax OK

View Slide

📓

View Slide

Lexing &

Parsing

View Slide

"hello world"
Ripper.lex
>

View Slide

View Slide

"hello world"
Ripper.lex
[

[[1, 0], :on_tstring_beg, "\"", BEG],

[[1, 1], :on_tstring_content, "hello world", BEG],

[[1, 12], :on_tstring_end, "\"", END],

[[1, 13], :on_nl, "\n", BEG]

]
>

View Slide

[


[[1, 1], :on_tstring_content, "hello world", BEG],


[[1, 13], :on_nl, "\n", BEG]

]

View Slide

while b != 0

if a > b

a = a - b

else

b = b - a

end

end

return a
Ripper.parse

View Slide

while b != 0

# if a > b

a = a - b

else

b = b - a

end

end

return a
Ripper.parse

:(

View Slide

🤔

View Slide

while b != 0

# if a > b

a = a - b

else

b = b - a

end

end

return a
Ripper.parse

:(

View Slide

while b != 0

# if a > b

# a = a - b

# else

# b = b - a

# end

end

return a
Ripper.parse

:)

View Slide

AI

View Slide

Artificial Intelligence
Algorithm

View Slide

Pathfinding
Credit: Factorio 'New
pathfinding algorithm'

View Slide

Credit @redblobgames: Introduction to the A-star Algorithm

View Slide

Pathfinding
Credit: Factorio 'New
pathfinding algorithm'

View Slide

Tree 
Search 
"Reti Draw" Endgame

Chess problem by

Richard Reti

(1921)

View Slide

Tree 
Search 
BOTVINNIK, M.M.

(1984).

"Computers in Chess
Solving Inexact Search
Problems"

View Slide

Non-optimal
Play

View Slide

View Slide

Checkmate
Black

View Slide

Optimal
Draw
Play

View Slide

View Slide

Draw

View Slide

Tree 
Search 
BOTVINNIK, M.M.

(1984).

"Computers in Chess
Solving Inexact Search
Problems"

View Slide

AI ~ Search

View Slide

View Slide

while b != 0

if a > b

a = a - b

else

b = b - a

end

end

return a
Ripper.parse

View Slide

while b != 0

# if a > b

a = a - b

else

b = b - a

end

end

return a
Ripper.parse

View Slide

❌🌳💔

View Slide

>
Dir.chdir("/tmp")

Dir.chdir("lol")

end
do???
do???
puts `ls`

View Slide

>
Dir.chdir("/tmp")

Dir.chdir("lol")

end
do
puts `ls`🥰

View Slide

while b != 0

# if a > b

# a = a - b

# else

# b = b - a

# end

end

return a
Ripper.parse

View Slide

🌳
❌ 💔

View Slide

demo

View Slide

module SyntaxErrorSearch

# Used for formatting invalid blocks

class DisplayInvalidBlocks

attr_reader :filename

def initialize(block_array, io: $stderr, filename: nil)

@filename = filename

@io = io

@blocks = block_array

@lines = @blocks.map(&:lines).flatten

@digit_count = @lines.last.line_number.to_s.length

@code_lines = @blocks.first.code_lines

@invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

end

def call

64: syntax error, unexpected end-of-input,
expecting `end'
📕
❌

View Slide

51 if line.empty?

52 "#{number.to_s}#{line}"

53 else

54 string = String.new

55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics

56 string << "#{number.to_s} "

57 string << line.to_s

58 string << "\e[0m"

59 string

60 end

61 end.join

62 end

63 end

64 end

View Slide

50 number = line.line_number.to_s.rjust(@digit_count)

51 if line.empty?


53 else






❯ 59 string

60 end

61 end.join

62 end

63 end

64 end

View Slide


51 if line.empty?


53 else





❯ 58 string << "\e[0m"

❯ 59 string

60 end

61 end.join

62 end

63 end

64 end

View Slide


51 if line.empty?


53 else




❯ 57 string << line.to_s

❯ 58 string << "\e[0m"

❯ 59 string

60 end

61 end.join

62 end

63 end

64 end

View Slide


51 if line.empty?


53 else



❯ 56 string << "#{number.to_s} "


❯ 58 string << "\e[0m"

❯ 59 string

60 end

61 end.join

62 end

63 end

64 end

View Slide


51 if line.empty?


53 else


❯ 55 string << "\e[1;3m" if @invalid_line_hash[line] # Bold, italics



❯ 58 string << "\e[0m"

❯ 59 string

60 end

61 end.join

62 end

63 end

64 end

View Slide


51 if line.empty?


53 else

❯ 54 string = String.new




❯ 58 string << "\e[0m"

❯ 59 string

60 end

61 end.join

62 end

63 end

64 end

View Slide


51 if line.empty?


53 else

60 end

61 end.join

62 end

63 end

64 end

📕
❌

View Slide

42 string << code_with_lines

43 string << "```\n"

44 string

45 end

46

47 def code_with_lines

48 @code_lines.map do |line|

49 next if line.hidden?


51 if line.empty?

❯ 52 "#{number.to_s}#{line}"

53 else

60 end

61 end.join

62 end

63 end

64 end

View Slide


43 string << "```\n"

44 string

45 end

46





51 if line.empty?

53 else

60 end

61 end.join

62 end

63 end

64 end

📕
❌

View Slide


43 string << "```\n"

44 string

45 end

46





❯ 51 if line.empty?

❯ 53 else

❯ 60 end

61 end.join

62 end

63 end

64 end

View Slide


43 string << "```\n"

44 string

45 end

46





61 end.join

62 end

63 end

64 end

📕
❌

View Slide


43 string << "```\n"

44 string

45 end

46



❯ 49 next if line.hidden?

❯ 50 number = line.line_number.to_s.rjust(@digit_count)

61 end.join

62 end

63 end

64 end

View Slide


43 string << "```\n"

44 string

45 end

46



61 end.join

62 end

63 end

64 end

📕
❌

View Slide


43 string << "```\n"

44 string

45 end

46


❯ 48 @code_lines.map do |line|

❯ 61 end.join

62 end

63 end

64 end

View Slide


43 string << "```\n"

44 string

45 end

46


62 end

63 end

64 end

📕
❌

View Slide


43 string << "```\n"

44 string

45 end

46

❯ 47 def code_with_lines

❯ 62 end

63 end

64 end

View Slide

40 string << "```\n"

41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename


43 string << "```\n"

44 string

45 end

46

63 end

64 end

📕
❌

View Slide

13

14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

15 end

16

35

36 def filename

37

38 def code_with_filename

39 string = String.new("")

40 string << "```\n"



43 string << "```\n"

44 string

45 end

46

63 end

64 end

View Slide

13


15 end

16

35

36 def filename

37



40 string << "```\n"



43 string << "```\n"

❯ 44 string

45 end

46

63 end

64 end

View Slide

13


15 end

16

35

36 def filename

37



40 string << "```\n"



❯ 43 string << "```\n"

❯ 44 string

45 end

46

63 end

64 end

View Slide

13


15 end

16

35

36 def filename

37



40 string << "```\n"


❯ 42 string << code_with_lines

❯ 43 string << "```\n"

❯ 44 string

45 end

46

63 end

64 end

View Slide

13


15 end

16

35

36 def filename

37



40 string << "```\n"

❯ 41 string << "#".rjust(@digit_count) + " filename: #{filename}\n\n" if filename


❯ 43 string << "```\n"

❯ 44 string

45 end

46

63 end

64 end

View Slide

13


15 end

16

35

36 def filename

37



❯ 40 string << "```\n"



❯ 43 string << "```\n"

❯ 44 string

45 end

46

63 end

64 end

View Slide

13


15 end

16

35

36 def filename

37


❯ 39 string = String.new("")

❯ 40 string << "```\n"



❯ 43 string << "```\n"

❯ 44 string

45 end

46

63 end

64 end

View Slide

13


15 end

16

35

36 def filename

37


45 end

46

63 end

64 end

📕
❌

View Slide

1 module SyntaxErrorSearch

3 class DisplayInvalidBlocks

4 attr_reader :filename

5

6 def initialize(block_array, io: $stderr, filename: nil)

7 @filename = filename

8 @io = io

9 @blocks = block_array

10 @lines = @blocks.map(&:lines).flatten

11 @digit_count = @lines.last.line_number.to_s.length

12 @code_lines = @blocks.first.code_lines

13


15 end

16

35

36 def filename

37

View Slide




5



8 @io = io




12 @code_lines = @blocks.first.code_lines

13

❯ 14 @invalid_line_hash = @lines.each_with_object({}) {|line, h| h[line] = true}

15 end

16

35

36 def filename

37

View Slide




5



8 @io = io




❯ 12 @code_lines = @blocks.first.code_lines

❯ 13


15 end

16

35

36 def filename

View Slide




5



8 @io = io



❯ 11 @digit_count = @lines.last.line_number.to_s.length


❯ 13


15 end

16

35

36 def filename

View Slide




5



8 @io = io


❯ 10 @lines = @blocks.map(&:lines).flatten



❯ 13


15 end

16

35

36 def filename

View Slide




5



8 @io = io

❯ 9 @blocks = block_array




❯ 13


15 end

16

35

36 def filename

View Slide




5



❯ 8 @io = io





❯ 13


15 end

16

35

36 def filename

View Slide




5


❯ 7 @filename = filename

❯ 8 @io = io





❯ 13


15 end

16

35

36 def filename

View Slide




5


15 end

16

35

36 def filename

37


45 end

46

63 end

64 end

📕
❌

View Slide




5


15 end

16

35

36 def filename

37

❯ 38 def code_with_filename

❯ 45 end

46

63 end

64 end

View Slide




5


15 end

16

35

36 def filename

37

46

63 end

64 end

📕
❌

View Slide




5

❯ 6 def initialize(block_array, io: $stderr, filename: nil)

❯ 15 end

16

35

36 def filename

37

46

63 end

64 end

View Slide

📕
❌



5

16

35

36 def filename

37

46

63 end

64 end

View Slide




❯ 5

❯ 16

❯ 35

❯ 36 def filename

❯ 37

❯ 46

63 end

64 end

View Slide




63 end

64 end

📕
✅

View Slide

Syntax O
K

🥰🥰🥰



63 end

64 end

📕
✅

View Slide


This code has a missing `end`. Ensure that all

syntax keywords (`def`, `do`, etc.) have a matching `end`.

file: /Users/rschneeman/Documents/projects/dead_end/spec/fixtures/this_project_extra_def.rb.txt

simplified:




❯ 36 def filename


❯ 45 end

63 end

64 end

View Slide

Internals

View Slide

module Kernel

module_function

def require(file)

dead_end_original_require(file)

rescue SyntaxError => e

DeadEnd.handle_error(e)

end

end
dead_end/auto.rb

View Slide

# ...

Timeout.timeout(timeout) do

record_dir ||= ENV["DEBUG"] ? "tmp" : nil

search = CodeSearch.new(source, record_dir: record_dir).call

end

# ...
dead_end/internals.rb

View Slide

# ...




end

# ...

View Slide

#

# ## Syntax error detection

#

# When the frontier holds the syntax error, we can stop searching

#

# search = CodeSearch.new(<<~EOM)

# def dog

# def lol

# end

# EOM

#

# search.call

#

# search.invalid_blocks.map(&:to_s) # =>

# # => ["def lol\n"]

#

code_search.rb

View Slide

end

# Main search loop

def call

sweep_heredocs

sweep_comments

until frontier.holds_all_syntax_errors?

@tick += 1

if frontier.expand?

expand_invalid_block

else

visit_new_blocks

end

end

@invalid_blocks.concat(frontier.detect_invalid_blocks )

code_search.rb

View Slide

end

# Main search loop

def call

sweep_heredocs

sweep_comments


@tick += 1

if frontier.expand?


else

visit_new_blocks

end

end

code_search.rb

View Slide

end

# Main search loop

def call

sweep_heredocs

sweep_comments


@tick += 1

if frontier.expand?


else

visit_new_blocks

end

end

code_search.rb
📕
✅
📕
❌

View Slide

>



❯ 5

❯ 16

❯ 35

❯ 36 def filename

❯ 37

❯ 46

63 end

64 end

📕
❌

View Slide

>



63 end

64 end

📕
✅

View Slide

# ...

end

module DeadEnd

# ...

def self.invalid?(source)

source = source.join if source.is_a?(Array)

source = source.to_s

Ripper.new(source).tap(&:parse).error?

end

# ...

end
dead_end.rb

View Slide

>




51 if line.empty?


53 else






59 string

60 end

61 end.join

62 end

63 end

64 end

View Slide

end

# Main search loop

def call

sweep_heredocs

sweep_comments


@tick += 1

if frontier.expand?


else

visit_new_blocks

end

end

code_search.rb

View Slide

end

# Main search loop

def call

sweep_heredocs

sweep_comments


@tick += 1

if frontier.expand?


else

visit_new_blocks

end

end


code_search.rb

View Slide

def expand?

return false if @frontier.empty?

# ...

# Expand all blocks before moving to unvisited lines

frontier_indent >= unvisited_indent

end

code_frontier.rb

View Slide

end

# Main search loop

def call

sweep_heredocs

sweep_comments


@tick += 1

if frontier.expand?


else

visit_new_blocks

end

end


code_search.rb

View Slide

# Parses the most indented lines into blocks that are marked

# and added to the frontier

def visit_new_blocks

max_indent = frontier.next_indent_line&.indent

while (line = frontier.next_indent_line) && (line.indent == max_indent)

@parse_blocks_from_indent_line.each_neighbor_block(frontier.next_indent_line) do |block|

record(block: block, name: "add")

block.mark_invisible if block.valid?

push(block, name: "add")

end

end

end

code_search.rb

View Slide

>




51 if line.empty?


53 else






❯ 59 string

60 end

61 end.join

62 end

63 end

64 end

View Slide

>




51 if line.empty?


53 else





❯ 58 string << "\e[0m"

❯ 59 string

60 end

61 end.join

62 end

63 end

64 end

View Slide










end

end

end

code_search.rb

View Slide

>




51 if line.empty?


53 else





❯ 58 string << "\e[0m"

❯ 59 string

60 end

61 end.join

62 end

63 end

64 end

View Slide

>




51 if line.empty?


53 else

60 end

61 end.join

62 end

63 end

64 end

View Slide










end

end

end

code_search.rb

View Slide

sweep(block: block, name: "comments")

end

# Main search loop

def call

sweep_heredocs

sweep_comments


@tick += 1

if frontier.expand?


else

visit_new_blocks

end

end

code_search.rb

View Slide

>
43 string << "```\n"

44 string

45 end

46





51 if line.empty?


53 else

60 end

61 end.join

62 end

63 end

64 end

View Slide

>
43 string << "```\n"

44 string

45 end

46





51 if line.empty?

53 else

60 end

61 end.join

62 end

63 end

64 end

View Slide


end

# Main search loop

def call

sweep_heredocs

sweep_comments


@tick += 1

if frontier.expand?


else

visit_new_blocks

end

end

code_search.rb

View Slide

code_search.rb

end

# Main search loop

def call

sweep_heredocs

sweep_comments


@tick += 1

if frontier.expand?


else

visit_new_blocks

end

end

View Slide

end

end

# Given an already existing block in the frontier, expand it to see

# if it contains our invalid syntax

def expand_invalid_block

block = frontier.pop

return unless block

record(block: block, name: "pop")

block = @block_expand.call(block)

push(block, name: "expand")

end

def sweep_heredocs

code_search.rb

View Slide

# frozen_string_literal: true

module DeadEnd

# This class is responsible for taking a code block that exists

# at a far indentaion and then iteratively increasing the block

# so that it captures everything within the same indentation block.

#

# def dog

# puts "bow"

# puts "wow"

# end

#

# block = BlockExpand.new(code_lines: code_lines)

# .call(CodeBlock.new(lines: code_lines[1]))

#

# puts block.to_s

# # => puts "bow"

block_expand.rb

View Slide

# puts "wow"

# end

#

class BlockExpand

def initialize(code_lines: )

@code_lines = code_lines

end

def call(block)

if (next_block = expand_neighbors(block, grab_empty: true))

return next_block

end

expand_indent(block)

end

def expand_indent(block)

block_expand.rb

View Slide


block = AroundBlockScan.new(code_lines: @code_lines, block: block)

.skip(:hidden?)

.stop_after_kw

.scan_adjacent_indent

.code_block

end

def expand_neighbors(block, grab_empty: true)

scan = AroundBlockScan.new(code_lines: @code_lines, block: block)

.skip(:hidden?)

.stop_after_kw

.scan_neighbors

# Slurp up empties

block_expand.rb

View Slide

>


43 string << "```\n"

44 string

45 end

46





51 if line.empty?


53 else

60 end

61 end.join

62 end

63 end

64 end

View Slide

>


43 string << "```\n"

44 string

45 end

46





❯ 51 if line.empty?

❯ 53 else

❯ 60 end

61 end.join

62 end

63 end

64 end

View Slide

>


43 string << "```\n"

44 string

45 end

46





61 end.join

62 end

63 end

64 end

View Slide


end

# Main search loop

def call

sweep_heredocs

sweep_comments


@tick += 1

if frontier.expand?


else

visit_new_blocks

end

end

code_search.rb

View Slide


block = AroundBlockScan.new(code_lines: @code_lines, block: block)

.skip(:hidden?)

.stop_after_kw

.scan_adjacent_indent

.code_block

end

def expand_neighbors(block, grab_empty: true)

scan = AroundBlockScan.new(code_lines: @code_lines, block: block)

.skip(:hidden?)

.stop_after_kw

.scan_neighbors

# Slurp up empties

block_expand.rb

View Slide

code_line.rb

class CodeLine

TRAILING_SLASH = ("\\" + $/).freeze

attr_reader :line, :index, :indent, :original_line

def initialize(line: , index:)

@original_line = line.freeze

@line = @original_line

if line.strip.empty?

@empty = true

@indent = 0

else

@empty = false

@indent = SpaceCount.indent(line)

end

@index = index

@status = nil # valid, invalid, unknown

View Slide

private def lex_detect!

lex_array = LexAll.new(source: line)

kw_count = 0

end_count = 0

lex_array.each_with_index do |lex, index|

next unless lex.type == :on_kw

case lex.token

when 'if', 'unless', 'while', 'until'

# Only count if/unless when it's not a "trailing" if/unless

kw_count += 1 if !lex.expr_label?

when 'def', 'case', 'for', 'begin', 'class', 'module', 'do'

kw_count += 1

when 'end'

end_count += 1

end

code_line.rb

View Slide

>
source_code = %q{

string = <<~EOM

"I am a here doc" if true

EOM

}

Ripper.lex(source_code)

View Slide

>
source_code = %q{

string = <<~EOM


EOM

}

Ripper.lex(source_code)
[[[1, 0], :on_ignored_nl, "\n", BEG],

[[2, 0], :on_ident, "string", CMDARG],

[[2, 6], :on_sp, " ", CMDARG],

[[2, 7], :on_op, "=", BEG],

[[2, 8], :on_sp, " ", BEG],

[[2, 9], :on_heredoc_beg, "<<~EOM", BEG],

[[2, 15], :on_nl, "\n", BEG],

[[3, 0], :on_tstring_content,

" \"I am a here doc\" if true\n",

BEG],

[[4, 0], :on_tstring_content, "EOM \n",

BEG]]

View Slide

>
source_code = %q{

string = <<~EOM


EOM

}

Ripper.lex(source_code.lines[2])

View Slide

>
[[[1, 0], :on_sp, " ", BEG],


[[1, 5], :on_tstring_content,

"I am a here doc", BEG],


[[1, 21], :on_sp, " ", END],

[[1, 22], :on_kw, "if", BEG|LABEL],

[[1, 24], :on_sp, " ", BEG|LABEL],

[[1, 25], :on_kw, "true", END],

[[1, 29], :on_nl, "\n", BEG]]
source_code = %q{

string = <<~EOM


EOM

}

Ripper.lex(source_code.lines[2])

View Slide

private def lex_detect!

lex_array = LexAll.new(source: line)

kw_count = 0

end_count = 0

lex_array.each_with_index do |lex, index|

next unless lex.type == :on_kw

case lex.token

when 'if', 'unless', 'while', 'until'

# Only count if/unless when it's not a "trailing" if/unless

kw_count += 1 if !lex.expr_label?

when 'def', 'case', 'for', 'begin', 'class', 'module', 'do'

kw_count += 1

when 'end'

end_count += 1

end

code_line.rb

View Slide


end

# Main search loop

def call

sweep_heredocs

sweep_comments


@tick += 1

if frontier.expand?


else

visit_new_blocks

end

end

code_search.rb

View Slide

>

15 end

16

17 def call

❯ 18 @io.puts <<~EOM

❯ 19

❯ 20 DeadEnd: A syntax error was detected

❯ 21

❯ 22 This code has an unmatched `end` this is caused by either

❯ 23 missing a syntax keyword (`def`, `do`, etc.) or inclusion

❯ 24 of an extra `end` line:

❯ 25 EOM

26

27 @io.puts(<<~EOM) if filename

28 file: #{filename}

29 EOM

30

31 @io.puts <<~EOM

View Slide

>

15 end

16

17 def call

26

27 @io.puts(<<~EOM) if filename

28 file: #{filename}

29 EOM

30

31 @io.puts <<~EOM

32 #{code_with_filename}

33 EOM

34 end

35

36 def filename

37



View Slide

>

15 end

16

17 def call

26

❯ 27 @io.puts(<<~EOM) if filename

❯ 28 file: #{filename}

❯ 29 EOM

30

31 @io.puts <<~EOM

32 #{code_with_filename}

33 EOM

34 end

35

36 def filename

37



View Slide

>

15 end

16

17 def call

26

30

❯ 31 @io.puts <<~EOM

❯ 32 #{code_with_filename}

❯ 33 EOM

34 end

35

36 def filename

37



40 string << "```\n"



View Slide

>

15 end

16

17 def call

26

30

34 end

35

36 def filename

37



40 string << "```\n"



43 string << "```\n"

44 string

45 end

View Slide


end

# Main search loop

def call

sweep_heredocs

sweep_comments


@tick += 1

if frontier.expand?


else

visit_new_blocks

end

end

code_search.rb

View Slide

$ dead_end bad.rb --record tmp/

View Slide

View Slide


end

# Main search loop

def call

sweep_heredocs

sweep_comments


@tick += 1

if frontier.expand?


else

visit_new_blocks

end

end

code_search.rb

View Slide

def call

sweep_heredocs

sweep_comments


@tick += 1

if frontier.expand?


else

visit_new_blocks

end

end


@invalid_blocks.sort_by! {|block| block.starts_at }

self

end

code_search.rb

View Slide

# Example:

#

# combination([:a, :b, :c, :d])

# # => [[:a], [:b], [:c], [:d],

# [:a, :b], [:a, :c], [:a, :d],

# [:b, :c], [:b, :d], [:c, :d],

# [:a, :b, :c], [:a, :b, :d], [:a, :c, :d],

# [:b, :c, :d],

# [:a, :b, :c, :d]]

def self.combination(array)

guesses = []

1.upto(array.length).each do |size|

guesses.concat(array.combination(size).to_a)

end

guesses

end

code_frontier.rb

View Slide

# Given that we know our syntax error exists somewhere in our frontier, we want to find

# the smallest possible set of blocks that contain all the syntax errors

def detect_invalid_blocks

self.class.combination(@frontier.select(&:invalid?)).detect do |block_array|

holds_all_syntax_errors?(block_array)

end || []

end

end

end

code_frontier.rb

View Slide

guesses

end

# Given that we know our syntax error exists somewhere in our frontier, we want to find

# the smallest possible set of blocks that contain all the syntax errors

def detect_invalid_blocks

self.class.combination(@frontier.select(&:invalid?)).detect do |block_array|

holds_all_syntax_errors?(block_array)

end || []

end

end

end

code_frontier.rb

View Slide




end

blocks = search.invalid_blocks

DisplayInvalidBlocks.new(

blocks: blocks,

filename: filename,

terminal: terminal,

code_lines: search.code_lines,

invalid_obj: invalid_type(source),

io: io

).call

dead_end/internals.rb

View Slide

>

This code has a missing `end`. Ensure that all

syntax keywords (`def`, `do`, etc.) have a matching `end`.

file: /Users/rschneeman/Documents/projects/dead_end/spec/fixtures/this_project_extr

simplified:




❯ 36 def filename


❯ 45 end

63 end

64 end

View Slide

dead_end

View Slide

Are you a
dev?

View Slide

View Slide

Are you an
open source
dev?

View Slide

View Slide

Are you a
Ruby
core dev?

View Slide

❤
🙏 🙏

View Slide

Lexing

Parsing

Syntax

SyntaxErrors

AI

Pathfinding and goal seeking

View Slide

View Slide

@schneems

View Slide

View Slide

Questions?

View Slide

[Kaigi] Beware the Dead End

[Kaigi] Beware the Dead End

Richard Schneeman

More Decks by Richard Schneeman

Featured

Transcript