Tweet
Tweet

Slide 1

Slide 1 text

Syntax Tree

Slide 2

Slide 2 text

Kevin Newton @kddnewton

Slide 3

Slide 3 text

Syntax Tree

Slide 4

Slide 4 text

What is Syntax Tree?

Slide 5

Slide 5 text

Syntax Tree concepts

Slide 6

Slide 6 text

Parser Syntax Tree concepts

Slide 7

Slide 7 text

Parser Syntax tree Syntax Tree concepts

Slide 8

Slide 8 text

Parser Syntax tree Visitor Syntax Tree concepts

Slide 9

Slide 9 text

Parser · Syntax tree · Visitor

Slide 10

Slide 10 text

Parser · Syntax tree · Visitor Parser

Slide 11

Slide 11 text

Matz is nice so we are nice. Parser · Syntax tree · Visitor

Slide 12

Slide 12 text

Lexical analysis Parser · Syntax tree · Visitor

Slide 13

Slide 13 text

Matz is nice so we are nice. Parser · Syntax tree · Visitor

Slide 14

Slide 14 text

Matz is nice so we are nice . Parser · Syntax tree · Visitor

Slide 15

Slide 15 text

is nice so we are nice . Matz Parser · Syntax tree · Visitor

Slide 16

Slide 16 text

is nice so we are nice . Matz Parser · Syntax tree · Visitor

Slide 17

Slide 17 text

Matz is nice we are nice . so Parser · Syntax tree · Visitor

Slide 18

Slide 18 text

Matz is nice so we are nice . Parser · Syntax tree · Visitor

Slide 19

Slide 19 text

Matz is nice so we are nice . Parser · Syntax tree · Visitor

Slide 20

Slide 20 text

Matz is nice so we are . nice Parser · Syntax tree · Visitor

Slide 21

Slide 21 text

Matz is nice so we are nice . Parser · Syntax tree · Visitor

Slide 22

Slide 22 text

Matz is nice so we are nice . Parser · Syntax tree · Visitor

Slide 23

Slide 23 text

Semantic analysis Parser · Syntax tree · Visitor

Slide 24

Slide 24 text

Parser · Syntax tree · Visitor is nice are nice

Slide 25

Slide 25 text

Grammar Parser · Syntax tree · Visitor

Slide 26

Slide 26 text

Grammar Parser · Syntax tree · Visitor verb-phrase : VERB ADJECTIVE

Slide 27

Slide 27 text

Parser · Syntax tree · Visitor is nice are nice

Slide 28

Slide 28 text

Parser · Syntax tree · Visitor is nice verb phrase verb phrase are nice

Slide 29

Slide 29 text

Parser · Syntax tree · Visitor Matz is nice verb phrase we verb phrase are nice

Slide 30

Slide 30 text

Grammar Parser · Syntax tree · Visitor verb-phrase : VERB ADJECTIVE

Slide 31

Slide 31 text

Grammar Parser · Syntax tree · Visitor verb-phrase : VERB ADJECTIVE 
 subject-phrase : NOUN verb-phrase

Slide 32

Slide 32 text

Parser · Syntax tree · Visitor Matz is nice verb phrase we verb phrase are nice

Slide 33

Slide 33 text

Parser · Syntax tree · Visitor Matz is nice subject phrase subject phrase verb phrase we verb phrase are nice

Slide 34

Slide 34 text

Parser · Syntax tree · Visitor Matz is nice so subject phrase subject phrase verb phrase we verb phrase are nice

Slide 35

Slide 35 text

Grammar Parser · Syntax tree · Visitor verb-phrase : VERB ADJECTIVE 
 subject-phrase : NOUN verb-phrase

Slide 36

Slide 36 text

Grammar Parser · Syntax tree · Visitor verb-phrase : VERB ADJECTIVE 
 subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase

Slide 37

Slide 37 text

Parser · Syntax tree · Visitor Matz is nice so subject phrase subject phrase verb phrase we verb phrase are nice

Slide 38

Slide 38 text

Parser · Syntax tree · Visitor Matz is nice so subord. conjunction subject phrase subject phrase verb phrase we verb phrase are nice

Slide 39

Slide 39 text

Parser · Syntax tree · Visitor Matz is nice so . subord. conjunction subject phrase subject phrase verb phrase we verb phrase are nice

Slide 40

Slide 40 text

verb-phrase : VERB ADJECTIVE 
 subject-phrase : NOUN verb-phrase 
 subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase Grammar Parser · Syntax tree · Visitor

Slide 41

Slide 41 text

verb-phrase : VERB ADJECTIVE 
 subject-phrase : NOUN verb-phrase 
 subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase 
 sentence : subordinating-conjunction PERIOD Grammar Parser · Syntax tree · Visitor

Slide 42

Slide 42 text

Parser · Syntax tree · Visitor Matz is nice so . subord. conjunction subject phrase subject phrase verb phrase we verb phrase are nice

Slide 43

Slide 43 text

Parser · Syntax tree · Visitor Matz is nice so . sentence subord. conjunction subject phrase subject phrase verb phrase we verb phrase are nice

Slide 44

Slide 44 text

verb-phrase : VERB ADJECTIVE 
 subject-phrase : NOUN verb-phrase 
 subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase 
 sentence : subordinating-conjunction PERIOD Grammar Parser · Syntax tree · Visitor

Slide 45

Slide 45 text

Parser · Syntax tree · Visitor Syntax tree

Slide 46

Slide 46 text

verb-phrase : VERB ADJECTIVE 
 subject-phrase : NOUN verb-phrase 
 subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase 
 sentence : subordinating-conjunction PERIOD Grammar Parser · Syntax tree · Visitor

Slide 47

Slide 47 text

Parser · Syntax tree · Visitor verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD

Slide 48

Slide 48 text

Parser · Syntax tree · Visitor class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD

Slide 49

Slide 49 text

Parser · Syntax tree · Visitor verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end

Slide 50

Slide 50 text

Parser · Syntax tree · Visitor class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end

Slide 51

Slide 51 text

Parser · Syntax tree · Visitor verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end

Slide 52

Slide 52 text

Parser · Syntax tree · Visitor verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end

Slide 53

Slide 53 text

Parser · Syntax tree · Visitor class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end

Slide 54

Slide 54 text

Parser · Syntax tree · Visitor class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end

Slide 55

Slide 55 text

Parser · Syntax tree · Visitor class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end

Slide 56

Slide 56 text

Parser · Syntax tree · Visitor class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end

Slide 57

Slide 57 text

Parser · Syntax tree · Visitor class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end

Slide 58

Slide 58 text

Parser · Syntax tree · Visitor class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end class Adjective attr_reader :value def initialize(value:) @value = value end end

Slide 59

Slide 59 text

Parser · Syntax tree · Visitor class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end class Adjective attr_reader :value def initialize(value:) @value = value end end

Slide 60

Slide 60 text

Parser · Syntax tree · Visitor class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end class Adjective attr_reader :value def initialize(value:) @value = value end end class Noun attr_reader :value def initialize(value:) @value = value end end

Slide 61

Slide 61 text

Parser · Syntax tree · Visitor class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end class Adjective attr_reader :value def initialize(value:) @value = value end end class Noun attr_reader :value def initialize(value:) @value = value end end

Slide 62

Slide 62 text

Parser · Syntax tree · Visitor class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end class Adjective attr_reader :value def initialize(value:) @value = value end end class Noun attr_reader :value def initialize(value:) @value = value end end class Conjunction attr_reader :value def initialize(value:) @value = value end end

Slide 63

Slide 63 text

Parser · Syntax tree · Visitor class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end class Adjective attr_reader :value def initialize(value:) @value = value end end class Noun attr_reader :value def initialize(value:) @value = value end end class Conjunction attr_reader :value def initialize(value:) @value = value end end

Slide 64

Slide 64 text

Parser · Syntax tree · Visitor class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end class Adjective attr_reader :value def initialize(value:) @value = value end end class Noun attr_reader :value def initialize(value:) @value = value end end class Conjunction attr_reader :value def initialize(value:) @value = value end end class Period attr_reader :value def initialize(value:) @value = value end end

Slide 65

Slide 65 text

Parser · Syntax tree · Visitor class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction : 
 subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end class Adjective attr_reader :value def initialize(value:) @value = value end end class Noun attr_reader :value def initialize(value:) @value = value end end class Conjunction attr_reader :value def initialize(value:) @value = value end end class Period attr_reader :value def initialize(value:) @value = value end end

Slide 66

Slide 66 text

Parser · Syntax tree · Visitor class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end class Adjective attr_reader :value def initialize(value:) @value = value end end class Noun attr_reader :value def initialize(value:) @value = value end end class Conjunction attr_reader :value def initialize(value:) @value = value end end class Period attr_reader :value def initialize(value:) @value = value end end

Slide 67

Slide 67 text

class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end class Adjective attr_reader :value def initialize(value:) @value = value end end class Noun attr_reader :value def initialize(value:) @value = value end end class Conjunction attr_reader :value def initialize(value:) @value = value end end class Period attr_reader :value def initialize(value:) @value = value end end Parser · Syntax tree · Visitor

Slide 68

Slide 68 text

class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end Parser · Syntax tree · Visitor

Slide 69

Slide 69 text

Parser · Syntax tree · Visitor class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end def deconstruct_keys(keys) { verb: verb, adjective: adjective } end def ==(other) other in VerbPhrase[ verb: ^(verb), adjective: ^(adjective) ] end end

Slide 70

Slide 70 text

class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end Parser · Syntax tree · Visitor

Slide 71

Slide 71 text

Parser · Syntax tree · Visitor class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end def copy(verb: self.verb, adjective: self.adjective) VerbPhrase.new(verb: verb, adjective: adjective) end end

Slide 72

Slide 72 text

class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end Parser · Syntax tree · Visitor

Slide 73

Slide 73 text

Parser · Syntax tree · Visitor class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end def accept(visitor) visitor.visit_verb_phrase(self) end def child_nodes [verb, adjective] end end

Slide 74

Slide 74 text

Parser · Syntax tree · Visitor class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end def accept(visitor) visitor.visit_verb_phrase(self) end def child_nodes [verb, adjective] end end

Slide 75

Slide 75 text

Visitor Parser · Syntax tree · Visitor

Slide 76

Slide 76 text

class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end def accept(visitor) visitor.visit_verb_phrase(self) end def child_nodes [verb, adjective] end end Parser · Syntax tree · Visitor

Slide 77

Slide 77 text

class VerbPhrase def accept(visitor) visitor.visit_verb_phrase(self) end end Parser · Syntax tree · Visitor

Slide 78

Slide 78 text

class VerbPhrase def accept(visitor) visitor.visit_verb_phrase(self) end end 
 class Visitor def visit_verb_phrase(node) # ... end end node.accept(Visitor.new) Parser · Syntax tree · Visitor

Slide 79

Slide 79 text

class VerbPhrase def accept(visitor) visitor.visit_verb_phrase(self) end end 
 class Visitor def visit_adjective(node) end def visit_conjunction(node) end def visit_noun(node) end def visit_verb(node) end def visit_punctuation(node) end def visit_sentence(node) end def visit_subordinating_conjunction(node) end def visit_subject_phrase(node) end def visit_verb_phrase(node) # ... end end node.accept(Visitor.new) Parser · Syntax tree · Visitor

Slide 80

Slide 80 text

class Visitor def visit_adjective(node) end def visit_conjunction(node) end def visit_noun(node) end def visit_verb(node) end def visit_punctuation(node) end def visit_sentence(node) end def visit_subordinating_conjunction(node) end def visit_subject_phrase(node) end def visit_verb_phrase(node) # ... end end node.accept(Visitor.new) Parser · Syntax tree · Visitor

Slide 81

Slide 81 text

class Visitor def visit(node) node.accept(self) end def visit_adjective(node) end def visit_conjunction(node) end def visit_noun(node) end def visit_verb(node) end def visit_punctuation(node) end def visit_sentence(node) end def visit_subordinating_conjunction(node) end def visit_subject_phrase(node) end def visit_verb_phrase(node) # ... end end node.accept(Visitor.new) Parser · Syntax tree · Visitor

Slide 82

Slide 82 text

Parser · Syntax tree · Visitor class Visitor def visit(node) node.accept(self) end def visit_adjective(node) end def visit_conjunction(node) end def visit_noun(node) end def visit_verb(node) end def visit_punctuation(node) end def visit_sentence(node) end def visit_subordinating_conjunction(node) end def visit_subject_phrase(node) end def visit_verb_phrase(node) node.child_nodes.each { |node| visit(node) } end end node.accept(Visitor.new)

Slide 83

Slide 83 text

Parser · Syntax tree · Visitor class Visitor def visit(node) node.accept(self) end def visit_adjective(node) end def visit_conjunction(node) end def visit_noun(node) end def visit_verb(node) end def visit_punctuation(node) end def visit_sentence(node) end def visit_subordinating_conjunction(node) end def visit_subject_phrase(node) end def visit_verb_phrase(node) node.child_nodes.each { |node| visit(node) } end end

Slide 84

Slide 84 text

Parser · Syntax tree · Visitor class Visitor def visit(node) node.accept(self) end def visit_child_nodes(node) node.child_nodes.each { |node| visit(node) } end def visit_adjective(node) end def visit_conjunction(node) end def visit_noun(node) end def visit_verb(node) end def visit_punctuation(node) end def visit_sentence(node) end def visit_subordinating_conjunction(node) end def visit_subject_phrase(node) end def visit_verb_phrase(node) visit_child_nodes(node) end end

Slide 85

Slide 85 text

Parser · Syntax tree · Visitor class Visitor def visit(node) node.accept(self) end def visit_child_nodes(node) node.child_nodes.each { |node| visit(node) } end def visit_adjective(node) end def visit_conjunction(node) end def visit_noun(node) end def visit_verb(node) end def visit_punctuation(node) end def visit_sentence(node) end def visit_subordinating_conjunction(node) end def visit_subject_phrase(node) end alias visit_verb_phrase visit_child_nodes end

Slide 86

Slide 86 text

Parser · Syntax tree · Visitor class Visitor def visit(node) node.accept(self) end def visit_child_nodes(node) node.child_nodes.each { |node| visit(node) } end alias visit_adjective visit_child_nodes alias visit_conjunction visit_child_nodes alias visit_noun visit_child_nodes alias visit_verb visit_child_nodes alias visit_punctuation visit_child_nodes alias visit_sentence visit_child_nodes alias visit_subordinating_conjunction 
 visit_child_nodes alias visit_subject_phrase visit_child_nodes alias visit_verb_phrase visit_child_nodes end

Slide 87

Slide 87 text

Parser · Syntax tree · Visitor

Slide 88

Slide 88 text

Parser · Syntax tree · Visitor class SentenceCountVisitor < Visitor attr_reader :count def initialize @count = 0 end def visit_sentence(node) @count += 1 super end end visitor = SentenceCountVisitor.new node.accept(visitor) puts visitor.count

Slide 89

Slide 89 text

Parser · Syntax tree · Visitor class PrettyPrintVisitor < Visitor attr_reader :q def initialize @q = PP.new(+"") end def visit_sentence(node) q.group(2, "sentence(", ")") do visit(node.phrase) visit(node.punctuation) end end end visitor = PrettyPrintVisitor.new node.accept(visitor) visitor.q.flush puts visitor.q.output

Slide 90

Slide 90 text

Parser · Syntax tree · Visitor class FormatterVisitor < Visitor attr_reader :q def initialize @q = PP.new(+"") end def visit_sentence(node) q.group do q.format(node.phrase) q.format(node.punctuation) end end end visitor = FormatterVisitor.new node.accept(visitor) visitor.q.flush puts visitor.q.output

Slide 91

Slide 91 text

What is Syntax Tree?

Slide 92

Slide 92 text

Syntax Tree

Slide 93

Slide 93 text

Object layer representing the result of parsing Ruby Syntax Tree

Slide 94

Slide 94 text

Object layer representing the result of parsing Ruby Tools to interact with and manipulate that layer Syntax Tree

Slide 95

Slide 95 text

Build a syntax tree Syntax Tree

Slide 96

Slide 96 text

Build a syntax tree Format the syntax tree Syntax Tree

Slide 97

Slide 97 text

Build a syntax tree Format the syntax tree Create a CLI Syntax Tree

Slide 98

Slide 98 text

Build a syntax tree Format the syntax tree Create a CLI Create a language server (LSP) Syntax Tree

Slide 99

Slide 99 text

Build a syntax tree Format the syntax tree Create a CLI Create a language server (LSP) Translate the syntax tree Syntax Tree

Slide 100

Slide 100 text

Build · Format · CLI · LSP · Translate Syntax Tree

Slide 101

Slide 101 text

Build a syntax tree Build · Format · CLI · LSP · Translate

Slide 102

Slide 102 text

Define each node Build · Format · CLI · LSP · Translate

Slide 103

Slide 103 text

Build · Format · CLI · LSP · Translate class VCall < Node attr_reader :value, :location, :comments def initialize(value:, location:, comments: []) @value = value @location = location @comments = comments end def accept(visitor) visitor.visit_vcall(self) end def child_nodes [value] end alias deconstruct child_nodes def deconstruct_keys(keys) { value: value, location: location, comments: comments } end end

Slide 104

Slide 104 text

Build · Format · CLI · LSP · Translate class VCall < Node attr_reader :value, :location, :comments def initialize(value:, location:, comments: []) @value = value @location = location @comments = comments end def accept(visitor) visitor.visit_vcall(self) end def child_nodes [value] end alias deconstruct child_nodes def deconstruct_keys(keys) { value: value, location: location, comments: comments } end end Named fields

Slide 105

Slide 105 text

Build · Format · CLI · LSP · Translate class VCall < Node attr_reader :value, :location, :comments def initialize(value:, location:, comments: []) @value = value @location = location @comments = comments end def accept(visitor) visitor.visit_vcall(self) end def child_nodes [value] end alias deconstruct child_nodes def deconstruct_keys(keys) { value: value, location: location, comments: comments } end end Named fields #location/#comments

Slide 106

Slide 106 text

Build · Format · CLI · LSP · Translate class VCall < Node attr_reader :value, :location, :comments def initialize(value:, location:, comments: []) @value = value @location = location @comments = comments end def accept(visitor) visitor.visit_vcall(self) end def child_nodes [value] end alias deconstruct child_nodes def deconstruct_keys(keys) { value: value, location: location, comments: comments } end end Named fields #location/#comments #accept/#child_nodes

Slide 107

Slide 107 text

Build · Format · CLI · LSP · Translate class VCall < Node attr_reader :value, :location, :comments def initialize(value:, location:, comments: []) @value = value @location = location @comments = comments end def accept(visitor) visitor.visit_vcall(self) end def child_nodes [value] end alias deconstruct child_nodes def deconstruct_keys(keys) { value: value, location: location, comments: comments } end end Named fields #location/#comments #accept/#child_nodes #deconstruct/ 
 #deconstruct_keys

Slide 108

Slide 108 text

Build · Format · CLI · LSP · Translate class VCall < Node attr_reader :value, :location, :comments def initialize(value:, location:, comments: []) @value = value @location = location @comments = comments end def accept(visitor) visitor.visit_vcall(self) end def child_nodes [value] end alias deconstruct child_nodes def deconstruct_keys(keys) { value: value, location: location, comments: comments } end end Named fields #location/#comments #accept/#child_nodes #deconstruct/ 
 #deconstruct_keys Immutable

Slide 109

Slide 109 text

Define each node Build · Format · CLI · LSP · Translate

Slide 110

Slide 110 text

Ripper::EVENTS.count # => 190 Define each node Build · Format · CLI · LSP · Translate

Slide 111

Slide 111 text

Ripper::EVENTS.count # => 190 SyntaxTree::Node.descendants.count 
 # => 162 Define each node Build · Format · CLI · LSP · Translate

Slide 112

Slide 112 text

Ripper::EVENTS.count # => 190 SyntaxTree::Node.descendants.count 
 # => 162 qwords_new/qwords_add -> SyntaxTree::QWords Define each node Build · Format · CLI · LSP · Translate

Slide 113

Slide 113 text

How we use ripper Build · Format · CLI · LSP · Translate

Slide 114

Slide 114 text

var_ref : user_variable { /*%%%*/ if (!($$ = gettable(p, $1, &@$))) $$ = NEW_BEGIN(0, &@$); /*% if (id_is_var(p, get_id($1))) { $$ = dispatch1(var_ref, $1); } else { $$ = dispatch1(vcall, $1); } %*/ } | keyword_variable { /*%%%*/ if (!($$ = gettable(p, $1, &@$))) $$ = NEW_BEGIN(0, &@$); /*% %*/ /*% ripper: var_ref!($1) %*/ } ; Build · Format · CLI · LSP · Translate

Slide 115

Slide 115 text

var_ref : user_variable { /*%%%*/ if (!($$ = gettable(p, $1, &@$))) $$ = NEW_BEGIN(0, &@$); /*% if (id_is_var(p, get_id($1))) { $$ = dispatch1(var_ref, $1); } else { $$ = dispatch1(vcall, $1); } %*/ } | keyword_variable { /*%%%*/ if (!($$ = gettable(p, $1, &@$))) $$ = NEW_BEGIN(0, &@$); /*% %*/ /*% ripper: var_ref!($1) %*/ } ; Build · Format · CLI · LSP · Translate

Slide 116

Slide 116 text

class Foo def bar baz end end Build · Format · CLI · LSP · Translate

Slide 117

Slide 117 text

class Foo def bar baz end end Build · Format · CLI · LSP · Translate class Parser < Ripper def on_const(value) = [:@const, value] def on_ident(value) = [:@ident, value] def on_const_ref(const) = [:const_ref, const] def on_vcall(ident) = [:vcall, ident] def on_class(const, superclass, bodystmt) [:class, [const, superclass, bodystmt]] end def on_def(name, params, bodystmt) [:def, [name, params, bodystmt]] end end

Slide 118

Slide 118 text

class Foo def bar baz end end Build · Format · CLI · LSP · Translate class Parser < Ripper def on_const(value) = [:@const, value] def on_ident(value) = [:@ident, value] def on_const_ref(const) = [:const_ref, const] def on_vcall(ident) = [:vcall, ident] def on_class(const, superclass, bodystmt) [:class, [const, superclass, bodystmt]] end def on_def(name, params, bodystmt) [:def, [name, params, bodystmt]] end end

Slide 119

Slide 119 text

class Foo def bar baz end end Build · Format · CLI · LSP · Translate class Parser < Ripper def on_const(value) = [:@const, value] def on_ident(value) = [:@ident, value] def on_const_ref(const) = [:const_ref, const] def on_vcall(ident) = [:vcall, ident] def on_class(const, superclass, bodystmt) [:class, [const, superclass, bodystmt]] end def on_def(name, params, bodystmt) [:def, [name, params, bodystmt]] end end

Slide 120

Slide 120 text

class Foo def bar baz end end Build · Format · CLI · LSP · Translate class Parser < Ripper def on_const(value) = [:@const, value] def on_ident(value) = [:@ident, value] def on_const_ref(const) = [:const_ref, const] def on_vcall(ident) = [:vcall, ident] def on_class(const, superclass, bodystmt) [:class, [const, superclass, bodystmt]] end def on_def(name, params, bodystmt) [:def, [name, params, bodystmt]] end end

Slide 121

Slide 121 text

class Foo def bar baz end end Build · Format · CLI · LSP · Translate class Parser < Ripper def on_const(value) = [:@const, value] def on_ident(value) = [:@ident, value] def on_const_ref(const) = [:const_ref, const] def on_vcall(ident) = [:vcall, ident] def on_class(const, superclass, bodystmt) [:class, [const, superclass, bodystmt]] end def on_def(name, params, bodystmt) [:def, [name, params, bodystmt]] end end

Slide 122

Slide 122 text

class Foo def bar baz end end Build · Format · CLI · LSP · Translate class Parser < Ripper def on_const(value) = [:@const, value] def on_ident(value) = [:@ident, value] def on_const_ref(const) = [:const_ref, const] def on_vcall(ident) = [:vcall, ident] def on_class(const, superclass, bodystmt) [:class, [const, superclass, bodystmt]] end def on_def(name, params, bodystmt) [:def, [name, params, bodystmt]] end end

Slide 123

Slide 123 text

class Foo def bar baz end end Build · Format · CLI · LSP · Translate class Parser < Ripper def on_const(value) = [:@const, value] def on_ident(value) = [:@ident, value] def on_const_ref(const) = [:const_ref, const] def on_vcall(ident) = [:vcall, ident] def on_class(const, superclass, bodystmt) [:class, [const, superclass, bodystmt]] end def on_def(name, params, bodystmt) [:def, [name, params, bodystmt]] end end

Slide 124

Slide 124 text

class Foo def bar baz end end Build · Format · CLI · LSP · Translate class Parser < Ripper def on_const(value) = [:@const, value] def on_ident(value) = [:@ident, value] def on_const_ref(const) = [:const_ref, const] def on_vcall(ident) = [:vcall, ident] def on_class(const, superclass, bodystmt) [:class, [const, superclass, bodystmt]] end def on_def(name, params, bodystmt) [:def, [name, params, bodystmt]] end end

Slide 125

Slide 125 text

Build · Format · CLI · LSP · Translate class Parser < Ripper def on_const(value) = [:@const, value] def on_ident(value) = [:@ident, value] def on_const_ref(const) = [:const_ref, const] def on_vcall(ident) = [:vcall, ident] def on_class(const, superclass, bodystmt) [:class, [const, superclass, bodystmt]] end def on_def(name, params, bodystmt) [:def, [name, params, bodystmt]] end end

Slide 126

Slide 126 text

Build · Format · CLI · LSP · Translate class Parser < Ripper def on_const(value) = [:@const, value] def on_ident(value) = [:@ident, value] def on_kw(value) = @stack << [:@kw, value] def on_const_ref(const) = [:const_ref, const] def on_vcall(ident) = [:vcall, ident] def on_class(const, superclass, bodystmt) kw = @stack.rfind { |token| token in [:@kw, "class"] } [:class, [@stack.delete(kw), const, superclass, bodystmt]] end def on_def(name, params, bodystmt) kw = @stack.rfind { |token| token in [:@kw, "def"] } [:def, [@stack.delete(kw), name, params, bodystmt]] end end

Slide 127

Slide 127 text

Build · Format · CLI · LSP · Translate class Parser < Ripper def on_const(value) = [:@const, value] def on_ident(value) = [:@ident, value] def on_kw(value) = @stack << [:@kw, value] def on_const_ref(const) = [:const_ref, const] def on_vcall(ident) = [:vcall, ident] def on_class(const, superclass, bodystmt) kw = @stack.rfind { |token| token in [:@kw, "class"] } [:class, [@stack.delete(kw), const, superclass, bodystmt]] end def on_def(name, params, bodystmt) kw = @stack.rfind { |token| token in [:@kw, "def"] } [:def, [@stack.delete(kw), name, params, bodystmt]] end end

Slide 128

Slide 128 text

Build · Format · CLI · LSP · Translate -> (positional; local1, local2) {}

Slide 129

Slide 129 text

Build · Format · CLI · LSP · Translate -> (positional; local1, local2) {} range = (location.start_char + 1)...location.end_char locals = lambda_locals(source[range]) Paren.new( lparen: params.lparen, contents: LambdaVar.new( params: params.contents, locals: locals, location: location ), location: params.location, comments: params.comments )

Slide 130

Slide 130 text

class Foo def bar baz end end Build · Format · CLI · LSP · Translate

Slide 131

Slide 131 text

Build · Format · CLI · LSP · Translate class Foo # comment def bar baz end end

Slide 132

Slide 132 text

Build · Format · CLI · LSP · Translate class Foo def bar # comment baz end end

Slide 133

Slide 133 text

Build · Format · CLI · LSP · Translate class Foo def bar baz # comment end end

Slide 134

Slide 134 text

Build · Format · CLI · LSP · Translate # comment class Foo def bar baz end end

Slide 135

Slide 135 text

Build · Format · CLI · LSP · Translate class Foo # comment def bar baz end end

Slide 136

Slide 136 text

Build · Format · CLI · LSP · Translate # comment1 class Foo # comment2 # comment3 def bar # comment4 baz # comment5 end end

Slide 137

Slide 137 text

Build · Format · CLI · LSP · Translate # comment1 class Foo # comment2 # comment3 def bar # comment4 baz # comment5 end end [ ]

Slide 138

Slide 138 text

Build · Format · CLI · LSP · Translate # comment1 class Foo # comment2 # comment3 def bar # comment4 baz # comment5 end end [ # comment1, ]

Slide 139

Slide 139 text

Build · Format · CLI · LSP · Translate # comment1 class Foo # comment2 # comment3 def bar # comment4 baz # comment5 end end [ # comment1, # comment2, ]

Slide 140

Slide 140 text

Build · Format · CLI · LSP · Translate # comment1 class Foo # comment2 # comment3 def bar # comment4 baz # comment5 end end [ # comment1, # comment2, # comment3, ]

Slide 141

Slide 141 text

Build · Format · CLI · LSP · Translate # comment1 class Foo # comment2 # comment3 def bar # comment4 baz # comment5 end end [ # comment1, # comment2, # comment3, # comment4, ]

Slide 142

Slide 142 text

Build · Format · CLI · LSP · Translate # comment1 class Foo # comment2 # comment3 def bar # comment4 baz # comment5 end end [ # comment1, # comment2, # comment3, # comment4, # comment5 ]

Slide 143

Slide 143 text

Build · Format · CLI · LSP · Translate [ # comment1, # comment2, # comment3, # comment4, # comment5 ] (program (statements ((class (const_ref (const "Foo")) (bodystmt (statements ((void_stmt), (def (ident "bar") (params) (bodystmt (statements ((vcall (ident "baz")))))))))))))

Slide 144

Slide 144 text

Build · Format · CLI · LSP · Translate (program (statements ((comment "# comment1"), (class (const_ref (const "Foo") ((comment "# comment2"))) (bodystmt (statements ((void_stmt), (comment "# comment3"), (def (ident "bar") (params ((comment "# comment4"))) (bodystmt (statements ((vcall (ident "baz") ((comment "# comment5"))))))))))))))

Slide 145

Slide 145 text

Walking the tree Build · Format · CLI · LSP · Translate

Slide 146

Slide 146 text

Build · Format · CLI · LSP · Translate class Visitor def visit(node) node&.accept(self) end def visit_child_nodes(node) node.child_nodes.map do |node| visit(node) end end end

Slide 147

Slide 147 text

Build · Format · CLI · LSP · Translate class VCall < Node def accept(visitor) visitor.visit_vcall(self) end end class Visitor def visit(node) node&.accept(self) end def visit_child_nodes(node) node.child_nodes.map do |node| visit(node) end end end

Slide 148

Slide 148 text

Build · Format · CLI · LSP · Translate class VCall < Node def accept(visitor) visitor.visit_vcall(self) end end # Visit an ARef node. alias visit_aref visit_child_nodes # Visit an ARefField node. alias visit_aref_field visit_child_nodes # Visit an Alias node. alias visit_alias visit_child_nodes # Visit an ArgBlock node. alias visit_arg_block visit_child_nodes # Visit an ArgParen node. alias visit_arg_paren visit_child_nodes # Visit an ArgStar node. alias visit_arg_star visit_child_nodes # Visit an Args node. alias visit_args visit_child_nodes # Visit an ArgsForward node. alias visit_args_forward visit_child_nodes # Visit an ArrayLiteral node. alias visit_array visit_child_nodes # Visit an AryPtn node. alias visit_aryptn visit_child_nodes # Visit an Assign node. alias visit_assign visit_child_nodes # Visit an Assoc node. alias visit_assoc visit_child_nodes # Visit an AssocSplat node. alias visit_assoc_splat visit_child_nodes # Visit a Backref node. alias visit_backref visit_child_nodes class Visitor def visit(node) node&.accept(self) end def visit_child_nodes(node) node.child_nodes.map do |node| visit(node) end end end

Slide 149

Slide 149 text

Build · Format · CLI · LSP · Translate class VCall < Node def accept(visitor) visitor.visit_vcall(self) end end alias visit_vcall visit_child_nodes # Visit a VoidStmt node. alias visit_void_stmt visit_child_nodes # Visit a When node. alias visit_when visit_child_nodes # Visit a While node. alias visit_while visit_child_nodes # Visit a WhileMod node. alias visit_while_mod visit_child_nodes # Visit a Word node. alias visit_word visit_child_nodes # Visit a Words node. alias visit_words visit_child_nodes # Visit a WordsBeg node. alias visit_words_beg visit_child_nodes # Visit a XString node. alias visit_xstring visit_child_nodes # Visit a XStringLiteral node. alias visit_xstring_literal visit_child_nodes # Visit a Yield node. alias visit_yield visit_child_nodes # Visit a Yield0 node. alias visit_yield0 visit_child_nodes # Visit a ZSuper node. alias visit_zsuper visit_child_nodes # Visit an EndContent node. alias visit___end__ visit_child_nodes class Visitor def visit(node) node&.accept(self) end def visit_child_nodes(node) node.child_nodes.map do |node| visit(node) end end end

Slide 150

Slide 150 text

Build · Format · CLI · LSP · Translate class RegexpVisitor < Visitor def visit_regexp_literal(node) # do something with regexp literal node here node.options # call super to continue walking the tree super end end

Slide 151

Slide 151 text

Build · Format · CLI · LSP · Translate class JSONVisitor < Visitor def visit_aref(node) node(node, "aref") do field("collection", node.collection) field("index", node.index) comments(node) end end def visit_aref_field(node) node(node, "aref_field") do field("collection", node.collection) field("index", node.index) comments(node) end end def visit_alias(node) node(node, "alias") do field("left", node.left) field("right", node.right) comments(node) end end

Slide 152

Slide 152 text

Format the syntax tree Build · Format · CLI · LSP · Translate

Slide 153

Slide 153 text

Algorithm Build · Format · CLI · LSP · Translate

Slide 154

Slide 154 text

prettyprint Algorithm Build · Format · CLI · LSP · Translate

Slide 155

Slide 155 text

prettyprint Build group, text, breakable to layout content Algorithm Build · Format · CLI · LSP · Translate

Slide 156

Slide 156 text

prettyprint Build group, text, breakable to layout content Break outermost groups to fit to print width Algorithm Build · Format · CLI · LSP · Translate

Slide 157

Slide 157 text

Build · Format · CLI · LSP · Translate [foo1, foo2, [bar1, bar2, bar3, bar4], foo3, foo4, [baz1, baz2]]

Slide 158

Slide 158 text

Build · Format · CLI · LSP · Translate [foo1, foo2, [bar1, bar2, bar3, bar4], foo3, foo4, [baz1, baz2]]

Slide 159

Slide 159 text

Build · Format · CLI · LSP · Translate [foo1, foo2, [bar1, bar2, bar3, bar4], foo3, foo4, [baz1, baz2]]

Slide 160

Slide 160 text

Build · Format · CLI · LSP · Translate [ foo1, foo2, [bar1, bar2, bar3, bar4], foo3, foo4, [baz1, baz2] ]

Slide 161

Slide 161 text

Build · Format · CLI · LSP · Translate [ foo1, foo2, [bar1, bar2, bar3, bar4], foo3, foo4, [baz1, baz2] ]

Slide 162

Slide 162 text

Build · Format · CLI · LSP · Translate [ foo1, foo2, [ bar1, bar2, bar3, bar4 ], foo3, foo4, [baz1, baz2] ]

Slide 163

Slide 163 text

breakGroup([group([text(["["]), indent([breakable, text(["foo1", ","]), breakable, text(["foo2", ","]), breakable, group([text(["["]), indent([breakable, text(["bar1", ","]), breakable, text(["bar2", ","]), breakable, text(["bar3", ","]), breakable, text(["bar4"])]), breakable, text(["]"])]), text([","]), breakable, text(["foo3", ","]), breakable, text(["foo4", ","]), breakable, ... Build · Format · CLI · LSP · Translate

Slide 164

Slide 164 text

breakGroup([group([text(["["]), indent([breakable, text(["foo1", ","]), breakable, text(["foo2", ","]), breakable, group([text(["["]), indent([breakable, text(["bar1", ","]), breakable, text(["bar2", ","]), breakable, text(["bar3", ","]), breakable, text(["bar4"])]), breakable, text(["]"])]), text([","]), breakable, text(["foo3", ","]), breakable, text(["foo4", ","]), breakable, ... Build · Format · CLI · LSP · Translate

Slide 165

Slide 165 text

breakGroup([group([text(["["]), indent([breakable, text(["foo1", ","]), breakable, text(["foo2", ","]), breakable, group([text(["["]), indent([breakable, text(["bar1", ","]), breakable, text(["bar2", ","]), breakable, text(["bar3", ","]), breakable, text(["bar4"])]), breakable, text(["]"])]), text([","]), breakable, text(["foo3", ","]), breakable, text(["foo4", ","]), breakable, ... Build · Format · CLI · LSP · Translate

Slide 166

Slide 166 text

breakGroup([group([text(["["]), indent([breakable, text(["foo1", ","]), breakable, text(["foo2", ","]), breakable, group([text(["["]), indent([breakable, text(["bar1", ","]), breakable, text(["bar2", ","]), breakable, text(["bar3", ","]), breakable, text(["bar4"])]), breakable, text(["]"])]), text([","]), breakable, text(["foo3", ","]), breakable, text(["foo4", ","]), breakable, ... Build · Format · CLI · LSP · Translate

Slide 167

Slide 167 text

breakGroup([group([text(["["]), indent([breakable, text(["foo1", ","]), breakable, text(["foo2", ","]), breakable, group([text(["["]), indent([breakable, text(["bar1", ","]), breakable, text(["bar2", ","]), breakable, text(["bar3", ","]), breakable, text(["bar4"])]), breakable, text(["]"])]), text([","]), breakable, text(["foo3", ","]), breakable, text(["foo4", ","]), breakable, ... Build · Format · CLI · LSP · Translate

Slide 168

Slide 168 text

I made too many additions to prettyprint Build · Format · CLI · LSP · Translate 😅😅😅

Slide 169

Slide 169 text

prettyprint Build · Format · CLI · LSP · Translate

Slide 170

Slide 170 text

prettier_print Build · Format · CLI · LSP · Translate

Slide 171

Slide 171 text

Build · Format · CLI · LSP · Translate if foo bar += baz end

Slide 172

Slide 172 text

Build · Format · CLI · LSP · Translate if foo bar += baz end

Slide 173

Slide 173 text

Build · Format · CLI · LSP · Translate bar += baz if foo

Slide 174

Slide 174 text

Build · Format · CLI · LSP · Translate if foo bar else baz end

Slide 175

Slide 175 text

Build · Format · CLI · LSP · Translate if foo bar else baz end

Slide 176

Slide 176 text

Build · Format · CLI · LSP · Translate foo ? bar : baz

Slide 177

Slide 177 text

Build · Format · CLI · LSP · Translate here_is_a_method_call(1, 2, <<~HEREDOC this is the content of the heredoc HEREDOC )

Slide 178

Slide 178 text

Build · Format · CLI · LSP · Translate here_is_a_method_call(1, 2, <<~HEREDOC this is the content of the heredoc HEREDOC )

Slide 179

Slide 179 text

Build · Format · CLI · LSP · Translate here_is_a_method_call(1, 2, <<~HEREDOC) this is the content of the heredoc HEREDOC

Slide 180

Slide 180 text

Build · Format · CLI · LSP · Translate def foo return [1, 2, 3].map { |element| element * 2 } # comment end

Slide 181

Slide 181 text

Build · Format · CLI · LSP · Translate def foo return [1, 2, 3].map { |element| element * 2 } # comment end

Slide 182

Slide 182 text

Build · Format · CLI · LSP · Translate def foo return( [1, 2, 3].map do |element| element * 2 end ) # comment end

Slide 183

Slide 183 text

Build · Format · CLI · LSP · Translate class If < Node def format(q) if node.consequent || node.statements.empty? q.group { format_break(q, force: true) } else q.group do q .if_break { format_break(q, force: false) } .if_flat do Parentheses.flat(q) do q.format(node.statements) q.text(" if ") q.format(node.predicate) end end end end end end

Slide 184

Slide 184 text

Build · Format · CLI · LSP · Translate class If < Node def format(q) if node.consequent || node.statements.empty? q.group { format_break(q, force: true) } else q.group do q .if_break { format_break(q, force: false) } .if_flat do Parentheses.flat(q) do q.format(node.statements) q.text(" if ") q.format(node.predicate) end end end end end end

Slide 185

Slide 185 text

Create a CLI Build · Format · CLI · LSP · Translate

Slide 186

Slide 186 text

Build · Format · CLI · LSP · Translate stree ast [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the AST corresponding to the given files stree check [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Check that the given files are formatted as syntax tree would format them stree debug [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Check that the given files can be formatted idempotently stree doc [--plugins=...] [-e SCRIPT] FILE Print out the doc tree that would be used to format the given files stree expr [-e SCRIPT] FILE Print out a pattern-matching Ruby expression that would match the first expression of the given files stree format [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the formatted version of the given files stree json [--plugins=...] [-e SCRIPT] FILE Print out the JSON representation of the given files stree match [--plugins=...] [-e SCRIPT] FILE Print out a pattern-matching Ruby expression that would match the given files stree help Display this help message stree lsp [--plugins=...] [--print-width=NUMBER] Run syntax tree in language server mode stree search PATTERN [-e SCRIPT] FILE Search for the given pattern in the given files stree version Output the current version of syntax tree stree write [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Read, format, and write back the source of the given files --plugins=... A comma-separated list of plugins to load. --print-width=NUMBER The maximum line width to use when formatting. -e SCRIPT Parse an inline Ruby string.

Slide 187

Slide 187 text

Build · Format · CLI · LSP · Translate stree ast [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the AST corresponding to the given files stree format [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the formatted version of the given files stree match [--plugins=...] [-e SCRIPT] FILE Print out a pattern-matching Ruby expression that would match the given files stree lsp [--plugins=...] [--print-width=NUMBER] Run syntax tree in language server mode stree search PATTERN [-e SCRIPT] FILE Search for the given pattern in the given files

Slide 188

Slide 188 text

Build · Format · CLI · LSP · Translate stree ast [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the AST corresponding to the given files stree format [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the formatted version of the given files stree match [--plugins=...] [-e SCRIPT] FILE Print out a pattern-matching Ruby expression that would match the given files stree lsp [--plugins=...] [--print-width=NUMBER] Run syntax tree in language server mode stree search PATTERN [-e SCRIPT] FILE Search for the given pattern in the given files

Slide 189

Slide 189 text

Build · Format · CLI · LSP · Translate $ stree ast -e '1+2'

Slide 190

Slide 190 text

Build · Format · CLI · LSP · Translate $ stree ast -e '1+2' (program (statements ((binary (int "1") + (int "2")))))

Slide 191

Slide 191 text

Build · Format · CLI · LSP · Translate stree ast [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the AST corresponding to the given files stree format [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the formatted version of the given files stree match [--plugins=...] [-e SCRIPT] FILE Print out a pattern-matching Ruby expression that would match the given files stree lsp [--plugins=...] [--print-width=NUMBER] Run syntax tree in language server mode stree search PATTERN [-e SCRIPT] FILE Search for the given pattern in the given files

Slide 192

Slide 192 text

Build · Format · CLI · LSP · Translate stree ast [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the AST corresponding to the given files stree format [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the formatted version of the given files stree match [--plugins=...] [-e SCRIPT] FILE Print out a pattern-matching Ruby expression that would match the given files stree lsp [--plugins=...] [--print-width=NUMBER] Run syntax tree in language server mode stree search PATTERN [-e SCRIPT] FILE Search for the given pattern in the given files

Slide 193

Slide 193 text

Build · Format · CLI · LSP · Translate $ stree format -e '1+2'

Slide 194

Slide 194 text

Build · Format · CLI · LSP · Translate $ stree format -e '1+2' 1 + 2

Slide 195

Slide 195 text

Build · Format · CLI · LSP · Translate stree ast [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the AST corresponding to the given files stree format [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the formatted version of the given files stree match [--plugins=...] [-e SCRIPT] FILE Print out a pattern-matching Ruby expression that would match the given files stree lsp [--plugins=...] [--print-width=NUMBER] Run syntax tree in language server mode stree search PATTERN [-e SCRIPT] FILE Search for the given pattern in the given files

Slide 196

Slide 196 text

Build · Format · CLI · LSP · Translate stree ast [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the AST corresponding to the given files stree format [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the formatted version of the given files stree match [--plugins=...] [-e SCRIPT] FILE Print out a pattern-matching Ruby expression that would match the given files stree lsp [--plugins=...] [--print-width=NUMBER] Run syntax tree in language server mode stree search PATTERN [-e SCRIPT] FILE Search for the given pattern in the given files

Slide 197

Slide 197 text

Build · Format · CLI · LSP · Translate $ stree match -e '1+2'

Slide 198

Slide 198 text

Build · Format · CLI · LSP · Translate $ stree match -e '1+2' SyntaxTree::Program[ statements: SyntaxTree::Statements[ body: [ SyntaxTree::Binary[ left: SyntaxTree::Int[value: "1"], operator: :+, right: SyntaxTree::Int[value: "2"] ] ] ] ]

Slide 199

Slide 199 text

Build · Format · CLI · LSP · Translate stree ast [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the AST corresponding to the given files stree format [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the formatted version of the given files stree match [--plugins=...] [-e SCRIPT] FILE Print out a pattern-matching Ruby expression that would match the given files stree lsp [--plugins=...] [--print-width=NUMBER] Run syntax tree in language server mode stree search PATTERN [-e SCRIPT] FILE Search for the given pattern in the given files

Slide 200

Slide 200 text

Build · Format · CLI · LSP · Translate stree ast [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the AST corresponding to the given files stree format [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the formatted version of the given files stree match [--plugins=...] [-e SCRIPT] FILE Print out a pattern-matching Ruby expression that would match the given files stree lsp [--plugins=...] [--print-width=NUMBER] Run syntax tree in language server mode stree search PATTERN [-e SCRIPT] FILE Search for the given pattern in the given files

Slide 201

Slide 201 text

Build · Format · CLI · LSP · Translate $ stree search Int -e '1+2'

Slide 202

Slide 202 text

Build · Format · CLI · LSP · Translate $ stree search Int -e '1+2' script:1:0: 1+2 script:1:2: 1+2

Slide 203

Slide 203 text

Build · Format · CLI · LSP · Translate $ stree search Int -e '1+2'

Slide 204

Slide 204 text

Build · Format · CLI · LSP · Translate $ stree search 'Int[value: "2"]' -e '1+2'

Slide 205

Slide 205 text

Build · Format · CLI · LSP · Translate $ stree search 'Int[value: "2"]' -e '1+2' 
 
 script:1:2: 1+2

Slide 206

Slide 206 text

Build · Format · CLI · LSP · Translate $ stree search <(stree match -e '1+2') \ 
 -e '1+2'

Slide 207

Slide 207 text

Build · Format · CLI · LSP · Translate $ stree search <(stree match -e '1+2') \ 
 -e '1+2' 
 
 script:1:0: 1+2

Slide 208

Slide 208 text

Build · Format · CLI · LSP · Translate stree ast [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the AST corresponding to the given files stree format [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the formatted version of the given files stree match [--plugins=...] [-e SCRIPT] FILE Print out a pattern-matching Ruby expression that would match the given files stree lsp [--plugins=...] [--print-width=NUMBER] Run syntax tree in language server mode stree search PATTERN [-e SCRIPT] FILE Search for the given pattern in the given files stree write [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Read, format, and write back the source of the given files

Slide 209

Slide 209 text

Build · Format · CLI · LSP · Translate stree ast [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the AST corresponding to the given files stree format [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the formatted version of the given files stree match [--plugins=...] [-e SCRIPT] FILE Print out a pattern-matching Ruby expression that would match the given files stree lsp [--plugins=...] [--print-width=NUMBER] Run syntax tree in language server mode stree search PATTERN [-e SCRIPT] FILE Search for the given pattern in the given files stree write [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Read, format, and write back the source of the given files

Slide 210

Slide 210 text

Create a language server Build · Format · CLI · LSP · Translate

Slide 211

Slide 211 text

Language server protocol Build · Format · CLI · LSP · Translate

Slide 212

Slide 212 text

Language server protocol Runs a JSON RPC server behind an editor Build · Format · CLI · LSP · Translate

Slide 213

Slide 213 text

Language server protocol Runs a JSON RPC server behind an editor textDocument/didChange textDocument/didOpen textDocument/didClose Build · Format · CLI · LSP · Translate

Slide 214

Slide 214 text

Language server protocol Runs a JSON RPC server behind an editor textDocument/didChange textDocument/didOpen textDocument/didClose textDocument/formatting Build · Format · CLI · LSP · Translate

Slide 215

Slide 215 text

Language server protocol Runs a JSON RPC server behind an editor textDocument/didChange textDocument/didOpen textDocument/didClose textDocument/formatting textDocument/inlayHint Build · Format · CLI · LSP · Translate

Slide 216

Slide 216 text

Build · Format · CLI · LSP · Translate

Slide 217

Slide 217 text

Shopify/ruby-lsp Build · Format · CLI · LSP · Translate

Slide 218

Slide 218 text

Shopify/ruby-lsp Document highlight Build · Format · CLI · LSP · Translate

Slide 219

Slide 219 text

Shopify/ruby-lsp Document highlight Document symbols Build · Format · CLI · LSP · Translate

Slide 220

Slide 220 text

Shopify/ruby-lsp Document highlight Document symbols Folding ranges Build · Format · CLI · LSP · Translate

Slide 221

Slide 221 text

Shopify/ruby-lsp Document highlight Document symbols Folding ranges Formatting Build · Format · CLI · LSP · Translate

Slide 222

Slide 222 text

Shopify/ruby-lsp Document highlight Document symbols Folding ranges Formatting Selection ranges Build · Format · CLI · LSP · Translate

Slide 223

Slide 223 text

Shopify/ruby-lsp Document highlight Document symbols Folding ranges Formatting Selection ranges Semantic highlighting Build · Format · CLI · LSP · Translate

Slide 224

Slide 224 text

Translate the syntax tree Build · Format · CLI · LSP · Translate

Slide 225

Slide 225 text

Translate the syntax tree Build · Format · CLI · LSP · Translate

Slide 226

Slide 226 text

Translate the syntax tree seattlerb/ruby_parser Build · Format · CLI · LSP · Translate

Slide 227

Slide 227 text

Build · Format · CLI · LSP · Translate class RubyParserVisitor < Visitor def visit_binary(node) left = visit(node.left) right = visit(node.right) case node in { operator: :and } s(:and, left, right) in { operator: :!~ } s(:not, s(:call, left, :=~, right)) else s(:call, left, node.operator, right) end end # ... end

Slide 228

Slide 228 text

Translate the syntax tree seattlerb/ruby_parser Build · Format · CLI · LSP · Translate

Slide 229

Slide 229 text

Translate the syntax tree seattlerb/ruby_parser whitequark/parser Build · Format · CLI · LSP · Translate

Slide 230

Slide 230 text

Build · Format · CLI · LSP · Translate class ParserVisitor < Visitor def visit_binary(node) left = visit(node.left) right = visit(node.right) case node in { operator: :and } s(:and, [left, right]) in { operator: :or } s(:or, [left, right]) else s(:send, [left, node.operator, right]) end end # ... end

Slide 231

Slide 231 text

Translate the syntax tree seattlerb/ruby_parser whitequark/parser Build · Format · CLI · LSP · Translate

Slide 232

Slide 232 text

Translate the syntax tree seattlerb/ruby_parser whitequark/parser YARV Build · Format · CLI · LSP · Translate

Slide 233

Slide 233 text

Build · Format · CLI · LSP · Translate class YARVVisitor < Visitor def visit_binary(node) case node.operator when :"&&" # ... when :"||" # ... else visit(node.left) visit(node.right) builder.send( node.operator, 1, VM_CALL_ARGS_SIMPLE ) end end # ... end

Slide 234

Slide 234 text

Translate the syntax tree seattlerb/ruby_parser whitequark/parser YARV Build · Format · CLI · LSP · Translate

Slide 235

Slide 235 text

Translate the syntax tree seattlerb/ruby_parser whitequark/parser YARV Ruby Build · Format · CLI · LSP · Translate

Slide 236

Slide 236 text

Build · Format · CLI · LSP · Translate class MutationVisitor < Visitor def visit_binary(node) node.copy( left: visit(node.left), right: visit(node.right) ) end end

Slide 237

Slide 237 text

Build · Format · CLI · LSP · Translate mutator = SyntaxTree.mutation do |mutator| query = <<~RUBY IfNode[predicate: Assign | OpAssign] | UnlessNode[predicate: Assign | OpAssign] RUBY mutator.mutate(query) do |node| node.copy(predicate: SyntaxTree::Paren.new( lparen: SyntaxTree::LParen.default, contents: node.predicate, location: node.predicate.location )) end end mutated = node.accept(mutator)

Slide 238

Slide 238 text

Translate the syntax tree seattlerb/ruby_parser whitequark/parser YARV Ruby Build · Format · CLI · LSP · Translate

Slide 239

Slide 239 text

Syntax Tree

Slide 240

Slide 240 text

Parser Syntax Tree

Slide 241

Slide 241 text

Parser Syntax tree Syntax Tree

Slide 242

Slide 242 text

Parser Syntax tree Visitor Syntax Tree

Slide 243

Slide 243 text

Syntax Tree

Slide 244

Slide 244 text

Syntax Tree Build a syntax tree

Slide 245

Slide 245 text

Syntax Tree Build a syntax tree Format the syntax tree

Slide 246

Slide 246 text

Syntax Tree Build a syntax tree Format the syntax tree Create a CLI

Slide 247

Slide 247 text

Syntax Tree Build a syntax tree Format the syntax tree Create a CLI Create a language server (LSP)

Slide 248

Slide 248 text

Syntax Tree Build a syntax tree Format the syntax tree Create a CLI Create a language server (LSP) Translate the syntax tree

Slide 249

Slide 249 text

Thank you!

Slide 250

Slide 250 text

Syntax Tree github.com/ruby-syntax-tree Kevin Newton @kddnewton