Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Syntax Tree (RubyKaigi)

Kevin Newton
September 09, 2022
930

Syntax Tree (RubyKaigi)

Syntax Tree is a new toolkit for interacting with the Ruby parse tree. It can be used to analyze, inspect, debug, and format your Ruby code. In this talk we'll walk through how it works, how to use it in your own applications, and the exciting future possibilities enabled by Syntax Tree.

Kevin Newton

September 09, 2022
Tweet

Transcript

  1. Kevin Newton
    @kddnewton

    View full-size slide

  2. Build a syntax tree
    Syntax Tree

    View full-size slide

  3. Build a syntax tree


    Format the syntax tree
    Syntax Tree

    View full-size slide

  4. Build a syntax tree


    Format the syntax tree


    Create a CLI
    Syntax Tree

    View full-size slide

  5. Build a syntax tree


    Format the syntax tree


    Create a CLI


    Create a language server (LSP)
    Syntax Tree

    View full-size slide

  6. Build a syntax tree


    Format the syntax tree


    Create a CLI


    Create a language server (LSP)


    Translate the syntax tree
    Syntax Tree

    View full-size slide

  7. Build · Format · CLI · LSP · Translate
    Syntax Tree

    View full-size slide

  8. Build a syntax tree
    Build · Format · CLI · LSP · Translate

    View full-size slide

  9. Define each node
    Build · Format · CLI · LSP · Translate

    View full-size slide

  10. Build · Format · CLI · LSP · Translate
    class VCall < Node


    attr_reader :value, :location, :comments


    def initialize(value:, location:, comments: [])


    @value = value


    @location = location


    @comments = comments


    end


    def accept(visitor)


    visitor.visit_vcall(self)


    end


    def child_nodes


    [value]


    end


    alias deconstruct child_nodes


    def deconstruct_keys(keys)


    { value: value, location: location, comments: comments }


    end


    end


    View full-size slide

  11. Build · Format · CLI · LSP · Translate
    class VCall < Node


    attr_reader :value, :location, :comments


    def initialize(value:, location:, comments: [])


    @value = value


    @location = location


    @comments = comments


    end


    def accept(visitor)


    visitor.visit_vcall(self)


    end


    def child_nodes


    [value]


    end


    alias deconstruct child_nodes


    def deconstruct_keys(keys)


    { value: value, location: location, comments: comments }


    end


    end


    Named fields

    View full-size slide

  12. Build · Format · CLI · LSP · Translate
    class VCall < Node


    attr_reader :value, :location, :comments


    def initialize(value:, location:, comments: [])


    @value = value


    @location = location


    @comments = comments


    end


    def accept(visitor)


    visitor.visit_vcall(self)


    end


    def child_nodes


    [value]


    end


    alias deconstruct child_nodes


    def deconstruct_keys(keys)


    { value: value, location: location, comments: comments }


    end


    end


    Named fields


    #location/#comments

    View full-size slide

  13. Build · Format · CLI · LSP · Translate
    class VCall < Node


    attr_reader :value, :location, :comments


    def initialize(value:, location:, comments: [])


    @value = value


    @location = location


    @comments = comments


    end


    def accept(visitor)


    visitor.visit_vcall(self)


    end


    def child_nodes


    [value]


    end


    alias deconstruct child_nodes


    def deconstruct_keys(keys)


    { value: value, location: location, comments: comments }


    end


    end


    Named fields


    #location/#comments


    #accept/#child_nodes

    View full-size slide

  14. Build · Format · CLI · LSP · Translate
    class VCall < Node


    attr_reader :value, :location, :comments


    def initialize(value:, location:, comments: [])


    @value = value


    @location = location


    @comments = comments


    end


    def accept(visitor)


    visitor.visit_vcall(self)


    end


    def child_nodes


    [value]


    end


    alias deconstruct child_nodes


    def deconstruct_keys(keys)


    { value: value, location: location, comments: comments }


    end


    end


    Named fields


    #location/#comments


    #accept/#child_nodes


    #deconstruct/

    #deconstruct_keys

    View full-size slide

  15. Build · Format · CLI · LSP · Translate
    class VCall < Node


    attr_reader :value, :location, :comments


    def initialize(value:, location:, comments: [])


    @value = value


    @location = location


    @comments = comments


    end


    def accept(visitor)


    visitor.visit_vcall(self)


    end


    def child_nodes


    [value]


    end


    alias deconstruct child_nodes


    def deconstruct_keys(keys)


    { value: value, location: location, comments: comments }


    end


    end


    Named fields


    #location/#comments


    #accept/#child_nodes


    #deconstruct/

    #deconstruct_keys


    Immutable

    View full-size slide

  16. Define each node
    Build · Format · CLI · LSP · Translate

    View full-size slide

  17. Ripper::EVENTS.count # => 190
    Define each node
    Build · Format · CLI · LSP · Translate

    View full-size slide

  18. Ripper::EVENTS.count # => 190


    SyntaxTree::Node.descendants.count

    # => 162
    Define each node
    Build · Format · CLI · LSP · Translate

    View full-size slide

  19. Ripper::EVENTS.count # => 190


    SyntaxTree::Node.descendants.count

    # => 162


    qwords_new/qwords_add ->
    SyntaxTree::QWords
    Define each node
    Build · Format · CLI · LSP · Translate

    View full-size slide

  20. How we use ripper
    Build · Format · CLI · LSP · Translate

    View full-size slide

  21. var_ref : user_variable


    {


    /*%%%*/


    if (!($$ = gettable(p, $1, &@$))) $$ = NEW_BEGIN(0, &@$);


    /*%


    if (id_is_var(p, get_id($1))) {


    $$ = dispatch1(var_ref, $1);


    }


    else {


    $$ = dispatch1(vcall, $1);


    }


    %*/


    }


    | keyword_variable


    {


    /*%%%*/


    if (!($$ = gettable(p, $1, &@$))) $$ = NEW_BEGIN(0, &@$);


    /*% %*/


    /*% ripper: var_ref!($1) %*/


    }


    ;


    Build · Format · CLI · LSP · Translate

    View full-size slide

  22. var_ref : user_variable


    {


    /*%%%*/


    if (!($$ = gettable(p, $1, &@$))) $$ = NEW_BEGIN(0, &@$);


    /*%


    if (id_is_var(p, get_id($1))) {


    $$ = dispatch1(var_ref, $1);


    }


    else {


    $$ = dispatch1(vcall, $1);


    }


    %*/


    }


    | keyword_variable


    {


    /*%%%*/


    if (!($$ = gettable(p, $1, &@$))) $$ = NEW_BEGIN(0, &@$);


    /*% %*/


    /*% ripper: var_ref!($1) %*/


    }


    ;


    Build · Format · CLI · LSP · Translate

    View full-size slide

  23. class Foo


    def bar


    baz


    end


    end


    Build · Format · CLI · LSP · Translate

    View full-size slide

  24. class Foo


    def bar


    baz


    end


    end


    Build · Format · CLI · LSP · Translate
    class Parser < Ripper


    def on_const(value) = [:@const, value]


    def on_ident(value) = [:@ident, value]


    def on_const_ref(const) = [:const_ref, const]


    def on_vcall(ident) = [:vcall, ident]


    def on_class(const, superclass, bodystmt)


    [:class, [const, superclass, bodystmt]]


    end


    def on_def(name, params, bodystmt)


    [:def, [name, params, bodystmt]]


    end


    end


    View full-size slide

  25. class Foo


    def bar


    baz


    end


    end


    Build · Format · CLI · LSP · Translate
    class Parser < Ripper


    def on_const(value) = [:@const, value]


    def on_ident(value) = [:@ident, value]


    def on_const_ref(const) = [:const_ref, const]


    def on_vcall(ident) = [:vcall, ident]


    def on_class(const, superclass, bodystmt)


    [:class, [const, superclass, bodystmt]]


    end


    def on_def(name, params, bodystmt)


    [:def, [name, params, bodystmt]]


    end


    end


    View full-size slide

  26. class Foo


    def bar


    baz


    end


    end


    Build · Format · CLI · LSP · Translate
    class Parser < Ripper


    def on_const(value) = [:@const, value]


    def on_ident(value) = [:@ident, value]


    def on_const_ref(const) = [:const_ref, const]


    def on_vcall(ident) = [:vcall, ident]


    def on_class(const, superclass, bodystmt)


    [:class, [const, superclass, bodystmt]]


    end


    def on_def(name, params, bodystmt)


    [:def, [name, params, bodystmt]]


    end


    end


    View full-size slide

  27. class Foo


    def bar


    baz


    end


    end


    Build · Format · CLI · LSP · Translate
    class Parser < Ripper


    def on_const(value) = [:@const, value]


    def on_ident(value) = [:@ident, value]


    def on_const_ref(const) = [:const_ref, const]


    def on_vcall(ident) = [:vcall, ident]


    def on_class(const, superclass, bodystmt)


    [:class, [const, superclass, bodystmt]]


    end


    def on_def(name, params, bodystmt)


    [:def, [name, params, bodystmt]]


    end


    end


    View full-size slide

  28. class Foo


    def bar


    baz


    end


    end


    Build · Format · CLI · LSP · Translate
    class Parser < Ripper


    def on_const(value) = [:@const, value]


    def on_ident(value) = [:@ident, value]


    def on_const_ref(const) = [:const_ref, const]


    def on_vcall(ident) = [:vcall, ident]


    def on_class(const, superclass, bodystmt)


    [:class, [const, superclass, bodystmt]]


    end


    def on_def(name, params, bodystmt)


    [:def, [name, params, bodystmt]]


    end


    end


    View full-size slide

  29. class Foo


    def bar


    baz


    end


    end


    Build · Format · CLI · LSP · Translate
    class Parser < Ripper


    def on_const(value) = [:@const, value]


    def on_ident(value) = [:@ident, value]


    def on_const_ref(const) = [:const_ref, const]


    def on_vcall(ident) = [:vcall, ident]


    def on_class(const, superclass, bodystmt)


    [:class, [const, superclass, bodystmt]]


    end


    def on_def(name, params, bodystmt)


    [:def, [name, params, bodystmt]]


    end


    end


    View full-size slide

  30. class Foo


    def bar


    baz


    end


    end


    Build · Format · CLI · LSP · Translate
    class Parser < Ripper


    def on_const(value) = [:@const, value]


    def on_ident(value) = [:@ident, value]


    def on_const_ref(const) = [:const_ref, const]


    def on_vcall(ident) = [:vcall, ident]


    def on_class(const, superclass, bodystmt)


    [:class, [const, superclass, bodystmt]]


    end


    def on_def(name, params, bodystmt)


    [:def, [name, params, bodystmt]]


    end


    end


    View full-size slide

  31. class Foo


    def bar


    baz


    end


    end


    Build · Format · CLI · LSP · Translate
    class Parser < Ripper


    def on_const(value) = [:@const, value]


    def on_ident(value) = [:@ident, value]


    def on_const_ref(const) = [:const_ref, const]


    def on_vcall(ident) = [:vcall, ident]


    def on_class(const, superclass, bodystmt)


    [:class, [const, superclass, bodystmt]]


    end


    def on_def(name, params, bodystmt)


    [:def, [name, params, bodystmt]]


    end


    end


    View full-size slide

  32. Build · Format · CLI · LSP · Translate
    class Parser < Ripper


    def on_const(value) = [:@const, value]


    def on_ident(value) = [:@ident, value]


    def on_const_ref(const) = [:const_ref, const]


    def on_vcall(ident) = [:vcall, ident]


    def on_class(const, superclass, bodystmt)


    [:class, [const, superclass, bodystmt]]


    end


    def on_def(name, params, bodystmt)


    [:def, [name, params, bodystmt]]


    end


    end


    View full-size slide

  33. Build · Format · CLI · LSP · Translate
    class Parser < Ripper


    def on_const(value) = [:@const, value]


    def on_ident(value) = [:@ident, value]


    def on_kw(value) = @stack << [:@kw, value]


    def on_const_ref(const) = [:const_ref, const]


    def on_vcall(ident) = [:vcall, ident]


    def on_class(const, superclass, bodystmt)


    kw = @stack.rfind { |token| token in [:@kw, "class"] }


    [:class, [@stack.delete(kw), const, superclass, bodystmt]]


    end


    def on_def(name, params, bodystmt)


    kw = @stack.rfind { |token| token in [:@kw, "def"] }


    [:def, [@stack.delete(kw), name, params, bodystmt]]


    end


    end


    View full-size slide

  34. Build · Format · CLI · LSP · Translate
    class Parser < Ripper


    def on_const(value) = [:@const, value]


    def on_ident(value) = [:@ident, value]


    def on_kw(value) = @stack << [:@kw, value]


    def on_const_ref(const) = [:const_ref, const]


    def on_vcall(ident) = [:vcall, ident]


    def on_class(const, superclass, bodystmt)


    kw = @stack.rfind { |token| token in [:@kw, "class"] }


    [:class, [@stack.delete(kw), const, superclass, bodystmt]]


    end


    def on_def(name, params, bodystmt)


    kw = @stack.rfind { |token| token in [:@kw, "def"] }


    [:def, [@stack.delete(kw), name, params, bodystmt]]


    end


    end


    View full-size slide

  35. Build · Format · CLI · LSP · Translate
    -> (positional; local1, local2) {}


    View full-size slide

  36. Build · Format · CLI · LSP · Translate
    -> (positional; local1, local2) {}


    range = (location.start_char + 1)...location.end_char


    locals = lambda_locals(source[range])


    Paren.new(


    lparen: params.lparen,


    contents:


    LambdaVar.new(


    params: params.contents,


    locals: locals,


    location: location


    ),


    location: params.location,


    comments: params.comments


    )


    View full-size slide

  37. class Foo


    def bar


    baz


    end


    end


    Build · Format · CLI · LSP · Translate

    View full-size slide

  38. Build · Format · CLI · LSP · Translate
    class Foo # comment


    def bar


    baz


    end


    end


    View full-size slide

  39. Build · Format · CLI · LSP · Translate
    class Foo


    def bar # comment


    baz


    end


    end


    View full-size slide

  40. Build · Format · CLI · LSP · Translate
    class Foo


    def bar


    baz # comment


    end


    end


    View full-size slide

  41. Build · Format · CLI · LSP · Translate
    # comment


    class Foo


    def bar


    baz


    end


    end


    View full-size slide

  42. Build · Format · CLI · LSP · Translate
    class Foo


    # comment


    def bar


    baz


    end


    end


    View full-size slide

  43. Build · Format · CLI · LSP · Translate
    # comment1


    class Foo # comment2


    # comment3


    def bar # comment4


    baz # comment5


    end


    end


    View full-size slide

  44. Build · Format · CLI · LSP · Translate
    # comment1


    class Foo # comment2


    # comment3


    def bar # comment4


    baz # comment5


    end


    end


    [


    ]

    View full-size slide

  45. Build · Format · CLI · LSP · Translate
    # comment1


    class Foo # comment2


    # comment3


    def bar # comment4


    baz # comment5


    end


    end


    [


    # comment1,


    ]

    View full-size slide

  46. Build · Format · CLI · LSP · Translate
    # comment1


    class Foo # comment2


    # comment3


    def bar # comment4


    baz # comment5


    end


    end


    [


    # comment1,


    # comment2,


    ]

    View full-size slide

  47. Build · Format · CLI · LSP · Translate
    # comment1


    class Foo # comment2


    # comment3


    def bar # comment4


    baz # comment5


    end


    end


    [


    # comment1,


    # comment2,


    # comment3,


    ]

    View full-size slide

  48. Build · Format · CLI · LSP · Translate
    # comment1


    class Foo # comment2


    # comment3


    def bar # comment4


    baz # comment5


    end


    end


    [


    # comment1,


    # comment2,


    # comment3,


    # comment4,


    ]

    View full-size slide

  49. Build · Format · CLI · LSP · Translate
    # comment1


    class Foo # comment2


    # comment3


    def bar # comment4


    baz # comment5


    end


    end


    [


    # comment1,


    # comment2,


    # comment3,


    # comment4,


    # comment5


    ]

    View full-size slide

  50. Build · Format · CLI · LSP · Translate
    [


    # comment1,


    # comment2,


    # comment3,


    # comment4,


    # comment5


    ]
    (program


    (statements


    ((class


    (const_ref (const "Foo"))


    (bodystmt


    (statements


    ((void_stmt),


    (def


    (ident "bar")


    (params)


    (bodystmt


    (statements


    ((vcall


    (ident "baz")))))))))))))


    View full-size slide

  51. Build · Format · CLI · LSP · Translate
    (program


    (statements


    ((comment "# comment1"),


    (class


    (const_ref (const "Foo") ((comment "# comment2")))


    (bodystmt


    (statements


    ((void_stmt),


    (comment "# comment3"),


    (def


    (ident "bar")


    (params ((comment "# comment4")))


    (bodystmt


    (statements


    ((vcall


    (ident "baz")


    ((comment "# comment5"))))))))))))))


    View full-size slide

  52. Walking the tree
    Build · Format · CLI · LSP · Translate

    View full-size slide

  53. Build · Format · CLI · LSP · Translate
    class Visitor


    def visit(node)


    node&.accept(self)


    end


    def visit_child_nodes(node)


    node.child_nodes.map do |node|


    visit(node)


    end


    end


    end


    View full-size slide

  54. Build · Format · CLI · LSP · Translate
    class VCall < Node


    def accept(visitor)


    visitor.visit_vcall(self)


    end


    end


    class Visitor


    def visit(node)


    node&.accept(self)


    end


    def visit_child_nodes(node)


    node.child_nodes.map do |node|


    visit(node)


    end


    end


    end


    View full-size slide

  55. Build · Format · CLI · LSP · Translate
    class VCall < Node


    def accept(visitor)


    visitor.visit_vcall(self)


    end


    end


    # Visit an ARef node.


    alias visit_aref visit_child_nodes


    # Visit an ARefField node.


    alias visit_aref_field visit_child_nodes


    # Visit an Alias node.


    alias visit_alias visit_child_nodes


    # Visit an ArgBlock node.


    alias visit_arg_block visit_child_nodes


    # Visit an ArgParen node.


    alias visit_arg_paren visit_child_nodes


    # Visit an ArgStar node.


    alias visit_arg_star visit_child_nodes


    # Visit an Args node.


    alias visit_args visit_child_nodes


    # Visit an ArgsForward node.


    alias visit_args_forward visit_child_nodes


    # Visit an ArrayLiteral node.


    alias visit_array visit_child_nodes


    # Visit an AryPtn node.


    alias visit_aryptn visit_child_nodes


    # Visit an Assign node.


    alias visit_assign visit_child_nodes


    # Visit an Assoc node.


    alias visit_assoc visit_child_nodes


    # Visit an AssocSplat node.


    alias visit_assoc_splat visit_child_nodes


    # Visit a Backref node.


    alias visit_backref visit_child_nodes


    class Visitor


    def visit(node)


    node&.accept(self)


    end


    def visit_child_nodes(node)


    node.child_nodes.map do |node|


    visit(node)


    end


    end


    end


    View full-size slide

  56. Build · Format · CLI · LSP · Translate
    class VCall < Node


    def accept(visitor)


    visitor.visit_vcall(self)


    end


    end



    alias visit_vcall visit_child_nodes


    # Visit a VoidStmt node.


    alias visit_void_stmt visit_child_nodes


    # Visit a When node.


    alias visit_when visit_child_nodes


    # Visit a While node.


    alias visit_while visit_child_nodes


    # Visit a WhileMod node.


    alias visit_while_mod visit_child_nodes


    # Visit a Word node.


    alias visit_word visit_child_nodes


    # Visit a Words node.


    alias visit_words visit_child_nodes


    # Visit a WordsBeg node.


    alias visit_words_beg visit_child_nodes


    # Visit a XString node.


    alias visit_xstring visit_child_nodes


    # Visit a XStringLiteral node.


    alias visit_xstring_literal visit_child_nodes


    # Visit a Yield node.


    alias visit_yield visit_child_nodes


    # Visit a Yield0 node.


    alias visit_yield0 visit_child_nodes


    # Visit a ZSuper node.


    alias visit_zsuper visit_child_nodes


    # Visit an EndContent node.


    alias visit___end__ visit_child_nodes


    class Visitor


    def visit(node)


    node&.accept(self)


    end


    def visit_child_nodes(node)


    node.child_nodes.map do |node|


    visit(node)


    end


    end


    end


    View full-size slide

  57. Build · Format · CLI · LSP · Translate
    class RegexpVisitor < Visitor


    def visit_regexp_literal(node)


    # do something with regexp literal node here


    node.options


    # call super to continue walking the tree


    super


    end


    end


    View full-size slide

  58. Build · Format · CLI · LSP · Translate
    class JSONVisitor < Visitor


    def visit_aref(node)


    node(node, "aref") do


    field("collection", node.collection)


    field("index", node.index)


    comments(node)


    end


    end


    def visit_aref_field(node)


    node(node, "aref_field") do


    field("collection", node.collection)


    field("index", node.index)


    comments(node)


    end


    end


    def visit_alias(node)


    node(node, "alias") do


    field("left", node.left)


    field("right", node.right)


    comments(node)


    end


    end

    View full-size slide

  59. Format the syntax tree
    Build · Format · CLI · LSP · Translate

    View full-size slide

  60. Algorithm
    Build · Format · CLI · LSP · Translate

    View full-size slide

  61. prettyprint
    Algorithm
    Build · Format · CLI · LSP · Translate

    View full-size slide

  62. prettyprint


    Build group, text, breakable to layout content
    Algorithm
    Build · Format · CLI · LSP · Translate

    View full-size slide

  63. prettyprint


    Build group, text, breakable to layout content


    Break outermost groups to fit to print width
    Algorithm
    Build · Format · CLI · LSP · Translate

    View full-size slide

  64. Build · Format · CLI · LSP · Translate
    [foo1, foo2, [bar1, bar2, bar3, bar4], foo3, foo4, [baz1, baz2]]


    View full-size slide

  65. Build · Format · CLI · LSP · Translate
    [foo1, foo2, [bar1, bar2, bar3, bar4], foo3, foo4, [baz1, baz2]]


    View full-size slide

  66. Build · Format · CLI · LSP · Translate
    [foo1, foo2, [bar1, bar2, bar3, bar4], foo3, foo4, [baz1, baz2]]


    View full-size slide

  67. Build · Format · CLI · LSP · Translate
    [


    foo1,


    foo2,


    [bar1, bar2, bar3, bar4],


    foo3,


    foo4,


    [baz1, baz2]


    ]


    View full-size slide

  68. Build · Format · CLI · LSP · Translate
    [


    foo1,


    foo2,


    [bar1, bar2, bar3, bar4],


    foo3,


    foo4,


    [baz1, baz2]


    ]


    View full-size slide

  69. Build · Format · CLI · LSP · Translate
    [


    foo1,


    foo2,


    [


    bar1,


    bar2,


    bar3,


    bar4


    ],


    foo3,


    foo4,


    [baz1, baz2]


    ]


    View full-size slide

  70. breakGroup([group([text(["["]),


    indent([breakable,


    text(["foo1", ","]),


    breakable,


    text(["foo2", ","]),


    breakable,


    group([text(["["]),


    indent([breakable,


    text(["bar1", ","]),


    breakable,


    text(["bar2", ","]),


    breakable,


    text(["bar3", ","]),


    breakable,


    text(["bar4"])]),


    breakable,


    text(["]"])]),


    text([","]),


    breakable,


    text(["foo3", ","]),


    breakable,


    text(["foo4", ","]),


    breakable,


    ...
    Build · Format · CLI · LSP · Translate

    View full-size slide

  71. breakGroup([group([text(["["]),


    indent([breakable,


    text(["foo1", ","]),


    breakable,


    text(["foo2", ","]),


    breakable,


    group([text(["["]),


    indent([breakable,


    text(["bar1", ","]),


    breakable,


    text(["bar2", ","]),


    breakable,


    text(["bar3", ","]),


    breakable,


    text(["bar4"])]),


    breakable,


    text(["]"])]),


    text([","]),


    breakable,


    text(["foo3", ","]),


    breakable,


    text(["foo4", ","]),


    breakable,


    ...
    Build · Format · CLI · LSP · Translate

    View full-size slide

  72. breakGroup([group([text(["["]),


    indent([breakable,


    text(["foo1", ","]),


    breakable,


    text(["foo2", ","]),


    breakable,


    group([text(["["]),


    indent([breakable,


    text(["bar1", ","]),


    breakable,


    text(["bar2", ","]),


    breakable,


    text(["bar3", ","]),


    breakable,


    text(["bar4"])]),


    breakable,


    text(["]"])]),


    text([","]),


    breakable,


    text(["foo3", ","]),


    breakable,


    text(["foo4", ","]),


    breakable,


    ...
    Build · Format · CLI · LSP · Translate

    View full-size slide

  73. breakGroup([group([text(["["]),


    indent([breakable,


    text(["foo1", ","]),


    breakable,


    text(["foo2", ","]),


    breakable,


    group([text(["["]),


    indent([breakable,


    text(["bar1", ","]),


    breakable,


    text(["bar2", ","]),


    breakable,


    text(["bar3", ","]),


    breakable,


    text(["bar4"])]),


    breakable,


    text(["]"])]),


    text([","]),


    breakable,


    text(["foo3", ","]),


    breakable,


    text(["foo4", ","]),


    breakable,


    ...
    Build · Format · CLI · LSP · Translate

    View full-size slide

  74. breakGroup([group([text(["["]),


    indent([breakable,


    text(["foo1", ","]),


    breakable,


    text(["foo2", ","]),


    breakable,


    group([text(["["]),


    indent([breakable,


    text(["bar1", ","]),


    breakable,


    text(["bar2", ","]),


    breakable,


    text(["bar3", ","]),


    breakable,


    text(["bar4"])]),


    breakable,


    text(["]"])]),


    text([","]),


    breakable,


    text(["foo3", ","]),


    breakable,


    text(["foo4", ","]),


    breakable,


    ...
    Build · Format · CLI · LSP · Translate

    View full-size slide

  75. I made too many additions
    to prettyprint
    Build · Format · CLI · LSP · Translate
    😅😅😅

    View full-size slide

  76. prettyprint
    Build · Format · CLI · LSP · Translate

    View full-size slide

  77. prettier_print
    Build · Format · CLI · LSP · Translate

    View full-size slide

  78. Build · Format · CLI · LSP · Translate
    if foo


    bar += baz


    end


    View full-size slide

  79. Build · Format · CLI · LSP · Translate
    if foo


    bar += baz


    end


    View full-size slide

  80. Build · Format · CLI · LSP · Translate
    bar += baz if foo


    View full-size slide

  81. Build · Format · CLI · LSP · Translate
    if foo


    bar


    else


    baz


    end


    View full-size slide

  82. Build · Format · CLI · LSP · Translate
    if foo


    bar


    else


    baz


    end


    View full-size slide

  83. Build · Format · CLI · LSP · Translate
    foo ? bar : baz


    View full-size slide

  84. Build · Format · CLI · LSP · Translate
    here_is_a_method_call(1, 2,


    <<~HEREDOC


    this is the content of the heredoc


    HEREDOC


    )


    View full-size slide

  85. Build · Format · CLI · LSP · Translate
    here_is_a_method_call(1, 2,


    <<~HEREDOC


    this is the content of the heredoc


    HEREDOC


    )


    View full-size slide

  86. Build · Format · CLI · LSP · Translate
    here_is_a_method_call(1, 2, <<~HEREDOC)


    this is the content of the heredoc


    HEREDOC


    View full-size slide

  87. Build · Format · CLI · LSP · Translate
    def foo


    return [1, 2, 3].map { |element| element * 2 } # comment


    end


    View full-size slide

  88. Build · Format · CLI · LSP · Translate
    def foo


    return [1, 2, 3].map { |element| element * 2 } # comment


    end


    View full-size slide

  89. Build · Format · CLI · LSP · Translate
    def foo


    return(


    [1, 2, 3].map do |element|


    element * 2


    end


    ) # comment


    end


    View full-size slide

  90. Build · Format · CLI · LSP · Translate
    class If < Node


    def format(q)


    if node.consequent || node.statements.empty?


    q.group { format_break(q, force: true) }


    else


    q.group do


    q


    .if_break { format_break(q, force: false) }


    .if_flat do


    Parentheses.flat(q) do


    q.format(node.statements)


    q.text(" if ")


    q.format(node.predicate)


    end


    end


    end


    end


    end


    end


    View full-size slide

  91. Build · Format · CLI · LSP · Translate
    class If < Node


    def format(q)


    if node.consequent || node.statements.empty?


    q.group { format_break(q, force: true) }


    else


    q.group do


    q


    .if_break { format_break(q, force: false) }


    .if_flat do


    Parentheses.flat(q) do


    q.format(node.statements)


    q.text(" if ")


    q.format(node.predicate)


    end


    end


    end


    end


    end


    end


    View full-size slide

  92. Create a CLI
    Build · Format · CLI · LSP · Translate

    View full-size slide

  93. Build · Format · CLI · LSP · Translate
    stree ast [OPTIONS] [FILE]


    Print out the AST corresponding to the given files


    stree check [OPTIONS] [FILE]


    Check that the given files are formatted as syntax tree would format them


    stree debug [OPTIONS] [FILE]


    Check that the given files can be formatted idempotently


    stree doc [OPTIONS] [FILE]


    Print out the doc tree that would be used to format the given files


    stree format [OPTIONS] [FILE]


    Print out the formatted version of the given files


    stree json [OPTIONS] [FILE]


    Print out the JSON representation of the given files


    stree match [OPTIONS] [FILE]


    Print out a pattern-matching Ruby expression that would match the given files


    stree help


    Display this help message


    stree lsp


    Run syntax tree in language server mode


    stree version


    Output the current version of syntax tree


    stree write [OPTIONS] [FILE]


    Read, format, and write back the source of the given files

    View full-size slide

  94. Build · Format · CLI · LSP · Translate
    stree ast [OPTIONS] [FILE]


    Print out the AST corresponding to the given files


    stree check [OPTIONS] [FILE]


    Check that the given files are formatted as syntax tree would format them


    stree debug [OPTIONS] [FILE]


    Check that the given files can be formatted idempotently


    stree doc [OPTIONS] [FILE]


    Print out the doc tree that would be used to format the given files


    stree format [OPTIONS] [FILE]


    Print out the formatted version of the given files


    stree json [OPTIONS] [FILE]


    Print out the JSON representation of the given files


    stree match [OPTIONS] [FILE]


    Print out a pattern-matching Ruby expression that would match the given files


    stree help


    Display this help message


    stree lsp


    Run syntax tree in language server mode


    stree version


    Output the current version of syntax tree


    stree write [OPTIONS] [FILE]


    Read, format, and write back the source of the given files

    View full-size slide

  95. Build · Format · CLI · LSP · Translate
    stree ast [OPTIONS] [FILE]


    Print out the AST corresponding to the given files


    stree check [OPTIONS] [FILE]


    Check that the given files are formatted as syntax tree would format them


    stree debug [OPTIONS] [FILE]


    Check that the given files can be formatted idempotently


    stree doc [OPTIONS] [FILE]


    Print out the doc tree that would be used to format the given files


    stree format [OPTIONS] [FILE]


    Print out the formatted version of the given files


    stree json [OPTIONS] [FILE]


    Print out the JSON representation of the given files


    stree match [OPTIONS] [FILE]


    Print out a pattern-matching Ruby expression that would match the given files


    stree help


    Display this help message


    stree lsp


    Run syntax tree in language server mode


    stree version


    Output the current version of syntax tree


    stree write [OPTIONS] [FILE]


    Read, format, and write back the source of the given files

    View full-size slide

  96. Build · Format · CLI · LSP · Translate
    stree ast [OPTIONS] [FILE]


    Print out the AST corresponding to the given files


    stree check [OPTIONS] [FILE]


    Check that the given files are formatted as syntax tree would format them


    stree debug [OPTIONS] [FILE]


    Check that the given files can be formatted idempotently


    stree doc [OPTIONS] [FILE]


    Print out the doc tree that would be used to format the given files


    stree format [OPTIONS] [FILE]


    Print out the formatted version of the given files


    stree json [OPTIONS] [FILE]


    Print out the JSON representation of the given files


    stree match [OPTIONS] [FILE]


    Print out a pattern-matching Ruby expression that would match the given files


    stree help


    Display this help message


    stree lsp


    Run syntax tree in language server mode


    stree version


    Output the current version of syntax tree


    stree write [OPTIONS] [FILE]


    Read, format, and write back the source of the given files

    View full-size slide

  97. Build · Format · CLI · LSP · Translate
    stree ast [OPTIONS] [FILE]


    Print out the AST corresponding to the given files


    stree check [OPTIONS] [FILE]


    Check that the given files are formatted as syntax tree would format them


    stree debug [OPTIONS] [FILE]


    Check that the given files can be formatted idempotently


    stree doc [OPTIONS] [FILE]


    Print out the doc tree that would be used to format the given files


    stree format [OPTIONS] [FILE]


    Print out the formatted version of the given files


    stree json [OPTIONS] [FILE]


    Print out the JSON representation of the given files


    stree match [OPTIONS] [FILE]


    Print out a pattern-matching Ruby expression that would match the given files


    stree help


    Display this help message


    stree lsp


    Run syntax tree in language server mode


    stree version


    Output the current version of syntax tree


    stree write [OPTIONS] [FILE]


    Read, format, and write back the source of the given files

    View full-size slide

  98. Plugins
    Build · Format · CLI · LSP · Translate

    View full-size slide

  99. Formatting plugins
    Build · Format · CLI · LSP · Translate

    View full-size slide

  100. Single quotes
    Formatting plugins
    Build · Format · CLI · LSP · Translate

    View full-size slide

  101. Single quotes


    Trailing commas
    Formatting plugins
    Build · Format · CLI · LSP · Translate

    View full-size slide

  102. Language plugins
    Build · Format · CLI · LSP · Translate

    View full-size slide

  103. Language plugins
    Build · Format · CLI · LSP · Translate
    rbs

    View full-size slide

  104. Language plugins
    Build · Format · CLI · LSP · Translate
    rbs


    haml

    View full-size slide

  105. Language plugins
    Build · Format · CLI · LSP · Translate
    rbs


    haml


    json

    View full-size slide

  106. Language plugins
    Build · Format · CLI · LSP · Translate
    rbs


    haml


    json


    xml

    View full-size slide

  107. Language plugins
    Build · Format · CLI · LSP · Translate
    rbs


    haml


    json


    xml


    css

    View full-size slide

  108. Build · Format · CLI · LSP · Translate
    stree ast [OPTIONS] [FILE]


    Print out the AST corresponding to the given files


    stree check [OPTIONS] [FILE]


    Check that the given files are formatted as syntax tree would format them


    stree debug [OPTIONS] [FILE]


    Check that the given files can be formatted idempotently


    stree doc [OPTIONS] [FILE]


    Print out the doc tree that would be used to format the given files


    stree format [OPTIONS] [FILE]


    Print out the formatted version of the given files


    stree json [OPTIONS] [FILE]


    Print out the JSON representation of the given files


    stree match [OPTIONS] [FILE]


    Print out a pattern-matching Ruby expression that would match the given files


    stree help


    Display this help message


    stree lsp


    Run syntax tree in language server mode


    stree version


    Output the current version of syntax tree


    stree write [OPTIONS] [FILE]


    Read, format, and write back the source of the given files

    View full-size slide

  109. Build · Format · CLI · LSP · Translate
    stree ast [OPTIONS] [FILE]


    Print out the AST corresponding to the given files


    stree check [OPTIONS] [FILE]


    Check that the given files are formatted as syntax tree would format them


    stree debug [OPTIONS] [FILE]


    Check that the given files can be formatted idempotently


    stree doc [OPTIONS] [FILE]


    Print out the doc tree that would be used to format the given files


    stree format [OPTIONS] [FILE]


    Print out the formatted version of the given files


    stree json [OPTIONS] [FILE]


    Print out the JSON representation of the given files


    stree match [OPTIONS] [FILE]


    Print out a pattern-matching Ruby expression that would match the given files


    stree help


    Display this help message


    stree lsp


    Run syntax tree in language server mode


    stree version


    Output the current version of syntax tree


    stree write [OPTIONS] [FILE]


    Read, format, and write back the source of the given files

    View full-size slide

  110. Create a


    language server
    Build · Format · CLI · LSP · Translate

    View full-size slide

  111. Language server protocol
    Build · Format · CLI · LSP · Translate

    View full-size slide

  112. Language server protocol
    Runs a JSON RPC server behind an editor
    Build · Format · CLI · LSP · Translate

    View full-size slide

  113. Language server protocol
    Runs a JSON RPC server behind an editor


    textDocument/didChange


    textDocument/didOpen


    textDocument/didClose
    Build · Format · CLI · LSP · Translate

    View full-size slide

  114. Language server protocol
    Runs a JSON RPC server behind an editor


    textDocument/didChange


    textDocument/didOpen


    textDocument/didClose


    textDocument/formatting
    Build · Format · CLI · LSP · Translate

    View full-size slide

  115. Language server protocol
    Runs a JSON RPC server behind an editor


    textDocument/didChange


    textDocument/didOpen


    textDocument/didClose


    textDocument/formatting


    textDocument/inlayHint
    Build · Format · CLI · LSP · Translate

    View full-size slide

  116. Build · Format · CLI · LSP · Translate

    View full-size slide

  117. Shopify/ruby-lsp
    Build · Format · CLI · LSP · Translate

    View full-size slide

  118. Shopify/ruby-lsp
    Document highlight
    Build · Format · CLI · LSP · Translate

    View full-size slide

  119. Shopify/ruby-lsp
    Document highlight


    Document symbols
    Build · Format · CLI · LSP · Translate

    View full-size slide

  120. Shopify/ruby-lsp
    Document highlight


    Document symbols


    Folding ranges
    Build · Format · CLI · LSP · Translate

    View full-size slide

  121. Shopify/ruby-lsp
    Document highlight


    Document symbols


    Folding ranges


    Formatting
    Build · Format · CLI · LSP · Translate

    View full-size slide

  122. Shopify/ruby-lsp
    Document highlight


    Document symbols


    Folding ranges


    Formatting


    Selection ranges
    Build · Format · CLI · LSP · Translate

    View full-size slide

  123. Shopify/ruby-lsp
    Document highlight


    Document symbols


    Folding ranges


    Formatting


    Selection ranges


    Semantic highlighting
    Build · Format · CLI · LSP · Translate

    View full-size slide

  124. Translate the


    syntax tree
    Build · Format · CLI · LSP · Translate

    View full-size slide

  125. Translate the syntax tree
    Build · Format · CLI · LSP · Translate

    View full-size slide

  126. Translate the syntax tree
    seattlerb/ruby_parser
    Build · Format · CLI · LSP · Translate

    View full-size slide

  127. Translate the syntax tree
    seattlerb/ruby_parser


    whitequark/parser
    Build · Format · CLI · LSP · Translate

    View full-size slide

  128. Translate the syntax tree
    seattlerb/ruby_parser


    whitequark/parser


    rubocop/rubocop-ast
    Build · Format · CLI · LSP · Translate

    View full-size slide

  129. Translate the syntax tree
    seattlerb/ruby_parser


    whitequark/parser


    rubocop/rubocop-ast


    ruby-syntax-tree/

    syntax_tree-translator
    Build · Format · CLI · LSP · Translate

    View full-size slide

  130. Syntax Tree


    github.com/ruby-syntax-tree
    Kevin Newton
    @kddnewton

    View full-size slide