YARP

Building good Ruby tooling is dependent on having a good parser. Today, it is difficult to use the CRuby parser to build tooling because of its lack of public interface and documentation. This has led to people using external gems and generally fragmenting the community.

The Yet Another Ruby Parser project is building a universal Ruby parser that can be used by all Ruby implementations and tools. It is documented, error tolerant, and performant. It can be used without linking against CRuby, which means it can be easily used by other projects.

This talk is about the YARP project's motivation, design, implementation, and results. Come to learn about the future of parsing Ruby.

Transcript

1. Yet Another
   
   
   Ruby Parser
   

   View Slide

2. Kevin Newton
   @kddnewton
   

   View Slide

3. Yet Another Ruby Parser
   

   View Slide

4. Yet Another Ruby Parser
   

   View Slide

5. Yet Another Ruby Parser
   Can parse 100% of:
   

   View Slide

6. Yet Another Ruby Parser
   Can parse 100% of:
   
   
   Shopify/shopify
   

   View Slide

7. Yet Another Ruby Parser
   Can parse 100% of:
   
   
   Shopify/shopify
   
   
   github/github
   

   View Slide

8. Yet Another Ruby Parser
   Can parse 100% of:
   
   
   Shopify/shopify
   
   
   github/github
   
   
   ruby/ruby
   

   View Slide

9. Yet Another Ruby Parser
   Can parse 100% of:
   
   
   Shopify/shopify
   
   
   github/github
   
   
   ruby/ruby
   
   
   The top 100 gems by download on rubygems.org
   

   View Slide

10. Yet Another Ruby Parser
    Can parse 100% of:
    
    
    Shopify/shopify
    
    
    github/github
    
    
    ruby/ruby
    
    
    The top 100 gems by download on rubygems.org
    
    
    Experimental fork of CRuby
    

    View Slide

11. Yet Another Ruby Parser
    Can parse 100% of:
    
    
    Shopify/shopify
    
    
    github/github
    
    
    ruby/ruby
    
    
    The top 100 gems by download on rubygems.org
    
    
    Experimental fork of CRuby
    
    
    Experimental feature in TruffleRuby
    

    View Slide

12. Yet Another Ruby Parser
    Can parse 100% of:
    
    
    Shopify/shopify
    
    
    github/github
    
    
    ruby/ruby
    
    
    The top 100 gems by download on rubygems.org
    
    
    Experimental fork of CRuby
    
    
    Experimental feature in TruffleRuby
    
    
    Experimental branch in JRuby
    

    View Slide

13. Yet Another Ruby Parser
    

    View Slide

14. Yet Another Ruby Parser
    Motivations
    

    View Slide

15. Error tolerance
    Yet Another Ruby Parser
    Motivations
    

    View Slide

16. Error tolerance
    
    
    Portability
    Yet Another Ruby Parser
    Motivations
    

    View Slide

17. Error tolerance
    
    
    Portability
    
    
    Maintainability
    Yet Another Ruby Parser
    Motivations
    

    View Slide

18. Error tolerance
    
    
    Portability
    
    
    Maintainability
    Yet Another Ruby Parser
    Motivations
    Development
    

    View Slide

19. Error tolerance
    
    
    Portability
    
    
    Maintainability
    
    
    Challenges
    Yet Another Ruby Parser
    Motivations
    Development
    

    View Slide

20. Error tolerance
    
    
    Portability
    
    
    Maintainability
    
    
    Challenges
    Yet Another Ruby Parser
    Motivations
    Development
    Path forward
    

    View Slide

21. Error tolerance
    
    
    Portability
    
    
    Maintainability
    
    
    Challenges
    
    
    Adoption
    Yet Another Ruby Parser
    Motivations
    Development
    Path forward
    

    View Slide

22. Error tolerance
    
    
    Portability
    
    
    Maintainability
    
    
    Challenges
    
    
    Adoption
    
    
    Future work
    Yet Another Ruby Parser
    Motivations
    Development
    Path forward
    

    View Slide

23. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

24. Error tolerance
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

25. Error tolerance
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

26. Error tolerance
    $ irb
    
    
    irb(main):001:0> foo + bar + <> baz
    
    
    .../irb/workspace.rb:119:in `eval': (irb):1: syntax
    error, unexpected '<' (SyntaxError)
    
    
    foo + bar + <> baz
    
    
    ^
    
    
    
    
    from .../exe/irb:11:in `'
    
    
    from .../bin/irb:25:in `load'
    
    
    from .../bin/irb:25:in `'
    
    
    irb(main):002:0>
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

27. Error tolerance
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

28. Error tolerance
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

29. Automatically insert tokens
    Error tolerance
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

30. Automatically insert tokens
    
    
    Automatically insert nodes
    Error tolerance
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

31. Error tolerance
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

32. Automatically insert tokens
    
    
    Automatically insert nodes
    
    
    Context-based recovery
    Error tolerance
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

33. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    class Foo
    
    
    def bar(baz)
    
    
    self.then { |value| value + baz *
    
    
    end
    
    
    end
    
    
    

    View Slide

34. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    class Foo
    
    
    def bar(baz)
    
    
    self.then { |value| value + baz *
    
    
    end
    
    
    end
    
    
    ClassNode
    

    View Slide

35. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    class Foo
    
    
    def bar(baz)
    
    
    self.then { |value| value + baz *
    
    
    end
    
    
    end
    
    
    ClassNode
    StatementsNode
    

    View Slide

36. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    class Foo
    
    
    def bar(baz)
    
    
    self.then { |value| value + baz *
    
    
    end
    
    
    end
    
    
    ClassNode
    StatementsNode
    DefNode
    

    View Slide

37. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    class Foo
    
    
    def bar(baz)
    
    
    self.then { |value| value + baz *
    
    
    end
    
    
    end
    
    
    ClassNode
    StatementsNode
    DefNode
    CallNode
    

    View Slide

38. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    class Foo
    
    
    def bar(baz)
    
    
    self.then { |value| value + baz *
    
    
    end
    
    
    end
    
    
    ClassNode
    StatementsNode
    DefNode
    CallNode
    SelfNode
    

    View Slide

39. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    class Foo
    
    
    def bar(baz)
    
    
    self.then { |value| value + baz *
    
    
    end
    
    
    end
    
    
    ClassNode
    StatementsNode
    DefNode
    CallNode
    SelfNode
    BlockNode
    

    View Slide

40. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    class Foo
    
    
    def bar(baz)
    
    
    self.then { |value| value + baz *
    
    
    end
    
    
    end
    
    
    ClassNode
    StatementsNode
    DefNode
    CallNode
    SelfNode
    BlockNode
    CallNode
    

    View Slide

41. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    class Foo
    
    
    def bar(baz)
    
    
    self.then { |value| value + baz *
    
    
    end
    
    
    end
    
    
    ClassNode
    StatementsNode
    DefNode
    CallNode
    SelfNode
    BlockNode
    CallNode
    ArgumentsNode
    

    View Slide

42. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    class Foo
    
    
    def bar(baz)
    
    
    self.then { |value| value + baz *
    
    
    end
    
    
    end
    
    
    ClassNode
    StatementsNode
    DefNode
    CallNode
    SelfNode
    BlockNode
    CallNode
    CallNode
    ArgumentsNode
    

    View Slide

43. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    class Foo
    
    
    def bar(baz)
    
    
    self.then { |value| value + baz *
    
    
    end
    
    
    end
    
    
    ClassNode
    StatementsNode
    DefNode
    CallNode
    SelfNode
    BlockNode
    CallNode
    CallNode
    ArgumentsNode
    ArgumentsNode
    

    View Slide

44. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    class Foo
    
    
    def bar(baz)
    
    
    self.then { |value| value + baz *
    
    
    end
    
    
    end
    
    
    ClassNode
    StatementsNode
    DefNode
    CallNode
    SelfNode
    BlockNode
    CallNode
    CallNode
    ArgumentsNode
    ArgumentsNode
    CLASS
    

    View Slide

45. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    class Foo
    
    
    def bar(baz)
    
    
    self.then { |value| value + baz *
    
    
    end
    
    
    end
    
    
    ClassNode
    StatementsNode
    DefNode
    CallNode
    SelfNode
    BlockNode
    CallNode
    CallNode
    ArgumentsNode
    ArgumentsNode
    CLASS
    CLASS,DEF
    

    View Slide

46. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    class Foo
    
    
    def bar(baz)
    
    
    self.then { |value| value + baz *
    
    
    end
    
    
    end
    
    
    ClassNode
    StatementsNode
    DefNode
    CallNode
    SelfNode
    BlockNode
    CallNode
    CallNode
    ArgumentsNode
    ArgumentsNode
    CLASS
    CLASS,DEF
    CLASS,DEF,BLOCK
    

    View Slide

47. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    class Foo
    
    
    def bar(baz)
    
    
    self.then { |value| value + baz *
    
    
    end
    
    
    end
    
    
    ClassNode
    StatementsNode
    DefNode
    CallNode
    SelfNode
    BlockNode
    CallNode
    CallNode
    ArgumentsNode
    ArgumentsNode
    CLASS
    CLASS,DEF
    CLASS,DEF,BLOCK
    

    View Slide

48. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    class Foo
    
    
    def bar(baz)
    
    
    self.then { |value| value + baz *
    
    
    end
    
    
    end
    
    
    ClassNode
    StatementsNode
    DefNode
    CallNode
    SelfNode
    BlockNode
    CallNode
    CallNode
    MissingNode
    ArgumentsNode
    ArgumentsNode
    CLASS
    CLASS,DEF
    CLASS,DEF,BLOCK
    

    View Slide

49. Portability
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

50. Portability
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

51. Portability
    CRuby
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

52. Portability
    parse.y
    CRuby
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

53. Portability
    parse.y
    ripper/parse.y
    CRuby
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

54. Portability
    parse.y
    ripper/parse.y
    CRuby Runtimes
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

55. Portability
    parse.y
    ripper/parse.y
    mruby
    CRuby Runtimes
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

56. Portability
    parse.y
    ripper/parse.y
    mruby
    JRuby
    CRuby Runtimes
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

57. Portability
    parse.y
    ripper/parse.y
    mruby
    JRuby
    TruffleRuby
    CRuby Runtimes
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

58. Portability
    parse.y
    ripper/parse.y
    mruby
    JRuby
    TruffleRuby
    ruruby
    CRuby Runtimes
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

59. Portability
    parse.y
    ripper/parse.y
    mruby
    JRuby
    TruffleRuby
    ruruby
    natalie
    CRuby Runtimes
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

60. Portability
    parse.y
    ripper/parse.y
    mruby
    JRuby
    TruffleRuby
    ruruby
    natalie
    CRuby Runtimes Tools
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

61. Portability
    parse.y
    ripper/parse.y
    mruby
    JRuby
    TruffleRuby
    ruruby
    natalie
    parser
    CRuby Runtimes Tools
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

62. Portability
    parse.y
    ripper/parse.y
    mruby
    JRuby
    TruffleRuby
    ruruby
    natalie
    parser
    ruby_parser
    CRuby Runtimes Tools
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

63. Portability
    parse.y
    ripper/parse.y
    mruby
    JRuby
    TruffleRuby
    ruruby
    natalie
    parser
    ruby_parser
    tree-sitter-ruby
    CRuby Runtimes Tools
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

64. Portability
    parse.y
    ripper/parse.y
    mruby
    JRuby
    TruffleRuby
    ruruby
    natalie
    parser
    ruby_parser
    tree-sitter-ruby
    sorbet
    CRuby Runtimes Tools
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

65. Portability
    parse.y
    ripper/parse.y
    mruby
    JRuby
    TruffleRuby
    ruruby
    natalie lib-ruby-parser
    parser
    ruby_parser
    tree-sitter-ruby
    sorbet
    CRuby Runtimes Tools
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

66. Portability
    kddnewton.com/parsing-ruby/
    parse.y
    ripper/parse.y
    mruby
    JRuby
    TruffleRuby
    ruruby
    natalie lib-ruby-parser
    parser
    ruby_parser
    tree-sitter-ruby
    sorbet
    CRuby Runtimes Tools
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

67. Portability
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

68. Portability
    syntax_tree
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

69. Portability
    syntax_tree
    rubyfmt
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

70. Portability
    syntax_tree
    rufo rubyfmt
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

71. Portability
    syntax_tree
    rufo rubyfmt
    syntax_suggest
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

72. Portability
    syntax_tree
    rufo rubyfmt
    ruby-lsp
    syntax_suggest
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

73. Portability
    syntax_tree
    rufo rubyfmt
    ruby-lsp
    rubocop
    syntax_suggest
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

74. Portability
    syntax_tree
    rufo rubyfmt
    ruby-lsp
    rubocop
    standard
    syntax_suggest
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

75. Portability
    syntax_tree
    rufo rubyfmt
    ruby-lsp
    unparser
    rubocop
    standard
    syntax_suggest
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

76. Portability
    syntax_tree
    rufo rubyfmt
    ruby-lsp
    unparser
    ruby-next
    rubocop
    standard
    syntax_suggest
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

77. Portability
    syntax_tree
    rufo rubyfmt
    ruby-lsp
    solargraph
    unparser
    ruby-next
    rubocop
    standard
    syntax_suggest
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

78. Portability
    syntax_tree
    rufo rubyfmt
    ruby-lsp
    solargraph
    unparser
    ruby-next
    rubocop
    standard
    steep
    syntax_suggest
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

79. Portability
    syntax_tree
    rufo rubyfmt
    ruby-lsp
    solargraph
    unparser
    ruby-next
    debride
    rubocop
    standard
    steep
    syntax_suggest
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

80. Portability
    syntax_tree
    rufo rubyfmt
    ruby-lsp
    solargraph
    unparser
    ruby-next
    flay
    debride
    rubocop
    standard
    steep
    syntax_suggest
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

81. Portability
    syntax_tree
    rufo rubyfmt
    ruby-lsp
    solargraph
    unparser
    ruby-next
    flog
    flay
    debride
    rubocop
    standard
    steep
    syntax_suggest
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

82. Portability
    syntax_tree
    rufo rubyfmt
    ruby-lsp
    solargraph
    unparser
    ruby-next
    flog
    flay
    debride
    fasterer
    rubocop
    standard
    steep
    syntax_suggest
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

83. Portability
    syntax_tree
    rufo rubyfmt
    ruby-lsp
    sorbet
    solargraph
    unparser
    ruby-next
    flog
    flay
    debride
    fasterer
    rubocop
    standard
    steep
    syntax_suggest
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

84. Portability
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

85. No reliance on CRuby internals (VALUE, ID, etc.)
    Portability
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

86. No reliance on CRuby internals (VALUE, ID, etc.)
    
    
    No reliance on any external packages or generators
    Portability
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

87. No reliance on CRuby internals (VALUE, ID, etc.)
    
    
    No reliance on any external packages or generators
    
    
    Shared logic: unescaping strings, pack API, etc.
    Portability
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

88. No reliance on CRuby internals (VALUE, ID, etc.)
    
    
    No reliance on any external packages or generators
    
    
    Shared logic: unescaping strings, pack API, etc.
    
    
    Serialization API
    Portability
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

89. #include "yarp.h"
    
    
    #include "org_yarp_Parser.h"
    
    
    JNIEXPORT jbyteArray JNICALL Java_org_yarp_Parser_parseAndSerialize(
    
    
    JNIEnv *env, jclass clazz, jbyteArray source) {
    
    
    jsize size = (*env)->GetArrayLength(env, source);
    
    
    jbyte* bytes = (*env)->GetByteArrayElements(env, source, NULL);
    
    
    yp_buffer_t buffer;
    
    
    yp_buffer_init(&buffer);
    
    
    yp_parse_serialize((char*) bytes, size, &buffer);
    
    
    (*env)->ReleaseByteArrayElements(env, source, bytes, JNI_ABORT);
    
    
    jbyteArray serialized = (*env)->NewByteArray(env, buffer.length);
    
    
    jbyte *value = buffer.value;
    
    
    (*env)->SetByteArrayRegion(env, serialized, 0, buffer.length, value);
    
    
    yp_buffer_free(&buffer);
    
    
    return serialized;
    
    
    }
    
    
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

90. #include "yarp.h"
    
    
    #include "org_yarp_Parser.h"
    
    
    JNIEXPORT jbyteArray JNICALL Java_org_yarp_Parser_parseAndSerialize(
    
    
    JNIEnv *env, jclass clazz, jbyteArray source) {
    
    
    jsize size = (*env)->GetArrayLength(env, source);
    
    
    jbyte* bytes = (*env)->GetByteArrayElements(env, source, NULL);
    
    
    yp_buffer_t buffer;
    
    
    yp_buffer_init(&buffer);
    
    
    yp_parse_serialize((char*) bytes, size, &buffer);
    
    
    (*env)->ReleaseByteArrayElements(env, source, bytes, JNI_ABORT);
    
    
    jbyteArray serialized = (*env)->NewByteArray(env, buffer.length);
    
    
    jbyte *value = buffer.value;
    
    
    (*env)->SetByteArrayRegion(env, serialized, 0, buffer.length, value);
    
    
    yp_buffer_free(&buffer);
    
    
    return serialized;
    
    
    }
    
    
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

91. Maintainability
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

92. Maintainability
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

93. Readability
    Maintainability
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

94. Readability
    
    
    Understandability
    Maintainability
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

95. Readability
    
    
    Understandability
    
    
    Ease of contribution
    Maintainability
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

96. Readability
    
    
    Understandability
    
    
    Ease of contribution
    
    
    Ease of change
    Maintainability
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

97. Readability
    
    
    Understandability
    
    
    Ease of contribution
    
    
    Ease of change
    
    
    Documentation
    Maintainability
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

98. Readability
    
    
    Understandability
    
    
    Ease of contribution
    
    
    Ease of change
    
    
    Documentation
    
    
    Test suites
    Maintainability
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

99. Maintainability
    Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
    

    View Slide

100. Is YARP more maintainable?
     Maintainability
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

101. Is YARP more maintainable?
     
     
     Can we make YARP more maintainable?
     Maintainability
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

102. Challenges
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

103. Challenges
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

104. Grammar ambiguities
     Challenges
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

105. Grammar ambiguities
     
     
     Unary/binary operators, spaces
     Challenges
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

106. Grammar ambiguities
     
     
     Unary/binary operators, spaces
     
     
     The `do` keyword
     Challenges
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

107. Grammar ambiguities
     
     
     Unary/binary operators, spaces
     
     
     The `do` keyword
     
     
     Newlines, comments, semicolons
     Challenges
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

108. Grammar ambiguities
     
     
     Unary/binary operators, spaces
     
     
     The `do` keyword
     
     
     Newlines, comments, semicolons
     
     
     Local variables and method calls
     Challenges
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

109. Grammar ambiguities
     
     
     Unary/binary operators, spaces
     
     
     The `do` keyword
     
     
     Newlines, comments, semicolons
     
     
     Local variables and method calls
     
     
     Encoding
     Challenges
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

110. Adoption
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

111. Adoption
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

112. Gem release
     Adoption
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

113. Gem release
     
     
     CRuby fork/compilation work
     Adoption
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

114. Gem release
     
     
     CRuby fork/compilation work
     
     
     JRuby exploring
     Adoption
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

115. Gem release
     
     
     CRuby fork/compilation work
     
     
     JRuby exploring
     
     
     TruffleRuby merged, actively working
     Adoption
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

116. Gem release
     
     
     CRuby fork/compilation work
     
     
     JRuby exploring
     
     
     TruffleRuby merged, actively working
     
     
     Other runtimes/tools
     Adoption
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

117. Gem release
     
     
     CRuby fork/compilation work
     
     
     JRuby exploring
     
     
     TruffleRuby merged, actively working
     
     
     Other runtimes/tools
     
     
     Compatibility with ripper
     Adoption
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

118. Future work
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

119. Future work
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

120. Error tolerance - forward scanning
     Future work
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

121. class Foo
     
     
     def bar(baz)
     
     
     baz.
     
     
     end
     
     
     end
     
     
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

122. class Foo
     
     
     def bar(baz)
     
     
     baz.
     
     
     end
     
     
     end
     
     
     baz.end Missing `end` keyword
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

123. class Foo
     
     
     def bar(baz)
     
     
     baz.
     
     
     end
     
     
     end
     
     
     baz.end
     baz.
     Missing `end` keyword
     Missing call message
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

124. Error tolerance - scanning forward
     Future work
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

125. Error tolerance - scanning forward
     
     
     Memory
     Future work
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

126. Error tolerance - scanning forward
     
     
     Memory
     
     
     Arena allocation, improve locality
     Future work
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

127. Error tolerance - scanning forward
     
     
     Memory
     
     
     Arena allocation, improve locality
     
     
     Reduce size of the tree
     Future work
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

128. Error tolerance - scanning forward
     
     
     Memory
     
     
     Arena allocation, improve locality
     
     
     Reduce size of the tree
     
     
     More nodes, more specialization
     Future work
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

129. Error tolerance - scanning forward
     
     
     Memory
     
     
     Arena allocation, improve locality
     
     
     Reduce size of the tree
     
     
     More nodes, more specialization
     
     
     Serialization - reduce size, variable length integers?
     Future work
     Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
     

     View Slide

130. Thank you!
     

     View Slide

131. Yet Another
     
     
     Ruby Parser
     
     
     github.com/Shopify/yarp
     Kevin Newton
     @kddnewton
     

     View Slide