Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Parsing Ruby (RubyKaigi)

Kevin Newton
September 09, 2021
92

Parsing Ruby (RubyKaigi)

Since Ruby's inception, there have been many different projects that parse Ruby code. This includes everything from development tools to Ruby implementations themselves. This talk dives into the technical details and tradeoffs of how each of these tools parses and subsequently understands your applications. After, we'll discuss how you can do the same with your own projects using the Ripper standard library. You'll see just how far we can take this library toward building useful development tools.

Kevin Newton

September 09, 2021
Tweet

Transcript

  1. twitter.com/kddnewton Parsing Ruby Ruby 0.06 1994-01-07 Fri Jan 7 15:23:20

    1994 Yukihiro Matsumoto (matz at nws119) * baseline - version 0.06.
  2. twitter.com/kddnewton Parsing Ruby Ruby 0.76 1995-05-19 Thu Jul 14 11:18:07

    1994 Yukihiro Matsumoto (matz@ix-02) * parse.y: Dictを⽣成する構⽂を追加. こちらを{..}にした. * parse.y: 配列を⽣成する構⽂を[..]に変更した. 過去のRubyスクリプ トとの互換性が保てないが, Dictを⽣成する構⽂を導⼊するに当たり, perl5に合わせて(意識して), 変更する時期は今しかないと考えた. *BACKWARD INCOMPATIBILITY*
  3. twitter.com/kddnewton Parsing Ruby Ruby 0.95 1995-12-21 Thu Nov 9 23:26:01

    1995 Yukihiro Matsumoto <[email protected]> * parse.y (f_arglist): メソッド定義の引数を括弧で括らなくても良い ようにした. Mon Aug 7 12:47:41 1995 Yukihiro Matsumoto <[email protected]> * parse.y: resque -> rescue.恥ずかしいがtypoを残しておくわけには いかないよなあ.なんで今まで気がつかなかったのか….
  4. twitter.com/kddnewton Parsing Ruby Ruby 1.0.961225 1996-12-25 Wed May 22 19:48:42

    1996 Yukihiro Matsumoto <[email protected]> * parse.y (superclass): スーパークラスの指定⼦を`:'から`<'に変更. Wed Mar 27 10:02:44 1996 Yukihiro Matsumoto <[email protected]> * parse.y: 予約語の変更 continue -> next
  5. twitter.com/kddnewton Parsing Ruby Ruby 1.0.971225 1997-12-25 Mon Apr 7 11:36:16

    1997 Yukihiro Matsumoto <[email protected]> * parse.y (primary): syntax to access singleton class. Thu Apr 3 02:12:31 1997 Yukihiro Matsumoto <[email protected]> * parse.y (parse_regx): new option //[nes] to specify character code for regexp literals. Last speci fi ed code option is valid.
  6. twitter.com/kddnewton Parsing Ruby Ruby 1.2.0 1998-12-25 • heredocs
 • =begin

    to =end
 • true and false
 • BEGIN and END
 • %w
 • Top-level constant access
 • ||= and &&=
  7. twitter.com/kddnewton Parsing Ruby Ruby 1.4.0 1999-08-13 • binary number literals


    • anonymous * in method de fi nitions
 • nested string interpolation
 • multibyte character identi fi ers
  8. twitter.com/kddnewton Parsing Ruby ruth 0.0.1 2001-01-10 • C extension
 •

    Ruby under the hood
 • Ruby::Interpreter.parse
  9. twitter.com/kddnewton Parsing Ruby Ruby 1.7.1 2001-06-01 • break and next

    now accept values
 • %w can escape spaces
 • rescue in singleton method bodies
  10. twitter.com/kddnewton Parsing Ruby JRuby 2001-09-10 • Java port of Ruby

    1.6
 • Rewrite actions in parse.y into Java
 • Rewrite standard library in Ruby
  11. twitter.com/kddnewton Parsing Ruby ripper 0.0.1 2001-10-20 • Rewrite parse.y to

    dispatch parser events
 • “Ripper is still early-alpha version”
  12. twitter.com/kddnewton Parsing Ruby Ruby 1.8.0 2003-08-04 • %W word lists


    • Dynamic symbols
 • Nested constant assignment
  13. twitter.com/kddnewton Parsing Ruby ruby_parser 1.0.0 2007-11-14 • Rewrite parse.y in

    Ruby, use racc
 • dawnscanner, debride, fasterer, fl ay, fl og, railroader, roodi
  14. twitter.com/kddnewton Parsing Ruby Ruby 1.9.0 2007-12-25 • Bison
 • YARV


    • Ripper merged
 • Lambda literals
 • Symbol hash keys
  15. twitter.com/kddnewton Parsing Ruby Ruby 1.9.1 2009-01-30 • encoding pragma
 •

    call shorthand
 • positional arguments after splat
  16. twitter.com/kddnewton Parsing Ruby Ruby Intermediate Language 2009-10-26 • Intermediate representation

    for powering semantic analysis
 • Used to implement type systems in OCaml
 • druby, rtc, rubydust
  17. twitter.com/kddnewton Parsing Ruby LASER 0.0.1 2010-08-27 • Undergraduate thesis
 •

    Linter, type system, documentation generation
 • Used ripper internally
  18. twitter.com/kddnewton Parsing Ruby Topaz • Ruby implementation targeting RPython
 •

    Rewrote parse.y making grammar rules into decorators 2012-04-07
  19. twitter.com/kddnewton Parsing Ruby Ruby 2.0.0 • Re fi nements
 •

    %i symbol lists
 • Keyword arguments 2013-02-24
  20. twitter.com/kddnewton Parsing Ruby parser • Published gem, parser API
 •

    Well-documented
 • covered, deep-cover, erb-lint, fast, opal, packwerk, querly, rdl, reek, rubocop, rubrowser, ruby-lint, ruby-next, ruby_detective, rubycritic, seeing_is_believing, standard, steep, unparser, vernacular, yoda 2013-04-15
  21. twitter.com/kddnewton Parsing Ruby Tru ffl eRuby • Originally forked JRuby


    • Graal dynamic compiler, Tru ffl e AST interpreter 2013-10-26
  22. twitter.com/kddnewton Parsing Ruby Ruby 2.1.0 • Required keyword arguments
 •

    Rational and complex literals
 • Frozen string literal su ffi x 2013-12-25
  23. twitter.com/kddnewton Parsing Ruby Ruby 2.3.0 • Frozen string literal pragma


    • <<~ heredocs
 • &. lonely operator 2015-12-25
  24. twitter.com/kddnewton Parsing Ruby Ruby 2.4.0 • Symbol#to_proc re fi nements


    • Top-level return
 • Multiple assignment in conditional 2016-12-25
  25. twitter.com/kddnewton Parsing Ruby typedruby • Type system in Rust, parser

    in C++
 • Grammar from Ruby 2.4, lexer from ruby_parser
 • Vendored in Sorbet 2017-02-26
  26. twitter.com/kddnewton Parsing Ruby Ruby 2.5.0 • String interpolation re fi

    nements
 • rescue and ensure at the block level 2017-12-25
  27. twitter.com/kddnewton Parsing Ruby Ruby 2.6.0 • RubyVM::AbstractSyntaxTree
 • Flip- fl

    op deprecated
 • Endless ranges
 • Non-ASCII constant names 2018-12-25
  28. twitter.com/kddnewton Parsing Ruby Ruby 2.7.0 • Flip- fl op undeprecated

    • Method reference operator • Keyword argument warning • No other keywords syntax • Beginless range • Pattern matching • Numbered parameters • Rightward assignment • Argument forwarding 2019-12-25
  29. twitter.com/kddnewton Parsing Ruby Ruby 3.0.0 • Keyword arguments
 • Single-line

    “endless” methods
 • “Find pattern” pattern matching
 • shareable_constant_value pragma
 • in keyword pattern matching 2020-12-25
  30. twitter.com/kddnewton Parsing Ruby Lessons • Very di ffi cult to

    keep up with core
 • Small changes in syntax can have wide-reaching e ff ects
 • So much subtle logic in parse.y, very di ff i cult to extend
  31. twitter.com/kddnewton Parsing Ruby parser • Tons of community adoption
 •

    Well-documented
 • Not 100% compatible, stu ff may break
 • Doesn’t ship with/test with core
  32. twitter.com/kddnewton Parsing Ruby ripper • Built into the parser generator


    • Well-tested in core
 • Ships with Ruby
 • No documentation
  33. twitter.com/kddnewton Parsing Ruby Language servers • castwide/solargraph (parser or RubyVM::AST)

    • mtsmfm/language_server-ruby (ripper)
 • rubyide/vscode-ruby (tree-sitter) • tomoasleep/yoda (parser)