Trying to Make Ruby's Parser Available as a Gem

Fukuoka RubyistKaigi 04

Gem: https://github.com/S-H-GAMELINKS/kanayago

Transcript

1. Trying to Make Ruby’s Parser Available as a Gem Shun Hiraoka Fukuoka RubyistKaigi 04 2024-09-07
2. About me S.H.(Shun Hiraoka) Hamada.rb ESM, Inc GitHub: S-H-GAMELINKS Mastodon: S_H_@gamelinks007.net
3. Fukuoka RubyistKaigi 04
4. Daily Activities
5. I often write code with Ruby and Ruby’s Parser
6. I’ll be talking about Ruby’s Parser
7. Agenda Motivation 1. Gem 2. Implementation 3. Future Prospects and Issues 4.
8. (1) Motivation
9. Want to use Ruby Parser The Ruby Parser is now available as a Universal Parser
10. But… Few cases use Universal Parser Interested if it works as a Universal Parser
11. Example yui-knk/ruby-parser
12. yui-knk/ruby-parser It may be the only gem using Universal Parser Only parsing specific code Not converting and returning the AST as Ruby objects
13. Inconveniences Need to modify the code and rebuild to try various Ruby scripts Unable to work with the AST as Ruby objects It’s a bit cumbersome to try parsing Ruby code
14. Current status Errors occur because some function symbols cannot be found undefined symbol: rb_xmalloc_mul_add
15. Want a way to verify the Universal Parser
16. If it doesn’t exist, make it I’ll better understand Ruby’s Parser by building it Above all, it sounds interesting
17. Consider the specs
18. Easy to test code Prism.parse Code can be passed as a string The parsed result can be accepted as an AST from Prism Prism.parse('1 + 1')
19. Doing the same would be good
20. Read the implementation of yui-knk/ruby- parser
21. Implementation Using an extracted Ruby Parser Building a custom adapter for the Universal Parser void parser_config_initialize(rb_parser_config_t *config) { config->calloc = calloc; config->malloc = malloc; config->alloc = malloc; config->alloc_n = alloc_n; config->sized_xfree = sized_xfree; // And other adapter function set }
22. Is there an easier way? rb_parser_params_new function Returns the Universal Parser adapter used in Ruby rb_parser_t * rb_parser_params_new(void) { return rb_ruby_parser_new(&rb_global_parser_config); }
23. How to convert the AST? It might be a good idea to return the AST as a Prism-like class I haven’t planned for it to be used in any future products Then, using Array and Hash should be fine for now
24. Implementation plan Import and use only the Ruby’s Parser Reuse the adapter that Ruby provides Convert the AST into an Array and Hash
25. (2) Gem
26. Kanayago
27. Origin of the name Inspired by Kanayago, the god of ironworking The process of converting the AST into an Array or Hash reminded me of refining and processing iron ore
28. About Kanayago Works on Ruby 3.4.0-dev Enabling the Universal Parser is also necessary Using Universal Parser, fully Ruby-compatible But only what’s convertible to Hash and Array
29. Support AST nodes Literal Object e.g. Integer, Float, Symbol, String… Local variables Instance variables Class and Method definition if and unless expression
30. Example Code Takes a String as an argument, similar to Prism Returns the AST as a Hash and Array result = Kanayago.parse('1 + 1') body = result[:NODE_SCOPE][:body] args = body[:NODE_OPCALL][:args] args[:NODE_LIST][0] # => { NODE_INTEGER: 1 }
31. (3) Implementation
32. Implementation Takes source code as a string through the argument Passes the accepted code to the Universal Parser and converts it into an AST Convert the AST into a Hash and Array
33. Kanayago.parse Kanayago.parse accepts the source code and passes it to kanayago_parse function module Kanayago def self.parse(source) kanayago_parse(source) end end
34. kanayago_parse Pass the code to the Universal Parser and accept the AST static VALUE kanayago_parse(VALUE self, VALUE source) { struct ruby_parser *parser; rb_parser_t *parser_params; // Set Ruby Parser struct... VALUE vast = rb_parser_compile_string(vparser, "main", source, 0); rb_ast_t *ast = rb_ruby_ast_data_get(vast); return ast_to_hash(ast->body.root); }
35. ast_to_hash Convert the AST into a Hash and Array static VALUE ast_to_hash(const NODE *node) { enum node_type type; // Check and Set node type... switch (type) { // And Other AST NODE case's... case NODE_INTEGER: case NODE_FLOAT: case NODE_RATIONAL: case NODE_IMAGINARY: case NODE_STR: case NODE_SYM: return node_literal_to_hash(node); default: return Qfalse; } }
36. Processing flow Ruby Parser C Ruby Universal Parser ast_to_hash kanayago_parse Kanayago.parse Universal Parser ast_to_hash kanayago_parse Kanayago.parse Given Ruby Code with String Pass to Ruby Code Return AbstractSyntaxTree Given AbstractSyntaxTree Return Converted Hash Pass to Hash
37. A few patches applied Wrapping a function Exporting a struct
38. Wrapping a function Wrapping some functions was necessary to prevent errors caused by undefined symbols static void * xmalloc_mul_add(size_t x, size_t y, size_t z) { return rb_xmalloc_mul_add(x, y, z); }
39. Exporting a struct It was required because of the use of the rb_parser_params_new function struct ruby_parser { rb_parser_t *parser_params; enum lex_type type; union { struct lex_pointer_string lex_str; struct { VALUE file; } lex_io; struct { VALUE ary; } lex_array; } data; };
40. (4)Future Prospects and Issues
41. I’ll nurture it like a bonsai
42. Issues Support for more AST nodes Avoid patching Ruby’s Parser Want to remove dependency on the Universal Parser adapter provided by Ruby
43. Support for more AST Expand AST support Kanayago.parse(<<~CODE) {name: :kanayago, method: :parse} in {name: } CODE #=> {:NODE_SCOPE=>{:args=>nil, :body=>false}}
44. Avoid patching I want to submit the function wrap as a patch to Ruby core Considering better approaches to struct export
45. Adapter dependency SEGV occurs when the adapter is modified So, I want to build a custom adapter
46. Conclusion With effort, Ruby’s Parser can be used as a Universal Parser And you can provide it as a gem
47. Finally We can use Ruby’s Parser as a Universal Parser Let’s all make more use of Ruby’s Parser!