Unexplored Region - parse.y -

RubyKaigi 2023 LT
https://rubykaigi.org/2023/presentations/lt/

Transcript

1. Unexplored Region - parse.y - May 11, 2023 in RubyKaigi 2023 @yui-knk Yuichiro Kaneko
2. RubyKaigi 2023
3. 5 sessions for Parser Day 1 Day 2 Day 3 LT
4. This is the time known as the “Great Parser Era” !
5. Everyone has interest in Parser
6. Rumors about parse.y
7. Demon Castle parse.y (2017) Rumors about parse.y
8. Monstrous lex_state (2017) Demon Castle parse.y (2017) Rumors about parse.y
9. Monstrous lex_state (2017) Demon Castle parse.y (2017) parse.y is “hell” (2019) Rumors about parse.y
10. Monstrous lex_state (2017) Demon Castle parse.y (2017) parse.y is “hell” (2019) The current parse.y is a hell (2021) Rumors about parse.y
11. Rumors about parse.y
12. Rumors about parse.y This is NOT a parser
13. Rumors about parse.y This is NOT a parser This is a parser
14. This is a quick & safe tour of unexplored region, “parse.y"
15. In 5 minute
16. About me • Yuichiro Kaneko • yui-knk (GitHub) / spikeolaf (Twitter) • Treasure Data • Engineering Manager of Applications Backend • CRuby committer • Mainly develop parser related features other than new syntax • RubyVM::AbstractSyntaxTree (2018, Ruby 2.6) • keep_tokens option (2022, Ruby 3.2) • error_tolerant option (2022, Ruby 3.2)
17. “parse.y” Who's Who in 2023
18. “parse.y” Who's Who in 2023 The patch monster
19. “parse.y” Who's Who in 2023 The patch monster The Creator of Ruby
20. “parse.y” Who's Who in 2023 The patch monster The Creator of Ruby The Organizer of TRICK
21. “parse.y” Who's Who in 2023 The patch monster The Creator of Ruby The Organizer of TRICK Me
22. I’m the weakest of the Big Four
23. Beginner Course Let's understand outline of “parse.y”
24. #1: The location of parse • Goto https://github.com/ruby/ruby • “parse.y” is on the top level
25. #2: Size of “parse.y” • Less than 15,000 • It’s not the largest one 0 5000 10000 15000 20000 common.mk io.c gc.c parse.y compile.c string.c tag: v3_2_0
26. #3: Structure of “parse.y” C codes Declarations Grammar rules C codes
27. #3: Structure of “parse.y” Declarations Grammar rules • From L.1329 to L.6118 • About 5,000 lines
28. #4: Grammar rules
29. #4: Grammar rules Almost same as BNF
30. #4: Grammar rules Almost same as BNF
31. #4: Grammar rules Almost same as BNF Action
32. #4: Grammar rules Almost same as BNF Action Comment
33. #4: Grammar rules Almost same as BNF Action Comment NODE_IF
34. #4: Grammar rules Almost same as BNF Action Comment NODE_IF condition “i == 1”
35. #4: Grammar rules Almost same as BNF Action Comment NODE_IF condition “i == 1” body “1”
36. #4: Grammar rules Almost same as BNF Action NODE_IF Comment condition “i == 1” body “1” else “0”
37. #4: Grammar rules Almost same as BNF Action NODE_IF Comment condition “i == 1” body “1” else “0” At sign means location
38. How's this? It is not so dif fi cult, right?
39. Guidebooks • shioimm/coe401_. “ͨͷ͍͠RubyͷߏจղੳπΞʔ”, March 2023. https:// speakerdeck.com/coe401_/tanosiirubynogou-wen-jie-xi-tua • aamine. “Rubyιʔείʔυ׬શղઆ” ୈ 2 ෦ʮߏจղੳʯ, July 2004. • https://i.loveruby.net/ja/rhg/book/ [JA] • https://ruby-hacking-guide.github.io/ [EN]
40. #5: dump=y option
41. Do I understand all of them?
42. Almost, unfortunately
43. #6: YFLAGS • make YFLAGS=" --report=states,itemsets,lookaheads,solved" miniruby • “parse.tmp.output" is generated
44. Now you’re a beginner of parser
45. Intermediate Course Let's understand parser_params and lexer state
46. “If you fi nd both data and code, you should fi rst investigate the data structure.” Introduction “Understanding data structure” https://ruby-hacking-guide.github.io/intro.html Iron Rule #1
47. What one should do is think toward speci fi c goals: “This part is needed to solve this task” “This code is for overcoming this problem” Chapter 11 Finite-state scanner “Understanding data structure” https://ruby-hacking-guide.github.io/contextual.html Iron Rule #2
48. #7: struct parser_params • Not small …
49. #7: parser_params 2004 • Subscribe now! https://github.com/ruby/ruby/commit/e77ddaf0d1d421da2f655832a45f237558e23115
50. #8: Lexer Buffer • Hospitality !!! • (But this the only diagram…)
51. #9: lastline / nextline • There is two lines, lastline and nextline • Why?
52. For here document?
53. NO
54. #9: lastline / nextline • Ruby sometimes ignores NL (internally this is called “tIGNORED_NL”) • Need to know which token appears next to determine ignore NL or not NL is ignored
55. #9: lastline / nextline 1. A pointer is on the end of the line
56. #9: lastline / nextline 1. A pointer is on the end of the line 2. Get next line
57. #9: lastline / nextline 1. A pointer is on the end of the line 2. Get next line 3. Check next token to determine generate NL
58. #9: lastline / nextline 1. A pointer is on the end of the line 2. Get next line 3. Check next token to determine generate NL 4. Go back to previous line and hold next line
59. #9: lastline / nextline 1. A pointer is on the end of the line 2. Get next line 3. Check next token to determine generate NL 4. Go back to previous line and hold next line 5. “nextline” is set to “lastline”
60. #10: parent_iseq • It’s weird because ISeq is a data structure generated from AST
61. #10: parent_iseq https://www.slideshare.net/mametter/trick-2022-results • This is a trick used in TRICK 2022
62. #10: parent_iseq • #eval binds the context around it • “Context” is represented as “parent_iseq”
63. #eval is evil
64. #11: Growing lex_state v3_2_0
65. #11: Growing lex_state v0_49 v3_2_0 https://github.com/ruby/ruby/blob/v0_49/parse.y • I love v0_49
66. Is this actually needed? • They are needed to distinguish these codes
67. Is this actually needed? • They are needed to distinguish these codes
68. Is this actually needed? • They are needed to distinguish these codes
69. Is this actually needed? • They are needed to distinguish these codes
70. Is this actually needed? • They are needed to distinguish these codes
71. Is this actually needed? • They are needed to distinguish these codes
72. How's this? It is not so dif fi cult, right?
73. Wrong, it’s complex.
74. Open Question: How to sort out lex_state
75. Advanced Course Let's understand actions
76. #4: Grammar rules Action
77. #12: Action is a local variable • Some rules has a lot of actions in the middle of right hand sides
78. #12: Action is a local variable “compstmt” is almost all of Ruby’s syntax
79. String interpolation is very useful, right?
80. With great power comes great responsibility
81. #12: Action is a local variable Save current values Restore values
82. #13: Semantic value is reusable
83. #13: Semantic value is reusable Reuse “->” semantic value because nobody uses it Restore the value
84. #14: lex_ctxt nonterminal symbol • This does not consume any tokens. This exists for just getting current value of “struct lex_context ctxt”!
85. Conclusions It's fun to read parse.y
86. See you next time at “parse.y”
87. Guidebooks • shioimm/coe401_. “ͨͷ͍͠RubyͷߏจղੳπΞʔ”, March 2023. https:// speakerdeck.com/coe401_/tanosiirubynogou-wen-jie-xi-tua • aamine. “Rubyιʔείʔυ׬શղઆ” ୈ 2 ෦ʮߏจղੳʯ, July 2004. • https://i.loveruby.net/ja/rhg/book/ [JA] • https://ruby-hacking-guide.github.io/ [EN] • େງ ३ “LRߏจղੳͷݪཧ”, Feb 2014. https://www.jstage.jst.go.jp/article/jssst/ 31/1/31_1_30/_pdf/-char/ja • A.V. ΤΠϗ ଞ “ίϯύΠϥ[ୈ2൛] ݪཧɾٕ๏ɾπʔϧ” αΠΤϯεࣾ