Slide 1
Slide 1 text
Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Make Parsers Compatible
Using Automata Learning
Slide 2
Slide 2 text
Hiroya Fujinami / @makenowjust
• Ph.D student
a
t H
a
suo L
a
b.,
NII (N
a
tion
a
l Institute of Inform
a
tics)
• Working p
a
rt-time
a
t STORES, Inc.
• Rese
a
rcher on
a
pplic
a
tions of
a
utom
a
t
a a
nd form
a
l l
a
ngu
a
ges
• Ruby committer
• Recent topic:
a
utom
a
t
a
le
a
rning
2
Birthd
a
y: 4/15
Slide 3
Slide 3 text
Hiroya Fujinami / @makenowjust
• Ph.D student
a
t H
a
suo L
a
b.,
NII (N
a
tion
a
l Institute of Inform
a
tics)
• Working p
a
rt-time
a
t STORES, Inc.
• Rese
a
rcher on
a
pplic
a
tions of
a
utom
a
t
a a
nd form
a
l l
a
ngu
a
ges
• Ruby committer
• Recent topic:
a
utom
a
t
a
le
a
rning
3
Slide 4
Slide 4 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Talk Summary
4
I found the comp
a
tibility issue (BUG) between Prism
a
nd p
a
rse.y
using
a
utom
a
t
a
theory.
$ ruby --parser=prism \
'x = ("a":); p x'
:a
BUG
Applying
a
utom
a
t
a
theory,
we need to obt
a
in
a
utom
a
t
a
corresponding to p
a
rsers.
For this purpose, we employ
a
utom
a
t
a
le
a
rning.
Slide 5
Slide 5 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Two Ruby Parsers:
parse.y and Prism
5
Slide 6
Slide 6 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
parse.y and Prism
Two Ruby Parsers
• Ruby (CRuby) h
a
s two p
a
rsers since 3.3.
• p
a
rse.y
• Prism (def
a
ult since 3.4)
• They c
a
n be switched with --parser option.
• We c
a
n see the p
a
rse tree with --dump=parsetree option.
6
Slide 7
Slide 7 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
traditional Ruby parser
parse.y
• Built on yacc (bison).
• Recently, bison is repl
a
ced with lrama (p
a
rser gener
a
tor written in Ruby).
7
$ ruby --parser=parse.y --dump=parsetree -e 'p 1'
###########################################################
## Do NOT use this node dump for any purpose other than ##
## debug and research. Compatibility is not guaranteed. ##
###########################################################
# @ NODE_SCOPE (id: 3, line: 1, location: (1,0)-(1,3))
# +- nd_tbl: (empty)
# +- nd_args:
# | (null node)
# +- nd_body:
$ wc -l parse.y
16023 parse.y
Slide 8
Slide 8 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
fresh Ruby parser implementation
Prism
• Written in C. F
a
ster th
a
n p
a
rse.y.
• Provides the better API to use it
a
s
a
libr
a
ry.
8
$ ruby --parser=prism --dump=parsetree -e 'p 1'
@ ProgramNode (location: (1,0)-(1,3))
+-- locals: []
+-- statements:
@ StatementsNode (location: (1,0)-(1,3))
+-- body: (length: 1)
+-- @ CallNode (location: (1,0)-(1,3))
+-- CallNodeFlags: ignore_visibility
+-- receiver: nil
+-- call_operator_loc: nil
+-- name: :p
Slide 9
Slide 9 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Two or More Parsers...
• Some libr
a
ries for p
a
rsing Ruby progr
a
ms:
• RubyVM::AbstractSyntaxTree
• ripper
• syntax_tree
• parser gem
• ...
9
Slide 10
Slide 10 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Great Parsers Era
10
Slide 11
Slide 11 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Q. Are they compatible?
11
Slide 12
Slide 12 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Solutions to Compatibility Issues
• M
a
ny unit tests
• Tests with re
a
l-world gems
• Fuzzing
• However, they
a
re not exh
a
ustive...
12
RubyGems libAFL
Slide 13
Slide 13 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Solutions to Compatibility Issues
13
theoretical way
Let's use
a
utom
a
t
a
theory!
The comp
a
tibility issue c
a
n be solved exh
a
ustively
using
a
utom
a
t
a
theory.
Tod
a
y's m
a
in topic
Slide 14
Slide 14 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Automata Theory
14
Slide 15
Slide 15 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Automaton
15
q0
q1
0 0
1
Def.
(Q, Σ, q0
, F, δ)
/0*1(0*10*1)*/
Slide 16
Slide 16 text
• Autom
a
ton = regul
a
r expression ᴷ
a
set of strings
• Set oper
a
tions
a
re e
ff
ectively comput
a
ble:
• intersection , union , di
ff
erence
• symmetric di
ff
erence (XOR)
A ∩ B A ∪ B A∖B
A △ B = (A∖B) ∪ (B∖A)
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Automata Theory
16
A △ B
Slide 17
Slide 17 text
• , ,
•
Q = QA
× QB
q0
= (q0,A
, q0,B
) δ((qA
, qB
), σ) = (δ(qA
, σ), δ(qB
, σ))
F = FA
× FB
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Automata Theory
17
A ∩ B
how to compute intersection (AND)
q0
q1
0 0
1
p0
p1
1 1
0
(q0
, p0
) (q0
, p1
)
0
(q1
, p0
) (q1
, p1
)
0
1 1
∩ =
Slide 18
Slide 18 text
• , ,
•
Q = QA
× QB
q0
= (q0,A
, q0,B
) δ((qA
, qB
), σ) = (δ(qA
, σ), δ(qB
, σ))
F = (FA
× (QB
∖FB
)) ∪ ((QA
∖FB
) × FB)
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Automata Theory
18
A △ B
how to compute symmetric di
ff
erence (XOR)
q0
q1
0 0
1
p0
p1
1 1
0
(q0
, p0
) (q0
, p1
)
0
(q1
, p0
) (q1
, p1
)
0
1 1
△ =
Slide 19
Slide 19 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Automata and Parsers
• is
a
n
a
utom
a
ton corresponding to
a
p
a
rser :
•
a
ccepts
a
word c
a
n p
a
rse
a
source code
Ap
p
Ap
w ⟺ p w
19
•
a
ccepts Prism.parse(w).success?
•
a
ccepts !Ripper.sexp(w).nil?
A
𝖯
𝗋
𝗂
𝗌 𝗆
w ⟺
A
𝗉
𝖺
𝗋 𝗌
𝖾
.
𝗒
w ⟺
Slide 20
Slide 20 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Automata and Parsers
• If the symmetric di
ff
is empty,
they
a
re comp
a
tible.
• If not, is
a
counterex
a
mple of comp
a
tibility.
A
𝖯
𝗋
𝗂
𝗌 𝗆
△ A
𝗉 𝖺
𝗋
𝗌 𝖾
.
𝗒
w ∈ A
𝖯 𝗋
𝗂 𝗌 𝗆
△ A
𝗉 𝖺
𝗋
𝗌 𝖾
.
𝗒
20
•
a
ccepts Prism.parse(w).success?
•
a
ccepts !Ripper.sexp(w).nil?
A
𝖯
𝗋
𝗂
𝗌 𝗆
w ⟺
A
𝗉
𝖺
𝗋 𝗌
𝖾
.
𝗒
w ⟺
A △ B
Slide 21
Slide 21 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
How to Find Compatibility Issues
21
Compute
A
𝖯 𝗋
𝗂 𝗌 𝗆
△ A
𝗉
𝖺
𝗋 𝗌 𝖾
.
𝗒
Prism
p
a
rse.y
q1
q1
A
𝖯𝗋𝗂 𝗌𝗆
A
𝗉𝖺𝗋𝗌
𝖾
.
𝗒
empty
not empty
no comp
a
tible
✨ issues ✨
report
the issue
obt
a
in
A
𝖯
𝗋 𝗂
𝗌 𝗆
obt
a
in
A
𝗉
𝖺 𝗋
𝗌 𝖾
.
𝗒
Q. How do we obt
a
in ?
Ap
Slide 22
Slide 22 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
A. Automata Learning
22
Q. How do we obtain ?
Ap
Slide 23
Slide 23 text
• Algorithm to infer
a
n
a
utom
a
ton from
a
bl
a
ck-box system.
• Angluin's L* is
a
typic
a
l
a
lgorithm for
a
utom
a
t
a
le
a
rning.
q1
q2
0 0
1
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Automata Learning
23
?
D
a
n
a
Angluin
Slide 24
Slide 24 text
te
a
cher
query
"01"
"11"
a
ccept
not
a
ccept
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Angluin's L*
• Angluin's L* is
a
ctive
a
utom
a
t
a
le
a
rning
a
lgorithm.
• Given te
a
cher queries inste
a
d of te
a
cher d
a
t
a
.
• 㲗 p
a
ssive
a
utom
a
t
a
le
a
rning (e.g. RPNI)
24
Active P
a
ssive
te
a
cher d
a
t
a
"01"
a
ccept
"11" not
a
ccept
…
active automata learning
Slide 25
Slide 25 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
two teacher queries: MQ and EQ
Angluin's L*
25
Membership query
𝖬
𝖰
(w)
Equiv
a
lence query
𝖤 𝖰
(H)
Checks whether is
a
ccepted.
w
Checks the inferred
a
utom
a
ton is comp
a
tible with the t
a
rget system.
If not, it returns
a
counterex
a
mple .
H
w
Is "0101"
a
ccepted?
Yes / No
Is this comp
a
tible with
the t
a
rget system?
Yes / cex w
q
0,1
Slide 26
Slide 26 text
T
a
rget system T
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
similar to Akinator
Angluin's L*
26
Membership query (MQ)
Equiv
a
lence query (EQ)
Angluin's L*
Is "0101"
a
ccepted?
I think of
q
0,1
cex w
Collect info.
Yes
Info. enough
Finish le
a
rning
Yes
/ No
Yes
/ cex w
Slide 27
Slide 27 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
How to Find Compatibility Issues
27
Compute
A
𝖯 𝗋
𝗂 𝗌 𝗆
△ A
𝗉
𝖺
𝗋 𝗌 𝖾
.
𝗒
Prism
p
a
rse.y
q1
A
𝖯𝗋 𝗂
𝗌 𝗆
q1
A
𝗉𝖺𝗋𝗌
𝖾
.
𝗒
empty
not empty
no comp
a
tible
✨ issues ✨
report
the issue
obt
a
in
A
𝖯
𝗋 𝗂
𝗌 𝗆
obt
a
in
A
𝗉
𝖺 𝗋
𝗌 𝖾
.
𝗒
a
utom
a
t
a
le
a
rning
te
a
cher
queries
for Prism
te
a
cher
queries
for p
a
rse.y
Q. How?
Slide 28
Slide 28 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
How to Obtain Teacher Queries
28
Membership query
𝖬
𝖰
(w)
Equiv
a
lence query
𝖤 𝖰
(H)
We c
a
n use the condition of .
Ap
We c
a
n obt
a
in from
using r
a
ndom testing or conform
a
nce testing (W-method).
𝖤
𝖰
(H)
𝖬
𝖰
(w)
• for Prism: Prism.parse(w).success?
• for p
a
rse.y: !Ripper.sexp(w).nil?
𝖬 𝖰
(w)
a
ccepts
H w
⟺
r
a
ndom
gener
a
tor
"0101"
"011"
"11001"
Slide 29
Slide 29 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
How to Find Compatibility Issues
29
Compute
A
𝖯 𝗋
𝗂 𝗌 𝗆
△ A
𝗉
𝖺
𝗋 𝗌 𝖾
.
𝗒
Prism
p
a
rse.y
q1
A
𝖯𝗋 𝗂
𝗌 𝗆
q1
A
𝗉𝖺𝗋𝗌
𝖾
.
𝗒
empty
not empty
no comp
a
tible
✨ issues ✨
report
the issue
obt
a
in
A
𝖯
𝗋 𝗂
𝗌 𝗆
obt
a
in
A
𝗉
𝖺 𝗋
𝗌 𝖾
.
𝗒
a
utom
a
t
a
le
a
rning
te
a
cher
queries
for Prism
te
a
cher
queries
for p
a
rse.y
Q. Is this possible?
Slide 30
Slide 30 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Angluin's L* is an algorithm
for regular language.
30
L* is for here.
regul
a
r
l
a
ngu
a
ge
context-free
l
a
ngu
a
ge
Chomsky
hier
a
rchy
No
a
m Chomsky (1928-)
Slide 31
Slide 31 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Unfortunately, Ruby's grammar is
not in regular language.
31
L* is for here.
regul
a
r
l
a
ngu
a
ge
context-free
l
a
ngu
a
ge
Chomsky
hier
a
rchy
Ruby's gr
a
mm
a
r
is here.
Or, out of there.
No
a
m Chomsky (1928-)
k
a
neko.y
Slide 32
Slide 32 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Automata Extensions
32
stoch
a
stic system
(M
a
rkov decision process)
0.3
0.7
a
symbolic
a
utom
a
ton
σ = a
σ ≠ a
register
a
utom
a
ton
q0
qx
𝖺 𝖼
𝗍
(x)
visibly pushdown
a
utom
a
ton
q0
')' / ( , '(')
q0
They
a
re le
a
rn
a
ble!
Slide 33
Slide 33 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Visibly Push-down Automaton (VPA)
• VPA is
a
restricted version of PDA (Push-Down Auotom
a
t
a
).
• In VPA, speci
f
ic ch
a
r
a
cters c
a
n only m
a
nipul
a
te the st
a
ck (visibly).
• e.g. ( c
a
n push
a
nd ) c
a
n pop.
• Reg VPL CFG
• VPL c
a
nnot represent
a
ll Ruby synt
a
x, but it is better th
a
n Reg.
⊆ ⊆
33
regul
a
r
l
a
ngu
a
ge
regul
a
r
l
a
ngu
a
ge
context-free
l
a
ngu
a
ge
VPL
Chomsky
hier
a
rchy
Slide 34
Slide 34 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
How to Find Compatibility Issues
34
Compute
A
𝖯 𝗋
𝗂 𝗌 𝗆
△ A
𝗉
𝖺
𝗋 𝗌 𝖾
.
𝗒
Prism
p
a
rse.y
q1
A
𝖯𝗋 𝗂
𝗌 𝗆
q1
A
𝗉𝖺𝗋𝗌
𝖾
.
𝗒
empty
not empty
no comp
a
tible
✨ issues ✨
report
the issue
obt
a
in
A
𝖯
𝗋 𝗂
𝗌 𝗆
obt
a
in
A
𝗉
𝖺 𝗋
𝗌 𝖾
.
𝗒
a
utom
a
t
a
le
a
rning
te
a
cher
queries
for Prism
te
a
cher
queries
for p
a
rse.y
Prism.parse(w).success?
!Ripper.sexp(w).nil?
L* extended for VPL
Slide 35
Slide 35 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Let's Try Automata Learning!
35
Slide 36
Slide 36 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Lernen
•
g
m
a
kenowjust/lernen
• implements m
a
ny
a
lgorithms for m
a
ny kinds of
a
utom
a
t
a
:
• L*, KV, DHC, L#
• for DFA, Moore, Me
a
ly, VPA, SPA.
• E
a
sy to use.
36
an automata learning library written in Ruby
Slide 37
Slide 37 text
• C
a
ll Lernen.learn!
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Easy to use
37
Learnen
automaton = Lernen.learn(alphabet: [0, 1]) do |word|
word.count(1) % 2 == 0
end
puts automaton.to_dot # =>
Slide 38
Slide 38 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Find Compatibility Issues
• It is h
a
rd to infer
a
n
a
utom
a
ton of the entire p
a
rser.
• Therefore, we limits only four ch
a
r
a
cters th
a
t c
a
n be used.
• "a", :, (, )
• (
a
nd ) c
a
n be nested.
38
using Lernen
Slide 39
Slide 39 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Find Compatibility Issues
39
Code
alphabet, call_alphabet, return_alphabet =
%w["a" :], %w[(], %w[)]
ripper_vpa = Lernen.learn(
alphabet:, call_alphabet:, return_alphabet:
) do |word|
!Ripper.sexp(word.join).nil?
end
prism_vpa = Lernen.learn(
alphabet:, call_alphabet:, return_alphabet:
) do |word|
Prism.parse(word.join).success?
end
Infer
a
nd
A
𝗉
𝖺
𝗋 𝗌
𝖾
.
𝗒
A
𝖯
𝗋
𝗂
𝗌 𝗆
Slide 40
Slide 40 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Find Compatibility Issues
40
Code
sep_word = Lernen::Automaton::VPA.find_separating_word(
alphabet, call_alphabet, return_alphabet,
ripper_vpa, prism_vpa
)
Compute A
𝗉
𝖺
𝗋 𝗌
𝖾
.
𝗒
△ A
𝖯 𝗋
𝗂 𝗌 𝗆
Slide 41
Slide 41 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Find Compatibility Issues
• Two VPAs
a
nd
a
re obt
a
ined:
• is 6 st
a
tes
a
nd is 9 st
a
tes.
A
𝗉
𝖺
𝗋 𝗌
𝖾
.
𝗒
A
𝖯
𝗋
𝗂
𝗌 𝗆
A
𝗉
𝖺
𝗋 𝗌
𝖾
.
𝗒
A
𝖯
𝗋
𝗂
𝗌 𝗆
41
Result
Slide 42
Slide 42 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Find Compatibility Issues
42
Result
A
𝗉𝖺𝗋𝗌
𝖾
.
𝗒
A
𝖯𝗋𝗂 𝗌𝗆
Slide 43
Slide 43 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Find Compatibility Issues
• Two VPAs
a
nd
a
re obt
a
ined:
• is 6 st
a
tes
a
nd is 9 st
a
tes.
• Computing ,
a
counterex
a
mple '("a":)' is found.
• This bug is reported (ruby/prism#3035),
a
nd
f
ixed
a
t Prism v1.1.0.
A
𝗉 𝖺
𝗋 𝗌
𝖾
.
𝗒
A
𝖯 𝗋
𝗂 𝗌 𝗆
A
𝗉 𝖺
𝗋 𝗌
𝖾
.
𝗒
A
𝖯 𝗋
𝗂 𝗌 𝗆
A
𝖯 𝗋
𝗂 𝗌 𝗆
△ A
𝗉
𝖺 𝗋
𝗌 𝖾
.
𝗒
43
Result
Autom
a
t
a
le
a
rning
is re
a
l!
$ ruby --parser=prism \
'x = ("a":); p x'
:a BUG
Slide 44
Slide 44 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Find Compatibility Issues
44
Result
Autom
a
t
a
le
a
rning
is re
a
l!
You c
a
n try
a
utom
a
t
a
le
a
rning!
Ex
a
mple use c
a
se:
When
a
Regexp is rewritten into
a
progr
a
m,
a
re they the s
a
me?
s =~ /AAA.*AAA\z/ =
s.contains("AAA") &&
s.end_with?("AAA")
Slide 45
Slide 45 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Future Work
45
Slide 46
Slide 46 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Towards Inferring the Entire Parsers
46
It is h
a
rd bec
a
use the entire p
a
rsers
a
re
L
a
rge
in the st
a
tes size
a
nd
the
a
lph
a
bet size
Complex
from Ruby's gr
a
mm
a
r
a
nd
a
nother problem:
Output
Slide 47
Slide 47 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Towards Inferring the Entire Parsers
47
L
a
rge
in the st
a
te size
a
nd
the
a
lph
a
bet size
MQ complexity of L* is
O(kn2)
is the
a
lph
a
bet size.
k
is the st
a
te size.
n
As grows, grows
a
t
a
n
a
cceler
a
ted r
a
te.
n n2
Slide 48
Slide 48 text
Complex
from Ruby's gr
a
mm
a
r
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Towards Inferring the Entire Parsers
48
Ruby's gr
a
mm
a
r is too complex.
e.g. v
a
ri
a
ble n
a
mes
a
re conetxt-sensitive.
foo /1
synt
a
x error
foo = 1
foo /1
v
a
lid
So f
a
r, there is no known
a
utom
a
t
a
le
a
rning for this kind of gr
a
mm
a
r.
Slide 49
Slide 49 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Towards Inferring the Entire Parsers
49
Current de
f
inition:
Ap
Output
a
ccepts
a
word c
a
n p
a
rse
a
source code
Ap
w ⟺ p w
Wh
a
t we w
a
nt:
output
a
g
a
inst p
a
rse result
a
g
a
inst by
Ap
[w] w = w p
M
a
ybe tree tr
a
nsducer is useful...?
Slide 50
Slide 50 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Towards Inferring the Entire Parsers
50
It is h
a
rd bec
a
use the entire p
a
rsers
a
re
L
a
rge
in the st
a
tes size
a
nd
the
a
lph
a
bet size
Complex
from Ruby's gr
a
mm
a
r
a
nd
a
nother problem:
Output
M
a
ny rese
a
rch topics
a
re still rem
a
ins
Slide 51
Slide 51 text
Make Parsers Compatible Using Automata Learning - Hiroya Fujinami @ RubyKaigi 2025 Matsuyama
Conclusion
51
1. Ruby's gr
a
mm
a
r is too complex to be inferred.
2. Ple
a
se remember th
a
t there
a
re
a
lgorithms th
a
t infer
a
utom
a
t
a
2. in the m
a
nner of the Akin
a
tor, n
a
mely
a
utom
a
t
a
le
a
rning
a
nd L*.
Th
a
nk you for listening!
$ ruby --parser=prism \
'x = ("a":); p x'
:a BUG