Reading and improving Pattern Matching in Ruby

Reading and improving
Pattern Matching in Ruby
2023.MAY YUKI TORII IN RUBYKAIGI

View Slide

Advertising Time
W
2
The sponsor Book
will be published 2023.May.17
 
from O’Reilly Japan
 
#yuandmagic
Event
#authorsrb

View Slide

Who am I?
Name: Yuki Torii

An ordinary Ruby programmer

Talked “Pattern Matching in Ruby” at RubyKaigi
2017. Implemented Toy pattern matching inRuby
 
(After that “Real” pattern matching is introduced by
Tsujimoto-san from Ruby2.7, 2019)
3

View Slide

The Goal of this talk
Sparking interest in the internals of Ruby among
ordinary Ruby programmers like myself.
4

View Slide

Agenda
Walk through the specs of pattern matching in Ruby

Know how to see the instructions for YALV (What’s the
instructions like?)

See The sequence of pattern matching built by the
instructions.

Try to speed up Pattern Matching of Array pattern
5

View Slide

What’s Pattern Matching in
Ruby like?

View Slide

Basic Pattern Matching styles
case
 
in
 
…
 
in
 
…

else

…
 
end
case / in expression user_ids = [3, 5]
 
case user_ids
 
in [1, 2]
 
:admins
 
in [3, 5]
 
:managers
 
else

:other_users
 
end

=> “managers”

View Slide

Basic Pattern Matching styles
=>
=> operator
in operator
in
response = {status: 404, message:
“Not Found”}
 
response => {message:}
 
p message

response => {other_key:}
 
#=> Raise NoMatchingPatternKeyError
#=> “Not Found”
“Not Found”}
 
p message if response in {message:}
 
 
response => {other_key:}

#=>nil
#=> “Not Found”

View Slide

Patterns
user_id = 1
 
p "admin" if user_id in 1
 
 
p “admin” if user_id in Integer
=> “admin”
Any RubyObject Pattern
=> “admin”

View Slide

Patterns
user_ids = [3, 5]
 
case user_ids
 
in [1, 2]
 
p “admins”
 
else

p “other users”
 
end
Hash Pattern
response = {status: 404}

case response

in {status: 404}

p “Not Found”
 
else

p response

end
=> “Not Found”
Array Pattern

View Slide

Patterns
user_ids = [5, 8, 1, 23]

p "admin is included" if user_ids in [*, 1, *]
Find Pattern
=> “admin is included”
user = {id: 2, role: "admin"}

p "user is admin" if user in {id: 1 } | {role: "admin"}
Combine pattern
=> “user is admin”

View Slide

“success!”}

case response

in {status: 404}

p “Not Found”

in {status: 200, message: }

p message

end
Patterns
Variable pattern
=> “success!”

View Slide

How is the pattern match
realized

In Ruby internal?

View Slide

From code to instructions
Ruby code

Abstract
Syntax Tree

(AST)
Instructions
Tokens
Tokenize Parse Compile
What is YARV instructions? : the instructions that Ruby virtual machine
executes.

View Slide

From code to instructions
Ruby code

Abstract
Syntax Tree

(AST)
Instructions
Tokens
the YARV instructions show how Ruby code work on VM

View Slide

How to see the instructions
On irb,

1. Compile your ruby code file and get a
RubyVM::InstructionSequence instance

2. See the instructions using #disasm
iseq = RubyVM::InstructionSequence.compile_
fi
le(“path/to/
user_ruby_code.rb“) 
 
pp iseq.disasm

View Slide

Example: Instructions of a simple
code
a = 1

a = a + 1
Code
"local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])\n"
+

"[ 1] a@0\n" +

"0000 putobject_INT2FIX_1_ ( 1)[Li]\n" +

"0001 setlocal_WC_0 a@0\n" +

"0003 getlocal_WC_0 a@0 ( 2)[Li]\n" +

"0005 putobject 2\n" +

"0007 opt_plus [CcCr]\n" +

"0009 dup\n" +


"0012 leave\n""
AST
Tokens
Instructions

View Slide

Means of instructions and stack
"local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
\n" +

"[ 1] a@0\n" +



"0003 getlocal_WC_0 a@0 ( 2)[Li]\n" +


"0007 opt_plus [CcCr]\

"0009 dup\n" +


"0012 leave\n"
Instructions
1
Stack
Put 1 on the stack
a = 1

a = a + 2
Code

View Slide

\n" +

"[ 1] a@0\n" +



"0003 getlocal_WC_0 a@0 ( 2)[Li]\n" +



"0009 dup\n" +


"0012 leave\n"
Instructions
1
Stack
a = 1

a = a + 2
Code
Pop the top of the stack and bind to local variable “a”

View Slide

\n" +

"[ 1] a@0\n" +



"0003 getlocal_WC_0 a@0 ( 2)[Li]\n" +



"0009 dup\n" +


"0012 leave\n"
Instructions
Stack
a = 1

a = a + 2
Code
Get local variable “a” ’s value and put it on stack
1

View Slide

\n" +

"[ 1] a@0\n" +



"0003 getlocal_WC_0 a@0 ( 2)[Li]\n" +



"0009 dup\n" +


"0012 leave\n"
Instructions
Stack
a = 1

a = a + 2
Code
1
Put 2 on stack
2

View Slide

\n" +

"[ 1] a@0\n" +



"0003 getlocal_WC_0 a@0 ( 2)[Li]\n" +



"0009 dup\n" +


"0012 leave\n"
Instructions
Stack
a = 1

a = a + 2
Code
3
Pop 2 and 1 then add them. Put the result

View Slide

\n" +

"[ 1] a@0\n" +



"0003 getlocal_WC_0 a@0 ( 2)[Li]\n" +



"0009 dup\n" +


"0012 leave\n"
Instructions
Stack
a = 1

a = a + 2
Code
3
Duplicate the top of the stack
3

View Slide

\n" +

"[ 1] a@0\n" +



"0003 getlocal_WC_0 a@0 ( 2)[Li]\n" +



"0009 dup\n" +


"0012 leave\n"
Instructions
Stack
a = 1

a = a + 2
Code
3
Pop the top of the stack and bind with local varialbe “a”
3

View Slide

Let’s check out the instructions of
This simple pattern matching code
user_ids = [3, 5]

case user_ids

in [1, 2]

p “admins”

en
Code

View Slide

Code and Instructions
user_ids = [3, 5]
 
case user_ids

in [1, 2]

p “admins”

en

"== disasm: #@./try.rb:1 (1,0)-(6,3)> (catch: FALSE)\n" +

"local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])\n" +

"[ 1] user_ids@0\n" +

"0000 duparray [3, 5] ( 1)[Li]\n" +

"0002 setlocal_WC_0 user_ids@0\n" +

"0004 putnil ( 4)[Li]\n" +

"0005 putnil\n" +

"0006 putobject false\n" +

"0008 putnil\n" +

"0009 putnil\n" +

"0010 getlocal_WC_0 user_ids@0 ( 3)\n" +

"0012 dup ( 4)\n" +

"0013 topn 2\n" +

"0015 branchnil 26\n" +

"0017 topn 2\n" +

"0019 branchunless 181\n" +

"0021 pop\n" +

"0022 topn 1\n" +

"0024 jump 63\n" +

"0026 dup\n" +

"0027 putobject :deconstruct\n" +

"0029 opt_send_without_block \n" +

"0031 setn 3\n" +

"0033 dup\n" +

"0034 branchif 52\n" +

"0036 putspecialobject 1\n" +

"0038 putobject \"%p does not respond to #deconstruct\"\n" +

"0040 topn 3\n" +


"0044 setn 5\n" +


"0048 setn 7\n" +

"0050 pop\n" +

"0051 pop\n" +



"0056 setn 2\n" +

"0058 dup\n" +

"0059 checktype T_ARRAY\n" +


"0063 dup\n" +

"0064 opt_length [CcCr]\n" +


"0068 opt_eq [CcCr]\n" +

"0070 dup\n" +



"0075 putobject \"%p length mismatch (given %p, expected %p)\"\n" +

"0077 topn 3\n" +

"0079 dup\n" +




"0086 setn 5\n" +


"0090 setn 7\n" +

"0092 pop\n" +

"0093 pop\n" +


"0096 dup\n" +

"0097 putobject_INT2FIX_0_\n" +

"0098 opt_aref [CcCr]\n" +


"0101 dupn 2\n" +

"0103 checkmatch 2\n" +

"0105 dup\n" +

"0106 branchif 126\n" +


"0110 putobject \"%p === %p does not return true\"\n" +

"0112 topn 3\n" +

"0114 topn 5\n" +


"0118 setn 7\n" +


"0122 setn 9\n" +

"0124 pop\n" +

"0125 pop\n" +

"0126 setn 2\n" +

"0128 pop\n" +

"0129 pop\n" +


"0132 dup\n" +




"0138 dupn 2\n" +


"0142 dup\n" +

"0143 branchif 163\n" +



"0149 topn 3\n" +

"0151 topn 5\n" +


"0155 setn 7\n" +


"0159 setn 9\n" +

"0161 pop\n" +

"0162 pop\n" +

Code
Instructions
Woops!
AST
Tokens

View Slide

Code and Instructions
user_ids = [3, 5]
 
case user_ids

in [1, 2]

p “admins”

en
irb(main):031:0> pp iseq.disasm

"== disasm: #@./try.rb:1 (1,0)-(6,3)> (catch: FALSE)\n" +

"local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])\n" +

"[ 1] user_ids@0\n" +

"0000 duparray [3, 5] ( 1)[Li]\n" +

"0002 setlocal_WC_0 user_ids@0\n" +

"0004 putnil ( 4)[Li]\n" +

"0005 putnil\n" +


"0008 putnil\n" +

"0009 putnil\n" +

"0010 getlocal_WC_0 user_ids@0 ( 3)\n" +

"0012 dup ( 4)\n" +

"0013 topn 2\n" +

"0015 branchnil 26\n" +

"0017 topn 2\n" +


"0021 pop\n" +

"0022 topn 1\n" +

"0024 jump 63\n" +

"0026 dup\n" +

"0027 putobject :deconstruct\n" +


"0031 setn 3\n" +

"0033 dup\n" +



"0038 putobject \"%p does not respond to #deconstruct\"\n" +

"0040 topn 3\n" +


"0044 setn 5\n" +


"0048 setn 7\n" +

"0050 pop\n" +

"0051 pop\n" +



"0056 setn 2\n" +

"0058 dup\n" +

"0059 checktype T_ARRAY\n" +


"0063 dup\n" +



"0068 opt_eq [CcCr]\n" +

"0070 dup\n" +



"0075 putobject \"%p length mismatch (given %p, expected %p)\"\n" +

"0077 topn 3\n" +

"0079 dup\n" +




"0086 setn 5\n" +


"0090 setn 7\n" +

"0092 pop\n" +

"0093 pop\n" +


"0096 dup\n" +




"0101 dupn 2\n" +


"0105 dup\n" +

"0106 branchif 126\n" +



"0112 topn 3\n" +

"0114 topn 5\n" +


"0118 setn 7\n" +


"0122 setn 9\n" +

"0124 pop\n" +

"0125 pop\n" +

"0126 setn 2\n" +

"0128 pop\n" +

"0129 pop\n" +


"0132 dup\n" +




"0138 dupn 2\n" +


"0142 dup\n" +

"0143 branchif 163\n" +



"0149 topn 3\n" +

"0151 topn 5\n" +


"0155 setn 7\n" +


"0159 setn 9\n" +

"0161 pop\n" +

Code
Instructions
AST
Tokens
Don't Panic,
 
Open compile.c

View Slide

compile.c makes instructions.
 
And more, Tsujimoto-san (who implemented pattern-
matching in rubyʣcommented how it works by ruby code
A part of compile.c A part of Tsujimoto-san’s excelent comments
if (use_rest_num) {

ADD_INSN1(ret, line_node, putobject,
INT2FIX(0)); /* allocate stack for rest_num */

ADD_INSN(ret, line_node, swap);

if (base_index) {

base_index++;

}

}

View Slide

This is the sequence of Pattern matching
read from the Instructions
 
and compile.c (with some detail omissions)
 
[Array Pattern and Hash Pattern]

View Slide

In the case of Array Pattern
user_ids = [3, 5]
 
case user_ids

in [1, 2]

:admins

en
Code
obj: [3, 5]
: [1, 2]
Does’nt Pattern have constant || has const and the const eq obj?
Does obj have #deconstruct?
Does obj#deconstruct return a Array object?
The result of obj#deconstruct’s length is
 
equal to Pattern’s one?
Check if each element of Pattern ===

each element of the result of obj#deconstruct
Y N
Type
 
Error
Match
Failed
N
N
N
N
Y
Y
Y
Y
Matching Success!
If any element is subpattern, instructions will be added recursively

View Slide

Q.What’s #deconstruct method?
A. The method to be called for inspecting the object matches Array
Pattern. It should return an Array object. (Array returns itself)
class Foo
 
def deconstruct

[1, 2]

end

end

foo = Foo.new

 
p "Matched!" if foo in [1, 2]
=> “Matched!”
class Array
 
def deconstruct

self

end

end

foo = [1, 2]

 
=> “Matched!”

View Slide

In the case of Hash Pattern
res = {status: 400}
 
case res
 
in {status: 200}
 
:success

end
Code
obj: {status: 400}
: {status: 200}
 
key_args: [:status]
Does’nt Pattern have constant or has const and the const eq obj
Does obj has #deconstruct_keys(key_args)?
Does obj#deconstruct_keys return a Hash object?
The result of obj#deconstruct_key’s keys includes all key_args?
Check if each element of Pattern matches

each element of the result of obj#deconstruct_keys
Y N
Type
 
Error
Match
Failed
N
N
N
N
Y
Y
Y
Y Match
Success!

View Slide

It’s time to try improving
Pattern Matching of Array
Pattern.

View Slide

Target: Array Pattern
class Array
 
def deconstruct

self

end

end

foo = [1, 2]

 
Does obj has #deconstruct?
 

Y
Y
Y
Array has #deconstruct by default, and return itself.

View Slide

Target: Array Pattern
 

Y
Y
Y
Are these checks and deconstruction
 
necessary for an object of Array?

View Slide

Change: Skip #deconstruct checks
when obj is a Array
The result of obj#deconstruct’s(or value of itself) length is
 

each element of the result of obj#deconstruct (or value of itself)
Y
Y
Y
Y
Is Obj an Array?
- LABEL *match_failed, *type_error, *deconstruct, *deconstructed;

+ LABEL *match_failed, *type_error, *deconstruct, *deconstructed,
*skip_deconstruct;

int i;

match_failed = NEW_LABEL(line);

type_error = NEW_LABEL(line);

deconstruct = NEW_LABEL(line);

deconstructed = NEW_LABEL(line);

+ skip_deconstruct = NEW_LABEL(line);

if (use_rest_num) {

ADD_INSN1(ret, line_node, putobject, INT2FIX(0)); /* allocate stack for
rest_num */

@@ -6439,9 +6440,19 @@ iseq_compile_pattern_each(rb_iseq_t *iseq, LINK_ANCHOR
*const ret, const NODE *c

CHECK(iseq_compile_pattern_constant(iseq, ret, node, match_failed,
in_single_pattern, base_index));

+ //skip deconstuct and use obj value when obj is array.

+ ADD_INSN(ret, line_node, dup);

+ ADD_INSN1(ret, line_node, checktype, INT2FIX(T_ARRAY));

+ ADD_INSNL(ret, line_node, branchif, skip_deconstruct);

CHECK(iseq_compile_array_deconstruct(iseq, ret, node, deconstruct,
deconstructed, match_failed, type_error, in_single_pattern, base_index,
use_deconstructed_cache));

+ ADD_LABEL(ret, skip_deconstruct);
N
diff of compile.c

View Slide

Change2: unify length check to
new instruction when obj is a Array
 

Y
Y
Y
Y
Is Obj an Array?
 
&& pattern.length eq obj.length
ADD_INSN(ret, line_node, dup);

- ADD_INSN1(ret, line_node, checktype, INT2FIX(T_ARRAY));

+ ADD_INSN2(ret, line_node, checkpatarray, INT2FIX(min_argc),
INT2FIX(apinfo->rest_arg != NULL));

ADD_INSNL(ret, line_node, branchif, skip_deconstruct);

CHECK(iseq_compile_array_deconstruct(iseq, ret, node,
deconstruct, deconstructed, match_failed, type_error, in_single_pattern,
base_index, use_deconstructed_cache));

- ADD_LABEL(ret, skip_deconstruct);

-

ADD_INSN(ret, line_node, dup);

ADD_SEND(ret, line_node, idLength, INT2FIX(0));

@@ -6464,6 +6477,8 @@ iseq_compile_pattern_each(rb_iseq_t *iseq,
LINK_ANCHOR *const ret, const NODE *c

}

ADD_INSNL(ret, line_node, branchunless, match_failed);

+ ADD_LABEL(ret, skip_deconstruct);

+
diff of compile.c

N

View Slide

Change2: unify length check to
new instruction when obj is a Array
 

Y
Y
Y
Y
Obj is a Array?
 
&& pattern.length eq obj.length
+DEFINE_INSN

+checkpatarray

+(rb_num_t length, rb_num_t remain_args)

+(VALUE obj)

+(VALUE val)

+{

+ VALUE ary;

+

+ if (RB_TYPE_P(obj, T_ARRAY) &&

+ (remain_args ? (rb_num_t)RARRAY_LEN(obj) >= length :
(rb_num_t)RARRAY_LEN(obj) == length)) {

+

+ // TODO: rede
fi
nition check

+ val = Qtrue;

+ }

+ else {

+ val = Qfalse;

+ }

+}

+
diff of insns.def
N

View Slide

Easier said than done
Debugging:
 
Writing out all of the instructions
 
and simulating stacks by hand.
 
Feeling with the heart of VM.

View Slide

Measurement
require 'benchmark'

obj = [:sym]

N = 1_000_000

Benchmark.bm{|x|

x.report{

N.times{

r = obj in [:sym]

r = obj in [:sym]

r = obj in [:sym]

r = obj in [:sym]

r = obj in [:sym]

}

}

}
user system total real

0.606022 0.000293 0.606315 ( 0.606855)
user system total real

0.294262 0.000462 0.294724 ( 0.295170)
After
Before
Achieved twice the speed!!

View Slide

Remaining challenge: checking
 
the case Array#deconstruct is overridden
Should not match, but match.
DEFINE_INSN

checkpatarray

(rb_num_t length, rb_num_t remain_args)

(VALUE obj)

(VALUE val)

{

VALUE ary;

if (RB_TYPE_P(obj, T_ARRAY) &&

(remain_args ? (rb_num_t)RARRAY_LEN(obj) >= length :
(rb_num_t)RARRAY_LEN(obj) == length)) {

// TODO: rede
fi
nition check

val = Qtrue;

}

else {

val = Qfalse;

}

}

class Array
 
def deconstruct

[1, 3]

end

end

foo = [1, 2]

 

View Slide

Do not override
“deconstruct” of Array!!!!!

View Slide

Summary of this talk
Instructions show how Ruby code works.

Feel with the heart of VM

Do not override Array#deconstruct
43

View Slide

What is fun of knowing
Ruby internal?

View Slide

We can know the favorite feature of the language
deeply

How it works

Various efforts to make execution better(ex.
Cache of deconstructed result)

If there is a chance, we can make the feature better
a bit.

View Slide

In other words, we can love Ruby more.

We; ordinary Ruby programmers can use Ruby
every day with much more love.

Let’s start reading your favorite feature in Ruby
Internal.

View Slide

Enjoy!

View Slide

Reading and improving Pattern Matching in Ruby

Reading and improving Pattern Matching in Ruby

やきとりい

More Decks by やきとりい

Other Decks in Programming

Featured

Transcript