Ruby 3x3 に向けたRubyのベンチマーク環境と成果 / Ruby Association Grant

by Takashi Kokubun

Slide 1

Slide 1 text

Ruby 3x3 ʹ޲͚ͨ RubyͷϕϯνϚʔΫ؀ڥͱ੒Ռ RubyΞιγΤʔγϣϯ։ൃॿ੒ ੒Ռใࠂձ @k0kubun

Slide 2

Slide 2 text

ࣗݾ঺հ GitHub, Twitter: k0kubun Rubyίϛολ: JITίϯύΠϥͷ։ൃ, ERBͷϝϯςφϯε ࡞ͬͨ: Hamlit, MItamae, md2key, xremap, LLRB

Slide 3

Slide 3 text

ࠓ೔࿩͢͜ͱ 1. ຊϓϩδΣΫτ͸ԿΛղܾ͔ͨ͠ 2. ϕϯνϚʔΫϝτϦΫεͷϞσϦϯά 3. ϓϥάΠϯΞʔΩςΫνϟ 4. ϓϩδΣΫτऴྃޙͷ׆ಈ

Slide 4

Slide 4 text

1. ຊϓϩδΣΫτ͸ԿΛղܾ͔ͨ͠

Slide 5

Slide 5 text

͜Ε·ͰͷRubyͷϕϯνϚʔΫ؀ڥ

Slide 6

Slide 6 text

ϒϩοΫݺͼग़͠ͷΦʔόʔϔου໰୊ ඪ४ͷbenchmarkϥΠϒϥϦ΋benchmark-ips΋ɺܭଌ୯Ґ͕ϒ ϩοΫʹͳ͍ͬͯΔ ϒϩοΫݺͼग़͠͸ී௨ͷϝιουݺͼग़͠૬౰ͷΦʔόʔ ϔου͕͋Δ ͦΕΑΓ΋࠷దԽ͞Ε໋ͨྩͷܭଌ͸?

Slide 7

Slide 7 text

ෳ਺ͷRuby࣮૷ؒͰͷൺֱͷ೉͠͞ ϒϩοΫͰಈతʹܭଌํ๏͕ࢦఆՄೳͩͱɺͦͷϒϩοΫΛ ผͷRubyΠϯλϓϦλʹ౉ͯ͠ܭଌ͢Δͷ͸ࠔ೉ ͔͠͠Ruby 3x3ͷͨΊʹ͸ɺॲཧܥࣗମͷมߋͷޮՌΛධՁ ͍ͨ͠

Slide 8

Slide 8 text

ཚཱ͢ΔϕϯνϚʔΧʔ ඪ४ͷbenchmarkϥΠϒϥϦ benchmark-ips RubyϦϙδτϦ಺ͷ benchmark/driver.rb RubyBenchͷ benchmark/driver.rb ͷϑΥʔΫ VladʹΑΔMJIT-benchmarksͷ࣮ߦΤϯδϯ

Slide 9

Slide 9 text

Ͱ͸Ͳ͏͢Δ͔?

Slide 10

Slide 10 text

࠷΋͜ͷ໰୊ͷղܾʹ͍ۙπʔϧ ඪ४ͷbenchmarkϥΠϒϥϦ benchmark-ips RubyϦϙδτϦ಺ͷ benchmark/driver.rb RubyBenchͷ benchmark/driver.rb ͷϑΥʔΫ VladʹΑΔMJIT-benchmarksͷ࣮ߦΤϯδϯ

Slide 11

Slide 11 text

benchmark/driver.rb ͱ͸ ෳ਺RubyόΠφϦؒͰൺֱ͕Ͱ͖ɺ࣮ߦ࣌ؒͷଞʹϝϞϦ࢖༻ྔΛܭଌͰ ͖Δ ϒϩοΫͰ͸ͳ͘whileϧʔϓΛ࢖͏ͨΊɺܭଌͷΦʔόʔϔου͕গͳ͍ ͨͩ͠RubyϦϙδτϦ಺ͷεΫϦϓτʹಛԽͨ͠ॲཧ͕ೖ͍ͬͯΔ benchmark/driver.rb Λ࢖͍΍ͯ͘͢͠օͰ࢖͑͹͍͍ͷͰ͸? ͦΕ͕ benchmark_driver.gem

Slide 12

Slide 12 text

࠾୒͞ΕͨϓϩδΣΫτ “ΑΓ౷ҰతͰ؆୯ͳRubyͷϕϯνϚʔΫ؀ڥͷ࣮ݱ" benchmark_driver.gem ͷػೳ֦ॆ Ruby 3x3 ʹ޲͚ͨϕϯνϚʔΫηοτͱCIͷ੔උ

Slide 13

Slide 13 text

benchmark_driver.gem ͷಛ௃

Slide 14

Slide 14 text

1. ͋ΒΏΔϝτϦΫεΛ͋ΒΏΔํ๏Ͱग़ྗͰ͖Δ ޙड़͢ΔϓϥάΠϯػߏʹΑΓɺ༷ʑͳܭଌํ๏Λ༷ʑͳํ ๏ͰूܭՄೳ σϑΥϧτͰ͸benchmark-ips෩ͷܭଌɾग़ྗ ಉ͡ग़ྗͷ··ϝϞϦ࢖༻ྔΛܭଌͨ͠ΓɺOptcarrotͷfps Λܭଌͨ͠ΓͰ͖Δ

Slide 15

Slide 15 text

2. ਫ਼౓ͷྑ͍ܭଌεΫϦϓτΛࣗಈͰੜ੒ whileϧʔϓΛ͢ΔεΫϦϓτΛಈతʹੜ੒͠ɺ͔ͭwhileϧʔϓ ʹ͔͔Δ࣌ؒ΋ܭଌͯࠩ͠͠Ҿ͘ ͦͷͨΊɺܭଌର৅͸ϒϩοΫͰ͸ͳ͘ඞͣจࣈྻͰ༩͑ΒΕΔ Rubyͷىಈ࣌ؒ΋ܭଌͯ͠͠·͏ΦϦδφϧͷbenchmark/ driver.rbͱҧ͍ɺੜ੒ͨ͠εΫϦϓτ಺Ͱ࣌ؒͷܭଌΛ͢Δ ىಈ࣌ؒͷϒϨͷӨڹΛड͚ʹ͍͘

Slide 16

Slide 16 text

3. RubyʹՃ͑ɺYAMLͰ؆͔ܿͭॊೈʹܭଌ͕ߦͳ͑Δ ಈతͳϒϩοΫͰ͸ͳ͘จࣈྻͰ༩͑ΔҎ্ɺΘ͟Θ͟RubyͰϕ ϯνϚʔΫΛఆٛͤ͞Δඞཁ͕΋͸΍ͳ͍ YAMLͷγϯλοΫεͰϕϯνϚʔΫΛఆٛ͢Δͱ୹͘ॻ͚Δ .travis.yml ΍ .circleci/conﬁg.yml ͘Β͍ͷؾܰ͞Ͱஔ͍ͯ΋Β͍͍ͨ Optcarrotʹ͸طʹYAML͕഑ஔ͞Ε͍ͯΔ

Slide 17

Slide 17 text

4. ෳ਺ͷRuby࣮૷ؒɺgemͷผόʔδϣϯؒͰൺֱͰ͖Δ RubyόΠφϦΛىಈͯ͠ܭଌ͢ΔͨΊɺෳ਺ͷRuby࣮૷ؒͰ ൺֱͰ͖ɺRuby 3x3ʹ࠷ద Ұ౓RubyΛىಈͨ͠Βͦ͜Ͱಉ͡gemͷෳ਺όʔδϣϯΛڞ ଘͤ͞Δ͜ͱ͸ෆՄೳ͕ͩɺbenchmark_driver.gemͰ͸ͦͷൺ ֱ͕Ͱ͖Δ

Slide 18

Slide 18 text

࣮ࡍͷ࢖༻ྫ

Slide 19

Slide 19 text

Haml 4 vs Haml 5

Slide 20

Slide 20 text

ERB Ruby 2.4 vs Ruby 2.5

Slide 21

Slide 21 text

Optcarrot

Slide 22

Slide 22 text

VS JRuby

Slide 23

Slide 23 text

VS TrufﬂeRuby

Slide 24

Slide 24 text

Docker Time.utc

Slide 25

Slide 25 text

Docker Time.utc w/ $TZ

Slide 26

Slide 26 text

Ruby 2.6 block.call

Slide 27

Slide 27 text

benchmark_driver.gemͷ։ൃ Ҏ֎ͷ੒Ռ෺

Slide 28

Slide 28 text

ϕϯνϚʔΫηοτͷ༻ҙ FluentdϕϯνϚʔΫͷࣗಈԽɾbenchmark_driverରԠ benchmark-driver/ﬂuentd-benchmark MJIT-benchmarksͷbenchmark_driverରԠ benchmark-driver/mjit-benchmarks Watson1978͞ΜͷϕϯνϚʔΫηοτͷม׵෺ benchmark-driver/ruby-method-benchmarks

Slide 29

Slide 29 text

benchmark-driver.github.io

Slide 30

Slide 30 text

benchmark-driver.github.io GitHub pages ϕϯνϚʔΫαʔόʔ cronͰruby/ruby ΛϙʔϦϯά + ϕϯνϚʔΫ middleman build git push Travis ϕϯνϚʔΫ݁Ռͷ YAMLΛอଘ͢Δ GitHubϦϙδτϦ Πϯλʔωοτ ݁ՌͷYAMLΛ git push webhook ݟΕΔ आΓͯΔαʔόʔͷ࢓༷Ͱ ֎෦͔ΒΞΫηεͰ͖ͳ͍

Slide 31

Slide 31 text

benchmark-driver.github.io benchmark-driver/ruby-builder, skybench, sky2-infra… RubyBenchͩͱϓϩδΣΫτظؒதʹ͍͡Γ౗͢ͷ͕೉͍͠ ͱߟ͑ͨͨΊɺࢼݧతʹผϓϩδΣΫτͱͯ࢝͠ಈ ࣮ࡍʹ͸ϓϩδΣΫτऴྃޙ੠Λ͔͚ͨΒ͙͢৭ʑͳݖݶ Λ΋Β͑ͨ ৄࡉ͸ޙ൒Ͱ࿩͠·͢

Slide 32

Slide 32 text

2. ϕϯνϚʔΫϝτϦΫεͷϞσϦϯά

Slide 33

Slide 33 text

ม਺ଟ͗͢໰୊ ܭଌର৅ͷεΫϦϓτୡ ࣄલ࣮ߦεΫϦϓτͷ૊߹ͤ ܭଌ͢ΔϝτϦΫεɺ͋Δ͍͸ܭଌํ๏ Rubyͷόʔδϣϯ gemͷόʔδϣϯ

Slide 34

Slide 34 text

ෳ਺ϝτϦΫεͷಉ࣌ܭଌ ܭଌ͢ΔεΫϦϓτͷҰཡͱRubyͷόʔδϣϯҰཡ͚ͩͰදʹͳ͍ͬͯ ͯɺෳ਺ͷϝτϦΫεΛࠞͥΔͱਏ͍ Ͱ΋user, systemͷCPU࣌ؒͱ࣮࣌ؒΛಉ࣌ʹग़͢Έ͍ͨͳͷ΋΍Γ͍ͨ (Ϟσϧ্͸ରԠ͚ͨ͠Ͳ࣮͸·ͩ͜Ε࣮૷ͯ͠ͳ͍) શͯͷϢʔεέʔεΛΧόʔͭͭ͠΋ഁ୼͍ͯ͠ͳ͍ϞσϧΛͲ͏ʹ͔ ࡞Εͳ͍͔?

Slide 35

Slide 35 text

v0.11~v0.12ݱࡏͷϞσϧ

Slide 36

Slide 36 text

v0.11~v0.12ݱࡏͷϞσϧ

Slide 37

Slide 37 text

v0.11~v0.12ݱࡏͷϞσϧ

Slide 38

Slide 38 text

ݕ౼ͨ͠ࣄ߲ HashͷΩʔ͸ඞͣΠϛϡʔλϒϧͳ΋ͷʹ͢Δ ͡Όͳ͍ͱOutputϓϥάΠϯ͕Hash͔Β஋Λࢀর͢Δ࣌ʹࠔΔ Outputͷ͠΍͢͞ΑΓRunnerͷ࣮૷͕ഁ୼͠ͳ͍ͷΛ༏ઌͨ͠ߏ଄ ಉ࣌ʹऔΒΕΔϝτϦΫε͕ͳΔ΂͍ۙ͘৔ॴʹདྷΔߏ଄ Output͸࠷ѱ࠷ޙʹ·ͱΊͯͲ͏ʹ͔Ͱ͖ΔͨΊ

Slide 39

Slide 39 text

3. ϓϥάΠϯΞʔΩςΫνϟ

Slide 40

Slide 40 text

աڈͷϓϥάΠϯࣄྫ ϓϥάΠϯαϙʔτͷ͋Δ๻ͷ͜Ε·ͰͷϓϩδΣΫτ cookpad/barbeque itamae-kitchen/mitamae benchmark-driver/benchmark-driver

Slide 41

Slide 41 text

ڞ௨఺: ࣗ෼͔͠ϓϥάΠϯॻ͍ͯͳ͍ ͔͠΋ࣗݾධՁͰbenchmark-driverͷΠϯλʔϑΣʔε͕Ұ൪ ΍͹͍ ͦ΋ͦ΋͋·Γந৅Խʹ޲͍ͯͳ͍ཁ݅ͳؾ΋͍ͯ͠Δ ͳͷͰ࿩൒෼ʹฉ͍͓͍͍ͯͯͩ͘͞

Slide 42

Slide 42 text

ϓϥάΠϯγεςϜͷ֓ཁ

Slide 43

Slide 43 text

2छྨͷϓϥάΠϯαϙʔτ RunnerϓϥάΠϯ: ܭଌํ๏ͱܭଌ͍ͨ͠ϝτϦΫεΛܾΊΔ OutputϓϥάΠϯ: ܭଌ݁ՌͷूܭΛߦͳ͏

Slide 44

Slide 44 text

RunnerϓϥάΠϯ Ruby API͔Β࣮ߦ͞ΕΔ͕ɺYAMLͷύʔε΋͜ͷRunnerϓ ϥάΠϯ͕ߦͳ͏ RunnerͷType͸benchmark-driverίϚϯυͰ༩͑ΒΕΔ͕ɺ YAMLͷதͰ΋ॻ͚ΔͷͰɺYAML಺ͰTypeΛݻఆͭͭ͠ಠ ࣗͷYAMLύʔαΛॻ͘ͱ޷͖์୊YAMLͷจ๏͕֦ுͰ͖ Δ

Slide 45

Slide 45 text

RunnerϓϥάΠϯ σϑΥϧτͷYAMLఆٛͷ΋ͷ: ips: ࣮ߦ࣌ؒ/࣮ߦճ਺Λܭଌ͢Δ once: ςετ༻ʹ1ճ͚ͩ૸ΒͤΔ time: ࣮ߦ࣌ؒ memory: Max Resident Set Size (ࠓ͸LinuxͷΈαϙʔτ)

Slide 46

Slide 46 text

RunnerϓϥάΠϯ ಠࣗͷYAMLఆٛͷ΋ͷ: recorded: record outputΛ࢖͏ͱੜ੒͞ΕΔYAMLͰɺಉ͡ܭ ଌ݁ՌΛԿ౓΋ग़ྗͰ͖Δ ruby_stdout: RubyίϚϯυͷstdoutΛύʔεͯ͠ϝτϦΫε ΛऔΔ

Slide 47

Slide 47 text

memory

Slide 48

Slide 48 text

memory

Slide 49

Slide 49 text

Optcarrot

Slide 50

Slide 50 text

Optcarrot

Slide 51

Slide 51 text

Runnerʹ͍ͭͯҙਤతʹαϙʔτΛ΍Ίͨػೳ Ruby͔Β࣮ߦ࣌ɺจࣈྻ͡Όͳͯ͘ϒϩοΫͰܭଌର৅Λ༩͑Δͷ΋α ϙʔτ͍ͯͨ͠ Ҡߦύε + ෳ਺όʔδϣϯ͕ࢦఆ͞Εͳ͚Ε͹evalͰ૸Βͤͨํ͕଎͍ ࣮͸࠷ॳ͸͜ͷػೳ͕Runnerͱͯ͠ϓϥάΠϯαϙʔτ͞Ε͍ͯͨ(3छ ྨͷϓϥάΠϯ͕͋ͬͨ) υϥΠόͱಉ໊͡લۭؒͰ૸ΒͤΔ෼࣮ߦ͕ΫϦʔϯ͡Όͳ͘ɺಉ͡εΫ ϦϓτΛ࣮ߦ͢Δʹ͸αϙʔτ͕ਏ͔ͬͨ͠ϝϦοτ΋ബ͍ͷͰ΍Ίͨ

Slide 52

Slide 52 text

v0.11~12ͷRunnerϓϥάΠϯͷॻ͖ํ YAMLΛந৅Խ͢ΔJobͱɺYAML͔ΒͦΕΛ࡞ΔJobParserΛఆٛ͢Δ #initializeʹ͸ɺCLIΦϓγϣϯΛ࣋ͭconﬁg, OutputΛந৅Խ͢Δ output, δϣϒͷϚτϦΫεΛ࡞Δcontexts͕༩͑ΒΕΔ #run ͸޷͖์୊࣮૷͍͍͕ͯ͠ɺoutputʹରͯ͠ϝιου܊Λܾ·ͬ ͨॱ൪ʹݺ͹ͳ͚Ε͹ͳΒͳ͍ #with_warmup, #with_benchmark, #with_job, #with_context, #report

Slide 53

Slide 53 text

OutputϓϥάΠϯ લड़ͨ͠ϞσϧΛड͚औͬͯूܭ͢Δ໾ׂ 1ͭܭଌ͢Δ͝ͱʹoutputʹ౉ͤΔΠϯλʔϑΣʔεͳͷͰɺ ݁Ռ͕ετϦʔϛϯάग़ྗͰ͖Δ શ෦ऴΘͬͨޙʹόϧΫͰग़ྗ͢ΔͨΊͷ؆୯ͳΠϯλʔ ϑΣʔε΋༻ҙ͍ͯ͠Δ

Slide 54

Slide 54 text

OutputϓϥάΠϯ ϏϧτΠϯͷϓϥάΠϯ compare: benchmark-ips෩ͷൺֱग़ྗ simple: benchmark/driver.rb ෩ग़ྗ markdown: GFMͷςʔϒϧग़ྗ record: ܭଌ݁ՌΛอଘͯ͠ผͷoutputʹྲྀ͢༻

Slide 55

Slide 55 text

OutputϓϥάΠϯ ඇެࣜϓϥάΠϯ gruff: gruff.gemͰάϥϑ͕ग़ྗͰ͖Δ rubybench: RubyBenchʹ݁ՌΛૹ৴

Slide 56

Slide 56 text

Markdown

Slide 57

Slide 57 text

gruff

Slide 58

Slide 58 text

ϓϥάΠϯʹؔ͢Δ஫ҙࣄ߲ v0ͷؒԿ౓΋ϓϥάΠϯΠϯλʔϑΣʔε͕มΘ͍ͬͯͯɺv1ʹ͢Δ·Ͱ͸ ֬ఆ͡Όͳ͍Ͱ͢ Runner Typeݻ༗ͷprelude͕άϩʔόϧͳcontexʹೖͬͯΔͷ͸Ҿ͖ണ͕ ͢༧ఆ ϓϥάΠϯΛ࡞ͬͯͩ͘͞Δํ͸ґଘΛ "benchmark_driver", "~> v0.X.Y" ʹ ౰໘͸͍͚ͯͨͩ͠Δͱॿ͔Γ·͢ ಉ͡ v0.X ͷؒ͸յ͞ͳ͍Α͏ʹ͍ͯ͠·͢

Slide 59

Slide 59 text

4. ϓϩδΣΫτऴྃޙͷ׆ಈ

Slide 60

Slide 60 text

RubyBench

Slide 61

Slide 61 text

RubyBenchͱ͸ RubyͷύϑΥʔϚϯεΛ؂ࢹ͢ΔCI benchmark-driver.github.io ΑΓ΋େ෼લ͔Β͋Δ

Slide 62

Slide 62 text

RubyBenchͷΞʔΩςΫνϟ RubyBench ϫʔΧʔ task-spooler → docker run ruby … GitHub fork for ruby/ruby RubyBench web Rails ΞϓϦέʔγϣϯ → Sidekiq GitHub ruby/ruby cronͰߋ৽ Webhook ssh tsp …

Slide 63

Slide 63 text

͜͜Λஔ͖׵͍͑ͨ RubyBench ϫʔΧʔ task-spooler → docker run ruby … GitHub fork for ruby/ruby RubyBench web Rails ΞϓϦέʔγϣϯ → Sidekiq GitHub ruby/ruby cronͰߋ৽ Webhook ssh tsp …

Slide 64

Slide 64 text

RubyBench΁ͷ౷߹ JITͱͷൺֱάϥϑ͸ޙ͔ΒRubyBenchʹ΋࣮૷͞Εͨ όάͬͯͨͷͰ௚ͨ͠Γͨ͠ benchmark_driver޲͚ͷϕϯνϚʔΫηοτ΍ਫ਼౓ͷྑ͍ܭଌػೳ͕ࠓޙ׆༻͠ ΍͍͢Α͏ɺϕϯνϚʔΧʔͷbenchmark_driver΁ͷஔ͖׵͑Λ΍͍ͬͯΔ طʹOptcarrot͸benchmark_driverͰܭଌ͞Ε͍ͯΔ ॏෳ͕ͪ͠ͳ࣮૷͕ϓϥάΠϯͰந৅ԽͰ͖͓ͯΓbenchmark_driverͷϓϥά Πϯػߏ͕׆༂͍ͯ͠Δ

Slide 65

Slide 65 text

RubyBench޲͚ʹɺϕϯνϚʔΫͷ֦ॆ΋΍͍͖ͬͯ·͢ VladͷMJIT޲͚ϕϯνϚʔΫूΛ௥Ճ༧ఆ Watson1978͞Μͷ֤ϝιου޲͚ϕϯνϚʔΫूΛม׵ͨ͠ ΋ͷ΋࢖Θ͍͔ͤͯͨͩ͘͞΋? https://github.com/benchmark-driver/ruby-method-benchmarks

Slide 66

Slide 66 text

·ͱΊ benchmark_driver ศརͰ͢ͷͰɺੋඇ͝ར༻͍ͩ͘͞ Rubyͷߴ଎Խ΍RubyBenchʹطʹຊ੒Ռ͕׆༻͞Ε͍ͯ·͢

Slide 67

Slide 67 text

ँࣙ ຊϓϩδΣΫτ͸ɺ࡫ాߞҰ༷Λ࢝Ίͱͨ͠RubyΞιγΤʔ γϣϯ༷ͷ͝ڠྗʹΑ࣮ͬͯݱ͢Δ͜ͱ͕Ͱ͖·ͨ͠ɻ͜ͷ ৔ΛआΓͯޚྱਃ্͛͠·͢ɻ