Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
Ruby 3x3 に向けたRubyのベンチマーク環境と成果 / Ruby Association Grant
Takashi Kokubun
July 07, 2018
Programming
10
12k
Ruby 3x3 に向けたRubyのベンチマーク環境と成果 / Ruby Association Grant
Rubyアソシエーション開発助成成果報告会
https://rubyassociation.doorkeeper.jp/events/74355
Takashi Kokubun
July 07, 2018
Tweet
Share
More Decks by Takashi Kokubun
See All by Takashi Kokubun
k0kubun
0
150
k0kubun
8
6k
k0kubun
2
740
k0kubun
0
440
k0kubun
13
8.8k
k0kubun
1
470
k0kubun
8
15k
k0kubun
1
180
k0kubun
5
1.2k
Other Decks in Programming
See All in Programming
azdaroth
0
170
wasabeef
1
570
hanakla
2
3k
mfunaki
1
110
kyonmm
2
2.2k
zsmb
1
130
kazuki19992
0
450
yokaze
0
280
466548
0
110
sullis
0
120
canon1ky
3
350
s_uryu
0
210
Featured
See All Featured
morganepeng
92
14k
tanoku
86
8.5k
lynnandtonic
271
16k
jonrohan
1021
380k
maltzj
500
36k
bermonpainter
342
26k
malarkey
192
8.6k
sachag
446
36k
sstephenson
144
12k
lara
16
2.6k
vanstee
116
4.8k
lauravandoore
11
1.3k
Transcript
Ruby 3x3 ʹ͚ͨ RubyͷϕϯνϚʔΫڥͱՌ RubyΞιγΤʔγϣϯ։ൃॿ Ռใࠂձ @k0kubun
ࣗݾհ GitHub, Twitter: k0kubun Rubyίϛολ: JITίϯύΠϥͷ։ൃ, ERBͷϝϯςφϯε ࡞ͬͨ: Hamlit, MItamae,
md2key, xremap, LLRB
ࠓ͢͜ͱ 1. ຊϓϩδΣΫτԿΛղܾ͔ͨ͠ 2. ϕϯνϚʔΫϝτϦΫεͷϞσϦϯά 3. ϓϥάΠϯΞʔΩςΫνϟ 4. ϓϩδΣΫτऴྃޙͷ׆ಈ
1. ຊϓϩδΣΫτԿΛղܾ͔ͨ͠
͜Ε·ͰͷRubyͷϕϯνϚʔΫڥ
ϒϩοΫݺͼग़͠ͷΦʔόʔϔου ඪ४ͷbenchmarkϥΠϒϥϦbenchmark-ipsɺܭଌ୯Ґ͕ϒ ϩοΫʹͳ͍ͬͯΔ ϒϩοΫݺͼग़͠ී௨ͷϝιουݺͼग़͠૬ͷΦʔόʔ ϔου͕͋Δ ͦΕΑΓ࠷దԽ͞Ε໋ͨྩͷܭଌ?
ෳͷRuby࣮ؒͰͷൺֱͷ͠͞ ϒϩοΫͰಈతʹܭଌํ๏͕ࢦఆՄೳͩͱɺͦͷϒϩοΫΛ ผͷRubyΠϯλϓϦλʹͯ͠ܭଌ͢Δͷࠔ ͔͠͠Ruby 3x3ͷͨΊʹɺॲཧܥࣗମͷมߋͷޮՌΛධՁ ͍ͨ͠
ཚཱ͢ΔϕϯνϚʔΧʔ ඪ४ͷbenchmarkϥΠϒϥϦ benchmark-ips RubyϦϙδτϦͷ benchmark/driver.rb RubyBenchͷ benchmark/driver.rb ͷϑΥʔΫ VladʹΑΔMJIT-benchmarksͷ࣮ߦΤϯδϯ
ͰͲ͏͢Δ͔?
࠷͜ͷͷղܾʹ͍ۙπʔϧ ඪ४ͷbenchmarkϥΠϒϥϦ benchmark-ips RubyϦϙδτϦͷ benchmark/driver.rb RubyBenchͷ benchmark/driver.rb ͷϑΥʔΫ VladʹΑΔMJIT-benchmarksͷ࣮ߦΤϯδϯ
benchmark/driver.rb ͱ ෳRubyόΠφϦؒͰൺֱ͕Ͱ͖ɺ࣮ߦ࣌ؒͷଞʹϝϞϦ༻ྔΛܭଌͰ ͖Δ ϒϩοΫͰͳ͘whileϧʔϓΛ͏ͨΊɺܭଌͷΦʔόʔϔου͕গͳ͍ ͨͩ͠RubyϦϙδτϦͷεΫϦϓτʹಛԽͨ͠ॲཧ͕ೖ͍ͬͯΔ benchmark/driver.rb Λ͍ͯ͘͢͠օͰ͍͍͑ͷͰ? ͦΕ͕ benchmark_driver.gem
࠾͞ΕͨϓϩδΣΫτ “ΑΓ౷ҰతͰ؆୯ͳRubyͷϕϯνϚʔΫڥͷ࣮ݱ" benchmark_driver.gem ͷػೳ֦ॆ Ruby 3x3 ʹ͚ͨϕϯνϚʔΫηοτͱCIͷඋ
benchmark_driver.gem ͷಛ
1. ͋ΒΏΔϝτϦΫεΛ͋ΒΏΔํ๏Ͱग़ྗͰ͖Δ ޙड़͢ΔϓϥάΠϯػߏʹΑΓɺ༷ʑͳܭଌํ๏Λ༷ʑͳํ ๏ͰूܭՄೳ σϑΥϧτͰbenchmark-ips෩ͷܭଌɾग़ྗ ಉ͡ग़ྗͷ··ϝϞϦ༻ྔΛܭଌͨ͠ΓɺOptcarrotͷfps Λܭଌͨ͠ΓͰ͖Δ
2. ਫ਼ͷྑ͍ܭଌεΫϦϓτΛࣗಈͰੜ whileϧʔϓΛ͢ΔεΫϦϓτΛಈతʹੜ͠ɺ͔ͭwhileϧʔϓ ʹ͔͔Δ࣌ؒܭଌͯࠩ͠͠Ҿ͘ ͦͷͨΊɺܭଌରϒϩοΫͰͳ͘ඞͣจࣈྻͰ༩͑ΒΕΔ Rubyͷىಈ࣌ؒܭଌͯ͠͠·͏ΦϦδφϧͷbenchmark/ driver.rbͱҧ͍ɺੜͨ͠εΫϦϓτͰ࣌ؒͷܭଌΛ͢Δ ىಈ࣌ؒͷϒϨͷӨڹΛड͚ʹ͍͘
3. RubyʹՃ͑ɺYAMLͰ؆͔ܿͭॊೈʹܭଌ͕ߦͳ͑Δ ಈతͳϒϩοΫͰͳ͘จࣈྻͰ༩͑ΔҎ্ɺΘ͟Θ͟RubyͰϕ ϯνϚʔΫΛఆٛͤ͞Δඞཁ͕ͳ͍ YAMLͷγϯλοΫεͰϕϯνϚʔΫΛఆٛ͢Δͱ͘ॻ͚Δ .travis.yml .circleci/config.yml ͘Β͍ͷؾܰ͞Ͱஔ͍ͯΒ͍͍ͨ OptcarrotʹطʹYAML͕ஔ͞Ε͍ͯΔ
4. ෳͷRuby࣮ؒɺgemͷผόʔδϣϯؒͰൺֱͰ͖Δ RubyόΠφϦΛىಈͯ͠ܭଌ͢ΔͨΊɺෳͷRuby࣮ؒͰ ൺֱͰ͖ɺRuby 3x3ʹ࠷ద ҰRubyΛىಈͨ͠Βͦ͜Ͱಉ͡gemͷෳόʔδϣϯΛڞ ଘͤ͞Δ͜ͱෆՄೳ͕ͩɺbenchmark_driver.gemͰͦͷൺ ֱ͕Ͱ͖Δ
࣮ࡍͷ༻ྫ
Haml 4 vs Haml 5
ERB Ruby 2.4 vs Ruby 2.5
Optcarrot
VS JRuby
VS TruffleRuby
Docker Time.utc
Docker Time.utc w/ $TZ
Ruby 2.6 block.call
benchmark_driver.gemͷ։ൃ Ҏ֎ͷՌ
ϕϯνϚʔΫηοτͷ༻ҙ FluentdϕϯνϚʔΫͷࣗಈԽɾbenchmark_driverରԠ benchmark-driver/fluentd-benchmark MJIT-benchmarksͷbenchmark_driverରԠ benchmark-driver/mjit-benchmarks Watson1978͞ΜͷϕϯνϚʔΫηοτͷม benchmark-driver/ruby-method-benchmarks
benchmark-driver.github.io
benchmark-driver.github.io GitHub pages ϕϯνϚʔΫαʔόʔ cronͰruby/ruby ΛϙʔϦϯά + ϕϯνϚʔΫ middleman build
git push Travis ϕϯνϚʔΫ݁Ռͷ YAMLΛอଘ͢Δ GitHubϦϙδτϦ Πϯλʔωοτ ݁ՌͷYAMLΛ git push webhook ݟΕΔ आΓͯΔαʔόʔͷ༷Ͱ ֎෦͔ΒΞΫηεͰ͖ͳ͍
benchmark-driver.github.io benchmark-driver/ruby-builder, skybench, sky2-infra… RubyBenchͩͱϓϩδΣΫτظؒதʹ͍͡Γ͢ͷ͕͍͠ ͱߟ͑ͨͨΊɺࢼݧతʹผϓϩδΣΫτͱͯ࢝͠ಈ ࣮ࡍʹϓϩδΣΫτऴྃޙΛ͔͚ͨΒ͙͢৭ʑͳݖݶ ΛΒ͑ͨ ৄࡉޙͰ͠·͢
2. ϕϯνϚʔΫϝτϦΫεͷϞσϦϯά
มଟ͗͢ ܭଌରͷεΫϦϓτୡ ࣄલ࣮ߦεΫϦϓτͷ߹ͤ ܭଌ͢ΔϝτϦΫεɺ͋Δ͍ܭଌํ๏ Rubyͷόʔδϣϯ gemͷόʔδϣϯ
ෳϝτϦΫεͷಉ࣌ܭଌ ܭଌ͢ΔεΫϦϓτͷҰཡͱRubyͷόʔδϣϯҰཡ͚ͩͰදʹͳ͍ͬͯ ͯɺෳͷϝτϦΫεΛࠞͥΔͱਏ͍ Ͱuser, systemͷCPU࣌ؒͱ࣮࣌ؒΛಉ࣌ʹग़͢Έ͍ͨͳͷΓ͍ͨ (Ϟσϧ্ରԠ͚ͨ͠Ͳ࣮·ͩ͜Ε࣮ͯ͠ͳ͍) શͯͷϢʔεέʔεΛΧόʔͭͭ͠ഁ͍ͯ͠ͳ͍ϞσϧΛͲ͏ʹ͔ ࡞Εͳ͍͔?
v0.11~v0.12ݱࡏͷϞσϧ
v0.11~v0.12ݱࡏͷϞσϧ
v0.11~v0.12ݱࡏͷϞσϧ
ݕ౼ͨ͠ࣄ߲ HashͷΩʔඞͣΠϛϡʔλϒϧͳͷʹ͢Δ ͡Όͳ͍ͱOutputϓϥάΠϯ͕Hash͔ΒΛࢀর͢Δ࣌ʹࠔΔ Outputͷ͢͠͞ΑΓRunnerͷ࣮͕ഁ͠ͳ͍ͷΛ༏ઌͨ͠ߏ ಉ࣌ʹऔΒΕΔϝτϦΫε͕ͳΔ͍ۙ͘ॴʹདྷΔߏ Output࠷ѱ࠷ޙʹ·ͱΊͯͲ͏ʹ͔Ͱ͖ΔͨΊ
3. ϓϥάΠϯΞʔΩςΫνϟ
աڈͷϓϥάΠϯࣄྫ ϓϥάΠϯαϙʔτͷ͋Δͷ͜Ε·ͰͷϓϩδΣΫτ cookpad/barbeque itamae-kitchen/mitamae benchmark-driver/benchmark-driver
ڞ௨: ͔ࣗ͠ϓϥάΠϯॻ͍ͯͳ͍ ͔ࣗ͠ݾධՁͰbenchmark-driverͷΠϯλʔϑΣʔε͕Ұ൪ ͍ ͦͦ͋·ΓநԽʹ͍ͯͳ͍ཁ݅ͳؾ͍ͯ͠Δ ͳͷͰʹฉ͍͓͍͍ͯͯͩ͘͞
ϓϥάΠϯγεςϜͷ֓ཁ
2छྨͷϓϥάΠϯαϙʔτ RunnerϓϥάΠϯ: ܭଌํ๏ͱܭଌ͍ͨ͠ϝτϦΫεΛܾΊΔ OutputϓϥάΠϯ: ܭଌ݁ՌͷूܭΛߦͳ͏
RunnerϓϥάΠϯ Ruby API͔Β࣮ߦ͞ΕΔ͕ɺYAMLͷύʔε͜ͷRunnerϓ ϥάΠϯ͕ߦͳ͏ RunnerͷTypebenchmark-driverίϚϯυͰ༩͑ΒΕΔ͕ɺ YAMLͷதͰॻ͚ΔͷͰɺYAMLͰTypeΛݻఆͭͭ͠ಠ ࣗͷYAMLύʔαΛॻ͘ͱ͖์YAMLͷจ๏͕֦ுͰ͖ Δ
RunnerϓϥάΠϯ σϑΥϧτͷYAMLఆٛͷͷ: ips: ࣮ߦ࣌ؒ/࣮ߦճΛܭଌ͢Δ once: ςετ༻ʹ1ճ͚ͩΒͤΔ time: ࣮ߦ࣌ؒ memory: Max
Resident Set Size (ࠓLinuxͷΈαϙʔτ)
RunnerϓϥάΠϯ ಠࣗͷYAMLఆٛͷͷ: recorded: record outputΛ͏ͱੜ͞ΕΔYAMLͰɺಉ͡ܭ ଌ݁ՌΛԿग़ྗͰ͖Δ ruby_stdout: RubyίϚϯυͷstdoutΛύʔεͯ͠ϝτϦΫε ΛऔΔ
memory
memory
Optcarrot
Optcarrot
Runnerʹ͍ͭͯҙਤతʹαϙʔτΛΊͨػೳ Ruby͔Β࣮ߦ࣌ɺจࣈྻ͡Όͳͯ͘ϒϩοΫͰܭଌରΛ༩͑Δͷα ϙʔτ͍ͯͨ͠ Ҡߦύε + ෳόʔδϣϯ͕ࢦఆ͞Εͳ͚ΕevalͰΒͤͨํ͕͍ ࣮࠷ॳ͜ͷػೳ͕Runnerͱͯ͠ϓϥάΠϯαϙʔτ͞Ε͍ͯͨ(3छ ྨͷϓϥάΠϯ͕͋ͬͨ) υϥΠόͱಉ໊͡લۭؒͰΒͤΔ࣮ߦ͕ΫϦʔϯ͡Όͳ͘ɺಉ͡εΫ ϦϓτΛ࣮ߦ͢Δʹαϙʔτ͕ਏ͔ͬͨ͠ϝϦοτബ͍ͷͰΊͨ
v0.11~12ͷRunnerϓϥάΠϯͷॻ͖ํ YAMLΛநԽ͢ΔJobͱɺYAML͔ΒͦΕΛ࡞ΔJobParserΛఆٛ͢Δ #initializeʹɺCLIΦϓγϣϯΛ࣋ͭconfig, OutputΛநԽ͢Δ output, δϣϒͷϚτϦΫεΛ࡞Δcontexts͕༩͑ΒΕΔ #run ͖์࣮͍͍͕ͯ͠ɺoutputʹରͯ͠ϝιου܊Λܾ·ͬ ͨॱ൪ʹݺͳ͚ΕͳΒͳ͍ #with_warmup,
#with_benchmark, #with_job, #with_context, #report
OutputϓϥάΠϯ લड़ͨ͠ϞσϧΛड͚औͬͯूܭ͢Δׂ 1ͭܭଌ͢Δ͝ͱʹoutputʹͤΔΠϯλʔϑΣʔεͳͷͰɺ ݁Ռ͕ετϦʔϛϯάग़ྗͰ͖Δ શ෦ऴΘͬͨޙʹόϧΫͰग़ྗ͢ΔͨΊͷ؆୯ͳΠϯλʔ ϑΣʔε༻ҙ͍ͯ͠Δ
OutputϓϥάΠϯ ϏϧτΠϯͷϓϥάΠϯ compare: benchmark-ips෩ͷൺֱग़ྗ simple: benchmark/driver.rb ෩ग़ྗ markdown: GFMͷςʔϒϧग़ྗ record:
ܭଌ݁ՌΛอଘͯ͠ผͷoutputʹྲྀ͢༻
OutputϓϥάΠϯ ඇެࣜϓϥάΠϯ gruff: gruff.gemͰάϥϑ͕ग़ྗͰ͖Δ rubybench: RubyBenchʹ݁ՌΛૹ৴
Markdown
gruff
ϓϥάΠϯʹؔ͢Δҙࣄ߲ v0ͷؒԿϓϥάΠϯΠϯλʔϑΣʔε͕มΘ͍ͬͯͯɺv1ʹ͢Δ·Ͱ ֬ఆ͡Όͳ͍Ͱ͢ Runner Typeݻ༗ͷprelude͕άϩʔόϧͳcontexʹೖͬͯΔͷҾ͖ണ͕ ͢༧ఆ ϓϥάΠϯΛ࡞ͬͯͩ͘͞ΔํґଘΛ "benchmark_driver", "~> v0.X.Y"
ʹ ໘͍͚ͯͨͩ͠Δͱॿ͔Γ·͢ ಉ͡ v0.X ͷؒյ͞ͳ͍Α͏ʹ͍ͯ͠·͢
4. ϓϩδΣΫτऴྃޙͷ׆ಈ
RubyBench
RubyBenchͱ RubyͷύϑΥʔϚϯεΛࢹ͢ΔCI benchmark-driver.github.io ΑΓେલ͔Β͋Δ
RubyBenchͷΞʔΩςΫνϟ RubyBench ϫʔΧʔ task-spooler → docker run ruby … GitHub
fork for ruby/ruby RubyBench web Rails ΞϓϦέʔγϣϯ → Sidekiq GitHub ruby/ruby cronͰߋ৽ Webhook ssh tsp …
͜͜Λஔ͖͍͑ͨ RubyBench ϫʔΧʔ task-spooler → docker run ruby … GitHub
fork for ruby/ruby RubyBench web Rails ΞϓϦέʔγϣϯ → Sidekiq GitHub ruby/ruby cronͰߋ৽ Webhook ssh tsp …
RubyBenchͷ౷߹ JITͱͷൺֱάϥϑޙ͔ΒRubyBenchʹ࣮͞Εͨ όάͬͯͨͷͰͨ͠Γͨ͠ benchmark_driver͚ͷϕϯνϚʔΫηοτਫ਼ͷྑ͍ܭଌػೳ͕ࠓޙ׆༻͠ ͍͢Α͏ɺϕϯνϚʔΧʔͷbenchmark_driverͷஔ͖͑Λ͍ͬͯΔ طʹOptcarrotbenchmark_driverͰܭଌ͞Ε͍ͯΔ ॏෳ͕ͪ͠ͳ࣮͕ϓϥάΠϯͰநԽͰ͖͓ͯΓbenchmark_driverͷϓϥά Πϯػߏ͕׆༂͍ͯ͠Δ
RubyBench͚ʹɺϕϯνϚʔΫͷ֦ॆ͍͖ͬͯ·͢ VladͷMJIT͚ϕϯνϚʔΫूΛՃ༧ఆ Watson1978͞Μͷ֤ϝιου͚ϕϯνϚʔΫूΛมͨ͠ ͷΘ͍͔ͤͯͨͩ͘͞? https://github.com/benchmark-driver/ruby-method-benchmarks
·ͱΊ benchmark_driver ศརͰ͢ͷͰɺੋඇ͝ར༻͍ͩ͘͞ RubyͷߴԽRubyBenchʹطʹຊՌ͕׆༻͞Ε͍ͯ·͢
ँࣙ ຊϓϩδΣΫτɺాߞҰ༷Λ࢝Ίͱͨ͠RubyΞιγΤʔ γϣϯ༷ͷ͝ڠྗʹΑ࣮ͬͯݱ͢Δ͜ͱ͕Ͱ͖·ͨ͠ɻ͜ͷ ΛआΓͯޚྱਃ্͛͠·͢ɻ