Ruby 3x3 に向けたRubyのベンチマーク環境と成果 / Ruby Association Grant

Ruby 3x3 に向けたRubyのベンチマーク環境と成果 / Ruby Association Grant

Rubyアソシエーション開発助成成果報告会
https://rubyassociation.doorkeeper.jp/events/74355

08d5432a5bc31e6d9edec87b94cb1db1?s=128

Takashi Kokubun

July 07, 2018
Tweet

Transcript

  1. Ruby 3x3 ʹ޲͚ͨ RubyͷϕϯνϚʔΫ؀ڥͱ੒Ռ RubyΞιγΤʔγϣϯ։ൃॿ੒ ੒Ռใࠂձ @k0kubun

  2. ࣗݾ঺հ GitHub, Twitter: k0kubun Rubyίϛολ: JITίϯύΠϥͷ։ൃ, ERBͷϝϯςφϯε ࡞ͬͨ: Hamlit, MItamae,

    md2key, xremap, LLRB
  3. ࠓ೔࿩͢͜ͱ 1. ຊϓϩδΣΫτ͸ԿΛղܾ͔ͨ͠ 2. ϕϯνϚʔΫϝτϦΫεͷϞσϦϯά 3. ϓϥάΠϯΞʔΩςΫνϟ 4. ϓϩδΣΫτऴྃޙͷ׆ಈ

  4. 1. ຊϓϩδΣΫτ͸ԿΛղܾ͔ͨ͠

  5. ͜Ε·ͰͷRubyͷϕϯνϚʔΫ؀ڥ

  6. ϒϩοΫݺͼग़͠ͷΦʔόʔϔου໰୊ ඪ४ͷbenchmarkϥΠϒϥϦ΋benchmark-ips΋ɺܭଌ୯Ґ͕ϒ ϩοΫʹͳ͍ͬͯΔ ϒϩοΫݺͼग़͠͸ී௨ͷϝιουݺͼग़͠૬౰ͷΦʔόʔ ϔου͕͋Δ ͦΕΑΓ΋࠷దԽ͞Ε໋ͨྩͷܭଌ͸?

  7. ෳ਺ͷRuby࣮૷ؒͰͷൺֱͷ೉͠͞ ϒϩοΫͰಈతʹܭଌํ๏͕ࢦఆՄೳͩͱɺͦͷϒϩοΫΛ ผͷRubyΠϯλϓϦλʹ౉ͯ͠ܭଌ͢Δͷ͸ࠔ೉ ͔͠͠Ruby 3x3ͷͨΊʹ͸ɺॲཧܥࣗମͷมߋͷޮՌΛධՁ ͍ͨ͠

  8. ཚཱ͢ΔϕϯνϚʔΧʔ ඪ४ͷbenchmarkϥΠϒϥϦ benchmark-ips RubyϦϙδτϦ಺ͷ benchmark/driver.rb RubyBenchͷ benchmark/driver.rb ͷϑΥʔΫ VladʹΑΔMJIT-benchmarksͷ࣮ߦΤϯδϯ

  9. Ͱ͸Ͳ͏͢Δ͔?

  10. ࠷΋͜ͷ໰୊ͷղܾʹ͍ۙπʔϧ ඪ४ͷbenchmarkϥΠϒϥϦ benchmark-ips RubyϦϙδτϦ಺ͷ benchmark/driver.rb RubyBenchͷ benchmark/driver.rb ͷϑΥʔΫ VladʹΑΔMJIT-benchmarksͷ࣮ߦΤϯδϯ

  11. benchmark/driver.rb ͱ͸ ෳ਺RubyόΠφϦؒͰൺֱ͕Ͱ͖ɺ࣮ߦ࣌ؒͷଞʹϝϞϦ࢖༻ྔΛܭଌͰ ͖Δ ϒϩοΫͰ͸ͳ͘whileϧʔϓΛ࢖͏ͨΊɺܭଌͷΦʔόʔϔου͕গͳ͍ ͨͩ͠RubyϦϙδτϦ಺ͷεΫϦϓτʹಛԽͨ͠ॲཧ͕ೖ͍ͬͯΔ benchmark/driver.rb Λ࢖͍΍ͯ͘͢͠օͰ࢖͑͹͍͍ͷͰ͸? ͦΕ͕ benchmark_driver.gem

  12. ࠾୒͞ΕͨϓϩδΣΫτ “ΑΓ౷ҰతͰ؆୯ͳRubyͷϕϯνϚʔΫ؀ڥͷ࣮ݱ" benchmark_driver.gem ͷػೳ֦ॆ Ruby 3x3 ʹ޲͚ͨϕϯνϚʔΫηοτͱCIͷ੔උ

  13. benchmark_driver.gem ͷಛ௃

  14. 1. ͋ΒΏΔϝτϦΫεΛ͋ΒΏΔํ๏Ͱग़ྗͰ͖Δ ޙड़͢ΔϓϥάΠϯػߏʹΑΓɺ༷ʑͳܭଌํ๏Λ༷ʑͳํ ๏ͰूܭՄೳ σϑΥϧτͰ͸benchmark-ips෩ͷܭଌɾग़ྗ ಉ͡ग़ྗͷ··ϝϞϦ࢖༻ྔΛܭଌͨ͠ΓɺOptcarrotͷfps Λܭଌͨ͠ΓͰ͖Δ

  15. 2. ਫ਼౓ͷྑ͍ܭଌεΫϦϓτΛࣗಈͰੜ੒ whileϧʔϓΛ͢ΔεΫϦϓτΛಈతʹੜ੒͠ɺ͔ͭwhileϧʔϓ ʹ͔͔Δ࣌ؒ΋ܭଌͯࠩ͠͠Ҿ͘ ͦͷͨΊɺܭଌର৅͸ϒϩοΫͰ͸ͳ͘ඞͣจࣈྻͰ༩͑ΒΕΔ Rubyͷىಈ࣌ؒ΋ܭଌͯ͠͠·͏ΦϦδφϧͷbenchmark/ driver.rbͱҧ͍ɺੜ੒ͨ͠εΫϦϓτ಺Ͱ࣌ؒͷܭଌΛ͢Δ ىಈ࣌ؒͷϒϨͷӨڹΛड͚ʹ͍͘

  16. 3. RubyʹՃ͑ɺYAMLͰ؆͔ܿͭॊೈʹܭଌ͕ߦͳ͑Δ ಈతͳϒϩοΫͰ͸ͳ͘จࣈྻͰ༩͑ΔҎ্ɺΘ͟Θ͟RubyͰϕ ϯνϚʔΫΛఆٛͤ͞Δඞཁ͕΋͸΍ͳ͍ YAMLͷγϯλοΫεͰϕϯνϚʔΫΛఆٛ͢Δͱ୹͘ॻ͚Δ .travis.yml ΍ .circleci/config.yml ͘Β͍ͷؾܰ͞Ͱஔ͍ͯ΋Β͍͍ͨ Optcarrotʹ͸طʹYAML͕഑ஔ͞Ε͍ͯΔ

  17. 4. ෳ਺ͷRuby࣮૷ؒɺgemͷผόʔδϣϯؒͰൺֱͰ͖Δ RubyόΠφϦΛىಈͯ͠ܭଌ͢ΔͨΊɺෳ਺ͷRuby࣮૷ؒͰ ൺֱͰ͖ɺRuby 3x3ʹ࠷ద Ұ౓RubyΛىಈͨ͠Βͦ͜Ͱಉ͡gemͷෳ਺όʔδϣϯΛڞ ଘͤ͞Δ͜ͱ͸ෆՄೳ͕ͩɺbenchmark_driver.gemͰ͸ͦͷൺ ֱ͕Ͱ͖Δ

  18. ࣮ࡍͷ࢖༻ྫ

  19. Haml 4 vs Haml 5

  20. ERB Ruby 2.4 vs Ruby 2.5

  21. Optcarrot

  22. VS JRuby

  23. VS TruffleRuby

  24. Docker Time.utc

  25. Docker Time.utc w/ $TZ

  26. Ruby 2.6 block.call

  27. benchmark_driver.gemͷ։ൃ Ҏ֎ͷ੒Ռ෺

  28. ϕϯνϚʔΫηοτͷ༻ҙ FluentdϕϯνϚʔΫͷࣗಈԽɾbenchmark_driverରԠ benchmark-driver/fluentd-benchmark MJIT-benchmarksͷbenchmark_driverରԠ benchmark-driver/mjit-benchmarks Watson1978͞ΜͷϕϯνϚʔΫηοτͷม׵෺ benchmark-driver/ruby-method-benchmarks

  29. benchmark-driver.github.io

  30. benchmark-driver.github.io GitHub pages ϕϯνϚʔΫαʔόʔ cronͰruby/ruby ΛϙʔϦϯά + ϕϯνϚʔΫ middleman build

    git push Travis ϕϯνϚʔΫ݁Ռͷ YAMLΛอଘ͢Δ GitHubϦϙδτϦ Πϯλʔωοτ ݁ՌͷYAMLΛ git push webhook ݟΕΔ आΓͯΔαʔόʔͷ࢓༷Ͱ ֎෦͔ΒΞΫηεͰ͖ͳ͍
  31. benchmark-driver.github.io benchmark-driver/ruby-builder, skybench, sky2-infra… RubyBenchͩͱϓϩδΣΫτظؒதʹ͍͡Γ౗͢ͷ͕೉͍͠ ͱߟ͑ͨͨΊɺࢼݧతʹผϓϩδΣΫτͱͯ࢝͠ಈ ࣮ࡍʹ͸ϓϩδΣΫτऴྃޙ੠Λ͔͚ͨΒ͙͢৭ʑͳݖݶ Λ΋Β͑ͨ ৄࡉ͸ޙ൒Ͱ࿩͠·͢

  32. 2. ϕϯνϚʔΫϝτϦΫεͷϞσϦϯά

  33. ม਺ଟ͗͢໰୊ ܭଌର৅ͷεΫϦϓτୡ ࣄલ࣮ߦεΫϦϓτͷ૊߹ͤ ܭଌ͢ΔϝτϦΫεɺ͋Δ͍͸ܭଌํ๏ Rubyͷόʔδϣϯ gemͷόʔδϣϯ

  34. ෳ਺ϝτϦΫεͷಉ࣌ܭଌ ܭଌ͢ΔεΫϦϓτͷҰཡͱRubyͷόʔδϣϯҰཡ͚ͩͰදʹͳ͍ͬͯ ͯɺෳ਺ͷϝτϦΫεΛࠞͥΔͱਏ͍ Ͱ΋user, systemͷCPU࣌ؒͱ࣮࣌ؒΛಉ࣌ʹग़͢Έ͍ͨͳͷ΋΍Γ͍ͨ (Ϟσϧ্͸ରԠ͚ͨ͠Ͳ࣮͸·ͩ͜Ε࣮૷ͯ͠ͳ͍) શͯͷϢʔεέʔεΛΧόʔͭͭ͠΋ഁ୼͍ͯ͠ͳ͍ϞσϧΛͲ͏ʹ͔ ࡞Εͳ͍͔?

  35. v0.11~v0.12ݱࡏͷϞσϧ

  36. v0.11~v0.12ݱࡏͷϞσϧ

  37. v0.11~v0.12ݱࡏͷϞσϧ

  38. ݕ౼ͨ͠ࣄ߲ HashͷΩʔ͸ඞͣΠϛϡʔλϒϧͳ΋ͷʹ͢Δ ͡Όͳ͍ͱOutputϓϥάΠϯ͕Hash͔Β஋Λࢀর͢Δ࣌ʹࠔΔ Outputͷ͠΍͢͞ΑΓRunnerͷ࣮૷͕ഁ୼͠ͳ͍ͷΛ༏ઌͨ͠ߏ଄ ಉ࣌ʹऔΒΕΔϝτϦΫε͕ͳΔ΂͍ۙ͘৔ॴʹདྷΔߏ଄ Output͸࠷ѱ࠷ޙʹ·ͱΊͯͲ͏ʹ͔Ͱ͖ΔͨΊ

  39. 3. ϓϥάΠϯΞʔΩςΫνϟ

  40. աڈͷϓϥάΠϯࣄྫ ϓϥάΠϯαϙʔτͷ͋Δ๻ͷ͜Ε·ͰͷϓϩδΣΫτ cookpad/barbeque itamae-kitchen/mitamae benchmark-driver/benchmark-driver

  41. ڞ௨఺: ࣗ෼͔͠ϓϥάΠϯॻ͍ͯͳ͍ ͔͠΋ࣗݾධՁͰbenchmark-driverͷΠϯλʔϑΣʔε͕Ұ൪ ΍͹͍ ͦ΋ͦ΋͋·Γந৅Խʹ޲͍ͯͳ͍ཁ݅ͳؾ΋͍ͯ͠Δ ͳͷͰ࿩൒෼ʹฉ͍͓͍͍ͯͯͩ͘͞

  42. ϓϥάΠϯγεςϜͷ֓ཁ

  43. 2छྨͷϓϥάΠϯαϙʔτ RunnerϓϥάΠϯ: ܭଌํ๏ͱܭଌ͍ͨ͠ϝτϦΫεΛܾΊΔ OutputϓϥάΠϯ: ܭଌ݁ՌͷूܭΛߦͳ͏

  44. RunnerϓϥάΠϯ Ruby API͔Β࣮ߦ͞ΕΔ͕ɺYAMLͷύʔε΋͜ͷRunnerϓ ϥάΠϯ͕ߦͳ͏ RunnerͷType͸benchmark-driverίϚϯυͰ༩͑ΒΕΔ͕ɺ YAMLͷதͰ΋ॻ͚ΔͷͰɺYAML಺ͰTypeΛݻఆͭͭ͠ಠ ࣗͷYAMLύʔαΛॻ͘ͱ޷͖์୊YAMLͷจ๏͕֦ுͰ͖ Δ

  45. RunnerϓϥάΠϯ σϑΥϧτͷYAMLఆٛͷ΋ͷ: ips: ࣮ߦ࣌ؒ/࣮ߦճ਺Λܭଌ͢Δ once: ςετ༻ʹ1ճ͚ͩ૸ΒͤΔ time: ࣮ߦ࣌ؒ memory: Max

    Resident Set Size (ࠓ͸LinuxͷΈαϙʔτ)
  46. RunnerϓϥάΠϯ ಠࣗͷYAMLఆٛͷ΋ͷ: recorded: record outputΛ࢖͏ͱੜ੒͞ΕΔYAMLͰɺಉ͡ܭ ଌ݁ՌΛԿ౓΋ग़ྗͰ͖Δ ruby_stdout: RubyίϚϯυͷstdoutΛύʔεͯ͠ϝτϦΫε ΛऔΔ

  47. memory

  48. memory

  49. Optcarrot

  50. Optcarrot

  51. Runnerʹ͍ͭͯҙਤతʹαϙʔτΛ΍Ίͨػೳ Ruby͔Β࣮ߦ࣌ɺจࣈྻ͡Όͳͯ͘ϒϩοΫͰܭଌର৅Λ༩͑Δͷ΋α ϙʔτ͍ͯͨ͠ Ҡߦύε + ෳ਺όʔδϣϯ͕ࢦఆ͞Εͳ͚Ε͹evalͰ૸Βͤͨํ͕଎͍ ࣮͸࠷ॳ͸͜ͷػೳ͕Runnerͱͯ͠ϓϥάΠϯαϙʔτ͞Ε͍ͯͨ(3छ ྨͷϓϥάΠϯ͕͋ͬͨ) υϥΠόͱಉ໊͡લۭؒͰ૸ΒͤΔ෼࣮ߦ͕ΫϦʔϯ͡Όͳ͘ɺಉ͡εΫ ϦϓτΛ࣮ߦ͢Δʹ͸αϙʔτ͕ਏ͔ͬͨ͠ϝϦοτ΋ബ͍ͷͰ΍Ίͨ

  52. v0.11~12ͷRunnerϓϥάΠϯͷॻ͖ํ YAMLΛந৅Խ͢ΔJobͱɺYAML͔ΒͦΕΛ࡞ΔJobParserΛఆٛ͢Δ #initializeʹ͸ɺCLIΦϓγϣϯΛ࣋ͭconfig, OutputΛந৅Խ͢Δ output, δϣϒͷϚτϦΫεΛ࡞Δcontexts͕༩͑ΒΕΔ #run ͸޷͖์୊࣮૷͍͍͕ͯ͠ɺoutputʹରͯ͠ϝιου܊Λܾ·ͬ ͨॱ൪ʹݺ͹ͳ͚Ε͹ͳΒͳ͍ #with_warmup,

    #with_benchmark, #with_job, #with_context, #report
  53. OutputϓϥάΠϯ લड़ͨ͠ϞσϧΛड͚औͬͯूܭ͢Δ໾ׂ 1ͭܭଌ͢Δ͝ͱʹoutputʹ౉ͤΔΠϯλʔϑΣʔεͳͷͰɺ ݁Ռ͕ετϦʔϛϯάग़ྗͰ͖Δ શ෦ऴΘͬͨޙʹόϧΫͰग़ྗ͢ΔͨΊͷ؆୯ͳΠϯλʔ ϑΣʔε΋༻ҙ͍ͯ͠Δ

  54. OutputϓϥάΠϯ ϏϧτΠϯͷϓϥάΠϯ compare: benchmark-ips෩ͷൺֱग़ྗ simple: benchmark/driver.rb ෩ग़ྗ markdown: GFMͷςʔϒϧग़ྗ record:

    ܭଌ݁ՌΛอଘͯ͠ผͷoutputʹྲྀ͢༻
  55. OutputϓϥάΠϯ ඇެࣜϓϥάΠϯ gruff: gruff.gemͰάϥϑ͕ग़ྗͰ͖Δ rubybench: RubyBenchʹ݁ՌΛૹ৴

  56. Markdown

  57. gruff

  58. ϓϥάΠϯʹؔ͢Δ஫ҙࣄ߲ v0ͷؒԿ౓΋ϓϥάΠϯΠϯλʔϑΣʔε͕มΘ͍ͬͯͯɺv1ʹ͢Δ·Ͱ͸ ֬ఆ͡Όͳ͍Ͱ͢ Runner Typeݻ༗ͷprelude͕άϩʔόϧͳcontexʹೖͬͯΔͷ͸Ҿ͖ണ͕ ͢༧ఆ ϓϥάΠϯΛ࡞ͬͯͩ͘͞Δํ͸ґଘΛ "benchmark_driver", "~> v0.X.Y"

    ʹ ౰໘͸͍͚ͯͨͩ͠Δͱॿ͔Γ·͢ ಉ͡ v0.X ͷؒ͸յ͞ͳ͍Α͏ʹ͍ͯ͠·͢
  59. 4. ϓϩδΣΫτऴྃޙͷ׆ಈ

  60. RubyBench

  61. RubyBenchͱ͸ RubyͷύϑΥʔϚϯεΛ؂ࢹ͢ΔCI benchmark-driver.github.io ΑΓ΋େ෼લ͔Β͋Δ

  62. RubyBenchͷΞʔΩςΫνϟ RubyBench ϫʔΧʔ task-spooler → docker run ruby … GitHub

    fork for ruby/ruby RubyBench web Rails ΞϓϦέʔγϣϯ → Sidekiq GitHub ruby/ruby cronͰߋ৽ Webhook ssh tsp …
  63. ͜͜Λஔ͖׵͍͑ͨ RubyBench ϫʔΧʔ task-spooler → docker run ruby … GitHub

    fork for ruby/ruby RubyBench web Rails ΞϓϦέʔγϣϯ → Sidekiq GitHub ruby/ruby cronͰߋ৽ Webhook ssh tsp …
  64. RubyBench΁ͷ౷߹ JITͱͷൺֱάϥϑ͸ޙ͔ΒRubyBenchʹ΋࣮૷͞Εͨ όάͬͯͨͷͰ௚ͨ͠Γͨ͠ benchmark_driver޲͚ͷϕϯνϚʔΫηοτ΍ਫ਼౓ͷྑ͍ܭଌػೳ͕ࠓޙ׆༻͠ ΍͍͢Α͏ɺϕϯνϚʔΧʔͷbenchmark_driver΁ͷஔ͖׵͑Λ΍͍ͬͯΔ طʹOptcarrot͸benchmark_driverͰܭଌ͞Ε͍ͯΔ ॏෳ͕ͪ͠ͳ࣮૷͕ϓϥάΠϯͰந৅ԽͰ͖͓ͯΓbenchmark_driverͷϓϥά Πϯػߏ͕׆༂͍ͯ͠Δ

  65. RubyBench޲͚ʹɺϕϯνϚʔΫͷ֦ॆ΋΍͍͖ͬͯ·͢ VladͷMJIT޲͚ϕϯνϚʔΫूΛ௥Ճ༧ఆ Watson1978͞Μͷ֤ϝιου޲͚ϕϯνϚʔΫूΛม׵ͨ͠ ΋ͷ΋࢖Θ͍͔ͤͯͨͩ͘͞΋? https://github.com/benchmark-driver/ruby-method-benchmarks

  66. ·ͱΊ benchmark_driver ศརͰ͢ͷͰɺੋඇ͝ར༻͍ͩ͘͞ Rubyͷߴ଎Խ΍RubyBenchʹطʹຊ੒Ռ͕׆༻͞Ε͍ͯ·͢

  67. ँࣙ ຊϓϩδΣΫτ͸ɺ࡫ాߞҰ༷Λ࢝Ίͱͨ͠RubyΞιγΤʔ γϣϯ༷ͷ͝ڠྗʹΑ࣮ͬͯݱ͢Δ͜ͱ͕Ͱ͖·ͨ͠ɻ͜ͷ ৔ΛआΓͯޚྱਃ্͛͠·͢ɻ