Ruby 3x3 に向けたRubyのベンチマーク環境と成果 / Ruby Association Grant

Transcript

1. Ruby 3x3 ʹ޲͚ͨ RubyͷϕϯνϚʔΫ؀ڥͱ੒Ռ RubyΞιγΤʔγϣϯ։ൃॿ੒ ੒Ռใࠂձ @k0kubun

2. ࣗݾ঺հ GitHub, Twitter: k0kubun Rubyίϛολ: JITίϯύΠϥͷ։ൃ, ERBͷϝϯςφϯε ࡞ͬͨ: Hamlit, MItamae,

   md2key, xremap, LLRB

3. ࠓ೔࿩͢͜ͱ 1. ຊϓϩδΣΫτ͸ԿΛղܾ͔ͨ͠ 2. ϕϯνϚʔΫϝτϦΫεͷϞσϦϯά 3. ϓϥάΠϯΞʔΩςΫνϟ 4. ϓϩδΣΫτऴྃޙͷ׆ಈ

4. 1. ຊϓϩδΣΫτ͸ԿΛղܾ͔ͨ͠

5. ͜Ε·ͰͷRubyͷϕϯνϚʔΫ؀ڥ

6. ϒϩοΫݺͼग़͠ͷΦʔόʔϔου໰୊ ඪ४ͷbenchmarkϥΠϒϥϦ΋benchmark-ips΋ɺܭଌ୯Ґ͕ϒ ϩοΫʹͳ͍ͬͯΔ ϒϩοΫݺͼग़͠͸ී௨ͷϝιουݺͼग़͠૬౰ͷΦʔόʔ ϔου͕͋Δ ͦΕΑΓ΋࠷దԽ͞Ε໋ͨྩͷܭଌ͸?

7. ෳ਺ͷRuby࣮૷ؒͰͷൺֱͷ೉͠͞ ϒϩοΫͰಈతʹܭଌํ๏͕ࢦఆՄೳͩͱɺͦͷϒϩοΫΛ ผͷRubyΠϯλϓϦλʹ౉ͯ͠ܭଌ͢Δͷ͸ࠔ೉ ͔͠͠Ruby 3x3ͷͨΊʹ͸ɺॲཧܥࣗମͷมߋͷޮՌΛධՁ ͍ͨ͠

8. ཚཱ͢ΔϕϯνϚʔΧʔ ඪ४ͷbenchmarkϥΠϒϥϦ benchmark-ips RubyϦϙδτϦ಺ͷ benchmark/driver.rb RubyBenchͷ benchmark/driver.rb ͷϑΥʔΫ VladʹΑΔMJIT-benchmarksͷ࣮ߦΤϯδϯ

9. Ͱ͸Ͳ͏͢Δ͔?

10. ࠷΋͜ͷ໰୊ͷղܾʹ͍ۙπʔϧ ඪ४ͷbenchmarkϥΠϒϥϦ benchmark-ips RubyϦϙδτϦ಺ͷ benchmark/driver.rb RubyBenchͷ benchmark/driver.rb ͷϑΥʔΫ VladʹΑΔMJIT-benchmarksͷ࣮ߦΤϯδϯ

11. benchmark/driver.rb ͱ͸ ෳ਺RubyόΠφϦؒͰൺֱ͕Ͱ͖ɺ࣮ߦ࣌ؒͷଞʹϝϞϦ࢖༻ྔΛܭଌͰ ͖Δ ϒϩοΫͰ͸ͳ͘whileϧʔϓΛ࢖͏ͨΊɺܭଌͷΦʔόʔϔου͕গͳ͍ ͨͩ͠RubyϦϙδτϦ಺ͷεΫϦϓτʹಛԽͨ͠ॲཧ͕ೖ͍ͬͯΔ benchmark/driver.rb Λ࢖͍΍ͯ͘͢͠օͰ࢖͑͹͍͍ͷͰ͸? ͦΕ͕ benchmark_driver.gem

12. ࠾୒͞ΕͨϓϩδΣΫτ “ΑΓ౷ҰతͰ؆୯ͳRubyͷϕϯνϚʔΫ؀ڥͷ࣮ݱ" benchmark_driver.gem ͷػೳ֦ॆ Ruby 3x3 ʹ޲͚ͨϕϯνϚʔΫηοτͱCIͷ੔උ

13. benchmark_driver.gem ͷಛ௃

14. 1. ͋ΒΏΔϝτϦΫεΛ͋ΒΏΔํ๏Ͱग़ྗͰ͖Δ ޙड़͢ΔϓϥάΠϯػߏʹΑΓɺ༷ʑͳܭଌํ๏Λ༷ʑͳํ ๏ͰूܭՄೳ σϑΥϧτͰ͸benchmark-ips෩ͷܭଌɾग़ྗ ಉ͡ग़ྗͷ··ϝϞϦ࢖༻ྔΛܭଌͨ͠ΓɺOptcarrotͷfps Λܭଌͨ͠ΓͰ͖Δ

15. 2. ਫ਼౓ͷྑ͍ܭଌεΫϦϓτΛࣗಈͰੜ੒ whileϧʔϓΛ͢ΔεΫϦϓτΛಈతʹੜ੒͠ɺ͔ͭwhileϧʔϓ ʹ͔͔Δ࣌ؒ΋ܭଌͯࠩ͠͠Ҿ͘ ͦͷͨΊɺܭଌର৅͸ϒϩοΫͰ͸ͳ͘ඞͣจࣈྻͰ༩͑ΒΕΔ Rubyͷىಈ࣌ؒ΋ܭଌͯ͠͠·͏ΦϦδφϧͷbenchmark/ driver.rbͱҧ͍ɺੜ੒ͨ͠εΫϦϓτ಺Ͱ࣌ؒͷܭଌΛ͢Δ ىಈ࣌ؒͷϒϨͷӨڹΛड͚ʹ͍͘

16. 3. RubyʹՃ͑ɺYAMLͰ؆͔ܿͭॊೈʹܭଌ͕ߦͳ͑Δ ಈతͳϒϩοΫͰ͸ͳ͘จࣈྻͰ༩͑ΔҎ্ɺΘ͟Θ͟RubyͰϕ ϯνϚʔΫΛఆٛͤ͞Δඞཁ͕΋͸΍ͳ͍ YAMLͷγϯλοΫεͰϕϯνϚʔΫΛఆٛ͢Δͱ୹͘ॻ͚Δ .travis.yml ΍ .circleci/config.yml ͘Β͍ͷؾܰ͞Ͱஔ͍ͯ΋Β͍͍ͨ Optcarrotʹ͸طʹYAML͕഑ஔ͞Ε͍ͯΔ

17. 4. ෳ਺ͷRuby࣮૷ؒɺgemͷผόʔδϣϯؒͰൺֱͰ͖Δ RubyόΠφϦΛىಈͯ͠ܭଌ͢ΔͨΊɺෳ਺ͷRuby࣮૷ؒͰ ൺֱͰ͖ɺRuby 3x3ʹ࠷ద Ұ౓RubyΛىಈͨ͠Βͦ͜Ͱಉ͡gemͷෳ਺όʔδϣϯΛڞ ଘͤ͞Δ͜ͱ͸ෆՄೳ͕ͩɺbenchmark_driver.gemͰ͸ͦͷൺ ֱ͕Ͱ͖Δ

18. ࣮ࡍͷ࢖༻ྫ

19. Haml 4 vs Haml 5

20. ERB Ruby 2.4 vs Ruby 2.5

21. Optcarrot

22. VS JRuby

23. VS TruffleRuby

24. Docker Time.utc

25. Docker Time.utc w/ $TZ

26. Ruby 2.6 block.call

27. benchmark_driver.gemͷ։ൃ Ҏ֎ͷ੒Ռ෺

28. ϕϯνϚʔΫηοτͷ༻ҙ FluentdϕϯνϚʔΫͷࣗಈԽɾbenchmark_driverରԠ benchmark-driver/fluentd-benchmark MJIT-benchmarksͷbenchmark_driverରԠ benchmark-driver/mjit-benchmarks Watson1978͞ΜͷϕϯνϚʔΫηοτͷม׵෺ benchmark-driver/ruby-method-benchmarks

29. benchmark-driver.github.io

30. benchmark-driver.github.io GitHub pages ϕϯνϚʔΫαʔόʔ cronͰruby/ruby ΛϙʔϦϯά + ϕϯνϚʔΫ middleman build

    git push Travis ϕϯνϚʔΫ݁Ռͷ YAMLΛอଘ͢Δ GitHubϦϙδτϦ Πϯλʔωοτ ݁ՌͷYAMLΛ git push webhook ݟΕΔ आΓͯΔαʔόʔͷ࢓༷Ͱ ֎෦͔ΒΞΫηεͰ͖ͳ͍

31. benchmark-driver.github.io benchmark-driver/ruby-builder, skybench, sky2-infra… RubyBenchͩͱϓϩδΣΫτظؒதʹ͍͡Γ౗͢ͷ͕೉͍͠ ͱߟ͑ͨͨΊɺࢼݧతʹผϓϩδΣΫτͱͯ࢝͠ಈ ࣮ࡍʹ͸ϓϩδΣΫτऴྃޙ੠Λ͔͚ͨΒ͙͢৭ʑͳݖݶ Λ΋Β͑ͨ ৄࡉ͸ޙ൒Ͱ࿩͠·͢

32. 2. ϕϯνϚʔΫϝτϦΫεͷϞσϦϯά

33. ม਺ଟ͗͢໰୊ ܭଌର৅ͷεΫϦϓτୡ ࣄલ࣮ߦεΫϦϓτͷ૊߹ͤ ܭଌ͢ΔϝτϦΫεɺ͋Δ͍͸ܭଌํ๏ Rubyͷόʔδϣϯ gemͷόʔδϣϯ

34. ෳ਺ϝτϦΫεͷಉ࣌ܭଌ ܭଌ͢ΔεΫϦϓτͷҰཡͱRubyͷόʔδϣϯҰཡ͚ͩͰදʹͳ͍ͬͯ ͯɺෳ਺ͷϝτϦΫεΛࠞͥΔͱਏ͍ Ͱ΋user, systemͷCPU࣌ؒͱ࣮࣌ؒΛಉ࣌ʹग़͢Έ͍ͨͳͷ΋΍Γ͍ͨ (Ϟσϧ্͸ରԠ͚ͨ͠Ͳ࣮͸·ͩ͜Ε࣮૷ͯ͠ͳ͍) શͯͷϢʔεέʔεΛΧόʔͭͭ͠΋ഁ୼͍ͯ͠ͳ͍ϞσϧΛͲ͏ʹ͔ ࡞Εͳ͍͔?

35. v0.11~v0.12ݱࡏͷϞσϧ

36. v0.11~v0.12ݱࡏͷϞσϧ

37. v0.11~v0.12ݱࡏͷϞσϧ

38. ݕ౼ͨ͠ࣄ߲ HashͷΩʔ͸ඞͣΠϛϡʔλϒϧͳ΋ͷʹ͢Δ ͡Όͳ͍ͱOutputϓϥάΠϯ͕Hash͔Β஋Λࢀর͢Δ࣌ʹࠔΔ Outputͷ͠΍͢͞ΑΓRunnerͷ࣮૷͕ഁ୼͠ͳ͍ͷΛ༏ઌͨ͠ߏ଄ ಉ࣌ʹऔΒΕΔϝτϦΫε͕ͳΔ΂͍ۙ͘৔ॴʹདྷΔߏ଄ Output͸࠷ѱ࠷ޙʹ·ͱΊͯͲ͏ʹ͔Ͱ͖ΔͨΊ

39. 3. ϓϥάΠϯΞʔΩςΫνϟ

40. աڈͷϓϥάΠϯࣄྫ ϓϥάΠϯαϙʔτͷ͋Δ๻ͷ͜Ε·ͰͷϓϩδΣΫτ cookpad/barbeque itamae-kitchen/mitamae benchmark-driver/benchmark-driver

41. ڞ௨఺: ࣗ෼͔͠ϓϥάΠϯॻ͍ͯͳ͍ ͔͠΋ࣗݾධՁͰbenchmark-driverͷΠϯλʔϑΣʔε͕Ұ൪ ΍͹͍ ͦ΋ͦ΋͋·Γந৅Խʹ޲͍ͯͳ͍ཁ݅ͳؾ΋͍ͯ͠Δ ͳͷͰ࿩൒෼ʹฉ͍͓͍͍ͯͯͩ͘͞

42. ϓϥάΠϯγεςϜͷ֓ཁ

43. 2छྨͷϓϥάΠϯαϙʔτ RunnerϓϥάΠϯ: ܭଌํ๏ͱܭଌ͍ͨ͠ϝτϦΫεΛܾΊΔ OutputϓϥάΠϯ: ܭଌ݁ՌͷूܭΛߦͳ͏

44. RunnerϓϥάΠϯ Ruby API͔Β࣮ߦ͞ΕΔ͕ɺYAMLͷύʔε΋͜ͷRunnerϓ ϥάΠϯ͕ߦͳ͏ RunnerͷType͸benchmark-driverίϚϯυͰ༩͑ΒΕΔ͕ɺ YAMLͷதͰ΋ॻ͚ΔͷͰɺYAML಺ͰTypeΛݻఆͭͭ͠ಠ ࣗͷYAMLύʔαΛॻ͘ͱ޷͖์୊YAMLͷจ๏͕֦ுͰ͖ Δ

45. RunnerϓϥάΠϯ σϑΥϧτͷYAMLఆٛͷ΋ͷ: ips: ࣮ߦ࣌ؒ/࣮ߦճ਺Λܭଌ͢Δ once: ςετ༻ʹ1ճ͚ͩ૸ΒͤΔ time: ࣮ߦ࣌ؒ memory: Max

    Resident Set Size (ࠓ͸LinuxͷΈαϙʔτ)

46. RunnerϓϥάΠϯ ಠࣗͷYAMLఆٛͷ΋ͷ: recorded: record outputΛ࢖͏ͱੜ੒͞ΕΔYAMLͰɺಉ͡ܭ ଌ݁ՌΛԿ౓΋ग़ྗͰ͖Δ ruby_stdout: RubyίϚϯυͷstdoutΛύʔεͯ͠ϝτϦΫε ΛऔΔ

47. memory

48. memory

49. Optcarrot

50. Optcarrot

51. Runnerʹ͍ͭͯҙਤతʹαϙʔτΛ΍Ίͨػೳ Ruby͔Β࣮ߦ࣌ɺจࣈྻ͡Όͳͯ͘ϒϩοΫͰܭଌର৅Λ༩͑Δͷ΋α ϙʔτ͍ͯͨ͠ Ҡߦύε + ෳ਺όʔδϣϯ͕ࢦఆ͞Εͳ͚Ε͹evalͰ૸Βͤͨํ͕଎͍ ࣮͸࠷ॳ͸͜ͷػೳ͕Runnerͱͯ͠ϓϥάΠϯαϙʔτ͞Ε͍ͯͨ(3छ ྨͷϓϥάΠϯ͕͋ͬͨ) υϥΠόͱಉ໊͡લۭؒͰ૸ΒͤΔ෼࣮ߦ͕ΫϦʔϯ͡Όͳ͘ɺಉ͡εΫ ϦϓτΛ࣮ߦ͢Δʹ͸αϙʔτ͕ਏ͔ͬͨ͠ϝϦοτ΋ബ͍ͷͰ΍Ίͨ

52. v0.11~12ͷRunnerϓϥάΠϯͷॻ͖ํ YAMLΛந৅Խ͢ΔJobͱɺYAML͔ΒͦΕΛ࡞ΔJobParserΛఆٛ͢Δ #initializeʹ͸ɺCLIΦϓγϣϯΛ࣋ͭconfig, OutputΛந৅Խ͢Δ output, δϣϒͷϚτϦΫεΛ࡞Δcontexts͕༩͑ΒΕΔ #run ͸޷͖์୊࣮૷͍͍͕ͯ͠ɺoutputʹରͯ͠ϝιου܊Λܾ·ͬ ͨॱ൪ʹݺ͹ͳ͚Ε͹ͳΒͳ͍ #with_warmup,

    #with_benchmark, #with_job, #with_context, #report

53. OutputϓϥάΠϯ લड़ͨ͠ϞσϧΛड͚औͬͯूܭ͢Δ໾ׂ 1ͭܭଌ͢Δ͝ͱʹoutputʹ౉ͤΔΠϯλʔϑΣʔεͳͷͰɺ ݁Ռ͕ετϦʔϛϯάग़ྗͰ͖Δ શ෦ऴΘͬͨޙʹόϧΫͰग़ྗ͢ΔͨΊͷ؆୯ͳΠϯλʔ ϑΣʔε΋༻ҙ͍ͯ͠Δ

54. OutputϓϥάΠϯ ϏϧτΠϯͷϓϥάΠϯ compare: benchmark-ips෩ͷൺֱग़ྗ simple: benchmark/driver.rb ෩ग़ྗ markdown: GFMͷςʔϒϧग़ྗ record:

    ܭଌ݁ՌΛอଘͯ͠ผͷoutputʹྲྀ͢༻

55. OutputϓϥάΠϯ ඇެࣜϓϥάΠϯ gruff: gruff.gemͰάϥϑ͕ग़ྗͰ͖Δ rubybench: RubyBenchʹ݁ՌΛૹ৴

56. Markdown

57. gruff

58. ϓϥάΠϯʹؔ͢Δ஫ҙࣄ߲ v0ͷؒԿ౓΋ϓϥάΠϯΠϯλʔϑΣʔε͕มΘ͍ͬͯͯɺv1ʹ͢Δ·Ͱ͸ ֬ఆ͡Όͳ͍Ͱ͢ Runner Typeݻ༗ͷprelude͕άϩʔόϧͳcontexʹೖͬͯΔͷ͸Ҿ͖ണ͕ ͢༧ఆ ϓϥάΠϯΛ࡞ͬͯͩ͘͞Δํ͸ґଘΛ "benchmark_driver", "~> v0.X.Y"

    ʹ ౰໘͸͍͚ͯͨͩ͠Δͱॿ͔Γ·͢ ಉ͡ v0.X ͷؒ͸յ͞ͳ͍Α͏ʹ͍ͯ͠·͢

59. 4. ϓϩδΣΫτऴྃޙͷ׆ಈ

60. RubyBench

61. RubyBenchͱ͸ RubyͷύϑΥʔϚϯεΛ؂ࢹ͢ΔCI benchmark-driver.github.io ΑΓ΋େ෼લ͔Β͋Δ

62. RubyBenchͷΞʔΩςΫνϟ RubyBench ϫʔΧʔ task-spooler → docker run ruby … GitHub

    fork for ruby/ruby RubyBench web Rails ΞϓϦέʔγϣϯ → Sidekiq GitHub ruby/ruby cronͰߋ৽ Webhook ssh tsp …

63. ͜͜Λஔ͖׵͍͑ͨ RubyBench ϫʔΧʔ task-spooler → docker run ruby … GitHub

    fork for ruby/ruby RubyBench web Rails ΞϓϦέʔγϣϯ → Sidekiq GitHub ruby/ruby cronͰߋ৽ Webhook ssh tsp …

64. RubyBench΁ͷ౷߹ JITͱͷൺֱάϥϑ͸ޙ͔ΒRubyBenchʹ΋࣮૷͞Εͨ όάͬͯͨͷͰ௚ͨ͠Γͨ͠ benchmark_driver޲͚ͷϕϯνϚʔΫηοτ΍ਫ਼౓ͷྑ͍ܭଌػೳ͕ࠓޙ׆༻͠ ΍͍͢Α͏ɺϕϯνϚʔΧʔͷbenchmark_driver΁ͷஔ͖׵͑Λ΍͍ͬͯΔ طʹOptcarrot͸benchmark_driverͰܭଌ͞Ε͍ͯΔ ॏෳ͕ͪ͠ͳ࣮૷͕ϓϥάΠϯͰந৅ԽͰ͖͓ͯΓbenchmark_driverͷϓϥά Πϯػߏ͕׆༂͍ͯ͠Δ

65. RubyBench޲͚ʹɺϕϯνϚʔΫͷ֦ॆ΋΍͍͖ͬͯ·͢ VladͷMJIT޲͚ϕϯνϚʔΫूΛ௥Ճ༧ఆ Watson1978͞Μͷ֤ϝιου޲͚ϕϯνϚʔΫूΛม׵ͨ͠ ΋ͷ΋࢖Θ͍͔ͤͯͨͩ͘͞΋? https://github.com/benchmark-driver/ruby-method-benchmarks

66. ·ͱΊ benchmark_driver ศརͰ͢ͷͰɺੋඇ͝ར༻͍ͩ͘͞ Rubyͷߴ଎Խ΍RubyBenchʹطʹຊ੒Ռ͕׆༻͞Ε͍ͯ·͢

67. ँࣙ ຊϓϩδΣΫτ͸ɺ࡫ాߞҰ༷Λ࢝Ίͱͨ͠RubyΞιγΤʔ γϣϯ༷ͷ͝ڠྗʹΑ࣮ͬͯݱ͢Δ͜ͱ͕Ͱ͖·ͨ͠ɻ͜ͷ ৔ΛआΓͯޚྱਃ্͛͠·͢ɻ