Slide 1

Slide 1 text

Improving the Benchmark library Benoit Daloze February 8, 2011

Slide 2

Slide 2 text

require “benchmark” # # benchmark.rb - a performance benchmarking library # # # Created by Gotoken ( gotoken@notwork .org). # # Documentation by Gotoken (original RD), Lyle Johnson (RDoc conversion), and # Gavin Sinclair (editing). #

Slide 3

Slide 3 text

Benchmark Probably one of the most useful tool in the standard library But, no maintainer Which is not a big problem, given it does not need to change much with time, and it has a simple basic API But, it could be better

Slide 4

Slide 4 text

API: realtime and measure Benchmark.realtime { sleep 1 } # => 1.0000250339508057 Benchmark.measure { sleep 1 } # => Benchmark ::Tms # user system total real 0.000000 0.000000 0.000000 ( 1.000104)

Slide 5

Slide 5 text

API: bm n = 5_000_000 Benchmark.bm(6) do |x| x.report("for:") { for i in 1..n; end } x.report("times:") { n.times {} } x.report("upto:") { 1. upto(n) {} } end user system total real for: 0.550000 0.000000 0.550000 ( 0.552241) times: 0.540000 0.000000 0.540000 ( 0.546636) upto: 0.530000 0.010000 0.540000 ( 0.552317)

Slide 6

Slide 6 text

API: bmbm n = 5_000_000 Benchmark.bmbm do |x| x.report("for:") { for i in 1..n; end } x.report("times:") { n.times {} } x.report("upto:") { 1. upto(n) {} } end Rehearsal ------------------------------------------ for: 0.570000 0.000000 0.570000 ( 0.568279) times: 0.530000 0.000000 0.530000 ( 0.537404) upto: 0.540000 0.000000 0.540000 ( 0.545920) --------------------------------- total: 1.640000sec user system total real for: 0.570000 0.010000 0.580000 ( 0.564945) times: 0.540000 0.000000 0.540000 ( 0.548760) upto: 0.550000 0.000000 0.550000 ( 0.548751)

Slide 7

Slide 7 text

API Benchmark realtime measure benchmark bm bmbm Report (yielded by bm and benchmark) item alias report Job (yielded by bmbm) item alias report list: list of items width: maximum width of items Tms operators + - * / format tms.format(’%u %r’)

Slide 8

Slide 8 text

What is it then ? The code is old (no blame, just a fact) There is some duplication The API could be improved

Slide 9

Slide 9 text

The code is old sum = Tms.new list.each { |i| sum += i } raise ArgumentError , "no block" unless iterator? printf("%s %s\n\n", "-"*some_length , ets) print "Rehearsal " puts ’-’*( some_length - "Rehearsal ".length)

Slide 10

Slide 10 text

Can you guess what it should be ?

Slide 11

Slide 11 text

The code is old - sum = Tms.new - list.each { |i| sum += i } + list.inject(Tms.new , :+) - raise ArgumentError , "no block" unless iterator? + # useless , yield will raise a LocalJumpError if no block is given - printf("%s %s\n\n", "-"*some_length , ets) + print "#{’-’* some_length} #{ets}\n\n" - print "Rehearsal " - puts ’-’*( some_length - "Rehearsal ".length) + puts ’Rehearsal ’.ljust(some_length , ’-’)

Slide 12

Slide 12 text

Update the code 18 commits to clean/update the code Status: “* lib/benchmark.rb: fix benchmarck to work with current ruby. patched by Benoit Daloze [ruby-core:33846] [ruby-dev:43143] merged from github.com/eregon/ruby/commits/benchmark” So it’s merged on trunk (r30747) With the first minitest/spec specs And a nice comment from Kosaki: “At minimum, your code is very clean and good readable. therefore I could find the test failure reason and fix it quickly.”

Slide 13

Slide 13 text

There is some duplication def bmbm(width = 0, &blk) # :yield: job job = Job.new(width) yield(job) width = job.width sync = STDOUT.sync STDOUT.sync = true # rehearsal print "Rehearsal " puts ’-’*( width+CAPTION.length - "Rehearsal ".length) list = [] job.list.each {|label ,item| print(label.ljust(width)) res = Benchmark :: measure (& item) print res.format () list.push res } sum = Tms.new; list.each {|i| sum += i} ets = sum.format("total: %tsec") printf("%s %s\n\n", "-"*( width+CAPTION.length -ets.length -1) , ets) # take print ’ ’*width , CAPTION list = [] ary = [] job.list.each {|label ,item| GC:: start print label.ljust(width) res = Benchmark :: measure (& item) print res.format () ary.push res list.push [label , res] } STDOUT.sync = sync ary end def benchmark(caption = "", label_width = nil , fmtstr = nil , *labels) # :yield: report sync = STDOUT.sync STDOUT.sync = true label_width ||= 0 fmtstr ||= FMTSTR raise ArgumentError , "no block" unless iterator? print caption results = yield(Report.new(label_width , fmtstr)) Array === results and results.grep(Tms).each {|t| print (( labels.shift || t.label || ""). ljust( label_width ), t.format(fmtstr)) } STDOUT.sync = sync end

Slide 14

Slide 14 text

After some cleaning def bmbm(width = 0, &blk) # :yield: job job = Job.new(width) yield(job) width = job.width sync = STDOUT.sync STDOUT.sync = true # rehearsal puts ’Rehearsal ’.ljust(width+CAPTION.length ,’-’) ets = job.list.inject(Tms.new) { |sum ,(label ,item)| print label.ljust(width) res = Benchmark.measure (& item) print res.format sum + res }. format("total: %tsec") print " #{ ets }\n\n".rjust(width+CAPTION.length +2,’-’) # take print ’ ’*width + CAPTION job.list.map { |label ,item| GC.start print label.ljust(width) Benchmark.measure (& item).tap { |res| print res. format } }. tap { STDOUT.sync = sync } end def benchmark(caption = "", label_width = nil , format = nil , *labels) # :yield: report sync = STDOUT.sync STDOUT.sync = true label_width ||= 0 format ||= FORMAT print ’ ’* label_width + caption report = Report.new(label_width , format) results = yield(report) Array === results and results.grep(Tms).each {|t| print (( labels.shift || t.label || ""). ljust( label_width ), t.format(format )) } STDOUT.sync = sync report.list end

Slide 15

Slide 15 text

After some refactoring def bmbm( label_width = nil) # :yield: report report = Report.new( label_width ) yield(report) width ||= report. label_width # rehearsal puts "Rehearsal ".ljust(width + CAPTION.length) report.run ets = report.sum.format("total: %tsec") print " #{ ets }\n\n" puts ’-’*( width+CAPTION.length) # take print ’ ’*width , CAPTION report.run(: with_gc) end def benchmark(caption = "", label_width = nil , format = FORMAT , *labels) # :yield: report report = Report.new(label_width , *labels) yield(report) label_width ||= report. label_width print ’ ’* label_width + caption results = report.run Array === results and results.grep( DelayedOperation ).each do |proc| tms = proc.compute print (labels.shift || tms.label || ""). ljust( label_width ), tms.format( format) end results end

Slide 16

Slide 16 text

So what’s the code doing? def bmbm(label_width = nil) # :yield: report report = Report.new(label_width) yield(report) width ||= report.label_width # rehearsal puts "Rehearsal ".ljust(width+CAPTION.length) report.run ets = report.sum.format("total: %tsec") print " #{ets}\n\n" puts ’-’*( width+CAPTION.length) # take print ’ ’*width , CAPTION report.run(: with_gc) end

Slide 17

Slide 17 text

Report and Job are very similar Report#report run immediately when you call and print the results Job#report just save the block and title, leaving bmbm do the job

Slide 18

Slide 18 text

The API could be improved module Benchmark def bm(label_width = 0, *labels , &blk) benchmark (...) end class Report def item(label , &blk) print label.ljust(@width) print tms = Benchmark.measure (&blk) tms # returns the measured time end end end

Slide 19

Slide 19 text

The API could be improved n = 5_000_000 Benchmark.bm(7, ">total:", ">avg:") do |x| f = x.report("for:") { for i in 1..n; end } t = x.report("times:") { n.times {} } u = x.report("upto:") { 1. upto(n) {} } [f+t+u, (f+t+u)/3] end user system total real for: 0.550000 0.000000 0.550000 ( 0.544705) times: 0.520000 0.000000 0.520000 ( 0.521256) upto: 0.520000 0.000000 0.520000 ( 0.519856) >total: 1.590000 0.000000 1.590000 ( 1.585816) >avg: 0.530000 0.000000 0.530000 ( 0.528605)

Slide 20

Slide 20 text

Idea of a new API Benchmark Report (yielded by bm, bmbm and benchmark) Job: created by Report#item alias report

Slide 21

Slide 21 text

A solution All blocks could be stored and run when the main block is closed But, we can not easily return the Tms Or ... We could with ...

Slide 22

Slide 22 text

DelayedOperation class Job attr_reader :label , :time include Delayable.on(: time) def run @time = Benchmark.measure(@label , &@block) end end a = Job.new(’upto ’) { 1. upto(n) {} } b = Job.new(’times ’) { n.times {} } sum = a + b # => # <# DelayedOperation #, +, #> a.run , b.run sum.compute # => 1.041112

Slide 23

Slide 23 text

Benchmark.compare Benchmark.bm do |x| x.compare(A.new , B.new) { |o| o.compute_sth } end # Or , it could be a new scope Benchmark.compare(A.new , B.new) do |x| x.report { |o| o.do_sth } end

Slide 24

Slide 24 text

Metaprogamming over-use Benchmark.bmbm do |x| [Syck , Psych ]. each do |impl| x.report("#{ impl }# dump") { N.times { @yaml = impl.dump(data) } } x.report("#{ impl }# load") { N.times { impl.load(@yaml) } } end end

Slide 25

Slide 25 text

With #compare Benchmark.compare(Syck , Psych) do |x| x.report (: dump) { |impl| N.times { @yaml = impl.dump(data) } } x.report (: load) { |impl| N.times { impl.load(@yaml) } } end

Slide 26

Slide 26 text

One-line ! Benchmark.compare Syck , Psych , dump: data , load: :dump

Slide 27

Slide 27 text

Current output Rehearsal --------------------------------------------- Syck#dump 0.500000 0.000000 0.500000 ( 0.503858) Syck#load 0.080000 0.000000 0.080000 ( 0.085303) Psych#dump 0.460000 0.000000 0.460000 ( 0.459000) Psych#load 0.290000 0.010000 0.300000 ( 0.289525) ------------------------------------ total: 1.340000sec user system total real Syck#dump 0.500000 0.000000 0.500000 ( 0.499590) Syck#load 0.070000 0.000000 0.070000 ( 0.079536) Psych#dump 0.460000 0.000000 0.460000 ( 0.457742) Psych#load 0.280000 0.000000 0.280000 ( 0.280588)

Slide 28

Slide 28 text

Comparative Output Syck Psych Syck/Psych dump 0.499590 0.457742 (1.0914) load 0.079536 0.280588 (0.2835)

Slide 29

Slide 29 text

Thanks for listening Any question ? What do you think ? structure: Benchmark → Report → Job DelayedOperation Benchmark.compare Do you have some idea to improve Benchmark ?