Improving the Benchmark library

Transcript

1. Improving the Benchmark library Benoit Daloze February 8, 2011
2. require “benchmark” # # benchmark.rb - a performance benchmarking library # # # Created by Gotoken ( gotoken@notwork .org). # # Documentation by Gotoken (original RD), Lyle Johnson (RDoc conversion), and # Gavin Sinclair (editing). #
3. Benchmark Probably one of the most useful tool in the standard library But, no maintainer Which is not a big problem, given it does not need to change much with time, and it has a simple basic API But, it could be better
4. API: realtime and measure Benchmark.realtime { sleep 1 } # => 1.0000250339508057 Benchmark.measure { sleep 1 } # => Benchmark ::Tms # user system total real 0.000000 0.000000 0.000000 ( 1.000104)
5. API: bm n = 5_000_000 Benchmark.bm(6) do |x| x.report("for:") { for i in 1..n; end } x.report("times:") { n.times {} } x.report("upto:") { 1. upto(n) {} } end user system total real for: 0.550000 0.000000 0.550000 ( 0.552241) times: 0.540000 0.000000 0.540000 ( 0.546636) upto: 0.530000 0.010000 0.540000 ( 0.552317)
6. API: bmbm n = 5_000_000 Benchmark.bmbm do |x| x.report("for:") { for i in 1..n; end } x.report("times:") { n.times {} } x.report("upto:") { 1. upto(n) {} } end Rehearsal ------------------------------------------ for: 0.570000 0.000000 0.570000 ( 0.568279) times: 0.530000 0.000000 0.530000 ( 0.537404) upto: 0.540000 0.000000 0.540000 ( 0.545920) --------------------------------- total: 1.640000sec user system total real for: 0.570000 0.010000 0.580000 ( 0.564945) times: 0.540000 0.000000 0.540000 ( 0.548760) upto: 0.550000 0.000000 0.550000 ( 0.548751)
7. API Benchmark realtime measure benchmark bm bmbm Report (yielded by bm and benchmark) item alias report Job (yielded by bmbm) item alias report list: list of items width: maximum width of items Tms operators + - * / format tms.format(’%u %r’)
8. What is it then ? The code is old (no blame, just a fact) There is some duplication The API could be improved
9. The code is old sum = Tms.new list.each { |i| sum += i } raise ArgumentError , "no block" unless iterator? printf("%s %s\n\n", "-"*some_length , ets) print "Rehearsal " puts ’-’*( some_length - "Rehearsal ".length)
10. Can you guess what it should be ?
11. The code is old - sum = Tms.new - list.each { |i| sum += i } + list.inject(Tms.new , :+) - raise ArgumentError , "no block" unless iterator? + # useless , yield will raise a LocalJumpError if no block is given - printf("%s %s\n\n", "-"*some_length , ets) + print "#{’-’* some_length} #{ets}\n\n" - print "Rehearsal " - puts ’-’*( some_length - "Rehearsal ".length) + puts ’Rehearsal ’.ljust(some_length , ’-’)
12. Update the code 18 commits to clean/update the code Status: “* lib/benchmark.rb: fix benchmarck to work with current ruby. patched by Benoit Daloze [ruby-core:33846] [ruby-dev:43143] merged from github.com/eregon/ruby/commits/benchmark” So it’s merged on trunk (r30747) With the first minitest/spec specs And a nice comment from Kosaki: “At minimum, your code is very clean and good readable. therefore I could find the test failure reason and fix it quickly.”
13. There is some duplication def bmbm(width = 0, &blk) # :yield: job job = Job.new(width) yield(job) width = job.width sync = STDOUT.sync STDOUT.sync = true # rehearsal print "Rehearsal " puts ’-’*( width+CAPTION.length - "Rehearsal ".length) list = [] job.list.each {|label ,item| print(label.ljust(width)) res = Benchmark :: measure (& item) print res.format () list.push res } sum = Tms.new; list.each {|i| sum += i} ets = sum.format("total: %tsec") printf("%s %s\n\n", "-"*( width+CAPTION.length -ets.length -1) , ets) # take print ’ ’*width , CAPTION list = [] ary = [] job.list.each {|label ,item| GC:: start print label.ljust(width) res = Benchmark :: measure (& item) print res.format () ary.push res list.push [label , res] } STDOUT.sync = sync ary end def benchmark(caption = "", label_width = nil , fmtstr = nil , *labels) # :yield: report sync = STDOUT.sync STDOUT.sync = true label_width ||= 0 fmtstr ||= FMTSTR raise ArgumentError , "no block" unless iterator? print caption results = yield(Report.new(label_width , fmtstr)) Array === results and results.grep(Tms).each {|t| print (( labels.shift || t.label || ""). ljust( label_width ), t.format(fmtstr)) } STDOUT.sync = sync end
14. After some cleaning def bmbm(width = 0, &blk) # :yield: job job = Job.new(width) yield(job) width = job.width sync = STDOUT.sync STDOUT.sync = true # rehearsal puts ’Rehearsal ’.ljust(width+CAPTION.length ,’-’) ets = job.list.inject(Tms.new) { |sum ,(label ,item)| print label.ljust(width) res = Benchmark.measure (& item) print res.format sum + res }. format("total: %tsec") print " #{ ets }\n\n".rjust(width+CAPTION.length +2,’-’) # take print ’ ’*width + CAPTION job.list.map { |label ,item| GC.start print label.ljust(width) Benchmark.measure (& item).tap { |res| print res. format } }. tap { STDOUT.sync = sync } end def benchmark(caption = "", label_width = nil , format = nil , *labels) # :yield: report sync = STDOUT.sync STDOUT.sync = true label_width ||= 0 format ||= FORMAT print ’ ’* label_width + caption report = Report.new(label_width , format) results = yield(report) Array === results and results.grep(Tms).each {|t| print (( labels.shift || t.label || ""). ljust( label_width ), t.format(format )) } STDOUT.sync = sync report.list end
15. After some refactoring def bmbm( label_width = nil) # :yield: report report = Report.new( label_width ) yield(report) width ||= report. label_width # rehearsal puts "Rehearsal ".ljust(width + CAPTION.length) report.run ets = report.sum.format("total: %tsec") print " #{ ets }\n\n" puts ’-’*( width+CAPTION.length) # take print ’ ’*width , CAPTION report.run(: with_gc) end def benchmark(caption = "", label_width = nil , format = FORMAT , *labels) # :yield: report report = Report.new(label_width , *labels) yield(report) label_width ||= report. label_width print ’ ’* label_width + caption results = report.run Array === results and results.grep( DelayedOperation ).each do |proc| tms = proc.compute print (labels.shift || tms.label || ""). ljust( label_width ), tms.format( format) end results end
16. So what’s the code doing? def bmbm(label_width = nil) # :yield: report report = Report.new(label_width) yield(report) width ||= report.label_width # rehearsal puts "Rehearsal ".ljust(width+CAPTION.length) report.run ets = report.sum.format("total: %tsec") print " #{ets}\n\n" puts ’-’*( width+CAPTION.length) # take print ’ ’*width , CAPTION report.run(: with_gc) end
17. Report and Job are very similar Report#report run immediately when you call and print the results Job#report just save the block and title, leaving bmbm do the job
18. The API could be improved module Benchmark def bm(label_width = 0, *labels , &blk) benchmark (...) end class Report def item(label , &blk) print label.ljust(@width) print tms = Benchmark.measure (&blk) tms # returns the measured time end end end
19. The API could be improved n = 5_000_000 Benchmark.bm(7, ">total:", ">avg:") do |x| f = x.report("for:") { for i in 1..n; end } t = x.report("times:") { n.times {} } u = x.report("upto:") { 1. upto(n) {} } [f+t+u, (f+t+u)/3] end user system total real for: 0.550000 0.000000 0.550000 ( 0.544705) times: 0.520000 0.000000 0.520000 ( 0.521256) upto: 0.520000 0.000000 0.520000 ( 0.519856) >total: 1.590000 0.000000 1.590000 ( 1.585816) >avg: 0.530000 0.000000 0.530000 ( 0.528605)
20. Idea of a new API Benchmark Report (yielded by bm, bmbm and benchmark) Job: created by Report#item alias report
21. A solution All blocks could be stored and run when the main block is closed But, we can not easily return the Tms Or ... We could with ...
22. DelayedOperation class Job attr_reader :label , :time include Delayable.on(: time) def run @time = Benchmark.measure(@label , &@block) end end a = Job.new(’upto ’) { 1. upto(n) {} } b = Job.new(’times ’) { n.times {} } sum = a + b # => # <# DelayedOperation #<Job upto >, +, #<Job times >> a.run , b.run sum.compute # => 1.041112
23. Benchmark.compare Benchmark.bm do |x| x.compare(A.new , B.new) { |o| o.compute_sth } end # Or , it could be a new scope Benchmark.compare(A.new , B.new) do |x| x.report { |o| o.do_sth } end
24. Metaprogamming over-use Benchmark.bmbm do |x| [Syck , Psych ]. each do |impl| x.report("#{ impl }# dump") { N.times { @yaml = impl.dump(data) } } x.report("#{ impl }# load") { N.times { impl.load(@yaml) } } end end
25. With #compare Benchmark.compare(Syck , Psych) do |x| x.report (: dump) { |impl| N.times { @yaml = impl.dump(data) } } x.report (: load) { |impl| N.times { impl.load(@yaml) } } end
26. One-line ! Benchmark.compare Syck , Psych , dump: data , load: :dump
27. Current output Rehearsal --------------------------------------------- Syck#dump 0.500000 0.000000 0.500000 ( 0.503858) Syck#load 0.080000 0.000000 0.080000 ( 0.085303) Psych#dump 0.460000 0.000000 0.460000 ( 0.459000) Psych#load 0.290000 0.010000 0.300000 ( 0.289525) ------------------------------------ total: 1.340000sec user system total real Syck#dump 0.500000 0.000000 0.500000 ( 0.499590) Syck#load 0.070000 0.000000 0.070000 ( 0.079536) Psych#dump 0.460000 0.000000 0.460000 ( 0.457742) Psych#load 0.280000 0.000000 0.280000 ( 0.280588)
28. Comparative Output Syck Psych Syck/Psych dump 0.499590 0.457742 (1.0914) load 0.079536 0.280588 (0.2835)
29. Thanks for listening Any question ? What do you think ? structure: Benchmark → Report → Job DelayedOperation Benchmark.compare Do you have some idea to improve Benchmark ?