Upgrade to Pro — share decks privately, control downloads, hide ads and more …

An introduction and future of Ruby coverage library

Yusuke Endoh
September 22, 2017

An introduction and future of Ruby coverage library

Yusuke Endoh

September 22, 2017
Tweet

More Decks by Yusuke Endoh

Other Decks in Programming

Transcript

  1. Yusuke Endoh (@mametter) • Ruby committer (2008—) • Full-time Ruby

    committer (2017—) • My goal: make Ruby programs robust – Test coverage, and type system?
  2. Method. Put the cream cheese into the mixing bowl. Put

    the sour cream into the mixing bowl. Put the white sugar into the mixing bowl. Put the yogrut into the mixing bowl. Put the unsalted butter into the mixing bowl. Combine the milk. Put the cake flour into the mixing bowl. Combine the corn starch. Put the brown sugar into the mixing bowl. Combine the egg whites. Combine the egg yolks. Put the lemon juice into the mixing bowl. Stir for 7 minutes. Liquify the contents of the mixing bowl. Pour contents of the mixing bowl into the baking dish Bake the cake mixture. Watch the cake mixture until baked. Serves 4. Cheese cake in Chef. Ingredients. 100 g cream cheese 97 g sour cream 107 g yogrut 112 g white sugar 11 g brown sugar 37 g unsalted butter 37 g cake flour 3 g corn starch 3 ml milk 3 egg yolks 3 egg whites 10 ml lemon juice 0 g cake mixture Cooking time: 80 minutes. ‘d’ ‘a’ ‘k’ ‘p’ 11*3*3=99 ‘c’ 37*3=111 ‘o’ ¥n ‘o’ Push characters ‘¥n’ ‘d’ ‘a’ ‘p’ ‘k’ ‘o’ ‘o’ ‘c’ into the stack Convert them to a string and “serve” it. data code
  3. Esoteric Recipe • Polyglot of Chef and real recipe –

    Japanese version https://cookpad.com/ recipe/4649810 – English version https://cookpad.com/us/ recipes/3335222
  4. My main contributions for Ruby • Implementation of some features:

    keyword arguments, deadlock detection, etc. • Release management for Ruby 1.9.2 and 2.0.0 • Optcarrot: A NES emulator for Ruby3x3 benchmark • Enhancement of the test suite of Ruby • coverage.so: the core library for coverage measurement ’06B ’07A ’07B ’08A 60 70 80 90 100 coverage (%) 70% 85% line coverage
  5. Today’s theme • An introduction of test coverage • An

    improvement plan of Ruby’s coverage measurement feature towards 2.5
  6. Survey [3/3] • Q. In those, do you use “coverage”?

    – Do you check the result of SimpleCov?
  7. Agenda • What is coverage • How to understand and

    use coverage • The current status of Ruby coverage feature • The future plan of Ruby coverage feature • Conclusion
  8. Agenda ☞ What is coverage • How to understand and

    use coverage • The current status of Ruby coverage feature • The future plan of Ruby coverage feature • Conclusion
  9. What is coverage? • A measure of “goodness” of a

    test suite – Also called “test coverage” or “code coverage” • Allows you: – Find untested code – Decide whether your test suite is good enough or not yet • (This is arguable, but I think we can use it as an advice) • Types of coverage – Function coverage, line coverage, branch coverage, …
  10. Function coverage • How many functions are executed by the

    tests # code def foo; …; end # ✓ def bar; …; end # ✗ def baz; …; end # ✓ # test foo baz 2/3 (67%) • Advantage • Easy to understand • Easy to visualize • Disadvantage • Too weak as a measure
  11. Line coverage • How many lines are executed # code

    def foo(x) # ✓ if x == 0 # ✓ p :foo # ✗ else p :bar # ✓ end end # test foo(1) 3/4 (75%) Non-significant line is ignored • Advantage • Easy to understand • Easy to visualize • Disadvantage • Still weak as a measure • foo() if x == 0
  12. Branch coverage • How many branches are taken true and

    false # code def foo(x) p :foo if x == 0 # ✓ p :bar if x < 2 # ✗ end # test foo(0) foo(1) 1/2 (50%) • Advantage • Relatively exhaustive • Disadvantage • Difficult to visualize true-case and false-case are both executed
  13. Coverage types Coverage type Easy to understand/ visualize Exhaustive Function

    coverage ◦ ✕ Line coverage ◦ △ Branch coverage △ ◦ Currently, Ruby supports only line coverage
  14. Other types of coverage • Condition coverage – How many

    conditions (not branches) are taken both true and false • Path coverage – How many paths are executed – Combinatorial explosion • Other advanced ones – Data-flow coverage – MC/DC coverage if a && b branch condition condition
  15. Trivia • “C0/C1/C2 coverages” have difference meanings to different people

    – C0 coverage = line coverage – C1 coverage = branch coverage or path coverage? – C2 coverage = condition coverage or path coverage?
  16. Coverage and Ruby • In Ruby, Coverage is crucial! –

    A test is the only way to ensure quality – Coverage is important to measure test goodness • Considering it, coverage is not used so much… – Coverage is not well-known? – It is not well-known how to use coverage? – Ruby’s coverage measurement feature is not enough? • I’d like to improve the situation with this talk
  17. Agenda • What is coverage ☞ How to understand and

    use coverage • The current status of Ruby coverage feature • The future plan of Ruby coverage feature • Conclusion
  18. What is a good test suite? • Covers the code

    – Coverage measures this • Also covers the design of program – Coverage does not measure this directly
  19. If coverage XX% is required as a goal… • Developers

    will 1. Pick untested code that looks easiest to cover 2. Write a trivial test just for the code 3. Iterate this procedure until XX% is achieved • It will result in trivial, not-so-good test suite – It may be exhaustive for the code itself, but – It won't be exhaustive for the design
  20. A good way to improve coverage • Developers should 1.

    Look at untested code 2. Consider what “test design” is insufficient 3. Write them – In consequence of them, the untested code is executed • It will result in good test suite – It will be exhaustive not only for the code but also for the design
  21. How many % is needed/enough? • It depends upon the

    module being tested – Aim 100% for a significant module (e.g., injure someone) – Don't worry too much for a non-significant module • It also depends upon cost performance – For non-significant module, write a test only if it is not so hard • Again: Coverage is not a goal
  22. Agenda • What is coverage • How to understand and

    use coverage ☞ The current status of Ruby coverage feature • The future plan of Ruby coverage feature • Conclusion
  23. SimpleCov • A wrapper library for coverage.so • Visualization with

    HTML • Useful features: merging, filtering, for Rails app • Author: Christoph Olszowka (@colszowka)
  24. Usage of SimpleCov • Write this at the top of

    test/test_helper.rb • Run the test suite • coverage/index.html will be produced – Note: SimpleCov cannot measure already- loaded files before SimpleCov.start require "simplecov" SimpleCov.start
  25. coverage.so • The only implementation of coverage measurement for Ruby

    1.9+ • SimpleCov is a wrapper for coverage.so • Author: me
  26. Basic usage # test.rb require "coverage" Coverage.start load "target.rb" p

    Coverage.result #=> {"target.rb"=> # [nil,nil,1,1,1,nil, # 0,nil,nil,nil,1]} Start measuring coverage Load the target file Stop measuring and get the result Coverage data
  27. Coverage data # target.rb def foo(x) if x == 0

    p 0 else p 1 end end foo(1) [nil, nil 1 1 0 nil 1 nil nil nil 1] nil: Non-significant line Number: Count executed Untested line!
  28. Method definition is code in Ruby # target.rb def foo(x)

    if x == 0 p 0 else p 1 end end [nil, nil 1 0 0 nil 0 nil nil] Method definition is counted as an execution (It is not a count of method invocation!)
  29. I regret the design of coverage.so • Support only line

    coverage • Excuse: I introduced it just for test enhancement of Ruby itself – I didn't deliberate the API for external project… • I have wanted to make the next version better ext/coverage/coverage.c: 69 /* Coverage provides coverage measurement feature for Ruby. 70 * This feature is experimental, so these APIs may be changed in future. 71 *
  30. Concov • CONtinuous COVerage – Detects temporal change (degradation) of

    coverage • Developed for monitoring Ruby's coverage • Author: me • Presented at RubyKaigi 2009, and then… – It has not been used by everyone (including me) – It was based on Ramaze (old web framework)!
  31. Coverage ecosystem for other languages • C/C++: GCOV/LCOV – Integrated

    with gcc: gcc -coverage target.c • Java: A lot of tools – Cobertura, Emma, Clover, JaCoCo – Integrated with CI tools and/or IDEs • JavaScript: Istanbul Jenkins Cobertura plugin LCOV result
  32. Agenda • What is coverage • How to understand and

    use coverage • The current status of Ruby coverage feature ☞ The future plan of Ruby coverage feature • Conclusion
  33. A plan towards Ruby 2.5 • Support function and branch

    coverage – There have been multiple requests and some PoC patches… • To make the API better, any comments are welcome – https://bugs.ruby-lang.org/issues/13901
  34. API: to start measuring # compatible layer Coverage.start Coverage.result #=>

    {"file.rb"=>[nil, 1, 0, …], … } # new API Coverage.start(lines: true) Coverage.result #=> {"file.rb" => { :lines => [nil, 1, 0, …] } }
  35. API: to start other types of coverage # enable branch

    and function coverage Coverage.start(lines:true, branches:true, methods:true) Coverage.result #=> {"file.rb" => { :lines => [nil, 1, 0, …], # :branches => {…}, # :methods => {…} } } # shorthand Coverage.start(:all)
  36. Coverage.result for branch coverage {"target1.rb"=> {:lines=>[…], :branches=>{ [:if, 0, 2]=>{

    [:then, 1, 3]=>2, [:else, 2, 5]=>1 } }, :methods=>{ [:test_if, 1]=>3 }}} # target1.rb 1: def test_if(x) 2: if x == 0 3: p "x == 0" 4: else 5: p "x != 0" 6: end 7: end 8: 9: test_if(0) 10: test_if(0) 11: test_if(1) From if at Line 2 Jumped to then clause at Line 3 twice Jumped to else clause at Line 5 once
  37. Coverage.result for branch coverage {"target2.rb"=> {:lines=>[1, 1, 1, nil, nil,

    1], :branches=> {[:if, 0, 2]=>{ [:then, 1, 2]=>1, [:else, 2, 2]=>0}, [:if, 3, 3]=>{ [:then, 4, 3]=>0, [:else, 5, 3]=>1}}, :methods=>{ [:test_if, 1]=>3 }}} # target2.rb 1: def test_if_oneline(x) 2: p "x == 0" if x == 0 3: p "x != 0" if x != 0 4: end 5: 6: test_if_oneline(0) Line coverage 100% Branch coverage tells you there are untested cases
  38. Discussion of API design • 100% compatible • [<label>, <numbering>,

    <line-no>] – e.g., [:if, 0, 1], [:while, 1, 1], [:case, 2, 1] – <numbering> is a unique ID to avoid conflicts for the case where there are multiple branches in one line • LCOV-style – Other candidates: • [<label>, <line-no>, <column-no>] – How to handle TAB character • [<label>, <offset from file head>] – Looks good, but hard to implement (I'll try later)
  39. Overhead of coverage measurement (just preliminary experiment) # Example 2

    1: foo() 2: foo() … 99: foo() 100: foo() Benchmark w/o cov. w/ cov. Overhead Example 1 0.322 μs 6.21 μs x19.3 Example 2 1.55 μs 7.16 μs x4.61 make test-all 485 s 550 s x1.13 # Example 1 1: x = 1 2: x = 1 … 99: x = 1 100: x = 1
  40. Demo • Applied the new coverage.so to Ruby • Integrated

    with C code coverage by GCOV and Visualized by LCOV Ruby code in stdlib make exam with gcc -coverage make exam COVERAGE=1 test- coverage .dat *.gcda gcov my script run test C code of MRI cov. data source aggregate lcov HTML
  41. Agenda • What is coverage • How to understand and

    use coverage • The current status of Ruby coverage feature • The future plan of Ruby coverage feature ☞ Conclusion
  42. Acknowledgement • @_ko1 • @t_wada • @kazu_cocoa • @moro •

    @makimoto • @dev_shia • @tanaka_akr • @nalsh • @spikeolaf • @k_tsj
  43. Conclusion • What is coverage, how important in Ruby, and

    how to understand coverage • The current status of Ruby's coverage measurement and ecosystem • A plan towards Ruby 2.5 and preliminary demo – Any comments are welcome! – https://bugs.ruby-lang.org/issues/13901
  44. Future work • Determine the API • define_method as method

    coverage • &. as branch coverage • Callsite coverage • Block coverage obj.foo.bar ary.map { …… }