An introduction and future of Ruby coverage library

RubyKaigi 2017 (19th Sep.)
https://rubykaigi.org/2017/presentations/mametter.html

Transcript

1. An introduction and future of Ruby coverage library RubyKaigi 2017 (19th Sep. ) Yusuke Endoh (@mametter)
2. Yusuke Endoh (@mametter) • Ruby committer (2008—) • Full-time Ruby committer (2017—) • My goal: make Ruby programs robust – Test coverage, and type system?
5. Method. Put the cream cheese into the mixing bowl. Put the sour cream into the mixing bowl. Put the white sugar into the mixing bowl. Put the yogrut into the mixing bowl. Put the unsalted butter into the mixing bowl. Combine the milk. Put the cake flour into the mixing bowl. Combine the corn starch. Put the brown sugar into the mixing bowl. Combine the egg whites. Combine the egg yolks. Put the lemon juice into the mixing bowl. Stir for 7 minutes. Liquify the contents of the mixing bowl. Pour contents of the mixing bowl into the baking dish Bake the cake mixture. Watch the cake mixture until baked. Serves 4. Cheese cake in Chef. Ingredients. 100 g cream cheese 97 g sour cream 107 g yogrut 112 g white sugar 11 g brown sugar 37 g unsalted butter 37 g cake flour 3 g corn starch 3 ml milk 3 egg yolks 3 egg whites 10 ml lemon juice 0 g cake mixture Cooking time: 80 minutes. ‘d’ ‘a’ ‘k’ ‘p’ 11*3*3=99 ‘c’ 37*3=111 ‘o’ ¥n ‘o’ Push characters ‘¥n’ ‘d’ ‘a’ ‘p’ ‘k’ ‘o’ ‘o’ ‘c’ into the stack Convert them to a string and “serve” it. data code
6. Esoteric Recipe • Polyglot of Chef and real recipe – Japanese version https://cookpad.com/ recipe/4649810 – English version https://cookpad.com/us/ recipes/3335222
7. My main contributions for Ruby • Implementation of some features: keyword arguments, deadlock detection, etc. • Release management for Ruby 1.9.2 and 2.0.0 • Optcarrot: A NES emulator for Ruby3x3 benchmark • Enhancement of the test suite of Ruby • coverage.so: the core library for coverage measurement ’06B ’07A ’07B ’08A 60 70 80 90 100 coverage (%) 70% 85% line coverage
8. Today’s theme • An introduction of test coverage • An improvement plan of Ruby’s coverage measurement feature towards 2.5
9. Survey [1/3] • Q. Do you use Ruby/RoR in production? – Raise your hand, please!
10. Survey [2/3] • Q. In those, do you test your code?
11. Survey [3/3] • Q. In those, do you use “coverage”? – Do you check the result of SimpleCov?
12. Agenda • What is coverage • How to understand and use coverage • The current status of Ruby coverage feature • The future plan of Ruby coverage feature • Conclusion
13. Agenda ☞ What is coverage • How to understand and use coverage • The current status of Ruby coverage feature • The future plan of Ruby coverage feature • Conclusion
14. What is coverage? • A measure of “goodness” of a test suite – Also called “test coverage” or “code coverage” • Allows you: – Find untested code – Decide whether your test suite is good enough or not yet • (This is arguable, but I think we can use it as an advice) • Types of coverage – Function coverage, line coverage, branch coverage, …
15. Function coverage • How many functions are executed by the tests # code def foo; …; end # ✓ def bar; …; end # ✗ def baz; …; end # ✓ # test foo baz 2/3 (67%) • Advantage • Easy to understand • Easy to visualize • Disadvantage • Too weak as a measure
16. Line coverage • How many lines are executed # code def foo(x) # ✓ if x == 0 # ✓ p :foo # ✗ else p :bar # ✓ end end # test foo(1) 3/4 (75%) Non-significant line is ignored • Advantage • Easy to understand • Easy to visualize • Disadvantage • Still weak as a measure • foo() if x == 0
17. Branch coverage • How many branches are taken true and false # code def foo(x) p :foo if x == 0 # ✓ p :bar if x < 2 # ✗ end # test foo(0) foo(1) 1/2 (50%) • Advantage • Relatively exhaustive • Disadvantage • Difficult to visualize true-case and false-case are both executed
18. Coverage types Coverage type Easy to understand/ visualize Exhaustive Function coverage ◦ ✕ Line coverage ◦ △ Branch coverage △ ◦ Currently, Ruby supports only line coverage
19. Other types of coverage • Condition coverage – How many conditions (not branches) are taken both true and false • Path coverage – How many paths are executed – Combinatorial explosion • Other advanced ones – Data-flow coverage – MC/DC coverage if a && b branch condition condition
20. Trivia • “C0/C1/C2 coverages” have difference meanings to different people – C0 coverage = line coverage – C1 coverage = branch coverage or path coverage? – C2 coverage = condition coverage or path coverage?
21. Coverage and Ruby • In Ruby, Coverage is crucial! – A test is the only way to ensure quality – Coverage is important to measure test goodness • Considering it, coverage is not used so much… – Coverage is not well-known? – It is not well-known how to use coverage? – Ruby’s coverage measurement feature is not enough? • I’d like to improve the situation with this talk
22. Agenda • What is coverage ☞ How to understand and use coverage • The current status of Ruby coverage feature • The future plan of Ruby coverage feature • Conclusion
23. What is a good test suite? • Covers the code – Coverage measures this • Also covers the design of program – Coverage does not measure this directly
24. How to understand coverage • Coverage is just a measure • Coverage is not a goal
25. If coverage XX% is required as a goal… • Developers will 1. Pick untested code that looks easiest to cover 2. Write a trivial test just for the code 3. Iterate this procedure until XX% is achieved • It will result in trivial, not-so-good test suite – It may be exhaustive for the code itself, but – It won't be exhaustive for the design
26. A good way to improve coverage • Developers should 1. Look at untested code 2. Consider what “test design” is insufficient 3. Write them – In consequence of them, the untested code is executed • It will result in good test suite – It will be exhaustive not only for the code but also for the design
27. How many % is needed/enough? • It depends upon the module being tested – Aim 100% for a significant module (e.g., injure someone) – Don't worry too much for a non-significant module • It also depends upon cost performance – For non-significant module, write a test only if it is not so hard • Again: Coverage is not a goal
28. Agenda • What is coverage • How to understand and use coverage ☞ The current status of Ruby coverage feature • The future plan of Ruby coverage feature • Conclusion
29. Ruby's coverage ecosystem • SimpleCov • coverage.so • Concov
30. SimpleCov • A wrapper library for coverage.so • Visualization with HTML • Useful features: merging, filtering, for Rails app • Author: Christoph Olszowka (@colszowka)
31. Usage of SimpleCov • Write this at the top of test/test_helper.rb • Run the test suite • coverage/index.html will be produced – Note: SimpleCov cannot measure already- loaded files before SimpleCov.start require "simplecov" SimpleCov.start
32. coverage.so • The only implementation of coverage measurement for Ruby 1.9+ • SimpleCov is a wrapper for coverage.so • Author: me
33. Basic usage # test.rb require "coverage" Coverage.start load "target.rb" p Coverage.result #=> {"target.rb"=> # [nil,nil,1,1,1,nil, # 0,nil,nil,nil,1]} Start measuring coverage Load the target file Stop measuring and get the result Coverage data
34. Coverage data # target.rb def foo(x) if x == 0 p 0 else p 1 end end foo(1) [nil, nil 1 1 0 nil 1 nil nil nil 1] nil: Non-significant line Number: Count executed Untested line!
35. Method definition is code in Ruby # target.rb def foo(x) if x == 0 p 0 else p 1 end end [nil, nil 1 0 0 nil 0 nil nil] Method definition is counted as an execution (It is not a count of method invocation!)
36. I regret the design of coverage.so • Support only line coverage • Excuse: I introduced it just for test enhancement of Ruby itself – I didn't deliberate the API for external project… • I have wanted to make the next version better ext/coverage/coverage.c: 69 /* Coverage provides coverage measurement feature for Ruby. 70 * This feature is experimental, so these APIs may be changed in future. 71 *
37. Concov • CONtinuous COVerage – Detects temporal change (degradation) of coverage • Developed for monitoring Ruby's coverage • Author: me • Presented at RubyKaigi 2009, and then… – It has not been used by everyone (including me) – It was based on Ramaze (old web framework)!
38. Concov reveals reality of Ruby dev. （Enumerable#join, 2009/07/07, M***） New feature introduced with no tests!
39. Concov reveals reality of Ruby dev. （File#size, 2009/02/25, M***）
40. Concov reveals reality of Ruby dev. （Etc::Passwd.each, 2009/02/19, N*****）
41. Concov reveals reality of Ruby dev. （Dir.home, 2009/02/03, N*****）
42. Concov reveals reality of Ruby dev. （Array#sort_by!, 2009/02/03, M***）
43. Concov reveals reality of Ruby dev. （rb_to_float, 2008/12/30, M***）
44. Coverage ecosystem for other languages • C/C++: GCOV/LCOV – Integrated with gcc: gcc -coverage target.c • Java: A lot of tools – Cobertura, Emma, Clover, JaCoCo – Integrated with CI tools and/or IDEs • JavaScript: Istanbul Jenkins Cobertura plugin LCOV result
45. Agenda • What is coverage • How to understand and use coverage • The current status of Ruby coverage feature ☞ The future plan of Ruby coverage feature • Conclusion
46. A plan towards Ruby 2.5 • Support function and branch coverage – There have been multiple requests and some PoC patches… • To make the API better, any comments are welcome – https://bugs.ruby-lang.org/issues/13901
47. API: to start measuring # compatible layer Coverage.start Coverage.result #=> {"file.rb"=>[nil, 1, 0, …], … } # new API Coverage.start(lines: true) Coverage.result #=> {"file.rb" => { :lines => [nil, 1, 0, …] } }
48. API: to start other types of coverage # enable branch and function coverage Coverage.start(lines:true, branches:true, methods:true) Coverage.result #=> {"file.rb" => { :lines => [nil, 1, 0, …], # :branches => {…}, # :methods => {…} } } # shorthand Coverage.start(:all)
49. Coverage.result for branch coverage {"target1.rb"=> {:lines=>[…], :branches=>{ [:if, 0, 2]=>{ [:then, 1, 3]=>2, [:else, 2, 5]=>1 } }, :methods=>{ [:test_if, 1]=>3 }}} # target1.rb 1: def test_if(x) 2: if x == 0 3: p "x == 0" 4: else 5: p "x != 0" 6: end 7: end 8: 9: test_if(0) 10: test_if(0) 11: test_if(1) From if at Line 2 Jumped to then clause at Line 3 twice Jumped to else clause at Line 5 once
50. Coverage.result for branch coverage {"target2.rb"=> {:lines=>[1, 1, 1, nil, nil, 1], :branches=> {[:if, 0, 2]=>{ [:then, 1, 2]=>1, [:else, 2, 2]=>0}, [:if, 3, 3]=>{ [:then, 4, 3]=>0, [:else, 5, 3]=>1}}, :methods=>{ [:test_if, 1]=>3 }}} # target2.rb 1: def test_if_oneline(x) 2: p "x == 0" if x == 0 3: p "x != 0" if x != 0 4: end 5: 6: test_if_oneline(0) Line coverage 100% Branch coverage tells you there are untested cases
51. Discussion of API design • 100% compatible • [<label>, <numbering>, <line-no>] – e.g., [:if, 0, 1], [:while, 1, 1], [:case, 2, 1] – <numbering> is a unique ID to avoid conflicts for the case where there are multiple branches in one line • LCOV-style – Other candidates: • [<label>, <line-no>, <column-no>] – How to handle TAB character • [<label>, <offset from file head>] – Looks good, but hard to implement (I'll try later)
52. Overhead of coverage measurement (just preliminary experiment) # Example 2 1: foo() 2: foo() … 99: foo() 100: foo() Benchmark w/o cov. w/ cov. Overhead Example 1 0.322 μs 6.21 μs x19.3 Example 2 1.55 μs 7.16 μs x4.61 make test-all 485 s 550 s x1.13 # Example 1 1: x = 1 2: x = 1 … 99: x = 1 100: x = 1
53. Demo • Applied the new coverage.so to Ruby • Integrated with C code coverage by GCOV and Visualized by LCOV Ruby code in stdlib make exam with gcc -coverage make exam COVERAGE=1 test- coverage .dat *.gcda gcov my script run test C code of MRI cov. data source aggregate lcov HTML
54. Jenkins Cobertura Plugin
55. Agenda • What is coverage • How to understand and use coverage • The current status of Ruby coverage feature • The future plan of Ruby coverage feature ☞ Conclusion
56. Acknowledgement • @_ko1 • @t_wada • @kazu_cocoa • @moro • @makimoto • @dev_shia • @tanaka_akr • @nalsh • @spikeolaf • @k_tsj
57. Conclusion • What is coverage, how important in Ruby, and how to understand coverage • The current status of Ruby's coverage measurement and ecosystem • A plan towards Ruby 2.5 and preliminary demo – Any comments are welcome! – https://bugs.ruby-lang.org/issues/13901
58. Future work • Determine the API • define_method as method coverage • &. as branch coverage • Callsite coverage • Block coverage obj.foo.bar ary.map { …… }