Slide 1

Slide 1 text

An introduction and future of Ruby coverage library RubyKaigi 2017 (19th Sep. ) Yusuke Endoh (@mametter)

Slide 2

Slide 2 text

Yusuke Endoh (@mametter) • Ruby committer (2008—) • Full-time Ruby committer (2017—) • My goal: make Ruby programs robust – Test coverage, and type system?

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

Method. Put the cream cheese into the mixing bowl. Put the sour cream into the mixing bowl. Put the white sugar into the mixing bowl. Put the yogrut into the mixing bowl. Put the unsalted butter into the mixing bowl. Combine the milk. Put the cake flour into the mixing bowl. Combine the corn starch. Put the brown sugar into the mixing bowl. Combine the egg whites. Combine the egg yolks. Put the lemon juice into the mixing bowl. Stir for 7 minutes. Liquify the contents of the mixing bowl. Pour contents of the mixing bowl into the baking dish Bake the cake mixture. Watch the cake mixture until baked. Serves 4. Cheese cake in Chef. Ingredients. 100 g cream cheese 97 g sour cream 107 g yogrut 112 g white sugar 11 g brown sugar 37 g unsalted butter 37 g cake flour 3 g corn starch 3 ml milk 3 egg yolks 3 egg whites 10 ml lemon juice 0 g cake mixture Cooking time: 80 minutes. ‘d’ ‘a’ ‘k’ ‘p’ 11*3*3=99 ‘c’ 37*3=111 ‘o’ ¥n ‘o’ Push characters ‘¥n’ ‘d’ ‘a’ ‘p’ ‘k’ ‘o’ ‘o’ ‘c’ into the stack Convert them to a string and “serve” it. data code

Slide 6

Slide 6 text

Esoteric Recipe • Polyglot of Chef and real recipe – Japanese version https://cookpad.com/ recipe/4649810 – English version https://cookpad.com/us/ recipes/3335222

Slide 7

Slide 7 text

My main contributions for Ruby • Implementation of some features: keyword arguments, deadlock detection, etc. • Release management for Ruby 1.9.2 and 2.0.0 • Optcarrot: A NES emulator for Ruby3x3 benchmark • Enhancement of the test suite of Ruby • coverage.so: the core library for coverage measurement ’06B ’07A ’07B ’08A 60 70 80 90 100 coverage (%) 70% 85% line coverage

Slide 8

Slide 8 text

Today’s theme • An introduction of test coverage • An improvement plan of Ruby’s coverage measurement feature towards 2.5

Slide 9

Slide 9 text

Survey [1/3] • Q. Do you use Ruby/RoR in production? – Raise your hand, please!

Slide 10

Slide 10 text

Survey [2/3] • Q. In those, do you test your code?

Slide 11

Slide 11 text

Survey [3/3] • Q. In those, do you use “coverage”? – Do you check the result of SimpleCov?

Slide 12

Slide 12 text

Agenda • What is coverage • How to understand and use coverage • The current status of Ruby coverage feature • The future plan of Ruby coverage feature • Conclusion

Slide 13

Slide 13 text

Agenda ☞ What is coverage • How to understand and use coverage • The current status of Ruby coverage feature • The future plan of Ruby coverage feature • Conclusion

Slide 14

Slide 14 text

What is coverage? • A measure of “goodness” of a test suite – Also called “test coverage” or “code coverage” • Allows you: – Find untested code – Decide whether your test suite is good enough or not yet • (This is arguable, but I think we can use it as an advice) • Types of coverage – Function coverage, line coverage, branch coverage, …

Slide 15

Slide 15 text

Function coverage • How many functions are executed by the tests # code def foo; …; end # ✓ def bar; …; end # ✗ def baz; …; end # ✓ # test foo baz 2/3 (67%) • Advantage • Easy to understand • Easy to visualize • Disadvantage • Too weak as a measure

Slide 16

Slide 16 text

Line coverage • How many lines are executed # code def foo(x) # ✓ if x == 0 # ✓ p :foo # ✗ else p :bar # ✓ end end # test foo(1) 3/4 (75%) Non-significant line is ignored • Advantage • Easy to understand • Easy to visualize • Disadvantage • Still weak as a measure • foo() if x == 0

Slide 17

Slide 17 text

Branch coverage • How many branches are taken true and false # code def foo(x) p :foo if x == 0 # ✓ p :bar if x < 2 # ✗ end # test foo(0) foo(1) 1/2 (50%) • Advantage • Relatively exhaustive • Disadvantage • Difficult to visualize true-case and false-case are both executed

Slide 18

Slide 18 text

Coverage types Coverage type Easy to understand/ visualize Exhaustive Function coverage ○ ✕ Line coverage ○ △ Branch coverage △ ○ Currently, Ruby supports only line coverage

Slide 19

Slide 19 text

Other types of coverage • Condition coverage – How many conditions (not branches) are taken both true and false • Path coverage – How many paths are executed – Combinatorial explosion • Other advanced ones – Data-flow coverage – MC/DC coverage if a && b branch condition condition

Slide 20

Slide 20 text

Trivia • “C0/C1/C2 coverages” have difference meanings to different people – C0 coverage = line coverage – C1 coverage = branch coverage or path coverage? – C2 coverage = condition coverage or path coverage?

Slide 21

Slide 21 text

Coverage and Ruby • In Ruby, Coverage is crucial! – A test is the only way to ensure quality – Coverage is important to measure test goodness • Considering it, coverage is not used so much… – Coverage is not well-known? – It is not well-known how to use coverage? – Ruby’s coverage measurement feature is not enough? • I’d like to improve the situation with this talk

Slide 22

Slide 22 text

Agenda • What is coverage ☞ How to understand and use coverage • The current status of Ruby coverage feature • The future plan of Ruby coverage feature • Conclusion

Slide 23

Slide 23 text

What is a good test suite? • Covers the code – Coverage measures this • Also covers the design of program – Coverage does not measure this directly

Slide 24

Slide 24 text

How to understand coverage • Coverage is just a measure • Coverage is not a goal

Slide 25

Slide 25 text

If coverage XX% is required as a goal… • Developers will 1. Pick untested code that looks easiest to cover 2. Write a trivial test just for the code 3. Iterate this procedure until XX% is achieved • It will result in trivial, not-so-good test suite – It may be exhaustive for the code itself, but – It won't be exhaustive for the design

Slide 26

Slide 26 text

A good way to improve coverage • Developers should 1. Look at untested code 2. Consider what “test design” is insufficient 3. Write them – In consequence of them, the untested code is executed • It will result in good test suite – It will be exhaustive not only for the code but also for the design

Slide 27

Slide 27 text

How many % is needed/enough? • It depends upon the module being tested – Aim 100% for a significant module (e.g., injure someone) – Don't worry too much for a non-significant module • It also depends upon cost performance – For non-significant module, write a test only if it is not so hard • Again: Coverage is not a goal

Slide 28

Slide 28 text

Agenda • What is coverage • How to understand and use coverage ☞ The current status of Ruby coverage feature • The future plan of Ruby coverage feature • Conclusion

Slide 29

Slide 29 text

Ruby's coverage ecosystem • SimpleCov • coverage.so • Concov

Slide 30

Slide 30 text

SimpleCov • A wrapper library for coverage.so • Visualization with HTML • Useful features: merging, filtering, for Rails app • Author: Christoph Olszowka (@colszowka)

Slide 31

Slide 31 text

Usage of SimpleCov • Write this at the top of test/test_helper.rb • Run the test suite • coverage/index.html will be produced – Note: SimpleCov cannot measure already- loaded files before SimpleCov.start require "simplecov" SimpleCov.start

Slide 32

Slide 32 text

coverage.so • The only implementation of coverage measurement for Ruby 1.9+ • SimpleCov is a wrapper for coverage.so • Author: me

Slide 33

Slide 33 text

Basic usage # test.rb require "coverage" Coverage.start load "target.rb" p Coverage.result #=> {"target.rb"=> # [nil,nil,1,1,1,nil, # 0,nil,nil,nil,1]} Start measuring coverage Load the target file Stop measuring and get the result Coverage data

Slide 34

Slide 34 text

Coverage data # target.rb def foo(x) if x == 0 p 0 else p 1 end end foo(1) [nil, nil 1 1 0 nil 1 nil nil nil 1] nil: Non-significant line Number: Count executed Untested line!

Slide 35

Slide 35 text

Method definition is code in Ruby # target.rb def foo(x) if x == 0 p 0 else p 1 end end [nil, nil 1 0 0 nil 0 nil nil] Method definition is counted as an execution (It is not a count of method invocation!)

Slide 36

Slide 36 text

I regret the design of coverage.so • Support only line coverage • Excuse: I introduced it just for test enhancement of Ruby itself – I didn't deliberate the API for external project… • I have wanted to make the next version better ext/coverage/coverage.c: 69 /* Coverage provides coverage measurement feature for Ruby. 70 * This feature is experimental, so these APIs may be changed in future. 71 *

Slide 37

Slide 37 text

Concov • CONtinuous COVerage – Detects temporal change (degradation) of coverage • Developed for monitoring Ruby's coverage • Author: me • Presented at RubyKaigi 2009, and then… – It has not been used by everyone (including me) – It was based on Ramaze (old web framework)!

Slide 38

Slide 38 text

Concov reveals reality of Ruby dev. (Enumerable#join, 2009/07/07, M***) New feature introduced with no tests!

Slide 39

Slide 39 text

Concov reveals reality of Ruby dev. (File#size, 2009/02/25, M***)

Slide 40

Slide 40 text

Concov reveals reality of Ruby dev. (Etc::Passwd.each, 2009/02/19, N*****)

Slide 41

Slide 41 text

Concov reveals reality of Ruby dev. (Dir.home, 2009/02/03, N*****)

Slide 42

Slide 42 text

Concov reveals reality of Ruby dev. (Array#sort_by!, 2009/02/03, M***)

Slide 43

Slide 43 text

Concov reveals reality of Ruby dev. (rb_to_float, 2008/12/30, M***)

Slide 44

Slide 44 text

Coverage ecosystem for other languages • C/C++: GCOV/LCOV – Integrated with gcc: gcc -coverage target.c • Java: A lot of tools – Cobertura, Emma, Clover, JaCoCo – Integrated with CI tools and/or IDEs • JavaScript: Istanbul Jenkins Cobertura plugin LCOV result

Slide 45

Slide 45 text

Agenda • What is coverage • How to understand and use coverage • The current status of Ruby coverage feature ☞ The future plan of Ruby coverage feature • Conclusion

Slide 46

Slide 46 text

A plan towards Ruby 2.5 • Support function and branch coverage – There have been multiple requests and some PoC patches… • To make the API better, any comments are welcome – https://bugs.ruby-lang.org/issues/13901

Slide 47

Slide 47 text

API: to start measuring # compatible layer Coverage.start Coverage.result #=> {"file.rb"=>[nil, 1, 0, …], … } # new API Coverage.start(lines: true) Coverage.result #=> {"file.rb" => { :lines => [nil, 1, 0, …] } }

Slide 48

Slide 48 text

API: to start other types of coverage # enable branch and function coverage Coverage.start(lines:true, branches:true, methods:true) Coverage.result #=> {"file.rb" => { :lines => [nil, 1, 0, …], # :branches => {…}, # :methods => {…} } } # shorthand Coverage.start(:all)

Slide 49

Slide 49 text

Coverage.result for branch coverage {"target1.rb"=> {:lines=>[…], :branches=>{ [:if, 0, 2]=>{ [:then, 1, 3]=>2, [:else, 2, 5]=>1 } }, :methods=>{ [:test_if, 1]=>3 }}} # target1.rb 1: def test_if(x) 2: if x == 0 3: p "x == 0" 4: else 5: p "x != 0" 6: end 7: end 8: 9: test_if(0) 10: test_if(0) 11: test_if(1) From if at Line 2 Jumped to then clause at Line 3 twice Jumped to else clause at Line 5 once

Slide 50

Slide 50 text

Coverage.result for branch coverage {"target2.rb"=> {:lines=>[1, 1, 1, nil, nil, 1], :branches=> {[:if, 0, 2]=>{ [:then, 1, 2]=>1, [:else, 2, 2]=>0}, [:if, 3, 3]=>{ [:then, 4, 3]=>0, [:else, 5, 3]=>1}}, :methods=>{ [:test_if, 1]=>3 }}} # target2.rb 1: def test_if_oneline(x) 2: p "x == 0" if x == 0 3: p "x != 0" if x != 0 4: end 5: 6: test_if_oneline(0) Line coverage 100% Branch coverage tells you there are untested cases

Slide 51

Slide 51 text

Discussion of API design • 100% compatible • [, , ] – e.g., [:if, 0, 1], [:while, 1, 1], [:case, 2, 1] – is a unique ID to avoid conflicts for the case where there are multiple branches in one line • LCOV-style – Other candidates: • [, , ] – How to handle TAB character • [, ] – Looks good, but hard to implement (I'll try later)

Slide 52

Slide 52 text

Overhead of coverage measurement (just preliminary experiment) # Example 2 1: foo() 2: foo() … 99: foo() 100: foo() Benchmark w/o cov. w/ cov. Overhead Example 1 0.322 μs 6.21 μs x19.3 Example 2 1.55 μs 7.16 μs x4.61 make test-all 485 s 550 s x1.13 # Example 1 1: x = 1 2: x = 1 … 99: x = 1 100: x = 1

Slide 53

Slide 53 text

Demo • Applied the new coverage.so to Ruby • Integrated with C code coverage by GCOV and Visualized by LCOV Ruby code in stdlib make exam with gcc -coverage make exam COVERAGE=1 test- coverage .dat *.gcda gcov my script run test C code of MRI cov. data source aggregate lcov HTML

Slide 54

Slide 54 text

Jenkins Cobertura Plugin

Slide 55

Slide 55 text

Agenda • What is coverage • How to understand and use coverage • The current status of Ruby coverage feature • The future plan of Ruby coverage feature ☞ Conclusion

Slide 56

Slide 56 text

Acknowledgement • @_ko1 • @t_wada • @kazu_cocoa • @moro • @makimoto • @dev_shia • @tanaka_akr • @nalsh • @spikeolaf • @k_tsj

Slide 57

Slide 57 text

Conclusion • What is coverage, how important in Ruby, and how to understand coverage • The current status of Ruby's coverage measurement and ecosystem • A plan towards Ruby 2.5 and preliminary demo – Any comments are welcome! – https://bugs.ruby-lang.org/issues/13901

Slide 58

Slide 58 text

Future work • Determine the API • define_method as method coverage • &. as branch coverage • Callsite coverage • Block coverage obj.foo.bar ary.map { …… }