Effort vs. Reward
Tests
Types
Simple Theorems
Low effort per theorem,
but there are lots of
simple theorems!
Covers a few states
Very low effort,
especially if types
inferred
Covers all states
Complex Theorems
Medium effort
Covers a few,
important states
High effort
Covers all states
I have a slightly different definition of test coverage than most. I don’t care which lines of code are executed nearly as much as I care about covering the
possible states of the system. Covering dead code is pointless. (click)
For simple theorems, like ‘are the arguments to this method all non-null,’ many people don’t bother testing, because there are so many, which is why we
have so many null reference errors. Inferred types in a rich type system cover this for free. (click) OTOH, total state space coverage for complex theorems
(“I am following SSL protocol to the letter”) with types can be difficult, although the effort pays off when needed. Tests might be “good enough.” Types
always cover all states, but complexity starts very low and extends to very high. Tests are in middle on coverage, effort, and complexity.