Intro to QuickCheck + Quviq EQC

Slides from the QuickCheck unsession at StrangeLoop 2013

Transcript

1. Introduction Quviq Erlang QuickCheck Intro to QuickCheck + Quviq EQC Sean Cribbs 2013-09-18 Wed
2. Introduction Quviq Erlang QuickCheck Outline 1 Introduction 2 Quviq Erlang QuickCheck
3. Introduction Quviq Erlang QuickCheck Problems with xUnit/BDD/etc Hand-coded test cases are tedious and brittle. When have you written enough of them? How do you know?
4. Introduction Quviq Erlang QuickCheck What is QuickCheck? A testing tool, but not like xUnit Write abstract properties, not specific examples Generates data to test over, with increasing entropy Shrinks failing cases to the smallest similar example
5. Introduction Quviq Erlang QuickCheck Sample properties Reversing a list twice should equal the original list. Reversing and sorting a list should preserve its length. Popping an element from a queue should reduce its size by one.
6. Introduction Quviq Erlang QuickCheck Sample property prop_reverse() -> ?FORALL(X, %% Binding list(int()), %% Generator %% Property lists:reverse(lists:reverse(X)) == X ).
7. Introduction Quviq Erlang QuickCheck Thinking in properties A “roundtrip”, e.g encode/decode? An existing implementation with similar behavior A relationship between inputs/outputs?
8. Introduction Quviq Erlang QuickCheck General advice Separate pure and impure code Write small functions Write functions that manipulate data, side-effect free
9. Introduction Quviq Erlang QuickCheck Quviq Erlang Quickcheck Caveat Quviq QuickCheck (“eqc”) is closed-source, licensed. On the other hand, it is probably the most advanced QuickCheck.
10. Introduction Quviq Erlang QuickCheck Other Erlang QuickChecks Triq Trifork, Apache 2.0 license PropEr Uppsala University, GPLv3 license
11. Introduction Quviq Erlang QuickCheck Building blocks ?FORALL macro wrapping the property eqc:quickcheck(Property) Generators for simple types
12. Introduction Quviq Erlang QuickCheck Setup -include_lib("eqc/include/eqc.hrl"). Imports the macros and functions from the eqc module and generators (eqc_gen)
13. Introduction Quviq Erlang QuickCheck Executing a property 3> eqc:quickcheck(eqc1:prop_reverse()). ............................................................ ............................. OK, passed 100 tests true
14. Introduction Quviq Erlang QuickCheck Run more examples 5> eqc:quickcheck(eqc:numtests(250, eqc1:prop_reverse())). ............................................................ ............................................................ ............................................................ ..................................... OK, passed 250 tests true
15. Introduction Quviq Erlang QuickCheck Failing test prop_delete() -> ?FORALL(Ls, non_empty(list(int())), ?FORALL(X, elements(Ls), not lists:member(X, lists:delete(X, Ls)))). 10> eqc:quickcheck(eqc1:prop_delete()). ....................................................Failed! [-15,-12,1,1,9,17] 1 Shrinking...(3 times) [1,1] 1 false
16. Introduction Quviq Erlang QuickCheck Generating inputs Simple: bool int, largeint, nat choose char real binary, bitstring function0, function1, function2, function3, function4 Compositions: list, orderedlist, shuffle default elements, oneof, frequency non_empty
17. Introduction Quviq Erlang QuickCheck Custom generators Static terms generate themselves Compound terms can contain generators Modifying a value before usage: ?LET(X, list(int()), lists:usort(X)) Discarding/filtering generated values: ?SUCHTHAT(X, int(), X > 10) Lazy generators: oneof([ int(), ?LAZY(gen_foo()) ]). Binding the generation size into the generator: gen_queue() -> ?SIZED(N, gen_queue(N)). gen_queue(0) -> queue:new(); gen_queue(N) -> queue:in(N, gen_queue(N-1)).
18. Introduction Quviq Erlang QuickCheck Sampling generators 13> eqc_gen:sample(eqc_gen:int()). 3 11 -12 -5 3 14 1 6 -13 -19 20
19. Introduction Quviq Erlang QuickCheck Sampling generators (2) 14> eqc_gen:sample(eqc1:gen_queue()). {[10,9,8,7,6,5,4,3,2],[1]} {[11,10,9,8,7,6,5,4,3,2],[1]} {[12,11,10,9,8,7,6,5,4,3,2],[1]} {[13,12,11,10,9,8,7,6,5,4,3,2],[1]} {[14,13,12,11,10,9,8,7,6,5,4,3,2],[1]} {[15,14,13,12,11,10,9,8,7,6,5,4,3,2],[1]} {[16,15,14,13,12,11,10,9,8,7,6,5,4,3,2],[1]} {[17,16,15,14,13,12,11,10,9,8,7,6,5,4,3,2],[1]} {[18,17,16,15,14,13,12,11,10,9,8,7,6,5,4,3,2],[1]} {[19,18,17,16,15,14,13,12,11,10,9,8,7,6,5,4,3,2],[1]} {[20,19,18,17,16,15,14,13,12,11,10,9,8,7,6,5,4,3,2],[1]}
20. Introduction Quviq Erlang QuickCheck Sample shrinking 15> eqc_gen:sampleshrink(eqc_gen:list(eqc_gen:int())). [-5,-8] --> [[],[-5],[-8],[0,-8],[2,-8],[4,-8],[5,-8],[-1,-8],[-3,-8],[-
21. Introduction Quviq Erlang QuickCheck Shrinking
22. Introduction Quviq Erlang QuickCheck Other cool features Stateful tests: eqc_statem & eqc_fsm Generate a sequence of commands Compute an abstract model Run the commands and compare results Driving other systems eqc_c: automagic binding of C code eqc_socket: send commands over sockets Testing parsers: eqc_grammar Create token generators Test roundtrips: AST -> pretty print -> lex/parse -> AST
23. Introduction Quviq Erlang QuickCheck PULSE Automates random interleavings of: message sending/delivery process birth/death Finds race conditions automatically! Reorderings of messages Receiver died while sender waits Timeouts, DOWN, EXIT arrive sooner/later