QuickCheck: First Dive

http://bucharestfp.ro/2014/08/27/types-in-haskell-quickcheck/

Transcript

1. QuickCheck First Dive 27.08.2014 • Dan Șerban • github.com/dserban
2. Let's start with WHY
3. Let's start with WHY Q: WHY QuickCheck? Q: WHY property-based testing?
4. A: Because testing is a very hard thing for developers to do
5. QuickCheck is 80/20
6. QuickCheck is 80/20 80% of the benefit of formal verification for 20% of the effort
7. Why is testing so hard? Think of a system with n features
8. n features 3-4 manually written units tests per feature O(n) Testing involves a linear amount of work But we all know you won't find all the bugs that way
9. Some bugs only manifest themselves when you use two distinct features in combination
10. Testing pairs of features How many tests do we need to write to test all pairs of features? Quadratically many How about those subtle bugs which only manifest themselves when 3 distinct features interact?
11. This is starting to sound like a lot of work
12. Testing triplets of features Are we really going to allow testing to take up O(n3) amount of effort?
13. And that's just for sequential code
14. We haven't even started talking about testing for intermittent race conditions
15. So how exactly does QuickCheck work?
22. There is a problem, however
23. The problem In traditional / unit testing, we write test cases by hand we specify the input exactly we can predict what the expected output should be
24. But we can't use expected results in randomized testing
25. When we generate test cases we don't know ahead of time what input will be sent to the API under test therefore we can't rely on expected results for the assert part
26. Question #1: How do we know whether an arbitrarily generated test has passed or failed?
27. Question #2: What is a general property?
28. Property: "Plus is a commutative operator." x + y == y + x
29. Property: "For each integer n, if n is even, then n+1 should be odd." \n -> even(n) ==> odd(n+1)
30. Property: "Compression should make things smaller" (size . compress) input < size input
31. Property: "Sorting a list twice in a row should give the same result as sorting only once." (sort . sort) list == sort list
32. Property: "The leftmost element in a sorted list is less than or equal to each of the first ten elements in the original list before the sort."
33. propSort myList = let headOfSortedList = (head . sort) myList in and [ headOfSortedList <= i | i <- take 10 myList ]
35. Property: "The following two implementations for the clamp function are equivalent."
36. clampA :: Int -> Int -> Int -> Int clampA lowerBound upperBound i | i > upperBound = upperBound | i < lowerBound = lowerBound | otherwise = i clampB :: Int -> Int -> Int -> Int clampB lowerBound upperBound i = (!! 1) . sort $ [lowerBound, upperBound, i]
37. Property: "The MD5 algorithm should always generate a string of length 32." $ echo -n "abc" | md5sum 900150983cd24fb0d6963f7d28e17f72 $
38. Property: "The MD5 algorithm should always generate a sufficient amount of entropy." $ echo -n "abc" | md5sum 900150983cd24fb0d6963f7d28e17f72 $
39. Question #3: How do I capture the essence of entropy so that I can write a general property for it? "Entropy" is a highly abstract concept.
40. $ echo -n "abc" | md5sum 900150983cd24fb0d6963f7d28e17f72 $ 900150983 cd 24 fb 0 d 6963 f 7 d 28 e 17 f 72 Key info here: 23 digits 9 letters
41. 23 digits and 9 letters. That's not at all balanced, therefore there is not enough entropy. Test failed.
42. Properties can be expressed for mission-critical systems too
43. "The software-powered semaphores at a simple intersection should never greenlight both roads, not even after being hit by 10 lightning strikes in a row."
44. Demo Time!
45. Follow me on GitHub github.com/dserban
46. QuickCheck pictures credit: John Hughes "Testing the Really Hard Stuff and Staying Sane"