Locaweb TechTalks 2015 - Random Testing: What it is and why it matters

Transcript

1. Random Testing MILHOUSE (RENAN RANELLI)
2. Outline  Introduction  Testing Is *hard*  What does it mean to *random test*  How do you even *random test* ?  Examples, examples and more examples  What's more out there
3. Software testing … is important
4. Software testing is HARD
5. Testing is hard  Testing shows the presence, not the absence of bugs – Edsger Dijkstra
6. Testing is hard  Testing shows the presence, not the absence of bugs – Edsger Dijkstra … that's why you never know when to *stop* testing
7. Testing is hard  Your tests are just as good as: Your input data Your assertions
8. Testing is hard  Your tests are just as good as: Your input data Your assertions
9. Testing is hard
10. Testing is hard
11. Testing is hard
12. What does it mean to random test ? … you choose your input data at random
13. Random testing is CHEAP
14. What does it mean to random test? [..] the technical, mathematical meaning of "random testing" refers to an explicit lack of "system" in the choice of test data, so that there is no correlation among different tests. – D. Hamlet
15. Random testing is GREAT for testing: - fault behavior, - data validation, -adherence to specs, - concurrency
16. What does it mean to random test?  Testing shows the presence, not the absence of bugs
17. What does it mean to random test?  Testing shows the presence, not the absence of bugs – Yep. But random testing does a pretty good job at showing their presence.
18. What does it mean to random test?  But random testing is not suited to be used as your primary testing source, since it lacks many of what we expect of a test suite: Reproducibility Fast to run Yadda yadda...
19. What does it mean to random test?  Instead, we are better of using it in conjunction with other tools.
20. What does it mean to random test? About the somewhat recent Heartbleed bug: Unit tests are unlikely to trigger edge cases and failed sanity checks. There is a class of tests that is known since decades that is, in my opinion, not used enough: fuzzy testing. The OpenSSL bug was definitely discoverable by sending different kind of OpenSSL packets with different randomized parameters, in conjunction with dynamic analysis tools like Valgrind. Salvatore Sanfilippo/@antirez (Author of *Redis*)
21. Testing is hard
22. What does it mean to random test?  Valgrind is an instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail [...]
23. How do you even random test ? … you choose your input data at random
24. Its an art actually.
25. How do you even random test? Any kind of testing is more-or-less like this:
26. How do you even random test?  Property based testing Is becoming quite popular due to the rise of functional programming (nerd revenge!) And also...
27. How do you even random test?
28. How do you even random test?
29. How do you even random test?
30. How do you even random test?
31. What does it mean to random test? The key problem is generating input that are part of the domain of the software under test. Random 01's is just plainly useless.
32. The one thing you need to understand: The input validity problem
33. What does it mean to random test?
34. What does it mean to random test?
35. What does it mean to random test? Most of the criticism to Random testing is about misapplication of the technique ignoring the input validity problem
36. QUICKCHECK!!!!
37. QUICKCHECK!!!!
38. How do you even random test?  There are reimplementations of Quickcheck in: C, C++, Chicken Scheme, Clojure, Common Lisp, D, Elm, Erlang, F#, Factor, Io, Java, Javascript, Node.js, Objective-C, Ocaml, Perl, Prolog, Python, R, Ruby, Rust, Scala, Scheme, Smalltalk, Standard ML and Swift.
39. DEMO TIME
40. Examples, examples, examples
41. Alright, but.... Is it worth it ? … or is it just for Haskell people?
42. Examples, examples, examples
43. Examples, examples, examples 1 3 4 2 5 6 8 7 Imagine a build tool: Waiting to execute Ready to execute Done Executing
44. Examples, examples, examples 1 3 4 2 5 6 8 7 Imagine a build tool: Waiting to execute Ready to execute Done Executing
45. Examples, examples, examples 1 3 4 2 5 6 8 7 Imagine a build tool: Waiting to execute Ready to execute Done Executing
46. Examples, examples, examples 1 3 4 2 5 6 8 7 Imagine a build tool: Waiting to execute Ready to execute Done Executing
47. Examples, examples, examples
48. Examples, examples, examples
49. Examples, examples, examples
50. Examples, examples, examples
51. Examples, examples, examples
52. Examples, examples, examples
53. Examples, examples, examples
54. Examples, examples, examples
57. Final Regards
58. Final Regards Random testing is no panacea. Understanding the input validity problem and your input data distribution is FUNDAMENTAL You need to think hard about your system in order to test it. There is no substitute to it.
59. What's more out there?
60. What's more out there?  You can't connect the dots looking forward. You can only connect them looking backwards – Steve Jobs
61. What's more out there?  You can't connect the dots looking forward. You can only connect them looking backwards – Steve Jobs
62. What's more out there?  There is a lot of literature out there about the topic, and many people doing crazy things.  The main challenge is always to devise a good test case generator. That is an art, and extremely context dependent.
63. What's more out there?
64. What's more out there?
65. What's more out there?  The bugs discovered by CSmith are/were far from trivial, and the process of diagnosing and fixing them resulted in a series of academic papers.
66. References: • Udacity's “software testing” course: https://www.udacity.com/course/software-testing--cs258 The paper that introduces Quickcheck to the world: http://www.eecs.northwestern.edu/~robby/courses/395- 495-2009-fall/quick.pdf Salvatore Sanfilippo's blog: http://antirez.com