Locaweb TechTalks 2015 - Random Testing: What it is and why it matters

Locaweb TechTalks 2015 - Random Testing: What it is and why it matters

4569aec00cb223b3fbf484f9e7ba1256?s=128

Renan Ranelli

September 10, 2015
Tweet

Transcript

  1. 2.

    Outline  Introduction  Testing Is *hard*  What does

    it mean to *random test*  How do you even *random test* ?  Examples, examples and more examples  What's more out there
  2. 5.

    Testing is hard  Testing shows the presence, not the

    absence of bugs – Edsger Dijkstra
  3. 6.

    Testing is hard  Testing shows the presence, not the

    absence of bugs – Edsger Dijkstra … that's why you never know when to *stop* testing
  4. 7.

    Testing is hard  Your tests are just as good

    as: Your input data Your assertions
  5. 8.

    Testing is hard  Your tests are just as good

    as: Your input data Your assertions
  6. 12.

    What does it mean to random test ? … you

    choose your input data at random
  7. 14.

    What does it mean to random test? [..] the technical,

    mathematical meaning of "random testing" refers to an explicit lack of "system" in the choice of test data, so that there is no correlation among different tests. – D. Hamlet
  8. 15.

    Random testing is GREAT for testing: - fault behavior, -

    data validation, -adherence to specs, - concurrency
  9. 16.

    What does it mean to random test?  Testing shows

    the presence, not the absence of bugs
  10. 17.

    What does it mean to random test?  Testing shows

    the presence, not the absence of bugs – Yep. But random testing does a pretty good job at showing their presence.
  11. 18.

    What does it mean to random test?  But random

    testing is not suited to be used as your primary testing source, since it lacks many of what we expect of a test suite: Reproducibility Fast to run Yadda yadda...
  12. 19.

    What does it mean to random test?  Instead, we

    are better of using it in conjunction with other tools.
  13. 20.

    What does it mean to random test? About the somewhat

    recent Heartbleed bug: Unit tests are unlikely to trigger edge cases and failed sanity checks. There is a class of tests that is known since decades that is, in my opinion, not used enough: fuzzy testing. The OpenSSL bug was definitely discoverable by sending different kind of OpenSSL packets with different randomized parameters, in conjunction with dynamic analysis tools like Valgrind. Salvatore Sanfilippo/@antirez (Author of *Redis*)
  14. 22.

    What does it mean to random test?  Valgrind is

    an instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail [...]
  15. 23.
  16. 26.

    How do you even random test?  Property based testing

    Is becoming quite popular due to the rise of functional programming (nerd revenge!) And also...
  17. 31.

    What does it mean to random test? The key problem

    is generating input that are part of the domain of the software under test. Random 01's is just plainly useless.
  18. 35.

    What does it mean to random test? Most of the

    criticism to Random testing is about misapplication of the technique ignoring the input validity problem
  19. 38.

    How do you even random test?  There are reimplementations

    of Quickcheck in: C, C++, Chicken Scheme, Clojure, Common Lisp, D, Elm, Erlang, F#, Factor, Io, Java, Javascript, Node.js, Objective-C, Ocaml, Perl, Prolog, Python, R, Ruby, Rust, Scala, Scheme, Smalltalk, Standard ML and Swift.
  20. 39.
  21. 41.

    Alright, but.... Is it worth it ? … or is

    it just for Haskell people?
  22. 43.

    Examples, examples, examples 1 3 4 2 5 6 8

    7 Imagine a build tool: Waiting to execute Ready to execute Done Executing
  23. 44.

    Examples, examples, examples 1 3 4 2 5 6 8

    7 Imagine a build tool: Waiting to execute Ready to execute Done Executing
  24. 45.

    Examples, examples, examples 1 3 4 2 5 6 8

    7 Imagine a build tool: Waiting to execute Ready to execute Done Executing
  25. 46.

    Examples, examples, examples 1 3 4 2 5 6 8

    7 Imagine a build tool: Waiting to execute Ready to execute Done Executing
  26. 55.
  27. 56.
  28. 58.

    Final Regards Random testing is no panacea. Understanding the input

    validity problem and your input data distribution is FUNDAMENTAL You need to think hard about your system in order to test it. There is no substitute to it.
  29. 60.

    What's more out there?  You can't connect the dots

    looking forward. You can only connect them looking backwards – Steve Jobs
  30. 61.

    What's more out there?  You can't connect the dots

    looking forward. You can only connect them looking backwards – Steve Jobs
  31. 62.

    What's more out there?  There is a lot of

    literature out there about the topic, and many people doing crazy things.  The main challenge is always to devise a good test case generator. That is an art, and extremely context dependent.
  32. 65.

    What's more out there?  The bugs discovered by CSmith

    are/were far from trivial, and the process of diagnosing and fixing them resulted in a series of academic papers.
  33. 66.

    References: • Udacity's “software testing” course: https://www.udacity.com/course/software-testing--cs258 The paper that

    introduces Quickcheck to the world: http://www.eecs.northwestern.edu/~robby/courses/395- 495-2009-fall/quick.pdf Salvatore Sanfilippo's blog: http://antirez.com