Locaweb TechTalks 2015 - Random Testing: What it is and why it matters

Locaweb TechTalks 2015 - Random Testing: What it is and why it matters

4569aec00cb223b3fbf484f9e7ba1256?s=128

Renan Ranelli

September 10, 2015
Tweet

Transcript

  1. Random Testing MILHOUSE (RENAN RANELLI)

  2. Outline  Introduction  Testing Is *hard*  What does

    it mean to *random test*  How do you even *random test* ?  Examples, examples and more examples  What's more out there
  3. Software testing … is important

  4. Software testing is HARD

  5. Testing is hard  Testing shows the presence, not the

    absence of bugs – Edsger Dijkstra
  6. Testing is hard  Testing shows the presence, not the

    absence of bugs – Edsger Dijkstra … that's why you never know when to *stop* testing
  7. Testing is hard  Your tests are just as good

    as: Your input data Your assertions
  8. Testing is hard  Your tests are just as good

    as: Your input data Your assertions
  9. Testing is hard

  10. Testing is hard

  11. Testing is hard

  12. What does it mean to random test ? … you

    choose your input data at random
  13. Random testing is CHEAP

  14. What does it mean to random test? [..] the technical,

    mathematical meaning of "random testing" refers to an explicit lack of "system" in the choice of test data, so that there is no correlation among different tests. – D. Hamlet
  15. Random testing is GREAT for testing: - fault behavior, -

    data validation, -adherence to specs, - concurrency
  16. What does it mean to random test?  Testing shows

    the presence, not the absence of bugs
  17. What does it mean to random test?  Testing shows

    the presence, not the absence of bugs – Yep. But random testing does a pretty good job at showing their presence.
  18. What does it mean to random test?  But random

    testing is not suited to be used as your primary testing source, since it lacks many of what we expect of a test suite: Reproducibility Fast to run Yadda yadda...
  19. What does it mean to random test?  Instead, we

    are better of using it in conjunction with other tools.
  20. What does it mean to random test? About the somewhat

    recent Heartbleed bug: Unit tests are unlikely to trigger edge cases and failed sanity checks. There is a class of tests that is known since decades that is, in my opinion, not used enough: fuzzy testing. The OpenSSL bug was definitely discoverable by sending different kind of OpenSSL packets with different randomized parameters, in conjunction with dynamic analysis tools like Valgrind. Salvatore Sanfilippo/@antirez (Author of *Redis*)
  21. Testing is hard

  22. What does it mean to random test?  Valgrind is

    an instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail [...]
  23. How do you even random test ? … you choose

    your input data at random
  24. Its an art actually.

  25. How do you even random test? Any kind of testing

    is more-or-less like this:
  26. How do you even random test?  Property based testing

    Is becoming quite popular due to the rise of functional programming (nerd revenge!) And also...
  27. How do you even random test?

  28. How do you even random test?

  29. How do you even random test?

  30. How do you even random test?

  31. What does it mean to random test? The key problem

    is generating input that are part of the domain of the software under test. Random 01's is just plainly useless.
  32. The one thing you need to understand: The input validity

    problem
  33. What does it mean to random test?

  34. What does it mean to random test?

  35. What does it mean to random test? Most of the

    criticism to Random testing is about misapplication of the technique ignoring the input validity problem
  36. QUICKCHECK!!!!

  37. QUICKCHECK!!!!

  38. How do you even random test?  There are reimplementations

    of Quickcheck in: C, C++, Chicken Scheme, Clojure, Common Lisp, D, Elm, Erlang, F#, Factor, Io, Java, Javascript, Node.js, Objective-C, Ocaml, Perl, Prolog, Python, R, Ruby, Rust, Scala, Scheme, Smalltalk, Standard ML and Swift.
  39. DEMO TIME

  40. Examples, examples, examples

  41. Alright, but.... Is it worth it ? … or is

    it just for Haskell people?
  42. Examples, examples, examples

  43. Examples, examples, examples 1 3 4 2 5 6 8

    7 Imagine a build tool: Waiting to execute Ready to execute Done Executing
  44. Examples, examples, examples 1 3 4 2 5 6 8

    7 Imagine a build tool: Waiting to execute Ready to execute Done Executing
  45. Examples, examples, examples 1 3 4 2 5 6 8

    7 Imagine a build tool: Waiting to execute Ready to execute Done Executing
  46. Examples, examples, examples 1 3 4 2 5 6 8

    7 Imagine a build tool: Waiting to execute Ready to execute Done Executing
  47. Examples, examples, examples

  48. Examples, examples, examples

  49. Examples, examples, examples

  50. Examples, examples, examples

  51. Examples, examples, examples

  52. Examples, examples, examples

  53. Examples, examples, examples

  54. Examples, examples, examples

  55. None
  56. None
  57. Final Regards

  58. Final Regards Random testing is no panacea. Understanding the input

    validity problem and your input data distribution is FUNDAMENTAL You need to think hard about your system in order to test it. There is no substitute to it.
  59. What's more out there?

  60. What's more out there?  You can't connect the dots

    looking forward. You can only connect them looking backwards – Steve Jobs
  61. What's more out there?  You can't connect the dots

    looking forward. You can only connect them looking backwards – Steve Jobs
  62. What's more out there?  There is a lot of

    literature out there about the topic, and many people doing crazy things.  The main challenge is always to devise a good test case generator. That is an art, and extremely context dependent.
  63. What's more out there?

  64. What's more out there?

  65. What's more out there?  The bugs discovered by CSmith

    are/were far from trivial, and the process of diagnosing and fixing them resulted in a series of academic papers.
  66. References: • Udacity's “software testing” course: https://www.udacity.com/course/software-testing--cs258 The paper that

    introduces Quickcheck to the world: http://www.eecs.northwestern.edu/~robby/courses/395- 495-2009-fall/quick.pdf Salvatore Sanfilippo's blog: http://antirez.com