Tweet
Tweet

Slide 1

Slide 1 text

Random Testing MILHOUSE (RENAN RANELLI)

Slide 2

Slide 2 text

Outline  Introduction  Testing Is *hard*  What does it mean to *random test*  How do you even *random test* ?  Examples, examples and more examples  What's more out there

Slide 3

Slide 3 text

Software testing … is important

Slide 4

Slide 4 text

Software testing is HARD

Slide 5

Slide 5 text

Testing is hard  Testing shows the presence, not the absence of bugs – Edsger Dijkstra

Slide 6

Slide 6 text

Testing is hard  Testing shows the presence, not the absence of bugs – Edsger Dijkstra … that's why you never know when to *stop* testing

Slide 7

Slide 7 text

Testing is hard  Your tests are just as good as: Your input data Your assertions

Slide 8

Slide 8 text

Testing is hard  Your tests are just as good as: Your input data Your assertions

Slide 9

Slide 9 text

Testing is hard

Slide 10

Slide 10 text

Testing is hard

Slide 11

Slide 11 text

Testing is hard

Slide 12

Slide 12 text

What does it mean to random test ? … you choose your input data at random

Slide 13

Slide 13 text

Random testing is CHEAP

Slide 14

Slide 14 text

What does it mean to random test? [..] the technical, mathematical meaning of "random testing" refers to an explicit lack of "system" in the choice of test data, so that there is no correlation among different tests. – D. Hamlet

Slide 15

Slide 15 text

Random testing is GREAT for testing: - fault behavior, - data validation, -adherence to specs, - concurrency

Slide 16

Slide 16 text

What does it mean to random test?  Testing shows the presence, not the absence of bugs

Slide 17

Slide 17 text

What does it mean to random test?  Testing shows the presence, not the absence of bugs – Yep. But random testing does a pretty good job at showing their presence.

Slide 18

Slide 18 text

What does it mean to random test?  But random testing is not suited to be used as your primary testing source, since it lacks many of what we expect of a test suite: Reproducibility Fast to run Yadda yadda...

Slide 19

Slide 19 text

What does it mean to random test?  Instead, we are better of using it in conjunction with other tools.

Slide 20

Slide 20 text

What does it mean to random test? About the somewhat recent Heartbleed bug: Unit tests are unlikely to trigger edge cases and failed sanity checks. There is a class of tests that is known since decades that is, in my opinion, not used enough: fuzzy testing. The OpenSSL bug was definitely discoverable by sending different kind of OpenSSL packets with different randomized parameters, in conjunction with dynamic analysis tools like Valgrind. Salvatore Sanfilippo/@antirez (Author of *Redis*)

Slide 21

Slide 21 text

Testing is hard

Slide 22

Slide 22 text

What does it mean to random test?  Valgrind is an instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail [...]

Slide 23

Slide 23 text

How do you even random test ? … you choose your input data at random

Slide 24

Slide 24 text

Its an art actually.

Slide 25

Slide 25 text

How do you even random test? Any kind of testing is more-or-less like this:

Slide 26

Slide 26 text

How do you even random test?  Property based testing Is becoming quite popular due to the rise of functional programming (nerd revenge!) And also...

Slide 27

Slide 27 text

How do you even random test?

Slide 28

Slide 28 text

How do you even random test?

Slide 29

Slide 29 text

How do you even random test?

Slide 30

Slide 30 text

How do you even random test?

Slide 31

Slide 31 text

What does it mean to random test? The key problem is generating input that are part of the domain of the software under test. Random 01's is just plainly useless.

Slide 32

Slide 32 text

The one thing you need to understand: The input validity problem

Slide 33

Slide 33 text

What does it mean to random test?

Slide 34

Slide 34 text

What does it mean to random test?

Slide 35

Slide 35 text

What does it mean to random test? Most of the criticism to Random testing is about misapplication of the technique ignoring the input validity problem

Slide 36

Slide 36 text

QUICKCHECK!!!!

Slide 37

Slide 37 text

QUICKCHECK!!!!

Slide 38

Slide 38 text

How do you even random test?  There are reimplementations of Quickcheck in: C, C++, Chicken Scheme, Clojure, Common Lisp, D, Elm, Erlang, F#, Factor, Io, Java, Javascript, Node.js, Objective-C, Ocaml, Perl, Prolog, Python, R, Ruby, Rust, Scala, Scheme, Smalltalk, Standard ML and Swift.

Slide 39

Slide 39 text

DEMO TIME

Slide 40

Slide 40 text

Examples, examples, examples

Slide 41

Slide 41 text

Alright, but.... Is it worth it ? … or is it just for Haskell people?

Slide 42

Slide 42 text

Examples, examples, examples

Slide 43

Slide 43 text

Examples, examples, examples 1 3 4 2 5 6 8 7 Imagine a build tool: Waiting to execute Ready to execute Done Executing

Slide 44

Slide 44 text

Examples, examples, examples 1 3 4 2 5 6 8 7 Imagine a build tool: Waiting to execute Ready to execute Done Executing

Slide 45

Slide 45 text

Examples, examples, examples 1 3 4 2 5 6 8 7 Imagine a build tool: Waiting to execute Ready to execute Done Executing

Slide 46

Slide 46 text

Examples, examples, examples 1 3 4 2 5 6 8 7 Imagine a build tool: Waiting to execute Ready to execute Done Executing

Slide 47

Slide 47 text

Examples, examples, examples

Slide 48

Slide 48 text

Examples, examples, examples

Slide 49

Slide 49 text

Examples, examples, examples

Slide 50

Slide 50 text

Examples, examples, examples

Slide 51

Slide 51 text

Examples, examples, examples

Slide 52

Slide 52 text

Examples, examples, examples

Slide 53

Slide 53 text

Examples, examples, examples

Slide 54

Slide 54 text

Examples, examples, examples

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

Final Regards

Slide 58

Slide 58 text

Final Regards Random testing is no panacea. Understanding the input validity problem and your input data distribution is FUNDAMENTAL You need to think hard about your system in order to test it. There is no substitute to it.

Slide 59

Slide 59 text

What's more out there?

Slide 60

Slide 60 text

What's more out there?  You can't connect the dots looking forward. You can only connect them looking backwards – Steve Jobs

Slide 61

Slide 61 text

What's more out there?  You can't connect the dots looking forward. You can only connect them looking backwards – Steve Jobs

Slide 62

Slide 62 text

What's more out there?  There is a lot of literature out there about the topic, and many people doing crazy things.  The main challenge is always to devise a good test case generator. That is an art, and extremely context dependent.

Slide 63

Slide 63 text

What's more out there?

Slide 64

Slide 64 text

What's more out there?

Slide 65

Slide 65 text

What's more out there?  The bugs discovered by CSmith are/were far from trivial, and the process of diagnosing and fixing them resulted in a series of academic papers.

Slide 66

Slide 66 text

References: ● Udacity's “software testing” course: https://www.udacity.com/course/software-testing--cs258 The paper that introduces Quickcheck to the world: http://www.eecs.northwestern.edu/~robby/courses/395- 495-2009-fall/quick.pdf Salvatore Sanfilippo's blog: http://antirez.com