Software Testing Techniques: Checking & Exploring

Slide 1

Slide 1 text

Hello, I’m Matt Obee I’m a Senior Software Tester  at Holiday Extras

Slide 2

Slide 2 text

Checking & Exploring: Testing in Agile projects

Slide 3

Slide 3 text

Testing in the past Exploratory Testing Scripted Testing

Slide 4

Slide 4 text

Testing Today Scripted Checking Testing

Slide 5

Slide 5 text

“Exploratory testing is not a testing technique. It's a way of thinking about testing. Any testing technique can be used in an exploratory way.”   Cem Kaner & James Bach (2004)

Slide 6

Slide 6 text

No freedom More freedom  Less formal Scale of Testing Freedom & Formality Less freedom  More formal CHECKING EXPLORING Test Cases Test Data Scenarios Checklist Mind Map Charter Tour Play Scripted Checking

Slide 7

Slide 7 text

Does it do everything that it’s supposed to do? ■ Asks closed (yes/no) questions; ■ Validation and verification; ■ Needs specification; ■ Often automated; ■ Developers do a lot of checking. Are there any risks or opportunities? ■ Asks open ended questions; ■ Does not need specification; ■ Results of the previous test inform the next; ■ Used to evaluate subjective qualities (e.g. usability and aesthetics); ■ Rarely automated; CHECKING EXPLORING

Slide 8

Slide 8 text

Tested = Checked + Explored   “You’re not done testing until you’ve checked that the software meets expectations and you’ve explored whether there are additional risks. A comprehensive test strategy incorporates both approaches.” Elisabeth Hendrickson (2013)

Slide 9

Slide 9 text

■ Complex multi-page forms ■ Complex processing systems ■ Highly-regulated industry and bureaucratic client ■ Accuracy, consistency and compliance ■ Overall project risk is high

Slide 10

Slide 10 text

No freedom More freedom  Less formal Less freedom  More formal Test Cases Test Data Checklist Scripted Checking

Slide 11

Slide 11 text

■ Not highly regulated ■ User experience is the main quality metric ■ Quality is subjective and not easy to measure ■ Relatively low risk but lots of opportunities

Slide 12

Slide 12 text

No freedom More freedom  Less formal Less freedom  More formal Test Data Scenarios Checklist Play Scripted Checking Personas

Slide 13

Slide 13 text

Recap: Checking & Exploring ■ Exploratory testing isn’t a technique; ■ All testing sits on the scale of freedom and formality; ■ Less formality means more freedom to explore; ■ Different projects need different levels of formality; ■ Tested = Checked + Explored

Slide 14

Slide 14 text

Checking

Slide 15

Slide 15 text

“Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.”   James Bach & Michael Bolton (2013)

Slide 16

Slide 16 text

Checking is boring. Make computers do it.

Slide 17

Slide 17 text

“Automate 100% of the tests that should be automated.” Alan Page (2013) Automate all the things!!1!

Slide 18

Slide 18 text

Computers are good at... ■ maths ■ simulation ■ generating data ■ validating against rules ■ doing repetitive tasks ■ multitasking ■ doing things quickly Computers suck at... ■ recognising patterns ■ learning and reacting ■ ignoring false positives ■ being random ■ curiosity ■ creativity ■ empathy ■ understanding context ■ common sense

Slide 19

Slide 19 text

Checking standards conformance with  linting and validation

Slide 20

Slide 20 text

Checking code logic  with unit tests

Slide 21

Slide 21 text

Checking that nothing has broken with  continuous integration  and regression tests

Slide 22

Slide 22 text

Checking for risks  and opportunities   in code review

Slide 23

Slide 23 text

Automation doesn’t replace human testers. Automation does the boring stuff so that humans have more time to do intelligent exploration.

Slide 24

Slide 24 text

Recap: Checking ■ Checks need explicit rules and success criteria; ■ Make computers do the boring stuff; ■ Don’t fear automation. It’s doesn’t replace you, it helps you do more of the testing that you enjoy and less of the stuff that you don’t;

Slide 25

Slide 25 text

Exploring

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

Checklists aren’t just for checks

Slide 33

Slide 33 text

istock/Aviation Week

Slide 34

Slide 34 text

Tours can change your perspective

Slide 35

Slide 35 text

50% of people don’t see the invisible gorilla. The same inattentional blindness happens when testing software. bit.ly/1jGOCoQ

Slide 36

Slide 36 text

The Guidebook Tour Staying on the main tourist trails Guidebook action by Jaymis Loveday

Slide 37

Slide 37 text

The Guidebook Tour ■ Do what the guidebook (instructions, manual) recommends; ■ Visit the top tourist destinations (the most popular features); ■ Don’t leave the main tourist trails (avoid advanced features); ■ Be lazy (don’t explore too much)

Slide 38

Slide 38 text

The Obsessive Compulsive Tour Doing the same thing over and over...

Slide 39

Slide 39 text

The Obsessive Compulsive Tour ■ Repeat actions ■ Click buttons more than once ■ Go back and forth between the same pages ■ Keep refreshing the page ■ Type in a field, delete it, type in it again ■ Make a selection, clear it, make the selection again ■ Make the same booking twice

Slide 40

Slide 40 text

The Backstreet Tour Going off the beaten track

Slide 41

Slide 41 text

The Backstreet Tour ■ Use the features that are rarely used ■ Take the routes that are rarely taken ■ Read the content that customers don’t normally read ■ Buy the products that fewer customers buy ■ Select the options that customers rarely select

Slide 42

Slide 42 text

The Bad Neighbourhood Tour Venturing into the rougher parts of town Light at the end of the Ghetto by Jenn Durfey

Slide 43

Slide 43 text

The Bad Neighbourhood Tour ■ Concentrate on areas that are known to be buggy (bugs cluster) ■ Explore nearby or related features ■ Do things that the application is known to struggle with

Slide 44

Slide 44 text

The Nervous Traveller Tour Being careful and scrutinise every detail Issica demonstrates the brace position by Karl Baron

Slide 45

Slide 45 text

The Nervous Traveller Tour ■ Look at everything with fresh eyes ■ Do things slowly and pay attention to the details ■ Read the microcopy (labels, instructions, error messages etc.) ■ Ask lots of questions and look for the answers ■ Be indecisive and change your mind ■ What do our customers worry about?

Slide 46

Slide 46 text

The Antisocial Tour Breaking the rules Bad Parking by xlibber

Slide 47

Slide 47 text

■ Leave mandatory form fields empty ■ Do things in the wrong order ■ Do the same things more than once ■ Start things but don’t finish them ■ Be impatient ■ Put JavaScript or SQL queries in form fields ■ Try to access member features as a non-member The Antisocial Tour

Slide 48

Slide 48 text

Variables Experimenting with   different conditions

Slide 49

Slide 49 text

A software tester walks into a bar. Orders a beer. Orders 9999 beers. Orders 1.5 beers. Orders -1 beers.   Orders 0 beers. Orders a fdsfdsa.   Tries to leave without paying.

Slide 50

Slide 50 text

How would you test this?

Slide 51

Slide 51 text

A few pen testing variables ■ Left and right handed users: try writing in either hand ■ Writing speed: try writing quickly and slowly ■ Writing pressure: try writing lightly and heavily ■ Surface: try writing on matte paper, glossy paper, cardboard, plastic etc. ■ Orientation: try writing on a table, on a wall, upside down etc. ■ Environment: trying writing in hot conditions, cold conditions, wind, rain etc. ■ Length of use: try writing a short note compared to a long essay ■ State: what happens when changing from one state to another?

Slide 52

Slide 52 text

Variables   in software

Slide 53

Slide 53 text

Odd numbers : Even numbers by Graham Horn Vary numbers

Slide 54

Slide 54 text

Vary numbers ■ Can you enter a negative number, e.g. “-1”? ■ Can you enter a decimal number, e.g. “1.5”? ■ Can you enter zero (e.g. “0”)? ■ Can you exceed the minimum or maximum limits? ■ What happens where there are zero, one or many things? ■ Goldilocks (too many, too few, just right) ■ What happens if you enter letters instead of numbers?

Slide 55

Slide 55 text

USS Yorktown

Slide 56

Slide 56 text

Amazon

Slide 57

Slide 57 text

Vary time

Slide 58

Slide 58 text

Vary time ■ Can you specify a date of birth in the future? ■ Can you schedule an event to happen in the past? ■ In a date range, can you set the ending earlier than the beginning? ■ What happens on leap days? ■ What happens on dates that don’t exist (e.g. 31st September)? ■ What happens if you reset time on the device forward and back? ■ What happens if you move between timezones?

Slide 59

Slide 59 text

Vary the format

Slide 60

Slide 60 text

Vary the format ■ What happens if you enter dates in different formats?   (e.g. 01/06/2015, 06/01/2015, 1/6/15, 1-6-15, 2015-06-01) ■ What happens if there’s a space in the number?   (e.g. “01303 815 300”) ■ What happens if a number includes special characters?   (e.g. “+44 (0)1303 815 300”) ■ Can you upload different file formats? (e.g. .gif, .jpg, .doc, .docx) ■ What happens if you enter numbers with different units of measurement?

Slide 61

Slide 61 text

Mars Climate Orbiter

Slide 62

Slide 62 text

Vary the journey Photo by Carsten Tolkmit

Slide 63

Slide 63 text

Vary the journey ■ Can you use the browser back and forward buttons? ■ What happens if you refresh the page? ■ What happens if you arrive via a bookmark? ■ Can you hack the URL? ■ Can you do things in a different order? ■ What happens if you have multiple tabs/windows running?

Slide 64

Slide 64 text

Vary the method of input Photo by utterlywrong

Slide 65

Slide 65 text

■ Can you use it without a mouse (i.e. keyboard only)? ■ What happens if you copy and paste instead of typing into fields? ■ Can you drag and drop content into fields? Vary the method of input

Slide 66

Slide 66 text

Vary the speed of input Photo by Nevit Dilmen

Slide 67

Slide 67 text

■ What happens if you interact with the application too quickly? ■ What happens if you interact with the application too slowly? ■ What happens if you don’t do anything for a length of time? Vary the speed of input

Slide 68

Slide 68 text

Therac-25

Slide 69

Slide 69 text

Vary Settings & States Control Panel by Les Chatfield

Slide 70

Slide 70 text

Vary Settings & States ■ What happens if JavaScript is disabled? ■ What happens if cookies are blocked? ■ What happens if you change the size of the browser window? ■ What happens if you change the browser font size? ■ Compare behaviour when logged in and logged out ■ What happens when the state changes? (e.g. a logged in user logging out, a logged out user logging in)

Slide 71

Slide 71 text

Testing Heuristics Cheat Sheet (PDF) bit.ly/1or9Inn

Slide 72

Slide 72 text

Recap: Exploring ■ Checklists are one great way to structure exploration; ■ Take tours to change your perspective; ■ Variables are key. Identify the variables and see what happens when you change them.

Slide 73

Slide 73 text

Takeaways ■ Tested = Checked + Explored  Check to make sure it’s doing what it should; explore to see if there are new risks or opportunities. ■ Make stupid computers do the boring checks so that intelligent humans have more time to explore. ■ Use checklists, take tours and experiment with variables to get everyone involved in testing sessions.

Slide 74

Slide 74 text

Thank you @mattobee