Software Testing Techniques: Checking & Exploring

In this presentation, we’ll think about how the role of software tester has evolved to serve the needs of modern Agile teams. We’ll look at how all testing activities sit somewhere on the sale of freedom and formality, and how successful projects include a healthy mix of formal checking and free exploration. Finally, we’ll learn how to bring structure to our exploratory testing efforts using sessions, tours and variables.

Transcript

1. Hello, I’m Matt Obee I’m a Senior Software Tester
 at Holiday Extras
2. Checking & Exploring: Testing in Agile projects
3. Testing in the past Exploratory Testing Scripted Testing
4. Testing Today Scripted Checking Testing
5. “Exploratory testing is not a testing technique. It's a way of thinking about testing. Any testing technique can be used in an exploratory way.” 
 Cem Kaner & James Bach (2004)
6. No freedom More freedom
 Less formal Scale of Testing Freedom & Formality Less freedom
 More formal CHECKING EXPLORING Test Cases Test Data Scenarios Checklist Mind Map Charter Tour Play Scripted Checking
7. Does it do everything that it’s supposed to do? ▪ Asks closed (yes/no) questions; ▪ Validation and verification; ▪ Needs specification; ▪ Often automated; ▪ Developers do a lot of checking. Are there any risks or opportunities? ▪ Asks open ended questions; ▪ Does not need specification; ▪ Results of the previous test inform the next; ▪ Used to evaluate subjective qualities (e.g. usability and aesthetics); ▪ Rarely automated; CHECKING EXPLORING
8. Tested = Checked + Explored 
 “You’re not done testing until you’ve checked that the software meets expectations and you’ve explored whether there are additional risks. A comprehensive test strategy incorporates both approaches.” Elisabeth Hendrickson (2013)
9. ▪ Complex multi-page forms ▪ Complex processing systems ▪ Highly-regulated industry and bureaucratic client ▪ Accuracy, consistency and compliance ▪ Overall project risk is high
10. No freedom More freedom
 Less formal Less freedom
 More formal Test Cases Test Data Checklist Scripted Checking
11. ▪ Not highly regulated ▪ User experience is the main quality metric ▪ Quality is subjective and not easy to measure ▪ Relatively low risk but lots of opportunities
12. No freedom More freedom
 Less formal Less freedom
 More formal Test Data Scenarios Checklist Play Scripted Checking Personas
13. Recap: Checking & Exploring ▪ Exploratory testing isn’t a technique; ▪ All testing sits on the scale of freedom and formality; ▪ Less formality means more freedom to explore; ▪ Different projects need different levels of formality; ▪ Tested = Checked + Explored
14. Checking
15. “Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.” 
 James Bach & Michael Bolton (2013)
16. Checking is boring. Make computers do it.
17. “Automate 100% of the tests that should be automated.” Alan Page (2013) Automate all the things!!1!
18. Computers are good at... ▪ maths ▪ simulation ▪ generating data ▪ validating against rules ▪ doing repetitive tasks ▪ multitasking ▪ doing things quickly Computers suck at... ▪ recognising patterns ▪ learning and reacting ▪ ignoring false positives ▪ being random ▪ curiosity ▪ creativity ▪ empathy ▪ understanding context ▪ common sense
19. Checking standards conformance with
 linting and validation
20. Checking code logic
 with unit tests
21. Checking that nothing has broken with
 continuous integration
 and regression tests
22. Checking for risks
 and opportunities 
 in code review
23. Automation doesn’t replace human testers. Automation does the boring stuff so that humans have more time to do intelligent exploration.
24. Recap: Checking ▪ Checks need explicit rules and success criteria; ▪ Make computers do the boring stuff; ▪ Don’t fear automation. It’s doesn’t replace you, it helps you do more of the testing that you enjoy and less of the stuff that you don’t;
25. Exploring
32. Checklists aren’t just for checks
33. istock/Aviation Week
34. Tours can change your perspective
35. 50% of people don’t see the invisible gorilla. The same inattentional blindness happens when testing software. bit.ly/1jGOCoQ
36. The Guidebook Tour Staying on the main tourist trails Guidebook action by Jaymis Loveday
37. The Guidebook Tour ▪ Do what the guidebook (instructions, manual) recommends; ▪ Visit the top tourist destinations (the most popular features); ▪ Don’t leave the main tourist trails (avoid advanced features); ▪ Be lazy (don’t explore too much)
38. The Obsessive Compulsive Tour Doing the same thing over and over...
39. The Obsessive Compulsive Tour ▪ Repeat actions ▪ Click buttons more than once ▪ Go back and forth between the same pages ▪ Keep refreshing the page ▪ Type in a field, delete it, type in it again ▪ Make a selection, clear it, make the selection again ▪ Make the same booking twice
40. The Backstreet Tour Going off the beaten track
41. The Backstreet Tour ▪ Use the features that are rarely used ▪ Take the routes that are rarely taken ▪ Read the content that customers don’t normally read ▪ Buy the products that fewer customers buy ▪ Select the options that customers rarely select
42. The Bad Neighbourhood Tour Venturing into the rougher parts of town Light at the end of the Ghetto by Jenn Durfey
43. The Bad Neighbourhood Tour ▪ Concentrate on areas that are known to be buggy (bugs cluster) ▪ Explore nearby or related features ▪ Do things that the application is known to struggle with
44. The Nervous Traveller Tour Being careful and scrutinise every detail Issica demonstrates the brace position by Karl Baron
45. The Nervous Traveller Tour ▪ Look at everything with fresh eyes ▪ Do things slowly and pay attention to the details ▪ Read the microcopy (labels, instructions, error messages etc.) ▪ Ask lots of questions and look for the answers ▪ Be indecisive and change your mind ▪ What do our customers worry about?
46. The Antisocial Tour Breaking the rules Bad Parking by xlibber
47. ▪ Leave mandatory form fields empty ▪ Do things in the wrong order ▪ Do the same things more than once ▪ Start things but don’t finish them ▪ Be impatient ▪ Put JavaScript or SQL queries in form fields ▪ Try to access member features as a non-member The Antisocial Tour
48. Variables Experimenting with 
 different conditions
49. A software tester walks into a bar. Orders a beer. Orders 9999 beers. Orders 1.5 beers. Orders -1 beers. 
 Orders 0 beers. Orders a fdsfdsa. 
 Tries to leave without paying.
50. How would you test this?
51. A few pen testing variables ▪ Left and right handed users: try writing in either hand ▪ Writing speed: try writing quickly and slowly ▪ Writing pressure: try writing lightly and heavily ▪ Surface: try writing on matte paper, glossy paper, cardboard, plastic etc. ▪ Orientation: try writing on a table, on a wall, upside down etc. ▪ Environment: trying writing in hot conditions, cold conditions, wind, rain etc. ▪ Length of use: try writing a short note compared to a long essay ▪ State: what happens when changing from one state to another?
52. Variables 
 in software
53. Odd numbers : Even numbers by Graham Horn Vary numbers
54. Vary numbers ▪ Can you enter a negative number, e.g. “-1”? ▪ Can you enter a decimal number, e.g. “1.5”? ▪ Can you enter zero (e.g. “0”)? ▪ Can you exceed the minimum or maximum limits? ▪ What happens where there are zero, one or many things? ▪ Goldilocks (too many, too few, just right) ▪ What happens if you enter letters instead of numbers?
55. USS Yorktown
56. Amazon
57. Vary time
58. Vary time ▪ Can you specify a date of birth in the future? ▪ Can you schedule an event to happen in the past? ▪ In a date range, can you set the ending earlier than the beginning? ▪ What happens on leap days? ▪ What happens on dates that don’t exist (e.g. 31st September)? ▪ What happens if you reset time on the device forward and back? ▪ What happens if you move between timezones?
59. Vary the format
60. Vary the format ▪ What happens if you enter dates in different formats? 
 (e.g. 01/06/2015, 06/01/2015, 1/6/15, 1-6-15, 2015-06-01) ▪ What happens if there’s a space in the number? 
 (e.g. “01303 815 300”) ▪ What happens if a number includes special characters? 
 (e.g. “+44 (0)1303 815 300”) ▪ Can you upload different file formats? (e.g. .gif, .jpg, .doc, .docx) ▪ What happens if you enter numbers with different units of measurement?
61. Mars Climate Orbiter
62. Vary the journey Photo by Carsten Tolkmit
63. Vary the journey ▪ Can you use the browser back and forward buttons? ▪ What happens if you refresh the page? ▪ What happens if you arrive via a bookmark? ▪ Can you hack the URL? ▪ Can you do things in a different order? ▪ What happens if you have multiple tabs/windows running?
64. Vary the method of input Photo by utterlywrong
65. ▪ Can you use it without a mouse (i.e. keyboard only)? ▪ What happens if you copy and paste instead of typing into fields? ▪ Can you drag and drop content into fields? Vary the method of input
66. Vary the speed of input Photo by Nevit Dilmen
67. ▪ What happens if you interact with the application too quickly? ▪ What happens if you interact with the application too slowly? ▪ What happens if you don’t do anything for a length of time? Vary the speed of input
68. Therac-25
69. Vary Settings & States Control Panel by Les Chatfield
70. Vary Settings & States ▪ What happens if JavaScript is disabled? ▪ What happens if cookies are blocked? ▪ What happens if you change the size of the browser window? ▪ What happens if you change the browser font size? ▪ Compare behaviour when logged in and logged out ▪ What happens when the state changes? (e.g. a logged in user logging out, a logged out user logging in)
71. Testing Heuristics Cheat Sheet (PDF) bit.ly/1or9Inn
72. Recap: Exploring ▪ Checklists are one great way to structure exploration; ▪ Take tours to change your perspective; ▪ Variables are key. Identify the variables and see what happens when you change them.
73. Takeaways ▪ Tested = Checked + Explored
 Check to make sure it’s doing what it should; explore to see if there are new risks or opportunities. ▪ Make stupid computers do the boring checks so that intelligent humans have more time to explore. ▪ Use checklists, take tours and experiment with variables to get everyone involved in testing sessions.
74. Thank you @mattobee