A Quick Introduction to Domain Testing: More than Quick Tests

A Quick Introduction to Domain Testing: More than Quick Tests

Domain Testing is an umbrella term for Boundary Analysis and Equivalence Class Partitioning. This is a quick introduction.


Chris Kenst

August 08, 2018


  1. 2.

    About Us Chris Kenst • Automation Engineer at BloomNation •

    Lead BBST Instructor for the AST • Runs AST’s webinar program • Maintains a open source list of software testing conferences at TestingConferences.org • Nominee for the Board of the AST Dwayne Green • Test Lead at 1-800 Contacts • Learning how to train testers through practice - Deliberate Practice in Testing
  2. 3.

    Objectives • Practice applying Domain Testing ◦ We develop skill

    over time, with practice and feedback. ◦ We hope this workshop provides some practice and feedback • Identify and characterize data types used by variables in the application • Determine variables that are good candidates for domain testing ◦ When looking at an application be able to exercise variables using data • Stretch Goal: Generate best representative data and put that information into a domain testing table
  3. 4.

    Straw Poll 1. How many people have heard of Domain

    Testing before? 2. What about Boundary Analysis or Equivalence Class Partitioning? 3. How many people here have applied either Domain Testing before (or Boundary Analysis or Equivalence Class Partitioning)? 4. How many people have heard of the term Quick tests or Quick attacks? 5. Has anyone taken the BBST Test Design course?
  4. 5.

    What are Quick Tests or Quick Attacks? • Quick Tests

    or Quick attacks are tests that don’t cost much to design, are based on some guesstimate for how the system could fail (risk-based) and don’t take much prior knowledge to apply. ◦ Great for starting your testing but can run out of steam quickly. What do you do then?
  5. 6.

    Test Design Overview • Different tests reveal different kinds of

    information • Techniques are a way to guide our thinking as we develop a group of tests ◦ Think of functional testing, specification testing, domain testing, et. al as techniques • Techniques have different attributes or characteristics (see right)
  6. 7.

    Test Design Overview • Boundary Analysis and Equivalence Class Partitioning

    are different techniques but they often go hand in hand. • Complete testing is impossible ◦ Any sufficiently complex application will have too many possible values to test ◦ Too many inputs for us to test with a reasonable amount of time ◦ How do we deal with this problem?
  7. 8.

    Why is it called Domain Testing? • In Mathematics a

    Domain is a set of possible values for a function ◦ Input domain ◦ Output domain • In Software Development, Domain Testing is an umbrella term for: ◦ Boundary analysis ◦ Equivalence class partitioning ◦ (Depending on the test design book you are reading you might this might also be called Domain Analysis or Input Domain Analysis)
  8. 9.

    What is Domain Testing? • It’s a risk-based sampling strategy

    for efficiently hunting for bugs in a programs handling of data ◦ Anytime you have a very large set of data you can use and want to reduce it to a smaller sample size • We can’t run every possible test for every value ◦ Complete testing is impossible • Instead we can select a few values we think will uncover issues ◦ Power is a characteristic we are looking for in all domain tests
  9. 10.

    Testing with Variables • Variables: a value or object that

    a program interacts with. • Data Types: Programmers define the types of their variables ◦ Integer ◦ Boolean ◦ Floating Point ◦ Strings ◦ Enumerated values ◦ Byte ◦ Char • Variables can have relationships with other variables
  10. 11.

    What is boundary analysis? • Boundaries are exactly what you

    think they are • Think of the Goldilocks rule: ◦ Too big ◦ Too small ◦ Just right ◦ But also.. Too strange? • Example using a Data Type: ◦ Integer Value (Int32) ▪ Too Big: 2147483648 ▪ Too Small: -2147483649 ▪ Just Right: 0 or 1 or 1000 ▪ Too Strange: A#$@! or (2-1) or 0.01 or +=1
  11. 12.

    What is boundary analysis? • Example 2: a password field

    ◦ Accepts a range of characters with a minimum of 5 and a maximum of 16 ◦ The size of the field has the following boundaries: ▪ 0 - 4: Too few characters ▪ 5 - 16: Just the right number of characters ▪ 17+: Too many characters
  12. 13.

    Exercise 1 - Variable Tour • Charter: Tour the Mortgage-Calc

    application looking for what you can change. Anything you can change is a variable. Find them all and make a list. Select one interesting variable and write down what values you think it can take. • Timebox: 20 Minutes
  13. 14.
  14. 15.

    Variables for MortgageCalc Input Variables: • Home Value • Loan

    Amount • Down Payment • Interest • Term • Property Tax • HOA Amount • Extra Payment Storage Variables: • NumberOfMonthlyPayments = Term * 12 MonthlyRate = Interest/12/100 Result Variables: • Monthly Taxes & Fees = (HomeValue * PropertyTax / 12) • Total Payment = Monthly Mortgage + Taxes & Fees) • Monthly Mortgage = (MonthlyRate * LoanAmount * (1 + MonthlyRate) ^ Number of MonthlyPayments) / ((1 + MonthlyRate) ^ NumberOfMonthlyPayments) - 1 https://en.wikipedia.org/wiki/Mortgage_calculator#Monthly_payment_formula
  15. 16.

    Expanding on Variables • We work with this idea of

    notational variables ◦ We analyze the program as if these variables and our understanding of them is correct but it may not be ◦ This makes the work we do subjective but valuable • Input, storage and result (output) variables ◦ Calculations should add up ◦ Where is the data going? ◦ Do any of these variables constrain or effect each other • Can you get past the input filter? ◦ Are you looking at the results?
  16. 17.

    Useful Data Types • Some data types are more applicable

    than others… • Good for Domain Testing ◦ Integers ◦ Floating Point, Fixed Point ◦ Strings ◦ Byte ◦ Char ◦ Arrays ◦ Date • Not good ◦ Binary or Boolean ◦ Enumerated
  17. 18.

    What is an equivalence class? • A set of values

    we think the program will handle the same way ◦ If one catches a bug, they all will ◦ If one doesn’t catch a bug, the others probably won’t either • We partition (or split) a domain (input or output) based on how we think the application will handle it. ◦ Valid Values ◦ Invalid Values ◦ Risks • Much of this partitioning is based on what data type we think this variable is
  18. 19.

    What is a best representative? • There are 2 main

    tasks for Domain Testing: ◦ Partition or split a domain into an equivalence class(es) ◦ Select a few representatives from each class to test with • A best representative of an equivalence class is a boundary or another value in the class that seems more likely than other values to cause a program failure. ◦ This is the idea that all domain tests should be powerful.
  19. 20.

    Exercise 2 - One Good Variable • Charter: Tour either

    MortgageCalc or ChrisReads application with the goal of identifying one variable you think is a good candidate for domain testing and identify the best representative data. Organize your work however you want, but list out the best representative data for the variable. • Timebox: 30 minutes
  20. 21.
  21. 23.

    Organizing your work • You will find so many variables

    and equivalence classes that you’ll need a way to organize them • We recommend: ◦ Using a Domain testing table or ◦ Creating an outline
  22. 24.

    When do we apply Domain Testing? • Going back to

    our Test Design overview, good test strategies will involve many techniques ◦ This means you are likely to apply domain testing combined with another technique like functional testing • For practice you might use an ET Tour focused on understanding the variables of a system during which you might try to apply this. ◦ This is exactly what we did ◦ You can create Exploratory Testing Charters to help
  23. 25.

    Scratching the Surface • We’ve covered a few things: ◦

    Identified and characterized variables based on Data Type ◦ Determined variables that are good candidates for domain testing ◦ Applied domain testing by ▪ Splitting a domain into equivalence classes ▪ Selected best Representatives • This goes wider and deeper ◦ BBST Test Design and BBST Domain Testing introduce a schema for helping with this technique that is 18 steps long
  24. 26.

    Practicing Further • We recommend practicing on your own applications

    either using Exploratory Tours or by integrating what you learned into your existing test strategy. • A few classes you could take: ◦ BBST Domain Testing -> runs once or twice per year ◦ BBST Test Design -> runs more frequently • Try a risk-based approach to Domain Testing (assignment)
  25. 27.

    References • Domain Testing: ◦ Beizer, Boris - Software Testing

    Techniques ◦ Copeland, Lee - A Practitioner’s Guide To Software Test Design ◦ Kaner, Cem - BBST Domain Testing Workbook ◦ Kaner, Cem - BBST Test Design Workbook ◦ Jorgensen, Paul - Software Testing: A Craftsman Approach, 3rd Edition • Quick tests: ◦ https://blog.gurock.com/beyond-quick-attacks/ ◦ https://searchsoftwarequality.techtarget.com/tip/Ten-quick-attacks-for-web-based-software ◦ https://www.kenst.com/2018/05/what-are-quicktests-and-when-are-they-used/ ◦ Heuristic Test Strategy Model ◦ Hendrickson’s Cheat Sheet