Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Rapid_Fire_Black_Box_Test_Techniques__1_.pdf

Jenny Bramble
April 25, 2023
27

 Rapid_Fire_Black_Box_Test_Techniques__1_.pdf

Jenny Bramble

April 25, 2023
Tweet

Transcript

  1. • Director of Quality Engineering, Pap a • Tester by

    nature and nurtur e • Fascinated by human computer interactio n • Pronouns are she/her Jenny Bramble @jennydoesthings
  2. @jennydoesthings @jennydoesthings Verify that systems under test preform in the

    ways that all the stakeholders expect, or help reset those expectations.
  3. @jennydoesthings @jennydoesthings Testing is usually talked about in terms of

    white box and black box testing (though grey box is gaining popularity)
  4. @jennydoesthings @jennydoesthings Black box test techniques are ways to test

    that don’t consider the internal workings of the system.
  5. @jennydoesthings @jennydoesthings The way systems are built has a direct

    impact on how we test them. This is sometimes called grey box testing.
  6. @jennydoesthings Equivalence Partitioning • Divide items under test into partitions

    (or classes) that are tested and treated the same wa y • Best for testing ranges and group s • Partitions are either valid or invali d • There is no added value from retesting the same partition with different value s • A single value belongs to only one partitio n • Valid and invalid partitions must be tested to have full coverage
  7. @jennydoesthings Equivalence Partitioning • A risk with this type of

    testing is that we assume all items in the portions will act the same. It’s acceptable risk because the time saved is worth the risk of finding a bug like this rarely . • Example : • We added some new needs to service requests under existing types . • We assumed every need under the type would act the same (properly assigned priority, closed as appropriate ) • Testing every new need would have been over-testin g • We had a typo so one need was acting incorrectly . • Should we have tested everything? NO! Finding this bug should not change our manual testing
  8. @jennydoesthings Equivalence Partitioning • An input takes a integer from

    1 - 1 0 • Valid partition: 1 - 1 0 • Invalid partitions: less than 1, more than 10, non-integer character s • A vet treats cats . • Valid partition: cat s • Invalid partition: any thing that isn’t a cat
  9. @jennydoesthings Equivalence Partitioning A Pal is paid $0 for the

    first ten miles they travel on a visit. After that, they are paid $1 for each mile up to 20 miles and $.25 for any mile after that. Assume integers.
  10. @jennydoesthings Equivalence Partitioning • A candy store offers a discount

    based on how much you buy : • .1-.5 lbs: no discoun t • .6-1.0 lbs: 5% discoun t • 1.1-2.0 lbs: 10% discoun t • 2.0+ lbs: 20% discoun t • What is each partition ? •
  11. @jennydoesthings Boundary Value Analysis (BVA) • Identify and test values

    at the boundaries of input s • Boundaries are where we often see special cases, unusual behavior—you know, edge cases • Two-value BVA: test minimum and maximum of the boundar y • Three-value BVA: test a value above the boundary, at the boundary, and below the boundary
  12. @jennydoesthings Boundary Value Analysis (BVA) A Pal is paid $0

    for the first ten miles they travel on a visit. After that, they are paid $1 for each mile up to 20 miles and $.25 for any mile after that. Assume integers.
  13. @jennydoesthings Boundary Value Analysis (BVA) • A candy store offers

    a discount based on how much you buy : • .1-.5 lbs: no discoun t • .6-1.0 lbs: 5% discoun t • 1.1-2.0 lbs: 10% discoun t • 2.0+ lbs: 20% discoun t • What is each boundary ? •
  14. @jennydoesthings Boundary Value Analysis (BVA) .1 - .5 .6 -

    1.0 1.1 - 2.0 2.0+ <.1 Invalid No discount 5% discount 10% discount 20% discount
  15. @jennydoesthings Decision Tables • Two inputs that act together to

    affect the resul t • Works very well for business rules, especially with some complexit y • List the inputs as a table and fill in the outcome s • Can show you where there are equivalence partitions to reduce th number of set cases needed
  16. @jennydoesthings Decision Tables • A Pal is paid $0 for

    the first ten miles they travel on a visit. After that, they are paid $1 for each mile up to 20 miles and $.25 for any mile after that if they are in North Carolina . • A Pal is paid $1 for the first ten miles they travel on a visit. After that, they are paid $2 for each mile up to 20 miles and $1 up to 25 miles, but aren’t paid for any miles after that if they are in California . • Pals in Portland do not get paid commute miles, but the rest of Oregon follows North Carolina rules . • Assume integers.
  17. @jennydoesthings Decision Tables 0-10 miles 11-20 miles 21-25 miles 25+

    miles North Carolina $0 $1 $0.25 $0.25 California $1 $2 $1 $0 Portland, OR $0 $0 $0 $0 Oregon $0 $1 $0.25 $0.25
  18. @jennydoesthings Decision Tables 0-10 miles 11-20 miles 21-25 miles 25+

    miles North Carolina $0 $1 $0.25 $0.25 California $1 $2 $1 $0 Portland, OR $0 $0 $0 $0 Oregon $0 $1 $0.25 $0.25
  19. @jennydoesthings Decision Tables 0-10 miles 11-20 miles 21-25 miles 25+

    miles NC, OR $0 $1 $0.25 $0.25 California $1 $2 $1 $0 Portland, OR $0 $0 $0 $0
  20. @jennydoesthings Decision Tables • A candy store offers a discount

    based on how much you buy : • .1-.5 lbs: no discoun t • .6-1.0 lbs: 5% discoun t • 1.1-2.0 lbs: 10% discoun t • 2.0+ lbs: 20% discoun t • Chocolates and lollipops follow the standard discount but fudge is not discounted above 2lbs. Also, gummy candy is on sale with an additional 2% discount . • What is the decision table ?
  21. @jennydoesthings Decision Tables > 0.1 0.1 - 0.5 lbs 0.6

    - 1.0 lbs 1.1 - 2.0 lbs 2.1 Chocolates Lollipops Invalid 0% 5% 10% 20% Gummy candy Invalid 2% 7% 12% 22% Fudge Invalid 0% 5% 10% 0%
  22. @jennydoesthings Pairwise/Combinatorial Testing • What’s the minimum combinations of inputs

    we can test to have a certain level of confidence that the software is not presenting major defects ? • Look at all your inputs and use equivalence partitioning to find any partitions (pay attention to valid/invalid, inputs with the same outputs ) • Set up a table with all the inputs listed starting with the variable that has the most inputs and ending with the leas t • As you fill out the test cases, you want to see all pairs represented somewhere in your test cases
  23. @jennydoesthings Pairwise/Combinatorial Testing • A visit can be virtual or

    in person . • If the visit is in person, it could be transportation or in home . • The visit can be for a health plan Papa, a direct consumer Papa, or a Caregiver Papa . • The visit can be one time or recurring on a schedule: daily, weekly, bi-weekly, monthly . • The objectives for the visit can be cleaning, tech help, or yard work. If the visit is transportation, it could be grocery shopping or doctor’s visit. All visits should have companionship . • The visit could be set to All Pals or Preferred Pals.
  24. @jennydoesthings Pairwise/Combinatorial Testing All combinations here would be 6x5x3x2x2x2 =

    720 test cases Objectives • Cleaning • Tech help • Yard work • Grocery shopping • Doctor’s visit • Companionship Schedule • One time • Daily • Weekly • Bi-weekly • Monthly Papa • Health plan • Direct consumer • Caregiver Visibility • All Pals • Preferred Pals Type • Virtual • Inperson Class • Transportation • In-home
  25. @jennydoesthings Pairwise/Combinatorial Testing Reduce some of the inputs. Now we’re

    down to 120! Objectives • Cleaning • Tech help • Yard work • Grocery shopping • Doctor’s visit • Companionship Schedule • One time • Reoccurring • Daily • Weekly • Bi-weekly • Monthly Papa • Health plan • Direct consumer • Caregiver • 3rd party pays Visibility • All Pals • Preferred Pals Type • Virtual • In person - transpo • In person - in home
  26. @jennydoesthings Pairwise/Combinatorial Testing • Now we start to fill in

    our table, starting with the variable that is the biggest, repeating it as many times as the next biggest . •
  27. @jennydoesthings Pairwise/Combinatorial Testing • Fill in the next column .

    • As you do, make sure that there is a representative of each pair of the previous and current column . • IE: for virtual visits, we should have a Self-Pay and 3rd Party option somewhere in the list.
  28. @jennydoesthings Pairwise/Combinatorial Testing • Continue filling the table, making sure

    that all pairs are represented somewhere . • This is the most important part of this style of testing so take your time.
  29. @jennydoesthings Pairwise/Combinatorial Testing • Now we’re down to 16 test

    cases ! • That’s doable so we’re done right ? • Nope, still too many test cases. Let’s continue to reduce it . • First: are any of these cases invalid?
  30. @jennydoesthings Pairwise/Combinatorial Testing • Make sure that all your pairs

    are still represented and add rows for any that aren’t . • Leave cells blank if the values don’t matter.
  31. @jennydoesthings Pairwise/Combinatorial Testing • 12 test cases, down from 720

    ! • This gets us full coverage of all pairs of inputs and will allow us to be confident that unless there is an interaction we aren’t aware of . • There are tools that will do this for you and that’s my go-to.
  32. @jennydoesthings Pairwise/Combinatorial Testing When buying candy, you have lots of

    options! You can buy chocolates, lollipops, fudge, or gummy candy. They can be packaged to go or you can eat them in store. If you get them to go, you can also get them gift wrapped. We offer blue, yellow, and green gift wrapping. The fudge and chocolate are available in gluten free versions which are an extra $1 per half pound. Candy may be purchased in .1lb increments from .1 lb to 10lbs using the previously described discount structure.
  33. @jennydoesthings State Transition Testing • This technique focuses on events

    that transition from one state to another. The system can be in only one state at a time . • There is a distinct event that results in the transition into our out of state but multiple events can move the system into the same state . • Coverage is described in switches : • One transition: 0-switch coverag e • Two transitions: 1-switch coverag e • If you are looking at more than one or two transitions, rethink your strategy or understanding of how your states affect each other . • This works best when states are influenced by the previous state and transition but not the entire chain of states
  34. @jennydoesthings State Transition Testing • Testing each transition once provides

    100% coverage . • We want to ensure we can make valid transitions and avoid invalid transitions . • You can use a flow diagram or a state table . • In a state table, list the states on the left and events across the top . • You need to have a clear idea of your states and the transition events.
  35. @jennydoesthings State Transition Testing Pal clicks in the Pal App

    Agent transitions the visit forward in Admin Agent transitions the visit backward in Admin Stuck Visit Bot takes action Pending Accepted Expired Accepted Confirmed, Pending Confirmed Pending Expired Confirmed Enroute, Pending Enroute Accepted Expired Enroute Started Started Confirmed Started Completed Completed (must provide override duration minutes) Enroute Completed Reviewed (Pal must write review) Reviewed, Terminated Started Reviewed Terminated Expired Terminated
  36. @jennydoesthings State Transition Testing • For each state, multiple the

    incoming and outgoing states then add up all your values. This is your starting point. (72 in our example! ) • Now write them down and look at the resulting cases. See if you can combine any or some are invalid . • Then decide if you need a long or short chain of test s • Consider if it matters how we got into the state or if we just need to be in the right state . • If we don’t need to move through the whole flow, then we shouldn’t.
  37. @jennydoesthings State Transition Testing • Generally, in our testing, we

    care about one particular state so focus on that . • Do we care what happened before it got into that state or do we only care about the visit being in that state? If we care about the states before, we may be making bad assumptions.
  38. @jennydoesthings @jennydoesthings Verify that systems under test preform in the

    ways that all the stakeholders expect, or help reset those expectations.
  39. @jennydoesthings @jennydoesthings Testing is usually talked about in terms of

    white box and black box testing (though grey box is gaining popularity)
  40. @jennydoesthings @jennydoesthings Black box test techniques are ways to test

    that don’t consider the internal workings of the system.
  41. @jennydoesthings @jennydoesthings The way systems are built has a direct

    impact on how we test them. This is sometimes called grey box testing.
  42. @jennydoesthings @jennydoesthings Unit testin g Scripted Testin g Ad Hoc

    Testin g Exploratory Testin g Integration Testin g Performance Testin g Smoke Testin g Regression Testin g User Acceptance Testing
  43. @jennydoesthings Unit Testing • A form of automated white box

    testing that lives deep in our codebas e • Generally, these run very quickly and are considered chea p • Tests the smallest unit of code possibl e • Generally written by developer s • Great place to test functionality like dates and math
  44. @jennydoesthings Scripted Testing • Sometimes referred to as ‘checking ’

    • Follows a prescribed script or test cas e • No room for investigatio n • Defects mean the scripts fai l • Test cases are generally scripted black box testin g • Automated tests are scripted white box testing
  45. @jennydoesthings Ad Hoc Testing • Randomly exploring the application or

    featur e • No real goal—just wandering around to see what you fin d • Most of our testing is ad hoc testing around feature s • Bug bashes are an example of ad hoc testin g • This is generally considered black or grey box testin g • Cannot be automated
  46. @jennydoesthings Exploratory Testing • Simultaneous learning, test design, and test

    executio n • Has a clear mission called a charte r • Take notes as you go or record session s • Hold off putting in or chasing defects until the session is ove r • Focuses on exploring with a ma p • This is generally considered black box testing • Cannot be automated
  47. @jennydoesthings Integration/End-to-End Testing • Follows a work flow from point

    A to point B • Generally done as a user or with a user type proces s • Tests the intersection of several modules, flows, or system s • Third party APIs, backend systems, front end systems, databases and more can be layers that need to be tested together • This can be white or black box testin g • Can be automated
  48. @jennydoesthings Performance Testing • Making sure that your systems perform

    as expected under given workload s • Several types of testing fall under this umbrella : • Load - expected loa d • Soak - expected load for an extended duratio n • Stress testing - higher than expected loa d • Spike testing - sudden influx of users or connection s • Volume testing - dramatic increase in data coming i n • If you don’t do it, your users wil l • This is white box testing . • Almost always scripted
  49. @jennydoesthings Smoke Testing • Checking quickly to ensure primary flows

    are workin g • Asks the basic questions we need to answer to ensure the base functionality is unchange d • Should be very short and quic k • Smoke tests are expected to pas s • This is generally black box testing • Can be automated
  50. @jennydoesthings Regression Testing • Conferring that no behavior has changed

    since the previous iteration of the softwar e • Often done before large change s • Regression is generally a heavy burden on the team and should be avoide d • Regression should take into account planned changes and should never be ‘test everything ’ • This is generally black box testin g • Can be automated (sort of)
  51. @jennydoesthings User Acceptance Testing (UAT) • Putting your software in

    front of your end users and in real world scenarios to ensure it meets their need s • Generally formalized as the team asks the end users to focus on specific flow s • Seeks to harden and find unknown requirements before releas e • This is considered black box testin g • Cannot be automated
  52. Do You Want Bugs? Because That’s How You Get Bugs.

    @jennydoesthings Jenny Brambl e Director of Engineerin g Papa
  53. @jennydoesthings User Acceptance Testing (UAT) • Putting your software in

    front of your end users and in real world scenarios to ensure it meets their need s • Generally formalized as the team asks the end users to focus on specific flow s • Seeks to harden and find unknown requirements before releas e • Cannot be automated