Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Rapid_Fire_Black_Box_Test_Techniques__1_.pdf

Jenny Bramble
April 25, 2023
24

 Rapid_Fire_Black_Box_Test_Techniques__1_.pdf

Jenny Bramble

April 25, 2023
Tweet

Transcript

  1. Rapid Fire Black
    Box Test Techniques
    @jennydoesthings
    Jenny Brambl
    e

    Director of Engineerin
    g

    Papa

    View full-size slide

  2. • Director of Quality Engineering,
    Pap
    a

    • Tester by nature and nurtur
    e

    • Fascinated by human computer
    interactio
    n

    • Pronouns are she/her
    Jenny Bramble
    @jennydoesthings

    View full-size slide

  3. @jennydoesthings
    @jennydoesthings
    What is testing?

    View full-size slide

  4. @jennydoesthings
    @jennydoesthings
    Verify that systems under test
    preform in the ways that all
    the stakeholders expect, or
    help reset those expectations.

    View full-size slide

  5. @jennydoesthings
    @jennydoesthings
    Testing is usually talked about
    in terms of white box and
    black box testing (though grey
    box is gaining popularity)

    View full-size slide

  6. @jennydoesthings
    @jennydoesthings
    White box testing
    techniques consider the
    internals of the system
    and how it’s built.

    View full-size slide

  7. @jennydoesthings
    @jennydoesthings
    Think of it as testing
    that knows the code
    on a deep level.

    View full-size slide

  8. @jennydoesthings
    @jennydoesthings
    Black box test techniques
    are ways to test that don’t
    consider the internal
    workings of the system.

    View full-size slide

  9. @jennydoesthings
    @jennydoesthings
    Haha, no one
    does that.

    View full-size slide

  10. @jennydoesthings
    @jennydoesthings
    The way systems are built has
    a direct impact on how we
    test them. This is sometimes
    called grey box testing.

    View full-size slide

  11. @jennydoesthings
    @jennydoesthings
    Today, I want to talk
    specifically about techniques
    that the industry considers
    black box techniques.

    View full-size slide

  12. @jennydoesthings
    @jennydoesthings
    Equivalence Partitionin
    g

    Boundary Value Analysi
    s

    Decision Table
    s

    Combinatorial Testin
    g

    State Transition Testing

    View full-size slide

  13. @jennydoesthings
    @jennydoesthings
    So let’s jump in.

    View full-size slide

  14. @jennydoesthings
    @jennydoesthings
    Warning: there’s a LOT of
    text on these slides so
    they can be a resource for
    later.

    View full-size slide

  15. @jennydoesthings
    Equivalence Partitioning
    • Divide items under test into partitions (or classes)
    that are tested and treated the same wa
    y

    • Best for testing ranges and group
    s

    • Partitions are either valid or invali
    d

    • There is no added value from retesting the same
    partition with different value
    s

    • A single value belongs to only one partitio
    n

    • Valid and invalid partitions must be tested to have
    full coverage

    View full-size slide

  16. @jennydoesthings
    Equivalence Partitioning
    • A risk with this type of testing is that we assume all items in the portions
    will act the same. It’s acceptable risk because the time saved is worth the
    risk of finding a bug like this rarely
    .

    • Example
    :

    • We added some new needs to service requests under existing types
    .

    • We assumed every need under the type would act the same (properly
    assigned priority, closed as appropriate
    )

    • Testing every new need would have been over-testin
    g

    • We had a typo so one need was acting incorrectly
    .

    • Should we have tested everything? NO! Finding this bug should not
    change our manual testing

    View full-size slide

  17. @jennydoesthings
    Equivalence Partitioning
    • An input takes a integer from 1 - 1
    0

    • Valid partition: 1 - 1
    0

    • Invalid partitions: less than 1, more than
    10, non-integer character
    s

    • A vet treats cats
    .

    • Valid partition: cat
    s

    • Invalid partition: any thing that isn’t a cat

    View full-size slide

  18. @jennydoesthings
    Equivalence Partitioning
    A Pal is paid $0 for the first ten
    miles they travel on a visit. After
    that, they are paid $1 for each mile
    up to 20 miles and $.25 for any
    mile after that. Assume integers.

    View full-size slide

  19. @jennydoesthings
    Equivalence Partitioning
    • A candy store offers a discount based on how
    much you buy
    :

    • .1-.5 lbs: no discoun
    t

    • .6-1.0 lbs: 5% discoun
    t

    • 1.1-2.0 lbs: 10% discoun
    t

    • 2.0+ lbs: 20% discoun
    t

    • What is each partition
    ?


    View full-size slide

  20. @jennydoesthings
    Boundary Value Analysis (BVA)
    • Identify and test values at the boundaries of input
    s

    • Boundaries are where we often see special cases,
    unusual behavior—you know, edge cases
    • Two-value BVA: test minimum and maximum of
    the boundar
    y

    • Three-value BVA: test a value above the
    boundary, at the boundary, and below the
    boundary

    View full-size slide

  21. @jennydoesthings
    Boundary Value Analysis (BVA)
    A Pal is paid $0 for the first ten
    miles they travel on a visit. After
    that, they are paid $1 for each mile
    up to 20 miles and $.25 for any
    mile after that. Assume integers.

    View full-size slide

  22. @jennydoesthings
    Boundary Value Analysis (BVA)
    0-10 11 - 20 20+
    $0/mile $1/mile $0.25/mile

    View full-size slide

  23. @jennydoesthings
    Boundary Value Analysis (BVA)
    • A candy store offers a discount based on how
    much you buy
    :

    • .1-.5 lbs: no discoun
    t

    • .6-1.0 lbs: 5% discoun
    t

    • 1.1-2.0 lbs: 10% discoun
    t

    • 2.0+ lbs: 20% discoun
    t

    • What is each boundary
    ?


    View full-size slide

  24. @jennydoesthings
    Boundary Value Analysis (BVA)
    .1 - .5 .6 - 1.0 1.1 - 2.0 2.0+
    <.1
    Invalid No discount 5% discount 10% discount 20% discount

    View full-size slide

  25. @jennydoesthings
    Decision Tables
    • Two inputs that act together to affect the resul
    t

    • Works very well for business rules, especially
    with some complexit
    y

    • List the inputs as a table and fill in the
    outcome
    s

    • Can show you where there are equivalence
    partitions to reduce th number of set cases
    needed

    View full-size slide

  26. @jennydoesthings
    Decision Tables
    • A Pal is paid $0 for the first ten miles they travel on a visit.
    After that, they are paid $1 for each mile up to 20 miles and
    $.25 for any mile after that if they are in North Carolina
    .

    • A Pal is paid $1 for the first ten miles they travel on a visit.
    After that, they are paid $2 for each mile up to 20 miles and
    $1 up to 25 miles, but aren’t paid for any miles after that if
    they are in California
    .

    • Pals in Portland do not get paid commute miles, but the
    rest of Oregon follows North Carolina rules
    .

    • Assume integers.

    View full-size slide

  27. @jennydoesthings
    Decision Tables
    0-10
    miles
    11-20
    miles
    21-25
    miles
    25+
    miles
    North
    Carolina
    $0 $1 $0.25 $0.25
    California $1 $2 $1 $0
    Portland,
    OR
    $0 $0 $0 $0
    Oregon $0 $1 $0.25 $0.25

    View full-size slide

  28. @jennydoesthings
    Decision Tables
    0-10
    miles
    11-20
    miles
    21-25
    miles
    25+
    miles
    North
    Carolina
    $0 $1 $0.25 $0.25
    California $1 $2 $1 $0
    Portland,
    OR
    $0 $0 $0 $0
    Oregon $0 $1 $0.25 $0.25

    View full-size slide

  29. @jennydoesthings
    Decision Tables
    0-10
    miles
    11-20
    miles
    21-25
    miles
    25+
    miles
    NC, OR $0 $1 $0.25 $0.25
    California $1 $2 $1 $0
    Portland,
    OR
    $0 $0 $0 $0

    View full-size slide

  30. @jennydoesthings
    Decision Tables
    • A candy store offers a discount based on how much you buy
    :

    • .1-.5 lbs: no discoun
    t

    • .6-1.0 lbs: 5% discoun
    t

    • 1.1-2.0 lbs: 10% discoun
    t

    • 2.0+ lbs: 20% discoun
    t

    • Chocolates and lollipops follow the standard discount but
    fudge is not discounted above 2lbs. Also, gummy candy is
    on sale with an additional 2% discount
    .

    • What is the decision table
    ?

    View full-size slide

  31. @jennydoesthings
    Decision Tables
    > 0.1
    0.1 - 0.5


    lbs
    0.6 - 1.0


    lbs
    1.1 - 2.0


    lbs
    2.1
    Chocolates


    Lollipops
    Invalid 0% 5% 10% 20%
    Gummy
    candy
    Invalid 2% 7% 12% 22%
    Fudge Invalid 0% 5% 10% 0%

    View full-size slide

  32. @jennydoesthings
    What if there ar
    e

    a LOT of inputs?

    View full-size slide

  33. @jennydoesthings
    Pairwise/Combinatorial Testing
    • What’s the minimum combinations of inputs we can test to
    have a certain level of confidence that the software is not
    presenting major defects
    ?

    • Look at all your inputs and use equivalence partitioning to
    find any partitions (pay attention to valid/invalid, inputs
    with the same outputs
    )

    • Set up a table with all the inputs listed starting with the
    variable that has the most inputs and ending with the leas
    t

    • As you fill out the test cases, you want to see all pairs
    represented somewhere in your test cases

    View full-size slide

  34. @jennydoesthings
    Pairwise/Combinatorial Testing
    • A visit can be virtual or in person
    .

    • If the visit is in person, it could be transportation or in home
    .

    • The visit can be for a health plan Papa, a direct consumer
    Papa, or a Caregiver Papa
    .

    • The visit can be one time or recurring on a schedule: daily,
    weekly, bi-weekly, monthly
    .

    • The objectives for the visit can be cleaning, tech help, or yard
    work. If the visit is transportation, it could be grocery shopping
    or doctor’s visit. All visits should have companionship
    .

    • The visit could be set to All Pals or Preferred Pals.

    View full-size slide

  35. @jennydoesthings
    Pairwise/Combinatorial Testing
    All combinations here would be 6x5x3x2x2x2 = 720 test cases
    Objectives
    • Cleaning

    • Tech help

    • Yard work

    • Grocery shopping

    • Doctor’s visit

    • Companionship
    Schedule
    • One time

    • Daily

    • Weekly

    • Bi-weekly

    • Monthly

    Papa
    • Health plan

    • Direct consumer

    • Caregiver
    Visibility
    • All Pals

    • Preferred Pals

    Type
    • Virtual

    • Inperson

    Class
    • Transportation

    • In-home

    View full-size slide

  36. @jennydoesthings
    Pairwise/Combinatorial Testing
    Reduce some of the inputs. Now we’re down to 120!
    Objectives
    • Cleaning

    • Tech help

    • Yard work

    • Grocery shopping

    • Doctor’s visit

    • Companionship
    Schedule
    • One time

    • Reoccurring

    • Daily

    • Weekly

    • Bi-weekly

    • Monthly

    Papa
    • Health plan

    • Direct consumer

    • Caregiver

    • 3rd party pays
    Visibility
    • All Pals

    • Preferred Pals

    Type
    • Virtual

    • In person - transpo

    • In person - in home

    View full-size slide

  37. @jennydoesthings
    Pairwise/Combinatorial Testing
    • Now we start to fill in our table,
    starting with the variable that is
    the biggest, repeating it as
    many times as the next biggest
    .


    View full-size slide

  38. @jennydoesthings
    Pairwise/Combinatorial Testing

    View full-size slide

  39. @jennydoesthings
    Pairwise/Combinatorial Testing
    • Fill in the next column
    .

    • As you do, make sure that there is a
    representative of each pair of the
    previous and current column
    .

    • IE: for virtual visits, we should have a
    Self-Pay and 3rd Party option
    somewhere in the list.

    View full-size slide

  40. @jennydoesthings
    Pairwise/Combinatorial Testing

    View full-size slide

  41. @jennydoesthings
    Pairwise/Combinatorial Testing
    • Continue filling the table, making
    sure that all pairs are represented
    somewhere
    .

    • This is the most important part of
    this style of testing so take your
    time.

    View full-size slide

  42. @jennydoesthings
    Pairwise/Combinatorial Testing

    View full-size slide

  43. @jennydoesthings
    Pairwise/Combinatorial Testing
    • Now we’re down to 16 test cases
    !

    • That’s doable so we’re done right
    ?

    • Nope, still too many test cases.
    Let’s continue to reduce it
    .

    • First: are any of these cases
    invalid?

    View full-size slide

  44. @jennydoesthings
    Pairwise/Combinatorial Testing

    View full-size slide

  45. @jennydoesthings
    Pairwise/Combinatorial Testing
    • Make sure that all your pairs
    are still represented and
    add rows for any that aren’t
    .

    • Leave cells blank if the
    values don’t matter.

    View full-size slide

  46. @jennydoesthings
    Pairwise/Combinatorial Testing

    View full-size slide

  47. @jennydoesthings
    Pairwise/Combinatorial Testing
    • 12 test cases, down from 720
    !

    • This gets us full coverage of all pairs of
    inputs and will allow us to be confident
    that unless there is an interaction we
    aren’t aware of
    .

    • There are tools that will do this for you
    and that’s my go-to.

    View full-size slide

  48. @jennydoesthings
    Pairwise/Combinatorial Testing
    When buying candy, you have lots of options! You can
    buy chocolates, lollipops, fudge, or gummy candy. They
    can be packaged to go or you can eat them in store. If
    you get them to go, you can also get them gift wrapped.
    We offer blue, yellow, and green gift wrapping. The
    fudge and chocolate are available in gluten free
    versions which are an extra $1 per half pound. Candy
    may be purchased in .1lb increments from .1 lb to 10lbs
    using the previously described discount structure.

    View full-size slide

  49. @jennydoesthings
    State Transition Testing
    • This technique focuses on events that transition from one state to
    another. The system can be in only one state at a time
    .

    • There is a distinct event that results in the transition into our out of
    state but multiple events can move the system into the same state
    .

    • Coverage is described in switches
    :

    • One transition: 0-switch coverag
    e

    • Two transitions: 1-switch coverag
    e

    • If you are looking at more than one or two transitions, rethink your
    strategy or understanding of how your states affect each other
    .

    • This works best when states are influenced by the previous state and
    transition but not the entire chain of states

    View full-size slide

  50. @jennydoesthings
    State Transition Testing
    • Testing each transition once provides 100% coverage
    .

    • We want to ensure we can make valid transitions and
    avoid invalid transitions
    .

    • You can use a flow diagram or a state table
    .

    • In a state table, list the states on the left and events
    across the top
    .

    • You need to have a clear idea of your states and the
    transition events.

    View full-size slide

  51. @jennydoesthings
    State Transition Testing
    Pal clicks in the Pal
    App
    Agent transitions the
    visit forward in
    Admin
    Agent transitions the
    visit backward in
    Admin
    Stuck Visit Bot takes
    action
    Pending Accepted Expired
    Accepted Confirmed, Pending Confirmed Pending Expired
    Confirmed Enroute, Pending Enroute Accepted Expired
    Enroute Started Started Confirmed
    Started Completed Completed (must
    provide override
    duration minutes)
    Enroute
    Completed Reviewed (Pal must
    write review)
    Reviewed, Terminated Started
    Reviewed Terminated
    Expired
    Terminated

    View full-size slide

  52. @jennydoesthings
    State Transition Testing

    View full-size slide

  53. @jennydoesthings
    State Transition Testing
    • For each state, multiple the incoming and outgoing states then
    add up all your values. This is your starting point. (72 in our
    example!
    )

    • Now write them down and look at the resulting cases. See if you
    can combine any or some are invalid
    .

    • Then decide if you need a long or short chain of test
    s

    • Consider if it matters how we got into the state or if we just need
    to be in the right state
    .

    • If we don’t need to move through the whole flow, then we
    shouldn’t.

    View full-size slide

  54. @jennydoesthings
    State Transition Testing
    • Generally, in our testing, we care about one
    particular state so focus on that
    .

    • Do we care what happened before it got into
    that state or do we only care about the visit
    being in that state? If we care about the
    states before, we may be making bad
    assumptions.

    View full-size slide

  55. @jennydoesthings
    @jennydoesthings
    Whew, that was a
    lot.

    View full-size slide

  56. @jennydoesthings
    @jennydoesthings
    Equivalence Partitionin
    g

    Boundary Value Analysi
    s

    Decision Table
    s

    Combinatorial Testin
    g

    State Transition Testing

    View full-size slide

  57. Questions?
    @jennydoesthings
    [email protected]
    • Twitter: @jennydoesthings

    View full-size slide

  58. So What is Testing,
    Anyway?
    @jennydoesthings
    Jenny Brambl
    e

    Director of Engineerin
    g

    Papa

    View full-size slide

  59. @jennydoesthings
    @jennydoesthings
    What is testing?

    View full-size slide

  60. @jennydoesthings
    @jennydoesthings
    Verify that systems under test
    preform in the ways that all
    the stakeholders expect, or
    help reset those expectations.

    View full-size slide

  61. @jennydoesthings
    @jennydoesthings
    Testing is usually talked about
    in terms of white box and
    black box testing (though grey
    box is gaining popularity)

    View full-size slide

  62. @jennydoesthings
    @jennydoesthings
    White box testing
    techniques consider the
    internals of the system
    and how it’s built.

    View full-size slide

  63. @jennydoesthings
    @jennydoesthings
    Think of it as testing
    that knows the code
    on a deep level.

    View full-size slide

  64. @jennydoesthings
    @jennydoesthings
    Black box test techniques
    are ways to test that don’t
    consider the internal
    workings of the system.

    View full-size slide

  65. @jennydoesthings
    @jennydoesthings
    Haha, no one
    does that.

    View full-size slide

  66. @jennydoesthings
    @jennydoesthings
    The way systems are built has
    a direct impact on how we
    test them. This is sometimes
    called grey box testing.

    View full-size slide

  67. @jennydoesthings
    @jennydoesthings
    Today, I want to talk about
    different types of testing
    so we can share
    vocabulary.

    View full-size slide

  68. @jennydoesthings
    @jennydoesthings
    Warning: there’s a LOT of
    text on these slides so
    they can be a resource for
    later.

    View full-size slide

  69. @jennydoesthings
    @jennydoesthings
    Unit testin
    g

    Scripted Testin
    g

    Ad Hoc Testin
    g

    Exploratory Testin
    g

    Integration Testin
    g

    Performance Testin
    g

    Smoke Testin
    g

    Regression Testin
    g

    User Acceptance Testing

    View full-size slide

  70. @jennydoesthings
    Unit Testing
    • A form of automated white box testing that lives
    deep in our codebas
    e

    • Generally, these run very quickly and are
    considered chea
    p

    • Tests the smallest unit of code possibl
    e

    • Generally written by developer
    s

    • Great place to test functionality like dates and math

    View full-size slide

  71. @jennydoesthings
    Unit Testing

    View full-size slide

  72. @jennydoesthings
    Scripted Testing
    • Sometimes referred to as ‘checking


    • Follows a prescribed script or test cas
    e

    • No room for investigatio
    n

    • Defects mean the scripts fai
    l

    • Test cases are generally scripted black box
    testin
    g

    • Automated tests are scripted white box testing

    View full-size slide

  73. @jennydoesthings
    Ad Hoc Testing
    • Randomly exploring the application or featur
    e

    • No real goal—just wandering around to see what you
    fin
    d

    • Most of our testing is ad hoc testing around feature
    s

    • Bug bashes are an example of ad hoc testin
    g

    • This is generally considered black or grey box testin
    g

    • Cannot be automated

    View full-size slide

  74. @jennydoesthings
    Exploratory Testing
    • Simultaneous learning, test design, and test executio
    n

    • Has a clear mission called a charte
    r

    • Take notes as you go or record session
    s

    • Hold off putting in or chasing defects until the session
    is ove
    r

    • Focuses on exploring with a ma
    p

    • This is generally considered black box testing


    • Cannot be automated

    View full-size slide

  75. @jennydoesthings
    Integration/End-to-End Testing
    • Follows a work flow from point A to point
    B

    • Generally done as a user or with a user type proces
    s

    • Tests the intersection of several modules, flows, or system
    s

    • Third party APIs, backend systems, front end systems,
    databases and more can be layers that need to be tested
    together


    • This can be white or black box testin
    g

    • Can be automated

    View full-size slide

  76. @jennydoesthings
    Performance Testing
    • Making sure that your systems perform as expected under given
    workload
    s

    • Several types of testing fall under this umbrella
    :

    • Load - expected loa
    d

    • Soak - expected load for an extended duratio
    n

    • Stress testing - higher than expected loa
    d

    • Spike testing - sudden influx of users or connection
    s

    • Volume testing - dramatic increase in data coming i
    n

    • If you don’t do it, your users wil
    l

    • This is white box testing
    .

    • Almost always scripted

    View full-size slide

  77. @jennydoesthings
    Smoke Testing
    • Checking quickly to ensure primary flows are workin
    g

    • Asks the basic questions we need to answer to
    ensure the base functionality is unchange
    d

    • Should be very short and quic
    k

    • Smoke tests are expected to pas
    s

    • This is generally black box testing


    • Can be automated

    View full-size slide

  78. @jennydoesthings
    Regression Testing
    • Conferring that no behavior has changed since the previous
    iteration of the softwar
    e

    • Often done before large change
    s

    • Regression is generally a heavy burden on the team and
    should be avoide
    d

    • Regression should take into account planned changes and
    should never be ‘test everything


    • This is generally black box testin
    g

    • Can be automated (sort of)

    View full-size slide

  79. @jennydoesthings
    User Acceptance Testing (UAT)
    • Putting your software in front of your end users and in
    real world scenarios to ensure it meets their need
    s

    • Generally formalized as the team asks the end users
    to focus on specific flow
    s

    • Seeks to harden and find unknown requirements
    before releas
    e

    • This is considered black box testin
    g

    • Cannot be automated

    View full-size slide

  80. Questions?
    @jennydoesthings
    [email protected]
    • Twitter: @jennydoesthings

    View full-size slide

  81. Do You Want Bugs?
    Because That’s How


    You Get Bugs.
    @jennydoesthings
    Jenny Brambl
    e

    Director of Engineerin
    g

    Papa

    View full-size slide

  82. @jennydoesthings
    @jennydoesthings
    What is testing?

    View full-size slide

  83. @jennydoesthings
    User Acceptance Testing (UAT)
    • Putting your software in front of your end users and
    in real world scenarios to ensure it meets their need
    s

    • Generally formalized as the team asks the end users
    to focus on specific flow
    s

    • Seeks to harden and find unknown requirements
    before releas
    e

    • Cannot be automated

    View full-size slide