Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding Cognitive Biases in Performance Measurement

Understanding Cognitive Biases in Performance Measurement

When measuring web performance, we often try to get a single number that we can trend over time. This may be the median page load time, hero image time, page speed score, or core web vitals score. But is it really that simple?

Users seldom visit just a single page on a site, so how do we account for varying performance across multiple pages? How do we tell which page’s performance impacts the overall user experience? How do various cognitive biases affect the user’s perception of our site’s performance?

As developers and data analysts, we have our own biases that affect how we look at the data and which problems we end up trying to solve. Often our measurements themselves may be affected by our confirmation bias.

In this talk, we go into different biases that may affect user perception as well as our ability to measure that perception, and ways in which to identify if our data exhibits these patterns.

Presented at: iJS Munich

Philip Tellis

October 26, 2022
Tweet

More Decks by Philip Tellis

Other Decks in Technology

Transcript

  1. Understanding Cognitive Biases
    in Performance Measurement
    Finding the factors that lead to abandonment

    View full-size slide

  2. Philip Tellis
    Principal RUM Distiller @ Akamai
    ● Analyses real user performance data from mPulse
    ● Author of the OpenSource boomerang RUM library
    twitter:@bluesmoon ⦿ github:@bluesmoon
    speakerdeck:@bluesmoon

    View full-size slide

  3. BIAS is an
    Expectation
    Our Journey Today...
    ★ Understanding Cognitive
    Biases
    ★ Signs of cognitive biases in
    browsing data
    ★ What can we do?

    View full-size slide

  4. Understanding Bias
    Good, Bad, Normal?

    View full-size slide

  5. Similarity Zero-Risk
    False Memory Expedience
    Experience Proximity
    Survivorship Negativity
    Safety Loss Aversion
    If you have a brain, you have bias.

    View full-size slide

  6. Bias stems from experience – It’s Normal
    ● Helps us learn
    Perceptual/Sensory Dissonance
    ● Keeps us safe
    Safety Bias, Loss Aversion, Negativity Bias
    ● Find our people
    Similarity Bias, Proximity Bias
    Boston Shipyard Artist’s Community

    View full-size slide

  7. https://upload.wikimedia.org/wikipedia/commons/6/65/Cognitive_bias_codex_en.svg

    View full-size slide

  8. Cognitive Biases
    ● Similarity Bias
    ● Expedience Bias
    ● Experience Bias
    ● Proximity Bias
    ● Safety Bias
    ● Serial-position effect
    ● False memory
    ● Duration neglect
    ● Peak–end rule
    ● Negativity bias
    ● Escalation of commitment
    ● Loss aversion
    ● Zero-risk bias
    ● Next-in-line effect
    ● Misattribution of memory
    ● Sunk cost
    ● Levels-of-processing
    ● Spacing effect

    View full-size slide

  9. Cognitive Biases - Related to Performance on the Web
    ● Similarity Bias
    ● Expedience Bias
    ● Experience Bias
    ● Proximity Bias
    ● Safety Bias
    ● Serial-position effect
    ● False memory
    ● Duration neglect
    ● Peak–end rule
    ● Negativity bias
    ● Escalation of commitment
    ● Loss aversion
    ● Zero-risk bias
    ● Next-in-line effect
    ● Misattribution of memory
    ● Sunk cost
    ● Levels-of-processing
    ● Spacing effect

    View full-size slide

  10. Cognitive Biases - This Talk
    ● Similarity Bias
    ● Expedience Bias
    ● Experience Bias
    ● Proximity Bias
    ● Safety Bias
    ● Serial-position effect
    ● False memory
    ● Duration neglect
    ● Peak–end rule
    ● Negativity bias
    ● Escalation of commitment
    ● Loss aversion
    ● Zero-risk bias
    ● Next-in-line effect
    ● Misattribution of memory
    ● Sunk cost
    ● Levels-of-processing
    ● Spacing effect

    View full-size slide

  11. Stats
    B
    reak
    A 500ms connection speed delay resulted in
    up to a 26% increase in peak frustration
    and up to an 8% decrease in engagement.
    Tammy Everts – The impact of network speed on emotional engagement

    View full-size slide

  12. some definitions
    Bounce Rate: Percentage of users on the site who leave after viewing one page.
    Retention Rate: Percentage of users on a particular page who remain on the
    site for at least one more page view.
    Conversion Rate: Percentage of users on the site who complete a goal or
    particular task.
    Goal: A task like a conversion, purchase, visiting a particular page, or viewing
    a certain number of pages.

    View full-size slide

  13. Serial-Position Effect
    …is the tendency of a person to recall the first and last items
    in a series best, and the middle items worst.
    Ebbinghaus, Hermann (1913). On memory: A contribution to experimental psychology

    View full-size slide

  14. Serial-Position Effect
    …is the tendency of a person to recall the first and last items
    in a series best, and the middle items worst.
    ● Retention rate might be a function of the first and latest pages
    ● The recency effect suggests that the latest page has a higher weight
    Ebbinghaus, Hermann (1913). On memory: A contribution to experimental psychology

    View full-size slide

  15. Peak-End Rule
    People judge an experience largely based on how they felt at
    its peak & at its end, rather than the sum or average of
    every moment of the experience.
    Kahneman, Daniel (2000). "Evaluation by moments, past and future"

    View full-size slide

  16. People judge an experience largely based on how they felt at
    its peak & at its end, rather than the sum or average of
    every moment of the experience.
    ● Retention rate depends on the best/worst and latest performance
    ● Conversion rate depends on the best/worst performance and that of the page
    just before the conversion
    Peak-End Rule
    Kahneman, Daniel (2000). "Evaluation by moments, past and future"

    View full-size slide

  17. Negativity Bias
    Even when of equal intensity, things of a more negative
    nature have a greater effect on one's psychological state and
    processes than neutral or positive things.
    Baumeister, Roy F
    .; Finkenauer, Catrin; Vohs, Kathleen D. (2001). "Bad is stronger than good"

    View full-size slide

  18. Negativity Bias
    Even when of equal intensity, things of a more negative
    nature have a greater effect on one's psychological state and
    processes than neutral or positive things.
    ● The ratio or average of worst experience to best experience should have an
    impact on conversion rate.
    ● Active Listening can confound the results
    Baumeister, Roy F
    .; Finkenauer, Catrin; Vohs, Kathleen D. (2001). "Bad is stronger than good"

    View full-size slide

  19. Escalation of Commitment / Sunk Cost
    An individual or group facing increasingly negative
    outcomes continue the behavior instead of altering course.
    A greater tendency to continue an endeavor once an
    investment in money, effort, or time has been made.
    Staw, Barry M. (1997). "The escalation of commitment: An update and appraisal"
    Arkes, Hal R.; Ayton, Peter (1999). "The sunk cost and Concorde effects"

    View full-size slide

  20. Escalation of Commitment / Sunk Cost
    An individual or group facing increasingly negative
    outcomes continue the behavior instead of altering course.
    A greater tendency to continue an endeavor once an
    investment in money, effort, or time has been made.
    ● High session length for really bad performing sessions
    ● Retention/conversion rate increases as session length increases
    Staw, Barry M. (1997). "The escalation of commitment: An update and appraisal"
    Arkes, Hal R.; Ayton, Peter (1999). "The sunk cost and Concorde effects"

    View full-size slide

  21. Hypotheses…
    ● The most recent experience is
    very impactful.
    ● The best and/or worst
    experiences are impactful.
    ● The first experience may be
    impactful.
    ● The amount of time someone
    stays on the site is impactful.
    Pacific Islander Navigation Map, Museum of Fine Arts, Boston
    https://www.flickr.com/photos/bluesmoon/1266590108/

    View full-size slide

  22. Stats
    B
    reak
    Wikipedia found that a 4% temporary
    improvement to page load time resulted in an
    equally temporary 1% increase in user
    satisfaction.
    Wiki Research: Analyzing Wikipedia Users’ Perceived Quality Of Experience

    View full-size slide

  23. Detecting Bias
    Identifying Cognitive Biases in Browsing Data

    View full-size slide

  24. ● Collection: Real user performance data collected with boomerang
    ● Sessions: Anonymous session ID attached to continuous sessions;
    discarded after 30 minutes of inactivity. Limited to sessions of 30
    pages or fewer.
    ● Samples: Analysis was done across multiple websites with millions
    of data points each.
    ● Timers: We looked at Page Load Time (PLT), Time to Interactive
    (TTI) and Largest Contentful Paint (LCP) for Full Page as well as
    Single Page Apps.
    Notes about the Data

    View full-size slide

  25. First, Last, Fastest, Slowest
    ● There is a strong negative correlation between
    conversion rate and the performance of the first page.
    3.5% @ 1.8s
    0.8% @ 18s
    1.6% @ 9s

    View full-size slide

  26. First, Last, Fastest, Slowest
    ● There is a strong negative correlation between
    conversion rate and the performance of the first page.
    ● The last page distribution has a negative correlation
    and appears multimodal, but it’s a 0.2pp delta.
    3.5% @ 1.8s
    0.4% @ 18s
    1% @ 9s

    View full-size slide

  27. First, Last, Fastest, Slowest
    ● There is a strong negative correlation between
    conversion rate and the performance of the first page.
    ● The last page distribution has a negative correlation
    and appears multimodal, but it’s a 0.2pp delta.
    ● The fastest page has to be really fast.
    10.5% @ 500ms
    0.4% @ 9s
    1% @ 5s

    View full-size slide

  28. Conversions x First, Last, Fastest, Slowest
    ● There is a strong negative correlation between
    conversion rate and the performance of the first page.
    ● The last page distribution has a negative correlation
    and appears multimodal, but it’s a 0.2pp delta.
    ● The fastest page has to be really fast. Too slow, and
    users bounce.
    ● Correlation with the slowest page is a little weird…

    View full-size slide

  29. 0.5% @ 1s
    3.4% @ 19s
    3% @ 5s
    ● It seems that conversions increase as performance gets worse
    ● It turns out that a slow experience is part of the conversion
    flow.
    ● The low conversion rate on the left is a result of bounces.
    Very fast pages are typically caused by JavaScript errors
    resulting in a mostly blank page.
    (we see the same when the fastest page is under 100ms)
    Is Slower Better?

    View full-size slide

  30. looking at the 2nd Slowest Instead…
    0.5% @ 1s
    1% @ 19s
    3.5% @ 2.8s
    1.9% @ 6s
    1.1% @ 12s

    View full-size slide

  31. Conversions x First, Last, Fastest, Slowest
    ● There is a strong negative correlation between
    conversion rate and the performance of the first page.
    ● The last page distribution has a negative correlation
    and appears multimodal, but it’s a 0.2pp delta.
    ● The fastest page has to be really fast. Too slow, and
    users bounce.
    ● The slowest page doesn’t matter, but you cannot have
    too many slow pages.

    View full-size slide

  32. Retention Rate x First, Last, Fastest, Slowest
    ● Retention Rate of a page varies based on the page.
    ● For Homepages and other Landing pages, the performance of the first page
    appears to be the biggest indicator of retention.
    ● For Product Detail, Category, and Search Results Pages, it’s a combination of
    the fastest & latest, and sometimes the first page.
    ● The worst and second worst performing pages do not have an impact on
    retention.

    View full-size slide

  33. Negativity Bias
    ● To determine if negativity bias is in play, we
    look at combinations of the best and 2nd
    worst performing pages.
    ● The ratio (worst/best) has a strong negative
    correlation with conversions.
    ● The geometric mean has a high, narrow peak.
    ● A heatmap shows low tolerance for deviations
    in the fastest load time, and inverse
    dependence between the fastest and slowest
    times.
    Ratio of Slowest to Fastest
    Geometric MEAN of Slowest & Fastest
    Fastest →
    ← Slowest
    1
    10
    20
    30
    40
    50
    60
    0 10 20 30 40 50
    0 0.4 0.7 1.0 1.3 1.6 1.9 2.2 2.5

    View full-size slide

  34. Uhh… What does all of that
    mean?

    View full-size slide

  35. Negativity Bias
    1
    10
    20
    30
    40
    50
    60
    0 0.4 0.7 1.0 1.3 1.6 1.9 2.2 2.5
    ● We have a practical lower bound on the fastest page
    ● We have a tolerable upper bound on the fastest page
    ● Slow pages are tolerated only when paired with a fast
    page that’s at least 15x faster.
    ● This results in an upper bound on the slowest page.
    Fastest →
    0 10 20 30 40 50
    ← Slowest

    View full-size slide

  36. A greater tendency to continue an
    endeavor once an investment in money,
    effort, or time has been made.
    Escalation of Commitment / Sunk Cost

    View full-size slide

  37. A greater tendency to continue an
    endeavor once an investment in money,
    effort, or time has been made.
    29% after 30
    0.6% after 5
    7.1% after 10
    21% after 20
    Pages ->
    Conversion Rate ->
    Escalation of Commitment / Sunk Cost

    View full-size slide

  38. Looking across Load Times…
    1
    10
    20
    30
    40
    50
    60
    70
    80
    90
    100
    110
    115
    0 10 20 30 40 50
    0.1s 2s 4s 6s 8s 10s 15s 20s 25s 30s

    View full-size slide

  39. Stats
    B
    reak
    The average rise in mobile users' heart rates caused
    by delayed web pages — equivalent to the anxiety
    of watching a horror movie alone.
    Ericsson ConsumerLab neuro research 2015
    38%

    View full-size slide

  40. Accounting for Bias
    What do we do with this knowledge?

    View full-size slide

  41. Focus performance improvements on
    a few key pages.

    View full-size slide

  42. The performance of the first page
    affects bounces.

    View full-size slide

  43. The performance of the fastest page
    and last page affects retention.

    View full-size slide

  44. The slowest page in a session should
    be no more than 15x the latency of
    the fastest page.

    View full-size slide

  45. Acknowledging when you didn’t meet the
    user’s expectations can alleviate negative
    perceptions.
    Practice Active Listening
    https://affect.media.mit.edu/pdfs/02.klein-moon-picard.pdf
    https://uxdesign.cc/the-fastest-way-to-pinpoint-frustrating-user-experiences-1f8b95bc94aa
    https://doi.org/10.1016/j.ijhcs.2004.01.002
    https://www.sciencedirect.com/science/article/abs/pii/S1071581904000060?via%3Dihub

    View full-size slide

  46. A fast page increase pages per session
    which in turn increase the likelihood
    of a conversion.

    View full-size slide

  47. Stats
    B
    reak
    Users are most patient when using the
    web from the office and least patient
    when using their phones.
    Median Lethal Frustration Index study in mPulse data

    View full-size slide

  48. Developer Bias
    Biases when studying the data

    View full-size slide

  49. https://upload.wikimedia.org/wikipedia/commons/6/65/Cognitive_bias_codex_en.svg

    View full-size slide

  50. Cognitive Biases – Developer Edition
    ● Amdahl's Law
    Assuming every millisecond is the same.
    ● Outcome Bias
    Choosing data that confirms past outcomes.
    ● Survivorship Bias
    Assuming what we’ve measured is all there is.
    ● Selection Bias
    Choosing dimensions based on our instincts.
    ● Pareidolia
    Preferring data that renders interesting shapes.
    ● Insensitivity to Sample Size
    Forgetting that smaller samples have larger variance.
    ● Clustering Illusion
    Seeing patterns in small samples where none exist.
    ● Confirmation Bias
    Choosing data that confirms our pre-existing beliefs.

    View full-size slide

  51. Ignoring Amdahl’s Law
    You may have read reports that say something like:
    “every 100ms decrease in homepage load
    time worked out to a 1% increase in
    conversion”
    Citation redacted to protect the innocent

    View full-size slide

  52. Survivorship Bias
    ● In 2012, Youtube made their site lighter but aggregate
    performance got worse.
    ● It turns out that new users who previously could not access the site
    were now coming in at the long tail.
    ● The site appeared slower in aggregate, but the number of users
    who could use it had gone up.
    Chris Zacharias: Page Weight Matters.

    View full-size slide

  53. Insensitivity To Sample Size
    We often get questions like:
    “Why is performance on tablets worse than
    performance on mobile devices?”
    It turns out that mobile generally has 50x the amount of traffic than tablets.
    That results in far less variance in the data.
    A customer recently asked me this question.

    View full-size slide

  54. Anscombe’s Quartet
    Anscombe's Quartet Frank Anscombe
    Plot of Anscombe's Quartet by Schutz & Avenue
    ● 4 data sets with the same summary statistics:
    ○ 𝜇
    x
    = 9, 𝜇
    y
    = 7.5
    ○ s
    x
    2 = 11, s
    y
    2 = 4.125
    ○ 𝜌
    x,y
    = 0.816
    ○ Linear Regression Line: y=3
    ○ ℝ2 = 0.67
    ● Anscombe’s Quartet shows us why it’s
    important to visualize data and not just look
    at summary stats

    View full-size slide

  55. End
    B
    reak
    You have been an awesome audience!
    Thank you!

    View full-size slide

  56. References
    ● Ebbinghaus, Hermann (1913). On memory: A contribution
    to experimental psychology
    ● Kahneman, Daniel (2000). "Evaluation by moments, past
    and future"
    ● Baumeister, Roy F
    .; Finkenauer, Catrin; Vohs, Kathleen D.
    (2001). "Bad is stronger than good"
    ● Staw, Barry M. (1997). "The escalation of commitment: An
    update and appraisal"
    ● Arkes, Hal R.; Ayton, Peter (1999). "The sunk cost and
    Concorde effects"
    ● The impact of network speed on emotional engagement
    ● Ericsson ConsumerLab neuro research 2015
    ● Wikipedia Paper on User Satisfaction v/s Performance
    ● Toward a more civilized design: studying the effects of
    computers that apologize
    ● The fastest way to pinpoint frustrating user experiences
    ● Serial-position effect
    ● Peak–end rule
    ● Negativity bias
    ● Escalation of commitment / Sunk cost
    ● Levels-of-processing
    ● Amdahl's Law
    ● Outcome Bias
    ● Survivorship Bias
    ● Selection Bias
    ● Pareidolia
    ● Insensitivity to Sample Size
    ● Clustering Illusion
    ● Confirmation Bias
    ● Time Saving Bias

    View full-size slide