Understanding Cognitive Biases in Performance Measurement Finding the factors that lead to abandonment

Philip Tellis Principal RUM Distiller @ Akamai ● Analyses real user performance data from mPulse ● Author of the OpenSource boomerang RUM library twitter:@bluesmoon ⦿ github:@bluesmoon speakerdeck:@bluesmoon

BIAS is an Expectation Our Journey Today... ★ Understanding Cognitive Biases ★ Signs of cognitive biases in browsing data ★ What can we do?

Understanding Bias Good, Bad, Normal?

Similarity Zero-Risk False Memory Expedience Experience Proximity Survivorship Negativity Safety Loss Aversion If you have a brain, you have bias.

Bias stems from experience – It’s Normal ● Helps us learn Perceptual/Sensory Dissonance ● Keeps us safe Safety Bias, Loss Aversion, Negativity Bias ● Find our people Similarity Bias, Proximity Bias Boston Shipyard Artist’s Community

https://upload.wikimedia.org/wikipedia/commons/6/65/Cognitive_bias_codex_en.svg

Cognitive Biases ● Similarity Bias ● Expedience Bias ● Experience Bias ● Proximity Bias ● Safety Bias ● Serial-position effect ● False memory ● Duration neglect ● Peak–end rule ● Negativity bias ● Escalation of commitment ● Loss aversion ● Zero-risk bias ● Next-in-line effect ● Misattribution of memory ● Sunk cost ● Levels-of-processing ● Spacing effect

Cognitive Biases - Related to Performance on the Web ● Similarity Bias ● Expedience Bias ● Experience Bias ● Proximity Bias ● Safety Bias ● Serial-position effect ● False memory ● Duration neglect ● Peak–end rule ● Negativity bias ● Escalation of commitment ● Loss aversion ● Zero-risk bias ● Next-in-line effect ● Misattribution of memory ● Sunk cost ● Levels-of-processing ● Spacing effect

Cognitive Biases - This Talk ● Similarity Bias ● Expedience Bias ● Experience Bias ● Proximity Bias ● Safety Bias ● Serial-position effect ● False memory ● Duration neglect ● Peak–end rule ● Negativity bias ● Escalation of commitment ● Loss aversion ● Zero-risk bias ● Next-in-line effect ● Misattribution of memory ● Sunk cost ● Levels-of-processing ● Spacing effect

Stats B reak A 500ms connection speed delay resulted in up to a 26% increase in peak frustration and up to an 8% decrease in engagement. Tammy Everts – The impact of network speed on emotional engagement

some definitions Bounce Rate: Percentage of users on the site who leave after viewing one page. Retention Rate: Percentage of users on a particular page who remain on the site for at least one more page view. Conversion Rate: Percentage of users on the site who complete a goal or particular task. Goal: A task like a conversion, purchase, visiting a particular page, or viewing a certain number of pages.

Serial-Position Effect …is the tendency of a person to recall the first and last items in a series best, and the middle items worst. Ebbinghaus, Hermann (1913). On memory: A contribution to experimental psychology

Serial-Position Effect …is the tendency of a person to recall the first and last items in a series best, and the middle items worst. ● Retention rate might be a function of the first and latest pages ● The recency effect suggests that the latest page has a higher weight Ebbinghaus, Hermann (1913). On memory: A contribution to experimental psychology

Peak-End Rule People judge an experience largely based on how they felt at its peak & at its end, rather than the sum or average of every moment of the experience. Kahneman, Daniel (2000). "Evaluation by moments, past and future"

People judge an experience largely based on how they felt at its peak & at its end, rather than the sum or average of every moment of the experience. ● Retention rate depends on the best/worst and latest performance ● Conversion rate depends on the best/worst performance and that of the page just before the conversion Peak-End Rule Kahneman, Daniel (2000). "Evaluation by moments, past and future"

Negativity Bias Even when of equal intensity, things of a more negative nature have a greater effect on one's psychological state and processes than neutral or positive things. Baumeister, Roy F .; Finkenauer, Catrin; Vohs, Kathleen D. (2001). "Bad is stronger than good"

Negativity Bias Even when of equal intensity, things of a more negative nature have a greater effect on one's psychological state and processes than neutral or positive things. ● The ratio or average of worst experience to best experience should have an impact on conversion rate. ● Active Listening can confound the results Baumeister, Roy F .; Finkenauer, Catrin; Vohs, Kathleen D. (2001). "Bad is stronger than good"

Escalation of Commitment / Sunk Cost An individual or group facing increasingly negative outcomes continue the behavior instead of altering course. A greater tendency to continue an endeavor once an investment in money, effort, or time has been made. Staw, Barry M. (1997). "The escalation of commitment: An update and appraisal" Arkes, Hal R.; Ayton, Peter (1999). "The sunk cost and Concorde effects"

Escalation of Commitment / Sunk Cost An individual or group facing increasingly negative outcomes continue the behavior instead of altering course. A greater tendency to continue an endeavor once an investment in money, effort, or time has been made. ● High session length for really bad performing sessions ● Retention/conversion rate increases as session length increases Staw, Barry M. (1997). "The escalation of commitment: An update and appraisal" Arkes, Hal R.; Ayton, Peter (1999). "The sunk cost and Concorde effects"

Hypotheses… ● The most recent experience is very impactful. ● The best and/or worst experiences are impactful. ● The first experience may be impactful. ● The amount of time someone stays on the site is impactful. Pacific Islander Navigation Map, Museum of Fine Arts, Boston https://www.flickr.com/photos/bluesmoon/1266590108/

Stats B reak Wikipedia found that a 4% temporary improvement to page load time resulted in an equally temporary 1% increase in user satisfaction. Wiki Research: Analyzing Wikipedia Users’ Perceived Quality Of Experience

Detecting Bias Identifying Cognitive Biases in Browsing Data

● Collection: Real user performance data collected with boomerang ● Sessions: Anonymous session ID attached to continuous sessions; discarded after 30 minutes of inactivity. Limited to sessions of 30 pages or fewer. ● Samples: Analysis was done across multiple websites with millions of data points each. ● Timers: We looked at Page Load Time (PLT), Time to Interactive (TTI) and Largest Contentful Paint (LCP) for Full Page as well as Single Page Apps. Notes about the Data

First, Last, Fastest, Slowest ● There is a strong negative correlation between conversion rate and the performance of the first page. 3.5% @ 1.8s 0.8% @ 18s 1.6% @ 9s

First, Last, Fastest, Slowest ● There is a strong negative correlation between conversion rate and the performance of the first page. ● The last page distribution has a negative correlation and appears multimodal, but it’s a 0.2pp delta. 3.5% @ 1.8s 0.4% @ 18s 1% @ 9s

First, Last, Fastest, Slowest ● There is a strong negative correlation between conversion rate and the performance of the first page. ● The last page distribution has a negative correlation and appears multimodal, but it’s a 0.2pp delta. ● The fastest page has to be really fast. 10.5% @ 500ms 0.4% @ 9s 1% @ 5s

Conversions x First, Last, Fastest, Slowest ● There is a strong negative correlation between conversion rate and the performance of the first page. ● The last page distribution has a negative correlation and appears multimodal, but it’s a 0.2pp delta. ● The fastest page has to be really fast. Too slow, and users bounce. ● Correlation with the slowest page is a little weird…

0.5% @ 1s 3.4% @ 19s 3% @ 5s ● It seems that conversions increase as performance gets worse ● It turns out that a slow experience is part of the conversion flow. ● The low conversion rate on the left is a result of bounces. Very fast pages are typically caused by JavaScript errors resulting in a mostly blank page. (we see the same when the fastest page is under 100ms) Is Slower Better?

looking at the 2nd Slowest Instead… 0.5% @ 1s 1% @ 19s 3.5% @ 2.8s 1.9% @ 6s 1.1% @ 12s

Conversions x First, Last, Fastest, Slowest ● There is a strong negative correlation between conversion rate and the performance of the first page. ● The last page distribution has a negative correlation and appears multimodal, but it’s a 0.2pp delta. ● The fastest page has to be really fast. Too slow, and users bounce. ● The slowest page doesn’t matter, but you cannot have too many slow pages.

Retention Rate x First, Last, Fastest, Slowest ● Retention Rate of a page varies based on the page. ● For Homepages and other Landing pages, the performance of the first page appears to be the biggest indicator of retention. ● For Product Detail, Category, and Search Results Pages, it’s a combination of the fastest & latest, and sometimes the first page. ● The worst and second worst performing pages do not have an impact on retention.

Negativity Bias ● To determine if negativity bias is in play, we look at combinations of the best and 2nd worst performing pages. ● The ratio (worst/best) has a strong negative correlation with conversions. ● The geometric mean has a high, narrow peak. ● A heatmap shows low tolerance for deviations in the fastest load time, and inverse dependence between the fastest and slowest times. Ratio of Slowest to Fastest Geometric MEAN of Slowest & Fastest Fastest → ← Slowest 1 10 20 30 40 50 60 0 10 20 30 40 50 0 0.4 0.7 1.0 1.3 1.6 1.9 2.2 2.5

Uhh… What does all of that mean?

Negativity Bias 1 10 20 30 40 50 60 0 0.4 0.7 1.0 1.3 1.6 1.9 2.2 2.5 ● We have a practical lower bound on the fastest page ● We have a tolerable upper bound on the fastest page ● Slow pages are tolerated only when paired with a fast page that’s at least 15x faster. ● This results in an upper bound on the slowest page. Fastest → 0 10 20 30 40 50 ← Slowest

A greater tendency to continue an endeavor once an investment in money, effort, or time has been made. Escalation of Commitment / Sunk Cost

A greater tendency to continue an endeavor once an investment in money, effort, or time has been made. 29% after 30 0.6% after 5 7.1% after 10 21% after 20 Pages -> Conversion Rate -> Escalation of Commitment / Sunk Cost

Looking across Load Times… 1 10 20 30 40 50 60 70 80 90 100 110 115 0 10 20 30 40 50 0.1s 2s 4s 6s 8s 10s 15s 20s 25s 30s

Stats B reak The average rise in mobile users' heart rates caused by delayed web pages — equivalent to the anxiety of watching a horror movie alone. Ericsson ConsumerLab neuro research 2015 38%

Accounting for Bias What do we do with this knowledge?

Focus performance improvements on a few key pages.

The performance of the first page affects bounces.

The performance of the fastest page and last page affects retention.

The slowest page in a session should be no more than 15x the latency of the fastest page.

Acknowledging when you didn’t meet the user’s expectations can alleviate negative perceptions. Practice Active Listening https://affect.media.mit.edu/pdfs/02.klein-moon-picard.pdf https://uxdesign.cc/the-fastest-way-to-pinpoint-frustrating-user-experiences-1f8b95bc94aa https://doi.org/10.1016/j.ijhcs.2004.01.002 https://www.sciencedirect.com/science/article/abs/pii/S1071581904000060?via%3Dihub

A fast page increase pages per session which in turn increase the likelihood of a conversion.

Stats B reak Users are most patient when using the web from the office and least patient when using their phones. Median Lethal Frustration Index study in mPulse data

Developer Bias Biases when studying the data

https://upload.wikimedia.org/wikipedia/commons/6/65/Cognitive_bias_codex_en.svg

Cognitive Biases – Developer Edition ● Amdahl's Law Assuming every millisecond is the same. ● Outcome Bias Choosing data that confirms past outcomes. ● Survivorship Bias Assuming what we’ve measured is all there is. ● Selection Bias Choosing dimensions based on our instincts. ● Pareidolia Preferring data that renders interesting shapes. ● Insensitivity to Sample Size Forgetting that smaller samples have larger variance. ● Clustering Illusion Seeing patterns in small samples where none exist. ● Confirmation Bias Choosing data that confirms our pre-existing beliefs.

Ignoring Amdahl’s Law You may have read reports that say something like: “every 100ms decrease in homepage load time worked out to a 1% increase in conversion” Citation redacted to protect the innocent

Survivorship Bias ● In 2012, Youtube made their site lighter but aggregate performance got worse. ● It turns out that new users who previously could not access the site were now coming in at the long tail. ● The site appeared slower in aggregate, but the number of users who could use it had gone up. Chris Zacharias: Page Weight Matters.

Insensitivity To Sample Size We often get questions like: “Why is performance on tablets worse than performance on mobile devices?” It turns out that mobile generally has 50x the amount of traffic than tablets. That results in far less variance in the data. A customer recently asked me this question.

Anscombe’s Quartet Anscombe's Quartet Frank Anscombe Plot of Anscombe's Quartet by Schutz & Avenue ● 4 data sets with the same summary statistics: ○ 𝜇 x = 9, 𝜇 y = 7.5 ○ s x 2 = 11, s y 2 = 4.125 ○ 𝜌 x,y = 0.816 ○ Linear Regression Line: y=3 ○ ℝ2 = 0.67 ● Anscombe’s Quartet shows us why it’s important to visualize data and not just look at summary stats

End B reak You have been an awesome audience! Thank you!

References ● Ebbinghaus, Hermann (1913). On memory: A contribution to experimental psychology ● Kahneman, Daniel (2000). "Evaluation by moments, past and future" ● Baumeister, Roy F .; Finkenauer, Catrin; Vohs, Kathleen D. (2001). "Bad is stronger than good" ● Staw, Barry M. (1997). "The escalation of commitment: An update and appraisal" ● Arkes, Hal R.; Ayton, Peter (1999). "The sunk cost and Concorde effects" ● The impact of network speed on emotional engagement ● Ericsson ConsumerLab neuro research 2015 ● Wikipedia Paper on User Satisfaction v/s Performance ● Toward a more civilized design: studying the effects of computers that apologize ● The fastest way to pinpoint frustrating user experiences ● Serial-position effect ● Peak–end rule ● Negativity bias ● Escalation of commitment / Sunk cost ● Levels-of-processing ● Amdahl's Law ● Outcome Bias ● Survivorship Bias ● Selection Bias ● Pareidolia ● Insensitivity to Sample Size ● Clustering Illusion ● Confirmation Bias ● Time Saving Bias