Using Analytics to Improve UX Paul McMahon @pwim

New Relic • Tells us if our site is up • Identifies slow response times

Sentry • Monitors application errors such as crashes • Less than 1% of users will report errors • Record user id if possible so you can follow up with the user after

Crazy Egg • Creates a heatmap of where users click on your page • Can be useful in identifying things that should be clickable, but aren't

Kiss Metrics • Record arbitrary events like “View Event Page”, “Start Registration”, and “Complete Registration” • Set arbitrary properties like “Subscription Plan” on events/users

Funnel Reports 
 (Kiss Metrics) • Track how users move through a series of steps towards a goal • Identify where people drop out, which gives you of ideas where you can focus UX effort • Track your changes over time

Measuring Improvements • Using data, we can come up with hypothesizes for UX improvements • Let’s imagine we are trying to improve Doorkeeper’s event page Existing Experiment

A/B Testing • Put each visitor into a control or experimental group randomly • Show existing version to control group,

The larger your sample size, the smaller the improvement you can measure • If we have 2000 people visit our event page, one version needs to outperform the other by 25% to be statistically significant • With 20000 people, it drops to 8% • Hypothesize appropriate sample size before running experiment

Measure conversion rate to the goal • Make sure you measure the conversion rate of what is actually your goal. • If you have a multi-step funnel, optimizations can increase the first step, but still lead to a lower overall conversion.

Read More • A/B testing is powerful but easy to abuse if you don’t understand the statistics behind it • See Jason Cohen’s Business of Software Conference talk “Why data will deceive you”.