Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using Analytics to Improve UX

Paul McMahon
May 01, 2014

Using Analytics to Improve UX

Paul McMahon

May 01, 2014


  1. New Relic • Tells us if our site is up

    • Identifies slow response times
  2. Sentry • Monitors application errors such as crashes • Less

    than 1% of users will report errors • Record user id if possible so you can follow up with the user after
  3. Crazy Egg • Creates a heatmap of where users click

    on your page • Can be useful in identifying things that should be clickable, but aren't
  4. Kiss Metrics • Record arbitrary events like “View Event Page”,

    “Start Registration”, and “Complete Registration” • Set arbitrary properties like “Subscription Plan” on events/users
  5. Funnel Reports 
 (Kiss Metrics) • Track how users move

    through a series of steps towards a goal • Identify where people drop out, which gives you of ideas where you can focus UX effort • Track your changes over time
  6. Measuring Improvements • Using data, we can come up with

    hypothesizes for UX improvements • Let’s imagine we are trying to improve Doorkeeper’s event page Existing Experiment
  7. A/B Testing • Put each visitor into a control or

    experimental group randomly • Show existing version to control group,
  8. The larger your sample size, the smaller the improvement you

    can measure • If we have 2000 people visit our event page, one version needs to outperform the other by 25% to be statistically significant • With 20000 people, it drops to 8% • Hypothesize appropriate sample size before running experiment
  9. Measure conversion rate to the goal • Make sure you

    measure the conversion rate of what is actually your goal. • If you have a multi-step funnel, optimizations can increase the first step, but still lead to a lower overall conversion.
  10. Read More • A/B testing is powerful but easy to

    abuse if you don’t understand the statistics behind it • See Jason Cohen’s Business of Software Conference talk “Why data will deceive you”.