Why Ben s Vanity for running A/B Tests Why The Pieces: Metrics & Experiments The Dashboard 30 minutes to our first experiment

Why: It’s Ruby and runs deep in the application Track and experiment with more than pageviews (e.g. validation errors, pageflows, changes to multiple pages or templates ) Easy for engineers to configure (double-edge sword) Store experiments in source code Ensure variations do not break things using standard CI flow before it gets to production (integrates with Rspec)

The pieces Metrics the objectives you want to influence Experiments the variants/changes you want to evaluate

The pieces: metrics - Define a metric with Ruby - Can be explicitly called (track!) or use existing ActiveRecord objects

The pieces: experiments - Choose a metric to influence - Define a list of variations - Define a success condition

The pieces: experiments (cont) - Branch your code

The Dashboard

30 minutes to our first experiment 1. Determine a measurable objective to influence 2. Create a Metric 3. Instrument the metric (if necessary) 4. Create an Experiment 5. Define the variants 6. Update tests to run through multiple variants (also helps inform resilient tests) 7.