"Is This Normal?" - Clearhead Opticon 2014 Presentation
A presentation by Clearhead co-founder, Ryan Garner, and our client, Jessica Vasbinder, from Warner Music Group, about the very real and very normal challenges of building a successful A/B testing program.
expectations. – You may (probably will) NOT materially improve your bottom line. ! • Risk (Disruption) & Reward (Lift) are correlated. – Small tests often provide small returns…especially for smaller businesses!
“No, start at the top of the funnel.” • “Test the whole funnel simultaneously.” • “NO, Find the bottlenecks IN the funnel.” • “Start with calls to action…copy, color, size.” • “Navigation is always a good place to test.” • “Forms, baby, forms.”
(hypotheses). • Invest significantly in their development. • Make the process open and accessible. • Store ideas and knowledge where the tribe can find it.
to the top. – it should need no real explanation. – no new acronyms! – Why? No hurt feelings & better ideas!! ! • Then use your own judgment. ! • Repeat regularly.
afford. – Can you? – How much traffic do you have? ! • Assuming limited resources and limited traffic, test the bigger ideas. – Don’t be afraid to bundle changes. (there - we said it) ! • If it grows the business, it grows the business.
you are pushing new code onto your production website. – Should just anyone be doing this? ! • Do you really want a high velocity, high returning optimization program? Of course you do! – You need your best developers doing the coding and testing. Not your marketers. – Even then, expect hiccups. ! • Monitor, monitor, monitor.. – Test monitoring is critical, especially the first 24 hours following a launch. – Data usually has a story. If yours makes absolutely no sense, your test could be broken.
• Stress the difference between statistical significance and observed lift. ! • Trust that there is improvement and move on. – The only accurate way to confirm lift is to re-test. – Do you really want to do that? ! • And if you must show something… – Show the full range of possibilities. – Add lots of these “****”.