A B 

:: also known as “Split Testing” a method of measuring the outcome of two versions of an action against each other to determine which one performs better. What is A/B Testing? 

Specifically in the world of websites and applications, A/B testing is an experiment where two or more variants of a page are shown to users at random, and statistical analysis is used to determine which variation performs better for a given conversion goal. What is A/B Testing? 

No content

To A/B test UI, we randomly send half of the users to the control UI and the other half of random users to the variation UI What is A/B Testing? 

Key Detail randomly segmenting users to either the control or the variation is vitally important. it must be actually random 

FAQ no, but you are significantly increasing volume of data you will need in order to see a statistically significant change in the data do the population sizes have to be 50/50? 

A/B Testing is also a form of statistical analysis called hypothesis testing. What is hypothesis testing? What is A/B Testing? 

What is A/B Testing? What is hypothesis testing? Hypothesis testing is the use of statistics to determine the probability that a given hypothesis is true. The usual process of hypothesis testing consists of four steps. 1. Formulate the null hypothesis 2. Identify a test statistic that can be used to assess the truth of they null hypothesis 3. Compute a p-value (probability that a test statistic at least as signi fi cant as the one observed would be obtained assuming the null hypothesis is true) 4. Compare p-value to acceptable signi fi cance value (a). if p <= a, null hypothesis is ruled out and alternative hypothesis is value 

what is a null hypothesis? What is A/B Testing? 

what is a null hypothesis? the hypothesis that given statistical significance you would get the same results not changing the test as you would if you changed it and that any variation is from random chance. basically: nothing special is happening here What is A/B Testing? 

what is the alternative hypothesis? What is A/B Testing? 

what is the alternative hypothesis? the contrary to the null hypothesis. if proven, the results are from a real effect and not purely from random chance. basically: what we changed made an real difference in the results What is A/B Testing? 

Test & Control i always hear “Test & Control”. What does this even mean? 

What does this even mean? Test is A Control is B A/B Testing == Test/Control Testing Test & Control 

“Test & Control” What is a control? the original version before you made a change 

“Test & Control” What is the variation? the modification of the original version 

FAQ Yes. That is called multivariate testing and often referred to as A/B/n testing. However you are significantly increasing the complexity of the math if you do this. A/B is two things. Can I test more than two 

No content

normal measurement of engagement Add to Cart 

Control Add to Cart 

Control Variation Add to Cart BUY NOW! 

WAIT what’s wrong with this test? Add to Cart BUY NOW! 

Key Detail test one variable 

Control Variation Add to Cart what is our Add to Cart 

Key Detail you must make a hypothesis 

Control Variation Add to Cart hypothesis: a purchase CTA with a green background will have higher engagement than one with a blue background. Add to Cart 

Control Variation Add to Cart Add to Cart hypothesis: a purchase CTA with a green background will have higher engagement than one with a blue background. 

Control Variation Lift Add to Cart Add to Cart hypothesis: a purchase CTA with a green background will have higher engagement than one with a blue background. 

Control Variation Lift Add to Cart Add to Cart hypothesis: a purchase CTA with a green background will have higher engagement than one with a blue background. PROVEN TRUE! 😄 

Control Variation Lift Add to Cart Add to Cart hypothesis: a purchase CTA with a green background will have higher engagement than one with a blue background. PROVEN TRUE?? or is it? 😰 

WHY? 

Key Detail statistical signi fi cance is important If you make an assertion with statistically insigni fi cant data, you are not proving anything. In fact, you could be communicating something that makes the situation worse than it is without change. 

what makes something statistically significant? Statistical Significance 

what makes something statistically significant? Statistical signi fi cance is the likelihood that the difference in measurements between a given variation and the baseline is not due to random chance. A result of an experiment is said to have statistical signi fi cance, or be statistically signi fi cant, if it is likely not caused by chance for a given statistical signi fi cance level. Statistical Significance 

If you rolled a double six, would you bet $1000 on rolling it again on the next try? 

1/6 1/6 If you rolled a double six, would you bet $1000 on rolling it again on the next try? 

1/6 1/6 1/6 x 1/6 = 1/36 chance If you rolled a double six, would you bet $1000 on rolling it again on the next try? 

No content

1/6 1/6 1/6 x 1/6 = 1/36 chance If you rolled a double six, would you bet $1000 on rolling it again on the next try? 

no, you wouldn’t. 

no, you wouldn’t. so why would you tell a client 75% of users were female if you only had 20 page views? 

that’s great addam… but i’m not teaching statistics at university for a reason. what am i supposed to do? Statistical Significance 

Useful Calculators https://www.evanmiller.org/ab-testing/sample-size.html http://www.abtestcalculator.com Statistical Significance 

could you repeat that? make a hypothesis i think X will do better than Y test one thing at a time i want to test changing the text and the color, but I’ll start with color. verify the samples are random ensure there’s no confirmation bias watch your sample size use the a/b test calculators online to determine optimal sample size verify statistical significance make sure you are not using stats to tell mathematically accurate lies iterate on the next hypothesis! it doesn’t stop after you’ve proven your first test. Use those results to make a new hypothesis and iterate.