Share
Embed
Transcript
45 slides
-
:: also known as “Split Testing” a method of measuring
the outcome of two versions of an action against each other to determine which one performs better. What is A/B Testing? -
Specifically in the world of websites and applications, A/B testing
is an experiment where two or more variants of a page are shown to users at random, and statistical analysis is used to determine which variation performs better for a given conversion goal. What is A/B Testing? -
To A/B test UI, we randomly send half of the
users to the control UI and the other half of random users to the variation UI What is A/B Testing? -
Key Detail randomly segmenting users to either the control or
the variation is vitally important. it must be actually random -
FAQ no, but you are significantly increasing volume of data
you will need in order to see a statistically significant change in the data do the population sizes have to be 50/50? -
A/B Testing is also a form of statistical analysis called
hypothesis testing. What is hypothesis testing? What is A/B Testing? -
What is A/B Testing? What is hypothesis testing? Hypothesis testing
is the use of statistics to determine the probability that a given hypothesis is true. The usual process of hypothesis testing consists of four steps. 1. Formulate the null hypothesis 2. Identify a test statistic that can be used to assess the truth of they null hypothesis 3. Compute a p-value (probability that a test statistic at least as signi fi cant as the one observed would be obtained assuming the null hypothesis is true) 4. Compare p-value to acceptable signi fi cance value (a). if p <= a, null hypothesis is ruled out and alternative hypothesis is value -
what is a null hypothesis? the hypothesis that given statistical
significance you would get the same results not changing the test as you would if you changed it and that any variation is from random chance. basically: nothing special is happening here What is A/B Testing? -
what is the alternative hypothesis? the contrary to the null
hypothesis. if proven, the results are from a real effect and not purely from random chance. basically: what we changed made an real difference in the results What is A/B Testing? -
Test & Control i always hear “Test & Control”. What
does this even mean? -
What does this even mean? Test is A Control is
B A/B Testing == Test/Control Testing Test & Control -
“Test & Control” What is a control? the original version
before you made a change -
“Test & Control” What is the variation? the modification of
the original version -
FAQ Yes. That is called multivariate testing and often referred
to as A/B/n testing. However you are significantly increasing the complexity of the math if you do this. A/B is two things. Can I test more than two -
Control Variation Add to Cart hypothesis: a purchase CTA with
a green background will have higher engagement than one with a blue background. Add to Cart -
Control Variation Add to Cart Add to Cart hypothesis: a
purchase CTA with a green background will have higher engagement than one with a blue background. -
Control Variation Lift Add to Cart Add to Cart hypothesis:
a purchase CTA with a green background will have higher engagement than one with a blue background. -
Control Variation Lift Add to Cart Add to Cart hypothesis:
a purchase CTA with a green background will have higher engagement than one with a blue background. PROVEN TRUE! 😄 -
Control Variation Lift Add to Cart Add to Cart hypothesis:
a purchase CTA with a green background will have higher engagement than one with a blue background. PROVEN TRUE?? or is it? 😰 -
Key Detail statistical signi fi cance is important If you
make an assertion with statistically insigni fi cant data, you are not proving anything. In fact, you could be communicating something that makes the situation worse than it is without change. -
what makes something statistically significant? Statistical signi fi cance is
the likelihood that the difference in measurements between a given variation and the baseline is not due to random chance. A result of an experiment is said to have statistical signi fi cance, or be statistically signi fi cant, if it is likely not caused by chance for a given statistical signi fi cance level. Statistical Significance -
If you rolled a double six, would you bet $1000
on rolling it again on the next try? -
1/6 1/6 If you rolled a double six, would you
bet $1000 on rolling it again on the next try? -
1/6 1/6 1/6 x 1/6 = 1/36 chance If you
rolled a double six, would you bet $1000 on rolling it again on the next try? -
1/6 1/6 1/6 x 1/6 = 1/36 chance If you
rolled a double six, would you bet $1000 on rolling it again on the next try? -
no, you wouldn’t. so why would you tell a client
75% of users were female if you only had 20 page views? -
that’s great addam… but i’m not teaching statistics at university
for a reason. what am i supposed to do? Statistical Significance -
could you repeat that? make a hypothesis i think X
will do better than Y test one thing at a time i want to test changing the text and the color, but I’ll start with color. verify the samples are random ensure there’s no confirmation bias watch your sample size use the a/b test calculators online to determine optimal sample size verify statistical significance make sure you are not using stats to tell mathematically accurate lies iterate on the next hypothesis! it doesn’t stop after you’ve proven your first test. Use those results to make a new hypothesis and iterate.