Sample 1 Probability Analysis Sample 2 alpha = 0.05 = 1 - 0.95 “Given that I have to live with the probability to observe the sample, the success rate on such observation must be 95% or greater”.
the shape of test (tails) Calculate the statistical score Compute the p-value Compare with signiﬁcance level Reject H0 and accept Ha Can’t reject H0 Smaller Greater
provided as result of two independent benchmarks and ﬁgure out if we have an evidence strong enough in order to demonstrate that some change actually improves our build time
again Compare measurements Let's do a statistical test for the difference of means calculated after two benchmarks and ﬁgure out if we have an evidence strong enough in order to demonstrate that some change actually improves our build time
again Compare measurements H0 : the mean after the improvement applied is statistically the same versus HA : the mean after the improvement applied is statistically lower than before