What Users Want, A/B testing explained [Devoxx Ukraine 2018]

@devoxxua @VladimirD_42 What Users Want A/B testing explained Vladimir Dejanović
23 November 2018

@VladimirD_42 Vladimir Dejanović Let’s Meet @devoxxua @VladimirD_42 [email protected] http://itshark.xyz github.com/vladimir-dejanovic

@VladimirD_42 Vladimir Dejanović Let’s Meet http://itshark.xyz @devoxxua

@VladimirD_42 Vladimir Dejanović Let’s Meet @devoxxua

@VladimirD_42 Vladimir Dejanović Let’s Meet and Others... @devoxxua

@VladimirD_42 @devoxxua

@VladimirD_42 @devoxxua Agenda

@VladimirD_42 @devoxxua Agenda Success vs Fail

@VladimirD_42 @devoxxua Agenda A/B Testing Success vs Fail

@VladimirD_42 @devoxxua Agenda A/B Testing Failing at A/B and limitations
Success vs Fail

@VladimirD_42 @devoxxua Agenda A/B Testing Questions Failing at A/B and
limitations Success vs Fail

@VladimirD_42 @devoxxua Success vs Fail

@VladimirD_42 @devoxxua ref: https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAAiMAAAAJDljNzE4YjZmLWJhNTAtNGM0My04Mzc1LTYxMDNlYWIzNDBhMw.jpg

@VladimirD_42 @devoxxua ref: http://www.neuresourcegroup.com.au/brainwaves-for-leaders-blog//2012/08/09/hate-long-meetings-try-the-10-minute-standing-meeting/

@VladimirD_42 @devoxxua ref: http://www.theinquirer.net/inquirer/news/2194702/nuance-announces-dragon-naturally-speaking-12-with-android-and-ios-support

@VladimirD_42 @devoxxua ref: https://d1o50x50snmhul.cloudfront.net/wp-content/uploads/2016/07/05160722/f14aba_cropped-800x533.jpg

@VladimirD_42 @devoxxua ref: https://d1o50x50snmhul.cloudfront.net/wp-content/uploads/2016/07/05160722/f14aba_cropped-800x533.jpg Highest Paid Person Opinion

@VladimirD_42 @devoxxua ref: http://gogmat.com/blog

@VladimirD_42 @devoxxua ref: https://macpeanut.wordpress.com/2015/08/12/crystal-balls/

@VladimirD_42 @devoxxua ref: http://marketingland.com/view-mobile-retail-holidays-past-present-112419

@VladimirD_42 @devoxxua ref: http://www.desktopanimewallpaper.com/wallpaper/Super-Saiyan/

@VladimirD_42 @devoxxua A/B Testing

@VladimirD_42 @devoxxua ref: http://www.truthmove.org/news-category/surveillance/

@VladimirD_42 @devoxxua ref: http://www.rootsrundeep.com/hypothesis.html

@VladimirD_42 @devoxxua ref: http://zehno.com/news-resources/the-more-you-know-ab-testing

@VladimirD_42 @devoxxua ref: http://www.astekweb.com/the-web/what-is-ab-testing/

@VladimirD_42 @devoxxua ref: http://www.pitstopmedia.com/sem/minimize-the-impact-of-website-testing-on-seo

@VladimirD_42 @devoxxua ref: https://petovera.com/you-are-probably-getting-stupid-results-from-ab-testing-heres-why/

@VladimirD_42 @devoxxua ref: http://www.footballhub.tk/champions-league-draw-round-16/

@VladimirD_42 @devoxxua ref: https://www.graydon.co.uk/blog/data-driven-marketing-for-data-driven-marketers

@VladimirD_42 @devoxxua ref: http://www.fanpop.com/clubs/mr-spock/images/10874060/title/mr-spock-photo

@VladimirD_42 @devoxxua ref: https://s-media-cache-ak0.pinimg.com/originals/d3/14/d2/d314d211d6fe7f8967819858e4fa5333.jpg

@VladimirD_42 @devoxxua Original

@VladimirD_42 @devoxxua Base Original

@VladimirD_42 @devoxxua Base Control Original

@VladimirD_42 @devoxxua Base Control Variant Original

@VladimirD_42 @devoxxua Base Control Variant Variant Original

@VladimirD_42 @devoxxua Base Control Variant Variant Experiment Original

@VladimirD_42 @devoxxua Process

@VladimirD_42 @devoxxua Visitors Process

@VladimirD_42 @devoxxua Visitors Conversion Process

@VladimirD_42 @devoxxua A

@VladimirD_42 @devoxxua A B

@VladimirD_42 @devoxxua ref: https://vignette2.wikia.nocookie.net/breakingbad/images/9/9d/1x02_-_Coin_flip.jpg

@VladimirD_42 @devoxxua A B

@VladimirD_42 @devoxxua B1 B1 A A A1 A1 B B

@VladimirD_42 @devoxxua ref: https://i.stack.imgur.com/sHtyh.png

@VladimirD_42 @devoxxua ref: http://brightcove.vo.llnwd.net/d21/unsecured/media/1033249144001/1033249144001_2257652098001_video-still-for-video-2255602883001.jpg

@VladimirD_42 @devoxxua ref: https://www.geek-vintage.com/wp-content/uploads/invention-gaston.jpg

@VladimirD_42 @devoxxua ref: http://www.ru.nl/publish/pages/733019/mathematics-blackboard-article.jpg

@VladimirD_42 @devoxxua Yes No A B Total

@VladimirD_42 @devoxxua Yes No A A_Total B Total

@VladimirD_42 @devoxxua Yes No A A_Total B B_Total Total

@VladimirD_42 @devoxxua Yes No A A_Total B B_Total Total Total_Num

@VladimirD_42 @devoxxua Yes No A A_Yes A_Total B B_Total Total
Total_Num

@VladimirD_42 @devoxxua Yes No A A_Yes A_No A_Total B B_Total
Total Total_Num

@VladimirD_42 @devoxxua Yes No A A_Yes A_No A_Total B B_Yes
B_Total Total Total_Num

B_No B_Total Total Total_Num

B_No B_Total Total Total_Yes Total_Num

B_No B_Total Total Total_Yes Total_No Total_Num

@VladimirD_42 @devoxxua ref: http://anthonyrealestate.net/anthony3/wp-content/uploads/2011/12/crunching-numbers.jpg

@VladimirD_42 @devoxxua Expectations Ex_A_Yes Ex_A_No Ex_B_Yes Ex_B_No

@VladimirD_42 @devoxxua Expectations Ex_A_Yes A_Total * Total_Yes / Total_Num Ex_A_No
Ex_B_Yes Ex_B_No

A_Total * Total_No / Total_Num Ex_B_Yes Ex_B_No

A_Total * Total_No / Total_Num Ex_B_Yes B_Total * Total_Yes / Total_Num Ex_B_No B_Total * Total_No / Total_Num

@VladimirD_42 @devoxxua G-Test

@VladimirD_42 @devoxxua G-Test 2 * Measured * log ( Measured
/Expected )

@VladimirD_42 @devoxxua G-Test 2 * ( A_Yes * log(A_Yes /
Ex_A_Yes) + A_No * log(A_No / Ex_A_No) + B_Yes * log(B_Yes / Ex_B_Yes) + B_No * log(B_No / Ex_B_No) )

@VladimirD_42 @devoxxua Chi-squared Probability Test

@VladimirD_42 @devoxxua Chi-squared Probability Test P = Chi_Square_Probability(1, G_Test)

@VladimirD_42 @devoxxua Chi-squared Probability Test P = Chi_Square_Probability(1, G_Test) 0
≤ P≤ 1

@VladimirD_42 @devoxxua Significant Difference

@VladimirD_42 @devoxxua Significant Difference Sig_Diff = 1- P

@VladimirD_42 @devoxxua Significant Difference Sig_Diff = 1- P Sig_Diff =
0.95 => 95% chance

@VladimirD_42 @devoxxua Failing at A/B and Limitations

@VladimirD_42 @devoxxua Failing at A/B Testing

@VladimirD_42 @devoxxua Failing at A/B Testing Visitor != Saw Experiment

@VladimirD_42 @devoxxua Failing at A/B Testing Visitor != Saw Experiment
Distribution for multiple experiments

@VladimirD_42 @devoxxua B1 B1 A A A1 A1 B B

@VladimirD_42 @devoxxua Failing at A/B Testing Errors interfering with Experiment

@VladimirD_42 @devoxxua Failing at A/B Testing

@VladimirD_42 @devoxxua Failing at A/B Testing Not looking at whole
process

@VladimirD_42 @devoxxua A/B Experiment

@VladimirD_42 @devoxxua A/B Experiment Issues

@VladimirD_42 @devoxxua Failing at A/B Testing Stopping experiment to fast

@VladimirD_42 @devoxxua Mon Thu Wed Tue Fri Sun Sat

@VladimirD_42 @devoxxua Failing at A/B Testing Understanding seasonality

@VladimirD_42 @devoxxua ref: http://aws-cf.caradisiac.com/prod/uploads/200401/ziriako_1073088347_gasapo1f.gif

@VladimirD_42 @devoxxua Failing at A/B Testing Using Experiment 1 data
to prove “Experiment 2”

@VladimirD_42 @devoxxua Limitations

@VladimirD_42 @devoxxua ref: http://www.tashakheiriddin.com/wp-content/uploads/2016/12/chicken-or-egg2.jpg

@VladimirD_42 @devoxxua ref: https://media3.s-nbcnews.com/j/newscms/2018_23/2455046/180605-big-bang-illustration-al-1141_c426a54e8d4f6d67f0b33d1abb671d40.fit-2000w.jpg

@VladimirD_42 @devoxxua Summary

@VladimirD_42 @devoxxua Summary Customer in center of everything

@VladimirD_42 @devoxxua Summary A/B testing Customer in center of everything

@VladimirD_42 @devoxxua Summary A/B testing Ways to fail Customer in
center of everything

@VladimirD_42 @devoxxua Summary A/B testing Limitation Ways to fail Customer
in center of everything

@VladimirD_42 @devoxxua Thank You

@VladimirD_42 Questions @VladimirD_42 [email protected] http://itshark.xyz @devoxxua

What Users Want, A/B testing explained [Devoxx ...

What Users Want, A/B testing explained [Devoxx Ukraine 2018]

More Decks by Vladimir Dejanovic

Other Decks in Programming

Featured

Transcript