Slide 1

Slide 1 text

#CodeteCON @VladimirD_42 What Users Want A/B testing explained Vladimir Dejanović 11 October 2018

Slide 2

Slide 2 text

@VladimirD_42 Vladimir Dejanović Let’s Meet @VladimirD_42 [email protected] http://itshark.xyz #CodeteCON

Slide 3

Slide 3 text

@VladimirD_42 Vladimir Dejanović Let’s Meet http://itshark.xyz #CodeteCON

Slide 4

Slide 4 text

@VladimirD_42 Vladimir Dejanović Let’s Meet #CodeteCON

Slide 5

Slide 5 text

@VladimirD_42 Vladimir Dejanović Let’s Meet and Others... #CodeteCON

Slide 6

Slide 6 text

@VladimirD_42 #CodeteCON Agenda

Slide 7

Slide 7 text

@VladimirD_42 #CodeteCON Agenda Success vs Fail

Slide 8

Slide 8 text

@VladimirD_42 #CodeteCON Agenda A/B Testing Success vs Fail

Slide 9

Slide 9 text

@VladimirD_42 #CodeteCON Agenda A/B Testing Failing at A/B and limitations Success vs Fail

Slide 10

Slide 10 text

@VladimirD_42 #CodeteCON Agenda A/B Testing Questions Failing at A/B and limitations Success vs Fail

Slide 11

Slide 11 text

@VladimirD_42 #CodeteCON Success vs Fail

Slide 12

Slide 12 text

@VladimirD_42 #CodeteCON

Slide 13

Slide 13 text

@VladimirD_42 #CodeteCON

Slide 14

Slide 14 text

@VladimirD_42 #CodeteCON

Slide 15

Slide 15 text

@VladimirD_42 #CodeteCON ref: https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAAiMAAAAJDljNzE4YjZmLWJhNTAtNGM0My04Mzc1LTYxMDNlYWIzNDBhMw.jpg

Slide 16

Slide 16 text

@VladimirD_42 #CodeteCON ref: http://www.neuresourcegroup.com.au/brainwaves-for-leaders-blog//2012/08/09/hate-long-meetings-try-the-10-minute-standing-meeting/

Slide 17

Slide 17 text

@VladimirD_42 #CodeteCON ref: http://www.theinquirer.net/inquirer/news/2194702/nuance-announces-dragon-naturally-speaking-12-with-android-and-ios-support

Slide 18

Slide 18 text

@VladimirD_42 #CodeteCON ref: https://d1o50x50snmhul.cloudfront.net/wp-content/uploads/2016/07/05160722/f14aba_cropped-800x533.jpg

Slide 19

Slide 19 text

@VladimirD_42 #CodeteCON ref: https://d1o50x50snmhul.cloudfront.net/wp-content/uploads/2016/07/05160722/f14aba_cropped-800x533.jpg Highest Paid Person Opinion

Slide 20

Slide 20 text

@VladimirD_42 #CodeteCON ref: http://gogmat.com/blog

Slide 21

Slide 21 text

@VladimirD_42 #CodeteCON ref: https://macpeanut.wordpress.com/2015/08/12/crystal-balls/

Slide 22

Slide 22 text

@VladimirD_42 #CodeteCON ref: http://marketingland.com/view-mobile-retail-holidays-past-present-112419

Slide 23

Slide 23 text

@VladimirD_42 #CodeteCON ref: http://www.desktopanimewallpaper.com/wallpaper/Super-Saiyan/

Slide 24

Slide 24 text

@VladimirD_42 #CodeteCON A/B Testing

Slide 25

Slide 25 text

@VladimirD_42 #CodeteCON ref: http://www.truthmove.org/news-category/surveillance/

Slide 26

Slide 26 text

@VladimirD_42 #CodeteCON ref: http://www.rootsrundeep.com/hypothesis.html

Slide 27

Slide 27 text

@VladimirD_42 #CodeteCON ref: http://zehno.com/news-resources/the-more-you-know-ab-testing

Slide 28

Slide 28 text

@VladimirD_42 #CodeteCON ref: http://www.astekweb.com/the-web/what-is-ab-testing/

Slide 29

Slide 29 text

@VladimirD_42 #CodeteCON ref: http://www.pitstopmedia.com/sem/minimize-the-impact-of-website-testing-on-seo

Slide 30

Slide 30 text

@VladimirD_42 #CodeteCON ref: https://petovera.com/you-are-probably-getting-stupid-results-from-ab-testing-heres-why/

Slide 31

Slide 31 text

@VladimirD_42 #CodeteCON ref: http://www.footballhub.tk/champions-league-draw-round-16/

Slide 32

Slide 32 text

@VladimirD_42 #CodeteCON ref: https://www.graydon.co.uk/blog/data-driven-marketing-for-data-driven-marketers

Slide 33

Slide 33 text

@VladimirD_42 #CodeteCON ref: http://www.fanpop.com/clubs/mr-spock/images/10874060/title/mr-spock-photo

Slide 34

Slide 34 text

@VladimirD_42 #CodeteCON

Slide 35

Slide 35 text

@VladimirD_42 #CodeteCON

Slide 36

Slide 36 text

@VladimirD_42 #CodeteCON

Slide 37

Slide 37 text

@VladimirD_42 #CodeteCON ref: https://s-media-cache-ak0.pinimg.com/originals/d3/14/d2/d314d211d6fe7f8967819858e4fa5333.jpg

Slide 38

Slide 38 text

@VladimirD_42 #CodeteCON Original

Slide 39

Slide 39 text

@VladimirD_42 #CodeteCON Base Original

Slide 40

Slide 40 text

@VladimirD_42 #CodeteCON Base Control Original

Slide 41

Slide 41 text

@VladimirD_42 #CodeteCON Base Control Variant Original

Slide 42

Slide 42 text

@VladimirD_42 #CodeteCON Base Control Variant Variant Original

Slide 43

Slide 43 text

@VladimirD_42 #CodeteCON Base Control Variant Variant Experiment Original

Slide 44

Slide 44 text

@VladimirD_42 #CodeteCON Process

Slide 45

Slide 45 text

@VladimirD_42 #CodeteCON Process

Slide 46

Slide 46 text

@VladimirD_42 #CodeteCON Process

Slide 47

Slide 47 text

@VladimirD_42 #CodeteCON Visitors Process

Slide 48

Slide 48 text

@VladimirD_42 #CodeteCON Visitors Process

Slide 49

Slide 49 text

@VladimirD_42 #CodeteCON Visitors Conversion Process

Slide 50

Slide 50 text

@VladimirD_42 #CodeteCON

Slide 51

Slide 51 text

@VladimirD_42 #CodeteCON

Slide 52

Slide 52 text

@VladimirD_42 #CodeteCON

Slide 53

Slide 53 text

@VladimirD_42 #CodeteCON

Slide 54

Slide 54 text

@VladimirD_42 #CodeteCON A

Slide 55

Slide 55 text

@VladimirD_42 #CodeteCON A B

Slide 56

Slide 56 text

@VladimirD_42 #CodeteCON A B

Slide 57

Slide 57 text

@VladimirD_42 #CodeteCON A B

Slide 58

Slide 58 text

@VladimirD_42 #CodeteCON ref: https://vignette2.wikia.nocookie.net/breakingbad/images/9/9d/1x02_-_Coin_flip.jpg

Slide 59

Slide 59 text

@VladimirD_42 #CodeteCON A B

Slide 60

Slide 60 text

@VladimirD_42 #CodeteCON B1 B1 A A A1 A1 B B

Slide 61

Slide 61 text

@VladimirD_42 #CodeteCON ref: https://i.stack.imgur.com/sHtyh.png

Slide 62

Slide 62 text

@VladimirD_42 #CodeteCON ref: http://brightcove.vo.llnwd.net/d21/unsecured/media/1033249144001/1033249144001_2257652098001_video-still-for-video-2255602883001.jpg

Slide 63

Slide 63 text

@VladimirD_42 #CodeteCON ref: https://www.geek-vintage.com/wp-content/uploads/invention-gaston.jpg

Slide 64

Slide 64 text

@VladimirD_42 #CodeteCON ref: http://www.ru.nl/publish/pages/733019/mathematics-blackboard-article.jpg

Slide 65

Slide 65 text

@VladimirD_42 #CodeteCON Yes No A B Total

Slide 66

Slide 66 text

@VladimirD_42 #CodeteCON Yes No A A_Total B Total

Slide 67

Slide 67 text

@VladimirD_42 #CodeteCON Yes No A A_Total B B_Total Total

Slide 68

Slide 68 text

@VladimirD_42 #CodeteCON Yes No A A_Total B B_Total Total Total_Num

Slide 69

Slide 69 text

@VladimirD_42 #CodeteCON Yes No A A_Yes A_Total B B_Total Total Total_Num

Slide 70

Slide 70 text

@VladimirD_42 #CodeteCON Yes No A A_Yes A_No A_Total B B_Total Total Total_Num

Slide 71

Slide 71 text

@VladimirD_42 #CodeteCON Yes No A A_Yes A_No A_Total B B_Yes B_Total Total Total_Num

Slide 72

Slide 72 text

@VladimirD_42 #CodeteCON Yes No A A_Yes A_No A_Total B B_Yes B_No B_Total Total Total_Num

Slide 73

Slide 73 text

@VladimirD_42 #CodeteCON Yes No A A_Yes A_No A_Total B B_Yes B_No B_Total Total Total_Yes Total_Num

Slide 74

Slide 74 text

@VladimirD_42 #CodeteCON Yes No A A_Yes A_No A_Total B B_Yes B_No B_Total Total Total_Yes Total_No Total_Num

Slide 75

Slide 75 text

@VladimirD_42 #CodeteCON ref: http://anthonyrealestate.net/anthony3/wp-content/uploads/2011/12/crunching-numbers.jpg

Slide 76

Slide 76 text

@VladimirD_42 #CodeteCON Expectations Ex_A_Yes Ex_A_No Ex_B_Yes Ex_B_No

Slide 77

Slide 77 text

@VladimirD_42 #CodeteCON Expectations Ex_A_Yes A_Total * Total_Yes / Total_Num Ex_A_No Ex_B_Yes Ex_B_No

Slide 78

Slide 78 text

@VladimirD_42 #CodeteCON Expectations Ex_A_Yes A_Total * Total_Yes / Total_Num Ex_A_No A_Total * Total_No / Total_Num Ex_B_Yes Ex_B_No

Slide 79

Slide 79 text

@VladimirD_42 #CodeteCON Expectations Ex_A_Yes A_Total * Total_Yes / Total_Num Ex_A_No A_Total * Total_No / Total_Num Ex_B_Yes B_Total * Total_Yes / Total_Num Ex_B_No B_Total * Total_No / Total_Num

Slide 80

Slide 80 text

@VladimirD_42 #CodeteCON G-Test

Slide 81

Slide 81 text

@VladimirD_42 #CodeteCON G-Test 2 * Measured * log ( Measured /Expected )

Slide 82

Slide 82 text

@VladimirD_42 #CodeteCON G-Test 2 * ( A_Yes * log(A_Yes / Ex_A_Yes) + A_No * log(A_No / Ex_A_No) + B_Yes * log(B_Yes / Ex_B_Yes) + B_No * log(B_No / Ex_B_No) )

Slide 83

Slide 83 text

@VladimirD_42 #CodeteCON Chi-squared Probability Test

Slide 84

Slide 84 text

@VladimirD_42 #CodeteCON Chi-squared Probability Test P = Chi_Square_Probability(1, G_Test)

Slide 85

Slide 85 text

@VladimirD_42 #CodeteCON Chi-squared Probability Test P = Chi_Square_Probability(1, G_Test) 0 ≤ P≤ 1

Slide 86

Slide 86 text

@VladimirD_42 #CodeteCON Significant Difference

Slide 87

Slide 87 text

@VladimirD_42 #CodeteCON Significant Difference Sig_Diff = 1- P

Slide 88

Slide 88 text

@VladimirD_42 #CodeteCON Significant Difference Sig_Diff = 1- P Sig_Diff = 0.95 => 95% chance

Slide 89

Slide 89 text

@VladimirD_42 #CodeteCON Failing at A/B and Limitations

Slide 90

Slide 90 text

@VladimirD_42 #CodeteCON Failing at A/B Testing

Slide 91

Slide 91 text

@VladimirD_42 #CodeteCON Failing at A/B Testing Visitor != Saw Experiment

Slide 92

Slide 92 text

@VladimirD_42 #CodeteCON Failing at A/B Testing Visitor != Saw Experiment Distribution for multiple experiments

Slide 93

Slide 93 text

@VladimirD_42 #CodeteCON B1 B1 A A A1 A1 B B

Slide 94

Slide 94 text

@VladimirD_42 #CodeteCON Failing at A/B Testing Errors interfering with Experiment

Slide 95

Slide 95 text

@VladimirD_42 #CodeteCON Failing at A/B Testing

Slide 96

Slide 96 text

@VladimirD_42 #CodeteCON Failing at A/B Testing Not looking at whole process

Slide 97

Slide 97 text

@VladimirD_42 #CodeteCON

Slide 98

Slide 98 text

@VladimirD_42 #CodeteCON A/B Experiment

Slide 99

Slide 99 text

@VladimirD_42 #CodeteCON A/B Experiment Issues

Slide 100

Slide 100 text

@VladimirD_42 #CodeteCON Failing at A/B Testing Stopping experiment to fast

Slide 101

Slide 101 text

@VladimirD_42 #CodeteCON Mon Thu Wed Tue Fri Sun Sat

Slide 102

Slide 102 text

@VladimirD_42 #CodeteCON Mon Thu Wed Tue Fri Sun Sat

Slide 103

Slide 103 text

@VladimirD_42 #CodeteCON Mon Thu Wed Tue Fri Sun Sat

Slide 104

Slide 104 text

@VladimirD_42 #CodeteCON Failing at A/B Testing Understanding seasonality

Slide 105

Slide 105 text

@VladimirD_42 #CodeteCON ref: http://aws-cf.caradisiac.com/prod/uploads/200401/ziriako_1073088347_gasapo1f.gif

Slide 106

Slide 106 text

@VladimirD_42 #CodeteCON Failing at A/B Testing Using Experiment 1 data to prove “Experiment 2”

Slide 107

Slide 107 text

@VladimirD_42 #CodeteCON Limitations

Slide 108

Slide 108 text

@VladimirD_42 #CodeteCON

Slide 109

Slide 109 text

@VladimirD_42 #CodeteCON ref: http://www.tashakheiriddin.com/wp-content/uploads/2016/12/chicken-or-egg2.jpg

Slide 110

Slide 110 text

@VladimirD_42 #CodeteCON ref: https://media3.s-nbcnews.com/j/newscms/2018_23/2455046/180605-big-bang-illustration-al-1141_c426a54e8d4f6d67f0b33d1abb671d40.fit-2000w.jpg

Slide 111

Slide 111 text

@VladimirD_42 #CodeteCON Thank You

Slide 112

Slide 112 text

@VladimirD_42 Questions @VladimirD_42 [email protected] http://itshark.xyz #CodeteCON