AB Test - Speaker Deck

Slide 1

Slide 1 text

A/B TEST IN GOD WE TRUST OTHERS MUST BRING DATA

Slide 2

Slide 2 text

GLOBO.COM

Slide 3

Slide 3 text

BOOKING.COM

Slide 4

Slide 4 text

LOOPINFINITO.COM.BR

Slide 5

Slide 5 text

COLETÂNEA FRONT-END

Slide 6

Slide 6 text

BASICS

Slide 7

Slide 7 text

AT GOOGLE USED TO SET OPTIMAL NUMBER ON SEARCH RESULTS AROUND 2000 ORIGIN

Slide 8

Slide 8 text

DEFINITION A WAY TO TEST YOUR PAGE AGAINST THE CURRENT DESIGN AND DETERMINE WHICH ONES PRODUCES POSITIVE RESULTS BASICS

Slide 9

Slide 9 text

B FLIP COIN A BASICS

Slide 10

Slide 10 text

HiPPO HIGHEST PAID PERSON OPINION

Slide 11

Slide 11 text

I THINK I KNOW

Slide 12

Slide 12 text

TEST CASUAL RELATIONSHIP INSULATE EXTERNAL FACTORS STANDARD FDA DRUG TEST BASICS

Slide 13

Slide 13 text

CORRELATION CAUSALITY *

Slide 14

Slide 14 text

AS İCE CREAM SALES İNCREASE, THE RATE OF DROWNİNG DEATHS İNCREASES SHARPLY. THEREFORE, İCE CREAM CONSUMPTİON CAUSES DROWNİNG. CORRELATION / CAUSALITY EXAMPLE 1 HTTP://EN.WİKİPEDİA.ORG/WİKİ/CORRELATİON_DOES_NOT_İMPLY_CAUSATİON

Slide 15

Slide 15 text

CORRELATION / CAUSALITY CAUSALITY CAUSALITY CORRELATION

Slide 16

Slide 16 text

PALM SİZE CORRELATES WİTH YOUR LİFE EXPECTANCY.   THE LARGER YOUR PALM, THE LESS YOU WİLL LİVE, ON AVERAGE. CORRELATION / CAUSALITY EXAMPLE 2

Slide 17

Slide 17 text

WOMEN HAVE SMALLER PALMS AND LİVE 6 YEARS LONGER ON AVERAGE CORRELATION / CAUSALITY CORRELATION

Slide 18

Slide 18 text

CONVERSION

Slide 19

Slide 19 text

DEFINITION THE POINT AT WHICH A USER TAKES THE DESIRED ACTION CONVERSION

Slide 20

Slide 20 text

FOR E-COMMERCE •COMPLETED PURCHASES •PRODUCT PAGE VIEW CONVERSION

Slide 21

Slide 21 text

FOR MEDIA •PAGE VIEW •ARTICLES READ CONVERSION

Slide 22

Slide 22 text

FOR DONATION •FORM COMPLETION CONVERSION

Slide 23

Slide 23 text

CASES

Slide 24

Slide 24 text

YOU ARE THE HiPPO

Slide 25

Slide 25 text

1 2 3 4 5 6 MSN REAL STATE HTTP://Aİ.STANFORD.EDU/~RONNYK/EXPTHİNKWEEK2009PUBLİC.PDF

Slide 26

Slide 26 text

6 MSN REAL STATE ↑9.7% HTTP://Aİ.STANFORD.EDU/~RONNYK/EXPTHİNKWEEK2009PUBLİC.PDF HTTP://Aİ.STANFORD.EDU/~RONNYK/EXPTHİNKWEEK2009PUBLİC.PDF

Slide 27

Slide 27 text

MSN HOME PAGE SEARCH BOX 1 2 HTTP://DL.ACM.ORG/CİTATİON.CFM?İD=1281295&CFID=431848911&CFTOKEN=49176768

Slide 28

Slide 28 text

MSN HOME PAGE SEARCH BOX 1 2 NEUTRAL RESULT HTTP://DL.ACM.ORG/CİTATİON.CFM?İD=1281295&CFID=431848911&CFTOKEN=49176768

Slide 29

Slide 29 text

MICROSOFT OFFICE ONLINE STORE 1 HTTP://WWW.EXP-PLATFORM.COM/DOCUMENTS/EXP_DMCASESTUDİES.PDF

Slide 30

Slide 30 text

MICROSOFT OFFICE ONLINE STORE 2 HTTP://WWW.EXP-PLATFORM.COM/DOCUMENTS/EXP_DMCASESTUDİES.PDF

Slide 31

Slide 31 text

MICROSOFT OFFICE ONLINE STORE ↓64% 2 HTTP://WWW.EXP-PLATFORM.COM/DOCUMENTS/EXP_DMCASESTUDİES.PDF

Slide 32

Slide 32 text

SAFESOFT SOLUTIONS 1 HTTPS://VWO.COM/BLOG/PROMİNENT-PRİCE-DİSPLAY-İNCREASES-LEADS-GENERATED-BY-100/ 2

Slide 33

Slide 33 text

2 SAFESOFT SOLUTIONS ↑100% HTTPS://VWO.COM/BLOG/PROMİNENT-PRİCE-DİSPLAY-İNCREASES-LEADS-GENERATED-BY-100/

Slide 34

Slide 34 text

HIGHRISE 1 2 HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL

Slide 35

Slide 35 text

HIGHRISE 2 ↑37.5% HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL

Slide 36

Slide 36 text

HIGHRISE 1 2 HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL

Slide 37

Slide 37 text

HIGHRISE 2 ↑102.5% HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL

Slide 38

Slide 38 text

HIGHRISE 3 HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL

Slide 39

Slide 39 text

HIGHRISE ↓22.72% HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL 3

Slide 40

Slide 40 text

HIGHRISE HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL

Slide 41

Slide 41 text

HIGHRISE ↑4.78% ↑3.49% ↓3.38% ↓0.8% ↑2.2% HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL

Slide 42

Slide 42 text

REFINEMENT / EXPLORATION HTTP://İNSİDEİNTERCOM.İO/WİREFRAMİNG-FOR-WEB-APPS/

Slide 43

Slide 43 text

NETFLIX BASE HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/

Slide 44

Slide 44 text

NETFLIX VARIANT HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/

Slide 45

Slide 45 text

JUNE 2011 NEW DESIGN WAS ROLLED OUT FOR 100% OF USERS NETFLIX HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/

Slide 46

Slide 46 text

PPL WENT MAD NETFLIX HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/

Slide 47

Slide 47 text

DATA DON’T LIE ↑ RETENTION ↑ ENGAGEMENT NETFLIX HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/

Slide 48

Slide 48 text

“WHAT PEOPLE SAY AND WHAT THEY DO ARE RARELY THE SAME.” — BRYAN GUMM NETFLIX HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/

Slide 49

Slide 49 text

LIVECHAT 1 2 DAN SİROKER & PETE KOOMEN. “A/B TESTİNG.”

Slide 50

Slide 50 text

LIVECHAT 2 ↑14.6% DAN SİROKER & PETE KOOMEN. “A/B TESTİNG.”

Slide 51

Slide 51 text

FRAMING EFFECTS *

Slide 52

Slide 52 text

DEFINITION AN EXAMPLE OF COGNİTİVE BİAS, İN WHİCH PEOPLE REACT TO A PARTİCULAR CHOİCE İN DİFFERENT WAYS DEPENDİNG ON WHETHER İT İS PRESENTED AS A LOSS OR AS A GAİN FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY)

Slide 53

Slide 53 text

PARTİCİPANTS WERE ASKED TO CHOOSE BETWEEN TWO TREATMENTS FOR 600 PEOPLE AFFECTED BY A DEADLY DİSEASE EXAMPLE 1 FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY)

Slide 54

Slide 54 text

RESEARCH TREATMENT A WAS PREDİCTED TO RESULT İN 400 DEATHS, WHEREAS TREATMENT B HAD A 33% CHANCE THAT NO ONE WOULD DİE BUT A 66% CHANCE THAT EVERYONE WOULD DİE FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY)

Slide 55

Slide 55 text

POSITIVE FRAMING TRATEMENT A SAVES 200 LİVES TRATEMENT B A 33% CHANCE OF SAVİNG ALL 600 PEOPLE, 66% POSSİBİLİTY OF SAVİNG NO ONE FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY)

Slide 56

Slide 56 text

NEGATIVE FRAMING TRATEMENT A 400 PEOPLE WİLL DİE TRATEMENT B A 33% CHANCE THAT NO PEOPLE WİLL DİE, 66% PROBABİLİTY THAT ALL 600 WİLL DİE FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY)

Slide 57

Slide 57 text

TRATMENT A CHOSEN BY 72% WHEN PRESENTED WİTH POSİTİVE FRAMİNG FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY) CHOSEN BY 22% WHEN PRESENTED WİTH NEGATIVE FRAMİNG

Slide 58

Slide 58 text

BOOKING.COM HTTP://BLOG.BOOKİNG.COM/HAMBURGER-MENU.HTML 1 2

Slide 59

Slide 59 text

BOOKING.COM 1 2 HTTP://BLOG.BOOKİNG.COM/HAMBURGER-MENU.HTML NEUTRAL RESULT

Slide 60

Slide 60 text

SUMMARY

Slide 61

Slide 61 text

DON’T BELIEVE THE GURUS WHAT WORKS FOR “EVERYBODY” MAY NOT WORK FOR YOU SUMMARY

Slide 62

Slide 62 text

DON’T BELIEVE THE HiPPO HiPPO DON’T THINK. THEY USED TO. YOU ARE BEING PAID TO DO IT SUMMARY

Slide 63

Slide 63 text

DON’T BELIEVE IN YOU YOU DON’T KNOW WHAT CONSUMER WANTS. YOU JUST HAVE HYPOTESHIS SUMMARY