AB Test

In God we trust
Others must bring data

Transcript

1. A/B TEST IN GOD WE TRUST OTHERS MUST BRING DATA
2. GLOBO.COM
3. BOOKING.COM
4. LOOPINFINITO.COM.BR
5. COLETÂNEA FRONT-END
6. BASICS
7. AT GOOGLE USED TO SET OPTIMAL NUMBER ON SEARCH RESULTS AROUND 2000 ORIGIN
8. DEFINITION A WAY TO TEST YOUR PAGE AGAINST THE CURRENT DESIGN AND DETERMINE WHICH ONES PRODUCES POSITIVE RESULTS BASICS
9. B FLIP COIN A BASICS
10. HiPPO HIGHEST PAID PERSON OPINION
11. I THINK I KNOW
12. TEST CASUAL RELATIONSHIP INSULATE EXTERNAL FACTORS STANDARD FDA DRUG TEST BASICS
13. CORRELATION CAUSALITY *
14. AS İCE CREAM SALES İNCREASE, THE RATE OF DROWNİNG DEATHS İNCREASES SHARPLY. THEREFORE, İCE CREAM CONSUMPTİON CAUSES DROWNİNG. CORRELATION / CAUSALITY EXAMPLE 1 HTTP://EN.WİKİPEDİA.ORG/WİKİ/CORRELATİON_DOES_NOT_İMPLY_CAUSATİON
15. CORRELATION / CAUSALITY CAUSALITY CAUSALITY CORRELATION
16. PALM SİZE CORRELATES WİTH YOUR LİFE EXPECTANCY. 
 THE LARGER YOUR PALM, THE LESS YOU WİLL LİVE, ON AVERAGE. CORRELATION / CAUSALITY EXAMPLE 2
17. WOMEN HAVE SMALLER PALMS AND LİVE 6 YEARS LONGER ON AVERAGE CORRELATION / CAUSALITY CORRELATION
18. CONVERSION
19. DEFINITION THE POINT AT WHICH A USER TAKES THE DESIRED ACTION CONVERSION
20. FOR E-COMMERCE •COMPLETED PURCHASES •PRODUCT PAGE VIEW CONVERSION
21. FOR MEDIA •PAGE VIEW •ARTICLES READ CONVERSION
22. FOR DONATION •FORM COMPLETION CONVERSION
23. CASES
24. YOU ARE THE HiPPO
25. 1 2 3 4 5 6 MSN REAL STATE HTTP://Aİ.STANFORD.EDU/~RONNYK/EXPTHİNKWEEK2009PUBLİC.PDF
26. 6 MSN REAL STATE ↑9.7% HTTP://Aİ.STANFORD.EDU/~RONNYK/EXPTHİNKWEEK2009PUBLİC.PDF HTTP://Aİ.STANFORD.EDU/~RONNYK/EXPTHİNKWEEK2009PUBLİC.PDF
27. MSN HOME PAGE SEARCH BOX 1 2 HTTP://DL.ACM.ORG/CİTATİON.CFM?İD=1281295&CFID=431848911&CFTOKEN=49176768
28. MSN HOME PAGE SEARCH BOX 1 2 NEUTRAL RESULT HTTP://DL.ACM.ORG/CİTATİON.CFM?İD=1281295&CFID=431848911&CFTOKEN=49176768
29. MICROSOFT OFFICE ONLINE STORE 1 HTTP://WWW.EXP-PLATFORM.COM/DOCUMENTS/EXP_DMCASESTUDİES.PDF
30. MICROSOFT OFFICE ONLINE STORE 2 HTTP://WWW.EXP-PLATFORM.COM/DOCUMENTS/EXP_DMCASESTUDİES.PDF
31. MICROSOFT OFFICE ONLINE STORE ↓64% 2 HTTP://WWW.EXP-PLATFORM.COM/DOCUMENTS/EXP_DMCASESTUDİES.PDF
32. SAFESOFT SOLUTIONS 1 HTTPS://VWO.COM/BLOG/PROMİNENT-PRİCE-DİSPLAY-İNCREASES-LEADS-GENERATED-BY-100/ 2
33. 2 SAFESOFT SOLUTIONS ↑100% HTTPS://VWO.COM/BLOG/PROMİNENT-PRİCE-DİSPLAY-İNCREASES-LEADS-GENERATED-BY-100/
34. HIGHRISE 1 2 HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL
35. HIGHRISE 2 ↑37.5% HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL
36. HIGHRISE 1 2 HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL
37. HIGHRISE 2 ↑102.5% HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL
38. HIGHRISE 3 HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL
39. HIGHRISE ↓22.72% HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL 3
40. HIGHRISE HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL
41. HIGHRISE ↑4.78% ↑3.49% ↓3.38% ↓0.8% ↑2.2% HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL
42. REFINEMENT / EXPLORATION HTTP://İNSİDEİNTERCOM.İO/WİREFRAMİNG-FOR-WEB-APPS/
43. NETFLIX BASE HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/
44. NETFLIX VARIANT HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/
45. JUNE 2011 NEW DESIGN WAS ROLLED OUT FOR 100% OF USERS NETFLIX HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/
46. PPL WENT MAD NETFLIX HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/
47. DATA DON’T LIE ↑ RETENTION ↑ ENGAGEMENT NETFLIX HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/
48. “WHAT PEOPLE SAY AND WHAT THEY DO ARE RARELY THE SAME.” — BRYAN GUMM NETFLIX HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/
49. LIVECHAT 1 2 DAN SİROKER & PETE KOOMEN. “A/B TESTİNG.”
50. LIVECHAT 2 ↑14.6% DAN SİROKER & PETE KOOMEN. “A/B TESTİNG.”
51. FRAMING EFFECTS *
52. DEFINITION AN EXAMPLE OF COGNİTİVE BİAS, İN WHİCH PEOPLE REACT TO A PARTİCULAR CHOİCE İN DİFFERENT WAYS DEPENDİNG ON WHETHER İT İS PRESENTED AS A LOSS OR AS A GAİN FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY)
53. PARTİCİPANTS WERE ASKED TO CHOOSE BETWEEN TWO TREATMENTS FOR 600 PEOPLE AFFECTED BY A DEADLY DİSEASE EXAMPLE 1 FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY)
54. RESEARCH TREATMENT A WAS PREDİCTED TO RESULT İN 400 DEATHS, WHEREAS TREATMENT B HAD A 33% CHANCE THAT NO ONE WOULD DİE BUT A 66% CHANCE THAT EVERYONE WOULD DİE FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY)
55. POSITIVE FRAMING TRATEMENT A SAVES 200 LİVES TRATEMENT B A 33% CHANCE OF SAVİNG ALL 600 PEOPLE, 66% POSSİBİLİTY OF SAVİNG NO ONE FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY)
56. NEGATIVE FRAMING TRATEMENT A 400 PEOPLE WİLL DİE TRATEMENT B A 33% CHANCE THAT NO PEOPLE WİLL DİE, 66% PROBABİLİTY THAT ALL 600 WİLL DİE FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY)
57. TRATMENT A CHOSEN BY 72% WHEN PRESENTED WİTH POSİTİVE FRAMİNG FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY) CHOSEN BY 22% WHEN PRESENTED WİTH NEGATIVE FRAMİNG
58. BOOKING.COM HTTP://BLOG.BOOKİNG.COM/HAMBURGER-MENU.HTML 1 2
59. BOOKING.COM 1 2 HTTP://BLOG.BOOKİNG.COM/HAMBURGER-MENU.HTML NEUTRAL RESULT
60. SUMMARY
61. DON’T BELIEVE THE GURUS WHAT WORKS FOR “EVERYBODY” MAY NOT WORK FOR YOU SUMMARY
62. DON’T BELIEVE THE HiPPO HiPPO DON’T THINK. THEY USED TO. YOU ARE BEING PAID TO DO IT SUMMARY
63. DON’T BELIEVE IN YOU YOU DON’T KNOW WHAT CONSUMER WANTS. YOU JUST HAVE HYPOTESHIS SUMMARY
64. BELIEVE IN DATA IT NEVER LIES SUMMARY
65. BE OPEN TO CHANGE EXPERIMENT OFTEN, FAIL FAST, BELIEVE IN DATA SUMMARY
66. TEST EVERYTHING THE MOST IMPORTANT SLIDE EVER
67. @ ME@CAIOGONDIM.COM CAIOGONDIM.COM @CAIO_GONDIM CONTACT