AB Test

212b8b6589f7e98dae14e6517101f427?s=47 Caio Gondim
September 26, 2014

AB Test

In God we trust
Others must bring data

212b8b6589f7e98dae14e6517101f427?s=128

Caio Gondim

September 26, 2014
Tweet

Transcript

  1. A/B TEST IN GOD WE TRUST OTHERS MUST BRING DATA

  2. GLOBO.COM

  3. BOOKING.COM

  4. LOOPINFINITO.COM.BR

  5. COLETÂNEA FRONT-END

  6. BASICS

  7. AT GOOGLE USED TO SET OPTIMAL NUMBER ON SEARCH RESULTS

    AROUND 2000 ORIGIN
  8. DEFINITION A WAY TO TEST YOUR PAGE AGAINST THE CURRENT

    DESIGN AND DETERMINE WHICH ONES PRODUCES POSITIVE RESULTS BASICS
  9. B FLIP COIN A BASICS

  10. HiPPO HIGHEST PAID PERSON OPINION

  11. I THINK I KNOW

  12. TEST CASUAL RELATIONSHIP INSULATE EXTERNAL FACTORS STANDARD FDA DRUG TEST

    BASICS
  13. CORRELATION CAUSALITY *

  14. AS İCE CREAM SALES İNCREASE, THE RATE OF DROWNİNG DEATHS

    İNCREASES SHARPLY. THEREFORE, İCE CREAM CONSUMPTİON CAUSES DROWNİNG. CORRELATION / CAUSALITY EXAMPLE 1 HTTP://EN.WİKİPEDİA.ORG/WİKİ/CORRELATİON_DOES_NOT_İMPLY_CAUSATİON
  15. CORRELATION / CAUSALITY CAUSALITY CAUSALITY CORRELATION

  16. PALM SİZE CORRELATES WİTH YOUR LİFE EXPECTANCY. 
 THE LARGER

    YOUR PALM, THE LESS YOU WİLL LİVE, ON AVERAGE. CORRELATION / CAUSALITY EXAMPLE 2
  17. WOMEN HAVE SMALLER PALMS AND LİVE 6 YEARS LONGER ON

    AVERAGE CORRELATION / CAUSALITY CORRELATION
  18. CONVERSION

  19. DEFINITION THE POINT AT WHICH A USER TAKES THE DESIRED

    ACTION CONVERSION
  20. FOR E-COMMERCE •COMPLETED PURCHASES •PRODUCT PAGE VIEW CONVERSION

  21. FOR MEDIA •PAGE VIEW •ARTICLES READ CONVERSION

  22. FOR DONATION •FORM COMPLETION CONVERSION

  23. CASES

  24. YOU ARE THE HiPPO

  25. 1 2 3 4 5 6 MSN REAL STATE HTTP://Aİ.STANFORD.EDU/~RONNYK/EXPTHİNKWEEK2009PUBLİC.PDF

  26. 6 MSN REAL STATE ↑9.7% HTTP://Aİ.STANFORD.EDU/~RONNYK/EXPTHİNKWEEK2009PUBLİC.PDF HTTP://Aİ.STANFORD.EDU/~RONNYK/EXPTHİNKWEEK2009PUBLİC.PDF

  27. MSN HOME PAGE SEARCH BOX 1 2 HTTP://DL.ACM.ORG/CİTATİON.CFM?İD=1281295&CFID=431848911&CFTOKEN=49176768

  28. MSN HOME PAGE SEARCH BOX 1 2 NEUTRAL RESULT HTTP://DL.ACM.ORG/CİTATİON.CFM?İD=1281295&CFID=431848911&CFTOKEN=49176768

  29. MICROSOFT OFFICE ONLINE STORE 1 HTTP://WWW.EXP-PLATFORM.COM/DOCUMENTS/EXP_DMCASESTUDİES.PDF

  30. MICROSOFT OFFICE ONLINE STORE 2 HTTP://WWW.EXP-PLATFORM.COM/DOCUMENTS/EXP_DMCASESTUDİES.PDF

  31. MICROSOFT OFFICE ONLINE STORE ↓64% 2 HTTP://WWW.EXP-PLATFORM.COM/DOCUMENTS/EXP_DMCASESTUDİES.PDF

  32. SAFESOFT SOLUTIONS 1 HTTPS://VWO.COM/BLOG/PROMİNENT-PRİCE-DİSPLAY-İNCREASES-LEADS-GENERATED-BY-100/ 2

  33. 2 SAFESOFT SOLUTIONS ↑100% HTTPS://VWO.COM/BLOG/PROMİNENT-PRİCE-DİSPLAY-İNCREASES-LEADS-GENERATED-BY-100/

  34. HIGHRISE 1 2 HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL

  35. HIGHRISE 2 ↑37.5% HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL

  36. HIGHRISE 1 2 HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL

  37. HIGHRISE 2 ↑102.5% HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL

  38. HIGHRISE 3 HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL

  39. HIGHRISE ↓22.72% HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL 3

  40. HIGHRISE HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL

  41. HIGHRISE ↑4.78% ↑3.49% ↓3.38% ↓0.8% ↑2.2% HTTPS://SİGNALVNOİSE.COM/POSTS/2991-BEHİND-THE-SCENES-AB-TESTİNG-PART-3-FİNAL

  42. REFINEMENT / EXPLORATION HTTP://İNSİDEİNTERCOM.İO/WİREFRAMİNG-FOR-WEB-APPS/

  43. NETFLIX BASE HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/

  44. NETFLIX VARIANT HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/

  45. JUNE 2011 NEW DESIGN WAS ROLLED OUT FOR 100% OF

    USERS NETFLIX HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/
  46. PPL WENT MAD NETFLIX HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/

  47. DATA DON’T LIE ↑ RETENTION ↑ ENGAGEMENT NETFLIX HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/

  48. “WHAT PEOPLE SAY AND WHAT THEY DO ARE RARELY THE

    SAME.” — BRYAN GUMM NETFLIX HTTP://BLOG.OPTİMİZELY.COM/2014/08/14/2-ALEXA-500-SİTE-REDESİGNS-THAT-SHOULD-İNSPİRE-YOU-TO-AB-TEST/
  49. LIVECHAT 1 2 DAN SİROKER & PETE KOOMEN. “A/B TESTİNG.”

  50. LIVECHAT 2 ↑14.6% DAN SİROKER & PETE KOOMEN. “A/B TESTİNG.”

  51. FRAMING EFFECTS *

  52. DEFINITION AN EXAMPLE OF COGNİTİVE BİAS, İN WHİCH PEOPLE REACT

    TO A PARTİCULAR CHOİCE İN DİFFERENT WAYS DEPENDİNG ON WHETHER İT İS PRESENTED AS A LOSS OR AS A GAİN FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY)
  53. PARTİCİPANTS WERE ASKED TO CHOOSE BETWEEN TWO TREATMENTS FOR 600

    PEOPLE AFFECTED BY A DEADLY DİSEASE EXAMPLE 1 FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY)
  54. RESEARCH TREATMENT A WAS PREDİCTED TO RESULT İN 400 DEATHS,

    WHEREAS TREATMENT B HAD A 33% CHANCE THAT NO ONE WOULD DİE BUT A 66% CHANCE THAT EVERYONE WOULD DİE FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY)
  55. POSITIVE FRAMING TRATEMENT A SAVES 200 LİVES TRATEMENT B A

    33% CHANCE OF SAVİNG ALL 600 PEOPLE, 66% POSSİBİLİTY OF SAVİNG NO ONE FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY)
  56. NEGATIVE FRAMING TRATEMENT A 400 PEOPLE WİLL DİE TRATEMENT B

    A 33% CHANCE THAT NO PEOPLE WİLL DİE, 66% PROBABİLİTY THAT ALL 600 WİLL DİE FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY)
  57. TRATMENT A CHOSEN BY 72% WHEN PRESENTED WİTH POSİTİVE FRAMİNG

    FRAMING EFFECT HTTP://EN.WİKİPEDİA.ORG/WİKİ/FRAMİNG_EFFECT_(PSYCHOLOGY) CHOSEN BY 22% WHEN PRESENTED WİTH NEGATIVE FRAMİNG
  58. BOOKING.COM HTTP://BLOG.BOOKİNG.COM/HAMBURGER-MENU.HTML 1 2

  59. BOOKING.COM 1 2 HTTP://BLOG.BOOKİNG.COM/HAMBURGER-MENU.HTML NEUTRAL RESULT

  60. SUMMARY

  61. DON’T BELIEVE THE GURUS WHAT WORKS FOR “EVERYBODY” MAY NOT

    WORK FOR YOU SUMMARY
  62. DON’T BELIEVE THE HiPPO HiPPO DON’T THINK. THEY USED TO.

    YOU ARE BEING PAID TO DO IT SUMMARY
  63. DON’T BELIEVE IN YOU YOU DON’T KNOW WHAT CONSUMER WANTS.

    YOU JUST HAVE HYPOTESHIS SUMMARY
  64. BELIEVE IN DATA IT NEVER LIES SUMMARY

  65. BE OPEN TO CHANGE EXPERIMENT OFTEN, FAIL FAST, BELIEVE IN

    DATA SUMMARY
  66. TEST EVERYTHING THE MOST IMPORTANT SLIDE EVER

  67. @ ME@CAIOGONDIM.COM CAIOGONDIM.COM @CAIO_GONDIM CONTACT