Ethics and Effective A/B Testing

Facebook and OK Cupid have recently been under public scrutiny for the experiments they run on their users. The truth is every web site of a considerable enough scale experiments on their users. Whether it’s searching for just the right shade of blue for the “Buy!” button or trying out a new algorithm to perfect the news feed, these are psychological experiments and the user is the subject. In this talk, you’ll be introduced to the basics of solid, overlapping, multivariate testing as well as a simple tool for enabling those experiments. This will enable you to perform more, better, safer experimentation. This talk will also include a healthy discussion on the ethics of experimentation to help steer you away from any possible controversy.

Transcript

1. Ethics, and Effective A/B Testing
2. Dusty Burwell
4. Buy! Buy!
5. Buy! Buy! Buy!
6. Buy! Buy! Buy! Buy!
7. Buy! Buy! Buy! Buy!
8. Buy! Obtain! Acquire! Purchase!
9. Ethics, and Effective Multivariate Testing
10. Buy! Buy!
13. http://bit.ly/freak-atmospherics
14. 7th Grade Science Mr. Burwell
15. Experiment
16. Natural Experiment vs. Controlled Experiment
17. Hypothesis
18. Hypothesis Buy! = $$$
19. Success Metrics
20. Success Metrics $$$
21. Success Metrics $$$
22. Success Metrics $$$
23. Variables
24. Variables Buy! Buy! vs
25. Control Group
26. Control Buy!
27. Variant (Treatment)
28. Variant Buy!
31. Ethics & Philosophy
41. :-o
42. :-o :-)
43. :-) :-o :-)
44. :-) :-o :-) :-)
45. :-) :-o :-) :-) :-) :-)
46. :-) :-) :-) :-) :-) :-)
47. :-( :-( :-( :-( :-( :-(
49. Love is Blind
50. The Power of Suggestion
51. Common Rule
52. The university’s Institutional Review Board approved all procedures.
53. We may use the information we receive about you for internal operations, including troubleshooting, data analysis, testing, research and service improvement. EULA, who?
54. But, everyone does it!
55. Explicit Consent
57. Moving fast and breaking things
58. Buy! Purchase! Buy! Buy!
59. Buy! Purchase! Buy! Buy! Color Text
60. Layers
61. Color Text
62. Buckets
63. Color
64. 0 1 Color 3 2 4 7 6 5 9 8
65. 0 1 Color 3 2 4 7 6 5 9 8 = % 10
66. 0 1 Color 3 2 4 7 6 5 9 8 = % 10
67. 0 1 Color 3 2 4 7 6 5 9 8 = % 10
68. 0 1 Color 3 2 4 7 6 5 9 8 = % 10
69. 0 1 Color 3 2 4 7 6 5 9 8 = % 10
70. 0 1 Color 3 2 4 7 6 5 9 8 = % 10
71. 0 1 Color 3 2 4 7 6 5 9 8 = % 10
72. Color
73. Color
74. Color
75. Color
76. Color
77. Color Text
78. Confounding
79. Color Text
80. Color Text = ( + Layer) % 10
81. Color Text = ( + Layer) % 10
82. Data Collection and Analysis
83. et al.
84. et al. https://vwo.com/ab-split-test-duration/
85. Caveats
90. Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better. I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that. I’ve grown tired of debating such miniscule design decisions. There are more exciting design problems in this world to tackle. –Douglas Bowman
93. Dusty Burwell @+ https://github.com/dustyburwell/config-proxy