Robust Experiment Setup

Building Robust Experiments while keeping the js footprint low.

Slides from my talk "Robust Experiment Setup" at WhichTestWon's (https://www.whichtestwon.com/) "The Live Event" in Austin 2015 : https://twitter.com/WhichTestWon/status/580371767870488576

Transcript

1. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. (Robust) Experiment Setup Keep Your Experiment Footprint Low Growth Hacker @ Atlassian Shreyas
2. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. building websites building experiments
3. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. websites experiments /home /home var_A var_B
4. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. websites experiments / /product /product/try / /product /product/try var_A var_B
5. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. websites experiments var_A var_B exp_1 exp_2 exp_3 var_A var_B var_A var_B
6. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. websites experiments Building Experiment Don’t usually have access to source code Source code can change while building the experiment after deploying the experiment
7. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. Experiment QA is harder Check all variations including control websites experiments You can’t (or shouldn’t) really do a hotfix after starting
8. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. building
9. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. 1. Understand Optimizely Execution Order diagram from “How Optimizely Works” experiment javascript, css before variation javascript
10. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. 1. Understand Optimizely Execution Order diagram from “How Optimizely Works” jQuery before native js vs jQuery(‘#btn’).hide(); document.getElementById(‘btn’).style.display=‘none’;
11. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. 1. Understand Optimizely Execution Order // is better than /* */ /* Comments written this way Will delay your code from executing until DOM ready */ // Comments written this way // on a single line will have // no impact on your tests vs
12. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. 2. “Use The Force, Luke” but wisely /* _optimizely_evaluate=force */ window.exp_wat161_top = 100; /* _optimizely_evaluate=safe */ jQuery(“#btn”).css({ “position”:”absolute", “top”:exp_wat161_top }); /* _optimizely_evaluate=force */ window.exp_wat161_top = 100; /* _optimizely_evaluate=force */ var exp_wat161_top = 100; vs
13. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. 3. The little things //auto-generated code in Optimizely $(".grid-3 > div:eq(2) > div:eq(1) > a:eq(0) > span:eq(0)").css({"position":"relative", "left":0, "top":0}); $(".grid-3 > div:eq(2) > div:eq(1) > a:eq(0) > span:eq(0)").css({"left":-48, "top":27}); vs reduce DOM lookups by caching jQuery elements jQuery(<selector>).css(<style1>); jQuery(<selector>).css(<style2>); jQuery(<selector>).css(<style3>); jQuery(<selector>) .css(<style1>) .css(<style2>) .css(<style3>);
14. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. 3. The little things visibility:hidden; > reduce render reflow display:none; jQuery(‘#elem’).detach(); > jQuery(‘#elem’).remove(); // batch DOM changes and perform them "offline" // avoid style triggered reflows more on reflows: http://www.phpied.com/rendering-repaint-reflowrelayout-restyle/
15. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. 3. The little things Use Experiment CSS .exp-wat123-var1 #elem:hover{ color: #bada55 } > jQuery(‘#elem’).hover( function(){ //mouseover }, function(){ //mouseout } ); jQuery(‘body’) .addClass(‘exp-wat123-var1’);
16. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. 4. Be aware of your source code / /products /ondemand/signup /ondemand/signup/form $(document).ready(); works magnolia $(document).ready(); doesn’t work AngularJS use history.pushState
17. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. 5. Embrace Namespaces & Closures keeps your variation code safe from your other parallel experiments from source code changes while experiment is running just good practice
18. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. 5. Embrace Namespaces & Closures ; // for safety from unclosed javascript expressions before var EXP = EXP || {}; //wac experiment namespace. EXP.wtf12918 = (function() { var init = function() { console.log("DEV LOG::building the experiment"); // to keep track of OPTIMIZELY's deployment variationScript(); }; var variationScript = function() { console.log("exp variation starts”); // your variation code goes here }; return { 'init': init }; })(); jQuery(document).ready(function() { var resetStyle = (function() { console.log("reset style"); jQuery('body').addClass('wtf12918'); })(); EXP.wtf12918.init(); }); full snippet at: https://gist.github.com/fb3aa18bad7a97ed5069.git
19. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. document.ready() … really?
20. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. document.ready() … really? “..premature optimization is the root of all evil”
21. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. document.ready() … really? —Donald Knuth: http://c2.com/cgi/wiki?PrematureOptimization “..premature optimization is the root of all evil”
22. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. 5. Some ugly hacks Make it slow /* _optimizely_evaluate=force */ document.getElementById(‘header’).style.visibility = ‘hidden’; // do changes /* _optimizely_evaluate=normal */ jQuery(‘document’).ready(function(){ window.setTimeout(function(){ jQuery("#header").css('visibility', 'visible'); }, 100); });
23. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. testing
24. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. 1. Cookies var_A var_B var_A var_B var_A var_B var_A var_B Live Experiment Experiment in QA qa_cookie=111 qa_cookie=222 qa_cookie=333 !qa_cookie
25. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. I have still not found an easy way to QA experiments Remember You can’t (or shouldn’t) really do a hotfix after starting
26. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. I have still not found an easy way to QA experiments Check control & variation probably ok to drop a browser. careful of the traffic loss. Check all the goals Note your office IPs could be blocked. ?optimizely_xEXPERIMENTID=VARIATIONINDEX ?optimizely_force_tracking=true
27. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. things I learned from optimizely source code
28. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. DATA in source code all active & draft experiments all active &draft variation code
29. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. DATA in source code all goals in active and draft exp
30. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. so… do regular housekeeping on your optimizely dashboard
31. © 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. Thanks to.. shreyas@atlassian.com | @seekshreyas Shreyas My wonderful colleagues at Atlassian The Live Event for hosting me You icons and pictograms used under creative commons license from: - Entypo pictograms by Daniel Bruce - Round Icons from Smashing Magazine