A/B Testing with Metriks

Quick lightning talk about using metriks to run a dead simple performance A/B test.

Transcript

1. Story Time Monday, October 1, 12
2. Story Time Monday, October 1, 12
3. Lazy Metrics <script>var start = new Date();</script> <img src="<%= content_url %>"> <script> $('img').load(function() { recordMetric("image-load", new Date() - start); }); </script> Monday, October 1, 12
4. Lazy Metrics <script>var start = new Date();</script> <img src="<%= content_url %>"> <script> $('img').load(function() { recordMetric("image-load", new Date() - start); }); </script> Monday, October 1, 12
5. Lazy Metrics <script>var start = new Date();</script> <img src="<%= content_url %>"> <script> $('img').load(function() { recordMetric("image-load", new Date() - start); }); </script> Monday, October 1, 12
6. Server # /metrics?name=image-load&value=123.45 get '/metrics' do name = params['name'] value = params['value'] Metriks.timer(name).update(value) content_type 'text/javascript' status 200 end Monday, October 1, 12
7. Server # /metrics?name=image-load&value=123.45 get '/metrics' do name = params['name'] value = params['value'] Metriks.timer(name).update(value) content_type 'text/javascript' status 200 end Monday, October 1, 12
8. Server # /metrics?name=image-load&value=123.45 get '/metrics' do name = params['name'] value = params['value'] Metriks.timer(name).update(value) content_type 'text/javascript' status 200 end Monday, October 1, 12
9. Client Side Monday, October 1, 12
10. Story Time Monday, October 1, 12
11. Story Time Monday, October 1, 12
12. Lazy A/B Test <% if fast = [true, false].sample %> <img src="<%= fast_content_url %>"> <% else %> <img src="<%= content_url %>"> <% end %> Monday, October 1, 12
13. Lazy A/B Test <script> $('img').load(function() { name = "image-load<%= fast ? '-test' : '' %>" recordMetric(name, new Date() - start); }); </script> Monday, October 1, 12
14. Lazy A/B Test <script> $('img').load(function() { name = "image-load<%= fast ? '-test' : '' %>" recordMetric(name, new Date() - start); }); </script> Monday, October 1, 12
15. Lazy A/B Test 10.5s 6.6s Monday, October 1, 12
16. New Relic 37ms Monday, October 1, 12
17. metriks 14ms 110ms Monday, October 1, 12
18. gem install metriks Thank me later. Monday, October 1, 12