DIY Synthetic: Private WebPagetest Magic

I gave this talk at Velocity NY 2013, and it covers the installation and use of a private WebPagetest instance, as well as the open source tool wpt-script which Etsy built.

Transcript

1. DIY Synthetic: Private WebPagetest Magic Velocity NY 2013 Jonathan Klein @jonathanklein Wednesday, October 16, 13
2. Slides, Links jkle.in/wpt Wednesday, October 16, 13
3. About Me Wednesday, October 16, 13
4. About Me • Performance Engineer At Etsy Wednesday, October 16, 13
5. About Me • Performance Engineer At Etsy • I write the Etsy Site Performance Reports Wednesday, October 16, 13
6. About Me • Performance Engineer At Etsy • I write the Etsy Site Performance Reports • Organize Boston Web Perf Meetup Group Wednesday, October 16, 13
7. Etsy Stats Wednesday, October 16, 13
8. Etsy Stats • World’s largest handmade marketplace Wednesday, October 16, 13
9. Etsy Stats • World’s largest handmade marketplace • 1.5 billion page views/month Wednesday, October 16, 13
10. Etsy Stats • World’s largest handmade marketplace • 1.5 billion page views/month • Almost $1B in sales last year Wednesday, October 16, 13
11. +160k to a page = +12% bounce rate on mobile Wednesday, October 16, 13
12. Synthetic Monitoring Wednesday, October 16, 13
13. Wednesday, October 16, 13
14. Synthetic Monitoring Real User Monitoring Browser Instrumentation Navigation Timing API Consistent trending over time Highly variable Largely in your control Last mile difficulties Great for identifying regressions Great for comparing across geographies/browsers Not super realistic “Real User Monitoring” Wednesday, October 16, 13
15. Synthetic Monitoring Real User Monitoring Browser Instrumentation Navigation Timing API Consistent trending over time Highly variable Largely in your control Last mile difficulties Great for identifying regressions Great for comparing across geographies/browsers Not super realistic “Real User Monitoring” Wednesday, October 16, 13
16. Synthetic Monitoring Real User Monitoring Browser Instrumentation Navigation Timing API Consistent trending over time Highly variable Largely in your control Last mile difficulties Great for identifying regressions Great for comparing across geographies/browsers Not super realistic “Real User Monitoring” Wednesday, October 16, 13
17. Synthetic Monitoring Real User Monitoring Browser Instrumentation Navigation Timing API Consistent trending over time Highly variable Largely in your control Last mile difficulties Great for identifying regressions Great for comparing across geographies/browsers Not super realistic “Real User Monitoring” Wednesday, October 16, 13
18. Synthetic Monitoring Real User Monitoring Browser Instrumentation Navigation Timing API Consistent trending over time Highly variable Largely in your control Last mile difficulties Great for identifying regressions Great for comparing across geographies/browsers Not super realistic “Real User Monitoring” Wednesday, October 16, 13
19. Synthetic Monitoring Real User Monitoring Browser Instrumentation Navigation Timing API Consistent trending over time Highly variable Largely in your control Last mile difficulties Great for identifying regressions Great for comparing across geographies/browsers Not super realistic “Real User Monitoring” Wednesday, October 16, 13
20. Wednesday, October 16, 13
21. $$$$ Wednesday, October 16, 13
22. Wednesday, October 16, 13
23. Background Wednesday, October 16, 13
24. Wednesday, October 16, 13
25. Wednesday, October 16, 13
26. Wednesday, October 16, 13
27. EC2 Test Agents Wednesday, October 16, 13
28. Wednesday, October 16, 13
29. No Built-in Automation Wednesday, October 16, 13
30. Wednesday, October 16, 13
31. Wednesday, October 16, 13
32. wpt-script Wednesday, October 16, 13
33. Wednesday, October 16, 13
34. Motivation Wednesday, October 16, 13
35. Why We Use It Wednesday, October 16, 13
36. Why We Use It • Wanted long term synthetic trending Wednesday, October 16, 13
37. Why We Use It • Wanted long term synthetic trending • Automated tests Wednesday, October 16, 13
38. Why We Use It • Wanted long term synthetic trending • Automated tests • More control Wednesday, October 16, 13
39. Why We Use It • Wanted long term synthetic trending • Automated tests • More control • Low cost Wednesday, October 16, 13
40. Easy Configuration Wednesday, October 16, 13
41. { "server": "http://www.yourwptinstall.com", "pending_dir": "private", "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"], "urls": { "label": "http://www.example.com", "other_label": "http://www.anotherexample.com" }, "graphite": "graphite.example.com", "logging_ns": "webpagetest.private", "splunkLog": "/var/log/webpagetest/results.log" } Wednesday, October 16, 13
42. { "server": "http://www.yourwptinstall.com", "pending_dir": "private", "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"], "urls": { "label": "http://www.example.com", "other_label": "http://www.anotherexample.com" }, "graphite": "graphite.example.com", "logging_ns": "webpagetest.private", "splunkLog": "/var/log/webpagetest/results.log" } Wednesday, October 16, 13
43. { "server": "http://www.yourwptinstall.com", "pending_dir": "private", "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"], "urls": { "label": "http://www.example.com", "other_label": "http://www.anotherexample.com" }, "graphite": "graphite.example.com", "logging_ns": "webpagetest.private", "splunkLog": "/var/log/webpagetest/results.log" } Wednesday, October 16, 13
44. { "server": "http://www.yourwptinstall.com", "pending_dir": "private", "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"], "urls": { "label": "http://www.example.com", "other_label": "http://www.anotherexample.com" }, "graphite": "graphite.example.com", "logging_ns": "webpagetest.private", "splunkLog": "/var/log/webpagetest/results.log" } Wednesday, October 16, 13
45. { "server": "http://www.yourwptinstall.com", "pending_dir": "private", "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"], "urls": { "label": "http://www.example.com", "other_label": "http://www.anotherexample.com" }, "graphite": "graphite.example.com", "logging_ns": "webpagetest.private", "splunkLog": "/var/log/webpagetest/results.log" } Wednesday, October 16, 13
46. # Run Test */25 * * * * php run.php Wednesday, October 16, 13
47. # Run Test */25 * * * * php run.php -c foo.conf Wednesday, October 16, 13
48. # Run Test */25 * * * * php run.php -c foo.conf # Get Results * * * * * php get_results.php Wednesday, October 16, 13
49. Advanced Features Wednesday, October 16, 13
50. { "server": "http://example.com", "pending_dir": "private", "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"], "urls": { "label": "http://www.example.com", "other_label": "http://www.webpagetest.org" }, "graphite": "graphite.example.com", "logging_ns": "webpagetest.private", "splunkLog": "/var/log/webpagetest/results.log", "prepend": "Login", "username": "someuser", "password": "somepassword", "run_options": { "video": 0 } } Wednesday, October 16, 13
51. { "server": "http://example.com", "pending_dir": "private", "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"], "urls": { "label": "http://www.example.com", "other_label": "http://www.webpagetest.org" }, "graphite": "graphite.example.com", "logging_ns": "webpagetest.private", "splunkLog": "/var/log/webpagetest/results.log", "prepend": ["BlockThirdParty", "bypass_cdn.txt"], "run_options": { "video": 0 } } Wednesday, October 16, 13
52. foreach ($prepends as $prepend) { if (file_exists($scripts_dir . $prepend)) { $script .= file_get_contents($scripts_dir . $prepend); } elseif (method_exists($this, $prepend)) { $script .= $this->$prepend(); } } Wednesday, October 16, 13
53. setDns site.etsystatic.com 38.106.64.123 setDns img0.etsystatic.com 38.106.64.125 setDns img1.etsystatic.com 38.106.64.125 setDns img2.etsystatic.com 38.106.64.125 setDns img3.etsystatic.com 38.106.64.125 setDns ny-image0.etsy.com 38.106.64.125 setDns ny-image1.etsy.com 38.106.64.125 setDns ny-image2.etsy.com 38.106.64.125 setDns ny-image3.etsy.com 38.106.64.125 bypass_cdn.txt Wednesday, October 16, 13
54. Gathering Results Wednesday, October 16, 13
55. <?php // Splunk $splunkLogger = new SplunkLogger($config['splunkLog'], $logging_ns); foreach ($results as $result) { $splunkLogger->log($result); } // Graphite $grapher = new Grapher($config['graphite'], $logging_ns); $grapher->graphResults($results); Wednesday, October 16, 13
56. Wednesday, October 16, 13
57. Wednesday, October 16, 13
58. Wednesday, October 16, 13
59. https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/metrics Wednesday, October 16, 13
60. Wednesday, October 16, 13
61. Wednesday, October 16, 13
62. Wednesday, October 16, 13
63. Disadvantages vs. Paid Tools Wednesday, October 16, 13
64. Disadvantages vs. Paid Tools • Harder to set up/maintain Wednesday, October 16, 13
65. Disadvantages vs. Paid Tools • Harder to set up/maintain • Functionality Wednesday, October 16, 13
66. Disadvantages vs. Paid Tools • Harder to set up/maintain • Functionality • Regions Wednesday, October 16, 13
67. Disadvantages vs. Paid Tools • Harder to set up/maintain • Functionality • Regions • Full page oriented Wednesday, October 16, 13
68. Disadvantages vs. Paid Tools • Harder to set up/maintain • Functionality • Regions • Full page oriented • Less consistent Wednesday, October 16, 13
69. Advantages vs. Paid Tools Wednesday, October 16, 13
70. Advantages vs. Paid Tools • Cost Wednesday, October 16, 13
71. Advantages vs. Paid Tools • Cost • Flexibility Wednesday, October 16, 13
72. Advantages vs. Paid Tools • Cost • Flexibility • People know the tool Wednesday, October 16, 13
73. Advantages vs. Paid Tools • Cost • Flexibility • People know the tool • Multi-page flows Wednesday, October 16, 13
74. Advantages vs. Paid Tools • Cost • Flexibility • People know the tool • Multi-page flows • Scripting Wednesday, October 16, 13
75. Our Use Cases Wednesday, October 16, 13
76. WebPagetest Catchpoint Wednesday, October 16, 13
77. WebPagetest Catchpoint • Multi-page flows Wednesday, October 16, 13
78. WebPagetest Catchpoint • Multi-page flows • Scripted tests Wednesday, October 16, 13
79. WebPagetest Catchpoint • Multi-page flows • Scripted tests • High volume testing Wednesday, October 16, 13
80. WebPagetest Catchpoint • Multi-page flows • Scripted tests • High volume testing • Ad-hoc tests Wednesday, October 16, 13
81. WebPagetest Catchpoint • Multi-page flows • Scripted tests • High volume testing • Ad-hoc tests • DNS Wednesday, October 16, 13
82. WebPagetest Catchpoint • Multi-page flows • Scripted tests • High volume testing • Ad-hoc tests • DNS • API Wednesday, October 16, 13
83. WebPagetest Catchpoint • Multi-page flows • Scripted tests • High volume testing • Ad-hoc tests • DNS • API • Single Object Tests Wednesday, October 16, 13
84. WebPagetest Catchpoint • Multi-page flows • Scripted tests • High volume testing • Ad-hoc tests • DNS • API • Single Object Tests • Multiple locations Wednesday, October 16, 13
85. WebPagetest Catchpoint • Multi-page flows • Scripted tests • High volume testing • Ad-hoc tests • DNS • API • Single Object Tests • Multiple locations • Comparisons Wednesday, October 16, 13
86. Future Work Wednesday, October 16, 13
87. Improving the Process Wednesday, October 16, 13
88. Improving the Process • Manage disk space Wednesday, October 16, 13
89. Improving the Process • Manage disk space • Multiple regions Wednesday, October 16, 13
90. Improving the Process • Manage disk space • Multiple regions • More canned scripts Wednesday, October 16, 13
91. Improving the Process • Manage disk space • Multiple regions • More canned scripts • Automate upgrades Wednesday, October 16, 13
92. Leveraging WebPagetest Wednesday, October 16, 13
93. Leveraging WebPagetest • Front-end SPOF Wednesday, October 16, 13
94. Leveraging WebPagetest • Front-end SPOF • User Timings (AFT) Wednesday, October 16, 13
95. Leveraging WebPagetest • Front-end SPOF • User Timings (AFT) • Speed Index Wednesday, October 16, 13
96. Wednesday, October 16, 13
97. Improving WebPagetest Wednesday, October 16, 13
98. WebPagetest + Vagrant Wednesday, October 16, 13
99. WebPagetest + Vagrant • Install VirtualBox Wednesday, October 16, 13
100. WebPagetest + Vagrant • Install VirtualBox • Install Vagrant Wednesday, October 16, 13
101. WebPagetest + Vagrant • Install VirtualBox • Install Vagrant • https://gist.github.com/jklein/6992296 Wednesday, October 16, 13
102. WebPagetest + Vagrant • Install VirtualBox • Install Vagrant • https://gist.github.com/jklein/6992296 •vagrant up Wednesday, October 16, 13
103. WebPagetest + Vagrant • Install VirtualBox • Install Vagrant • https://gist.github.com/jklein/6992296 •vagrant up • http://localhost:8080 Wednesday, October 16, 13
104. Wednesday, October 16, 13
105. 800MB Wednesday, October 16, 13
106. webpagetest-api Wednesday, October 16, 13
107. Wednesday, October 16, 13
108. Both Are Good Options Wednesday, October 16, 13
109. Results > Tools Wednesday, October 16, 13
110. Thanks! Wednesday, October 16, 13
111. @jonathanklein jonathan@etsy.com www.etsy.com/careers Get in Touch Wednesday, October 16, 13