HubSpot Tech Talk - DIY Synthetic

A talk I gave at a HubSpot tech talk in Boston.

Transcript

1. DIY Synthetic: Private WebPagetest Magic HubSpot Tech Talk Jonathan Klein @jonathanklein Wednesday, December 11, 13
2. Slides, Links jkle.in/hubspot Wednesday, December 11, 13
3. About Me Wednesday, December 11, 13
4. About Me • Performance Engineer At Etsy Wednesday, December 11, 13
5. About Me • Performance Engineer At Etsy • I write the Etsy Site Performance Reports Wednesday, December 11, 13
6. About Me • Performance Engineer At Etsy • I write the Etsy Site Performance Reports • Organize Boston Web Perf Meetup Group Wednesday, December 11, 13
7. Etsy Stats Wednesday, December 11, 13
8. Etsy Stats • World’s largest handmade marketplace Wednesday, December 11, 13
9. Etsy Stats • World’s largest handmade marketplace • 1.5 billion page views/month Wednesday, December 11, 13
10. Etsy Stats • World’s largest handmade marketplace • 1.5 billion page views/month • Almost $1B in sales last year Wednesday, December 11, 13
11. +160k to a page = +12% bounce rate on mobile Wednesday, December 11, 13
12. Synthetic Monitoring Wednesday, December 11, 13
13. Wednesday, December 11, 13
14. Synthetic Monitoring Real User Monitoring Browser Instrumentation Navigation Timing API Consistent trending over time Highly variable Largely in your control Last mile difficulties Great for identifying regressions Great for comparing across geographies/browsers Not super realistic “Real User Monitoring” Wednesday, December 11, 13
15. Synthetic Monitoring Real User Monitoring Browser Instrumentation Navigation Timing API Consistent trending over time Highly variable Largely in your control Last mile difficulties Great for identifying regressions Great for comparing across geographies/browsers Not super realistic “Real User Monitoring” Wednesday, December 11, 13
16. Synthetic Monitoring Real User Monitoring Browser Instrumentation Navigation Timing API Consistent trending over time Highly variable Largely in your control Last mile difficulties Great for identifying regressions Great for comparing across geographies/browsers Not super realistic “Real User Monitoring” Wednesday, December 11, 13
17. Synthetic Monitoring Real User Monitoring Browser Instrumentation Navigation Timing API Consistent trending over time Highly variable Largely in your control Last mile difficulties Great for identifying regressions Great for comparing across geographies/browsers Not super realistic “Real User Monitoring” Wednesday, December 11, 13
18. Synthetic Monitoring Real User Monitoring Browser Instrumentation Navigation Timing API Consistent trending over time Highly variable Largely in your control Last mile difficulties Great for identifying regressions Great for comparing across geographies/browsers Not super realistic “Real User Monitoring” Wednesday, December 11, 13
19. Synthetic Monitoring Real User Monitoring Browser Instrumentation Navigation Timing API Consistent trending over time Highly variable Largely in your control Last mile difficulties Great for identifying regressions Great for comparing across geographies/browsers Not super realistic “Real User Monitoring” Wednesday, December 11, 13
20. Wednesday, December 11, 13
21. $$$$ Wednesday, December 11, 13
22. Wednesday, December 11, 13
23. Background Wednesday, December 11, 13
24. Wednesday, December 11, 13
25. Wednesday, December 11, 13
26. Wednesday, December 11, 13
27. EC2 Test Agents Wednesday, December 11, 13
28. Wednesday, December 11, 13
29. No Built-in Automation Wednesday, December 11, 13
30. Wednesday, December 11, 13
31. Wednesday, December 11, 13
32. wpt-script Wednesday, December 11, 13
33. Wednesday, December 11, 13
34. Motivation Wednesday, December 11, 13
35. Why We Use It Wednesday, December 11, 13
36. Why We Use It • Wanted long term synthetic trending Wednesday, December 11, 13
37. Why We Use It • Wanted long term synthetic trending • Automated tests Wednesday, December 11, 13
38. Why We Use It • Wanted long term synthetic trending • Automated tests • More control Wednesday, December 11, 13
39. Why We Use It • Wanted long term synthetic trending • Automated tests • More control • Low cost Wednesday, December 11, 13
40. Easy Configuration Wednesday, December 11, 13
41. { "server": "http://www.yourwptinstall.com", "pending_dir": "private", "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"], "urls": { "label": "http://www.example.com", "other_label": "http://www.anotherexample.com" }, "graphite": "graphite.example.com", "logging_ns": "webpagetest.private", "splunkLog": "/var/log/webpagetest/results.log" } Wednesday, December 11, 13
42. { "server": "http://www.yourwptinstall.com", "pending_dir": "private", "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"], "urls": { "label": "http://www.example.com", "other_label": "http://www.anotherexample.com" }, "graphite": "graphite.example.com", "logging_ns": "webpagetest.private", "splunkLog": "/var/log/webpagetest/results.log" } Wednesday, December 11, 13
43. { "server": "http://www.yourwptinstall.com", "pending_dir": "private", "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"], "urls": { "label": "http://www.example.com", "other_label": "http://www.anotherexample.com" }, "graphite": "graphite.example.com", "logging_ns": "webpagetest.private", "splunkLog": "/var/log/webpagetest/results.log" } Wednesday, December 11, 13
44. { "server": "http://www.yourwptinstall.com", "pending_dir": "private", "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"], "urls": { "label": "http://www.example.com", "other_label": "http://www.anotherexample.com" }, "graphite": "graphite.example.com", "logging_ns": "webpagetest.private", "splunkLog": "/var/log/webpagetest/results.log" } Wednesday, December 11, 13
45. { "server": "http://www.yourwptinstall.com", "pending_dir": "private", "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"], "urls": { "label": "http://www.example.com", "other_label": "http://www.anotherexample.com" }, "graphite": "graphite.example.com", "logging_ns": "webpagetest.private", "splunkLog": "/var/log/webpagetest/results.log" } Wednesday, December 11, 13
46. # Run Test */25 * * * * php run.php Wednesday, December 11, 13
47. # Run Test */25 * * * * php run.php -c foo.conf Wednesday, December 11, 13
48. # Run Test */25 * * * * php run.php -c foo.conf # Get Results * * * * * php get_results.php Wednesday, December 11, 13
49. Advanced Features Wednesday, December 11, 13
50. { "server": "http://example.com", "pending_dir": "private", "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"], "urls": { "label": "http://www.example.com", "other_label": "http://www.webpagetest.org" }, "graphite": "graphite.example.com", "logging_ns": "webpagetest.private", "splunkLog": "/var/log/webpagetest/results.log", "prepend": "Login", "username": "someuser", "password": "somepassword", "run_options": { "video": 0 } } Wednesday, December 11, 13
51. { "server": "http://example.com", "pending_dir": "private", "locations": ["US_East_wptdriver:Firefox", "US_East_wptdriver:Chrome"], "urls": { "label": "http://www.example.com", "other_label": "http://www.webpagetest.org" }, "graphite": "graphite.example.com", "logging_ns": "webpagetest.private", "splunkLog": "/var/log/webpagetest/results.log", "prepend": ["BlockThirdParty", "bypass_cdn.txt"], "run_options": { "video": 0 } } Wednesday, December 11, 13
52. foreach ($prepends as $prepend) { if (file_exists($scripts_dir . $prepend)) { $script .= file_get_contents($scripts_dir . $prepend); } elseif (method_exists($this, $prepend)) { $script .= $this->$prepend(); } } Wednesday, December 11, 13
53. setDns site.etsystatic.com 38.106.64.123 setDns img0.etsystatic.com 38.106.64.125 setDns img1.etsystatic.com 38.106.64.125 setDns ny-image0.etsy.com 38.106.64.125 setDns ny-image1.etsy.com 38.106.64.125 bypass_cdn.txt Wednesday, December 11, 13
54. Gathering Results Wednesday, December 11, 13
55. <?php // Splunk $splunkLogger = new SplunkLogger($config['splunkLog'], $logging_ns); foreach ($results as $result) { $splunkLogger->log($result); } // Graphite $grapher = new Grapher($config['graphite'], $logging_ns); $grapher->graphResults($results); Wednesday, December 11, 13
56. Wednesday, December 11, 13
57. Wednesday, December 11, 13
58. Wednesday, December 11, 13
59. https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/metrics Wednesday, December 11, 13
60. Wednesday, December 11, 13
61. Wednesday, December 11, 13
62. Wednesday, December 11, 13
63. Disadvantages vs. Paid Tools Wednesday, December 11, 13
64. Disadvantages vs. Paid Tools • Harder to set up/maintain Wednesday, December 11, 13
65. Disadvantages vs. Paid Tools • Harder to set up/maintain • Functionality Wednesday, December 11, 13
66. Disadvantages vs. Paid Tools • Harder to set up/maintain • Functionality • Regions Wednesday, December 11, 13
67. Disadvantages vs. Paid Tools • Harder to set up/maintain • Functionality • Regions • Full page oriented Wednesday, December 11, 13
68. Disadvantages vs. Paid Tools • Harder to set up/maintain • Functionality • Regions • Full page oriented • Less consistent Wednesday, December 11, 13
69. Advantages vs. Paid Tools Wednesday, December 11, 13
70. Advantages vs. Paid Tools • Cost Wednesday, December 11, 13
71. Advantages vs. Paid Tools • Cost • Flexibility Wednesday, December 11, 13
72. Advantages vs. Paid Tools • Cost • Flexibility • People know the tool Wednesday, December 11, 13
73. Advantages vs. Paid Tools • Cost • Flexibility • People know the tool • Multi-page flows Wednesday, December 11, 13
74. Advantages vs. Paid Tools • Cost • Flexibility • People know the tool • Multi-page flows • Scripting Wednesday, December 11, 13
75. Our Use Cases Wednesday, December 11, 13
76. WebPagetest Catchpoint Wednesday, December 11, 13
77. WebPagetest Catchpoint • Multi-page flows Wednesday, December 11, 13
78. WebPagetest Catchpoint • Multi-page flows • Scripted tests Wednesday, December 11, 13
79. WebPagetest Catchpoint • Multi-page flows • Scripted tests • High volume testing Wednesday, December 11, 13
80. WebPagetest Catchpoint • Multi-page flows • Scripted tests • High volume testing • Ad-hoc tests Wednesday, December 11, 13
81. WebPagetest Catchpoint • Multi-page flows • Scripted tests • High volume testing • Ad-hoc tests • DNS Wednesday, December 11, 13
82. WebPagetest Catchpoint • Multi-page flows • Scripted tests • High volume testing • Ad-hoc tests • DNS • API Wednesday, December 11, 13
83. WebPagetest Catchpoint • Multi-page flows • Scripted tests • High volume testing • Ad-hoc tests • DNS • API • Single Object Tests Wednesday, December 11, 13
84. WebPagetest Catchpoint • Multi-page flows • Scripted tests • High volume testing • Ad-hoc tests • DNS • API • Single Object Tests • Multiple locations Wednesday, December 11, 13
85. WebPagetest Catchpoint • Multi-page flows • Scripted tests • High volume testing • Ad-hoc tests • DNS • API • Single Object Tests • Multiple locations • Comparisons Wednesday, December 11, 13
86. Future Work Wednesday, December 11, 13
87. Improving the Process Wednesday, December 11, 13
88. Improving the Process • Manage disk space Wednesday, December 11, 13
89. Improving the Process • Manage disk space • Multiple regions Wednesday, December 11, 13
90. Improving the Process • Manage disk space • Multiple regions • More canned scripts Wednesday, December 11, 13
91. Improving the Process • Manage disk space • Multiple regions • More canned scripts • Automate upgrades Wednesday, December 11, 13
92. Leveraging WebPagetest Wednesday, December 11, 13
93. Leveraging WebPagetest • Front-end SPOF Wednesday, December 11, 13
94. Leveraging WebPagetest • Front-end SPOF • User Timings Wednesday, December 11, 13
95. Leveraging WebPagetest • Front-end SPOF • User Timings • Speed Index Wednesday, December 11, 13
96. Wednesday, December 11, 13
97. Speed Index Wednesday, December 11, 13
98. Speed Index • Synthetic only Wednesday, December 11, 13
99. Speed Index • Synthetic only • Used heavily inside Google Wednesday, December 11, 13
100. Speed Index • Synthetic only • Used heavily inside Google • The best metric for perceived load time Wednesday, December 11, 13
101. Speed Index • Synthetic only • Used heavily inside Google • The best metric for perceived load time • (this is what we care about) Wednesday, December 11, 13
102. Improving WebPagetest Wednesday, December 11, 13
103. WebPagetest + Vagrant Wednesday, December 11, 13
104. WebPagetest + Vagrant • Install VirtualBox Wednesday, December 11, 13
105. WebPagetest + Vagrant • Install VirtualBox • Install Vagrant Wednesday, December 11, 13
106. WebPagetest + Vagrant • Install VirtualBox • Install Vagrant • https://gist.github.com/jklein/6992296 Wednesday, December 11, 13
107. WebPagetest + Vagrant • Install VirtualBox • Install Vagrant • https://gist.github.com/jklein/6992296 •vagrant up Wednesday, December 11, 13
108. WebPagetest + Vagrant • Install VirtualBox • Install Vagrant • https://gist.github.com/jklein/6992296 •vagrant up • http://localhost:8080 Wednesday, December 11, 13
109. Wednesday, December 11, 13
110. 800MB Wednesday, December 11, 13
111. WebPagetest Relay -- locations.ini [WPT_Dulles_IE8] browser=IE 8 label="WebPagetest.org Dulles, VA - IE8" relayServer="http://www.webpagetest.org/" relayKey=<your API key> relayLocation=Dulles_IE8 Wednesday, December 11, 13
112. webpagetest-api Wednesday, December 11, 13
113. Wednesday, December 11, 13
114. Both Are Good Options Wednesday, December 11, 13
115. Results > Tools Wednesday, December 11, 13
116. Thanks! Wednesday, December 11, 13
117. @jonathanklein jonathan@etsy.com www.meetup.com/Web-Performance-Boston/ Get in Touch Wednesday, December 11, 13