Making Maps: The Role of Frontend Infrastructure at Etsy

Video: http://vimeo.com/109912254
Given October 9th at Fronteers 2014 in Amsterdam.

The scope of Etsy's frontend is massive - our JavaScript codebase has grown by 50% in the last year to more than 3,000 files totaling almost 800,000 lines. Because of Etsy's dedication to continuous deployment, the code running in production changes 25 to 50 times every day. Because of our experimentation-driven development cycle, there may be multiple production versions of our features at any given time. These factors can lead to uncertainty and fear with rolling out upgrades, deleting old code, or confidently making changes. I share how the Frontend Infrastructure team works within this ecosystem to mitigate those risks, manage the asset build pipeline, build tools to understand our frontend, automate migrations & delete as much code as possible.

Transcript

1. Daniel Espeset @danielespeset Fronteers Amsterdam, October 2014 The Role of Frontend Infrastructure at Etsy Making Maps
2. talks.desp.in/fronteers2014 Resources for this talk
3. Daniel Espeset @danielespeset Frontend Infrastructure 3 Daniel Espeset Daniel Na Takashi Mizohata Etsy Seth Walker Sufian Rhazi
5. Daniel Espeset @danielespeset 1 Million Active Sellers 26 Million Active Listings 5
6. 6 Etsy Engineering
7. Daniel Espeset @danielespeset 7 Anybody can touch anything
8. Daniel Espeset @danielespeset 8 Blameless postmortems
9. Daniel Espeset @danielespeset 9 Everyone deploys
10. 10 Continuous Deployment
11. Daniel Espeset @danielespeset "The Button"
12. pages/search_results.php if (Feature::isEnabled('my_new_feature')) { // TODO build this part } else { // Existing code } 20 21 22 .. 100 12 config/production.php // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = ( 'enabled' => 0 ); 890 891 892 893
13. 13 desp .join pushbot has changed topic to: "<prod> seth dna* gio* | desp" pushbot desp jklein : You're up desp .in pushbot has changed topic to: "desp jklein" pushbot has changed topic to: "desp* jklein" jklein .in pushbot has changed topic to: "desp* jklein*" pushbot desp jklein : everyone is ready
19. 19 25-50Pushes a day
20. 20 config/production.php // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = ( 'enabled' => 0, 'group' => 9903 ); 890 891 892 893 894
21. 21 config/production.php // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = ( 'enabled' => 0, 'group' => 9903, 'admin' => 'on' ); 890 891 892 893 894 895
22. 22 config/production.php // Owner: despeset, frontend infrastructure server_config['my_new_feature'] = ( 'enabled' => 50, 'group' => 9903, 'admin' => 'on' ); 890 891 892 893 894 895
23. 23 Continuous Experimentation
24. Daniel Espeset @danielespeset 24 idea code release idea code release? A/B test idea prototype release A/B test validate A/B test refine 2005 2010 2012 Shamelessly stolen from Dan McKinley mcfunley.com/data-driven-products-now Evolution of Continuous Experimentation
25. Daniel Espeset @danielespeset 25 • Engineers are trusted and have massive access • The production codebase is constantly changing • That codebase is filled with experiments TL;DR
26. Daniel Espeset @danielespeset 26
27. Daniel Espeset @danielespeset 27 Our frontend is made up of 300
    different “pages”
28. Daniel Espeset @danielespeset 28 At any given time we may be running dozens of experiments per page
29. Daniel Espeset @danielespeset 29 4,096 212 =
30. Daniel Espeset @danielespeset 30 212 300 x = 1.2 M
31. Daniel Espeset @danielespeset 31 Reasoning about our frontend can be hard
32. Daniel Espeset @danielespeset 32 Deleting code is often scary
33. Daniel Espeset @danielespeset 33 Testing changes is difficult
34. 34 Frontend Infrastructure
35. Daniel Espeset @danielespeset 35 source code deploy build system
36. Daniel Espeset @danielespeset 36 source code production build deploy build system
37. Daniel Espeset @danielespeset 37 Frontend Infrastructure
38. 38 Builda [Build Assets]
39. Daniel Espeset @danielespeset • • Wraps RequireJS AMD compiler • Parallelizes work across CPU cores • Serves fully compiled assets on development VMS 39 Builda [Build Assets]
40. Daniel Espeset @danielespeset 40 Techniques for serving built assets in dev • Build on the fly in response to HTTP requests for asset files
41. Daniel Espeset @danielespeset 41 Techniques for serving built assets in dev • Build on the fly in response to HTTP requests for asset files • Watch FS for any changes and rebuild everything
42. Daniel Espeset @danielespeset 42 Techniques for serving built assets in dev • Build on the fly in response to HTTP requests for asset files • Watch FS for any changes and rebuild everything • Watch FS for any changes and only rebuild affected targets
43. Daniel Espeset @danielespeset 43 Techniques for serving built assets in dev • Build on the fly in response to HTTP requests for asset files • Watch FS for any changes and rebuild everything • Watch FS for any changes and only rebuild affected targets this is... complicated
44. Daniel Espeset @danielespeset 44 source code built assets Builda Source file is changed
45. Daniel Espeset @danielespeset 45 source code built assets Builda Builda looks up dependencies
46. Daniel Espeset @danielespeset 46 source code built assets Builda Only rebuilds files that are now stale
47. Daniel Espeset @danielespeset This solution gave us something valuable 47 A dependency graph
48. 48 dependency_graph.json "mobile/modules/rwd": { "info": { "version": "20140815195505" }, "children": { "common/etsy.namespace": "amd" }, "parents": { "mobile/base-container": "amd" } } 201 202 203 204 205 206 207 208 209 210 211
49. 49 Ranger
50. Daniel Espeset @danielespeset 50 + + JS graph render graph CSS graph Ranger combines 3 dependency graphs
59. Daniel Espeset @danielespeset Ranger gave us something valuable 59 The set of files that aren't used
60. 60 Shrinkray
61. Daniel Espeset @danielespeset • Pick a stylesheet loaded on the page 61 Shrinkray - clientside CSS analysis
62. Daniel Espeset @danielespeset • Pick a stylesheet loaded on the page • Randomly pick 50 selectors via CSSOM 62 Shrinkray - clientside CSS analysis
63. Daniel Espeset @danielespeset • Pick a stylesheet loaded on the page • Randomly pick 50 selectors via CSSOM • Search for matches with document.querySelector 63 Shrinkray - clientside CSS analysis
64. Daniel Espeset @danielespeset • Pick a stylesheet loaded on the page • Randomly pick 50 selectors via CSSOM • Search for matches with document.querySelector • Send results to server 64 Shrinkray - clientside CSS analysis
65. Daniel Espeset @danielespeset • Pick a stylesheet loaded on the page • Randomly pick 50 selectors via CSSOM • Search for matches with document.querySelector • Send results to server • Aggregate data with Map Reduce 65 Shrinkray - clientside CSS analysis
67. 67 Moving to SCSS
68. Daniel Espeset @danielespeset 68 Existing preprocessor inlines @import, versions images
69. Daniel Espeset @danielespeset 69 Linted for syntax (sorta)
70. 70 Every color defined in our CSS
71. Daniel Espeset @danielespeset • No @extends ever • No @mixins (for now) • Nesting level < 4 • Lints to enforce these and strict code-style adherence 71 Restricting SCSS functionality
72. Daniel Espeset @danielespeset • @import statements needed changed • Converter fixes 171,244 scss-lint errors • Had to resolve hundreds of invalid CSS rules & selectors 72 Converter script
73. Daniel Espeset @danielespeset 73 Will fixing this break something? color: rgb(25555, -10, 0);
74. Daniel Espeset @danielespeset 74 What about this one? color: rgb(0, 0, 0, 0);
75. Daniel Espeset @danielespeset 75 CSS Source Legacy CSS Build Legacy Builda
76. Daniel Espeset @danielespeset 76 CSS Source Legacy CSS Build Legacy Builda Converter SCSS Builda New CSS Build
77. Daniel Espeset @danielespeset 77 CSS Source Legacy CSS Build Legacy Builda Converter SCSS Builda New CSS Build AST Differ Parse to AST Parse to AST
78. Daniel Espeset @danielespeset 78 CSS Source Legacy CSS Build Legacy Builda Converter SCSS Builda 100% New CSS Build
79. Daniel Espeset @danielespeset 79 CSS Source Legacy CSS Build Legacy Builda Converter SCSS Builda 100% Etsy New CSS Build
80. Daniel Espeset @danielespeset 80 CSS Source Legacy CSS Build Legacy Builda Converter SCSS Builda 50% 50% New CSS Build
81. Daniel Espeset @danielespeset 81 CSS Source Legacy CSS Build New CSS Build Legacy Builda Converter SCSS Builda 100%
82. Daniel Espeset @danielespeset 82 SCSS Source New CSS Build SCSS Builda 100%
83. What have we learned?
84. Daniel Espeset @danielespeset 84 Disposability > Modularity
85. Daniel Espeset @danielespeset 85 The frontend is getting more complex everywhere.
86. Daniel Espeset @danielespeset 86 We need new strategies to manage and understand it.
87. Daniel Espeset @danielespeset 87 The browser is part of your distributed system, not just a client you support.
88. Daniel Espeset @danielespeset 88 The more extensible your tool the more useful it will be.