JSCONF 2019: WE STARTED USING WEBPACK AND IT TOOK A WHILE

There are so many benefits to using a modern JavaScript build system, but getting one to play nicely with a very old, very large codebase is not always easy. I'll go over some of the more interesting problems we ran into at Etsy when migrating to Webpack from a system written in-house. We'll talk about working in a large codebase without running out of memory, localizing our JavaScript into 15 languages without building it 15 times, and why 4 milliseconds kept us from launching the new system for 3 months.

This is a talk I gave at JSConf 2019. With any luck, there will be a video of it up soon.

Transcript

1. 1 My name is Salem Hilal @technoheads on twitter We started using Webpack (Thank you Katie, and thanks so much to JSConf for letting me give this talk) Hi everyone! My name is Salem (like the cat), my pronouns are he/him, and I work at Etsy (we are very much hiring). This is my very first conference and my very first talk, so if you have any feedback for me, I would absolutely welcome it. I want to talk about how we migrated from an in-house written build system to Webpack
2. 2 @technoheads on twitter We started using Webpack and My name is Salem Hilal And specifically,
3. 3 @technoheads on twitter but it’s mostly shitposting We started using Webpack and it took a while. My name is Salem Hilal I wanna talk about why that took us so long to do.
4. 4 Photo credit: Dobravel Office Furniture So, I work at Etsy on a team called the
5. 5 Web Photo credit: Dobravel Office Furniture Web
6. 6 Web platform Photo credit: Dobravel Office Furniture Platform team. If you’ve ever worked at a place that has a Frontend Infrastructure team, it’s likely very similar. Our whole job is to manage how Etsy writes and builds stuff like CSS and, more relevantly, JavaScript. This talk is essentially about how we moved from one build system to another. That doesn’t mean it’s gonna be about how to write a config file or how to make ES6 run in IE8. My goal here today is to talk about the weird problems we unearthed, how we fixed them, and where my hair went in the process.
7. 7 1. Etsy and Builda 2. Why we picked Webpack 3. Development 4. Production (localization) 5. Rolling it out This whole talk splits into roughly 5 parts: 1. I’m going to first tell you a bit about Etsy and our old build system, Builda 2. I’ll talk for a bit about why Webpack looked like a sick bet 3. I’ll go over some headaches we had in development 4. And production, 5. And finally, I’ll talk about a 4 ms bug we encountered while rolling Webpack out that delayed our launch by three months.
8. 8 Before I really get into it, there are two big ol’ caveats that are worth mentioning:
9. 9 1. Knowing JavaScript, this talk is probably dated already First, the decisions we made were true for us at the time we made them. There may now be even better alternatives, but they either didn't exist at the time, or they weren't battle-tested enough to rely on in production yet. JS changes fast, which is the whole reason we wanted a flexible build system in the first place.
10. 10 1. Knowing JavaScript, this talk is probably dated already 2. My team is dope And 2: All of the things I'm gonna talk about were team efforts. I know a whole lot about how Etsy's front-end infrastructure works, but I'm speaking on behalf of a team that collectively built and researched all of the things I’m going to talk about. They all kick ass and I love working with them.
11. 11 ETSY AND BUILDA TIMING IS EVERYTHING BY SALEM Anyway, let’s get going then. I'd like to start by talking a bit about Etsy's codebase, and Etsy’s old build system. It's hard to talk about migrating build systems without giving you a little bit of an idea about what we were migrating from.
12. 12 @technoheads ETSY AND BUILDA Etsy's codebase is a lot like any other codebase at a medium-sized company; it was once small and weird,
13. 13 @technoheads ETSY AND BUILDA and then it stayed weird but got big. Some things scaled well and some things didn't.
14. 14 @technoheads ETSY AND BUILDA Most of our code lives in one big monorepo, This is to say that all of the code that makes Etsy show up in a web browser, from our API to our CSS, lives in one big repository. We call our monorepo “EtsyWeb”. This has worked surprisingly well for us. It has helped facilitate learning, and it has made finding examples and reusing patterns extremely simple. However, one thing that didn't scale well was
15. 15 (JAVASCRIPT) @technoheads ETSY AND BUILDA the JavaScript part of our codebase, and more specifically...
16. 16 Photo credit: globaltrashsolutions.com @technoheads ETSY AND BUILDA our JavaScript build system.
17. 17 (TH IS IS A TR A SH C O M PA C TO R ) Photo credit: globaltrashsolutions.com @technoheads ETSY AND BUILDA This is a trash compactor just to drive the metaphor home.
18. 18 Photo credit: globaltrashsolutions.com @technoheads ETSY AND BUILDA That system was named Builda
19. 19 Photo credit: globaltrashsolutions.com @technoheads ETSY AND BUILDA Because it built
20. 20 Photo credit: globaltrashsolutions.com @technoheads ETSY AND BUILDA All of our assets
21. 21 Photo credit: globaltrashsolutions.com @technoheads ETSY AND BUILDA It was written in house more than eight years ago and has existed in various incarnations since. For all intents and purposes, it did its job pretty well for a long time.
22. 22 @technoheads ETSY AND BUILDA For those of you who may be unfamiliar with build systems, a JavaScript build system is something that takes
23. 23 (THESE ARE FILES) @technoheads ETSY AND BUILDA our javascript files
24. 24 @technoheads ETSY AND BUILDA Resolves their dependencies, bundles them together, Make them as efficient and as small as possible
25. 25 @technoheads ETSY AND BUILDA And make them work in a bunch of different locales and browsers all over the world. This wasn't to say that our JavaScript code was particularly sophisticated when Builda was first written.
26. 26 (A week ago) @technoheads ETSY AND BUILDA [HIT PLAY] When Builda was first written, Etsy wasn’t the sleek, interactive homepage that you can see here, complete with image carousels, slick overlays, cool animations, and a responsive navigation bar.
27. 27 (Like 2010 or something) @technoheads ETSY AND BUILDA It looked more like this. JavaScript was decoration on top of a highly server-driven experience. Having one file depend on another was relatively infrequent, and when that did happen, it was usually for large, globally-scoped libraries (like jQuery). For a long time, our code didn't take a lot of work to turn into a production-ready asset; few dependencies needed to be resolved, and nothing needed to be transpiled. Production builds very little time. At Etsy, that's very important; we deploy code dozens of times a day, and slowing down deployment is something we avoid at all costs.
28. 28 MONOREPO ME, DEVELOPING @technoheads ETSY AND BUILDA Developing JavaScript was similarly simple.
29. 29 G ET /app.js (please) MONOREPO @technoheads ETSY AND BUILDA ME, DEVELOPING If we loaded a page in development that requested some JavaScript,
30. 30 G ET /app.js (please) MONOREPO @technoheads ETSY AND BUILDA ME, DEVELOPING that JavaScript file would be built from source
31. 31 MONOREPO CODE BUNDLE @technoheads ETSY AND BUILDA ME, DEVELOPING and sent back in one request.
32. 32 MONOREPO BUILDLIST CODE BUNDLE @technoheads ETSY AND BUILDA ME, DEVELOPING To keep track of all the possible files we might request, we kept a list of all of them, which we called the Buildlist. Whenever we served a file, we’d make sure it was in that list somewhere. The buildlist was used in production to determine the entirety of what we need to build when we deployed our code.
33. 33 MONOREPO BUILDLIST ?? ?? ?? G ET /bob-ross.js @technoheads ETSY AND BUILDA ME, DEVELOPING In development, if a requested file wasn't in the buildlist,
34. 34 MONOREPO BUILDLIST JUST AN EXCEPTION ☢ @technoheads ETSY AND BUILDA ME, DEVELOPING we'd respond with some JS that threw an exception, telling you to add your file to the list. This ensured that we could build essentially any file on demand, while making sure that we knew the extent of what we needed to build when we were ready to go to production.
35. 35 @technoheads ETSY AND BUILDA We used require.js to manage and resolve our dependencies. Require.js is pretty simple. We send a copy of it to the browser, followed by our own code.
36. 36 @technoheads ETSY AND BUILDA Require.js allows our code to then use AMD-style module definitions, as you can see here. AMD stands for “Asynchronous Module Definition”, which is just a syntax for defining modules and dependencies
37. 37 @technoheads ETSY AND BUILDA We define an array of imports, which Require resolves for us.
38. 38 @technoheads ETSY AND BUILDA Those imports then get passes to a callback as arguments.
39. 39 @technoheads ETSY AND BUILDA All of our own code then goes into the body of the callback, where it has access to all of the imports. Builda’s job was to make sure that all of the necessary modules needed on a page were bundled into one JavaScript file, so that Require.js had access to everything it needed in the browser.
40. 45 @technoheads ETSY AND BUILDA At some point, React came along. For those of you unfamiliar, React is a library that makes writing large client-side apps a lot easier.
41. 46 (2016) @technoheads ETSY AND BUILDA Some time around 2016, we decided React would be a really good fit for some of our seller tools.
42. 47 @technoheads ETSY AND BUILDA When we decided to introduce React code to our codebase, everyone was excited. React, however, strongly relies on JSX, a syntax that looks like
43. 48 @technoheads ETSY AND BUILDA HTML, controversially inlined into JavaScript. If we wanted to be able to use this syntax, we needed our build system to do more than just glue files together; it had to also be able to convert JSX back into JavaScript.
44. 49 @technoheads ETSY AND BUILDA So, we used an old version of the React compiler,
45. 50 @technoheads ETSY AND BUILDA and we non-trivially bolted it on to Builda.
46. 51 @technoheads ETSY AND BUILDA This allowed us to turn React’s custom JSX syntax into plain ol’ JavaScript. There was one problem with this setup, however.
47. 52 MONOREPO G ET /bob-ross.js @technoheads ETSY AND BUILDA Because we built every file whenever it was requested, rather than whenever some part of it was edited, we had no way of knowing if it had actually changed or not since the last request.
48. 53 MONOREPO G ET /bob-ross.js @technoheads ETSY AND BUILDA So, we had to rebuild every asset every time we wanted it. Transpiling JSX into JavaScript adds a bit of work to this step, so assets that relied on React heavily ended up taking a good bit longer to build and get served.
49. 54 MONOREPO @technoheads ETSY AND BUILDA But JSX was new
50. 55 MONOREPO @technoheads ETSY AND BUILDA and wasn't used in many places yet, so very few people felt the pain of long build and rebuild times in development. And besides, how bad could it get? It's just JavaScript.
51. 56 MONOREPO @technoheads ETSY AND BUILDA We all know that didn’t last long. All of these quirks together meant that loading something large, like a single-page app, implied rebuilding the entire thing from scratch every time the page reloaded. Plus, our codebase was only getting bigger. At last count, we have over 1000 separate JavaScript assets to output, made up of over 11,000 separate files. React code was used in a lot more places, and it started to take the better part of a minute to show up on the page in development, which made iterating extremely tedious.
52. 57 @technoheads ETSY AND BUILDA On top of all this, developers were starting to ask for the ability to use ES6 syntax and features
53. 58 MONOREPO which felt like an absolute non-starter. React support was difficult to implement, and even though it worked, it certainly wasn’t sustainable. Repeating this process with ES6 felt hopeless. We knew we needed to do something.
54. 59 THE SOLUTION TO EVERY ONE OF OUR PROBLEMS [WATER BREAK] I’m gonna take a sec to drink some water because it’s extremely good for you. if you haven’t, in 15 seconds or less, introduce yourself to the people sitting next to you It was around this time that people were getting excited about a new, open-sourced JavaScript build system.
55. 60 HAVE YOU CONSIDERED USING WEBPACK? @technoheads It was called “Webpack”
56. 61 You may have heard of it
57. 62 HAVE YOU CONSIDERED USING WEBPACK? @technoheads Any new developer we hired asked if they could use it, and the internet made it sound like every tech company that used JavaScript also used Webpack. While it's always healthy to be skeptical of the things that it sounds like every other tech company is doing, Webpack seemed interesting.
58. 63 HAVE YOU CONSIDERED USING WEBPACK? @technoheads It sounded a lot like Builda in that it
59. 64 1. Builds stuff HAVE YOU CONSIDERED USING WEBPACK? @technoheads builds JavaScript assets, but it also sounded very different in that it...
60. 1. Builds stuff 2. Development is chill 65 HAVE YOU CONSIDERED USING WEBPACK? @technoheads had a robust development experience,
61. 66 1. Builds stuff 2. Development is chill 3. Learns new tricks HAVE YOU CONSIDERED USING WEBPACK? @technoheads it was highly extensible,
62. 67 1. Builds stuff 2. Development is chill 3. Learns new tricks 4. Optimizes code HAVE YOU CONSIDERED USING WEBPACK? @technoheads and it had a bunch of built in performance optimizations for our code.
63. 68 1. Builds stuff 2. Development is chill 3. Learns new tricks 4. Optimizes code 5. Makes fast HAVE YOU CONSIDERED USING WEBPACK? @technoheads Most of all, it was supposed to be
64. 69 1. Builds stuff 2. Development is chill 3. Learns new tricks 4. Optimizes code 5. Makes fast HAVE YOU CONSIDERED USING WEBPACK? @technoheads a lot faster at building things, particularly in development
65. 70 1. Builds stuff 2. Development is chill 3. Learns new tricks 4. Optimizes code 5. Makes fast HAVE YOU CONSIDERED USING WEBPACK? @technoheads So, we did a little research,
66. 71 1. Builds stuff 2. Development is chill 3. Learns new tricks 4. Optimizes code 5. Makes fast HAVE YOU CONSIDERED USING WEBPACK? @technoheads evaluated a bunch of alternatives,
67. 72 1. Builds stuff 2. Development is chill 3. Learns new tricks 4. Optimizes code 5. Makes fast photo credit: Cape Breton Post HAVE YOU CONSIDERED USING WEBPACK? @technoheads and decided to go down the path of building all of our JavaScript with Webpack instead of Builda. Our goal was to create a new system that would act a lot like Builda.
68. 73 HAVE YOU CONSIDERED USING WEBPACK? @technoheads We had two requirements for anything that we would want to adopt
69. 74 limited development interaction HAVE YOU CONSIDERED USING WEBPACK? @technoheads First and foremost, it would have to be able to take care of development builds without much input from developers. One of the nicest things about Builda was that, despite its slow build speed, you could navigate around the entire site in development and never once think about manually building any JavaScript.
70. 75 limited development interaction ~5 minute production builds HAVE YOU CONSIDERED USING WEBPACK? @technoheads The other requirement was that it would have to build our production code in about 5 minutes. As I mentioned before, Etsy deploys code to production dozens of times every day, which is a core part of how we work. If our builds took much longer than 5 minutes, we’d slow down Etsy’s ability to deploy quickly, which is a non-starter.
71. 76 @technoheads HAVE YOU CONSIDERED USING WEBPACK? Webpack is cool for us because it's highly configurable.
72. 77 BUILDLIST @technoheads HAVE YOU CONSIDERED USING WEBPACK? Instead of a buildlist,
73. 78 WEBPACK.CONFIG @technoheads HAVE YOU CONSIDERED USING WEBPACK? Webpack uses a configuration file (usually called something like webpack.config). It is in this same file that you specify every aspect of Webpack’s behavior,
74. 79 WEBPACK.CONFIG @technoheads HAVE YOU CONSIDERED USING WEBPACK? from how you’d like to support templates
75. 80 WEBPACK.CONFIG @technoheads HAVE YOU CONSIDERED USING WEBPACK? to what sort of performance optimizations you’d like to make.
76. 81 @technoheads HAVE YOU CONSIDERED USING WEBPACK? This often times means really large,
77. 82 @technoheads HAVE YOU CONSIDERED USING WEBPACK? Really scary config files, which isn't always the right idea for small projects. But for a large codebase with weird quirks everywhere, having something that could adapt and fit our code
78. 83 @technoheads HAVE YOU CONSIDERED USING WEBPACK? was invaluable. We spent a while writing out a really nice Webpack dot config file. We also spent a while writing a bunch of plugins to Webpack to emmulate various features that Builda supported which, at the time, included everything from transpiling templates to inlining SVG icons. After spending hours setting things up just right, we finally got Webpack to build all of our code.
79. 84 but HAVE YOU CONSIDERED USING WEBPACK? @technoheads But it wasn’t quite what we expected
80. 85 @technoheads HAVE YOU CONSIDERED USING WEBPACK? It took over half an hour,
81. 86 @technoheads HAVE YOU CONSIDERED USING WEBPACK? it ate up 20 gigs of our server’s memory and maxed out all 32 of its processors,
82. 87 @technoheads HAVE YOU CONSIDERED USING WEBPACK? And it had an unclear progress indicator, making it hard to tell what was making any of it take so long. This is me tweeting after a hard day of trying to figure out what made webpack take so long.
83. 88 @technoheads HAVE YOU CONSIDERED USING WEBPACK? We definitely had a lot to learn about improving Webpack's performance, but at the end of the day, we needed to change a lot of things about how our codebase looked and about how Webpack worked with our code if we wanted to use it at Etsy with on-demand development builds and 5 minute production builds.
84. 89 DEVELOPMENT , [WATER BREAK] in 15 seconds or less, what’s your favorite text editor to use? Our first order of business was to get a development workflow
85. 90 DEVELOPMENT , EXCEPT GOOD that would work better than Builda’s. If engineers could develop with Webpack, and validate that their code still worked with Builda, we could at least improve development times while we fiddled with production. We wouldn't be able to offer new features like ES6 syntax yet, since Builda was still building our production code, but maybe we could get build times down for the poor engineers waiting the better part of a minute for their React code to show up on the screen. To do this, we first had to figure out why our builds took up so much memory and so much time.
86. 91 MONOREPO @technoheads DEVELOPMENT When Webpack first starts,
87. 92 MONOREPO EVERY CODE BUNDLE @technoheads DEVELOPMENT it completes a full build of all your code first
88. 93 MONOREPO EVERY CODE BUNDLE @technoheads DEVELOPMENT and then enters "watch" mode, where it monitors your source files for changes
89. 94 MONOREPO EVERY CODE BUNDLE V2 V2 @technoheads DEVELOPMENT and kicks off a partial rebuild when it sees one. This pattern allows development to be extremely fast relative to complete rebuilds. We figured that a longer initial build coupled with rebuilds of a second or two is probably much better than consistently sluggish builds, but if that initial build took a half hour and ate up all of our memory, a two second rebuild probably wouldn't matter all that much. On top of that, our codebase was large and only getting larger, so whatever solution we came up with had to scale.
90. 95 BEST_APP.JS SICK_PROGRAM.JS @technoheads DEVELOPMENT Why is it necessary to do a complete build of all of your code first, you might ask? Webpack makes inferences about your code in the context of your whole project. For example, it is smart enough
91. 96 BEST_APP.JS SICK_PROGRAM.JS @technoheads DEVELOPMENT to group commonly requested modules together,
92. 97 BEST_APP.JS SICK_PROGRAM.JS SHARED.JS @technoheads DEVELOPMENT so that they can be cached between pages more efficiently. It can only make optimizations like that if it understands your whole project, not just your individual files. This is good if your project is nicely scoped, but it makes a lot less sense in a monorepo, where everything from our internal tools to our homepage code live.
93. 98 Powerpuff Girls © Cartoon Network @technoheads DEVELOPMENT We stated by trying to make Webpack faster and less resource intensive in general, something that's hard to do both of at the same time. We got a lot of mileage from caching plugins like cache-loader
94. 99 Powerpuff Girls © Cartoon Network @technoheads DEVELOPMENT and HardSource,
95. 100 Powerpuff Girls © Cartoon Network @technoheads DEVELOPMENT and allowing Webpack to process our modules in parallel using plugins like HappyPack was certainly a good idea,
96. 101 Powerpuff Girls © Cartoon Network @technoheads DEVELOPMENT but at the end of the day, we still had a prohibitively heavy first build time.
97. 102 @technoheads DEVELOPMENT We also looked at Webpack's development server for potential solutions. The development server, by design, chooses to keep everything in memory, including its finished output files. For us, this meant that our whole codebase, plus all the transpiling information about our whole codebase, plus all the built files in our whole codebase, had to fit in memory.
98. 103 @technoheads DEVELOPMENT There unfortunately isn't a way to turn this behavior off without modifying the development server's source code (which is really simple actually, and I'd definitely recommend taking a look at how the development middleware, in particular, works, if you are curious). Even if there was a way to turn this behavior off, it wasn’t clear that it would have been enough to save us from the size of our codebase over time.
99. 104 MONOREPO @technoheads DEVELOPMENT All of this put together made it sound like it was probably a good idea to keep Webpack from building the whole codebase at once.
100. 105 MONOREPO, BUT IN PIECES @technoheads DEVELOPMENT One really easy solution was to just make a few different configs for different areas of the codebase,
101. 106 MONOREPO, BUT IN PIECES @technoheads DEVELOPMENT and only built the one that was being used at the time. We can make each region be small enough so that memory is reasonable and initial build times are closer to a minute. For us, this meant splitting our codebase into 11 regions. We had regions for things like our seller tools, the core buyer experience, and our internal tools. That would have worked, but we wanted to emulate one of Builda's nicer features; quietly building files without any additional interaction from the developer.
102. 107 MONOREPO, BUT IN PIECES Photo credit: https://flic.kr/p/cEJpCY @technoheads DEVELOPMENT Juggling a bunch of configurations just to browse around Etsy on your work machine isn’t ideal. This meant that we likely needed to write something on our own.
103. 108 The Office © NBC So, we wrote this thing we call Kevin.
104. 109 The Office © NBC Kevin is a plugin for the Express web server framework, and it's only job is to manage a bunch of Webpack instances. Why did we name it kevin? Because we thought it was funny.
105. 110 MONOREPO GET /app.js (please) (still in pieces) @technoheads DEVELOPMENT When a request for a file comes in,
106. 111 MONOREPO GET /app.js (please) @technoheads DEVELOPMENT Kevin determines which Webpack config is responsible for building that file (if any),
107. 112 MONOREPO GET /app.js (please) @technoheads DEVELOPMENT and then starts a Webpack instance for that config.
108. 113 MONOREPO GET /app.js (please) CODE BUNDLE @technoheads DEVELOPMENT Once the compiler is finished,
109. 114 MONOREPO GET /app.js (please) CODE BUNDLE @technoheads DEVELOPMENT it stays running in watch mode, rebuilding files as they're edited, much like with Webpack's dev server.
110. 115 MONOREPO GET /something-else.js @technoheads DEVELOPMENT If Kevin gets a request for another file in a different config,
111. 116 MONOREPO GET /something-else.js CODE BUNDLE @technoheads DEVELOPMENT Kevin will spin up another Webpack instance to handle it.
112. 117 MONOREPO CODE BUNDLE ☠ GET /something-else.js @technoheads DEVELOPMENT If there are ever too many compilers running, Kevin will shut down a compiler, based on a frecency algorithm (which is frequency plus recency). This keeps us from building the whole codebase, and it has the added benefit of making sure that areas of the codebase that are related to each other get grouped into the same config.
113. 118 @technoheads DEVELOPMENT The only extra thing to consider is that really long first build.
114. 119 @technoheads DEVELOPMENT If a request for a JavaScript file
115. 120 @technoheads DEVELOPMENT (timeout) times out after 30 seconds (the default in Firefox, for example), and starting up a Webpack compiler takes closer to a minute, we're gonna end up timing out in development for what seems to be no reason.
116. 121 @technoheads DEVELOPMENT Our solution was to just serve some static JavaScript that shows an overlay with a list of what's being built, if we just started up an instance of Webpack. It’s modeled after the Dominos Pizza Tracker We also expose some data about the status of every active compiler, so the overlay is able to monitor the status of the first build, and reload the page whenever it's done. The end result is a fully automatic process that keeps memory usage low and keeps engineers informed about the state of a built, without requiring them to worry about managing a build system. With any luck, Kevin will be available as open source software very soon.
117. 122 PRODUCTION [WATER BREAK] In 15 seconds or less, tell a neighbor about what you did for adventure day Next, lemme talk about productionizing our code.
118. 123 PRODUCTION LOCALIZATION Specifically, I want to talk about localization, since there’s a lot of other, much less interesting things that went into getting our code production ready.
119. 124 @technoheads LOCALIZATION At this point in our journey to Webpack,
120. 125 The Office, still © NBC @technoheads LOCALIZATION development was roughly working. We even started to onboard a handful of developers so they could give us some feedback
121. 126 @technoheads LOCALIZATION (and so that they could develop a lot faster). Because we weren’t building production code with BuildaPack just yet, we weren’t ready to enable things like ES6 support. But the benefits Webpack brought to development speed were wins in and of themself. Our next order of business was to get production builds to run quickly
122. 127 @technoheads LOCALIZATION (remember, 5 minute builds were our goal).
123. 128 EVERY CODE BUNDLE @technoheads LOCALIZATION A working development environment meant that we were at least able to successfully build our code, but we still needed to productionize it.
124. 129 Photo credit: Broan @technoheads LOCALIZATION This involved minifying it,
125. 130 @technoheads LOCALIZATION but more critically it meant localizing all of our assets into 11 different languages.
126. 131 131 MONOREPO TRANSLATIONS translate(“hello”) @technoheads LOCALIZATION Localization is a little tricky with Webpack. The general assumption is that for every locale you want to support, you need to kick off a separate Webpack build. Webpack’s localization plugin uses a file that contains localized strings (that you provide) and your code uses a function call to get access to those strings. It seems simple enough; using this method, constant strings can be defined in one file and swapped out between builds, without your JavaScript changing.
127. 132 MONOREPO @technoheads LOCALIZATION There's an implication here that's not totally obvious though; Webpack ends up with a different configuration for each locale you support, because the plugin needs to be configured differently for each language. This means that in order to build 11 languages, you need to run 11 different webpack builds.
128. 133 @technoheads LOCALIZATION For Etsy, one build on our beefy primary deployment server (32 cores, 64 gigs of ram) could take as long as 5 or 6 minutes. We could maybe run two builds in parallel in eight or nine minutes, but that would still mean that all of our builds would take something like an hour. We were also constrained by the fact that we were actively moving all of Etsy from hardware in data centers to the cloud. While this was happening, we couldn’t just ask for new hardware, let alone a dozen or so of the beefiest servers we had. If we were trying to keep pace with Builda, which could get our code ready in something like five miuntes, we were gonna have to cheat a little.
129. 134 @technoheads LOCALIZATION Cheating is weirdly more ok than you'd think here, since Builda essentially cheats too. When it needs to localize a bunch of code into 11 different locales, Builda doesn't build the same code 11 times.
130. 135 @technoheads LOCALIZATION Instead, it inserts placeholder strings around the places where translated strings need to go.
131. 136 ]::TRANSLATION_TIME::[See ya later!]::TRANSLATION_TIME::[ @technoheads LOCALIZATION Later, when the build is mostly done, it goes back through each file,
132. 137 ]::TRANSLATION_TIME::[See ya later!]::TRANSLATION_TIME::[ @technoheads LOCALIZATION finds those placeholders,
133. 138 ]::TRANSLATION TIME::[See ya later!]::TRANSLATION TIME::[ @technoheads LOCALIZATION and swaps them out...
134. 139 au revoir mon petit chou-fleur @technoheads LOCALIZATION for actual localized strings in every language.
135. 140 @technoheads LOCALIZATION Compared with Webpack's
136. 141 @technoheads LOCALIZATION (now-deprecated) internationalization plugin, this seemed like a way faster approach. I actually didn’t realize it was deprecated until I started writing this talk. Anyways, we tweaked our plugins that were responsible for providing translated strings and had them insert plain ol' double quote string placeholders instead. Webpack would unknowingly build bundles with these strings inlined, which, being strings, would be left alone through the entire build process. Once the build finished, we'd be left with a bunch of production-ready files with a bunch of placeholders in them.
137. 142 From there, a plugin would iterate through our assets, do a find-and-replace, and write a copy of the source to disk with the new locale inlined. This was easier than it sounds:
138. 143 @technoheads LOCALIZATION we were able to just take our source code and call .split with our placeholder.
139. 144 @technoheads LOCALIZATION The resulting array would contain original source code in the even indexes,
140. 145 @technoheads LOCALIZATION and the string that needed translating in the odd indexes. We'd iterate through the array, pick the odd indexes,
141. 146 @technoheads LOCALIZATION and translate them. In case you’re curious, we get our translations from a separate service that we maintain, which dumps static translation files onto our development, deployment, and production servers.
142. 147 NBC owns the rights to this particular Kevin @technoheads LOCALIZATION This method was so fast, in fact, that we were able to provide localization in development just by translating each file as it was requested, rather than building each file into every possible language. Thanks to this method, we were able to get our production builds ready in well under 5 minutes.
143. 148 4ms DELAYED OUR LAUNCH [WATER BREAK] Tell your neighbor (in 15 seconds or less) if you can speak more than one language For the last part of this talk, lemme tell you a bit about 4 milliseconds that kept us from launching this whole migration 3 months earlier than we actually did. We were finally at the point where we felt ready to try and swap out our build systems on production traffic. Development looked good, production was building quickly, and a bunch of end-to-end tests showed that everything was still working as expected.
144. 149 INDIANA JONES is a trademark of LUCASFILM LTD Build systems are really, really, really hard to swap though.
145. 150 (remember these are files) 4ms MADE US SAD @technoheads A new build system has to work in every language, for every page on the site.
146. 151 4ms MADE US SAD @technoheads And it also has to work in every single browser
147. 152 4ms MADE US SAD @technoheads No matter how old
148. 153 4ms MADE US SAD @technoheads Or cutting edge it may be. If our pages doesn’t work, people will stop using our site, and we’ll probably lose money. So what do you do when you want to gain confidence in a huge change?
149. 154 O h yes an A/B test 4ms MADE US SAD @technoheads You run an a/b test.
150. 155 EXPERIMENT 4ms MADE US SAD @technoheads We actually ran five separate ones, if I’m gonna be totally honest.
151. 156 EXPERIMENT 4ms MADE US SAD @technoheads And for us, all of them looked really good. You can see there that all of our important metrics say “No change” next to them, which is actually as good as we hoped.
152. 157 and yet somehow 4ms MADE US SAD @technoheads And yet in spite of all this, we had some pretty alarming changes...
153. 158 These are our top-secret performance metrics 4ms MADE US SAD @technoheads ...in every one of our browser performance metrics. Every page was running way, way slower than expected.
154. 159 w hat in sw eet heck 4ms MADE US SAD @technoheads Some pages were over 14% slower! That’s unheard of. It’s a general rule of thumb that if your site’s performance gets worse, so does your site’s money. We had to figure out what was going wrong before we could launch Webpack.
155. 160 4ms MADE US SAD @technoheads Before I go on, let’s talk about some of our metrics really quick. Some of you may already know this stuff, some of you may not, but I hope it’ll be a good refresher nonetheless. At Etsy, we look particularly at two client-side performance metrics:
156. 161 4ms MADE US SAD @technoheads DOMContentLoaded
157. 162 4ms MADE US SAD @technoheads And Page Load
158. 163 When the browser parses our HTML, it goes through line at a time
159. 164 And evaluates everything as it goes.
160. 165 For most resources like this stylesheet, it downloads and executes it before continuing to parse the page
161. 166 Some things, like this image tag,
162. 167 will kick off a request, but will not block the browser from continuing to parse the page
163. 168 Here is our javascript, at the very bottom. Like CSS, JavaScript is downloaded and executed before the browser can continue.
164. 169 • Network requests • setTimeout Some javascript code, like network requests or setTimeout calls, do not block the browser, and are added to the list of tasks a browser can take care of at a later time.
165. 170 Finally, when the browser finishes parsing all of our HTML,
166. 171 DOMContentLoaded! it fires the DOMContentLoaded event, which says that the structure of the page is complete, even if we’re still waiting for some other resources (like that image)
167. 172 etsy.comshop/catHairAndTeeth Once all of our subresources load (like that image), and when the browser has run out of tasks to immediately take care of,
168. 173 page load! it fires the “page load” event, effectively marking the page as “done”
169. 174 load (or pageLoad) The load event is fired when the whole page has loaded, including all dependent resources, and there are no outstanding scripts waiting to be executed. DOMContentLoaded The DOMContentLoaded event fires when the initial HTML document has been completely loaded and parsed, without waiting for stylesheets (sometimes), images, and subframes to finish loading. 4ms MADE US SAD @technoheads To recap: DOMContentLoaded fires when the page’s structure is in place, and load fires when everything is loaded, and no scripts are waiting to be executed (also v important) Both of these metrics were consistently slower for us. This raised two questions
170. 175 1. What could we have missed? 4ms MADE US SAD @technoheads What thing is so different, yet so small that we didn’t notice?
171. 176 1. What could we have missed? 2. Why are people still buying things? 4ms MADE US SAD @technoheads If something so different is happening, why the hell are people buying things at all?
172. 177 investigation time. 4ms MADE US SAD @technoheads We started out by investigating our performance monitoring code for bugs. We also double-checked that we weren’t somehow sending more code to users than we thought we were. When both of these investigations turned up no new leads, we started to wonder if Webpack’s code was somehow RUNNING differently, despite otherwise looking the same. To confirm this hunch, we decided to profile our JavaScript as it ran in the browser.
173. 178 Builda 4ms MADE US SAD @technoheads This is a flame graph for the listing page, which is the page that shows an item for sale on Etsy. This graph shows our code, built with Builda. Here, the horizontal axis represents time And vertical access represents layers of execution. Every rectangle represents a function call or a piece of execution, and everything below it is a function call sub-task related to it.
174. 179 “Evaluate Script (main.[version].en-US.js)” Builda 4ms MADE US SAD @technoheads This particular slice of the graph shows what the browser does when it parses main.js, the primary javascript file on this page. Very quiet, not much going on
175. 180 “Evaluate Script (main.[version].en-US.js)” (it took 18ms here) Builda 4ms MADE US SAD @technoheads The whole thing doesn’t even take long (18ms) Now what does Webpack look like for the same code?
176. 181 Webpack 4ms MADE US SAD @technoheads It looks… hectic.
177. 182 Webpack “Evaluate Script (main.[version].en-US.js)” 4ms MADE US SAD @technoheads This is the same file as in the previous graph.
178. 183 Webpack “Evaluate Script (main.[version].en-US.js)” (it took a whopping 65ms here) But it’s taking almost 4 times as long, and a lot more is clearly happening
179. 184 Builda Webpack 4ms MADE US SAD @technoheads For kicks, here are those two graphs side by side. Clearly, Builda-built code appeared to be doing way less than Webpack-built code.
180. 185 Builda Webpack 4ms MADE US SAD @technoheads This is concerning.
181. 186 yes but how come 4ms MADE US SAD @technoheads So, why? Even if our conversion numbers look good, this huge difference may mean we’re missing an obvious performance enhancement!
182. 187 4ms MADE US SAD @technoheads Let’s take another look at the whole flame graph
183. 188 (Builda) 4ms MADE US SAD @technoheads This one is builda’s
184. 189 (Builda) 4ms MADE US SAD @technoheads This is roughly the area we were looking at before, where our browser downloaded, parsed, and executed our JavaScript.
185. 190 (Builda) 4ms MADE US SAD @technoheads This is where DOMContentLoaded is fired
186. 191 (Builda) 4ms MADE US SAD @technoheads This here is page load
187. 192 (Builda) 4ms MADE US SAD @technoheads And this right here is apparently where the contents of our JavaScript file actually get executed
188. 193 4ms MADE US SAD @technoheads Let me back up a little here and zoom in.
189. 194 4ms MADE US SAD @technoheads See this rectangle right here?
190. 195 This is code from our javascript file, running later than we thought. 4ms MADE US SAD @technoheads This little rectangle is the body of our JavaScript file, which, for some reason, is getting executed a lot later than we thought So it seems like Builda’s code is somehow running well after the page load event has fired, which doesn’t seem to be the case with Webpack’s code.
191. BAD PERF BAD MEASURE MENT In other words, our performance metrics were not accounting for the time it took to actually run our JavaScript code in the browser, when we built that code with Builda. Code built with Webpack did not have this problem.
192. 197 1. What could we have missed? 2. Why are people still buying shit? 4ms MADE US SAD @technoheads So we know why people are still buying stuff: the same things were happening in the browser, but our measurements moved.
193. 198 1. What could we have missed? 2. Why are people still buying shit? 4ms MADE US SAD @technoheads We still need to know why, though, mostly cuz we were curoius.
194. 199 load (or pageLoad) The load event is fired when the whole page has loaded, including all dependent resources, and there are no outstanding scripts waiting to be executed. DOMContentLoaded The DOMContentLoaded event fires when the initial HTML document has been completely loaded and parsed, without waiting for stylesheets, images, and subframes to finish loading. 4ms MADE US SAD @technoheads Let’s look back at our two metrics, DCL and page load
195. 200 load (or pageLoad) The load event is fired when the whole page has loaded, including all dependent resources, and there are no outstanding scripts waiting to be executed. DOMContentLoaded The DOMContentLoaded event fires when the initial HTML document has been completely loaded and parsed, without waiting for stylesheets, images, and subframes to finish loading. load (or pageLoad) The load event is fired when the whole page has loaded, including all dependent resources, and there are no outstanding scripts waiting to be executed. 4ms MADE US SAD @technoheads Specifically, look at load
196. 201 load (or pageLoad) The load event is fired when the whole page has loaded, including all dependent resources, and there are no outstanding scripts waiting to be executed. DOMContentLoaded The DOMContentLoaded event fires when the initial HTML document has been completely loaded and parsed, without waiting for stylesheets, images, and subframes to finish loading. load (or pageLoad) The load event is fired when the whole page has loaded, including all dependent resources, and there are no outstanding scripts waiting to be executed. 4ms MADE US SAD @technoheads Specifically specifically, look at this bit here where it says: “there are no outstanding scripts waiting to be executed” MAYBE the load event could fire if the browser runs out of things to do. What if something was telling our scripts to wait a second before running, tricking the browser into thinking the page was ready?
197. 202 4ms MADE US SAD @technoheads On that note, let’s REALLY QUICKLY dig into the meat of require.js. As a reminder, Require.js is the library that Builda was built around. We send a copy of it to our browsers, where it’s responsible for starting up our code and managing our code’s dependencies.
198. 203 4ms MADE US SAD @technoheads Here’s where require.js kicks off our code’s execution
199. 204 4ms MADE US SAD @technoheads But it’s wrapped in this nextTick thing. What’s that?
200. 205 4ms MADE US SAD @technoheads If we dig further, it looks like it’s just…
201. 206 4ms MADE US SAD @technoheads a 4ms set timeout call. That would tell the browser that our code can be executed at least 4ms later This isn’t a hard and fast time; the browser is totally allowed to push things back a little if it needs. In other words, Require.js tells the browser that none of our code needs to be run right away.
202. 207 4ms MADE US SAD @technoheads Require.js might do this because it gives the browser a second to finish loading the page before starting to execute any JavaScript.
203. 208 4ms MADE US SAD @technoheads It also might be because using setTimeout to make your perf numbers better was really popular in 2010.
204. 209 4ms MADE US SAD @technoheads It’s hard to know for sure, but we do know that WebPack does not do this behavior, which means its performance numbers are a lot more trustworthy. So knowing that this is the bug, what happens next?
205. 210 1. No we did not just setTimeout() everything 4ms MADE US SAD @technoheads For one, we did not modify all of our code to run inside setTimeout calls. Making our numbers better means nothing if the user experience remains exactly the same, and adding setTimeout calls just hides our real performance time from our metrics.
206. 211 1. No we did not just setTimeout() everything 2. Yes we did add a bunch of our own made up metrics 4ms MADE US SAD @technoheads But we did add some new metrics to help validate that Webpack code wasn’t actually that slow. We went to the bottom of the main js file on our top five pages and added a timer there, which roughly measured how long it took for us to download and run that JS file to completion.
207. 212 4ms MADE US SAD @technoheads We then graphed those timers between both Builda and Webpack Yellow is webpack, and lower is better.
208. 213 4ms MADE US SAD @technoheads Our team was very happy for at last, Webpack was ready for prime time.
209. 214 WHAT DID WE LEARN? With everything said and done, it took us almost a year and a half to move to Webpack. We really hoped that it would be a silver bullet that would just solve all our problems, but at the end of the day, we had a lot to learn about build systems, performance, and our own codebase, regardless of what we migrated to.
210. 215 @technoheads WHAT DID WE LEARN I felt like there should be a three-point slide with our learnings, which was so so difficult to write given how many things we learned during this migration. But if I had to just pick three things, I think it’d look like this:
211. 216 1. Your code only gets bigger @technoheads WHAT DID WE LEARN 1. Your codebase gets bigger, not smaller. If you’re trying to find small performance wins to get some build time under some threshold, it probably won’t last. A solution that anticipates your problem growing twice in size will always win out in the long run.
212. 217 1. Your code only gets bigger 2. Question best practices @technoheads WHAT DID WE LEARN 2. Question what other companies, websites, and articles claim to be best practices! What works well for one person may not always work well everywhere. If we accepted that the Webpack internationalization plugin was our only option, we’d still be waiting an hour for our JavaScript to build in all of our languages.
213. 218 1. Your code only gets bigger 2. Question best practices 3. It’s ok if it takes time @technoheads WHAT DID WE LEARN And finally, if you can afford to take extra time on a big, risky project, you absolutely should. One of the nicer parts of our migration was that no one was critically stuck by our work. We really hated that a little metrics bug cost us months of time when we were so close to being done, but we ended up learning much more about performance, the browser, and our own codebase because of it. At the end of the day, after almost a year and a half of wild problems and hair loss,
214. 219 1. Your code only gets bigger 2. Question best practices 3. It’s ok if it takes time @technoheads WHAT DID WE LEARN Webpack is still pretty great software.
215. 220 Thanks (@technoheads) If you have any questions, hit me up on slack or on twitter. If you like this stuff, my team is hiring. And of course, thank you all so much.