JSCONF 2019: WE STARTED USING WEBPACK AND IT TOOK A WHILE

1
My name is Salem Hilal
@technoheads on twitter
We started using Webpack
(Thank you Katie, and thanks so much to JSConf for letting me give this talk)
Hi everyone!
My name is Salem (like the cat), my pronouns are he/him, and I work at Etsy (we are
very much hiring).
This is my very first conference and my very first talk, so if you have any feedback for
me, I would absolutely welcome it.
I want to talk about how we migrated from an in-house written build system
to Webpack

View Slide

2
@technoheads on twitter
and
And specifically,

View Slide

3
@technoheads on twitter but it’s mostly shitposting
and
it took a while.
I wanna talk about why that took us so long to do.

View Slide

4
Photo credit: Dobravel Office Furniture
So, I work at Etsy on a team called the

View Slide

5
Web
Web

View Slide

6
Web
platform
Platform team.
If you’ve ever worked at a place that has a Frontend Infrastructure team, it’s likely
very similar. Our whole job is to manage how Etsy writes and builds stuff like CSS
and, more relevantly, JavaScript.
This talk is essentially about how we moved from one build system to another. That
doesn’t mean it’s gonna be about how to write a config file or how to make ES6 run in
IE8. My goal here today is to talk about the weird problems we unearthed, how we
fixed them, and where my hair went in the process.

View Slide

7
1. Etsy and Builda
2. Why we picked Webpack
3. Development
4. Production (localization)
5. Rolling it out
This whole talk splits into roughly 5 parts:
1. I’m going to first tell you a bit about Etsy and our old build system, Builda
2. I’ll talk for a bit about why Webpack looked like a sick bet
3. I’ll go over some headaches we had in development
4. And production,
5. And finally, I’ll talk about a 4 ms bug we encountered while rolling Webpack
out that delayed our launch by three months.

View Slide

8
Before I really get into it, there are two big ol’ caveats that are worth mentioning:

View Slide

9
1. Knowing JavaScript,
this talk is probably dated
already
First, the decisions we made were true for us at the time we made them. There may
now be even better alternatives, but they either didn't exist at the time, or they weren't
battle-tested enough to rely on in production yet. JS changes fast, which is the whole
reason we wanted a flexible build system in the first place.

View Slide

10
1. Knowing JavaScript,
this talk is probably dated
already
2. My team is dope
And 2:
All of the things I'm gonna talk about were team efforts. I know a whole lot about how
Etsy's front-end infrastructure works, but I'm speaking on behalf of a team that
collectively built and researched all of the things I’m going to talk about. They all kick
ass and I love working with them.

View Slide

11
ETSY AND
BUILDA
TIMING IS EVERYTHING
BY SALEM
Anyway, let’s get going then.
I'd like to start by talking a bit about Etsy's codebase, and Etsy’s old build system. It's
hard to talk about migrating build systems without giving you a little bit of an idea
about what we were migrating from.

View Slide

12
@technoheads
ETSY AND BUILDA
Etsy's codebase is a lot like any other codebase at a medium-sized company; it was
once small and weird,

View Slide

13
@technoheads
ETSY AND BUILDA
and then it stayed weird but got big. Some things scaled well and some things didn't.

View Slide

14
@technoheads
ETSY AND BUILDA
Most of our code lives in one big monorepo, This is to say that all of the code that
makes Etsy show up in a web browser, from our API to our CSS, lives in one big
repository.
We call our monorepo “EtsyWeb”.
This has worked surprisingly well for us. It has helped facilitate learning, and it has
made finding examples and reusing patterns extremely simple.
However, one thing that didn't scale well was

View Slide

15
(JAVASCRIPT)
@technoheads
ETSY AND BUILDA
the JavaScript part of our codebase, and more specifically...

View Slide

16
Photo credit: globaltrashsolutions.com
@technoheads
ETSY AND BUILDA
our JavaScript build system.

View Slide

17
(TH
IS
IS
A
TR
A
SH
C
O
M
PA
C
TO
R
)
@technoheads
ETSY AND BUILDA
This is a trash compactor just to drive the metaphor home.

View Slide

18
@technoheads
ETSY AND BUILDA
That system was named Builda

View Slide

19
@technoheads
ETSY AND BUILDA
Because it built

View Slide

20
@technoheads
ETSY AND BUILDA
All of our assets

View Slide

21
@technoheads
ETSY AND BUILDA
It was written in house more than eight years ago and has existed in various
incarnations since.
For all intents and purposes, it did its job pretty well for a long time.

View Slide

22
@technoheads
ETSY AND BUILDA
For those of you who may be unfamiliar with build systems, a JavaScript build system
is something that takes

View Slide

23
(THESE ARE FILES)
@technoheads
ETSY AND BUILDA
our javascript files

View Slide

24
@technoheads
ETSY AND BUILDA
Resolves their dependencies, bundles them together,
Make them as efficient and as small as possible

View Slide

25
@technoheads
ETSY AND BUILDA
And make them work in a bunch of different locales and browsers all over the world.
This wasn't to say that our JavaScript code was particularly sophisticated when Builda
was first written.

View Slide

26
(A week ago)
@technoheads
ETSY AND BUILDA
[HIT PLAY]
When Builda was first written, Etsy wasn’t the sleek, interactive homepage that you
can see here, complete with image carousels, slick overlays, cool animations, and a
responsive navigation bar.

View Slide

27
(Like 2010 or something)
@technoheads
ETSY AND BUILDA
It looked more like this. JavaScript was decoration on top of a highly server-driven
experience.
Having one file depend on another was relatively infrequent, and when that did
happen, it was usually for large, globally-scoped libraries (like jQuery).
For a long time, our code didn't take a lot of work to turn into a production-ready
asset; few dependencies needed to be resolved, and nothing needed to be transpiled.
Production builds very little time. At Etsy, that's very important; we deploy code
dozens of times a day, and slowing down deployment is something we avoid at all
costs.

View Slide

28
MONOREPO
ME, DEVELOPING
@technoheads
ETSY AND BUILDA
Developing JavaScript was similarly simple.

View Slide

29
G
ET
/app.js
(please)
MONOREPO
@technoheads
ETSY AND BUILDA
ME, DEVELOPING
If we loaded a page in development that requested some JavaScript,

View Slide

30
G
ET
/app.js
(please)
MONOREPO
@technoheads
ETSY AND BUILDA
ME, DEVELOPING
that JavaScript file would be built from source

View Slide

31
MONOREPO
CODE BUNDLE
@technoheads
ETSY AND BUILDA
ME, DEVELOPING
and sent back in one request.

View Slide

32
MONOREPO
BUILDLIST
CODE BUNDLE
@technoheads
ETSY AND BUILDA
ME, DEVELOPING
To keep track of all the possible files we might request, we kept a list of all of them,
which we called the Buildlist.
Whenever we served a file, we’d make sure it was in that list somewhere.
The buildlist was used in production to determine the entirety of what we need to build
when we deployed our code.

View Slide

33
MONOREPO
BUILDLIST
??
??
??
G
ET
/bob-ross.js
@technoheads
ETSY AND BUILDA
ME, DEVELOPING
In development, if a requested file wasn't in the buildlist,

View Slide

34
MONOREPO
BUILDLIST
JUST AN EXCEPTION
☢

@technoheads
ETSY AND BUILDA
ME, DEVELOPING
we'd respond with some JS that threw an exception, telling you to add your file to the
list.
This ensured that we could build essentially any file on demand, while making sure
that we knew the extent of what we needed to build when we were ready to go to
production.

View Slide

35
@technoheads
ETSY AND BUILDA
We used require.js to manage and resolve our dependencies. Require.js is pretty
simple.
We send a copy of it to the browser, followed by our own code.

View Slide

36
@technoheads
ETSY AND BUILDA
Require.js allows our code to then use AMD-style module definitions, as you can see
here.
AMD stands for “Asynchronous Module Definition”, which is just a syntax for defining
modules and dependencies

View Slide

37
@technoheads
ETSY AND BUILDA
We define an array of imports, which Require resolves for us.

View Slide

38
@technoheads
ETSY AND BUILDA
Those imports then get passes to a callback as arguments.

View Slide

39
@technoheads
ETSY AND BUILDA
All of our own code then goes into the body of the callback, where it has access to all
of the imports.
Builda’s job was to make sure that all of the necessary modules needed on a page
were bundled into one JavaScript file,
so that Require.js had access to everything it needed in the browser.

View Slide

45
@technoheads
ETSY AND BUILDA
At some point, React came along. For those of you unfamiliar, React is a library that
makes writing large client-side apps a lot easier.

View Slide

46
(2016)
@technoheads
ETSY AND BUILDA
Some time around 2016, we decided React would be a really good fit for some of our
seller tools.

View Slide

47
@technoheads
ETSY AND BUILDA
When we decided to introduce React code to our codebase, everyone was excited.
React, however, strongly relies on JSX, a syntax that looks like

View Slide

48
@technoheads
ETSY AND BUILDA
HTML, controversially inlined into JavaScript. If we wanted to be able to use this
syntax, we needed our build system to do more than just glue files together;
it had to also be able to convert JSX back into JavaScript.

View Slide

49
@technoheads
ETSY AND BUILDA
So, we used an old version of the React compiler,

View Slide

50
@technoheads
ETSY AND BUILDA
and we non-trivially bolted it on to Builda.

View Slide

51
@technoheads
ETSY AND BUILDA
This allowed us to turn React’s custom JSX syntax into plain ol’ JavaScript.
There was one problem with this setup, however.

View Slide

52
MONOREPO
G
ET
/bob-ross.js
@technoheads
ETSY AND BUILDA
Because we built every file whenever it was requested, rather than whenever some
part of it was edited, we had no way of knowing if it had actually changed or not since
the last request.

View Slide

53
MONOREPO

G
ET
/bob-ross.js

@technoheads
ETSY AND BUILDA
So, we had to rebuild every asset every time we wanted it.
Transpiling JSX into JavaScript adds a bit of work to this step,
so assets that relied on React heavily ended up taking a good bit longer to build and
get served.

View Slide

54
MONOREPO
@technoheads
ETSY AND BUILDA
But JSX was new

View Slide

55
MONOREPO
@technoheads
ETSY AND BUILDA
and wasn't used in many places yet, so very few people felt the pain of long build and
rebuild times in development. And besides, how bad could it get? It's just JavaScript.

View Slide

56
MONOREPO

@technoheads
ETSY AND BUILDA
We all know that didn’t last long.
All of these quirks together meant that loading something large, like a single-page
app, implied rebuilding the entire thing from scratch every time the page reloaded.
Plus, our codebase was only getting bigger. At last count, we have over 1000
separate JavaScript assets to output, made up of over 11,000 separate files.
React code was used in a lot more places, and it started to take the better part of a
minute to show up on the page in development, which made iterating extremely
tedious.

View Slide

57
@technoheads
ETSY AND BUILDA
On top of all this, developers were starting to ask for the ability to use ES6 syntax and
features

View Slide

58
MONOREPO
which felt like an absolute non-starter.
React support was difficult to implement, and even though it worked, it certainly
wasn’t sustainable.
Repeating this process with ES6 felt hopeless.
We knew we needed to do something.

View Slide

59
THE SOLUTION TO
EVERY ONE OF
OUR PROBLEMS
[WATER BREAK] I’m gonna take a sec to drink some water because it’s extremely
good for you.
if you haven’t, in 15 seconds or less, introduce yourself to the people sitting next to
you
It was around this time that people were getting excited about a new, open-sourced
JavaScript build system.

View Slide

60
HAVE YOU CONSIDERED USING WEBPACK?
@technoheads
It was called “Webpack”

View Slide

61
You may have heard of it

View Slide

62
@technoheads
Any new developer we hired asked if they could use it, and the internet made it sound
like every tech company that used JavaScript also used Webpack.
While it's always healthy to be skeptical of the things that it sounds like every other
tech company is doing, Webpack seemed interesting.

View Slide

63
@technoheads
It sounded a lot like Builda in that it

View Slide

64
1. Builds stuff
@technoheads
builds JavaScript assets, but it also sounded very different in that it...

View Slide

1. Builds stuff
2. Development is chill
65
@technoheads
had a robust development experience,

View Slide

66
1. Builds stuff
3. Learns new tricks
@technoheads
it was highly extensible,

View Slide

67
1. Builds stuff
4. Optimizes code
@technoheads
and it had a bunch of built in performance optimizations for our code.

View Slide

68
1. Builds stuff
4. Optimizes code
5. Makes fast
@technoheads
Most of all, it was supposed to be

View Slide

69
1. Builds stuff
4. Optimizes code
5. Makes fast
@technoheads
a lot faster at building things, particularly in development

View Slide

70
1. Builds stuff
4. Optimizes code
5. Makes fast
@technoheads
So, we did a little research,

View Slide

71
1. Builds stuff
4. Optimizes code
5. Makes fast
@technoheads
evaluated a bunch of alternatives,

View Slide

72
1. Builds stuff
4. Optimizes code
5. Makes fast
photo credit: Cape Breton Post
@technoheads
and decided to go down the path of building all of our JavaScript with Webpack
instead of Builda.
Our goal was to create a new system that would act a lot like Builda.

View Slide

73
@technoheads
We had two requirements for anything that we would want to adopt

View Slide

74
limited
development
interaction
@technoheads
First and foremost, it would have to be able to take care of development builds
without much input from developers.
One of the nicest things about Builda was that, despite its slow build speed, you could
navigate around the entire site in development
and never once think about manually building any JavaScript.

View Slide

75
limited
development
interaction
~5 minute
production
builds
@technoheads
The other requirement was that it would have to build our production code in about 5
minutes.
As I mentioned before, Etsy deploys code to production dozens of times every day,
which is a core part of how we work.
If our builds took much longer than 5 minutes, we’d slow down Etsy’s ability to deploy
quickly, which is a non-starter.

View Slide

76
@technoheads
Webpack is cool for us because it's highly configurable.

View Slide

77
BUILDLIST
@technoheads
Instead of a buildlist,

View Slide

78
WEBPACK.CONFIG
@technoheads
Webpack uses a configuration file (usually called something like webpack.config). It is
in this same file that you specify every aspect of Webpack’s behavior,

View Slide

79
WEBPACK.CONFIG
@technoheads
from how you’d like to support templates

View Slide

80
WEBPACK.CONFIG
@technoheads
to what sort of performance optimizations you’d like to make.

View Slide

81
@technoheads
This often times means really large,

View Slide

82

@technoheads
Really scary config files,
which isn't always the right idea for small projects.
But for a large codebase with weird quirks everywhere, having something that could
adapt and fit our code

View Slide

83

@technoheads
was invaluable.
We spent a while writing out a really nice Webpack dot config file. We also spent a
while writing a bunch of plugins to Webpack to emmulate various features that Builda
supported which, at the time, included everything from
transpiling templates
to inlining SVG icons.
After spending hours setting things up just right, we finally got Webpack to build all of
our code.

View Slide

84
but
@technoheads
But it wasn’t quite what we expected

View Slide

85
@technoheads
It took over half an hour,

View Slide

86
@technoheads
it ate up 20 gigs of our server’s memory and maxed out all 32 of its processors,

View Slide

87
@technoheads
And it had an unclear progress indicator, making it hard to tell what was making any
of it take so long.
This is me tweeting after a hard day of trying to figure out what made webpack take
so long.

View Slide

88
@technoheads
We definitely had a lot to learn about improving Webpack's performance,
but at the end of the day, we needed to change a lot of things about how our
codebase looked
and about how Webpack worked with our code if we wanted to use it at Etsy with
on-demand development builds and 5 minute production builds.

View Slide

89
DEVELOPMENT
,
[WATER BREAK] in 15 seconds or less, what’s your favorite text editor to use?
Our first order of business was to get a development workflow

View Slide

90
DEVELOPMENT
,
EXCEPT GOOD
that would work better than Builda’s.
If engineers could develop with Webpack,
and validate that their code still worked with Builda,
we could at least improve development times while we fiddled with production.
We wouldn't be able to offer new features like ES6 syntax yet, since Builda was still
building our production code,
but maybe we could get build times down for the poor engineers waiting the better
part of a minute for their React code to show up on the screen. To do this, we first had
to figure out why our builds took up so much memory and so much time.

View Slide

91
MONOREPO
@technoheads
DEVELOPMENT
When Webpack first starts,

View Slide

92
MONOREPO
EVERY CODE BUNDLE
@technoheads
DEVELOPMENT
it completes a full build of all your code first

View Slide

93
MONOREPO
EVERY CODE BUNDLE
@technoheads
DEVELOPMENT
and then enters "watch" mode, where it monitors your source files for changes

View Slide

94
MONOREPO
EVERY CODE BUNDLE
V2
V2
@technoheads
DEVELOPMENT
and kicks off a partial rebuild when it sees one.
This pattern allows development to be extremely fast relative to complete rebuilds.
We figured that a longer initial build coupled with rebuilds of a second or two is
probably much better than consistently sluggish builds,
but if that initial build took a half hour and ate up all of our memory, a two second
rebuild probably wouldn't matter all that much.
On top of that, our codebase was large and only getting larger, so whatever solution
we came up with had to scale.

View Slide

95
BEST_APP.JS SICK_PROGRAM.JS
@technoheads
DEVELOPMENT
Why is it necessary to do a complete build of all of your code first, you might ask?
Webpack makes inferences about your code in the context of your whole project.
For example, it is smart enough

View Slide

96
@technoheads
DEVELOPMENT
to group commonly requested modules together,

View Slide

97
SHARED.JS
@technoheads
DEVELOPMENT
so that they can be cached between pages more efficiently.
It can only make optimizations like that if it understands your whole project, not just
your individual files. This is good if your project is nicely scoped, but it makes a lot
less sense in a monorepo, where everything from our internal tools to our homepage
code live.

View Slide

98
Powerpuff Girls © Cartoon Network
@technoheads
DEVELOPMENT
We stated by trying to make Webpack faster and less resource intensive in general,
something that's hard to do both of at the same time. We got a lot of mileage from
caching plugins like cache-loader

View Slide

99
@technoheads
DEVELOPMENT
and HardSource,

View Slide

100
@technoheads
DEVELOPMENT
and allowing Webpack to process our modules in parallel using plugins like
HappyPack was certainly a good idea,

View Slide

101
@technoheads
DEVELOPMENT
but at the end of the day, we still had a prohibitively heavy first build time.

View Slide

102
@technoheads
DEVELOPMENT
We also looked at Webpack's development server for potential solutions.
The development server, by design, chooses to keep everything in memory, including
its finished output files. For us, this meant that our whole codebase, plus all the
transpiling information about our whole codebase, plus all the built files in our whole
codebase, had to fit in memory.

View Slide

103
@technoheads
DEVELOPMENT
There unfortunately isn't a way to turn this behavior off without modifying the
development server's source code (which is really simple actually, and I'd definitely
recommend taking a look at how the development middleware, in particular, works, if
you are curious).
Even if there was a way to turn this behavior off, it wasn’t clear that it would have
been enough to save us from the size of our codebase over time.

View Slide

104
MONOREPO
@technoheads
DEVELOPMENT
All of this put together made it sound like it was probably a good idea to keep
Webpack from building the whole codebase at once.

View Slide

105
MONOREPO, BUT IN PIECES
@technoheads
DEVELOPMENT
One really easy solution was to just make a few different configs for different areas of
the codebase,

View Slide

106
@technoheads
DEVELOPMENT
and only built the one that was being used at the time. We can make each region be
small enough so that memory is reasonable and initial build times are closer to a
minute.
For us, this meant splitting our codebase into 11 regions. We had regions for things
like our seller tools, the core buyer experience, and our internal tools.
That would have worked, but we wanted to emulate one of Builda's nicer features;
quietly building files without any additional interaction from the developer.

View Slide

107
Photo credit: https://flic.kr/p/cEJpCY
@technoheads
DEVELOPMENT
Juggling a bunch of configurations just to browse around Etsy on your work machine
isn’t ideal.
This meant that we likely needed to write something on our own.

View Slide

108
The Office © NBC
So, we wrote this thing we call Kevin.

View Slide

109
The Office © NBC
Kevin is a plugin for the Express web server framework, and it's only job is to manage
a bunch of Webpack instances.
Why did we name it kevin? Because we thought it was funny.

View Slide

110
MONOREPO
GET /app.js (please)
(still in pieces)
@technoheads
DEVELOPMENT
When a request for a file comes in,

View Slide

111
MONOREPO
@technoheads
DEVELOPMENT
Kevin determines which Webpack config is responsible for building that file (if any),

View Slide

112
MONOREPO
@technoheads
DEVELOPMENT
and then starts a Webpack instance for that config.

View Slide

113
MONOREPO
CODE BUNDLE
@technoheads
DEVELOPMENT
Once the compiler is finished,

View Slide

114
MONOREPO
CODE BUNDLE
@technoheads
DEVELOPMENT
it stays running in watch mode, rebuilding files as they're edited, much like with
Webpack's dev server.

View Slide

115
MONOREPO
GET /something-else.js
@technoheads
DEVELOPMENT
If Kevin gets a request for another file in a different config,

View Slide

116
MONOREPO
CODE BUNDLE
@technoheads
DEVELOPMENT
Kevin will spin up another Webpack instance to handle it.

View Slide

117
MONOREPO
CODE BUNDLE
☠
@technoheads
DEVELOPMENT
If there are ever too many compilers running, Kevin will shut down a compiler, based
on a frecency algorithm (which is frequency plus recency).
This keeps us from building the whole codebase, and it has the added benefit of
making sure that areas of the codebase that are related to each other get grouped
into the same config.

View Slide

118
@technoheads
DEVELOPMENT
The only extra thing to consider is that really long first build.

View Slide

119
@technoheads
DEVELOPMENT
If a request for a JavaScript file

View Slide

120
@technoheads
DEVELOPMENT
(timeout)
times out after 30 seconds (the default in Firefox, for example),
and starting up a Webpack compiler takes closer to a minute,
we're gonna end up timing out in development for what seems to be no reason.

View Slide

121
@technoheads
DEVELOPMENT
Our solution was to just serve some static JavaScript that shows an overlay with a list
of what's being built, if we just started up an instance of Webpack.
It’s modeled after the Dominos Pizza Tracker
We also expose some data about the status of every active compiler, so the overlay is
able to monitor the status of the first build, and reload the page whenever it's done.
The end result is a fully automatic process that keeps memory usage low and keeps
engineers informed about the state of a built, without requiring them to worry about
managing a build system.
With any luck, Kevin will be available as open source software very soon.

View Slide

122
PRODUCTION
[WATER BREAK] In 15 seconds or less, tell a neighbor about what you did for
adventure day
Next, lemme talk about productionizing our code.

View Slide

123
PRODUCTION
LOCALIZATION
Specifically, I want to talk about localization,
since there’s a lot of other, much less interesting things that went into getting our code
production ready.

View Slide

124
@technoheads
LOCALIZATION
At this point in our journey to Webpack,

View Slide

125
The Office, still © NBC
@technoheads
LOCALIZATION
development was roughly working.
We even started to onboard a handful of developers so they could give us some
feedback

View Slide

126
@technoheads
LOCALIZATION
(and so that they could develop a lot faster).
Because we weren’t building production code with BuildaPack just yet, we weren’t
ready to enable things like ES6 support.
But the benefits Webpack brought to development speed were wins in and of
themself.
Our next order of business was to get production builds to run quickly

View Slide

127
@technoheads
LOCALIZATION
(remember, 5 minute builds were our goal).

View Slide

128
EVERY CODE BUNDLE
@technoheads
LOCALIZATION
A working development environment meant that we were at least able to successfully
build our code, but we still needed to productionize it.

View Slide

129
Photo credit: Broan
@technoheads
LOCALIZATION
This involved minifying it,

View Slide

130
@technoheads
LOCALIZATION
but more critically it meant localizing all of our assets into 11 different languages.

View Slide

131
131
MONOREPO

TRANSLATIONS
translate(“hello”)
@technoheads
LOCALIZATION
Localization is a little tricky with Webpack. The general assumption is that for every
locale you want to support, you need to kick off a separate Webpack build. Webpack’s
localization plugin uses a file that contains localized strings (that you provide) and
your code uses a function call to get access to those strings.
It seems simple enough; using this method, constant strings can be defined in one file
and swapped out between builds, without your JavaScript changing.

View Slide

132
MONOREPO

@technoheads
LOCALIZATION
There's an implication here that's not totally obvious though; Webpack ends up with a
different configuration for each locale you support, because the plugin needs to be
configured differently for each language. This means that in order to build 11
languages, you need to run 11 different webpack builds.

View Slide

133
@technoheads
LOCALIZATION
For Etsy, one build on our beefy primary deployment server (32 cores, 64 gigs of ram)
could take as long as 5 or 6 minutes. We could maybe run two builds in parallel in
eight or nine minutes, but that would still mean that all of our builds would take
something like an hour.
We were also constrained by the fact that we were actively moving all of Etsy
from hardware in data centers
to the cloud.
While this was happening, we couldn’t just ask for new hardware, let alone a dozen
or so of the beefiest servers we had.
If we were trying to keep pace with Builda, which could get our code ready in
something like five miuntes, we were gonna have to cheat a little.

View Slide

134
@technoheads
LOCALIZATION
Cheating is weirdly more ok than you'd think here, since Builda essentially cheats too.
When it needs to localize a bunch of code into 11 different locales, Builda doesn't
build the same code 11 times.

View Slide

135
@technoheads
LOCALIZATION
Instead, it inserts placeholder strings around the places where translated strings need
to go.

View Slide

136
]::TRANSLATION_TIME::[See ya
later!]::TRANSLATION_TIME::[
@technoheads
LOCALIZATION
Later, when the build is mostly done, it goes back through each file,

View Slide

137
]::TRANSLATION_TIME::[See ya later!]::TRANSLATION_TIME::[
@technoheads
LOCALIZATION
finds those placeholders,

View Slide

138
]::TRANSLATION TIME::[See ya later!]::TRANSLATION TIME::[
@technoheads
LOCALIZATION
and swaps them out...

View Slide

139
au revoir mon petit chou-fleur
@technoheads
LOCALIZATION
for actual localized strings in every language.

View Slide

140
@technoheads
LOCALIZATION
Compared with Webpack's

View Slide

141
@technoheads
LOCALIZATION
(now-deprecated) internationalization plugin, this seemed like a way faster approach.
I actually didn’t realize it was deprecated until I started writing this talk.
Anyways, we tweaked our plugins that were responsible for providing translated
strings and had them insert plain ol' double quote string placeholders instead.
Webpack would unknowingly build bundles with these strings inlined, which, being
strings, would be left alone through the entire build process.
Once the build finished, we'd be left with a bunch of production-ready files with a
bunch of placeholders in them.

View Slide

142
From there, a plugin would iterate through our assets, do a find-and-replace, and
write a copy of the source to disk with the new locale inlined.
This was easier than it sounds:

View Slide

143
@technoheads
LOCALIZATION
we were able to just take our source code and call .split with our placeholder.

View Slide

144
@technoheads
LOCALIZATION
The resulting array would contain original source code in the even indexes,

View Slide

145
@technoheads
LOCALIZATION
and the string that needed translating
in the odd indexes.
We'd iterate through the array, pick the odd indexes,

View Slide

146
@technoheads
LOCALIZATION
and translate them.
In case you’re curious, we get our translations from a separate service that we
maintain, which dumps static translation files onto our development, deployment, and
production servers.

View Slide

147
NBC owns the rights to this particular Kevin
@technoheads
LOCALIZATION
This method was so fast, in fact, that we were able to provide localization in
development
just by translating each file as it was requested, rather than building each file into
every possible language.
Thanks to this method, we were able to get our production builds ready in well under
5 minutes.

View Slide

148
4ms DELAYED
OUR LAUNCH
[WATER BREAK] Tell your neighbor (in 15 seconds or less) if you can speak more
than one language
For the last part of this talk, lemme tell you a bit about 4 milliseconds that kept us from
launching this whole migration 3 months earlier than we actually did.
We were finally at the point where we felt ready to try and swap out our build systems
on production traffic. Development looked good, production was building quickly, and
a bunch of end-to-end tests showed that everything was still working as expected.

View Slide

149
INDIANA JONES is a trademark of LUCASFILM LTD
Build systems are really, really, really hard to swap though.

View Slide

150
(remember these are files)
4ms MADE US SAD
@technoheads
A new build system has to work in every language, for every page on the site.

View Slide

151
4ms MADE US SAD
@technoheads
And it also has to work in every single browser

View Slide

152
4ms MADE US SAD
@technoheads
No matter how old

View Slide

153
4ms MADE US SAD
@technoheads
Or cutting edge it may be.
If our pages doesn’t work, people will stop using our site, and we’ll probably lose
money.
So what do you do when you want to gain confidence in a huge change?

View Slide

154
O
h
yes
an
A/B
test
4ms MADE US SAD
@technoheads
You run an a/b test.

View Slide

155
EXPERIMENT
4ms MADE US SAD
@technoheads
We actually ran five separate ones, if I’m gonna be totally honest.

View Slide

156
EXPERIMENT
4ms MADE US SAD
@technoheads
And for us, all of them looked really good.
You can see there that all of our important metrics say “No change” next to them,
which is actually as good as we hoped.

View Slide

157
and yet somehow
4ms MADE US SAD
@technoheads
And yet in spite of all this, we had some pretty alarming changes...

View Slide

158
These are our top-secret
performance metrics
4ms MADE US SAD
@technoheads
...in every one of our browser performance metrics.
Every page was running way, way slower than expected.

View Slide

159
w
hat in
sw
eet heck
4ms MADE US SAD
@technoheads
Some pages were over 14% slower! That’s unheard of.
It’s a general rule of thumb that if your site’s performance gets worse, so does your
site’s money.
We had to figure out what was going wrong before we could launch Webpack.

View Slide

160
4ms MADE US SAD
@technoheads
Before I go on, let’s talk about some of our metrics really quick.
Some of you may already know this stuff, some of you may not, but I hope it’ll be a
good refresher nonetheless.
At Etsy, we look particularly at two client-side performance metrics:

View Slide

161
4ms MADE US SAD
@technoheads
DOMContentLoaded

View Slide

162
4ms MADE US SAD
@technoheads
And Page Load

View Slide

163
When the browser parses our HTML, it goes through line at a time

View Slide

164
And evaluates everything as it goes.

View Slide

165
For most resources like this stylesheet,
it downloads and executes it before continuing to parse the page

View Slide

166
Some things,
like this image tag,

View Slide

167
will kick off a request,
but will not block the browser from continuing to parse the page

View Slide

168
Here is our javascript, at the very bottom.
Like CSS, JavaScript is downloaded and executed before the browser can continue.

View Slide

169
● Network requests
● setTimeout
Some javascript code,
like network requests or setTimeout calls,
do not block the browser,
and are added to the list of tasks a browser can take care of at a later time.

View Slide

170
Finally, when the browser finishes parsing all of our HTML,

View Slide

171
DOMContentLoaded!
it fires the DOMContentLoaded event,
which says that the structure of the page is complete,
even if we’re still waiting for some other resources (like that image)

View Slide

172
etsy.comshop/catHairAndTeeth
Once all of our subresources load (like that image), and when the browser has run out
of tasks to immediately take care of,

View Slide

173
page load!
it fires the “page load” event, effectively marking the page as “done”

View Slide

174
load (or pageLoad)
The load event is fired when the whole page has loaded, including
all dependent resources, and there are no outstanding scripts
waiting to be executed.
DOMContentLoaded
The DOMContentLoaded event fires when the initial HTML
document has been completely loaded and parsed, without
waiting for stylesheets (sometimes), images, and subframes to
finish loading.
4ms MADE US SAD
@technoheads
To recap:
DOMContentLoaded fires when the page’s structure is in place,
and load fires when everything is loaded, and no scripts are waiting to be executed
(also v important)
Both of these metrics were consistently slower for us. This raised two questions

View Slide

175
1. What could we have
missed?
4ms MADE US SAD
@technoheads
What thing is so different, yet so small that we didn’t notice?

View Slide

176
missed?
2. Why are people still
buying things?
4ms MADE US SAD
@technoheads
If something so different is happening, why the hell are people buying things at all?

View Slide

177
investigation time.
4ms MADE US SAD
@technoheads
We started out by investigating our performance monitoring code for bugs.
We also double-checked that we weren’t somehow sending more code to users than
we thought we were.
When both of these investigations turned up no new leads, we started to wonder if
Webpack’s code was somehow RUNNING differently, despite otherwise looking the
same.
To confirm this hunch, we decided to profile our JavaScript as it ran in the browser.

View Slide

178
Builda
4ms MADE US SAD
@technoheads
This is a flame graph for the listing page, which is the page that shows an item for
sale on Etsy. This graph shows our code, built with Builda.
Here, the horizontal axis represents time
And vertical access represents layers of execution.
Every rectangle represents a function call or a piece of execution, and everything
below it is a function call sub-task related to it.

View Slide

179
“Evaluate Script (main.[version].en-US.js)”
Builda
4ms MADE US SAD
@technoheads
This particular slice of the graph shows what the browser does when it parses
main.js, the primary javascript file on this page.
Very quiet, not much going on

View Slide

180
(it took 18ms here)
Builda
4ms MADE US SAD
@technoheads
The whole thing doesn’t even take long (18ms)
Now what does Webpack look like for the same code?

View Slide

181
Webpack
4ms MADE US SAD
@technoheads
It looks… hectic.

View Slide

182
Webpack
4ms MADE US SAD
@technoheads
This is the same file as in the previous graph.

View Slide

183
Webpack
(it took a whopping 65ms here)
But it’s taking almost 4 times as long, and a lot more is clearly happening

View Slide

184
Builda Webpack
4ms MADE US SAD
@technoheads
For kicks, here are those two graphs side by side.
Clearly, Builda-built code appeared to be doing way less than Webpack-built code.

View Slide

185
Builda Webpack
4ms MADE US SAD
@technoheads
This is concerning.

View Slide

186
yes but how come
4ms MADE US SAD
@technoheads
So, why?
Even if our conversion numbers look good, this huge difference may mean we’re
missing an obvious performance enhancement!

View Slide

187
4ms MADE US SAD
@technoheads
Let’s take another look at the whole flame graph

View Slide

188
(Builda)
4ms MADE US SAD
@technoheads
This one is builda’s

View Slide

189
(Builda)
4ms MADE US SAD
@technoheads
This is roughly the area we were looking at before, where our browser downloaded,
parsed, and executed our JavaScript.

View Slide

190
(Builda)
4ms MADE US SAD
@technoheads
This is where DOMContentLoaded is fired

View Slide

191
(Builda)
4ms MADE US SAD
@technoheads
This here is page load

View Slide

192
(Builda)
4ms MADE US SAD
@technoheads
And this right here is apparently where the contents of our JavaScript file actually get
executed

View Slide

193
4ms MADE US SAD
@technoheads
Let me back up a little here and zoom in.

View Slide

194
4ms MADE US SAD
@technoheads
See this rectangle right here?

View Slide

195
This is code from our
javascript ﬁle, running
later than we thought.
4ms MADE US SAD
@technoheads
This little rectangle is the body of our JavaScript file, which, for some reason, is
getting executed a lot later than we thought
So it seems like Builda’s code is somehow running well after the page load event has
fired, which doesn’t seem to be the case with Webpack’s code.

View Slide

BAD
PERF
BAD
MEASURE
MENT
In other words, our performance metrics were not accounting for the time it took to
actually run our JavaScript code in the browser, when we built that code with Builda.
Code built with Webpack did not have this problem.

View Slide

197
missed?
buying shit?
4ms MADE US SAD
@technoheads
So we know why people are still buying stuff:
the same things were happening in the browser, but our measurements moved.

View Slide

198
missed?
buying shit?
4ms MADE US SAD
@technoheads
We still need to know why, though, mostly cuz we were curoius.

View Slide

199
load (or pageLoad)
DOMContentLoaded
waiting for stylesheets, images, and subframes to ﬁnish loading.
4ms MADE US SAD
@technoheads
Let’s look back at our two metrics, DCL and page load

View Slide

200
load (or pageLoad)
DOMContentLoaded
load (or pageLoad)
4ms MADE US SAD
@technoheads
Specifically, look at load

View Slide

201
load (or pageLoad)
DOMContentLoaded
load (or pageLoad)
4ms MADE US SAD
@technoheads
Specifically specifically, look at this bit here where it says:
“there are no outstanding scripts waiting to be executed”
MAYBE the load event could fire if the browser runs out of things to do.
What if something was telling our scripts to wait a second before running, tricking the
browser into thinking the page was ready?

View Slide

202
4ms MADE US SAD
@technoheads
On that note, let’s REALLY QUICKLY dig into the meat of require.js.
As a reminder, Require.js is the library that Builda was built around. We send a copy
of it to our browsers,
where it’s responsible for starting up our code and managing our code’s
dependencies.

View Slide

203
4ms MADE US SAD
@technoheads
Here’s where require.js kicks off our code’s execution

View Slide

204
4ms MADE US SAD
@technoheads
But it’s wrapped in this nextTick thing. What’s that?

View Slide

205
4ms MADE US SAD
@technoheads
If we dig further, it looks like it’s just…

View Slide

206
4ms MADE US SAD
@technoheads
a 4ms set timeout call.
That would tell the browser that our code can be executed at least 4ms later
This isn’t a hard and fast time; the browser is totally allowed to push things back a
little if it needs.
In other words, Require.js tells the browser that none of our code needs to be run
right away.

View Slide

207
4ms MADE US SAD
@technoheads
Require.js might do this because it gives the browser a second to finish loading the
page before starting to execute any JavaScript.

View Slide

208
4ms MADE US SAD
@technoheads
It also might be because using setTimeout to make your perf numbers better was
really popular in 2010.

View Slide

209
4ms MADE US SAD
@technoheads
It’s hard to know for sure, but we do know that WebPack does not do this behavior,
which means its performance numbers are a lot more trustworthy.
So knowing that this is the bug, what happens next?

View Slide

210
1. No we did not just
setTimeout() everything
4ms MADE US SAD
@technoheads
For one, we did not modify all of our code to run inside setTimeout calls.
Making our numbers better means nothing if the user experience remains exactly the
same,
and adding setTimeout calls just hides our real performance time from our metrics.

View Slide

211
1. No we did not just
setTimeout() everything
2. Yes we did add a bunch of
our own made up metrics
4ms MADE US SAD
@technoheads
But we did add some new metrics to help validate that Webpack code wasn’t actually
that slow.
We went to the bottom of the main js file on our top five pages and added a timer
there,
which roughly measured how long it took for us to download and run that JS file to
completion.

View Slide

212
4ms MADE US SAD
@technoheads
We then graphed those timers between both Builda and Webpack
Yellow is webpack, and lower is better.

View Slide

213
4ms MADE US SAD
@technoheads
Our team was very happy for at last, Webpack was ready for prime time.

View Slide

214
WHAT DID WE
LEARN?
With everything said and done, it took us almost a year and a half to move to
Webpack.
We really hoped that it would be a silver bullet that would just solve all our problems,
but at the end of the day,
we had a lot to learn about build systems, performance, and our own codebase,
regardless of what we migrated to.

View Slide

215
@technoheads
WHAT DID WE LEARN
I felt like there should be a three-point slide with our learnings,
which was so so difficult to write given how many things we learned during this
migration.
But if I had to just pick three things, I think it’d look like this:

View Slide

216
1. Your code only gets bigger
@technoheads
WHAT DID WE LEARN
1. Your codebase gets bigger, not smaller. If you’re trying to find small
performance wins to get some build time under some threshold, it probably
won’t last.
A solution that anticipates your problem growing twice in size will always win
out in the long run.

View Slide

217
2. Question best practices
@technoheads
WHAT DID WE LEARN
2. Question what other companies, websites, and articles claim to be best practices!
What works well for one person may not always work well everywhere.
If we accepted that the Webpack internationalization plugin was our only option, we’d
still be waiting an hour for our JavaScript to build in all of our languages.

View Slide

218
3. It’s ok if it takes time
@technoheads
WHAT DID WE LEARN
And finally, if you can afford to take extra time on a big, risky project, you absolutely
should.
One of the nicer parts of our migration was that no one was critically stuck by our
work.
We really hated that a little metrics bug cost us months of time when we were so
close to being done, but we ended up learning much more
about performance,
the browser,
and our own codebase because of it.
At the end of the day,
after almost a year and a half of wild problems and hair loss,

View Slide

219
3. It’s ok if it takes time
@technoheads
WHAT DID WE LEARN
Webpack is still pretty great software.

View Slide

220
Thanks
(@technoheads)
If you have any questions, hit me up on slack or on twitter. If you like this stuff, my
team is hiring.
And of course, thank you all so much.

View Slide

JSCONF 2019: WE STARTED USING WEBPACK AND IT TOOK A WHILE

JSCONF 2019: WE STARTED USING WEBPACK AND IT TOOK A WHILE

Salem

Other Decks in Programming

Featured

Transcript