Scaling Webpack to Thousands of Concurrent Builds

Building JavaScript for the web is easier than it's ever been with babel, webpack & rollup doing all the heavy lifting for individual builds – but what if you need a lot of builds across multiple environments like development, test, and production? Introduced at Node.js Interactive 2016 Warehouse.ai is an open-source distributed build system supporting webpack, babel, and browserify that aims to make Serverless front-end deployments & rollback easy.

This talk will cover the challenges (and solutions) for Warehouse to scale webpack & npm install to thousands of daily builds with high bursts of concurrency during peak hours.

Transcript

2. WEBPACK Scaling To Thousands of CONCURRENT BUILDS
3. WHY, HELLO THERE CHARLIE HERE
4. Who am I? Just
5. SENIOR DIR., UX PLATFORM @ GoDaddy
6. SENIOR DIR., UX PLATFORM @ GoDaddy BOARD MEMBER, NODE.JS FOUNDATION FORMERLY FOUNDER at NODEJITSU @INDEXZERO GITHUB TWITTER SO WHAT’S ALL THIS THEN?
7. What am I going to talk about? 5 1 Define the problem 2 Webpack it up, webpack it in 3 Initial solutions & naive approaches 4 DNS & HTTP Starvation 5 Putting it all together
8. LET’S DEFINE THE PROBLEM ITSELF
9. WEBPACK service as a
10. LET’S TALK webpack about
11. YOUR CODE
12. YOUR CODE “WEBPACK”
13. BUNDLE BUNDLE BUNDLE BUNDLE
14. BUNDLE BUNDLE BUNDLE BUNDLE there is more Of course under the covers
15. F I L E S TO DISK written
16. CPU BUILD TIME and lots of
17. PLACEHOLDER
18. PLACEHOLDER MINIFICATION
19. Copyright© 2017 Godaddy Inc. All Rights Reserved.
20. Copyright© 2017 Godaddy Inc. All Rights Reserved. CONFIG CONFIG lots of
22. WANT THE DEVELOPERS THE HOMER CAR
24. HELL
25. HELL webpack.config.js is other people’s
26. LET’S GET BACK TO THE PROBLEM ITSELF
27. WEBPACK service as a
29. It must be very reproducible
31. npm install Store output as a tarball
32. Your code
33. Your code npm install
34. Your code npm install store tarball
35. Your code npm install fetch tarbll store tarball
36. Your code webpack npm install fetch tarbll store tarball
37. Your code webpack npm install fetch tarbll store tarball store files
39. npm run build But what output should we store?
41. THE FIRST OF MANY SMALL QUESTIONS
42. NPM PUBLISH PAYLOAD npm install registry-mock -g ./registry-echo PROTIP! LEARN HOW THIS WORKS
43. a naive solution consider
44. Your computer
45. Your computer npm install requests
46. Your computer store tarball npm install requests
47. Your computer store tarball npm install requests webpack requests
48. Your computer store tarball store files npm install requests webpack requests
49. it was that
 simple if only
50. Your computer
51. Your computer npm install requests
52. Your computer store tarball npm install requests
53. Your computer store tarball npm install requests webpack requests
54. Your computer store tarball store files npm install requests webpack requests
55. Your computer store tarball store files npm install requests webpack requests DNS Caching
56. CONSIDER scale it at
57. Everyone’s computer
58. Everyone’s computer npm install requests webpack requests
59. from a pool of servers Everyone’s computer npm install requests webpack requests
60. from a pool of servers Everyone’s computer npm install requests webpack requests DNS Caching overloads one machine
62. STARVATION
63. about queues? what
64. PLACEHOLDER durability
65. PLACEHOLDER durability DURABILITY? PERSISTENCE?
66. PLACEHOLDER durability IN ORDER? OUT OF ORDER? DURABILITY? PERSISTENCE? DELIVERY GUARANTEES?
67. characteristics the
68. characteristics the of build systems
69. IDEMPOTENT
70. IDEMPOTENT ORDER DOES NOT MATTER
71. IDEMPOTENT ORDER DOES NOT MATTER EVENTUALLY CONSISTENT
72. IDEMPOTENT ORDER DOES NOT MATTER EVENTUALLY CONSISTENT NSQ MEETS ALL THESE
73. PUTTING together it all
74. Warehouse.ai lessons from PROD
75. carpenterd carpenterd-worker and
76. Your computer npm publish
77. Your computer npm publish Warehouse.ai
78. Your computer npm publish Warehouse.ai npm install requests carpenterd npm publish completes to any registry
79. carpenterd npm install
80. carpenterd npm install store tarball webpack requests
81. carpenterd npm install store tarball webpack requests webpack executes
82. carpenterd npm install store tarball webpack requests store files webpack executes
83. carpenterd npm install store tarball webpack requests store files webpack executes
84. carpenterd npm install store tarball webpack requests store files webpack executes P U L L PUSH don’t
85. FUTURES
87. WHAT IS SCALE?
88. spikes within Handling system workload
89. ...WRAPPING UP...
90. ...WRAPPING UP... GITHUB.COM/GODADDY/WAREHOUSE.AI GITHUB.COM/GODADDY/CARPENTERD GITHUB.COM/WAREHOUSEAI SPECIAL THANKS TO JARRETT CRUGER GEOFF WALL, MICHAEL LUTHER, AND ALL CONTRIBUTORS!
91. MAY THE SOURCE BE WITH YOU ICONS CREATED BY DINOSOFT LABS FROM THE NOUN PROJECT