Scaling Webpack to Thousands of Concurrent Builds

Scaling Webpack to Thousands of Concurrent Builds

Building JavaScript for the web is easier than it's ever been with babel, webpack & rollup doing all the heavy lifting for individual builds – but what if you need a lot of builds across multiple environments like development, test, and production? Introduced at Node.js Interactive 2016 Warehouse.ai is an open-source distributed build system supporting webpack, babel, and browserify that aims to make Serverless front-end deployments & rollback easy.

This talk will cover the challenges (and solutions) for Warehouse to scale webpack & npm install to thousands of daily builds with high bursts of concurrency during peak hours.

D43e8ea63b61e7669ded5b9d3c2e980f?s=128

Charlie Robbins

October 11, 2018
Tweet

Transcript

  1. None
  2. WEBPACK Scaling To Thousands of CONCURRENT BUILDS

  3. WHY, HELLO THERE CHARLIE HERE

  4. Who am I? Just

  5. SENIOR DIR., UX PLATFORM @ GoDaddy

  6. SENIOR DIR., UX PLATFORM @ GoDaddy BOARD MEMBER, NODE.JS FOUNDATION

    FORMERLY FOUNDER at NODEJITSU @INDEXZERO GITHUB TWITTER SO WHAT’S ALL THIS THEN?
  7. What am I going to talk about? 5 1 Define

    the problem 2 Webpack it up, webpack it in 3 Initial solutions & naive approaches 4 DNS & HTTP Starvation 5 Putting it all together
  8. LET’S DEFINE THE PROBLEM ITSELF

  9. WEBPACK service as a

  10. LET’S TALK webpack about

  11. YOUR CODE

  12. YOUR CODE “WEBPACK”

  13. BUNDLE BUNDLE BUNDLE BUNDLE

  14. BUNDLE BUNDLE BUNDLE BUNDLE there is more Of course under

    the covers
  15. F I L E S TO DISK written

  16. CPU BUILD TIME and lots of

  17. PLACEHOLDER

  18. PLACEHOLDER MINIFICATION

  19. Copyright© 2017 Godaddy Inc. All Rights Reserved.

  20. Copyright© 2017 Godaddy Inc. All Rights Reserved. CONFIG CONFIG lots

    of
  21. None
  22. WANT THE DEVELOPERS THE HOMER CAR

  23. None
  24. HELL

  25. HELL webpack.config.js is other people’s

  26. LET’S GET BACK TO THE PROBLEM ITSELF

  27. WEBPACK service as a

  28. None
  29. It must be very reproducible

  30. None
  31. npm install Store output as a tarball

  32. Your code

  33. Your code npm install

  34. Your code npm install store tarball

  35. Your code npm install fetch tarbll store tarball

  36. Your code webpack npm install fetch tarbll store tarball

  37. Your code webpack npm install fetch tarbll store tarball store

    files
  38. None
  39. npm run build But what output should we store?

  40. None
  41. THE FIRST OF MANY SMALL QUESTIONS

  42. NPM PUBLISH PAYLOAD npm install registry-mock -g ./registry-echo PROTIP! LEARN

    HOW THIS WORKS
  43. a naive solution consider

  44. Your computer

  45. Your computer npm install requests

  46. Your computer store tarball npm install requests

  47. Your computer store tarball npm install requests webpack requests

  48. Your computer store tarball store files npm install requests webpack

    requests
  49. it was that
 simple if only

  50. Your computer

  51. Your computer npm install requests

  52. Your computer store tarball npm install requests

  53. Your computer store tarball npm install requests webpack requests

  54. Your computer store tarball store files npm install requests webpack

    requests
  55. Your computer store tarball store files npm install requests webpack

    requests DNS Caching
  56. CONSIDER scale it at

  57. Everyone’s computer

  58. Everyone’s computer npm install requests webpack requests

  59. from a pool of servers Everyone’s computer npm install requests

    webpack requests
  60. from a pool of servers Everyone’s computer npm install requests

    webpack requests DNS Caching overloads one machine
  61. None
  62. STARVATION

  63. about queues? what

  64. PLACEHOLDER durability

  65. PLACEHOLDER durability DURABILITY? PERSISTENCE?

  66. PLACEHOLDER durability IN ORDER? OUT OF ORDER? DURABILITY? PERSISTENCE? DELIVERY

    GUARANTEES?
  67. characteristics the

  68. characteristics the of build systems

  69. IDEMPOTENT

  70. IDEMPOTENT ORDER DOES NOT MATTER

  71. IDEMPOTENT ORDER DOES NOT MATTER EVENTUALLY CONSISTENT

  72. IDEMPOTENT ORDER DOES NOT MATTER EVENTUALLY CONSISTENT NSQ MEETS ALL

    THESE
  73. PUTTING together it all

  74. Warehouse.ai lessons from PROD

  75. carpenterd carpenterd-worker and

  76. Your computer npm publish

  77. Your computer npm publish Warehouse.ai

  78. Your computer npm publish Warehouse.ai npm install requests carpenterd npm

    publish completes to any registry
  79. carpenterd npm install

  80. carpenterd npm install store tarball webpack requests

  81. carpenterd npm install store tarball webpack requests webpack executes

  82. carpenterd npm install store tarball webpack requests store files webpack

    executes
  83. carpenterd npm install store tarball webpack requests store files webpack

    executes
  84. carpenterd npm install store tarball webpack requests store files webpack

    executes P U L L PUSH don’t
  85. FUTURES

  86. None
  87. WHAT IS SCALE?

  88. spikes within Handling system workload

  89. ...WRAPPING UP...

  90. ...WRAPPING UP... GITHUB.COM/GODADDY/WAREHOUSE.AI GITHUB.COM/GODADDY/CARPENTERD GITHUB.COM/WAREHOUSEAI SPECIAL THANKS TO JARRETT CRUGER

    GEOFF WALL, MICHAEL LUTHER, AND ALL CONTRIBUTORS!
  91. MAY THE SOURCE BE WITH YOU ICONS CREATED BY DINOSOFT

    LABS FROM THE NOUN PROJECT