Unbundling the JavaScript module bundler - Codemotion Rome 2018

Today we all use Webpack (right?), but I remember a time when you had to manually copy-paste JavaScript files to create a package of libraries you could use in your frontend application. Many years have passed since then and the landscape of module bundlers evolved significantly along with the evolution of JavaScript and Node.js. In this talk, I will try to uncover some JavaScript module history and illustrate how a module bundler actually works, so that the next time you will use Webpack you will be able to understand what's going on behind the scenes.

Transcript

1. Unbundling the JavaScript module bundler Unbundling the JavaScript module bundler { Luciano Mammino ( Luciano Mammino ( ) ) @loige @loige ROME - APRIL 13/14 2018 loige.link/bundle-rome 1
2. loige.link/bundle-rome 2
3. Luciano... who Luciano... who Find me online: - (@loige) - (lmammino) - - (loige.co) Twitter GitHub Linkedin Blog Solution Architect at with @mariocasciaro with @andreaman87 with @Podgeypoos79 3
4. 1. A sample app 2. what is a dependency 3. JavaScript module systems 4. How a module bundler works 5. Webpack in 2 minutes! Agenda Agenda 4
5. http://poo.loige.co 5
6. App features App features 6
7. Dynamic DOM manipulation React, Angular & Vue are so... overrated! 7
8. Dynamic Favicon favico.js 8
9. Custom animated tooltips tippy.js 9
10. Confetti rainfall dom-confetti 10
11. Persist state through Local Storage + UUIDs + store2 uuidjs 11
12. 7 Requests only for the JS code! 12
13. current scenario current scenario zepto@1.2.0/dist/zepto.min.js uuidjs@4.0.3/dist/uuid.core.js store2@2.7.0/dist/store2.min.js tippy.js@2.2.3/dist/tippy.all.min.js confetti@0.0.10/lib/main.js favico.js@0.3.10/favico­0.3.10.min.js 13
14. Ideal scenario Ideal scenario zepto@1.2.0/dist/zepto.min.js uuidjs@4.0.3/dist/uuid.core.js store2@2.7.0/dist/store2.min.js tippy.js@2.2.3/dist/tippy.all.min.js confetti@0.0.10/lib/main.js favico.js@0.3.10/favico­0.3.10.min.js vendors.js + + + + + = 14
15. How to do this? How to do this? 15
16. ./buildVendors.sh > vendors.js buildVendors.sh $ 16
17. npm install ­­global lumpy $ github.com/lmammino/lumpy 17
18. # lumpy.txt https://unpkg.com/zepto@1.2.0/dist/zepto.min.js https://unpkg.com/uuidjs@4.0.3/dist/uuid.core.js https://unpkg.com/store2@2.7.0/dist/store2.min.js https://unpkg.com/tippy.js@2.2.3/dist/tippy.all.min.js https://unpkg.com/confetti­js@0.0.11/dist/index.min.js https://unpkg.com/dom­confetti@0.0.10/lib/main.js https://unpkg.com/favico.js@0.3.10/favico­0.3.10.min.js Lumpy allows you to define all the vendors in a text file 18
19. lumpy build $ 1. Downloads the files from lumpy.txt (and caches them) 2. Concatenates the content of the files 3. Minifies the resulting source code (using ) 4. Saves the resulting content in vendors.js babel-minify 19
20. 7 requests 2 requests 20
21. Concatenation Concatenation + + Minification Minification 21
22. This is good... This is good... ... in 2008 ... in 2008 was was 22
23. Today... Today... We can do better! We can do better! 23
24. Updating dependencies should be easy Updating dependencies should be easy We don't want to worry about dependencies We don't want to worry about dependencies of dependencies of dependencies We don't have to worry about the order of We don't have to worry about the order of imports imports Today... Today... 24
25. Dependency Dependency (or coupling) a state in which one object uses a function of another object — Wikipedia 25
26. Reusable dependencies... Reusable dependencies... Modules! Modules! 26
27. Modules Modules The bricks for structuring non-trivial applications, but also the main mechanism to enforce information hiding by keeping private all the functions and variables that are not explicitly marked to be exported — * Node.js Design Patterns (Second Edition) * yeah, I quite like quoting my stuff... 27
28. Meet my friend Meet my friend I.I.F.E. I.I.F.E. (Immediately Invoked Function Expression) loige.link/iife 28
29. We generally define a function this way We generally define a function this way const sum = (a, b) => a + b sum(a, b) { return a + b } or or then, at some point, we execute it... then, at some point, we execute it... const four = sum(2, 2) 29
30. A function in JS creates an isolated scope A function in JS creates an isolated scope (a, b) => { const secretString = "Hello" return a + b } console.log(secretString) // undefined secretString secretString is not visible outside the function is not visible outside the function 30
31. )(arg1, arg2) IIFE allows you to define an isolated IIFE allows you to define an isolated scope that executes itself scope that executes itself (arg1, arg2) => { // do stuff here const iAmNotVisibleOutside = true } ( 31
32. IIFE is a recurring pattern in IIFE is a recurring pattern in JavaScript modules JavaScript modules 32
33. Let's implement a module Let's implement a module that provides: that provides: Information hiding Information hiding exported functionalities exported functionalities 33
34. const myModule = (() => { const privateFoo = () => { /* ... */ } const privateBar = [ /* ... */ ] const exported = { publicFoo: () => { /* ... */ }, publicBar: [ /* ... */ ] }; return exported })() A module myModule.publicFoo() myModule.publicBar[0] myModule.privateFoo // undefined myModule.privateBar // undefined privateFoo // undefined privateBar // undefined 34
35. We want modules to be We want modules to be reusable reusable across different apps and across different apps and organisations... organisations... ...we need ...we need A A STANDARD MODULE STANDARD MODULE format! format! 35
36. Module system features Module system features Must have Simple syntax for import / export Information hiding Allows to define modules in separate files Modules can import from other modules (nested dependencies) 36
37. Module system features Module system features Nice to have Ability to import module subsets Avoid naming collision Asynchronous module loading Seamless support for Browsers & Server-side 37
38. xkcd.com/927 38
39. JavaScript module systems JavaScript module systems globals CommonJS (Node.js) AMD (Require.js / Dojo) UMD ES2015 Modules (ESM) Many others (SystemJS, ...) 39
40. Globals Globals var $, jQuery $ = jQuery = (() => { return { /* ... */ } })() // ... use $ or jQuery in the global scope $.find('.button').remove() 40
41. Globals Globals Might generate naming collisions (e.g. $ overrides browser global variable) Import order is important Cannot import parts of modules 41
42. // or import single functionality const { concat } = require('./loDash') concat([1], [2], [3]) // app.js // import full module const _ = require('./loDash') _.concat([1], [2], [3]) // loDash.js const loDash = { /* ... */ } module.exports = loDash CommonJS CommonJS module app using module 42
43. CommonJS CommonJS No naming collisions (imported modules can be renamed) Huge repository of modules through Synchronous import only Works natively on the server side only (Node.js) NPM 43
44. AMD ( AMD ( ) ) Asynchronous Module Definition Asynchronous Module Definition Require.js Require.js // jquery-1.9.0.js define( 'jquery', ['sizzle', 'jqueryUI'], function (sizzle, jqueryUI) { // Returns the exported value return function () { // ... } } ) module 44
45. AMD ( AMD ( ) ) Asynchronous Module Definition Asynchronous Module Definition Require.js Require.js // app.js // define paths requirejs.config({ baseUrl: 'js/lib', paths: { jquery: 'jquery-1.9.0' } }) define(['jquery'], function ($) { // this is executed only when jquery // and its deps are loaded }); app 45
46. AMD ( AMD ( ) ) Asynchronous Module Definition Asynchronous Module Definition Require.js Require.js Asynchronous modules Works on Browsers and Server side Very verbose and convoluted syntax (my opinion™) 46
47. UMD UMD Universal Module Definition Universal Module Definition A module definition that is compatible with Global modules, CommonJS & AMD github.com/umdjs/umd 47
48. (function (root, factory) { if (typeof exports === 'object') { // CommonJS module.exports = factory(require('dep')) } else if (typeof define === 'function' && define.amd) { // AMD define(['dep'], function (dep) { return (root.returnExportsGlobal = factory(dep)) }) } else { // Global Variables root.myModule = factory(root.dep) } }(this, function (dep) { // Your actual module return {} })) loige.link/umd 48
49. Allows you to define modules that can be used by almost any module loader Complex, the wrapper code is almost impossible to write manually UMD UMD Universal Module Definition Universal Module Definition 49
50. // calculator.js const add = (num1, num2) => num1 + num2 const sub = (num1, num2) => num1 - num2 const div = (num1, num2) => num1 / num2 const mul = (num1, num2) => num1 * num2 export { add, sub, div, mul } // app.js import { add } from './calculator' console.log(add(2,2)) // 4 ES2015 modules ES2015 modules module app import specific functionality 50
51. // index.html <html> <body> <!-- ... --> <script type="module"> import { add } from 'calculator.js' console.log(add(2,2)) // 4 </script> </body> </html> ES2015 modules ES2015 modules "works" in some modern browsers 51
52. ES2015 modules ES2015 modules Syntactically very similar to CommonJS... BUT import & export are static (allow static analysis of dependencies) It is a (still work in progress) standard format Works (almost) seamlessly in browsers & servers 52
53. ES2015 modules ES2015 modules Cool & broad subject, it would deserve it's own talk Wanna know more? / syntax reference import export ECMAScript modules in browsers ES modules: A cartoon deep-dive ES Modules in Node Today! 53
54. So many options... So many options... Current most used practice: Use CommonJS or ES2015 & create "compiled bundles" 54
55. Let's try to use Let's try to use CommonJS in the CommonJS in the browser browser 55
56. const $ = require('zepto') const tippy = require('tippy.js') const UUID = require('uuidjs') const { confetti } = require('dom-confetti/src/main') const store = require('store2') const Favico = require('favico.js') !(function () { const colors = ['#a864fd', '#29cdff', '#78ff44', '#ff718d', '#fdff6a'] const todoApp = (rootEl, opt = {}) => { const todos = opt.todos || [] let completedTasks = opt.completedTasks || 0 const onChange = opt.onChange || (() => {}) const list = rootEl.find('.todo-list') const footer = rootEl.find('.footer') const todoCount = footer.find('.todo-count') const insertInput = rootEl.find('.add-todo-box input') const insertBtn = rootEl.find('.add-todo-box button') const render = () => { let tips list.html('') The browser doesn't know how to process require. It doesn't support CommonJS! 56
57. Module Bundler Module Bundler A tool that takes modules with dependencies and emits static assets representing those modules Those static assets can be processed by browsers! 57
58. Dependency graph Dependency graph A graph built by connecting every module with its direct dependencies. app dependency A dependency B dependency A2 shared dependency 58
59. A module bundler has to: A module bundler has to: Construct the dependency graph (Dependency Resolution) Assemble the modules in the graph into a single executable asset (Packing) 59
60. // app.js const calculator = require('./calculator') const log = require('./log') log(calculator('2 + 2 / 4')) // log.js module.exports = console.log // calculator.js const parser = require('./parser') const resolver = require('./resolver') module.exports = (expr) => resolver(parser(expr)) // parser.js module.exports = (expr) => { /* ... */ } // resolver.js module.exports = (tokens) => { /* ... */ } Dependency resolution Dependency resolution app calculator log parser resolver 60 (entrypoint) (1) (2) (3) (4) (5)
61. During During dependency resolution dependency resolution, , the bundler creates a the bundler creates a modules map modules map { } './app': (module, require) => { … }, './calculator': (module, require) => { … }, './log': (module, require) => { … }, './parser': (module, require) => { … }, './resolver': (module, require) => { … }, const parser = require('./parser');const resolver = require('./resolver');module.exports = (expr) => resolver(parser(expr)) 61
62. Packing Packing .js { ­­­ : ­­­ ­­­ : ­­­­ ­­­ : ­­ } Modules map Modules map Single executable Single executable JS file JS file 62
63. Packed executable file Packed executable file ((modulesMap) => { const require = (name) => { const module = { exports: {} } modulesMap[name](module, require) return module.exports } require('./app') })( { './app': (module, require) => { … }, './calculator': (module, require) => { … }, './log': (module, require) => { … }, './parser': (module, require) => { … }, './resolver': (module, require) => { … } } ) 63
64. Now you know how Now you know how Module bundlers work! Module bundlers work! And how to convert code written And how to convert code written using using CommonJS CommonJS to a single file to a single file that works in the browser that works in the browser 64
65. A challenge for you! A challenge for you! If you do, ... I'll have a prize for you! TIP: you can use to parse JavaScript files (look for require and module.exports) and to map relative module paths to actual files in the filesystem. let me know acorn resolve Can you build a (simple) module bundler from scratch? Can you build a (simple) module bundler from scratch? 65
66. A state of the art module A state of the art module bundler for the web bundler for the web 66
67. npm install webpack webpack­cli 67
68. webpack app.js yep, recent versions of Webpack work without config! 68
69. webpack ­­mode=development app.js Do not compress the code and add annotations! 69
70. loige.link/sample-webpacked-file 70
71. Webpack concepts Webpack concepts Entry point: the starting file for dependency resolution. Output: the destination file (bundled file). Loaders: algorithms to parse different file types and convert them into executable javascript (e.g. babel, typescript, but also CSS, images or other static assets) Plugins: do extra things (e.g. generate a wrapping HTML or analysis tools) 71
72. const { resolve, join } = require('path') const CompressionPlugin = require('compression-webpack-plugin') module.exports = { entry: './app.js', output: { path: resolve(join(__dirname, 'build')), filename: 'app.js' }, module: { rules: [ { test: /\.js$/, exclude: /(node_modules|bower_components)/, use: { loader: 'babel-loader', options: { presets: [ ['@babel/preset-env'] ] } } } ] }, plugins: [ new CompressionPlugin() ] } Webpack.config.js Plugins Uses a plugin that generates a gzipped copy of every emitted file. 72
73. ...And you can do more! ...And you can do more! Dev Server Dev Server Tree shaking Tree shaking Dependencies analytics Dependencies analytics Source maps Source maps Async require / module splitting Async require / module splitting 73
74. Module bundlers are your friends Module bundlers are your friends Now you know how they work, they are not (really) magic! Start small and add more when needed If you try to build your own you'll learn a lot more! 74
75. Grazie! Grazie! loige.link/bundle-rome Special thanks to: , , (reviewers) and (inspiration - check out his and his ) @Podgeypoos79 @andreaman87 @mariocasciaro @MarijnJH amazing book workshop on JS modules 75