To ESM is human: The journeyman's guide to ES Modules in Node.js

There’s only one thing more complicated than ES Modules on browsers – ES Modules in Node.js. But how did we get here? How do modules even work? What are the merits and demerits around these conflicting module systems and why does everyone get so emotional while talking about them?

Join me, a Node.js core collaborator and an avid contributor to V8 in order to know all these and more. You will come away with a deep understanding of how the module systems work internally in Node.js and what are the challenges facing these systems, moving forwards.

Transcript

1. You’re “hooked”! A guide to using React Hooks in production 1 Ujjwal Sharma @ryzokuken
2. To ESM is human The Journeyman’s Guide to Modules in Node.js 2 Ujjwal Sharma @ryzokuken
3. Ujjwal Sharma (@ryzokuken) •Igalia - CE Student (Compilers Group) •Node.js – Core Collaborator •Electron – Maintainer (Upgrades WG) •V8 and TC39 Contributor •Google Summer of Code •Student •Speaker !3 @ryzokuken
4. DISCLAIMER •The cake is a lie. •This presentation is purely educational. •There are no grand discoveries, no revelations, no clever innovations. •Just systems, trusty old systems. !4
5. Then… Why? !5
6. !6
7. !7
8. – Lin Clark “Many JavaScript developers know that ES modules have been controversial. But few actually understand how ES modules work.” !8
9. BUT WHAT’S WRONG WITH THAT !9
10. !10
11. !11
12. !12
13. QUESTIONS 1. What are ES Modules? 2. What do they achieve? Why are they necessary? 3. What are the issues with ESM? How are they being resolved/mitigated? 4. What are our alternatives? !13
14. LET’S REVISIT THE FUNDAMENTALS !14
15. What is a module? !15 @ryzokuken
16. module noun. /ˈmɒdjuːl/ Modules are the fundamental building blocks of the code structure. !16 @ryzokuken
17. But how did we get here? Let’s talk history. !17 @ryzokuken
18. 1996 !18 @ryzokuken
19. JavaScript 1.0 released March 1996 !19 @ryzokuken
20. Era of the <script> tags begins. !20 @ryzokuken
21. JavaScript code too long? Try breaking up your code into multiple files. !21 @ryzokuken
22. Benefits: • Basic Modularity Drawbacks: • No Dependency Resolution • Namespace Pollution !22 @ryzokuken
23. Feb 2005 !23 @ryzokuken
24. PrototypeJS Feb 2005 !24 @ryzokuken
25. Aug 2006 !25 @ryzokuken
26. Oct 2009 !26 @ryzokuken
27. Apr 2012 !27 @ryzokuken
28. 999+ global variables Use IIFEs? 1 global variable !28 @ryzokuken
29. Benefits: • Basic Modularity • (Less) Namespace Pollution Drawbacks: • No Dependency Resolution • Namespace Pollution • Messy Syntax? !29 @ryzokuken
30. !30
31. “JavaScript on the server” 2009 !31 @ryzokuken
32. ServerJS ➡ CommonJS !32 @ryzokuken
33. Benefits: •Proper Modularity •Independent of HTML Drawbacks: •Synchronous require !33 @ryzokuken
34. Fact Node.js CommonJS Frontend Implementations: •Browserify •Webpack !34 @ryzokuken
35. CommonJS Frontend JavaScript Idea Let’s make a proposal! Module/Transfer/C !35 @ryzokuken
36. Asynchronous Module Definition circa 2010 !36 @ryzokuken
37. Oct 2010 !37 @ryzokuken
38. Oct 2010 !38 starts using AMD @ryzokuken
39. 2010 !39 James Burke, dissatisfied with how Dojo worked, created @ryzokuken
40. All problems are solved! Let’s go home. !40 @ryzokuken
41. Not quite. !41 @ryzokuken
42. Problems before AMD and CJS: •No dependency resolution •Namespace pollution !42 @ryzokuken
43. Problems after AMD: •Verbosity •Strictness pertaining to order •#justHTTP1.1Things !43 @ryzokuken
44. Nobody: JS Developers: We will just use CJS on the browser. !44 @ryzokuken
45. AMD Creators: *Create AMD* JavaScript Developers: !45 @ryzokuken
46. !46 @ryzokuken
47. Rise of Module Bundlers !47 Write CJS, get a single bundle file @ryzokuken
48. Now everyone uses CJS on both the server and the browser? !48 @ryzokuken
49. Congratulations! Order has been restored to the galaxy. !49 @ryzokuken
50. Module Authors: But how do I enable people to use my module without worrying about which format? !50 @ryzokuken
51. Universal Module Definition Late 2011 !51 @ryzokuken
52. !52 @ryzokuken
53. TC39 !53 @ryzokuken @ryzokuken
54. ECMAScript Modules July 2014 !54 @ryzokuken
55. Static Imports ESM ⚔ CommonJS ESM ⚔ Node.js Support for Dynamic Imports !55 @ryzokuken
56. Static Imports 1. Static Analysis 2. Tree Shaking Dynamic variant !56 @ryzokuken
57. Q: So everything was perfect, right? A: Not Quite. !57 @ryzokuken
58. ECOSYSTEM DISRUPTION !58 @ryzokuken
59. !59 @ryzokuken
60. Now let’s see how it all works behind the scenes !60 @ryzokuken @ryzokuken
61. Fun Fact The modules systems in Node.js are mostly written in JavaScript 61 @ryzokuken
62. VM Fundamentals !62 @ryzokuken @ryzokuken
63. Isolate A VM instance with its own heap. Chrome V8 Node.js V8 Chrome Node.js V8 Electron !63 @ryzokuken
64. Q: What’s the job of the Isolate? A: Heap allocation and garbage collection, mainly. !64 @ryzokuken
65. Aside: How does it all work? !65 @ryzokuken
66. C++ Stack Isolate Heap JS Land Object Object Object ??? Handle Scope Pointer? Handle Handle Handle Handle !66 @ryzokuken
67. Context A sandboxed execution context with its own set of built-in objects and functions. Isolate Context Context JavaScript Code !67 @ryzokuken
68. Process Thread Thread Thread Worker Worker Worker libuv V8 libuv V8 libuv V8 Event Loop Isolate Event Loop Isolate Event Loop Isolate Heap+GC Heap+GC Heap+GC Node.js Environment Node.js Environment Node.js Environment node stdlib node stdlib node stdlib Context Context Context Context Context Context Context Context Context JS builtins JS builtins JS builtins JS builtins JS builtins JS builtins JS builtins JS builtins JS builtins !68 @ryzokuken
69. How do Loaders work? •Fetch •Transform •Evaluation Hook !69 @ryzokuken
70. !70 Resolve Return Instantiate Cache Cached? @ryzokuken
71. CommonJS Loader !71 @ryzokuken
72. CommonJS has Synchronous Load
 + Inline Execution !72 @ryzokuken
73. !73 require Module.prototype.require @ryzokuken
74. !74 Module.prototype.require Module._load @ryzokuken
75. !75 1.filename := Module._resolveFilename(…) @ryzokuken
76. !76 1.filename := Module._resolveFilename(…) 2.cachedModule := Module._cache[filename] 3.if cachedModule: return cachedModule.exports @ryzokuken
77. !77 1.filename := Module._resolveFilename(…) 2.cachedModule := Module._cache[filename] 3.if cachedModule: return cachedModule.exports 4.[if nativeModule: return mod.compileForPublicLoader(...)] @ryzokuken
78. !78 1.filename := Module._resolveFilename(…) 2.cachedModule := Module._cache[filename] 3.if cachedModule: return cachedModule.exports 4.[if nativeModule: return mod.compileForPublicLoader(...)] 5.module := new Module(filename, …) @ryzokuken
79. !79 1.filename := Module._resolveFilename(…) 2.cachedModule := Module._cache[filename] 3.if cachedModule: return cachedModule.exports 4.[if nativeModule: return mod.compileForPublicLoader(...)] 5.module := new Module(filename, …) 6.Module._cache[filename] = module @ryzokuken
80. !80 1.filename := Module._resolveFilename(…) 2.cachedModule := Module._cache[filename] 3.if cachedModule: return cachedModule.exports 4.[if nativeModule: return mod.compileForPublicLoader(...)] 5.module := new Module(filename, …) 6.Module._cache[filename] = module 7.module.load(filename) 8.if throw, delete from cache 9.otherwise return module.exports @ryzokuken
81. !81 1.filename := Module._resolveFilename(…) 2.cachedModule := Module._cache[filename] 3.if cachedModule: return cachedModule.exports 4.[if nativeModule: return mod.compileForPublicLoader(...)] 5.module := new Module(filename, …) 6.Module._cache[filename] = module 7.module.load(filename) 8.if throw, delete from cache 9.otherwise return module.exports @ryzokuken
82. !82 Module.load 1.Find appropriate extension. 2. Take appropriate action. @ryzokuken
83. !83 And what exactly does Module._compile do anyway? @ryzokuken
84. !84 ❌ @ryzokuken
85. !85 ✅ vm.compileFunction ✅ @ryzokuken
86. ES Modules Loader !86 @ryzokuken
87. ES Modules have Asynchronous Load + Synchronous Execution !87 @ryzokuken
88. !88 @ryzokuken
89. !89 And guess what? Loader.getModuleJob is pretty much exactly inline with CJS @ryzokuken
90. !90 1.this.resolve(…args) 2.job := this.moduleMap.get(url) 3.If job: return job 4.job = new ModuleJob(…) 5.moduleMap.set(url, job) 6.return job @ryzokuken
91. !91 And what’s inside ModuleJob? @ryzokuken
92. CommonJS is easy ESM is easy Interoperability is HARD !92
93. Problems •How to detect if a module is ESM or CommonJS? • How to import CommonJS modules from ESM? • How to import ESM modules from CommonJS? !93 @ryzokuken
94. Answers • .mjs (sorry) • Dynamic Imports • ??? !94 @ryzokuken
95. !95 @ryzokuken
96. Special Thanks Myles Borins (@MylesBorins) Anna (@addaleax) Shelley Vohr (@codebytere) Dmitry Makhnev (@DmitryMakhnev) The Organizers !96 @ryzokuken
97. Хвала вам! !97