Hands-on Native ESM @ JSConf JP 2022

See Original version on Google Slides for copy-pasting and links.

See Materials

Transcript

1. © 2022 Wantedly, Inc. Hands-on Native ESM Understanding Node.js module discrepancy Nov. 26 2022 - Masaki Hara https://github.com/qnighy https://www.wantedly.com/id/qnighy
2. The aim of this workshop • Understand how CJS and ESM (does not smoothly) interoperate in Node.js • Understand how it becomes more difficult when module budler is involved © 2022 Wantedly, Inc. It does not cover the entire solution, but it will nonetheless help you migrating to Native ESM.
3. Related materials • 実践 Node.js Native ESM — Wantedlyでのアプリケー ション移行事例[1] • Node.jsのネイティブES Modulesサポートが抱える問題を 解決するBabelプラグインを書いた[2] • Native ESM + TypeScript 拡張子問題: 歯にものが挟まっ たようなスッキリしない書き流し[3] © 2022 Wantedly, Inc. [1] https://www.wantedly.com/companies/wantedly/post_articles/410531 [2] https://zenn.dev/qnighy/articles/6267716578c76d [3] https://zenn.dev/qnighy/articles/19603f11d5f264
4. Advertisement We are looking for JS ecosystem enthusiasts! © 2022 Wantedly, Inc. WebpackやBabelの設定を通じて開発を加速させたい人募集 ! - Wantedly[1] [1] https://www.wantedly.com/projects/1145492
5. Advertisement Our frontend team also welcomes you! © 2022 Wantedly, Inc. 今見ているWantedlyのWebフロントエンドをもっと良くしませんか？ - Wantedly[1] [1] https://www.wantedly.com/projects/1159802
6. Hands-on guide • Materials can be found at: https://github.com/wantedly/hands-on-native- esm-2022 • Requires Node.js ≧ 16 • I will demonstrate chapters 1 to 4. • Read and exercise the remaining chapters (5 to 11) if you are interested. © 2022 Wantedly, Inc.
7. 1: Let’s try ESM! © 2022 Wantedly, Inc.
8. Try Native ESM • Node.js has two module types: ◦ ESM = ES Modules (*.mjs, *.js) ◦ CJS = CommonJS Modules (*.cjs, *.js) • They interoperate badly 😵 • You usually write JS in ESM but execute it as CJS. © 2022 Wantedly, Inc. (as opposed to Native ESM)
9. Try Native ESM Try ESM import/export © 2022 Wantedly, Inc. export default function(x) { return x * x; } import square from "./lib.mjs"; console.log(square(2)); lib.mjs app.mjs
10. Try Native ESM Try ESM import/export © 2022 Wantedly, Inc. export default function(x) { return x * x; } import square from "./lib.mjs"; console.log(square(2)); lib.mjs app.mjs $ node app.mjs 4
11. Try Native ESM Open https://babeljs.io/repl © 2022 Wantedly, Inc.
12. Try Native ESM Try CJS transpilation © 2022 Wantedly, Inc. "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); exports.default = _default; function _default(x) { return x * x; } "use strict"; var _a = _interopRequireDefault(require("./a.cjs")); function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; } console.log((0, _a.default)(2)); lib.cjs (before Babel) app.cjs (before Babel) export default function(x) { return x * x; } import square from "./lib.cjs"; console.log(square(2));
13. Try Native ESM Try CJS transpilation © 2022 Wantedly, Inc. "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); exports.default = _default; function _default(x) { return x * x; } "use strict"; var _a = _interopRequireDefault(require("./lib.cjs")); function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; } console.log((0, _a.default)(2)); lib.cjs app.cjs
14. Try Native ESM Try CJS transpilation © 2022 Wantedly, Inc. "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); exports.default = _default; function _default(x) { return x * x; } "use strict"; var _a = _interopRequireDefault(require("./lib.cjs")); function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; } console.log((0, _a.default)(2)); lib.cjs app.cjs $ node app.cjs 4
15. Try Native ESM CJS → ESM © 2022 Wantedly, Inc. "use strict"; var _a = _interopRequireDefault(require("./lib.mjs")); function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; } console.log((0, _a.default)(2)); app.cjs export default function(x) { return x * x; } lib.mjs
16. Try Native ESM CJS → ESM © 2022 Wantedly, Inc. "use strict"; var _a = _interopRequireDefault(require("./lib.mjs")); function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; } console.log((0, _a.default)(2)); app.cjs export default function(x) { return x * x; } lib.mjs $ node app.cjs node:internal/modules/cjs/loader:1031 throw new ERR_REQUIRE_ESM(filename, true); ^ Error [ERR_REQUIRE_ESM]: require() of ES Module lib.mjs not supported. Instead change the require of lib.mjs to a dynamic import() which is available in all CommonJS modules.
17. Try Native ESM ESM → CJS © 2022 Wantedly, Inc. "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); exports.default = _default; function _default(x) { return x * x; } lib.cjs import square from "./lib.cjs"; console.log(square(2)); app.mjs
18. Try Native ESM ESM → CJS © 2022 Wantedly, Inc. "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); exports.default = _default; function _default(x) { return x * x; } lib.cjs import square from "./lib.cjs"; console.log(square(2)); app.mjs $ node app.mjs app.mjs:3 console.log(square(2)); ^ TypeError: square is not a function at app.mjs:3:13
19. There is no easy path • Library-first upgrade: essential difficulty ◦ Synchronous vs. Asynchronous • Application-first upgrade: avoidable pitfalls ◦ Default import problem • Dual package: difficulty in some packages ◦ State sharing problem © 2022 Wantedly, Inc.
20. But we must do it • Some libraries are already in Native ESM format. ◦ node-fetch ◦ d3 ◦ etc. • Depending applications are (almost) forced to do the migration. © 2022 Wantedly, Inc.
21. 2: Module detection © 2022 Wantedly, Inc.
22. How Node.js determines module types • Node.js determines[1] module types using: ◦ File extensions (*.cjs / *.mjs) ◦ Package metadata (“type”: “module”) if it is *.js © 2022 Wantedly, Inc. *.cjs *.js *.mjs “type”: “commonjs” CJS CJS ESM no “type” field CJS CJS ESM “type”: “module” CJS ESM ESM Conversely, they are not taken into account: • File contents • How the import had been routed (e.g. exports map) [1] https://nodejs.org/api/packages.html#packagejson-and-file-extensions
23. How Node.js determines module types Check module types © 2022 Wantedly, Inc. if (typeof await /0/["test"] === "function") { console.log("I'm ESM"); } else if (typeof require === "function") { console.log("I'm CJS"); } else { console.log("I'm other"); } module-type.js
24. Check module types © 2022 Wantedly, Inc. if (typeof await /0/["test"] === "function") { console.log("I'm ESM"); } else if (typeof require === "function") { console.log("I'm CJS"); } else { console.log("I'm other"); } module-type.js $ node module-type.js I’m CJS $ node module-type.cjs I’m CJS $ node module-type.mjs I’m ESM How Node.js determines module types
25. Module type depends on package.json © 2022 Wantedly, Inc. { "type": "module" } package.json How Node.js determines module types
26. Module type depends on package.json © 2022 Wantedly, Inc. { "type": "module" } package.json $ node module-type.js I’m ESM $ node module-type.cjs I’m CJS $ node module-type.mjs I’m ESM How Node.js determines module types
27. 3: Semantics of execution © 2022 Wantedly, Inc. Module asynchrony
28. Semantics of execution • Modules are executed ◦ synchronously in CJS ◦ asynchronously in ESM • Consequence👉 CJS cannot import ESM[1] © 2022 Wantedly, Inc. [1]: Precisely speaking, it can wait for the imported module to finish.
29. Do a time-consuming initialization © 2022 Wantedly, Inc. import { setTimeout } from "node:timers/promises"; await setTimeout(3000); // For Node.js < 16.0.0 // await new Proise((r) => setTimeout(r, 3000)); console.log("slept 1s"); sleep.mjs Explicit asynchrony: top-level await
30. Dependent module may also be asynchronous © 2022 Wantedly, Inc. import { setTimeout } from "node:timers/promises"; await setTimeout(3000); export const value = 42; sleep-lib.mjs Implied asynchrony
31. Dependent module may also be asynchronous © 2022 Wantedly, Inc. import { value } from "./sleep-lib.mjs"; console.log("value =", value); sleep-app.mjs Implied asynchrony
32. Dependent module may also be asynchronous © 2022 Wantedly, Inc. const { value } = await import("./sleep-lib.mjs"); console.log("value =", value); sleep-app.mjs (alternative) Expanding implied asynchrony
33. Require — this is a mere function © 2022 Wantedly, Inc. const { value } = require("./sleep-lib.mjs"); console.log("value =", value); sleep-app.cjs Discrepancy in asynchrony Error!
34. Require — this is a mere function © 2022 Wantedly, Inc. const { value } = require("./sleep-lib.mjs"); console.log("value =", value); sleep-app.cjs Discrepancy in asynchrony Error! $ node sleep-app.cjs node:internal/modules/cjs/loader:1031 throw new ERR_REQUIRE_ESM(filename, true); ^ Error [ERR_REQUIRE_ESM]: require() of ES Module sleep-lib.mjs not supported.
35. TLA — this is not possible © 2022 Wantedly, Inc. const { value } = await import("./sleep-lib.mjs"); console.log("value =", value); sleep-app.cjs Discrepancy in asynchrony Error!
36. TLA — this is not possible © 2022 Wantedly, Inc. const { value } = await import("./sleep-lib.mjs"); console.log("value =", value); sleep-app.cjs Discrepancy in asynchrony Error! $ node sleep-app.cjs sleep-app.cjs:1 const { value } = await import("./sleep-lib.mjs"); ^^^^^ SyntaxError: await is only valid in async functions and the top level bodies of modules
37. Import-then © 2022 Wantedly, Inc. import("./sleep-lib.mjs").then(({ value }) => { console.log("value =", value); }); sleep-app.cjs Discrepancy in asynchrony
38. Import-then: it’s too late for exporting © 2022 Wantedly, Inc. import("./sleep-lib.mjs").then(({ value }) => { console.log("value =", value); exports.value2 = value * 2; }); sleep-app.cjs Discrepancy in asynchrony Too late!
39. 4: Default export semantics © 2022 Wantedly, Inc.
40. Default export semantics • Default export is ◦ fancy namespace export in CJS ◦ special named export in ESM • Consequence👉 incompatible translations • My library node-cjs-interop[1] helps here. © 2022 Wantedly, Inc. [1] https://github.com/qnighy/node-cjs-interop
41. Default export semantics © 2022 Wantedly, Inc. Default Named export default f; export { f }; module.exports = f; exports.f = f;
42. Default export semantics from importer’s viewpoint © 2022 Wantedly, Inc. import * as ns from ""; ✅ Namespace ❌ Default ✅ Named import f from ""; import { f } from ""; const ns = require(""); const f = require(""); const { f } = require(""); = ∈
43. Default export semantics from importer’s viewpoint © 2022 Wantedly, Inc. import * as ns from ""; ✅ Namespace ❌ Default ✅ Named import { default as f } from ""; import { f } from ""; const ns = require(""); const f = require(""); const { f } = require(""); = ∈
44. Default export interop 1: Node.js rule (CJS-preserving) © 2022 Wantedly, Inc. import * as ns from ""; import f from ""; import { f } from ""; const ns = require(""); const f = require(""); const { f } = require(""); = ∈ (except “default”) (except “default”)
45. Default export interop 2: simple mapping (ESM-preserving) © 2022 Wantedly, Inc. import * as ns from ""; import f from ""; import { f } from ""; const ns = require(""); const f = require(""); const { f } = require(""); = ∈ (Loss of information)
46. Default export interop 3: Babel mapping © 2022 Wantedly, Inc. var _react = _interopRequireDefault(require("react")); function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; } Simple mapping (ESM-preserving)
47. Default export interop 3: Babel mapping © 2022 Wantedly, Inc. var _react = _interopRequireDefault(require("react")); function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; } Node.js rule (CJS-preserving)
48. Default export interop – tool support © 2022 Wantedly, Inc. Node.js rule Babel rule (hybrid) Simple mapping Node.js ✅ ❌ but see: node-cjs-interop[1] ❌ Webpack javascript/esm javascript/auto ❌ Babel interop = node interop = babel interop = none TypeScript ❌ esModuleInterop = true esModuleInterop = false [1] https://github.com/qnighy/node-cjs-interop
49. 4.1: Default export semantics (1) © 2022 Wantedly, Inc. Default export in ESM
50. Default export in ESM Default export and default import in ESM © 2022 Wantedly, Inc. export default function(x) { return x * x; } import square from "./lib.mjs"; console.log(square(2)); lib.mjs app.mjs
51. Default export in ESM Default export and default import in ESM © 2022 Wantedly, Inc. export default function(x) { return x * x; } import square from "./lib.mjs"; console.log(square(2)); lib.mjs app.mjs $ node app.mjs 4
52. Default export in ESM Default import is a syntax sugar © 2022 Wantedly, Inc. export default function(x) { return x * x; } import { default as square } from "./lib.mjs"; console.log(square(2)); lib.mjs app.mjs
53. Default export in ESM Default import is a syntax sugar © 2022 Wantedly, Inc. export default function(x) { return x * x; } import { default as square } from "./lib.mjs"; console.log(square(2)); lib.mjs app.mjs $ node app.mjs 4
54. Default export in ESM Default export is a syntax sugar © 2022 Wantedly, Inc. function square(x) { return x * x; } export { square as default }; import square from "./lib.mjs"; console.log(square(2)); lib.mjs app.mjs
55. Default export in ESM Default export is a syntax sugar © 2022 Wantedly, Inc. function square(x) { return x * x; } export { square as default }; import square from "./lib.mjs"; console.log(square(2)); lib.mjs app.mjs $ node app.mjs 4
56. 4.2: Default export semantics (2) © 2022 Wantedly, Inc. Default export in CJS
57. Default export in CJS Default export and default import in CJS © 2022 Wantedly, Inc. module.exports = 42; const x = require("./lib.cjs"); console.log(x); lib.cjs app.cjs
58. Default export in CJS Default export and default import in CJS © 2022 Wantedly, Inc. module.exports = 42; const x = require("./lib.cjs"); console.log(x); lib.cjs app.cjs $ node app.mjs 42
59. Node.js rule for default export Default export interop © 2022 Wantedly, Inc. module.exports = 42; import x from "./app.cjs"; console.log(x); lib.cjs app.mjs
60. Node.js rule for default export Default export interop © 2022 Wantedly, Inc. module.exports = 42; import x from "./app.cjs"; console.log(x); lib.cjs app.mjs $ node app.mjs 42
61. 4.3: Default export semantics (3) © 2022 Wantedly, Inc. Named exports in CJS
62. Named exports in CJS Named exports in CJS © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; const m = require("./lib.cjs"); console.log(m); lib.cjs app.cjs
63. Named exports in CJS Named exports in CJS © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; const m = require("./lib.cjs"); console.log(m); lib.cjs app.cjs $ node app.cjs { foo: 1, bar: 2 }
64. Node.js rule for named exports Named exports interop © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2); lib.cjs app.mjs
65. Node.js rule for named exports Named exports interop © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2); lib.cjs app.mjs $ node app.mjs { foo: 1, bar: 2 } [Module: null prototype] { bar: 2, default: { foo: 1, bar: 2 }, foo: 1 }
66. Node.js rule for named “default” exports Named “default” exports interop © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; exports.default = 3; import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2); lib.cjs app.mjs
67. Node.js rule for named “default” exports Named “default” exports interop © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; exports.default = 3; import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2); lib.cjs app.mjs $ node app.mjs { foo: 1, bar: 2, default: 3 } [Module: null prototype] { bar: 2, default: { foo: 1, bar: 2, default: 3 }, foo: 1 }
68. 4.4: Default export semantics (4) © 2022 Wantedly, Inc. Babel-style interop
69. Mixing CJS named and default exports Named + default exports in CJS © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; module.exports = 3; lib.cjs import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2); app.mjs
70. Mixing CJS named and default exports Named + default exports in CJS © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; module.exports = 3; lib.cjs import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2); app.mjs $ node app.cjs 3 [Module: null prototype] { bar: undefined, default: 3, foo: undefined }
71. Babel-style interop Special flag for simple mapping © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; exports.default = 3; exports.__esModule = true; lib.cjs app.mjs import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2);
72. Babel-style interop Special flag for simple mapping © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; exports.default = 3; exports.__esModule = true; var m2 = _interopRequireWildcard( require("./lib.cjs")); console.log(m2.default); console.log(m2); lib.cjs app.cjs (transpiled from app.mjs)
73. Babel-style interop Special flag for simple mapping © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; exports.default = 3; exports.__esModule = true; "use strict"; var m2 = _interopRequireWildcard(require("./lib.cjs")); function _getRequireWildcardCache(nodeInterop) { if (typeof WeakMap !== "function") return null; var cacheBabelInterop = new WeakMap(); var cacheNodeInterop = new WeakMap(); return (_getRequireWildcardCache = function (nodeInterop) { return nodeInterop ? cacheNodeInterop : cacheBabelInterop; })(nodeInterop); } function _interopRequireWildcard(obj, nodeInterop) { if (!nodeInterop && obj && obj.__esModule) { return obj; } if (obj === null || typeof obj !== "object" && typeof obj !== "function") { return { default: obj }; } var cache = _getRequireWildcardCache(nodeInterop); if (cache && cache.has(obj)) { return cache.get(obj); } var newObj = {}; var hasPropertyDescriptor = Object.defineProperty && Object.getOwnPropertyDescriptor; for (var key in obj) { if (key !== "default" && Object.prototype.hasOwnProperty.call(obj, key)) { var desc = hasPropertyDescriptor ? Object.getOwnPropertyDescriptor(obj, key) : null; if (desc && (desc.get || desc.set)) { Object.defineProperty(newObj, key, desc); } else { newObj[key] = obj[key]; } } } newObj.default = obj; if (cache) { cache.set(obj, newObj); } return newObj; } console.log(m2.default); console.log(m2); lib.cjs app.cjs (transpiled from app.mjs)
74. Babel-style interop Special flag for simple mapping © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; exports.default = 3; exports.__esModule = true; "use strict"; var m2 = _interopRequireWildcard(require("./lib.cjs")); function _getRequireWildcardCache(nodeInterop) { if (typeof WeakMap !== "function") return null; var cacheBabelInterop = new WeakMap(); var cacheNodeInterop = new WeakMap(); return (_getRequireWildcardCache = function (nodeInterop) { return nodeInterop ? cacheNodeInterop : cacheBabelInterop; })(nodeInterop); } function _interopRequireWildcard(obj, nodeInterop) { if (!nodeInterop && obj && obj.__esModule) { return obj; } if (obj === null || typeof obj !== "object" && typeof obj !== "function") { return { default: obj }; } var cache = _getRequireWildcardCache(nodeInterop); if (cache && cache.has(obj)) { return cache.get(obj); } var newObj = {}; var hasPropertyDescriptor = Object.defineProperty && Object.getOwnPropertyDescriptor; for (var key in obj) { if (key !== "default" && Object.prototype.hasOwnProperty.call(obj, key)) { var desc = hasPropertyDescriptor ? Object.getOwnPropertyDescriptor(obj, key) : null; if (desc && (desc.get || desc.set)) { Object.defineProperty(newObj, key, desc); } else { newObj[key] = obj[key]; } } } newObj.default = obj; if (cache) { cache.set(obj, newObj); } return newObj; } console.log(m2.default); console.log(m2); lib.cjs app.cjs (transpiled from app.mjs) $ node app.cjs 3 { foo: 1, bar: 2, default: 3, __esModule: true }
75. Babel-style interop Node.js does not support __esModule © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; exports.default = 3; exports.__esModule = true; lib.cjs import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2); app.mjs
76. Babel-style interop Node.js does not support __esModule © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; exports.default = 3; exports.__esModule = true; lib.cjs import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2); app.mjs $ node app.mjs { foo: 1, bar: 2, default: 3 } [Module: null prototype] { __esModule: true, bar: 2, default: { foo: 1, bar: 2, default: 3 }, foo: 1 }
77. End of demonstration Enjoy the remaining exercises if you are interested in. © 2022 Wantedly, Inc.
78. 5: Export analysis © 2022 Wantedly, Inc.
79. Export analysis • Static analysis is mandatory in Native ESM, and it also applies to CJS imported from ESM. • It sometimes fails to detect exports in CJS. © 2022 Wantedly, Inc.
80. Export analysis Node.js analyzes ESM exports before execution © 2022 Wantedly, Inc. console.log("Initializing..."); export const foo = 1; lib.mjs import { bar } from "./lib.mjs"; console.log(bar); app.mjs
81. Export analysis Node.js analyzes ESM exports before execution © 2022 Wantedly, Inc. console.log("Initializing..."); export const foo = 1; lib.mjs import { bar } from "./lib.mjs"; console.log(bar); app.mjs $ node app.mjs app.mjs:1 import { bar } from "./lib.mjs"; ^^^ SyntaxError: The requested module './lib.mjs' does not provide an export named 'bar'
82. Export analysis Node.js analyzes ESM exports before execution © 2022 Wantedly, Inc. console.log("Initializing..."); export const foo = 1; lib.mjs import { bar } from "./lib.mjs"; console.log(bar); app.mjs $ node app.mjs app.mjs:1 import { bar } from "./lib.mjs"; ^^^ SyntaxError: The requested module './lib.mjs' does not provide an export named 'bar' No “Initializing…” printed
83. Export analysis It analyzes CJS too (when imported from ESM) © 2022 Wantedly, Inc. console.log("Initializing..."); exports.foo = 1; lib.cjs import { bar } from "./lib.cjs"; console.log(bar); app.mjs
84. Export analysis It analyzes CJS too (when imported from ESM) © 2022 Wantedly, Inc. console.log("Initializing..."); exports.foo = 1; lib.cjs import { bar } from "./lib.cjs"; console.log(bar); app.mjs $ node app.mjs app.mjs:1 import { bar } from "./lib.cjs"; ^^^ SyntaxError: Named export 'bar' not found. The requested module './lib.cjs' is a CommonJS module, which may not support all module.exports as named exports.
85. Export analysis It analyzes CJS too (when imported from ESM) © 2022 Wantedly, Inc. console.log("Initializing..."); exports.bar = 2; lib.cjs import { bar } from "./app.cjs"; console.log(bar); app.mjs
86. Export analysis It analyzes CJS too (when imported from ESM) © 2022 Wantedly, Inc. console.log("Initializing..."); exports.bar = 2; lib.cjs import { bar } from "./app.cjs"; console.log(bar); app.mjs $ node app.mjs Initializing... 2
87. Export analysis It analyzes CJS too (when imported from ESM) © 2022 Wantedly, Inc. console.log("Initializing..."); exports["bar"] = 2; lib.cjs import { bar } from "./lib.cjs"; console.log(bar); app.mjs
88. Export analysis It analyzes CJS too (when imported from ESM) © 2022 Wantedly, Inc. console.log("Initializing..."); exports["bar"] = 2; lib.cjs import { bar } from "./lib.cjs"; console.log(bar); app.mjs $ node app.mjs Initializing... 2
89. Export analysis Not all exports can be analyzed © 2022 Wantedly, Inc. console.log("Initializing..."); exports["ba" + "r"] = 2; lib.cjs import { bar } from "./lib.cjs"; console.log(bar); app.mjs
90. Export analysis Not all exports can be analyzed © 2022 Wantedly, Inc. console.log("Initializing..."); exports["ba" + "r"] = 2; lib.cjs import { bar } from "./lib.cjs"; console.log(bar); app.mjs $ node app.mjs app.mjs:1 import { bar } from "./lib.cjs"; ^^^ SyntaxError: Named export 'bar' not found. The requested module './lib.cjs' is a CommonJS module, which may not support all module.exports as named exports.
91. Export analysis rules • Rules can be found at cjs-module-lexer[1] © 2022 Wantedly, Inc. [1] https://github.com/nodejs/cjs-module-lexer exports.foo = 1; module.exports.foo = 1; exports["foo"] = 1; module.exports = { foo }; module.exports = { foo: bar }; Object.defineProperty(exports, "foo", ); { value: 1 } { enumerable: true, value: 1 } { get() { return bar; } } { get: function() { return bar; } } { get() { return m.bar; } } { get() { return m["bar"]; } }
92. Export analysis rules • Rules can be found at cjs-module-lexer[1] © 2022 Wantedly, Inc. [1] https://github.com/nodejs/cjs-module-lexer module.exports = require("mod"); module.exports = { ...require("mod") }; __export(require("mod")); __exportStar(require("mod"), exports); var m = require("mod"); const m = require("mod"); let m = require("mod"); var m = _interopRequireWildcard (require("mod")); Object.keys(m).forEach(...); +
93. 6: Live exports © 2022 Wantedly, Inc.
94. Live binding of exports • In ESM, imported variables reference the latest value of the original variable. • This is not the case with CJS interop in Node.js. © 2022 Wantedly, Inc.
95. Live binding of exports counter evaluates to the latest value © 2022 Wantedly, Inc. export let counter = 0; export function countUp() { counter++; } lib.mjs import { counter, countUp } from "./lib.cjs"; console.log(counter); countUp(); console.log(counter); app.mjs
96. Live binding of exports counter evaluates to the latest value © 2022 Wantedly, Inc. export let counter = 0; export function countUp() { counter++; } lib.mjs import { counter, countUp } from "./lib.cjs"; console.log(counter); countUp(); console.log(counter); app.mjs $ node app.mjs 0 1
97. Live binding of exports counter evaluates to the value when CJS is imported © 2022 Wantedly, Inc. exports.counter = 0; exports.countUp = function() { exports.counter++; } lib.cjs import { counter, countUp } from "./lib.cjs"; console.log(counter); countUp(); console.log(counter); app.mjs
98. Live binding of exports counter evaluates to the value when CJS is imported © 2022 Wantedly, Inc. exports.counter = 0; exports.countUp = function() { exports.counter++; } lib.cjs import { counter, countUp } from "./lib.cjs"; console.log(counter); countUp(); console.log(counter); app.mjs $ node app.mjs 0 0
99. 7: Module path resolution © 2022 Wantedly, Inc.
100. Module path resolution • import() no longer automatically adds extensions. • It no longer automatically resolves directories as “index.js”. • It no longer resolves “folder main”. • *.cjs is not added for either CJS or ESM. © 2022 Wantedly, Inc.
101. Module path resolution “.js” is added in CJS © 2022 Wantedly, Inc. console.log("imported"); lib.js require("./lib"); app.cjs
102. Module path resolution “.js” is added in CJS © 2022 Wantedly, Inc. console.log("imported"); lib.js require("./lib"); app.cjs $ node app.cjs imported
103. Module path resolution “.js” is no longer added in ESM © 2022 Wantedly, Inc. console.log("imported"); lib.js import "./lib"; app.mjs
104. Module path resolution “.js” is no longer added in ESM © 2022 Wantedly, Inc. console.log("imported"); lib.js import "./lib"; app.mjs $ node app.mjs node:internal/process/esm_loader:97 internalBinding('errors').triggerUncaughtE xception( ^ Error [ERR_MODULE_NOT_FOUND]: Cannot find module 'lib' imported from app.mjs Did you mean to import ../lib.js?
105. Module path resolution ./index.js is added in CJS © 2022 Wantedly, Inc. console.log("imported"); lib/index.js require("./lib"); app.cjs
106. Module path resolution ./index.js is added in CJS © 2022 Wantedly, Inc. console.log("imported"); lib/index.js require("./lib"); app.cjs $ node app.cjs imported
107. Module path resolution ./index.js is no longer added in ESM © 2022 Wantedly, Inc. console.log("imported"); lib/index.js import "./lib"; app.mjs
108. Module path resolution ./index.js is no longer added in ESM © 2022 Wantedly, Inc. console.log("imported"); lib/index.js import "./lib"; app.mjs $ node app.mjs node:internal/process/esm_loader:97 internalBinding('errors').triggerUncaughtExcep tion( ^ Error [ERR_UNSUPPORTED_DIR_IMPORT]: Directory import 'lib' is not supported resolving ES modules imported from app.mjs Did you mean to import ../lib/index.js?
109. Module path resolution “Folder main” in CJS © 2022 Wantedly, Inc. console.log("imported"); lib/main.js require("./lib"); app.cjs { "main": "./main.js" } lib/package.json
110. Module path resolution “Folder main” in CJS © 2022 Wantedly, Inc. console.log("imported"); lib/main.js require("./lib"); app.cjs { "main": "./main.js" } lib/package.json $ node app.cjs imported
111. Module path resolution “Folder main” is no longer available © 2022 Wantedly, Inc. console.log("imported"); lib/main.js import "./lib"; app.mjs { "main": "./main.js" } lib/package.json
112. Module path resolution “Folder main” is no longer available © 2022 Wantedly, Inc. console.log("imported"); lib/main.js import "./lib"; app.mjs { "main": "./main.js" } lib/package.json $ node app.mjs node:internal/process/esm_loader:97 internalBinding('errors').triggerUncaughtExcep tion( ^ Error [ERR_UNSUPPORTED_DIR_IMPORT]: Directory import 'lib' is not supported resolving ES modules imported from app.mjs Did you mean to import ../lib/main.js?
113. Module path resolution “Package main” is still valid © 2022 Wantedly, Inc. console.log("imported"); node_modules/lib/main.js import "lib"; app.mjs { "main": "./main.js" } node_modules/lib/package.json
114. Module path resolution “Package main” is still valid © 2022 Wantedly, Inc. console.log("imported"); node_modules/lib/main.js import "lib"; app.mjs { "main": "./main.js" } node_modules/lib/package.json $ node app.mjs imported
115. Module path resolution “.cjs” is not added even from CJS © 2022 Wantedly, Inc. console.log("imported"); lib.cjs require("./lib"); app.cjs
116. Module path resolution “.cjs” is not added even from CJS © 2022 Wantedly, Inc. console.log("imported"); lib.cjs require("./lib"); app.cjs $ node app.mjs node:internal/modules/cjs/loader:988 throw err; ^ Error: Cannot find module './lib
117. 8: Module metadata © 2022 Wantedly, Inc.
118. Module metadata • These CJS variables are usually available in to-be-transpiled ESM code: ◦ require ◦ exports ◦ module ◦ __filename ◦ __dirname • This is no longer the case in Native ESM. © 2022 Wantedly, Inc.
119. Module metadata CJS metavariables are not available © 2022 Wantedly, Inc. console.log(`I'm ${__filename}`); export {}; meta.mjs
120. Module metadata CJS metavariables are not available © 2022 Wantedly, Inc. console.log(`I'm ${__filename}`); export {}; meta.mjs $ node meta.mjs file:///<dir>/meta.mjs:1 console.log(`I'm ${__filename}`); ^ ReferenceError: __filename is not defined in ES module scope
121. Module metadata They are usually transpiled as-is © 2022 Wantedly, Inc. "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); console.log(`I'm ${__filename}`); meta.cjs (transpiled from meta.mjs)
122. Module metadata They are usually transpiled as-is © 2022 Wantedly, Inc. "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); console.log(`I'm ${__filename}`); meta.cjs (transpiled from meta.mjs) $ node meta.cjs I’m <dir>/meta.cjs
123. Module metadata Use import.meta instead in Native ESM © 2022 Wantedly, Inc. console.log(`I'm ${import.meta.url}`); export {}; meta.mjs
124. Module metadata Use import.meta instead in Native ESM © 2022 Wantedly, Inc. console.log(`I'm ${import.meta.url}`); export {}; meta.mjs $ node meta.mjs I’m file://<dir>/meta.mjs
125. 9: Dual package © 2022 Wantedly, Inc.
126. Dual package • Export maps allow packages to provide different module types at once. • There is a pitfall when the package is stateful (or impure in some sense). © 2022 Wantedly, Inc.
127. Dual package Native ESM dual package © 2022 Wantedly, Inc. { "name": "esm-test", "exports": { "import": "./lib.mjs", "require": "./lib.cjs", } } package.json console.log("ESM entrypoint"); lib.mjs console.log("CJS entrypoint"); lib.cjs
128. Dual package Using import and require © 2022 Wantedly, Inc. import "esm-test"; app.mjs require("esm-test"); app.cjs package.json lib.mjs lib.cjs
129. Dual package Using import and require © 2022 Wantedly, Inc. import "esm-test"; app.mjs require("esm-test"); app.cjs package.json lib.mjs lib.cjs $ node app.mjs ESM entrypoint $ node app.cjs CJS entrypoint
130. Dual package Double side effects © 2022 Wantedly, Inc. import "esm-test"; import "./app.cjs"; app.mjs require("esm-test"); app.cjs package.json lib.mjs lib.cjs
131. Dual package Double side effects © 2022 Wantedly, Inc. import "esm-test"; import "./app.cjs"; app.mjs require("esm-test"); app.cjs package.json lib.mjs lib.cjs $ node app.mjs ESM entrypoint CJS entrypoint Imagine this is state initialization. There are two different states now!
132. 10: Modules in Webpack © 2022 Wantedly, Inc.
133. Modules in bundlers • ESM story in Node.js gets more complicated when module bundlers are involved. • In this workshop, we use Webpack (version 5) as an example because it is still the most influential bundler. © 2022 Wantedly, Inc.
134. Modules in Webpack • Webpack generally adheres to Node.js behavior. • One big difference: Webpack has two different ESM modes. • It complicates the matter in default export interop. © 2022 Wantedly, Inc.
135. Autodetected ESM (Fake ESM) • Webpack has “auto detection” module mode. • It semantically behaves like CJS, even if the module is detected to be ESM. © 2022 Wantedly, Inc. *.cjs *.js *.mjs “type”: “commonjs” CJS CJS ESM no “type” field CJS Auto ESM “type”: “module” CJS ESM ESM In Webpack terminology: • CJS = javascript/dynamic • Auto = javascript/auto • ESM = javascript/esm
136. Autodetected ESM (Fake ESM) • Webpack has “auto detection” module mode. • It semantically behaves like CJS, even if the module is detected to be ESM. © 2022 Wantedly, Inc. *.cjs *.js *.mjs “type”: “commonjs” CJS CJS ESM no “type” field CJS Auto ESM “type”: “module” CJS ESM ESM *.cjs *.js *.mjs “type”: “commonjs” CJS CJS ESM no “type” field CJS CJS ESM “type”: “module” CJS ESM ESM Recap: Node.js behavior
137. Resolution for autodetected ESM • Special lookup rule to detect Webpack © 2022 Wantedly, Inc. { "module": "esm/main.js", "main": "main.js", "exports": { "module": "./esm/main.js", "default": "./main.js" } } package.json This is usually CJS This is usually an ESM module to be auto-detected by Webpack
138. Resolution for autodetected ESM • Two types of dual package © 2022 Wantedly, Inc. { "exports": { "module": "./esm/main.js", "default": "./main.js" } } package.json { "exports": { "import": "./esm/main.mjs", "require": "./main.cjs" } } package.json Webpack uses ESM. Node.js always uses CJS. Webpack uses ESM. Node.js (import) uses ESM. Node.js (require) uses CJS.
139. Default export semantics in Node.js © 2022 Wantedly, Inc. CJS from Babel (*.js, *.cjs) Native ESM (*.js, *.mjs) Breaks default export
140. Default export semantics in Webpack © 2022 Wantedly, Inc. Autodetected ESM (*.js) CJS from Babel (*.js, *.cjs) Native ESM (*.js, *.mjs) Preserves default export Preserves default export Breaks default export
141. Set up Webpack Install Webpack © 2022 Wantedly, Inc. {} package.json $ yarn add --dev webpack webpack-cli Or: npm install --dev webpack webpack-cli
142. Set up Webpack Install Webpack © 2022 Wantedly, Inc. { "devDependencies": { "webpack": "^5.75.0", "webpack-cli": "^5.0.0" } } package.json (result)
143. Set up Webpack Check import result © 2022 Wantedly, Inc. exports.default = 1; exports.__esModule = true; lib1.js import a from "./lib1.js"; console.log("a =", a); export default 1; lib2.js import b from "./lib1.js"; import c from "./lib2.js"; console.log("b =", b); console.log("c =", c); app.mjs
144. Set up Webpack Check import result © 2022 Wantedly, Inc. exports.default = 1; exports.__esModule = true; lib1.js import a from "./lib1.js"; console.log("a =", a); export default 1; lib2.js import b from "./lib1.js"; import c from "./lib2.js"; console.log("b =", b); console.log("c =", c); app.mjs $ yarn webpack ./app.mjs --mode production --target node -o . (..snip..) webpack 5.75.0 compiled successfully in 122 ms ✨ Done in 0.50s.
145. Set up Webpack Check import result © 2022 Wantedly, Inc. exports.default = 1; exports.__esModule = true; lib1.js import a from "./lib1.js"; console.log("a =", a); export default 1; lib2.js import b from "./lib1.js"; import c from "./lib2.js"; console.log("b =", b); console.log("c =", c); app.mjs $ node main.js a = 1 b = { default: 1, __esModule: true } c = 1
146. 11: Meta APIs for modules © 2022 Wantedly, Inc.
147. Meta APIs for modules • Node.js implements `import` and `require` separately. ◦ Not just the parser, but the whole module lifecycle is implemented differently. • Consequently, they expose different APIs for ESM and CJS. © 2022 Wantedly, Inc.
148. Loader Hook: CJS • require.extensions[2] exists for this. • Isn’t it deprecated? Yes, but even @babel/register continues using it. • Things could easily go wrong with this API. Use the pirate[2] package for reliability. © 2022 Wantedly, Inc. [1] https://nodejs.org/api/modules.html#requireextensions [2] https://www.npmjs.com/package/pirates
149. Loader Hook: CJS © 2022 Wantedly, Inc. const oldLoader = require.extensions[".js"]; require.extensions[".js"] = function(mod, filename) { const oldCompile = mod._compile; mod._compile = function(code, filename) { const newCode = `console.log(__filename);` + code; return oldCompile.call(this, newCode, filename); }; oldLoader(mod, filename); }; register.cjs
150. Loader Hook: CJS © 2022 Wantedly, Inc. module.exports = 42; lib.cjs const x = require("./lib.cjs"); console.log(x); app.cjs register.cjs
151. Loader Hook: CJS © 2022 Wantedly, Inc. module.exports = 42; lib.cjs const x = require("./lib.cjs"); console.log(x); app.cjs register.cjs $ node -r ./register.cjs app.cjs app.cjs lib.cjs 42
152. Loader Hook: ESM • Conversely, Node.js has a (truly) public loader API[1] for ESM. • It is still experimental and subject to change. © 2022 Wantedly, Inc. [1] https://nodejs.org/api/esm.html#loaders
153. Loader Hook: ESM © 2022 Wantedly, Inc. export async function load(url, context, nextLoad) { const m = await nextLoad(url, context); if (m.format !== "module") return m; if (typeof m.source !== "string") { m.source = new TextDecoder().decode(m.source); } m.source = `console.log(import.meta.url); ${m.source}`; return m; } loader.mjs
154. Loader Hook: ESM © 2022 Wantedly, Inc. export default 42; lib.mjs import x from "./lib.mjs"; console.log(x); app.mjs loader.mjs
155. Loader Hook: ESM © 2022 Wantedly, Inc. export default 42; lib.mjs import x from "./lib.cjs"; console.log(x); app.mjs loader.mjs $ node --experimental-loader ./loader.mjs ./app.mjs file://<dir>/app.cjs file://<dir>/lib.cjs 42
156. Parsing modules • Want to re-implement module loader? ◦ Jest does this • For CJS, Function suffices. • For ESM, we need vm.Module (experimental). © 2022 Wantedly, Inc.
157. Parsing CJS modules © 2022 Wantedly, Inc. const fs = require("node:fs"); const m = { exports: {} }; const f = new Function( "exports", "require", "module", "__filename", "__dirname", fs.readFileSync("lib.cjs", "utf-8")); f(m.exports, null, m, "", ""); console.log(m.exports); app.cjs
158. Parsing CJS modules © 2022 Wantedly, Inc. app.cjs module.exports = 42; lib.cjs
159. Parsing CJS modules © 2022 Wantedly, Inc. app.cjs module.exports = 42; lib.cjs $ node app.cjs 42
160. Parsing ESM modules © 2022 Wantedly, Inc. import fs from "node:fs"; import vm from "node:vm"; const m = new vm.SourceTextModule( fs.readFileSync("lib.mjs", "utf-8")); await m.link(() => {}); await m.evaluate(); console.log(m.namespace.default); app.mjs
161. Parsing ESM modules © 2022 Wantedly, Inc. app.mjs export default 42; lib.mjs
162. Parsing ESM modules © 2022 Wantedly, Inc. app.mjs export default 42; lib.mjs $ node --experimental-vm-modules app.cjs 42 (node:10167) ExperimentalWarning: VM Modules is an experimental feature. This feature could change at any time
163. Key takeaways © 2022 Wantedly, Inc.
164. Key takeaways • There are dialects of ES Modules: ◦ transpiled to CJS and then run ◦ fed to Node.js as-is ◦ fed to Webpack as autodected ESM • Lots of things complicate interoperation between CJS and ESM • But now your hands remember what is happening behind the scenes. Nothing to fear. © 2022 Wantedly, Inc.
165. Appendix A package.json © 2022 Wantedly, Inc.
166. package.json roles in Node.js © 2022 Wantedly, Inc. Fields Which package.json to use Module type detection type Nearest ancestor of the resolved module Package exports exports main `segment` or `@scope/segment` part of the request Folder main (only in CJS/require) main The requested directory Package imports imports Nearest ancestor of the requesting module Self name exports Nearest ancestor of the requesting module
167. Appendix B Module polyglot © 2022 Wantedly, Inc.
168. Recap Snippet to check module types © 2022 Wantedly, Inc. if (typeof await /0/["test"] === "function") { console.log("I'm ESM"); } else if (typeof require === "function") { console.log("I'm CJS"); } else { console.log("I'm other"); } module-type.js
169. How ESM check works © 2022 Wantedly, Inc. typeof await /0/["test"] === "function" typeof (await (/0/["test"])) === "function" ((typeof await) / 0) / ["test"] === "function" = RegExp.prototype.test no-op variable (nonexistent) = “undefined” cast to NaN ESM: [~Yield, +Await, ~Return] CJS: [~Yield, ~Await, +Return]
170. Advanced application (not recommended for production use!) Module polyglot © 2022 Wantedly, Inc. "use strict"; typeof await /[//]/; export default /* ]; module.exports = /**/ 42; module-polyglot.js