Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Hands-on Native ESM @ JSConf JP 2022

Masaki Hara
November 26, 2022

Hands-on Native ESM @ JSConf JP 2022

See Original version on Google Slides for copy-pasting and links.

See Materials

Masaki Hara

November 26, 2022
Tweet

More Decks by Masaki Hara

Other Decks in Programming

Transcript

  1. © 2022 Wantedly, Inc. Hands-on Native ESM Understanding Node.js module

    discrepancy Nov. 26 2022 - Masaki Hara https://github.com/qnighy https://www.wantedly.com/id/qnighy
  2. The aim of this workshop • Understand how CJS and

    ESM (does not smoothly) interoperate in Node.js • Understand how it becomes more difficult when module budler is involved © 2022 Wantedly, Inc. It does not cover the entire solution, but it will nonetheless help you migrating to Native ESM.
  3. Related materials • 実践 Node.js Native ESM — Wantedlyでのアプリケー ション移行事例[1]

    • Node.jsのネイティブES Modulesサポートが抱える問題を 解決するBabelプラグインを書いた[2] • Native ESM + TypeScript 拡張子問題: 歯にものが挟まっ たようなスッキリしない書き流し[3] © 2022 Wantedly, Inc. [1] https://www.wantedly.com/companies/wantedly/post_articles/410531 [2] https://zenn.dev/qnighy/articles/6267716578c76d [3] https://zenn.dev/qnighy/articles/19603f11d5f264
  4. Advertisement We are looking for JS ecosystem enthusiasts! © 2022

    Wantedly, Inc. WebpackやBabelの設定を通じて開発を加速させたい人募集 ! - Wantedly[1] [1] https://www.wantedly.com/projects/1145492
  5. Advertisement Our frontend team also welcomes you! © 2022 Wantedly,

    Inc. 今見ているWantedlyのWebフロントエンドをもっと良くしませんか? - Wantedly[1] [1] https://www.wantedly.com/projects/1159802
  6. Hands-on guide • Materials can be found at: https://github.com/wantedly/hands-on-native- esm-2022

    • Requires Node.js ≧ 16 • I will demonstrate chapters 1 to 4. • Read and exercise the remaining chapters (5 to 11) if you are interested. © 2022 Wantedly, Inc.
  7. 1: Let’s try ESM! © 2022 Wantedly, Inc.

  8. Try Native ESM • Node.js has two module types: ◦

    ESM = ES Modules (*.mjs, *.js) ◦ CJS = CommonJS Modules (*.cjs, *.js) • They interoperate badly 😵 • You usually write JS in ESM but execute it as CJS. © 2022 Wantedly, Inc. (as opposed to Native ESM)
  9. Try Native ESM Try ESM import/export © 2022 Wantedly, Inc.

    export default function(x) { return x * x; } import square from "./lib.mjs"; console.log(square(2)); lib.mjs app.mjs
  10. Try Native ESM Try ESM import/export © 2022 Wantedly, Inc.

    export default function(x) { return x * x; } import square from "./lib.mjs"; console.log(square(2)); lib.mjs app.mjs $ node app.mjs 4
  11. Try Native ESM Open https://babeljs.io/repl © 2022 Wantedly, Inc.

  12. Try Native ESM Try CJS transpilation © 2022 Wantedly, Inc.

    "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); exports.default = _default; function _default(x) { return x * x; } "use strict"; var _a = _interopRequireDefault(require("./a.cjs")); function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; } console.log((0, _a.default)(2)); lib.cjs (before Babel) app.cjs (before Babel) export default function(x) { return x * x; } import square from "./lib.cjs"; console.log(square(2));
  13. Try Native ESM Try CJS transpilation © 2022 Wantedly, Inc.

    "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); exports.default = _default; function _default(x) { return x * x; } "use strict"; var _a = _interopRequireDefault(require("./lib.cjs")); function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; } console.log((0, _a.default)(2)); lib.cjs app.cjs
  14. Try Native ESM Try CJS transpilation © 2022 Wantedly, Inc.

    "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); exports.default = _default; function _default(x) { return x * x; } "use strict"; var _a = _interopRequireDefault(require("./lib.cjs")); function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; } console.log((0, _a.default)(2)); lib.cjs app.cjs $ node app.cjs 4
  15. Try Native ESM CJS → ESM © 2022 Wantedly, Inc.

    "use strict"; var _a = _interopRequireDefault(require("./lib.mjs")); function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; } console.log((0, _a.default)(2)); app.cjs export default function(x) { return x * x; } lib.mjs
  16. Try Native ESM CJS → ESM © 2022 Wantedly, Inc.

    "use strict"; var _a = _interopRequireDefault(require("./lib.mjs")); function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; } console.log((0, _a.default)(2)); app.cjs export default function(x) { return x * x; } lib.mjs $ node app.cjs node:internal/modules/cjs/loader:1031 throw new ERR_REQUIRE_ESM(filename, true); ^ Error [ERR_REQUIRE_ESM]: require() of ES Module lib.mjs not supported. Instead change the require of lib.mjs to a dynamic import() which is available in all CommonJS modules.
  17. Try Native ESM ESM → CJS © 2022 Wantedly, Inc.

    "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); exports.default = _default; function _default(x) { return x * x; } lib.cjs import square from "./lib.cjs"; console.log(square(2)); app.mjs
  18. Try Native ESM ESM → CJS © 2022 Wantedly, Inc.

    "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); exports.default = _default; function _default(x) { return x * x; } lib.cjs import square from "./lib.cjs"; console.log(square(2)); app.mjs $ node app.mjs app.mjs:3 console.log(square(2)); ^ TypeError: square is not a function at app.mjs:3:13
  19. There is no easy path • Library-first upgrade: essential difficulty

    ◦ Synchronous vs. Asynchronous • Application-first upgrade: avoidable pitfalls ◦ Default import problem • Dual package: difficulty in some packages ◦ State sharing problem © 2022 Wantedly, Inc.
  20. But we must do it • Some libraries are already

    in Native ESM format. ◦ node-fetch ◦ d3 ◦ etc. • Depending applications are (almost) forced to do the migration. © 2022 Wantedly, Inc.
  21. 2: Module detection © 2022 Wantedly, Inc.

  22. How Node.js determines module types • Node.js determines[1] module types

    using: ◦ File extensions (*.cjs / *.mjs) ◦ Package metadata (“type”: “module”) if it is *.js © 2022 Wantedly, Inc. *.cjs *.js *.mjs “type”: “commonjs” CJS CJS ESM no “type” field CJS CJS ESM “type”: “module” CJS ESM ESM Conversely, they are not taken into account: • File contents • How the import had been routed (e.g. exports map) [1] https://nodejs.org/api/packages.html#packagejson-and-file-extensions
  23. How Node.js determines module types Check module types © 2022

    Wantedly, Inc. if (typeof await /0/["test"] === "function") { console.log("I'm ESM"); } else if (typeof require === "function") { console.log("I'm CJS"); } else { console.log("I'm other"); } module-type.js
  24. Check module types © 2022 Wantedly, Inc. if (typeof await

    /0/["test"] === "function") { console.log("I'm ESM"); } else if (typeof require === "function") { console.log("I'm CJS"); } else { console.log("I'm other"); } module-type.js $ node module-type.js I’m CJS $ node module-type.cjs I’m CJS $ node module-type.mjs I’m ESM How Node.js determines module types
  25. Module type depends on package.json © 2022 Wantedly, Inc. {

    "type": "module" } package.json How Node.js determines module types
  26. Module type depends on package.json © 2022 Wantedly, Inc. {

    "type": "module" } package.json $ node module-type.js I’m ESM $ node module-type.cjs I’m CJS $ node module-type.mjs I’m ESM How Node.js determines module types
  27. 3: Semantics of execution © 2022 Wantedly, Inc. Module asynchrony

  28. Semantics of execution • Modules are executed ◦ synchronously in

    CJS ◦ asynchronously in ESM • Consequence👉 CJS cannot import ESM[1] © 2022 Wantedly, Inc. [1]: Precisely speaking, it can wait for the imported module to finish.
  29. Do a time-consuming initialization © 2022 Wantedly, Inc. import {

    setTimeout } from "node:timers/promises"; await setTimeout(3000); // For Node.js < 16.0.0 // await new Proise((r) => setTimeout(r, 3000)); console.log("slept 1s"); sleep.mjs Explicit asynchrony: top-level await
  30. Dependent module may also be asynchronous © 2022 Wantedly, Inc.

    import { setTimeout } from "node:timers/promises"; await setTimeout(3000); export const value = 42; sleep-lib.mjs Implied asynchrony
  31. Dependent module may also be asynchronous © 2022 Wantedly, Inc.

    import { value } from "./sleep-lib.mjs"; console.log("value =", value); sleep-app.mjs Implied asynchrony
  32. Dependent module may also be asynchronous © 2022 Wantedly, Inc.

    const { value } = await import("./sleep-lib.mjs"); console.log("value =", value); sleep-app.mjs (alternative) Expanding implied asynchrony
  33. Require — this is a mere function © 2022 Wantedly,

    Inc. const { value } = require("./sleep-lib.mjs"); console.log("value =", value); sleep-app.cjs Discrepancy in asynchrony Error!
  34. Require — this is a mere function © 2022 Wantedly,

    Inc. const { value } = require("./sleep-lib.mjs"); console.log("value =", value); sleep-app.cjs Discrepancy in asynchrony Error! $ node sleep-app.cjs node:internal/modules/cjs/loader:1031 throw new ERR_REQUIRE_ESM(filename, true); ^ Error [ERR_REQUIRE_ESM]: require() of ES Module sleep-lib.mjs not supported.
  35. TLA — this is not possible © 2022 Wantedly, Inc.

    const { value } = await import("./sleep-lib.mjs"); console.log("value =", value); sleep-app.cjs Discrepancy in asynchrony Error!
  36. TLA — this is not possible © 2022 Wantedly, Inc.

    const { value } = await import("./sleep-lib.mjs"); console.log("value =", value); sleep-app.cjs Discrepancy in asynchrony Error! $ node sleep-app.cjs sleep-app.cjs:1 const { value } = await import("./sleep-lib.mjs"); ^^^^^ SyntaxError: await is only valid in async functions and the top level bodies of modules
  37. Import-then © 2022 Wantedly, Inc. import("./sleep-lib.mjs").then(({ value }) => {

    console.log("value =", value); }); sleep-app.cjs Discrepancy in asynchrony
  38. Import-then: it’s too late for exporting © 2022 Wantedly, Inc.

    import("./sleep-lib.mjs").then(({ value }) => { console.log("value =", value); exports.value2 = value * 2; }); sleep-app.cjs Discrepancy in asynchrony Too late!
  39. 4: Default export semantics © 2022 Wantedly, Inc.

  40. Default export semantics • Default export is ◦ fancy namespace

    export in CJS ◦ special named export in ESM • Consequence👉 incompatible translations • My library node-cjs-interop[1] helps here. © 2022 Wantedly, Inc. [1] https://github.com/qnighy/node-cjs-interop
  41. Default export semantics © 2022 Wantedly, Inc. Default Named export

    default f; export { f }; module.exports = f; exports.f = f;
  42. Default export semantics from importer’s viewpoint © 2022 Wantedly, Inc.

    import * as ns from ""; ✅ Namespace ❌ Default ✅ Named import f from ""; import { f } from ""; const ns = require(""); const f = require(""); const { f } = require(""); = ∈
  43. Default export semantics from importer’s viewpoint © 2022 Wantedly, Inc.

    import * as ns from ""; ✅ Namespace ❌ Default ✅ Named import { default as f } from ""; import { f } from ""; const ns = require(""); const f = require(""); const { f } = require(""); = ∈
  44. Default export interop 1: Node.js rule (CJS-preserving) © 2022 Wantedly,

    Inc. import * as ns from ""; import f from ""; import { f } from ""; const ns = require(""); const f = require(""); const { f } = require(""); = ∈ (except “default”) (except “default”)
  45. Default export interop 2: simple mapping (ESM-preserving) © 2022 Wantedly,

    Inc. import * as ns from ""; import f from ""; import { f } from ""; const ns = require(""); const f = require(""); const { f } = require(""); = ∈ (Loss of information)
  46. Default export interop 3: Babel mapping © 2022 Wantedly, Inc.

    var _react = _interopRequireDefault(require("react")); function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; } Simple mapping (ESM-preserving)
  47. Default export interop 3: Babel mapping © 2022 Wantedly, Inc.

    var _react = _interopRequireDefault(require("react")); function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; } Node.js rule (CJS-preserving)
  48. Default export interop – tool support © 2022 Wantedly, Inc.

    Node.js rule Babel rule (hybrid) Simple mapping Node.js ✅ ❌ but see: node-cjs-interop[1] ❌ Webpack javascript/esm javascript/auto ❌ Babel interop = node interop = babel interop = none TypeScript ❌ esModuleInterop = true esModuleInterop = false [1] https://github.com/qnighy/node-cjs-interop
  49. 4.1: Default export semantics (1) © 2022 Wantedly, Inc. Default

    export in ESM
  50. Default export in ESM Default export and default import in

    ESM © 2022 Wantedly, Inc. export default function(x) { return x * x; } import square from "./lib.mjs"; console.log(square(2)); lib.mjs app.mjs
  51. Default export in ESM Default export and default import in

    ESM © 2022 Wantedly, Inc. export default function(x) { return x * x; } import square from "./lib.mjs"; console.log(square(2)); lib.mjs app.mjs $ node app.mjs 4
  52. Default export in ESM Default import is a syntax sugar

    © 2022 Wantedly, Inc. export default function(x) { return x * x; } import { default as square } from "./lib.mjs"; console.log(square(2)); lib.mjs app.mjs
  53. Default export in ESM Default import is a syntax sugar

    © 2022 Wantedly, Inc. export default function(x) { return x * x; } import { default as square } from "./lib.mjs"; console.log(square(2)); lib.mjs app.mjs $ node app.mjs 4
  54. Default export in ESM Default export is a syntax sugar

    © 2022 Wantedly, Inc. function square(x) { return x * x; } export { square as default }; import square from "./lib.mjs"; console.log(square(2)); lib.mjs app.mjs
  55. Default export in ESM Default export is a syntax sugar

    © 2022 Wantedly, Inc. function square(x) { return x * x; } export { square as default }; import square from "./lib.mjs"; console.log(square(2)); lib.mjs app.mjs $ node app.mjs 4
  56. 4.2: Default export semantics (2) © 2022 Wantedly, Inc. Default

    export in CJS
  57. Default export in CJS Default export and default import in

    CJS © 2022 Wantedly, Inc. module.exports = 42; const x = require("./lib.cjs"); console.log(x); lib.cjs app.cjs
  58. Default export in CJS Default export and default import in

    CJS © 2022 Wantedly, Inc. module.exports = 42; const x = require("./lib.cjs"); console.log(x); lib.cjs app.cjs $ node app.mjs 42
  59. Node.js rule for default export Default export interop © 2022

    Wantedly, Inc. module.exports = 42; import x from "./app.cjs"; console.log(x); lib.cjs app.mjs
  60. Node.js rule for default export Default export interop © 2022

    Wantedly, Inc. module.exports = 42; import x from "./app.cjs"; console.log(x); lib.cjs app.mjs $ node app.mjs 42
  61. 4.3: Default export semantics (3) © 2022 Wantedly, Inc. Named

    exports in CJS
  62. Named exports in CJS Named exports in CJS © 2022

    Wantedly, Inc. exports.foo = 1; exports.bar = 2; const m = require("./lib.cjs"); console.log(m); lib.cjs app.cjs
  63. Named exports in CJS Named exports in CJS © 2022

    Wantedly, Inc. exports.foo = 1; exports.bar = 2; const m = require("./lib.cjs"); console.log(m); lib.cjs app.cjs $ node app.cjs { foo: 1, bar: 2 }
  64. Node.js rule for named exports Named exports interop © 2022

    Wantedly, Inc. exports.foo = 1; exports.bar = 2; import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2); lib.cjs app.mjs
  65. Node.js rule for named exports Named exports interop © 2022

    Wantedly, Inc. exports.foo = 1; exports.bar = 2; import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2); lib.cjs app.mjs $ node app.mjs { foo: 1, bar: 2 } [Module: null prototype] { bar: 2, default: { foo: 1, bar: 2 }, foo: 1 }
  66. Node.js rule for named “default” exports Named “default” exports interop

    © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; exports.default = 3; import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2); lib.cjs app.mjs
  67. Node.js rule for named “default” exports Named “default” exports interop

    © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; exports.default = 3; import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2); lib.cjs app.mjs $ node app.mjs { foo: 1, bar: 2, default: 3 } [Module: null prototype] { bar: 2, default: { foo: 1, bar: 2, default: 3 }, foo: 1 }
  68. 4.4: Default export semantics (4) © 2022 Wantedly, Inc. Babel-style

    interop
  69. Mixing CJS named and default exports Named + default exports

    in CJS © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; module.exports = 3; lib.cjs import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2); app.mjs
  70. Mixing CJS named and default exports Named + default exports

    in CJS © 2022 Wantedly, Inc. exports.foo = 1; exports.bar = 2; module.exports = 3; lib.cjs import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2); app.mjs $ node app.cjs 3 [Module: null prototype] { bar: undefined, default: 3, foo: undefined }
  71. Babel-style interop Special flag for simple mapping © 2022 Wantedly,

    Inc. exports.foo = 1; exports.bar = 2; exports.default = 3; exports.__esModule = true; lib.cjs app.mjs import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2);
  72. Babel-style interop Special flag for simple mapping © 2022 Wantedly,

    Inc. exports.foo = 1; exports.bar = 2; exports.default = 3; exports.__esModule = true; var m2 = _interopRequireWildcard( require("./lib.cjs")); console.log(m2.default); console.log(m2); lib.cjs app.cjs (transpiled from app.mjs)
  73. Babel-style interop Special flag for simple mapping © 2022 Wantedly,

    Inc. exports.foo = 1; exports.bar = 2; exports.default = 3; exports.__esModule = true; "use strict"; var m2 = _interopRequireWildcard(require("./lib.cjs")); function _getRequireWildcardCache(nodeInterop) { if (typeof WeakMap !== "function") return null; var cacheBabelInterop = new WeakMap(); var cacheNodeInterop = new WeakMap(); return (_getRequireWildcardCache = function (nodeInterop) { return nodeInterop ? cacheNodeInterop : cacheBabelInterop; })(nodeInterop); } function _interopRequireWildcard(obj, nodeInterop) { if (!nodeInterop && obj && obj.__esModule) { return obj; } if (obj === null || typeof obj !== "object" && typeof obj !== "function") { return { default: obj }; } var cache = _getRequireWildcardCache(nodeInterop); if (cache && cache.has(obj)) { return cache.get(obj); } var newObj = {}; var hasPropertyDescriptor = Object.defineProperty && Object.getOwnPropertyDescriptor; for (var key in obj) { if (key !== "default" && Object.prototype.hasOwnProperty.call(obj, key)) { var desc = hasPropertyDescriptor ? Object.getOwnPropertyDescriptor(obj, key) : null; if (desc && (desc.get || desc.set)) { Object.defineProperty(newObj, key, desc); } else { newObj[key] = obj[key]; } } } newObj.default = obj; if (cache) { cache.set(obj, newObj); } return newObj; } console.log(m2.default); console.log(m2); lib.cjs app.cjs (transpiled from app.mjs)
  74. Babel-style interop Special flag for simple mapping © 2022 Wantedly,

    Inc. exports.foo = 1; exports.bar = 2; exports.default = 3; exports.__esModule = true; "use strict"; var m2 = _interopRequireWildcard(require("./lib.cjs")); function _getRequireWildcardCache(nodeInterop) { if (typeof WeakMap !== "function") return null; var cacheBabelInterop = new WeakMap(); var cacheNodeInterop = new WeakMap(); return (_getRequireWildcardCache = function (nodeInterop) { return nodeInterop ? cacheNodeInterop : cacheBabelInterop; })(nodeInterop); } function _interopRequireWildcard(obj, nodeInterop) { if (!nodeInterop && obj && obj.__esModule) { return obj; } if (obj === null || typeof obj !== "object" && typeof obj !== "function") { return { default: obj }; } var cache = _getRequireWildcardCache(nodeInterop); if (cache && cache.has(obj)) { return cache.get(obj); } var newObj = {}; var hasPropertyDescriptor = Object.defineProperty && Object.getOwnPropertyDescriptor; for (var key in obj) { if (key !== "default" && Object.prototype.hasOwnProperty.call(obj, key)) { var desc = hasPropertyDescriptor ? Object.getOwnPropertyDescriptor(obj, key) : null; if (desc && (desc.get || desc.set)) { Object.defineProperty(newObj, key, desc); } else { newObj[key] = obj[key]; } } } newObj.default = obj; if (cache) { cache.set(obj, newObj); } return newObj; } console.log(m2.default); console.log(m2); lib.cjs app.cjs (transpiled from app.mjs) $ node app.cjs 3 { foo: 1, bar: 2, default: 3, __esModule: true }
  75. Babel-style interop Node.js does not support __esModule © 2022 Wantedly,

    Inc. exports.foo = 1; exports.bar = 2; exports.default = 3; exports.__esModule = true; lib.cjs import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2); app.mjs
  76. Babel-style interop Node.js does not support __esModule © 2022 Wantedly,

    Inc. exports.foo = 1; exports.bar = 2; exports.default = 3; exports.__esModule = true; lib.cjs import m1 from "./lib.cjs"; import * as m2 from "./lib.cjs"; console.log(m1); console.log(m2); app.mjs $ node app.mjs { foo: 1, bar: 2, default: 3 } [Module: null prototype] { __esModule: true, bar: 2, default: { foo: 1, bar: 2, default: 3 }, foo: 1 }
  77. End of demonstration Enjoy the remaining exercises if you are

    interested in. © 2022 Wantedly, Inc.
  78. 5: Export analysis © 2022 Wantedly, Inc.

  79. Export analysis • Static analysis is mandatory in Native ESM,

    and it also applies to CJS imported from ESM. • It sometimes fails to detect exports in CJS. © 2022 Wantedly, Inc.
  80. Export analysis Node.js analyzes ESM exports before execution © 2022

    Wantedly, Inc. console.log("Initializing..."); export const foo = 1; lib.mjs import { bar } from "./lib.mjs"; console.log(bar); app.mjs
  81. Export analysis Node.js analyzes ESM exports before execution © 2022

    Wantedly, Inc. console.log("Initializing..."); export const foo = 1; lib.mjs import { bar } from "./lib.mjs"; console.log(bar); app.mjs $ node app.mjs app.mjs:1 import { bar } from "./lib.mjs"; ^^^ SyntaxError: The requested module './lib.mjs' does not provide an export named 'bar'
  82. Export analysis Node.js analyzes ESM exports before execution © 2022

    Wantedly, Inc. console.log("Initializing..."); export const foo = 1; lib.mjs import { bar } from "./lib.mjs"; console.log(bar); app.mjs $ node app.mjs app.mjs:1 import { bar } from "./lib.mjs"; ^^^ SyntaxError: The requested module './lib.mjs' does not provide an export named 'bar' No “Initializing…” printed
  83. Export analysis It analyzes CJS too (when imported from ESM)

    © 2022 Wantedly, Inc. console.log("Initializing..."); exports.foo = 1; lib.cjs import { bar } from "./lib.cjs"; console.log(bar); app.mjs
  84. Export analysis It analyzes CJS too (when imported from ESM)

    © 2022 Wantedly, Inc. console.log("Initializing..."); exports.foo = 1; lib.cjs import { bar } from "./lib.cjs"; console.log(bar); app.mjs $ node app.mjs app.mjs:1 import { bar } from "./lib.cjs"; ^^^ SyntaxError: Named export 'bar' not found. The requested module './lib.cjs' is a CommonJS module, which may not support all module.exports as named exports.
  85. Export analysis It analyzes CJS too (when imported from ESM)

    © 2022 Wantedly, Inc. console.log("Initializing..."); exports.bar = 2; lib.cjs import { bar } from "./app.cjs"; console.log(bar); app.mjs
  86. Export analysis It analyzes CJS too (when imported from ESM)

    © 2022 Wantedly, Inc. console.log("Initializing..."); exports.bar = 2; lib.cjs import { bar } from "./app.cjs"; console.log(bar); app.mjs $ node app.mjs Initializing... 2
  87. Export analysis It analyzes CJS too (when imported from ESM)

    © 2022 Wantedly, Inc. console.log("Initializing..."); exports["bar"] = 2; lib.cjs import { bar } from "./lib.cjs"; console.log(bar); app.mjs
  88. Export analysis It analyzes CJS too (when imported from ESM)

    © 2022 Wantedly, Inc. console.log("Initializing..."); exports["bar"] = 2; lib.cjs import { bar } from "./lib.cjs"; console.log(bar); app.mjs $ node app.mjs Initializing... 2
  89. Export analysis Not all exports can be analyzed © 2022

    Wantedly, Inc. console.log("Initializing..."); exports["ba" + "r"] = 2; lib.cjs import { bar } from "./lib.cjs"; console.log(bar); app.mjs
  90. Export analysis Not all exports can be analyzed © 2022

    Wantedly, Inc. console.log("Initializing..."); exports["ba" + "r"] = 2; lib.cjs import { bar } from "./lib.cjs"; console.log(bar); app.mjs $ node app.mjs app.mjs:1 import { bar } from "./lib.cjs"; ^^^ SyntaxError: Named export 'bar' not found. The requested module './lib.cjs' is a CommonJS module, which may not support all module.exports as named exports.
  91. Export analysis rules • Rules can be found at cjs-module-lexer[1]

    © 2022 Wantedly, Inc. [1] https://github.com/nodejs/cjs-module-lexer exports.foo = 1; module.exports.foo = 1; exports["foo"] = 1; module.exports = { foo }; module.exports = { foo: bar }; Object.defineProperty(exports, "foo", ); { value: 1 } { enumerable: true, value: 1 } { get() { return bar; } } { get: function() { return bar; } } { get() { return m.bar; } } { get() { return m["bar"]; } }
  92. Export analysis rules • Rules can be found at cjs-module-lexer[1]

    © 2022 Wantedly, Inc. [1] https://github.com/nodejs/cjs-module-lexer module.exports = require("mod"); module.exports = { ...require("mod") }; __export(require("mod")); __exportStar(require("mod"), exports); var m = require("mod"); const m = require("mod"); let m = require("mod"); var m = _interopRequireWildcard (require("mod")); Object.keys(m).forEach(...); +
  93. 6: Live exports © 2022 Wantedly, Inc.

  94. Live binding of exports • In ESM, imported variables reference

    the latest value of the original variable. • This is not the case with CJS interop in Node.js. © 2022 Wantedly, Inc.
  95. Live binding of exports counter evaluates to the latest value

    © 2022 Wantedly, Inc. export let counter = 0; export function countUp() { counter++; } lib.mjs import { counter, countUp } from "./lib.cjs"; console.log(counter); countUp(); console.log(counter); app.mjs
  96. Live binding of exports counter evaluates to the latest value

    © 2022 Wantedly, Inc. export let counter = 0; export function countUp() { counter++; } lib.mjs import { counter, countUp } from "./lib.cjs"; console.log(counter); countUp(); console.log(counter); app.mjs $ node app.mjs 0 1
  97. Live binding of exports counter evaluates to the value when

    CJS is imported © 2022 Wantedly, Inc. exports.counter = 0; exports.countUp = function() { exports.counter++; } lib.cjs import { counter, countUp } from "./lib.cjs"; console.log(counter); countUp(); console.log(counter); app.mjs
  98. Live binding of exports counter evaluates to the value when

    CJS is imported © 2022 Wantedly, Inc. exports.counter = 0; exports.countUp = function() { exports.counter++; } lib.cjs import { counter, countUp } from "./lib.cjs"; console.log(counter); countUp(); console.log(counter); app.mjs $ node app.mjs 0 0
  99. 7: Module path resolution © 2022 Wantedly, Inc.

  100. Module path resolution • import() no longer automatically adds extensions.

    • It no longer automatically resolves directories as “index.js”. • It no longer resolves “folder main”. • *.cjs is not added for either CJS or ESM. © 2022 Wantedly, Inc.
  101. Module path resolution “.js” is added in CJS © 2022

    Wantedly, Inc. console.log("imported"); lib.js require("./lib"); app.cjs
  102. Module path resolution “.js” is added in CJS © 2022

    Wantedly, Inc. console.log("imported"); lib.js require("./lib"); app.cjs $ node app.cjs imported
  103. Module path resolution “.js” is no longer added in ESM

    © 2022 Wantedly, Inc. console.log("imported"); lib.js import "./lib"; app.mjs
  104. Module path resolution “.js” is no longer added in ESM

    © 2022 Wantedly, Inc. console.log("imported"); lib.js import "./lib"; app.mjs $ node app.mjs node:internal/process/esm_loader:97 internalBinding('errors').triggerUncaughtE xception( ^ Error [ERR_MODULE_NOT_FOUND]: Cannot find module 'lib' imported from app.mjs Did you mean to import ../lib.js?
  105. Module path resolution ./index.js is added in CJS © 2022

    Wantedly, Inc. console.log("imported"); lib/index.js require("./lib"); app.cjs
  106. Module path resolution ./index.js is added in CJS © 2022

    Wantedly, Inc. console.log("imported"); lib/index.js require("./lib"); app.cjs $ node app.cjs imported
  107. Module path resolution ./index.js is no longer added in ESM

    © 2022 Wantedly, Inc. console.log("imported"); lib/index.js import "./lib"; app.mjs
  108. Module path resolution ./index.js is no longer added in ESM

    © 2022 Wantedly, Inc. console.log("imported"); lib/index.js import "./lib"; app.mjs $ node app.mjs node:internal/process/esm_loader:97 internalBinding('errors').triggerUncaughtExcep tion( ^ Error [ERR_UNSUPPORTED_DIR_IMPORT]: Directory import 'lib' is not supported resolving ES modules imported from app.mjs Did you mean to import ../lib/index.js?
  109. Module path resolution “Folder main” in CJS © 2022 Wantedly,

    Inc. console.log("imported"); lib/main.js require("./lib"); app.cjs { "main": "./main.js" } lib/package.json
  110. Module path resolution “Folder main” in CJS © 2022 Wantedly,

    Inc. console.log("imported"); lib/main.js require("./lib"); app.cjs { "main": "./main.js" } lib/package.json $ node app.cjs imported
  111. Module path resolution “Folder main” is no longer available ©

    2022 Wantedly, Inc. console.log("imported"); lib/main.js import "./lib"; app.mjs { "main": "./main.js" } lib/package.json
  112. Module path resolution “Folder main” is no longer available ©

    2022 Wantedly, Inc. console.log("imported"); lib/main.js import "./lib"; app.mjs { "main": "./main.js" } lib/package.json $ node app.mjs node:internal/process/esm_loader:97 internalBinding('errors').triggerUncaughtExcep tion( ^ Error [ERR_UNSUPPORTED_DIR_IMPORT]: Directory import 'lib' is not supported resolving ES modules imported from app.mjs Did you mean to import ../lib/main.js?
  113. Module path resolution “Package main” is still valid © 2022

    Wantedly, Inc. console.log("imported"); node_modules/lib/main.js import "lib"; app.mjs { "main": "./main.js" } node_modules/lib/package.json
  114. Module path resolution “Package main” is still valid © 2022

    Wantedly, Inc. console.log("imported"); node_modules/lib/main.js import "lib"; app.mjs { "main": "./main.js" } node_modules/lib/package.json $ node app.mjs imported
  115. Module path resolution “.cjs” is not added even from CJS

    © 2022 Wantedly, Inc. console.log("imported"); lib.cjs require("./lib"); app.cjs
  116. Module path resolution “.cjs” is not added even from CJS

    © 2022 Wantedly, Inc. console.log("imported"); lib.cjs require("./lib"); app.cjs $ node app.mjs node:internal/modules/cjs/loader:988 throw err; ^ Error: Cannot find module './lib
  117. 8: Module metadata © 2022 Wantedly, Inc.

  118. Module metadata • These CJS variables are usually available in

    to-be-transpiled ESM code: ◦ require ◦ exports ◦ module ◦ __filename ◦ __dirname • This is no longer the case in Native ESM. © 2022 Wantedly, Inc.
  119. Module metadata CJS metavariables are not available © 2022 Wantedly,

    Inc. console.log(`I'm ${__filename}`); export {}; meta.mjs
  120. Module metadata CJS metavariables are not available © 2022 Wantedly,

    Inc. console.log(`I'm ${__filename}`); export {}; meta.mjs $ node meta.mjs file:///<dir>/meta.mjs:1 console.log(`I'm ${__filename}`); ^ ReferenceError: __filename is not defined in ES module scope
  121. Module metadata They are usually transpiled as-is © 2022 Wantedly,

    Inc. "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); console.log(`I'm ${__filename}`); meta.cjs (transpiled from meta.mjs)
  122. Module metadata They are usually transpiled as-is © 2022 Wantedly,

    Inc. "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); console.log(`I'm ${__filename}`); meta.cjs (transpiled from meta.mjs) $ node meta.cjs I’m <dir>/meta.cjs
  123. Module metadata Use import.meta instead in Native ESM © 2022

    Wantedly, Inc. console.log(`I'm ${import.meta.url}`); export {}; meta.mjs
  124. Module metadata Use import.meta instead in Native ESM © 2022

    Wantedly, Inc. console.log(`I'm ${import.meta.url}`); export {}; meta.mjs $ node meta.mjs I’m file://<dir>/meta.mjs
  125. 9: Dual package © 2022 Wantedly, Inc.

  126. Dual package • Export maps allow packages to provide different

    module types at once. • There is a pitfall when the package is stateful (or impure in some sense). © 2022 Wantedly, Inc.
  127. Dual package Native ESM dual package © 2022 Wantedly, Inc.

    { "name": "esm-test", "exports": { "import": "./lib.mjs", "require": "./lib.cjs", } } package.json console.log("ESM entrypoint"); lib.mjs console.log("CJS entrypoint"); lib.cjs
  128. Dual package Using import and require © 2022 Wantedly, Inc.

    import "esm-test"; app.mjs require("esm-test"); app.cjs package.json lib.mjs lib.cjs
  129. Dual package Using import and require © 2022 Wantedly, Inc.

    import "esm-test"; app.mjs require("esm-test"); app.cjs package.json lib.mjs lib.cjs $ node app.mjs ESM entrypoint $ node app.cjs CJS entrypoint
  130. Dual package Double side effects © 2022 Wantedly, Inc. import

    "esm-test"; import "./app.cjs"; app.mjs require("esm-test"); app.cjs package.json lib.mjs lib.cjs
  131. Dual package Double side effects © 2022 Wantedly, Inc. import

    "esm-test"; import "./app.cjs"; app.mjs require("esm-test"); app.cjs package.json lib.mjs lib.cjs $ node app.mjs ESM entrypoint CJS entrypoint Imagine this is state initialization. There are two different states now!
  132. 10: Modules in Webpack © 2022 Wantedly, Inc.

  133. Modules in bundlers • ESM story in Node.js gets more

    complicated when module bundlers are involved. • In this workshop, we use Webpack (version 5) as an example because it is still the most influential bundler. © 2022 Wantedly, Inc.
  134. Modules in Webpack • Webpack generally adheres to Node.js behavior.

    • One big difference: Webpack has two different ESM modes. • It complicates the matter in default export interop. © 2022 Wantedly, Inc.
  135. Autodetected ESM (Fake ESM) • Webpack has “auto detection” module

    mode. • It semantically behaves like CJS, even if the module is detected to be ESM. © 2022 Wantedly, Inc. *.cjs *.js *.mjs “type”: “commonjs” CJS CJS ESM no “type” field CJS Auto ESM “type”: “module” CJS ESM ESM In Webpack terminology: • CJS = javascript/dynamic • Auto = javascript/auto • ESM = javascript/esm
  136. Autodetected ESM (Fake ESM) • Webpack has “auto detection” module

    mode. • It semantically behaves like CJS, even if the module is detected to be ESM. © 2022 Wantedly, Inc. *.cjs *.js *.mjs “type”: “commonjs” CJS CJS ESM no “type” field CJS Auto ESM “type”: “module” CJS ESM ESM *.cjs *.js *.mjs “type”: “commonjs” CJS CJS ESM no “type” field CJS CJS ESM “type”: “module” CJS ESM ESM Recap: Node.js behavior
  137. Resolution for autodetected ESM • Special lookup rule to detect

    Webpack © 2022 Wantedly, Inc. { "module": "esm/main.js", "main": "main.js", "exports": { "module": "./esm/main.js", "default": "./main.js" } } package.json This is usually CJS This is usually an ESM module to be auto-detected by Webpack
  138. Resolution for autodetected ESM • Two types of dual package

    © 2022 Wantedly, Inc. { "exports": { "module": "./esm/main.js", "default": "./main.js" } } package.json { "exports": { "import": "./esm/main.mjs", "require": "./main.cjs" } } package.json Webpack uses ESM. Node.js always uses CJS. Webpack uses ESM. Node.js (import) uses ESM. Node.js (require) uses CJS.
  139. Default export semantics in Node.js © 2022 Wantedly, Inc. CJS

    from Babel (*.js, *.cjs) Native ESM (*.js, *.mjs) Breaks default export
  140. Default export semantics in Webpack © 2022 Wantedly, Inc. Autodetected

    ESM (*.js) CJS from Babel (*.js, *.cjs) Native ESM (*.js, *.mjs) Preserves default export Preserves default export Breaks default export
  141. Set up Webpack Install Webpack © 2022 Wantedly, Inc. {}

    package.json $ yarn add --dev webpack webpack-cli Or: npm install --dev webpack webpack-cli
  142. Set up Webpack Install Webpack © 2022 Wantedly, Inc. {

    "devDependencies": { "webpack": "^5.75.0", "webpack-cli": "^5.0.0" } } package.json (result)
  143. Set up Webpack Check import result © 2022 Wantedly, Inc.

    exports.default = 1; exports.__esModule = true; lib1.js import a from "./lib1.js"; console.log("a =", a); export default 1; lib2.js import b from "./lib1.js"; import c from "./lib2.js"; console.log("b =", b); console.log("c =", c); app.mjs
  144. Set up Webpack Check import result © 2022 Wantedly, Inc.

    exports.default = 1; exports.__esModule = true; lib1.js import a from "./lib1.js"; console.log("a =", a); export default 1; lib2.js import b from "./lib1.js"; import c from "./lib2.js"; console.log("b =", b); console.log("c =", c); app.mjs $ yarn webpack ./app.mjs --mode production --target node -o . (..snip..) webpack 5.75.0 compiled successfully in 122 ms ✨ Done in 0.50s.
  145. Set up Webpack Check import result © 2022 Wantedly, Inc.

    exports.default = 1; exports.__esModule = true; lib1.js import a from "./lib1.js"; console.log("a =", a); export default 1; lib2.js import b from "./lib1.js"; import c from "./lib2.js"; console.log("b =", b); console.log("c =", c); app.mjs $ node main.js a = 1 b = { default: 1, __esModule: true } c = 1
  146. 11: Meta APIs for modules © 2022 Wantedly, Inc.

  147. Meta APIs for modules • Node.js implements `import` and `require`

    separately. ◦ Not just the parser, but the whole module lifecycle is implemented differently. • Consequently, they expose different APIs for ESM and CJS. © 2022 Wantedly, Inc.
  148. Loader Hook: CJS • require.extensions[2] exists for this. • Isn’t

    it deprecated? Yes, but even @babel/register continues using it. • Things could easily go wrong with this API. Use the pirate[2] package for reliability. © 2022 Wantedly, Inc. [1] https://nodejs.org/api/modules.html#requireextensions [2] https://www.npmjs.com/package/pirates
  149. Loader Hook: CJS © 2022 Wantedly, Inc. const oldLoader =

    require.extensions[".js"]; require.extensions[".js"] = function(mod, filename) { const oldCompile = mod._compile; mod._compile = function(code, filename) { const newCode = `console.log(__filename);` + code; return oldCompile.call(this, newCode, filename); }; oldLoader(mod, filename); }; register.cjs
  150. Loader Hook: CJS © 2022 Wantedly, Inc. module.exports = 42;

    lib.cjs const x = require("./lib.cjs"); console.log(x); app.cjs register.cjs
  151. Loader Hook: CJS © 2022 Wantedly, Inc. module.exports = 42;

    lib.cjs const x = require("./lib.cjs"); console.log(x); app.cjs register.cjs $ node -r ./register.cjs app.cjs app.cjs lib.cjs 42
  152. Loader Hook: ESM • Conversely, Node.js has a (truly) public

    loader API[1] for ESM. • It is still experimental and subject to change. © 2022 Wantedly, Inc. [1] https://nodejs.org/api/esm.html#loaders
  153. Loader Hook: ESM © 2022 Wantedly, Inc. export async function

    load(url, context, nextLoad) { const m = await nextLoad(url, context); if (m.format !== "module") return m; if (typeof m.source !== "string") { m.source = new TextDecoder().decode(m.source); } m.source = `console.log(import.meta.url); ${m.source}`; return m; } loader.mjs
  154. Loader Hook: ESM © 2022 Wantedly, Inc. export default 42;

    lib.mjs import x from "./lib.mjs"; console.log(x); app.mjs loader.mjs
  155. Loader Hook: ESM © 2022 Wantedly, Inc. export default 42;

    lib.mjs import x from "./lib.cjs"; console.log(x); app.mjs loader.mjs $ node --experimental-loader ./loader.mjs ./app.mjs file://<dir>/app.cjs file://<dir>/lib.cjs 42
  156. Parsing modules • Want to re-implement module loader? ◦ Jest

    does this • For CJS, Function suffices. • For ESM, we need vm.Module (experimental). © 2022 Wantedly, Inc.
  157. Parsing CJS modules © 2022 Wantedly, Inc. const fs =

    require("node:fs"); const m = { exports: {} }; const f = new Function( "exports", "require", "module", "__filename", "__dirname", fs.readFileSync("lib.cjs", "utf-8")); f(m.exports, null, m, "", ""); console.log(m.exports); app.cjs
  158. Parsing CJS modules © 2022 Wantedly, Inc. app.cjs module.exports =

    42; lib.cjs
  159. Parsing CJS modules © 2022 Wantedly, Inc. app.cjs module.exports =

    42; lib.cjs $ node app.cjs 42
  160. Parsing ESM modules © 2022 Wantedly, Inc. import fs from

    "node:fs"; import vm from "node:vm"; const m = new vm.SourceTextModule( fs.readFileSync("lib.mjs", "utf-8")); await m.link(() => {}); await m.evaluate(); console.log(m.namespace.default); app.mjs
  161. Parsing ESM modules © 2022 Wantedly, Inc. app.mjs export default

    42; lib.mjs
  162. Parsing ESM modules © 2022 Wantedly, Inc. app.mjs export default

    42; lib.mjs $ node --experimental-vm-modules app.cjs 42 (node:10167) ExperimentalWarning: VM Modules is an experimental feature. This feature could change at any time
  163. Key takeaways © 2022 Wantedly, Inc.

  164. Key takeaways • There are dialects of ES Modules: ◦

    transpiled to CJS and then run ◦ fed to Node.js as-is ◦ fed to Webpack as autodected ESM • Lots of things complicate interoperation between CJS and ESM • But now your hands remember what is happening behind the scenes. Nothing to fear. © 2022 Wantedly, Inc.
  165. Appendix A package.json © 2022 Wantedly, Inc.

  166. package.json roles in Node.js © 2022 Wantedly, Inc. Fields Which

    package.json to use Module type detection type Nearest ancestor of the resolved module Package exports exports main `segment` or `@scope/segment` part of the request Folder main (only in CJS/require) main The requested directory Package imports imports Nearest ancestor of the requesting module Self name exports Nearest ancestor of the requesting module
  167. Appendix B Module polyglot © 2022 Wantedly, Inc.

  168. Recap Snippet to check module types © 2022 Wantedly, Inc.

    if (typeof await /0/["test"] === "function") { console.log("I'm ESM"); } else if (typeof require === "function") { console.log("I'm CJS"); } else { console.log("I'm other"); } module-type.js
  169. How ESM check works © 2022 Wantedly, Inc. typeof await

    /0/["test"] === "function" typeof (await (/0/["test"])) === "function" ((typeof await) / 0) / ["test"] === "function" = RegExp.prototype.test no-op variable (nonexistent) = “undefined” cast to NaN ESM: [~Yield, +Await, ~Return] CJS: [~Yield, ~Await, +Return]
  170. Advanced application (not recommended for production use!) Module polyglot ©

    2022 Wantedly, Inc. "use strict"; typeof await /[//]/; export default /* ]; module.exports = /**/ 42; module-polyglot.js