Unlocking Lockfiles - A Developers Guide to Package Management

Find the full speaker notes on https://csi.lk/talks/unlocking-lockfiles.html

A deep dive into package management, lockfiles and how to manage them

The easiest way for me to explain package management is to go through how we got here and explain some concepts and the needs along the way

Transcript

1. UNLOCKING LOCKFILES A DEVELOPERS GUID PACKAGE MANAGEM Another CodingWithCallum ses
2. WHAT ARE W COVERING?
3. WHAT ARE W COVERING? • History
4. WHAT ARE W COVERING? • History • Fundamentals
5. HISTORY LESSON
6. HISTORY LESSON NEED (>2010
7. HISTORY LESSON NEED (>2010 • include libraries via <script>
8. HISTORY LESSON NEED (>2010 • include libraries via <script> • copypasta code from stackover
9. HISTORY LESSON NEED (>2010 an extremely embarrassing exam
10. HISTORY LESSON BIRTH (2010
11. HISTORY LESSON BIRTH (2010 • npm is created
12. HISTORY LESSON BIRTH (2010 • npm is created • Originally designed for No
13. HISTORY LESSON BIRTH (2010 • npm is created • Originally designed for No • Created standardised regis
14. HISTORY LESSON BIRTH (2010 • npm is created • Originally designed for No • Created standardised regis • package.json created
15. HISTORY LESSON BIRTH (2010 another... extremely embarrassing e
16. HISTORY LESSON BIRTH (2010
17. HISTORY LESSON BIRTH (2010 • Recursive dependency resol
18. HISTORY LESSON BIRTH (2010 • Recursive dependency resol • 🐢🐢🐢🐢
19. HISTORY LESSON BIRTH (2010 • Recursive dependency resol • 🐢🐢🐢🐢 • Nested node_modules
20. HISTORY LESSON BIRTH (2010 callums-bad-code/ ├── node_modules/ │ └── A/ │ ├── package.json │ └── node_modules/ │ └── B/ │ ├── package.json │ └── node_modules/ │ └── C/ │ └── package.json
21. HISTORY LESSON BIRTH (2010
22. HISTORY LESSON BIRTH (2010 • This was pretty revolutionary
23. HISTORY LESSON BIRTH (2010 • This was pretty revolutionary • node_modules-black-ho
24. HISTORY LESSON BIRTH (2010 • This was pretty revolutionary • node_modules-black-ho • Poor windows
25. HISTORY LESSON RISE (2012-20
26. HISTORY LESSON RISE (2012-20 • Bower / RequireJS / Browserfy / Gru
27. HISTORY LESSON RISE (2012-20 • Bower / RequireJS / Browserfy / Gru • npm gets used for frontend things al
28. HISTORY LESSON RISE (2012-20 • Bower / RequireJS / Browserfy / Gru • npm gets used for frontend things al • node_modules folder becomes tru
29. HISTORY LESSON FLAT (2016)
30. HISTORY LESSON FLAT (2016) • yarn released (from Facebook
31. HISTORY LESSON FLAT (2016) • yarn released (from Facebook ▪ lockfiles are born
32. HISTORY LESSON FLAT (2016) • yarn released (from Facebook ▪ lockfiles are born ▪ offline mode
33. HISTORY LESSON FLAT (2016) • yarn released (from Facebook ▪ lockfiles are born ▪ offline mode ▪ security (spoliers!)
34. HISTORY LESSON FLAT (2016) • yarn released (from Facebook ▪ lockfiles are born ▪ offline mode ▪ security (spoliers!) • npm creates package-lock.
35. HISTORY LESSON FLAT (2016) • yarn released (from Facebook ▪ lockfiles are born ▪ offline mode ▪ security (spoliers!) • npm creates package-lock. • flat dependency to avoid duplic
36. HISTORY LESSON FLAT (2016) node_modules/ ├── package-a/ │ └── node_modules/ │ └── package-c/ (version 1.0.0) └── package-b/ └── node_modules/ └── package-c/ (version 1.0.0)
37. HISTORY LESSON FLAT (2016) node_modules/ ├── package-a/ ├── package-b/ └── package-c/ (version 1.0.0, shared by both pa
38. HISTORY LESSON FLAT (2016) node_modules/ ├── package-a/ ├── package-b/ │ └── node_modules/ │ └── package-c/ (version 2.0.0) └── package-c/ (version 1.0.0, used by package-a
39. HISTORY LESSON FLAT (2016) yarn's new security model
40. HISTORY LESSON FLAT (2016) yarn's new security model • checksums
41. HISTORY LESSON FLAT (2016) yarn's new security model • checksums • offline mode
42. HISTORY LESSON FLAT (2016) yarn's new security model • checksums • offline mode • resolution
43. HISTORY LESSON FLAT (2016) yarn's new security model • checksums • offline mode • resolution • license checks
44. HISTORY LESSON FLAT (2016) yarn's new security model • checksums • offline mode • resolution • license checks • script execution
45. HISTORY LESSON FLAT (2016) yarn's new security model • checksums • offline mode • resolution • license checks • script execution • auditing
46. HISTORY LESSON FLAT (2016) Phantom dependencies (I ain't afraid of (Using things not listed in package Your package.json: { "dependencies": { "package- Package A's package.json: { "dependencies": { "p node_modules/ ├── package-a/ └── package-b/ (hoisted from package-a/node_mod
47. HISTORY LESSON MODERN (201
48. HISTORY LESSON MODERN (201 • pnpm created
49. HISTORY LESSON MODERN (201 • pnpm created ▪ content-addressable stor
50. HISTORY LESSON MODERN (201 • pnpm created ▪ content-addressable stor ▪ symlink all the things
51. HISTORY LESSON MODERN (201 • pnpm created ▪ content-addressable stor ▪ symlink all the things • solved "phantom" dependen
52. HISTORY LESSON MODERN (201 • pnpm created ▪ content-addressable stor ▪ symlink all the things • solved "phantom" dependen • yarn (v2 berry) releases Plug
53. HISTORY LESSON MODERN (201 • pnpm created ▪ content-addressable stor ▪ symlink all the things • solved "phantom" dependen • yarn (v2 berry) releases Plug • Deno switches to URL import
54. HISTORY LESSON MODERN (201 pnpm Content-Addressable Sto
55. HISTORY LESSON MODERN (201 pnpm Content-Addressable Sto • Global store in ~/.pnpm-stor
56. HISTORY LESSON MODERN (201 pnpm Content-Addressable Sto • Global store in ~/.pnpm-stor ▪ All packages x versions are s
57. HISTORY LESSON MODERN (201 pnpm Content-Addressable Sto • Global store in ~/.pnpm-stor ▪ All packages x versions are s • Hash-based addressing
58. HISTORY LESSON MODERN (201 pnpm Content-Addressable Sto • Global store in ~/.pnpm-stor ▪ All packages x versions are s • Hash-based addressing ▪ Unique + Free Integrity chec
59. HISTORY LESSON MODERN (201 pnpm Content-Addressable Sto • Global store in ~/.pnpm-stor ▪ All packages x versions are s • Hash-based addressing ▪ Unique + Free Integrity chec • Immutable
60. HISTORY LESSON MODERN (201 pnpm Symlinks
61. HISTORY LESSON MODERN (201 pnpm Symlinks • Symlink local to the Global
62. HISTORY LESSON MODERN (201 pnpm Symlinks • Symlink local to the Global • Strict node module structur
63. HISTORY LESSON MODERN (201 pnpm Symlinks • Symlink local to the Global • Strict node module structur • Multi-level linking
64. HISTORY LESSON MODERN (201 pnpm Symlinks • Symlink local to the Global • Strict node module structur • Multi-level linking ▪ symlinks on top of sym
65. HISTORY LESSON MODERN (201 node_modules/ ├── express -> ./.pnpm/express@4.17.1/node_modul └── .pnpm/ ├── express@4.17.1/ │ └── node_modules/ │ ├── express/ (actual link to global │ ├── body-parser -> ../../body-parser │ └── ... (other express dependencies) ├── body-parser@1.19.0/ │ └── node_modules/ │ ├── body-parser/ (actual link to gl │ └── ... (body-parser dependencies) └── ... (other packages)
66. HISTORY LESSON MODERN (201 yarn (v2 berry) Plug'n'play
67. HISTORY LESSON MODERN (201 yarn (v2 berry) Plug'n'play • Solves the same problems I just
68. HISTORY LESSON MODERN (201 yarn (v2 berry) Plug'n'play • Solves the same problems I just • Peer dependency issues
69. HISTORY LESSON MODERN (201 yarn (v2 berry) Plug'n'play • Solves the same problems I just • Peer dependency issues • Zero install
70. HISTORY LESSON TODAY (2024
71. HISTORY LESSON TODAY (2024 • Package management is everywhere
72. HISTORY LESSON TODAY (2024 • Package management is everywhere • Security concerns are now just supp attacks
73. HISTORY LESSON TODAY (2024 • Package management is everywhere • Security concerns are now just supp attacks • Monorepo management
74. HISTORY LESSON TODAY (2024 • Package management is everywhere • Security concerns are now just supp attacks • Monorepo management • The rise of bun
75. FUNDAMENTALS
76. FUNDAMENTALS SEMVER
77. FUNDAMENTALS SEMVER • Basic format: MAJOR.MINOR.PATC 2.3.1)
78. FUNDAMENTALS SEMVER • Basic format: MAJOR.MINOR.PATC 2.3.1) • MAJOR: Breaking changes
79. FUNDAMENTALS SEMVER • Basic format: MAJOR.MINOR.PATC 2.3.1) • MAJOR: Breaking changes • MINOR: New features, no breaking c
80. FUNDAMENTALS SEMVER • Basic format: MAJOR.MINOR.PATC 2.3.1) • MAJOR: Breaking changes • MINOR: New features, no breaking c • PATCH: Bug fixes, no new features or changes
81. COMMON VERS RANGES
83. COMMON VERS RANGES • Exact version: "react": "17.0.
85. COMMON VERS RANGES • Exact version: "react": "17.0. ▪ Only use exactly version 17.0.2
87. COMMON VERS RANGES • Exact version: "react": "17.0. ▪ Only use exactly version 17.0.2 • Caret (^): "react": "^17.0.2"
89. COMMON VERS RANGES • Exact version: "react": "17.0. ▪ Only use exactly version 17.0.2 • Caret (^): "react": "^17.0.2" ▪ Allow updates to any 17.x.x vers 18.0.0 (MINOR / PATCH)
91. COMMON VERS RANGES • Exact version: "react": "17.0. ▪ Only use exactly version 17.0.2 • Caret (^): "react": "^17.0.2" ▪ Allow updates to any 17.x.x vers 18.0.0 (MINOR / PATCH) • Tilde (~): "react": "~17.0.2"
93. COMMON VERS RANGES • Exact version: "react": "17.0. ▪ Only use exactly version 17.0.2 • Caret (^): "react": "^17.0.2" ▪ Allow updates to any 17.x.x vers 18.0.0 (MINOR / PATCH) • Tilde (~): "react": "~17.0.2" ▪ Allow updates to 17.0.x but not only)
95. COMMON VERS RANGES • Exact version: "react": "17.0. ▪ Only use exactly version 17.0.2 • Caret (^): "react": "^17.0.2" ▪ Allow updates to any 17.x.x vers 18.0.0 (MINOR / PATCH) • Tilde (~): "react": "~17.0.2" ▪ Allow updates to 17.0.x but not only)
96. • Wildcard (*): "react": "17.*.* "react": "17.*"
97. COMMON VERS RANGES • Exact version: "react": "17.0. ▪ Only use exactly version 17.0.2 • Caret (^): "react": "^17.0.2" ▪ Allow updates to any 17.x.x vers 18.0.0 (MINOR / PATCH) • Tilde (~): "react": "~17.0.2" ▪ Allow updates to 17.0.x but not only)
98. • Wildcard (*): "react": "17.*.* "react": "17.*" ▪ Any version starting with 17
99. UN-COMMON VERSION RANG
101. UN-COMMON VERSION RANG • Greater than (>): "react": ">17
103. UN-COMMON VERSION RANG • Greater than (>): "react": ">17 ▪ Any version higher than 17.0.0
105. UN-COMMON VERSION RANG • Greater than (>): "react": ">17 ▪ Any version higher than 17.0.0 • Greater than or equal (>=): "react ">=17.0.0"
107. UN-COMMON VERSION RANG • Greater than (>): "react": ">17 ▪ Any version higher than 17.0.0 • Greater than or equal (>=): "react ">=17.0.0" ▪ Version 17.0.0 or higher
109. UN-COMMON VERSION RANG • Greater than (>): "react": ">17 ▪ Any version higher than 17.0.0 • Greater than or equal (>=): "react ">=17.0.0" ▪ Version 17.0.0 or higher • Less than (<): "react": "<18.0.
111. UN-COMMON VERSION RANG • Greater than (>): "react": ">17 ▪ Any version higher than 17.0.0 • Greater than or equal (>=): "react ">=17.0.0" ▪ Version 17.0.0 or higher • Less than (<): "react": "<18.0. ▪ Any version lower than 18.0.0
113. UN-COMMON VERSION RANG • Greater than (>): "react": ">17 ▪ Any version higher than 17.0.0 • Greater than or equal (>=): "react ">=17.0.0" ▪ Version 17.0.0 or higher • Less than (<): "react": "<18.0. ▪ Any version lower than 18.0.0 • Range: "react": ">=16.0.0 <
115. UN-COMMON VERSION RANG • Greater than (>): "react": ">17 ▪ Any version higher than 17.0.0 • Greater than or equal (>=): "react ">=17.0.0" ▪ Version 17.0.0 or higher • Less than (<): "react": "<18.0. ▪ Any version lower than 18.0.0 • Range: "react": ">=16.0.0 <
116. ▪ Between 16.0.0 and 18.0.0 (exclu
117. UN-COMMON VERSION RANG • Greater than (>): "react": ">17 ▪ Any version higher than 17.0.0 • Greater than or equal (>=): "react ">=17.0.0" ▪ Version 17.0.0 or higher • Less than (<): "react": "<18.0. ▪ Any version lower than 18.0.0 • Range: "react": ">=16.0.0 <
118. ▪ Between 16.0.0 and 18.0.0 (exclu • OR: "react": "15.0.0 || 16
119. UN-COMMON VERSION RANG • Greater than (>): "react": ">17 ▪ Any version higher than 17.0.0 • Greater than or equal (>=): "react ">=17.0.0" ▪ Version 17.0.0 or higher • Less than (<): "react": "<18.0. ▪ Any version lower than 18.0.0 • Range: "react": ">=16.0.0 <
120. ▪ Between 16.0.0 and 18.0.0 (exclu • OR: "react": "15.0.0 || 16 ▪ Either exactly 15.0.0 or 16.0.0
121. FUNDAMENTALS LOCKFILE
122. FUNDAMENTALS LOCKFILE • Locking / Freezing your dependenci
123. FUNDAMENTALS LOCKFILE • Locking / Freezing your dependenci • Same immutable state of dependen
124. FUNDAMENTALS LOCKFILE • Locking / Freezing your dependenci • Same immutable state of dependen • Stop drift of dependencies between
125. FUNDAMENTALS UPDATING TH LOCKFILE
126. FUNDAMENTALS UPDATING TH LOCKFILE • Install dependencies within defin
127. FUNDAMENTALS UPDATING TH LOCKFILE • Install dependencies within defin ▪ "react": "^18.3.1"
128. FUNDAMENTALS UPDATING TH LOCKFILE • Install dependencies within defin ▪ "react": "^18.3.1" ◦ registry has react@18
129. FUNDAMENTALS UPDATING TH LOCKFILE • Install dependencies within defin ▪ "react": "^18.3.1" ◦ registry has react@18 • Update lockfile to point to react
130. FUNDAMENTALS UPDATING TH LOCKFILE • Install dependencies within defin ▪ "react": "^18.3.1" ◦ registry has react@18 • Update lockfile to point to react • package.json stays the same
131. FUNDAMENTALS RENOVATE
132. FUNDAMENTALS RENOVATE • Update the package.js
133. FUNDAMENTALS RENOVATE • Update the package.js ▪ Within defined range
134. FUNDAMENTALS RENOVATE • Update the package.js ▪ Within defined range • Update lockfile
135. PHEW we made it
136. QUESTIONS • How does pnpm handle workspace resolution? • You didn't go into peer dependencie them? • How is your beard not grey by now?