lerna - migrate your package zoo to a monorepo

Do you have a zoo of packages? Living the pre-release-testing-maintenance hell? Move everything to a single repository using lernaJS and have more time to things that matter.

Transcript

1. lerna migrate your package zoo to a monorepo lerna - monorepo | @arndissler 1
2. Do you have a zoo of packages? Living the pre-release-testing-maintenance hell? Move everything to a single repository using lernaJS and have more time to things that matter. lerna - monorepo | @arndissler 2
3. hi, I'm arnd lerna - monorepo | @arndissler 3
4. I'm working as a developer at SinnerSchrader lerna - monorepo | @arndissler 4
5. get started lerna - monorepo | @arndissler 5
6. the current situation application A uses package B package B uses package C lerna - monorepo | @arndissler 6
7. imagine the following Every package is checked out in the correct version & branches Bug in package C is well documented, but complex. Feature in package B should be just a small change. lerna - monorepo | @arndissler 7
8. imagine the following Every package is checked out in the correct version & branches Bug in package C is well documented, but complex. Feature in package B should be just a small change. easy doin' lerna - monorepo | @arndissler 8
9. so let's see how we get this done lerna - monorepo | @arndissler 9
10. our masterplan fix bug in package C code feature in package B build the application lerna - monorepo | @arndissler 10
11. done lerna - monorepo | @arndissler 11
12. the workflow start bug fixing in package C • fix problems in package C, commit code • run tests, formatter & linter • push the code, file code reviews for package C • increment version number, build pre-release • push to registry, QA starts manual testing lerna - monorepo | @arndissler 12
13. the workflow start feature development in package B • npm install package-C-pre-release • develop the feature • run tests, formatter & linter • push the code, file code reviews for package B • increment version number, build pre-release • push to registry, QA starts manual testing lerna - monorepo | @arndissler 13
14. the workflow build testable pre-release application • npm install package-B-pre-release • maybe some coding is needed • run tests, formatter & linter • push the code, file code reviews for app A • increment version number, build pre-release • push to registry, QA starts manual testing • guess what happens next? lerna - monorepo | @arndissler 14
15. QA reports a bug in package-C-prerelease lerna - monorepo | @arndissler 15
16. lerna - monorepo | @arndissler 16
17. what the workflow really means lerna - monorepo | @arndissler 17
18. "[...] the waterfall model maintains that one should move to a phase only when its preceding phase is reviewed and verified." https://en.wikipedia.org/wiki/Waterfall_model lerna - monorepo | @arndissler 18
19. only three packages lerna - monorepo | @arndissler 19
20. but we have twenty+ packages lerna - monorepo | @arndissler 20
21. what we need { some tooling that keeps all changes in one „bracket“ to build, test, fix, review, merge and deploy the whole thing } lerna - monorepo | @arndissler 21
22. possible solutions • a bunch of shell scripts • faster reviews & tests • awesome superpowers, e.g. • write bug-free code • everything in one repository monolithic repository lerna - monorepo | @arndissler 22
23. everything in one repository monorepo to the rescue lerna - monorepo | @arndissler 23
24. what is a monorepo? { A mono repo is a folder structure, where several npm packages are maintained within one git repository. } lerna - monorepo | @arndissler 24
25. monorepo sound good how do we handle this? { There are tools for everything. } lerna - monorepo | @arndissler 25
26. tools for handling monorepos { • shell scripts • builder • lerna } lerna - monorepo | @arndissler 26
27. what is builder { Builder is a meta tool for controlling npm workflows. If you’re working on multiple similar repos, it allows you to define tasks and dependencies in a single shared source of truth. } lerna - monorepo | @arndissler 27
28. what is lerna { Lerna is a tool that optimizes the workflow around managing multi-package repositories with git and npm (or yarn) } lerna - monorepo | @arndissler 28
29. who is using lerna? babel & jest lerna - monorepo | @arndissler 29
30. what about the project structure? lerna - monorepo | @arndissler 30
31. the lerna package structure monorepo/ package.json lerna.json packages/ package-A/ package.json package-B/ package.json lerna - monorepo | @arndissler 31
32. working with the monorepo tooling lerna - monorepo | @arndissler 32
33. the most important things (an overview) lerna - monorepo | @arndissler 33
34. what can lerna do for us { • script management • package linking • version number management • publishing } lerna - monorepo | @arndissler 34
35. running scripts lerna - monorepo | @arndissler 35
36. run scripts across packages run script format in all packages that supports it: ! lerna run format lerna - monorepo | @arndissler 36
37. run a script in specific repos run script generate only in package-c: ! lerna run generate --scope=package-c -- --interfaces Example for running code generation script for generating interfaces. --scope is a global parameter that accept a globbing pattern for package names lerna - monorepo | @arndissler 37
38. adding new dependencies to add new dependency, simply run ! lerna add <dependecy> this will install dependency in all packages --scope is respected lerna - monorepo | @arndissler 38
39. cleaning up to prune all node_modules, just run ! lerna clean or ! lerna exec -- rm -rf ./node_modules lerna - monorepo | @arndissler 39
40. what's up(dated) in large repos, it's always good to know what packages are affected: ! lerna updated lerna info version 2.10.2 lerna info Checking for updated packages... lerna info Comparing with v1.0.1. lerna info Checking for prereleased packages... lerna info result - package-c lerna - monorepo | @arndissler 40
41. package linking lerna - monorepo | @arndissler 41
42. install all the things to install all dependencies, just run ! lerna bootstrap • symlinks all internal packages within the monorepo • installs external dependencies • run prepublish and prepare on the packages lerna - monorepo | @arndissler 42
43. versioning & publishing lerna - monorepo | @arndissler 43
44. versioning modes standard one version number for all packages in the repo independent every package has it's own version number lerna - monorepo | @arndissler 44
45. versioning & publishing ! lerna publish will do the following for us • increment version key in lerna.json (if needed) • update the package.json for each package • create git commit & tag with the new version • publish everything on npm lerna - monorepo | @arndissler 45
46. versionize all the things to just set the version number ! lerna publish --skip-git --skip-npm ? Select a new version (currently 1.0.1) (Use arrow keys) › Patch (1.0.2) Minor (1.1.0) Major (2.0.0) Prepatch (1.0.2-0) Preminor (1.1.0-0) Premajor (2.0.0-0) Prerelease Custom lerna - monorepo | @arndissler 46
47. versionize all the things lerna can also use semantic versioning ! lerna publish --skip-npm --skip-git --cd-version minor lerna info version 2.10.2 lerna info current version 1.0.1 lerna info Checking for updated packages... lerna info Comparing with v1.0.1. lerna info Checking for prereleased packages... Changes: - package-c: 1.0.1 => 1.1.0 lerna - monorepo | @arndissler 47
48. publish all the things lerna can determine the version number by conventional commits lerna publish --conventional-commits • for a fix, it's like a semver patch version bump • for feature, it's like a semver minor version bump • if it's breaking change, it's like a semver major version bump lerna - monorepo | @arndissler 48
49. publish all the things to build a pre-release, just run ! lerna publish --canary • basically the same as publish • except the version number is bumped to the next minor version, a meta suffix and the current commit hash, e.g. 1.7.2-alpha.ba751a0e lerna - monorepo | @arndissler 49
50. woah how do we get there lerna - monorepo | @arndissler 50
51. migrate to monorepo starting from scratch to create a new package: mkdir -p packages/new-package cd packages/new-package npm init lerna - monorepo | @arndissler 51
52. migrate to monorepo import repositories to import another repository as a package lerna import ../path/to/repo be warned • it takes time & will consume massive memory • maybe it won't work (at least for long running projects) lerna - monorepo | @arndissler 52
53. migrate to monorepo tmp="$(pwd)" git filter-branch --index-filter ' git ls-files -s | gsed "s,\t,&'"packages/package-to-import"'/," | GIT_INDEX_FILE="$GIT_INDEX_FILE.new" git update-index --index-info && mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE" ' HEAD cd "$path" git remote add -f "remoteImportedPackage" "file://$tmp/.git" git fetch "remoteImportedPackage" git merge --allow-unrelated-histories -m \ "Merge repo package-to-import into master" \ --no-edit "remoteImportedPackage/master" git remote remove "remoteImportedPackage" lerna - monorepo | @arndissler 53
54. migrate to monorepo time range: month before migration • harmonize the npm scripts • harmonize dependency versions • adjust versioning tools (e.g. standard-version) • write documentation -- FAQ how to get things done -- give examples • document workflow changes lerna - monorepo | @arndissler 54
55. but the most important thing lerna - monorepo | @arndissler 55
56. talk to the people lerna - monorepo | @arndissler 56
57. migrate to monorepo talk to the people • talk to the team/teams, they know about their edge cases • take their concerns seriously • involve them in the process • keep your changes transparent • don't overrule team decisions lerna - monorepo | @arndissler 57
58. migrate to monorepo time range: weeks before migration • plan the downtime in product/package teams • setup your CI environment, check permissions • make test runs • plan a rollback strategy • spot the elephant in the room • automate as much as possible lerna - monorepo | @arndissler 58
59. migrate to monorepo time range: days before migration • write reminders to the teams • talk to the people • plan assistance on solving problems lerna - monorepo | @arndissler 59
60. migrate to monorepo time range: liftoff • make sure everyone saved and pushed changes • kickoff the migration script • set all migrated repos to 'read only' • push the monorepo • start the CI pipelines • manual test, if needed lerna - monorepo | @arndissler 60
61. migrate to monorepo time range: right after liftoff • make sure, if the repo is accessible • create a branch, change & push something • inform the teams about the success • say THANK YOU lerna - monorepo | @arndissler 61
62. but... when there is light there must be darkness lerna - monorepo | @arndissler 62
63. the downsides It's going to be LARGE in several ways • number of total stored bytes • number of commits • number of tracked files • number of branches • number of tags • number of tests lerna - monorepo | @arndissler 63
64. the consequences of large repos • cloning the repo will slow down • switching branches may slow down • tracking changed files may slow down • build times may increase • CI may need more resources • it's more likely that you need low-level git commands • your editor may not work anymore lerna - monorepo | @arndissler 64
65. how to keep it maintainable • avoid long running branches & remove branches when they are merged • avoid large files • document everything, even worse decisions • write a bullet proof workflow documentation • consider split the repo (again) into separate, logical units • ask yourself and the teams, if you're going in the right direction lerna - monorepo | @arndissler 65
66. Thanks for listening @arndissler lerna - monorepo | @arndissler 66