Developing in a Large Monorepo

In Microsoft, we have a very large TypeScript-based git repository where over 250 developers build and write code for high-value frontend components which is used across all Microsoft365 products like Outlook, Office, Bing, SharePoint, etc.It contains over a 100 npm packages, containing over a million lines of TypeScript code. Co-locating these components encourages collaboration and sharing code across teams very easier.

In this talk, we’ll focus on the tooling used and code organization to make the development easier, fast and reliable and will not discuss in detail unit tests and automation for a better focus on development.

Transcript

1. JSConf Korea 2020 Developing in a Large Monorepo
2. Jai Santhosh Engineering Manager @jaisanth
3. Microsoft Search
4. Re-using Components They are the user’s most important, cross-app and cross-platform interactions in the Office Suite. High-Value Experiences These experiences must feel familiar across apps and platforms, but also look and feel consistent within a platform and within an app. Coherence
5. People Card
6. People Card
7. People Card Card Conversations Documents Events Organization Analytics Contact
8. Write new code Test locally Create a PR Receive Peer Review Verify build and test reports Merge PR with main Checkin new code Deploy through rings to production Listen and measure for feedback Propose next steps Write new code Agility Inner Loop Outer Loop
9. Web Win32 UWP iOS Mac Android src src src src src src Agility
10. Web Win32 UWP iOS Mac Android src Agility
11. Agility & Coherence
12. Monorepo
13. What is a Monorepo? A repository with multiple packages or projects • Other repositories can re-use small parts of our code • Better code re-use unit (better tooling) Singularity across multiple packages • Developer agility • Code consistency as conventions are shared • Unified processes, versioned centrally • Code discoverability • ... Why in one repository? • Various projects or packages can co-exist. • Micro platform with an entire application stack in one place. Not necessarily a monolith
14. Using separate packages Re-using can be made easier with singular responsibility packages. Can be published easily. Enables cross-team usage of required packages. Can manage releases easily.
15. You need reliable tooling! Package Management Linters Build system / Infrastructure Testing Tools Cross-package orchestrators and more… Package Management Cross-package orchestrators
16. Package Management
17. JavaScript package JavaScript Repo Repo’s external dependencies
18. package.json NPM Yarn Files on disk NodeJS require() is read by which writes is resolved by which finds
19. require("react") • /User/../git/Monorepo/packages/people-core/node_modules/react • /User/../git/Monorepo/packages/node_modules/react • /User/../git/Monorepo/node_modules/react • /User/../git/node_modules/react • /User/../node_modules/react • /node_modules/react Node Module Resolution
20. package-1 package-2 Repository 1 Repository 2
21. package-1 package-2 Repository mono monorepo
22. package-1 package-2 Repository mono monorepo A (1.0) C (1.0) B (1.0) B (2.0) B (1.0)
23. package-1 package-2 monorepo A (1.0) C (1.0) B (1.0) B (2.0) node_modules packages package-1 package-2 node_modules symlinks https://yarnpkg.com/blog/2018/02/15/nohoist/ Package Hoisting
24. rush yarn workspaces pnpm workspaces Lerna Package Management - Tools Multiple popular tools for package management, let’s check how yarn workspaces works.
25. • Set "private": true • Add workspace packages • Optionally, you can add a nohoist list of packages as well • Run yarn from the root of the monorepo. • Package installation time for about 150+ packages came down from ~20mins to ~6mins. monorepo/package.json Yarn Workspaces
26. Cross-package Orchestration
27. rush Lerna Shell scripts Gulp Cross-package Orchestration - Tools Perform operations on files across multiple packages, usually from the root folder
28. monorepo/lerna.json • Add "packages": [<<paths>>] • Specify npmClient • Set useWorkspaces value. • lerna run [script] - Run the script across packages Lerna
29. Lerna - Example
30. Lerna - Example
31. • We use test dev apps for each of the experiences for quick dev validation. • lerna run --scope [<packages>] [script] Lerna - Scoping
32. • lerna publish - create new release versions of packages that have been updated. • lerna diff [package?] - Diff package(s) since the last release. • lerna changed – Check which packages have been changed since last release. • lerna version – Bump version of packages changed since the last release. Lerna - Versioning
33. • Brought down our build times by more than 5x. • Good community adoption • babeljs • create-react-app • react-router • jest • and many more... Lerna - Wins
34. Summing it up… • Pick the right tools for your monorepo, based on the complexity of the packages, to improve the developer experience. • For linting, unit/functional/perf testing you can combine it with tools like lerna to easily run these commands across packages. • If you are using TypeScript, you will have to handle TS compilation and for a better developer experience, manage typings for better Intellisense.
35. References https://medium.com/@vincentbailly/evolving-a-large-typescript-repository-4a32c61e94fd https://rushjs.io/ https://lerna.js.org/ https://yarnpkg.com/lang/en/docs/workspaces/ Image courtesy: @redbadgerteam, Adobe Stock images
36. Thank you! Jai Santhosh Engineering Manager @jaisanth