Monorepos - Why and How

Introductory presentation about tooling around monorepos presented at the Elixir Meetup Berlin. Rough overview about Bazel build, Buck build, Pants and Please build.

Transcript

1. Monorepos Why and How presented at Elixir Berlin Meetup on 2017/07/13 @ Blacklane
2. Me • Roman Heinrich (@mindreframer on all social accounts) • Ruby / Devops / React / Elixir / Go dev
3. Disclaimer • This talk does not have any special relationship to Elixir • But it will (hopefully) broaden your horizon a bit and might help in future decisions
4. What • What is a monorepo and how do you eat it?
5. Mono != Monolith • Most startups start with a monolithic implementation • By definition also a “monorepo” • The most pragmatic decision on start and great for quick shipping
6. What • Some stay with monolithic apps for the life of the product / company • This is not that bad, when approached carefully and with good modularisation in mind
7. What • Multi-repo discussion arises when SOA / microservices enter the field • Usually it looks reasonable to stick with 1 service -> 1 repo approach • This is familiar and we know how to deal with basic requirements for simple projects
8. What • But things start be become quite tricky down the road • It becomes hard / impossible to test applications end-2- end • Running the “application” locally is very hard, when you have to work against a pinned versions of other services
9. What • The number of repositories explodes and it feels like herding cats • Impossible to tell whether something is used or not • So we just keep adding them, because we are afraid to break something… • Onboarding of new colleagues takes a lot of time • Many other non-trivial problems…
10. What • Dependencies are implicit or are encoded in the CI - pipelines • Sometimes you might have a diagram of the overall architecture that is updated occasionally and is just a MAP, but not the real thing (might be incorrect without noticing it)
11. Magic Trick • Explicit dependency graph (even cross-languages) • Programmatically accessible and when used properly very-very helpful • Content dependent hashing, not related to GIT SHAs or similar things
12. Dependency Graph • This makes your build predictable and very fast • It allows you to statically analyse dependencies and do awesome things with it
13. Alternative • Single big repository + special tooling
14. BIG 3 • Google (https://bazel.build/) • Facebook (https://buckbuild.com/) • Twitter (http://www.pantsbuild.org/)
15. BAZEL • In Java • Stripe • Dropbox • Pinterest • Improbable • Google
16. BUCK • also Java • https://buckbuild.com/concept/what_makes_buck_so_fast.html
17. PANTS • Mostly Python • V2 will be in Rust! • http://www.pantsbuild.org/powered_by.html • https://github.com/pantsbuild/pants/tree/master/src/rust/engine
18. Origin • The idea for a huge monorepo originated from Google and was popularised by Ex-Googlers, when they joined other companies
19. GOOGLE • https://arxiv.org/pdf/1702.01715.pdf - Software Engineering at Google (2017/01) • Most of Google’s code is stored in a single unified source- code repository, and is accessible to all software engineers at Google • Exception: some high-value or security-critical pieces of code for which read access is locked down more tightly
20. GOOGLE • As of January 2015, this 86 terabyte repository contained a billion files, including over 9 million source code files containing a total of 2 billion lines of source code, with a history of 35 million commits and a change rate of 40 thousand commits per work day
21. Example BUILD file • http://www.pantsbuild.org/build_files.html
22. PLZ build • https://please.build/quickstart.html • New contender • Also from ex-Googlers • By thoughtmachine.net • Vault OS - An Operating System for Banks
23. PLZ build • A single Go binary
24. PLZ build
25. PLZ build
26. WHY USE PLEASE INSTEAD OF BAZEL, BUCK OR PANTS? • All four of these systems are quite closely related in the scheme of things, being inspired by (or in Bazel's case, a direct open sourcing of) Google's Blaze. • Several of us had worked at Google and used Blaze extensively there; we were excited about it being open sourced as Bazel but by then we were already using Please internally. It's a great system but we have slightly different goals, specifically we're aiming Please at being lighter weight and pushing the boundaries of what can be done within the BUILD language. Since Please is written in Go there's no runtime dependency on the JVM.
27. PLZ Example • https://github.com/mindreframer/please-go-example
28. Demo • https://github.com/mindreframer/please-go-example
29. Links • https://github.com/gophersgang/go-kitchensink/blob/ master/docs/monorepo.md
30. Monorepo - Pros • Very scalable approach people / code base wise • You probably won't find any hard blockers here, since it was proven to work for companies like Facebook / Twitter / Google • It is easier to contribute across teams because you work on the same physical codebase • Predictable fast builds, high team morale
31. Monorepo - Cons • Requires an all-in decision • Requires quite some tooling to become productive • Some operations for daily work (GIT / DVCS) might be slower than with smaller codebases • Not much experience in real life if you did not work for the couple bigger companies using it
32. Fin Thank you for listening! Q & A