Slide 1

Slide 1 text

Gareth Rushgrove The Monorepo Storing your source code has never been so much fun

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

@garethr

Slide 4

Slide 4 text

- A recent history of code organisation - The cost of a new repo - Monorepo examples - Build tools for monorepos

Slide 5

Slide 5 text

A recent history Of code organization

Slide 6

Slide 6 text

A single team with a monolithic application in a single repository

Slide 7

Slide 7 text

Multiple teams sharing a monolithic application in a single repository

Slide 8

Slide 8 text

Multiple teams with separate applications in a single repository

Slide 9

Slide 9 text

Multiple teams with separate applications in separate repositories

Slide 10

Slide 10 text

Multiple teams with many separate applications in many separate repositories

Slide 11

Slide 11 text

Multiple teams with many separate applications microservices in many separate repositories

Slide 12

Slide 12 text

A single team with many microservice in many repositories

Slide 13

Slide 13 text

Many teams with many applications in many repositories

Slide 14

Slide 14 text

Many teams with many applications in one big Monorepo

Slide 15

Slide 15 text

Enter monorepo What if we aimed for fewer repositories

Slide 16

Slide 16 text

Monorepo: A repository which contains more than one logical project

Slide 17

Slide 17 text

In some cases the monorepo contains nearly all of the projects in a given organisation

Slide 18

Slide 18 text

The point is not that you should have a single project, but that one repository can contain multiple distinct projects David R. MacIver

Slide 19

Slide 19 text

Why

Slide 20

Slide 20 text

- Simplified organisation - Simplified dependencies - Tooling - Cross-project changes https://danluu.com/monorepo/

Slide 21

Slide 21 text

- Single lint, build, test process - Easy to coordinate changes - Single place to report issues - Easier to setup dev environment - Integration tests https://github.com/babel/babel/blob/master/doc/design/monorepo.md

Slide 22

Slide 22 text

You can make atomic refactors across the entire codebase without worrying about versioning issues Ben Maurer, Facebook

Slide 23

Slide 23 text

Embrace change

Slide 24

Slide 24 text

All of the above benefits of having one repository also extend to having fewer repositories David R. MacIver

Slide 25

Slide 25 text

The cost of a new repo Downsides of so many code repositories

Slide 26

Slide 26 text

It’s very easy to press the button

Slide 27

Slide 27 text

24 million users on GitHub have 67 million repos Octoverse 2017

Slide 28

Slide 28 text

New repos spring into existence every day without you even knowing about it. Microsoft

Slide 29

Slide 29 text

Juggling a multimodule project over multiple repos is like trying to teach a newborn baby how to ride a bike https://github.com/babel/babel/blob/master/doc/design/monorepo.md

Slide 30

Slide 30 text

If project A depends on library B, you have to encode which revision of library B you use. Ben Maurer, Facebook

Slide 31

Slide 31 text

Worse, if Project A also depends on C and D both of which use library B you have to ensure that the version of B that C and D depend on are compatible. Ben Maurer, Facebook

Slide 32

Slide 32 text

- End-to-end testing - Running whole applications locally - Is that repository used? - Implicit dependencies in CI config - High cost of onboarding Roman Heinrich, Monorepos - Why and How

Slide 33

Slide 33 text

Examples Observations from public and private repos

Slide 34

Slide 34 text

Huge companies often talk about their monorepos

Slide 35

Slide 35 text

People from huge companies often talk about their monorepos

Slide 36

Slide 36 text

Google (computer science version)

Slide 37

Slide 37 text

Google (business version)

Slide 38

Slide 38 text

Scaling Mercurial at Facebook

Slide 39

Slide 39 text

Microsoft claim the largest git repo on the planet

Slide 40

Slide 40 text

There are two sides to Google Infrastructure for Everyone Else

Slide 41

Slide 41 text

I’m more interested in Smaller teams and organisations, open source projects and widely available tools

Slide 42

Slide 42 text

Put another way Do monorepos scale down as well as scale up?

Slide 43

Slide 43 text

foresquare public monorepo

Slide 44

Slide 44 text

foursquare/fsqio 16 contributors, 15 components, 3 languages

Slide 45

Slide 45 text

At Puppet we ran a small experiment. 1 team. 6 people. 7 components. 4 languages.

Slide 46

Slide 46 text

Tools for monorepos Build tools, migration and SaaS

Slide 47

Slide 47 text

How do monorepos affect tooling?

Slide 48

Slide 48 text

Build tools

Slide 49

Slide 49 text

Most build systems start small and simple, but over time grow into hairy monsters that few dare to touch Mokhov, Mitchell, Peyton Jones, Marlow

Slide 50

Slide 50 text

Non-recursive Make considered harmful

Slide 51

Slide 51 text

Buck from Facebook

Slide 52

Slide 52 text

BUCK file java_binary( name = "hello-buck-java", main_class = "com.facebook.buck.demo.HelloBuckJava", deps = [":main"], ) java_library( name = "main", srcs = glob(["*.java"]), )

Slide 53

Slide 53 text

Pants from Twitter

Slide 54

Slide 54 text

Pants BUILD file python_library( name = 'pants', sources = globs('*.py'), dependencies = [ '3rdparty/python:pantsbuild.pants', ], )

Slide 55

Slide 55 text

Bazel from Google

Slide 56

Slide 56 text

Bazel BUILD file cc_library( name = "hello-greet", srcs = ["hello-greet.cc"], hdrs = ["hello-greet.h"], ) cc_binary( name = "hello-world", srcs = ["hello-world.cc"], deps = [ ":hello-greet", ], )

Slide 57

Slide 57 text

I Make We used Make

Slide 58

Slide 58 text

At smaller scale picking one build tool is probably more important than which one

Slide 59

Slide 59 text

Migration tools

Slide 60

Slide 60 text

You can move from multi-repos to a monorepo, without losing history or work-in-progress

Slide 61

Slide 61 text

tomono - create a monorepo from many repos

Slide 62

Slide 62 text

Maintain history from separate repos with tomono $ cat repos.txt [email protected]:mycompany/service-one.git one [email protected]:mycompany/service-two.git two [email protected]:mycompany/service-three.git three $ cat repos.txt | tomono.sh

Slide 63

Slide 63 text

You can keep your monorepo private and still engage with open source

Slide 64

Slide 64 text

split.sh - split a repository into read-only repos

Slide 65

Slide 65 text

fbshipit - copy commits between repositories

Slide 66

Slide 66 text

JavaScript Monorepos

Slide 67

Slide 67 text

Oao - Javascript monorepo management tool

Slide 68

Slide 68 text

Yarn workspaces for monorepos

Slide 69

Slide 69 text

Rush from Microsoft

Slide 70

Slide 70 text

Lerna - a tool for managing Javascript monorepos

Slide 71

Slide 71 text

SaaS products

Slide 72

Slide 72 text

GitHub code owners

Slide 73

Slide 73 text

CODEOWNERS.md # Lines starting with '#' are comments. # Each line is a file pattern followed by one or more owners. # These owners will be the default owners for everything in the repo. * @garethr # Order is important. The last matching pattern has the most precedence. # So if a pull request only touches javascript files, only these owners # will be requested to review. *.js @octocat @github/js # You can also use email addresses if you prefer. docs/* [email protected]

Slide 74

Slide 74 text

Travis CI added Build Stages

Slide 75

Slide 75 text

Explicit build matrix in Travis matrix: include: - go: 1.8 language: go install: cd consumer script: - make dependencies test - jdk: oraclejdk8 language: clojure services: docker install: cd reporter script: - make ensure-table test - python: 3.6 language: python install: cd data script: - make test - services: docker

Slide 76

Slide 76 text

Conclusions If all you remember is...

Slide 77

Slide 77 text

Monorepos are not just for Google, Facebook and Microsoft and Twitter

Slide 78

Slide 78 text

Learn monorepo and multi-repo approaches to find out what works best for you

Slide 79

Slide 79 text

No silver bullet Fred Brooks

Slide 80

Slide 80 text

I’d argue that we don’t really yet have a good handle on how to scale that area that exists at the intersection of engineering and human organization Peter Seibel, Twitter

Slide 81

Slide 81 text

Code organisation will only get more important as more people write ever more code

Slide 82

Slide 82 text

Any questions? And thanks for listening