Tweet
Tweet

Slide 1

Slide 1 text

Building Android Apps At Scale and Speed Simon Stewart @shs96c Software Engineer, Facebook

Slide 2

Slide 2 text

One Guiding Principle

Slide 3

Slide 3 text

We don't want people waiting on computers <

Slide 4

Slide 4 text

We want computers waiting on people <

Slide 5

Slide 5 text

Scalable Source Organisation

Slide 6

Slide 6 text

Server Sharing Code Today Android c++ iOS

Slide 7

Slide 7 text

Server Sharing Code Today Android iOS c++ c++

Slide 8

Slide 8 text

Server Sharing Code Today Android iOS c++ c++ c++

Slide 9

Slide 9 text

Sharing Code Tomorrow fbsource c++

Slide 10

Slide 10 text

All our source in a single repository (a "monorepo")

Slide 11

Slide 11 text

Why a monorepo? Easier: code sharing, versioning, contributing Simpler: large scale changes Consistent: uniform tooling

Slide 12

Slide 12 text

Scalable Source Control

Slide 13

Slide 13 text

Mercurial >1M >100K Source control commands per day commits per week

Slide 14

Slide 14 text

Large Repos Are Slow ▪Lots of commits a day Pulls are slow Checkouts are slow ▪Years of history Clones are slow Requires lots of disk space

Slide 15

Slide 15 text

We want computers waiting on people <

Slide 16

Slide 16 text

Shallow Checkouts Clone just the latest version Work locally without complete history Need more history? Download on demand HEAD HEAD-1 HEAD-2 HEAD-3

Slide 17

Slide 17 text

Sparse Checkouts Work on just the files you need Build system integrations knows how to check out more ~/fbsource ios/... common/... ~/fbsource/.hg

Slide 18

Slide 18 text

Watchman OS independent filesystem change notification Integrates with mercurial File checking operations scale linearly with number of files changed

Slide 19

Slide 19 text

Remote Filelog Clone and pull just download metadata Download files when required Clones and pulls: 10x faster

Slide 20

Slide 20 text

Large Repos Are Fast 50 times faster than git 2000 hg improvements Up to More than

Slide 21

Slide 21 text

Scalable Builds

Slide 22

Slide 22 text

Buck ▪Next generation build system ▪Designed for modern hardware ▪SSD ▪Plentiful RAM ▪Many cores ▪Automatically parallelizes builds ▪And it's Open Source

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

Build Files java_binary( name = 'example-binary', deps = [ ':example', ], ) java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], ) Build file. Typically named BUCK

Slide 25

Slide 25 text

Build Files java_binary( name = 'example-binary', deps = [ ':example', ], ) java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], ) Build file. Typically named BUCK A build rule of type "java_binary"

Slide 26

Slide 26 text

Build Files java_binary( name = 'example-binary', deps = [ ':example', ], ) java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], ) Build file. Typically named BUCK A build rule of type "java_binary" A target defined in the same build file

Slide 27

Slide 27 text

Build Files java_binary( name = 'example-binary', deps = [ ':example', ], ) java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], ) Build file. Typically named BUCK A build rule of type "java_binary" A target defined in the same build file Build rule of type "java_library"

Slide 28

Slide 28 text

Build Files java_binary( name = 'example-binary', deps = [ ':example', ], ) java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], ) Build file. Typically named BUCK A build rule of type "java_binary" A target defined in the same build file A target defined in another build file Build rule of type "java_library"

Slide 29

Slide 29 text

Build Files java_binary( name = 'example-binary', deps = [ ':example', ], ) java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], ) Build file. Typically named BUCK A build rule of type "java_binary" A target defined in the same build file A target defined in another build file Build rule of type "java_library" All targets have visibility

Slide 30

Slide 30 text

Target and Action Graph Target Graph: a naive representation of targets defined in build files. Action Graph: an optimized graph of actions to perform as the build

Slide 31

Slide 31 text

Language-Aware Smarts public class Foo { private String someField; public void doSomething(int times) { System.out.println("Wow! " + times); } } Buck calculates the ABI, and keys rebuilds of dependencies off that Packaging rules that depend on this will always be re-run.

Slide 32

Slide 32 text

Language-Aware Smarts public class Foo { private boolean theCakeIsALie = true; public void doSomething(int times) { System.out.println("Wow! " + times); } } Private field changed: ABI remains the same as before

Slide 33

Slide 33 text

Language-Aware Smarts public class Foo { private boolean theCakeIsALie = true; public void doSomething(int num) { System.out.println("Wow! " + num); } } Parameter name changed: ABI remains the same as before.

Slide 34

Slide 34 text

Language-Aware Smarts public class Foo { private boolean theCakeIsALie = true; public void doSomething(String num) { System.out.println("Wow! " + num); } } Parameter type changed: ABI changes too!

Slide 35

Slide 35 text

Language-Aware Smarts public class Foo { private boolean theCakeIsALie = true; public void doSomething(String num) { System.err.println("Wow! " + num); } } Body of method changed: ABI remains the same.

Slide 36

Slide 36 text

Pervasive Caching • Only 3% of each build is unique. • Local and distributed caches slash build times.

Slide 37

Slide 37 text

Exopackage ▪Exopackage is an optimization for building and installing development builds of Android apps.

Slide 38

Slide 38 text

Buck Smarts Build rule knows all of the inputs that can affect its output Changes to dx and proguard for faster compilation Java ABI smarts Android resource smarts Daemon to watch file changes

Slide 39

Slide 39 text

Scalable Continuous Integration

Slide 40

Slide 40 text

<10 Minutes Feedback to Developers >1000 test results per second 5 Years machine work per day

Slide 41

Slide 41 text

Life Cycle of a Diff Write Code Test Locally Diff and review Sandcastle Land

Slide 42

Slide 42 text

Sandcastle Rapid High-Signal Frequent

Slide 43

Slide 43 text

Sandcastle Design Single Master Master Worker Worker Worker Worker

Slide 44

Slide 44 text

Sandcastle Design Single Master Distributed Queue Master Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Distributed Queue

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

Test Runs Flakiness Speed Coverage

Slide 47

Slide 47 text

Landcastle Land Target Determinator Test Test Test Push

Slide 48

Slide 48 text

We don't want people waiting on computers <

Slide 49

Slide 49 text

We want computers waiting on people <

Slide 50

Slide 50 text

Summary Tool OSS? Link! Buck Yes https://facebook.github.io/buck/ Mercurial Yes http://mercurial.selenic.com/ Watchman Yes https://facebook.github.io/watchman/ hgwatchman Yes https://bitbucket.org/facebook/hgwatchman remote filelog Yes https://bitbucket.org/facebook/remotefilelog Sandcastle No