Reduce Build Times and Get Home Eariler

Slide 1

Slide 1 text

Reduce Build Times and Get Home Earlier Yusei Nishiyama @yuseinishiyama

Slide 2

Slide 2 text

• iOS, Android, Web • 63 countries • 17 languages • 63M+ MAU (Japan) • 35M+ MAU (Outside Japan) https://info.cookpad.com/en https://cookpad.com/en

Slide 3

Slide 3 text

Distributed Team

Slide 4

Slide 4 text

Try Error

Slide 5

Slide 5 text

Try Error

Slide 6

Slide 6 text

Let’s improve the registration screen…

Slide 7

Slide 7 text

Let’s improve the registration screen… • Implement

Slide 8

Slide 8 text

Let’s improve the registration screen… • Implement • Tweak constraints

Slide 9

Slide 9 text

Let’s improve the registration screen… • Implement • Tweak constraints • Test with older iOS versions

Slide 10

Slide 10 text

Let’s improve the registration screen… • Implement • Tweak constraints • Test with older iOS versions • Fix tests

Slide 11

Slide 11 text

Let’s improve the registration screen… • Implement • Tweak constraints • Test with older iOS versions • Fix tests • Wait for CI

Slide 12

Slide 12 text

How many times   did you build so far?

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

You are not efﬁcient

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

Slide 19

Slide 19 text

3 steps to improve • Measurement • Visualization • Improvement

Slide 20

Slide 20 text

What I will “not” talk about • Annotate all types explicitly • Write as much as code in a single ﬁle • Get your Mac Pro

Slide 21

Slide 21 text

3 steps to improve • Measurement • Visualization • Improvement

Slide 22

Slide 22 text

“Premature optimization is the root of all evil.”    — Donald Knuth —

Slide 23

Slide 23 text

Display build duration $ defaults write com.apple.dt.Xcode ShowBuildOperationDuration YES

Slide 24

Slide 24 text

Build Process • Dependency Manager (CocoaPods, Carthage) • Compile source ﬁles • Run Scripts (Linter, Formatter)

Slide 25

Slide 25 text

cookpad/dokumi • Open source • Advanced Lint tool • Comments issues on a pull request • Runs on CI server

Slide 26

Slide 26 text

Violated linter rules Failed tests

Slide 27

Slide 27 text

require 'benchmark'    def measure(*args, &block) start_time = Time.now tms = Benchmark.measure(args, &block) @data << ExecutionLog.new(args, tms, start_time) end Benchmark

Slide 28

Slide 28 text

Build Process • Dependency Manager (CocoaPods, Carthage) • Compile source ﬁles • Runs Scripts (Linter, Formatter)

Slide 29

Slide 29 text

Build Process • Dependency Manager (CocoaPods, Carthage) • Compile source ﬁles • Function A • Function B • … • Runs Scripts (Linter, Formatter)

Slide 30

Slide 30 text

Warn long function bodies

Slide 31

Slide 31 text

Time function bodies $ pbpaste | grep '.[0-9]ms' | sort -nr | head -20

Slide 32

Slide 32 text

RobertGummesson/ BuildTimeAnalyzer-for-Xcode

Slide 33

Slide 33 text

Is that data easy to manipulate? • Integrate data collection into our build process • Flexible format which doesn’t not restrict how we use it

Slide 34

Slide 34 text

giginet/xcproﬁler • A CLI tool to time body functions • Easy to integrate into build process • Flexible output (stdout, JSON, Custom)

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

require 'xcprofiler' profiler = Xcprofiler::Profiler.by_product_name('MyApp') profiler.reporters = [ Xcprofiler::StandardOutputReporter.new(limit: 20, order: :time), Xcprofiler::JSONReporter.new(output_path: 'result.json'), Xcprofiler::BlockReporter.new do |executions| do_something(executions) end, ] profiler.report! Custom Reporter

Slide 37

Slide 37 text

xactivitylog • You can access build logs under DerivedData • xactivitylog is just gzip. You can check the content as a plain text just by unzipping it

Slide 38

Slide 38 text

3 steps to improve • Measurement • Visualization • Improvement

Slide 39

Slide 39 text

CI Server cookpad/dokumi

Slide 40

Slide 40 text

CI Server cookpad/dokumi

Slide 41

Slide 41 text

CI Server cookpad/dokumi

Slide 42

Slide 42 text

CI Server cookpad/dokumi

Slide 43

Slide 43 text

CI Server cookpad/dokumi

Slide 44

Slide 44 text

InﬂuxDB • Only purpose-built platform for any time-based data • Written in Go • Built-in HTTP API • SQL-like query language • Answer queries in real-time • Rich client libraries

Slide 45

Slide 45 text

CI Server cookpad/dokumi

Slide 46

Slide 46 text

Grafana • Rich metrics dashboard and editor • Fast rendering • Support InﬂuxDB as a data source • Query editor

Slide 47

Slide 47 text

Build time visualization

Slide 48

Slide 48 text

Other metrics

Slide 49

Slide 49 text

3 steps to improve • Measurement • Visualization • Improvement

Slide 50

Slide 50 text

Test with Real Data $ cloc    github.com/AlDanial/cloc v 1.72 T=3.40 s (153.3 files/s, 16646.6 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Swift 497 8944 3122 43831 JSON 18 0 0 522 Objective C 3 37 5 166 C/C++ Header 4 19 6 27 ------------------------------------------------------------------------------- SUM: 522 9000 3133 44546 -------------------------------------------------------------------------------

Slide 51

Slide 51 text

Test with Real Data • 40,000 lines of Swift code • CocoaPods • 38 direct dependencies and 60 libraries • Linter in build phases • CI archives the app and distributes it for internal testing

Slide 52

Slide 52 text

Machine specs • CI • Mac mini (Late 2014) • 3 GHz Intel Core i7 • Memory 16GB • 256GB SSD • Local • MBP (Retina, 15-inch, Late 2013) • 2.3 GHz Intel Core i7 • Memory 16 GB • 256GB SSD

Slide 53

Slide 53 text

(Local clean build: 330s)

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

Whole Module Optimization

Slide 56

Slide 56 text

WMO without Optimization

Slide 57

Slide 57 text

(Local clean build: 160s)

Slide 58

Slide 58 text

https://swift.org/blog/whole-module-optimizations/

Slide 59

Slide 59 text

post_install do |installer| installer.pods_project.targets.each do |target| target.build_configurations.each do |config| config.build_settings['SWIFT_OPTIMIZATION_LEVEL'] = '-Owholemodule' unless config.name == 'Release' other_swift_flags = config.build_settings['OTHER_SWIFT_FLAGS'] || ['$(inherited)'] other_swift_flags << '-Onone' config.build_settings['OTHER_SWIFT_FLAGS'] = other_swift_flags end end end end

Slide 60

Slide 60 text

(Local clean build: 118s)

Slide 61

Slide 61 text

Time Function Bodies

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

post_install do |installer| installer.pods_project.targets.each do |target| target.build_configurations.each do |config| config.build_settings['SWIFT_OPTIMIZATION_LEVEL'] = '-Owholemodule' unless config.name == 'Release' other_swift_flags = config.build_settings['OTHER_SWIFT_FLAGS'] || ['$(inherited)'] other_swift_flags << '-Onone' other_swift_flags << '-Xfrontend' other_swift_flags << '-debug-time-function-bodies' config.build_settings['OTHER_SWIFT_FLAGS'] = other_swift_flags end end end end

Slide 64

Slide 64 text

(Local clean build: 130s)

Slide 65

Slide 65 text

Slide 66

Slide 66 text

Carthage + github "ReactiveX/RxSwift" ~> 3.4 + github "RxSwiftCommunity/RxRealm" + github "realm/realm-cocoa" ~> 2.4 + github "danielgindi/Charts" ~> 3.0.2 - pod 'RxSwift', '3.3.1' - pod 'RxCocoa', '3.3.1' - pod 'RxRealm', '0.5.2' - pod 'RealmSwift', '2.4.4' Cartﬁle Podﬁle

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

No content

Slide 69

Slide 69 text

(Local clean build: 68s)

Slide 70

Slide 70 text

Partial Lint lint() { while read filename; do if [[ "${filename##*.}" == "swift" ]]; then ${SWIFT_LINT} lint --path "${filename}" --quiet fi done } FORK_POINT=`git merge-base --fork-point origin/develop` git diff $FORK_POINT --name-only | lint

Slide 71

Slide 71 text

(Local clean build: 55s)

Slide 72

Slide 72 text

Do you really need to build your entire app?

Slide 73

Slide 73 text

No content

Slide 74

Slide 74 text

Playground

Slide 75

Slide 75 text

Playground

Slide 76

Slide 76 text

Real World Prototyping class ViewController: UITableViewController { let myView = MyView() override func viewDidLoad() { super.viewDidLoad() view.backgroundColor = .myColor view.addSubview(myView) } }

Slide 77

Slide 77 text

You need “existing” code for prototyping

Slide 78

Slide 78 text

Embedded Playground

Slide 79

Slide 79 text

No content

Slide 80

Slide 80 text

No content

Slide 81

Slide 81 text

No content

Slide 82

Slide 82 text

No content

Slide 83

Slide 83 text

Framework for Playground

Slide 84

Slide 84 text

No content

Slide 85

Slide 85 text

No content

Slide 86

Slide 86 text

No content

Slide 87

Slide 87 text

No content

Slide 88

Slide 88 text

No content

Slide 89

Slide 89 text

How to use it in Playground import UIKit @testable import MyPlayground let color = UIColor.myGreen

Slide 90

Slide 90 text

No content

Slide 91

Slide 91 text

Code Modularization import UIKit @testable import YourUIKit let color = UIColor.myGreen

Slide 92

Slide 92 text

Framework Oriented Architecture Feature A Design Feature B Feature C Network Persistence Core

Slide 93

Slide 93 text

Microservice Architecture https://martinfowler.com/articles/microservices.html

Slide 94

Slide 94 text

Easy to prototype

Slide 95

Slide 95 text

High testability

Slide 96

Slide 96 text

Autonomous Teams

Slide 97

Slide 97 text

Independent Technology Stack   (Swift, Objc, React Native, Flux)

Slide 98

Slide 98 text

Complex Workﬂow

Slide 99

Slide 99 text

Dynamic Frameworks make your app slow to launch

Slide 100

Slide 100 text

dylib Loading Time Total pre-main time: 4.2 seconds (100.0%) dylib loading time: 4.0 seconds (94.4%) rebase/binding time: 93.32 milliseconds (2.1%) ObjC setup time: 44.64 milliseconds (1.0%) initializer time: 98.30 milliseconds (2.3%)

Slide 101

Slide 101 text

WWDC16 406   “Optimizing App Startup Time"

Slide 102

Slide 102 text

Recap • You can get detailed build time information • Visualization helps you to identify the bottleneck • You can improve build time in many ways (WMO and Carthage are highly effective) • Modularization makes your prototyping easier

Slide 103

Slide 103 text

References • http://techlife.cookpad.com/entry/2015/06/04/dokumi-en • http://developear.com/blog/2016/12/30/Speed-Swift- Compilation.html • https://swift.org/blog/whole-module-optimizations/ • https://medium.com/@LogMaestro/adding-playgrounds-to- your-xcode-project-79d5ea0c7087 • http://frameworkoriented.io/ • https://developer.apple.com/videos/play/wwdc2016/406/

Slide 104

Slide 104 text

I’ll summarize this talk on our engineering blog  https://sourcediving.com/

Slide 105

Slide 105 text

The presentation is available on  https://speakerdeck.com/yuseinishiyama/reduce-build-times- and-get-home-eariler

Slide 106

Slide 106 text

Any Question?