Building iOS apps at scale (Mobilization)

Working with a large code base in a large distributed team involves a lot of challenges. You need to deal with complex workflows, slow build times, communications across different time zones, etc.

In this talk, Yusei will share how development teams can tackle these issues and speed up daily development. This talk will also cover the following topics:

- Workflow automation with Fastlane
- Code review with Danger and SwiftLint
- Collecting and visualizing code metrics with InfluxDB and Grafana
- Build time reduction
- Code modularization

Transcript

1. Building iOS apps at scale Yusei Nishiyama @yuseinishiyama
2. Introduction
3. • iOS, Android, Web • 67 countries • 21 languages • 60M monthly users (Japan) • 30M+ monthly users 
 (Outside Japan) https://info.cookpad.com/en https://cookpad.com/en
4. Distributed Team
5. Working at Cookpad is … • Working in a large team • Working with large code base
6. Agenda • Working in a large team • Automation with Fastlane • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
7. Agenda • Working in a large team • Automation with Fastlane • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
8. • Working in a large team • Automation with Fastlane • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
9. • Someone broke the build • We can’t release today because {name} is on vacation • Code review never finishes
10. Automation with CI • Testing • Nitpicking • Beta distribution • Release management
11. CI Server
12. CI Server
13. CI Server
14. CI Server
15. • Working in a large team • Automation with Fastlane • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
16. Fastlane in Cookpad • 7 Fastfiles • 28 lanes • 740 lines • 5 plugins
17. Sync dSYMs version = get_version_number build_number = get_build_number # Download dSYM files from iTC into a temporary directory download_dsyms(version: version, build_number: build_number) # Upload them to Crashlytics upload_symbols_to_crashlytics # Delete the temporary dSYM files clean_build_artifacts
18. Release the App Spaceship::Tunes.login app_identifier = CredentialsManager::AppfileConfig.try_fetch_value(:app_identifier) edit_version = Spaceship::Application.find(app_identifier).edit_version version_to_release = edit_version.version UI.user_error! "No pending verion to release" unless edit_version.app_status == "Pending Developer Release" edit_version.release! UI.success "Version #{version_to_release} is released " slack( channel: "#release", attachment_properties: { pretext: ":appleinc: *Cookpad iOS #{version_to_release} - Released*” }) refresh_dsyms
19. Triggers beta distribution
20. Check Labels desc "Check if pr is marked as beta distribution" private_lane :beta_distribution do labels.any? { |label| label == 'beta distribution' } end desc "Get labels of pull request" private_lane :labels do pr_number = ENV['PR_NUMBER'] repo = ENV['GITHUB_PULL_REQUEST_REPO'] result = github_api( http_method: 'GET', path: "/repos/#{repo}/issues/#{pr_number}/labels" ) result[:json].map { |label| label['name'] } end
21. Internal testers Developers Beta apps Feedback
22. Internal testers Developers v 1.0.0.1 Feedback v 1.0.0.2 v 2.0.0.0 v 2.1.0.2 Which version should I download? and what needs to be tested?
23. Append badge on icon if is_release? next elsif is_rc? title = 'RC' number = get_version_number elsif is_beta? title = 'Beta' number = sh('git rev-parse --short=7 HEAD').strip elsif is_pr? title = 'PR' number = ENV.fetch('PR_NUMBER') end badge(shield: “#{title}-#{number}-blue", dark: true, no_badge: true)
24. We are going to release a new version soon. Please check the latest RC.
25. • Working in a large team • Automation with Fastlane • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
28. CI Server
29. Danger Danger runs during your CI process, and gives teams the chance to automate common code review chores. This provides another logical step in your build, through this Danger can help lint your rote tasks in daily code review. You can use Danger to codify your teams norms. Leaving humans to think about harder problems. She does this by leaving messages inside your PRs based on rules that you create with the Ruby scripting language. Over time, as rules are adhered to, the message is amended to reflect the current state of the code review http://danger.systems/ruby/
30. Dangerfile github.dismiss_out_of_range_messages({ error: false, warning: true, message: true, markdown: true }) xcode_summary.inline_mode = true xcode_summary.ignored_files = ['Pods/**'] xcode_summary.ignored_results { |result| result.message.start_with? 'ld' } log = File.join('logs', 'test.json') xcode_summary.report log if File.exist?(log) swiftlint.binary_path = './Pods/SwiftLint/swiftlint' swiftlint.lint_files inline_mode: true
32. • Working in a large team • Automation with Fastlane • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
33. CI Server
34. InfluxDB • Only purpose-built platform for any time-based data • Written in Go • Built-in HTTP API • SQL-like query language • Answer queries in real-time • Rich client libraries
35. Post Code Coverage lane :post_coverage do rate = export_coverage if rate influxdb(table_name: table_name, values: {coverage: rate}) slack(message: "Code coverage: #{rate.to_f.round(4) * 100}%", channel: “#ios- notifications") end end private_lane :export_coverage do fastlane_require 'nokogiri' output_directory = 'coverage_report' reports = Dir.glob(File.join(ROOT_PATH, output_directory, '*.xml')) slather(use_bundle_exec: true, output_directory: output_directory) if reports.empty? nil else xml = Nokogiri::XML(File.open(reports.first)) coverage = xml.xpath('//coverage').first coverage.attr("line-rate").to_f end end
36. CI Server
37. Grafana • Rich metrics dashboard and editor • Fast rendering • Support InfluxDB as a data source • Query editor
38. Visualize Code Coverage
39. Other metrics
41. • Working in a large team • Automation with Fastlane • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
42. Let’s improve the registration screen…
43. Let’s improve the registration screen… • Implement
44. Let’s improve the registration screen… • Implement • Tweak constraints
45. Let’s improve the registration screen… • Implement • Tweak constraints • Test with older iOS versions
46. Let’s improve the registration screen… • Implement • Tweak constraints • Test with older iOS versions • Fix tests
47. Let’s improve the registration screen… • Implement • Tweak constraints • Test with older iOS versions • Fix tests • Wait for CI
48. How many times 
 did you build so far?
49. • Working in a large team • Automation with Fastlane • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
50. Reduce Build Time • Measurement • Improvement
51. What I will “not” talk about • Annotate all types explicitly • Write as much as code in a single file • Get your Mac Pro
52. Reduce Build Time • Measurement • Improvement
53. “Premature optimization is the root of all evil.”
 
 — Donald Knuth —
54. Display build duration $ defaults write com.apple.dt.Xcode ShowBuildOperationDuration YES
55. Build Process • Dependency Manager (CocoaPods, Carthage) • Compile source files • Run Scripts (Linter, Formatter)
56. fastlane/report.xml <?xml version="1.0" encoding="UTF-8"?> <testsuites> <testsuite name="fastlane.lanes"> <testcase classname="fastlane.lanes" name="00: default_platform" time="0.002078"> </testcase> <testcase classname="fastlane.lanes" name="01: bundle outdated fastlane --strict" time="8.59854"> </testcase> <testcase classname="fastlane.lanes" name="02: ensure_git_status_clean" time="0.017918"> </testcase> <testcase classname="fastlane.lanes" name="03: Switch to ios bump_build_number lane" time="0.001982"> </testcase> … <testcase classname="fastlane.lanes" name="17: hockey" time="263.575088"> </testcase> <testcase classname="fastlane.lanes" name="18: clean_build_artifacts" time="0.002326"> </testcase> </testsuite> </testsuites>
60. Build Process • Dependency Manager (CocoaPods, Carthage) • Compile source files • Run Scripts (Linter, Formatter)
61. Build Process • Dependency Manager (CocoaPods, Carthage) • Compile source files • Function A • Function B • … • Runs Scripts (Linter, Formatter)
62. Warn long function bodies
63. Time function bodies $ pbpaste | grep '.[0-9]ms' | sort -nr | head -20
64. xcactivitylog • You can access build logs under DerivedData • xcactivitylog is just gzip. You can check the content as a plain text just by unzipping it
65. giginet/danger-xcprofiler (in Dangerfile)
 
 xcprofiler.thresholds = { warn: 50, fail: 500 } xcprofiler.report ‘ProductName'
66. Reduce Build Time • Measurement • Improvement
67. Test with Real Data $ cloc
 
 github.com/AlDanial/cloc v 1.72 T=3.40 s (153.3 files/s, 16646.6 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Swift 497 8944 3122 43831 JSON 18 0 0 522 Objective C 3 37 5 166 C/C++ Header 4 19 6 27 ------------------------------------------------------------------------------- SUM: 522 9000 3133 44546 -------------------------------------------------------------------------------
68. Test with Real Data • 40,000 lines of Swift code • CocoaPods • 38 direct dependencies and 60 libraries • Linter in build phases • CI archives the app and distributes it for internal testing
69. (Local clean build: 330s)
71. Whole Module Optimization
72. WMO without Optimization
73. (Local clean build: 160s)
74. Time Function Bodies
75. F
76. Carthage + github "ReactiveX/RxSwift" ~> 3.4 + github "RxSwiftCommunity/RxRealm" + github "realm/realm-cocoa" ~> 2.4 + github "danielgindi/Charts" ~> 3.0.2 - pod 'RxSwift', '3.3.1' - pod 'RxCocoa', '3.3.1' - pod 'RxRealm', '0.5.2' - pod 'RealmSwift', '2.4.4' Cartfile Podfile
77. (Local clean build: 68s)
78. Xcode 9 New Build System
79. • Xcode 9 • Swift 3.2 • New build system • No significant improvement in build time
80. Do you really need to build your entire app?
82. Prototyping with Playground • Test your idea in Playground 
 (interface, design, reproducing a bug, etc.) • Once your idea becomes concrete, move those code into the application project
83. Playground
84. Playground
85. You need “existing” code 
 for prototyping class ViewController: UITableViewController { let myView = MyView() override func viewDidLoad() { super.viewDidLoad() view.backgroundColor = .myColor view.addSubview(myView) } }
86. Build your own UI library import UIKit @testable import YourUIKit let color = UIColor.myGreen
87. • Working in a large team • Automation with Fastlane • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
88. Microservice Architecture https://martinfowler.com/articles/microservices.html
89. Code Modularization Feature A Design Feature B Feature C Network Persistence Core
90. Benefits • Strict access control • Faster incremental build/indexing
91. Autonomous teams https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/
92. Repository Strategy • Single Repository • Multiple Repositories
93. Single repository • Simple workflow • Can’t open-source
94. Multiple Repositories • Complex workflow • Code Review Process • Easy to open-source
95. Dependency Management • CocoaPods • git submodule • Carthage • git subtree
96. CocoaPods • Easy in use • Black box in nature
97. git submodule • Simple (!= Easy) • Full control • Manual works • Complex workflow
98. Carthage • Flexible • Pre-build • Not all libraries support it • ABI stability
99. https://github.com/apple/swift-evolution#development-major-version--swift-50
100. git subtree • An alternative to git submodule • Clones a subproject and merge it into the parent project • `git subtree push` goes through the commits and picking the changes that should go to a subproject
101. git subtree
102. git subtree • Requires no change to workflow • Need to learn a new merge strategy • Contributing back to upstream is a bit tricky https://medium.com/@v/git-subtrees-a-tutorial-6ff568381844
103. Lesson Learned • Breaking existing implicit dependencies is hard • Starting from the single repository • Once modules become stable, consider • managing them in separated repositories • using pre-build versions
104. Pitfall
105. dylib Loading Time Total pre-main time: 4.2 seconds (100.0%) dylib loading time: 4.0 seconds (94.4%) rebase/binding time: 93.32 milliseconds (2.1%) ObjC setup time: 44.64 milliseconds (1.0%) initializer time: 98.30 milliseconds (2.3%)
106. Pitfall
107. Xcode 9 supports Swift Static Library https://download.developer.apple.com/Developer_Tools/Xcode_9_beta_4/ Xcode_9_beta_4_Release_Notes.pdf
108. Recap • Automate daily workflows with CI and Fastlane • Reduce workload of code review with Danger • Visualize code metrics you want to improve • Revising project settings may improve build time dramatically • Code modularization improves build/index time and helps you getting well structured code/teams
109. Thank you! Yusei Nishiyama @yuseinishiyama https://speakerdeck.com/yuseinishiyama/building-ios-apps-at-scale-mobilization