Building iOS apps at scale

Building iOS apps at scale

Working with a large code base in a large distributed team involves a lot of challenges. You need to deal with complex workflows, slow build times, communications across different time zones, etc.

In this talk, Yusei will share how development teams can tackle these issues and speed up daily development. This talk will also cover the following topics:

- Workflow automation with Fastlane
- Code review with Danger and SwiftLint
- Collecting and visualizing code metrics with InfluxDB and Grafana
- Build time reduction
- Code modularization

562e29ba057361b2b944bd7bbd274887?s=128

Yusei Nishiyama

September 06, 2017
Tweet

Transcript

  1. Building iOS apps at scale Yusei Nishiyama @yuseinishiyama

  2. • iOS, Android, Web • 67 countries • 21 languages

    • 60M monthly users (Japan) • 30M+ monthly users 
 (Outside Japan) https://info.cookpad.com/en https://cookpad.com/en
  3. Distributed Team

  4. Working at Cookpad is … • Working in a large

    team • Working with large code base
  5. Working at Cookpad is … • Working in a large

    team • Working with a large code base
  6. • Someone broke the build • We can’t release today

    because {name} is on vacation • Code review never finishes
  7. Automation with CI • Testing • Nitpicking • Beta distribution

    • Release management
  8. CI Server

  9. CI Server

  10. CI Server

  11. CI Server

  12. None
  13. Fastlane in Cookpad • 7 Fastfiles • 28 lanes •

    740 lines • 5 plugins
  14. Sync dSYMs desc "Refresh dSYMs" lane :refresh_dsyms do version =

    get_version_number build_number = get_build_number # Download dSYM files from iTC into a temporary directory download_dsyms(version: version, build_number: build_number) # Upload them to Crashlytics upload_symbols_to_crashlytics # Delete the temporary dSYM files clean_build_artifacts end
  15. Release the App Spaceship::Tunes.login app_identifier = CredentialsManager::AppfileConfig.try_fetch_value(:app_identifier) edit_version = Spaceship::Application.find(app_identifier).edit_version

    version_to_release = edit_version.version UI.user_error! "No pending verion to release" unless edit_version.app_status == "Pending Developer Release" edit_version.release! UI.success "Version #{version_to_release} is released " slack( channel: "#release", attachment_properties: { pretext: ":appleinc: *Cookpad iOS #{version_to_release} - Released*” }) refresh_dsyms
  16. Triggers beta distribution

  17. Check Labels desc "Check if pr is marked as beta

    distribution" private_lane :beta_distribution do labels.any? { |label| label == 'beta distribution' } end desc "Get labels of pull request" private_lane :labels do pr_number = ENV['PR_NUMBER'] repo = ENV['GITHUB_PULL_REQUEST_REPO'] result = github_api( http_method: 'GET', path: "/repos/#{repo}/issues/#{pr_number}/labels" ) result[:json].map { |label| label['name'] } end
  18. Notify a New Build desc "Notify new beta distribution" private_lane

    :notify_build do messages = "A new version is available" pr_number = ENV['PR_NUMBER'] repo = ENV['GITHUB_PULL_REQUEST_REPO'] github_api( http_method: 'POST', path: "/repos/#{repo}/issues/#{pr_number}/comments", body: { "body": message } ) end
  19. None
  20. None
  21. None
  22. CI Server

  23. Danger Danger runs during your CI process, and gives teams

    the chance to automate common code review chores. This provides another logical step in your build, through this Danger can help lint your rote tasks in daily code review. You can use Danger to codify your teams norms. Leaving humans to think about harder problems. She does this by leaving messages inside your PRs based on rules that you create with the Ruby scripting language. Over time, as rules are adhered to, the message is amended to reflect the current state of the code review http://danger.systems/ruby/
  24. Dangerfile github.dismiss_out_of_range_messages({ error: false, warning: true, message: true, markdown: true

    }) xcode_summary.inline_mode = true xcode_summary.ignored_files = ['Pods/**'] xcode_summary.ignored_results { |result| result.message.start_with? 'ld' } log = File.join('logs', 'test.json') xcode_summary.report log if File.exist?(log) swiftlint.binary_path = './Pods/SwiftLint/swiftlint' swiftlint.lint_files inline_mode: true
  25. None
  26. CI Server

  27. InfluxDB • Only purpose-built platform for any time-based data •

    Written in Go • Built-in HTTP API • SQL-like query language • Answer queries in real-time • Rich client libraries
  28. Post Code Coverage lane :post_coverage do rate = export_coverage if

    rate influxdb(table_name: table_name, values: {coverage: rate}) slack(message: "Code coverage: #{rate.to_f.round(4) * 100}%", channel: “#ios- notifications") end end private_lane :export_coverage do fastlane_require 'nokogiri' output_directory = 'coverage_report' reports = Dir.glob(File.join(ROOT_PATH, output_directory, '*.xml')) slather(use_bundle_exec: true, output_directory: output_directory) if reports.empty? nil else xml = Nokogiri::XML(File.open(reports.first)) coverage = xml.xpath('//coverage').first coverage.attr("line-rate").to_f end end
  29. CI Server

  30. Grafana • Rich metrics dashboard and editor • Fast rendering

    • Support InfluxDB as a data source • Query editor
  31. Visualize Code Coverage

  32. Other metrics

  33. Working at Cookpad is … • Working in a large

    team • Working with a large code base
  34. Let’s improve the registration screen…

  35. Let’s improve the registration screen… • Implement

  36. Let’s improve the registration screen… • Implement • Tweak constraints

  37. Let’s improve the registration screen… • Implement • Tweak constraints

    • Test with older iOS versions
  38. Let’s improve the registration screen… • Implement • Tweak constraints

    • Test with older iOS versions • Fix tests
  39. Let’s improve the registration screen… • Implement • Tweak constraints

    • Test with older iOS versions • Fix tests • Wait for CI
  40. How many times 
 did you build so far?

  41. Reduce Build Time • Measurement • Improvement

  42. What I will “not” talk about • Annotate all types

    explicitly • Write as much as code in a single file • Get your Mac Pro
  43. Reduce Build Time • Measurement • Improvement

  44. “Premature optimization is the root of all evil.”
 
 —

    Donald Knuth —
  45. Display build duration $ defaults write com.apple.dt.Xcode ShowBuildOperationDuration YES

  46. Build Process • Dependency Manager (CocoaPods, Carthage) • Compile source

    files • Run Scripts (Linter, Formatter)
  47. fastlane/report.xml <?xml version="1.0" encoding="UTF-8"?> <testsuites> <testsuite name="fastlane.lanes"> <testcase classname="fastlane.lanes" name="00:

    default_platform" time="0.002078"> </testcase> <testcase classname="fastlane.lanes" name="01: bundle outdated fastlane --strict" time="8.59854"> </testcase> <testcase classname="fastlane.lanes" name="02: ensure_git_status_clean" time="0.017918"> </testcase> <testcase classname="fastlane.lanes" name="03: Switch to ios bump_build_number lane" time="0.001982"> </testcase> … <testcase classname="fastlane.lanes" name="17: hockey" time="263.575088"> </testcase> <testcase classname="fastlane.lanes" name="18: clean_build_artifacts" time="0.002326"> </testcase> </testsuite> </testsuites>
  48. None
  49. None
  50. None
  51. Build Process • Dependency Manager (CocoaPods, Carthage) • Compile source

    files • Run Scripts (Linter, Formatter)
  52. Build Process • Dependency Manager (CocoaPods, Carthage) • Compile source

    files • Function A • Function B • … • Runs Scripts (Linter, Formatter)
  53. Warn long function bodies

  54. Time function bodies $ pbpaste | grep '.[0-9]ms' | sort

    -nr | head -20
  55. xcactivitylog • You can access build logs under DerivedData •

    xcactivitylog is just gzip. You can check the content as a plain text just by unzipping it
  56. giginet/danger-xcprofiler (in Dangerfile)
 
 xcprofiler.thresholds = { warn: 50, fail:

    500 } xcprofiler.report ‘ProductName'
  57. Reduce Build Time • Measurement • Improvement

  58. Test with Real Data $ cloc
 
 github.com/AlDanial/cloc v 1.72

    T=3.40 s (153.3 files/s, 16646.6 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Swift 497 8944 3122 43831 JSON 18 0 0 522 Objective C 3 37 5 166 C/C++ Header 4 19 6 27 ------------------------------------------------------------------------------- SUM: 522 9000 3133 44546 -------------------------------------------------------------------------------
  59. Test with Real Data • 40,000 lines of Swift code

    • CocoaPods • 38 direct dependencies and 60 libraries • Linter in build phases • CI archives the app and distributes it for internal testing
  60. (Local clean build: 330s)

  61. None
  62. Whole Module Optimization

  63. WMO without Optimization

  64. (Local clean build: 160s)

  65. https://swift.org/blog/whole-module-optimizations/

  66. Time Function Bodies

  67. F

  68. Carthage + github "ReactiveX/RxSwift" ~> 3.4 + github "RxSwiftCommunity/RxRealm" +

    github "realm/realm-cocoa" ~> 2.4 + github "danielgindi/Charts" ~> 3.0.2 - pod 'RxSwift', '3.3.1' - pod 'RxCocoa', '3.3.1' - pod 'RxRealm', '0.5.2' - pod 'RealmSwift', '2.4.4' Cartfile Podfile
  69. (Local clean build: 68s)

  70. Do you really need to build your entire app?

  71. None
  72. Prototyping with Playground • Test your idea in Playground 


    (interface, design, reproducing a bug, etc.) • Once your idea becomes concrete, move those code into the application project
  73. Playground

  74. Playground

  75. You need “existing” code 
 for prototyping class ViewController: UITableViewController

    { let myView = MyView() override func viewDidLoad() { super.viewDidLoad() view.backgroundColor = .myColor view.addSubview(myView) } }
  76. Code Modularization import UIKit @testable import YourUIKit let color =

    UIColor.myGreen
  77. Code Modularization Feature A Design Feature B Feature C Network

    Persistence Core
  78. Microservice Architecture https://martinfowler.com/articles/microservices.html

  79. Benefits • Strict access control • Faster incremental build/indexing •

    Clear separation
  80. Repository Strategy • Single Repository • Multiple Repositories

  81. Single repository • Simple workflow • Can’t open-source

  82. Multiple Repositories • Complex workflow • Code Review Process •

    Easy to open-source
  83. Dependency Management • CocoaPods • git submodule • Carthage •

    git subtree
  84. CocoaPods • Easy in use • Black box in nature

  85. git submodule • Simple (!= Easy) • Full control •

    Manual works • Complex workflow
  86. Carthage • Flexible • Pre-build • Not all libraries support

    it • ABI stability
  87. https://github.com/apple/swift-evolution#development-major-version--swift-50

  88. git subtree • An alternative to git submodule • Clones

    a subproject and merge it into the parent project • `git subtree push` goes through the commits and picking the changes that should go to a subproject
  89. git subtree

  90. git subtree • Requires no change to workflow • Need

    to learn a new merge strategy • Contributing back to upstream is a bit tricky
  91. Lesson Learned • Breaking existing implicit dependencies is hard •

    Starting from the single repository • Once modules become stable, consider • managing them in separated repositories • using pre-build versions
  92. Case Study: Single repo + CocoaPods

  93. Directory Structure . "## MainApp $ "## MainApp $ $

    "## AppDelegate.swift $ $ "## Info.plist $ $ &## ViewController.swift $ "## MainApp.xcodeproj $ &## MainAppTests $ "## Info.plist $ &## MainAppTests.swift "## Library $ "## Library $ $ "## Info.plist $ $ "## Library.h $ $ &## Library.swift $ "## Library.xcodeproj $ &## LibraryTests $ "## Info.plist $ &## LibraryTests.swift "## MainApp.xcworkspace "## Podfile "## Podfile.lock "## Pods &## README.md
  94. Podfile source 'https://github.com/CocoaPods/Specs.git' use_frameworks! platform :ios, '9.3' workspace 'MainApp.xcworkspace' target

    'MainApp' do project 'MainApp/MainApp.xcodeproj' # Dependencies of the MainApp end target 'Library' do project 'Library/Library.xcodeproj' # Dependencies of the Library end
  95. Workspace Structure

  96. Link it!

  97. Find Implicit Dependencies

  98. Pitfall

  99. dylib Loading Time Total pre-main time: 4.2 seconds (100.0%) dylib

    loading time: 4.0 seconds (94.4%) rebase/binding time: 93.32 milliseconds (2.1%) ObjC setup time: 44.64 milliseconds (1.0%) initializer time: 98.30 milliseconds (2.3%)
  100. Pitfall

  101. Xcode 9 supports Swift Static Library https://download.developer.apple.com/Developer_Tools/Xcode_9_beta_4/ Xcode_9_beta_4_Release_Notes.pdf

  102. Recap • Automate daily workflows with CI and Fastlane •

    Reduce workload of code review with Danger • Visualize code metrics you want to improve • Revising project settings may improve build time dramatically • Code modularization improves build/index time and helps you getting well structured code/teams
  103. The presentation is available on
 https://speakerdeck.com/yuseinishiyama/building-ios-apps-at-scale

  104. Thank you! Yusei Nishiyama @yuseinishiyama