Pro Yearly is on sale from $80 to $50! »

Building iOS apps at scale (Mobilization)

Building iOS apps at scale (Mobilization)

Working with a large code base in a large distributed team involves a lot of challenges. You need to deal with complex workflows, slow build times, communications across different time zones, etc.

In this talk, Yusei will share how development teams can tackle these issues and speed up daily development. This talk will also cover the following topics:

- Workflow automation with Fastlane
- Code review with Danger and SwiftLint
- Collecting and visualizing code metrics with InfluxDB and Grafana
- Build time reduction
- Code modularization

562e29ba057361b2b944bd7bbd274887?s=128

Yusei Nishiyama

October 21, 2017
Tweet

Transcript

  1. Building iOS apps at scale Yusei Nishiyama @yuseinishiyama

  2. Introduction

  3. • iOS, Android, Web • 67 countries • 21 languages

    • 60M monthly users (Japan) • 30M+ monthly users 
 (Outside Japan) https://info.cookpad.com/en https://cookpad.com/en
  4. Distributed Team

  5. Working at Cookpad is … • Working in a large

    team • Working with large code base
  6. Agenda • Working in a large team • Automation with

    Fastlane • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
  7. Agenda • Working in a large team • Automation with

    Fastlane • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
  8. • Working in a large team • Automation with Fastlane

    • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
  9. • Someone broke the build • We can’t release today

    because {name} is on vacation • Code review never finishes
  10. Automation with CI • Testing • Nitpicking • Beta distribution

    • Release management
  11. CI Server

  12. CI Server

  13. CI Server

  14. CI Server

  15. • Working in a large team • Automation with Fastlane

    • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
  16. Fastlane in Cookpad • 7 Fastfiles • 28 lanes •

    740 lines • 5 plugins
  17. Sync dSYMs version = get_version_number build_number = get_build_number # Download

    dSYM files from iTC into a temporary directory download_dsyms(version: version, build_number: build_number) # Upload them to Crashlytics upload_symbols_to_crashlytics # Delete the temporary dSYM files clean_build_artifacts
  18. Release the App Spaceship::Tunes.login app_identifier = CredentialsManager::AppfileConfig.try_fetch_value(:app_identifier) edit_version = Spaceship::Application.find(app_identifier).edit_version

    version_to_release = edit_version.version UI.user_error! "No pending verion to release" unless edit_version.app_status == "Pending Developer Release" edit_version.release! UI.success "Version #{version_to_release} is released " slack( channel: "#release", attachment_properties: { pretext: ":appleinc: *Cookpad iOS #{version_to_release} - Released*” }) refresh_dsyms
  19. Triggers beta distribution

  20. Check Labels desc "Check if pr is marked as beta

    distribution" private_lane :beta_distribution do labels.any? { |label| label == 'beta distribution' } end desc "Get labels of pull request" private_lane :labels do pr_number = ENV['PR_NUMBER'] repo = ENV['GITHUB_PULL_REQUEST_REPO'] result = github_api( http_method: 'GET', path: "/repos/#{repo}/issues/#{pr_number}/labels" ) result[:json].map { |label| label['name'] } end
  21. Internal testers Developers Beta apps Feedback

  22. Internal testers Developers v 1.0.0.1 Feedback v 1.0.0.2 v 2.0.0.0

    v 2.1.0.2 Which version should I download? and what needs to be tested?
  23. Append badge on icon if is_release? next elsif is_rc? title

    = 'RC' number = get_version_number elsif is_beta? title = 'Beta' number = sh('git rev-parse --short=7 HEAD').strip elsif is_pr? title = 'PR' number = ENV.fetch('PR_NUMBER') end badge(shield: “#{title}-#{number}-blue", dark: true, no_badge: true)
  24. We are going to release a new version soon. Please

    check the latest RC.
  25. • Working in a large team • Automation with Fastlane

    • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
  26. None
  27. None
  28. CI Server

  29. Danger Danger runs during your CI process, and gives teams

    the chance to automate common code review chores. This provides another logical step in your build, through this Danger can help lint your rote tasks in daily code review. You can use Danger to codify your teams norms. Leaving humans to think about harder problems. She does this by leaving messages inside your PRs based on rules that you create with the Ruby scripting language. Over time, as rules are adhered to, the message is amended to reflect the current state of the code review http://danger.systems/ruby/
  30. Dangerfile github.dismiss_out_of_range_messages({ error: false, warning: true, message: true, markdown: true

    }) xcode_summary.inline_mode = true xcode_summary.ignored_files = ['Pods/**'] xcode_summary.ignored_results { |result| result.message.start_with? 'ld' } log = File.join('logs', 'test.json') xcode_summary.report log if File.exist?(log) swiftlint.binary_path = './Pods/SwiftLint/swiftlint' swiftlint.lint_files inline_mode: true
  31. None
  32. • Working in a large team • Automation with Fastlane

    • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
  33. CI Server

  34. InfluxDB • Only purpose-built platform for any time-based data •

    Written in Go • Built-in HTTP API • SQL-like query language • Answer queries in real-time • Rich client libraries
  35. Post Code Coverage lane :post_coverage do rate = export_coverage if

    rate influxdb(table_name: table_name, values: {coverage: rate}) slack(message: "Code coverage: #{rate.to_f.round(4) * 100}%", channel: “#ios- notifications") end end private_lane :export_coverage do fastlane_require 'nokogiri' output_directory = 'coverage_report' reports = Dir.glob(File.join(ROOT_PATH, output_directory, '*.xml')) slather(use_bundle_exec: true, output_directory: output_directory) if reports.empty? nil else xml = Nokogiri::XML(File.open(reports.first)) coverage = xml.xpath('//coverage').first coverage.attr("line-rate").to_f end end
  36. CI Server

  37. Grafana • Rich metrics dashboard and editor • Fast rendering

    • Support InfluxDB as a data source • Query editor
  38. Visualize Code Coverage

  39. Other metrics

  40. None
  41. • Working in a large team • Automation with Fastlane

    • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
  42. Let’s improve the registration screen…

  43. Let’s improve the registration screen… • Implement

  44. Let’s improve the registration screen… • Implement • Tweak constraints

  45. Let’s improve the registration screen… • Implement • Tweak constraints

    • Test with older iOS versions
  46. Let’s improve the registration screen… • Implement • Tweak constraints

    • Test with older iOS versions • Fix tests
  47. Let’s improve the registration screen… • Implement • Tweak constraints

    • Test with older iOS versions • Fix tests • Wait for CI
  48. How many times 
 did you build so far?

  49. • Working in a large team • Automation with Fastlane

    • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
  50. Reduce Build Time • Measurement • Improvement

  51. What I will “not” talk about • Annotate all types

    explicitly • Write as much as code in a single file • Get your Mac Pro
  52. Reduce Build Time • Measurement • Improvement

  53. “Premature optimization is the root of all evil.”
 
 —

    Donald Knuth —
  54. Display build duration $ defaults write com.apple.dt.Xcode ShowBuildOperationDuration YES

  55. Build Process • Dependency Manager (CocoaPods, Carthage) • Compile source

    files • Run Scripts (Linter, Formatter)
  56. fastlane/report.xml <?xml version="1.0" encoding="UTF-8"?> <testsuites> <testsuite name="fastlane.lanes"> <testcase classname="fastlane.lanes" name="00:

    default_platform" time="0.002078"> </testcase> <testcase classname="fastlane.lanes" name="01: bundle outdated fastlane --strict" time="8.59854"> </testcase> <testcase classname="fastlane.lanes" name="02: ensure_git_status_clean" time="0.017918"> </testcase> <testcase classname="fastlane.lanes" name="03: Switch to ios bump_build_number lane" time="0.001982"> </testcase> … <testcase classname="fastlane.lanes" name="17: hockey" time="263.575088"> </testcase> <testcase classname="fastlane.lanes" name="18: clean_build_artifacts" time="0.002326"> </testcase> </testsuite> </testsuites>
  57. None
  58. None
  59. None
  60. Build Process • Dependency Manager (CocoaPods, Carthage) • Compile source

    files • Run Scripts (Linter, Formatter)
  61. Build Process • Dependency Manager (CocoaPods, Carthage) • Compile source

    files • Function A • Function B • … • Runs Scripts (Linter, Formatter)
  62. Warn long function bodies

  63. Time function bodies $ pbpaste | grep '.[0-9]ms' | sort

    -nr | head -20
  64. xcactivitylog • You can access build logs under DerivedData •

    xcactivitylog is just gzip. You can check the content as a plain text just by unzipping it
  65. giginet/danger-xcprofiler (in Dangerfile)
 
 xcprofiler.thresholds = { warn: 50, fail:

    500 } xcprofiler.report ‘ProductName'
  66. Reduce Build Time • Measurement • Improvement

  67. Test with Real Data $ cloc
 
 github.com/AlDanial/cloc v 1.72

    T=3.40 s (153.3 files/s, 16646.6 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Swift 497 8944 3122 43831 JSON 18 0 0 522 Objective C 3 37 5 166 C/C++ Header 4 19 6 27 ------------------------------------------------------------------------------- SUM: 522 9000 3133 44546 -------------------------------------------------------------------------------
  68. Test with Real Data • 40,000 lines of Swift code

    • CocoaPods • 38 direct dependencies and 60 libraries • Linter in build phases • CI archives the app and distributes it for internal testing
  69. (Local clean build: 330s)

  70. None
  71. Whole Module Optimization

  72. WMO without Optimization

  73. (Local clean build: 160s)

  74. Time Function Bodies

  75. F

  76. Carthage + github "ReactiveX/RxSwift" ~> 3.4 + github "RxSwiftCommunity/RxRealm" +

    github "realm/realm-cocoa" ~> 2.4 + github "danielgindi/Charts" ~> 3.0.2 - pod 'RxSwift', '3.3.1' - pod 'RxCocoa', '3.3.1' - pod 'RxRealm', '0.5.2' - pod 'RealmSwift', '2.4.4' Cartfile Podfile
  77. (Local clean build: 68s)

  78. Xcode 9 New Build System

  79. • Xcode 9 • Swift 3.2 • New build system

    • No significant improvement in build time
  80. Do you really need to build your entire app?

  81. None
  82. Prototyping with Playground • Test your idea in Playground 


    (interface, design, reproducing a bug, etc.) • Once your idea becomes concrete, move those code into the application project
  83. Playground

  84. Playground

  85. You need “existing” code 
 for prototyping class ViewController: UITableViewController

    { let myView = MyView() override func viewDidLoad() { super.viewDidLoad() view.backgroundColor = .myColor view.addSubview(myView) } }
  86. Build your own UI library import UIKit @testable import YourUIKit

    let color = UIColor.myGreen
  87. • Working in a large team • Automation with Fastlane

    • Code review with Danger • Code metrics collection • Working with a large code base • Build time reduction • Code modularization
  88. Microservice Architecture https://martinfowler.com/articles/microservices.html

  89. Code Modularization Feature A Design Feature B Feature C Network

    Persistence Core
  90. Benefits • Strict access control • Faster incremental build/indexing

  91. Autonomous teams https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/

  92. Repository Strategy • Single Repository • Multiple Repositories

  93. Single repository • Simple workflow • Can’t open-source

  94. Multiple Repositories • Complex workflow • Code Review Process •

    Easy to open-source
  95. Dependency Management • CocoaPods • git submodule • Carthage •

    git subtree
  96. CocoaPods • Easy in use • Black box in nature

  97. git submodule • Simple (!= Easy) • Full control •

    Manual works • Complex workflow
  98. Carthage • Flexible • Pre-build • Not all libraries support

    it • ABI stability
  99. https://github.com/apple/swift-evolution#development-major-version--swift-50

  100. git subtree • An alternative to git submodule • Clones

    a subproject and merge it into the parent project • `git subtree push` goes through the commits and picking the changes that should go to a subproject
  101. git subtree

  102. git subtree • Requires no change to workflow • Need

    to learn a new merge strategy • Contributing back to upstream is a bit tricky https://medium.com/@v/git-subtrees-a-tutorial-6ff568381844
  103. Lesson Learned • Breaking existing implicit dependencies is hard •

    Starting from the single repository • Once modules become stable, consider • managing them in separated repositories • using pre-build versions
  104. Pitfall

  105. dylib Loading Time Total pre-main time: 4.2 seconds (100.0%) dylib

    loading time: 4.0 seconds (94.4%) rebase/binding time: 93.32 milliseconds (2.1%) ObjC setup time: 44.64 milliseconds (1.0%) initializer time: 98.30 milliseconds (2.3%)
  106. Pitfall

  107. Xcode 9 supports Swift Static Library https://download.developer.apple.com/Developer_Tools/Xcode_9_beta_4/ Xcode_9_beta_4_Release_Notes.pdf

  108. Recap • Automate daily workflows with CI and Fastlane •

    Reduce workload of code review with Danger • Visualize code metrics you want to improve • Revising project settings may improve build time dramatically • Code modularization improves build/index time and helps you getting well structured code/teams
  109. Thank you! Yusei Nishiyama @yuseinishiyama https://speakerdeck.com/yuseinishiyama/building-ios-apps-at-scale-mobilization