Working at Scale

Working with a large code base in a large distributed team involves a lot of challenges. You need to deal with complex workflows, slow build times, etc.

In this talk, Yusei will share how development teams can tackle these issues and speed up daily development. This talk will also cover the following topics:

- Workflow automation with Fastlane
- Code review with Danger and SwiftLint
- Collecting and visualizing code metrics with InfluxDB and Grafana
Build time reduction
- Code modularization

Transcript

1. Working at scale
   How to save time with CI and build time reduction
   Yusei Nishiyama
   @yuseinishiyama
   

   View Slide

2. • iOS, Android, Web
   • 67 countries
   • 21 languages
   • 60M monthly users (Japan)
   • 30M+ monthly users 

   (Outside Japan)
   https://info.cookpad.com/en
   https://cookpad.com/en
   

   View Slide

3. Distributed Team
   

   View Slide

4. Working at Cookpad is …
   • Working in a large team
   • Working with large code base
   

   View Slide

5. Working at Cookpad is …
   • Working in a large team
   • Working with a large code base
   

   View Slide

6. • Someone broke the build
   • We can’t release today because {name} is on
   vacation
   • Code review never finishes
   

   View Slide

7. Automation with CI
   • Testing
   • Nitpicking
   • Beta distribution
   • Release management
   

   View Slide

8. CI Server
   

   View Slide

9. CI Server
   

   View Slide

10. CI Server
    

    View Slide

11. CI Server
    

    View Slide

12. Fastlane in Cookpad
    • 7 Fastfiles
    • 28 lanes
    • 740 lines
    • 5 plugins
    

    View Slide

13. Sync dSYMs
    desc "Refresh dSYMs"
    lane :refresh_dsyms do
    version = get_version_number
    build_number = get_build_number
    # Download dSYM files from iTC into a temporary directory
    download_dsyms(version: version, build_number: build_number)
    # Upload them to Crashlytics
    upload_symbols_to_crashlytics
    # Delete the temporary dSYM files
    clean_build_artifacts
    end
    

    View Slide

14. Release the App
    Spaceship::Tunes.login
    app_identifier =
    CredentialsManager::AppfileConfig.try_fetch_value(:app_identifier)
    edit_version = Spaceship::Application.find(app_identifier).edit_version
    version_to_release = edit_version.version
    UI.user_error! "No pending verion to release" unless
    edit_version.app_status == "Pending Developer Release"
    edit_version.release!
    UI.success "Version #{version_to_release} is released "
    slack(
    channel: "#release",
    attachment_properties: {
    pretext: ":appleinc: *Cookpad iOS #{version_to_release} - Released*”
    })
    refresh_dsyms
    

    View Slide

15. Triggers beta distribution
    

    View Slide

16. Check Labels
    desc "Check if pr is marked as beta distribution"
    private_lane :beta_distribution do
    labels.any? { |label| label == 'beta distribution' }
    end
    desc "Get labels of pull request"
    private_lane :labels do
    pr_number = ENV['PR_NUMBER']
    repo = ENV['GITHUB_PULL_REQUEST_REPO']
    result = github_api(
    http_method: 'GET',
    path: "/repos/#{repo}/issues/#{pr_number}/labels"
    )
    result[:json].map { |label| label['name'] }
    end
    

    View Slide

17. Notify a New Build
    desc "Notify new beta distribution"
    private_lane :notify_build do
    messages = "A new version is available"
    pr_number = ENV['PR_NUMBER']
    repo = ENV['GITHUB_PULL_REQUEST_REPO']
    github_api(
    http_method: 'POST',
    path: "/repos/#{repo}/issues/#{pr_number}/comments",
    body: { "body": message }
    )
    end
    

    View Slide

18. View Slide

19. CI Server
    

    View Slide

20. Dangerfile
    github.dismiss_out_of_range_messages({
    error: false,
    warning: true,
    message: true,
    markdown: true
    })
    xcode_summary.inline_mode = true
    xcode_summary.ignored_files = ['Pods/**']
    xcode_summary.ignored_results { |result|
    result.message.start_with? 'ld'
    }
    log = File.join('logs', 'test.json')
    xcode_summary.report log if File.exist?(log)
    swiftlint.binary_path = './Pods/SwiftLint/swiftlint'
    swiftlint.lint_files inline_mode: true
    

    View Slide

21. View Slide

22. CI Server
    

    View Slide

23. InfluxDB
    • Only purpose-built platform for any time-based data
    • Written in Go
    • Built-in HTTP API
    • SQL-like query language
    • Answer queries in real-time
    • Rich client libraries
    

    View Slide

24. Post Code Coverage
    lane :post_coverage do
    rate = export_coverage
    if rate
    influxdb(table_name: table_name, values: {coverage: rate})
    slack(message: "Code coverage: #{rate.to_f.round(4) * 100}%", channel: “#ios-
    notifications")
    end
    end
    private_lane :export_coverage do
    fastlane_require 'nokogiri'
    output_directory = 'coverage_report'
    reports = Dir.glob(File.join(ROOT_PATH, output_directory, '*.xml'))
    slather(use_bundle_exec: true, output_directory: output_directory)
    if reports.empty?
    nil
    else
    xml = Nokogiri::XML(File.open(reports.first))
    coverage = xml.xpath('//coverage').first
    coverage.attr("line-rate").to_f
    end
    end
    

    View Slide

25. CI Server
    

    View Slide

26. Grafana
    • Rich metrics dashboard and editor
    • Fast rendering
    • Support InfluxDB as a data source
    • Query editor
    

    View Slide

27. Visualize Code Coverage
    

    View Slide

28. Other metrics
    

    View Slide

29. Working at Cookpad is …
    • Working in a large team
    • Working with a large code base
    

    View Slide

30. How many times 

    do you build every day?
    

    View Slide

31. Reduce Build Time
    • Measurement
    • Improvement
    

    View Slide

32. What I will “not” talk about
    • Annotate all types explicitly
    • Write as much as code in a single file
    • Get your Mac Pro
    

    View Slide

33. Reduce Build Time
    • Measurement
    • Improvement
    

    View Slide

34. “Premature optimization is
    the root of all evil.”

    

    — Donald Knuth —
    

    View Slide

35. Display build duration
    $ defaults write com.apple.dt.Xcode ShowBuildOperationDuration YES
    

    View Slide

36. Build Process
    • Dependency Manager (CocoaPods, Carthage)
    • Compile source files
    • Run tests
    

    View Slide

37. fastlane/report.xml
    
    
    
    
    
    time="8.59854">
    
    
    
    time="0.001982">
    
    …
    
    
    
    
    
    
    

    View Slide

38. View Slide

39. View Slide

40. View Slide

41. Build Process
    • Dependency Manager (CocoaPods, Carthage)
    • Compile source files
    • Run tests
    

    View Slide

42. Build Process
    • Dependency Manager (CocoaPods, Carthage)
    • Compile source files
    • Function A
    • Function B
    • …
    • Runs tests
    

    View Slide

43. Warn long function bodies
    

    View Slide

44. Time function bodies
    $ pbpaste | grep '.[0-9]ms' | sort -nr | head -20
    

    View Slide

45. xcactivitylog
    • You can access build logs under DerivedData
    • xcactivitylog is just gzip. You can check the
    content as a plain text just by unzipping it
    

    View Slide

46. giginet/danger-xcprofiler
    (in Dangerfile)

    

    xcprofiler.thresholds = {
    warn: 50,
    fail: 500
    }
    xcprofiler.report ‘ProductName'
    

    View Slide

47. Reduce Build Time
    • Measurement
    • Improvement
    

    View Slide

48. Test with Real Data
    $ cloc

    

    github.com/AlDanial/cloc v 1.72 T=3.40 s (153.3 files/s, 16646.6 lines/s)
    -------------------------------------------------------------------------------
    Language files blank comment code
    -------------------------------------------------------------------------------
    Swift 497 8944 3122 43831
    JSON 18 0 0 522
    Objective C 3 37 5 166
    C/C++ Header 4 19 6 27
    -------------------------------------------------------------------------------
    SUM: 522 9000 3133 44546
    -------------------------------------------------------------------------------
    

    View Slide

49. Test with Real Data
    • 40,000 lines of Swift code
    • CocoaPods
    • 38 direct dependencies and 60 libraries
    • Linter in build phases
    • CI archives the app and distributes it for internal
    testing
    

    View Slide

50. (Local clean build: 330s)
    

    View Slide

51. View Slide

52. Whole Module Optimization
    

    View Slide

53. WMO without Optimization
    

    View Slide

54. (Local clean build: 160s)
    

    View Slide

55. Time Function Bodies
    

    View Slide

56. F
    

    View Slide

57. Carthage
    + github "ReactiveX/RxSwift" ~> 3.4
    + github "RxSwiftCommunity/RxRealm"
    + github "realm/realm-cocoa" ~> 2.4
    + github "danielgindi/Charts" ~> 3.0.2
    - pod 'RxSwift', '3.3.1'
    - pod 'RxCocoa', '3.3.1'
    - pod 'RxRealm', '0.5.2'
    - pod 'RealmSwift', '2.4.4'
    Cartfile
    Podfile
    

    View Slide

58. https://github.com/apple/swift-evolution#development-major-version--swift-50
    

    View Slide

59. (Local clean build: 68s)
    

    View Slide

60. Xcode 9 New Build System
    

    View Slide

61. • Xcode 9
    • Swift 3.2
    • New build system
    • No significant improvement in build time
    

    View Slide

62. Do you really need to build
    your entire app?
    

    View Slide

63. Code Modularization
    Feature A
    Design
    Feature B Feature C
    Network Persistence
    Core
    

    View Slide

64. Benefits
    • Faster incremental build/indexing
    • Strict access control
    • Clear separation
    

    View Slide

65. Lessons Learned
    • Breaking existing implicit dependencies is hard
    • Starting from the single repository
    • Once modules become stable, consider
    • managing them in separated repositories
    • using pre-build versions
    

    View Slide

66. Pitfall
    

    View Slide

67. dylib Loading Time
    Total pre-main time: 4.2 seconds (100.0%)
    dylib loading time: 4.0 seconds (94.4%)
    rebase/binding time: 93.32 milliseconds (2.1%)
    ObjC setup time: 44.64 milliseconds (1.0%)
    initializer time: 98.30 milliseconds (2.3%)
    

    View Slide

68. Pitfall
    

    View Slide

69. Xcode 9 supports
    Swift Static Library
    https://download.developer.apple.com/Developer_Tools/Xcode_9_beta_4/
    Xcode_9_beta_4_Release_Notes.pdf
    

    View Slide

70. Recap
    • Automate daily workflows with CI and Fastlane
    • Reduce workload of code review with Danger
    • Visualize code metrics you want to improve
    • Revising project settings may improve build time
    dramatically
    • Code modularization improves build/index time and
    helps you get well structured code/teams
    

    View Slide

71. The presentation is available on

    
    https://speakerdeck.com/yuseinishiyama/working-at-scale
    

    View Slide

72. Thank you!
    Yusei Nishiyama
    @yuseinishiyama
    

    View Slide