$30 off During Our Annual Pro Sale. View Details »

Caching, a simple solution to speeding up build times

Caching, a simple solution to speeding up build times

Build times can be long, excruciatingly long, especially when dealing with Swift. A simple solution to mitigate the problem is caching pre-built dependencies. Rome is a tool that can help with that in combination with Carthage.

Tommaso Piazza

May 17, 2017
Tweet

More Decks by Tommaso Piazza

Other Decks in Programming

Transcript

  1. Caching,
    a simple solution to speeding up build times
    Rome, a cache for Carthage
    Tommaso Piazza - @tmpz
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  2. The Problem
    • A lot of frameworks
    • Long build times per framework
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  3. Like this much
    git "ssh://[email protected]:7999/cppdsp/feingoldtech-cpp-vadsegmentor" "d702e1e120f1f930abc568508386845b470596d2"
    git "ssh://[email protected]:7999/iossdk/feingoldtech-ios-analytics.git" "306969537850ba16aee481227097c47b82b16813"
    git "ssh://[email protected]:7999/iossdk/feingoldtech-ios-askmd.git" "d884fded5543834b5d40740af447614f4f6d6366"
    git "ssh://[email protected]:7999/iossdk/feingoldtech-ios-auth.git" "v5.0.0"
    git "ssh://[email protected]:7999/iossdk/feingoldtech-ios-core.git" "v1.0.0"
    git "ssh://[email protected]:7999/iossdk/feingoldtech-ios-geonames.git" "a942405b0b692d3062fee3d4a4d283aa41f50ddf"
    git "ssh://[email protected]:7999/iossdk/feingoldtech-ios-lig.git" "97320c62e2e5b0849c463a402d4646ae28c10659"
    git "ssh://[email protected]:7999/iossdk/feingoldtech-ios-realage.git" "65fa4fc13a6384d9ab01575b62671698341c7b75"
    git "ssh://[email protected]:7999/iossdk/feingoldtech-ios-rest.git" "b2e413019e4f0725fc97b0b7e734bbfaf2c8b9e4"
    git "ssh://[email protected]:7999/iossdk/feingoldtech-ios-search.git" "67adc70c1e66b31962162fa73e8cb8cd0fab1842"
    git "ssh://[email protected]:7999/iossdk/feingoldtech-ios-sleep.git" "3b79e4dd252dd69548eb828e3e17e2f0fd763060"
    git "ssh://[email protected]:7999/iossdk/feingoldtech-ios-voice.git" "7cfa3955a02e3889efaa50f01809f97d5d2d3c21"
    github "Alamofire/Alamofire" "4.4.0"
    github "IFTTT/RazzleDazzle" "0.1.5"
    github "JohnEstropia/CoreStore" "3.1.0"
    github "Marxon13/M13Checkbox" "3.1.3"
    github "OpenShelter/Tyro" "v0.1.0"
    github "ResearchKit/ResearchKit" "b04a5aa6a4794bc35c28233d9f182a355b19cb0c"
    github "SnapKit/SnapKit" "3.2.0"
    github "bitstadium/HockeySDK-iOS" "3.8.6"
    github "evgenyneu/SigmaSwiftStatistics" "5.0.1"
    github "facebook/pop" "1.0.9"
    github "kishikawakatsumi/KeychainAccess" "v3.0.2"
    github "marketplacer/Cosmos" "9.0.0"
    github "netbe/Koloda" "79fbd1ab40545c1996ae20600b74c9c5a1ee905d"
    github "typelift/Swiftz" "0.6.3"
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  4. More Problems
    • Frameworks versions change very frequently (every build)
    • Long app build times (~45m)
    • Change branch, rebuild at different hash
    • Sharing
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  5. The obvious solution
    • Cache
    • Producer / Consumer flow
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  6. Existing solutions - 1
    • Download/Upload entire build folder (> 500 MB)
    • Cache key computed from entire folder
    • Changing one library invalidates the cache
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  7. Existing solutions - 2
    • Attach build to Github release
    • Github is not an option for everyone
    • Tag every build
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  8. Rome Features
    • Cache just what you need
    • Per dependency cache keys derived from Cartfile.resolved
    • Cache redundancy
    • local folder
    • Amazon S3
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  9. More Rome Features
    • Probe cache for presence of single key
    • Blacklist some keys
    • Selective Download/Upload
    • 1,2,3... all frameworks
    • Per platform cache
    • iOS, tvOS, MacOS, watchOS
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  10. Even More Rome Features
    Cached artifacts
    • Frameworks
    • dSYMs
    • Carthage .version files
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  11. Automation Ready
    • Use on CI
    • Use with fastlane
    • @netbe's fastlane-plugin-rome
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  12. Alone?
    Useful for a one man team
    • Change branch
    • Pull framework version from cache
    • Enjoy
    • No rebuilds, no waiting
    • Productive, not painful experience
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  13. Demo Time
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  14. Rome Commands
    • Upload
    rome upload [Framework1, Framework2 ...] [--platform iOS,MacOS,tvOS,watchOS]
    • Download
    rome download [Framework1, Framework2 ...] [--platform iOS,MacOS,tvOS,watchOS]
    • List
    rome list [--missing | --present] [--platform iOS,MacOS,tvOS,watchOS]
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  15. Workflow example
    • Download available frameworks
    • Build only what is missing
    • Upload what was built
    • rome list --missing --platform ios
    | awk '{print $1}'
    | xargs carthage build --platform ios
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  16. Set up checklist
    • Cartfile.resolved
    • Romefile
    And you're done!
    • When using S3
    • ~/.aws/config or environment variable
    • ~/.aws/credentials or environment variable
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  17. Romefile
    [Cache] # Mandatory section
    S3-Bucket = feingoldtech-development-ios # Optional
    local = ~/Library/Caches/Rome # Optional, but at least one of the two
    [RepositoryMap] # Optional section
    HanekeSwift = Haneke
    HockeySDK-iOS = HockeySDK
    feingoldtech-cpp-vadsegmentor = VadSegmentor, SegmentorUtils # Multiple target support
    feingoldtech-ios-core = FGCore
    feingoldtech-ios-auth = FGAuth
    feingoldtech-ios-realage = FGRealAge
    feingoldtech-ios-voice = FGVoice
    feingoldtech-ios-lig = FGLiG
    feingoldtech-ios-geonames = FGGeoNames
    feingoldtech-ios-search = FGSearch
    [IgnoreMap] # Optional section
    GDCWebServer = GDCWebServer
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  18. Minimal Romefile
    [Cache]
    local = ~/Library/Caches/Rome
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  19. Why Romefile? - Cache Section
    To specify cache locations and names
    [Cache] # Mandatory section
    S3-Bucket = feingoldtech-development-ios # Optional
    local = ~/Library/Caches/Rome # Optional, but at least one of the two
    [RepositoryMap]
    ...
    [IgnoreMap]
    ...
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  20. Why Romefile? - RepositoryMap Section
    Not needed if dependencies follow
    Organisation\FrameworkName
    naming pattern.
    From Cartfile.resolved
    git "ssh://...:7999/cppdsp/feingoldtech-cpp-vadsegmentor" "d702...6d2"
    github "bitstadium/HockeySDK-iOS" "3.8.6"
    • No
    Organisation\FrameworkName
    pattern
    • Framework name is FGVoice

    Organisation\FrameworkName
    pattern
    • Framework Name is HockeySDK
    • Multiple targets per project
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  21. Why Romefile? - IgnoreMap Section
    Selectively blacklist targets for a Frameworks in Cartfile.resolved
    github "jspahrsummers/xcconfigs"
    [Cache]
    ...
    [RepositoryMap]
    ...
    [IgnoreMap]
    xcconfigs = xcconfigs
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  22. Roadmap
    • Auto generation of Romefile
    • Google Cloud Storage
    • Upload current framework (like carthage archive)
    • Suggestions ?
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide

  23. Thanks!
    https://github.com/blender/Rome
    https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz

    View Slide