Caching, a simple solution to speeding up build times

Build times can be long, excruciatingly long, especially when dealing with Swift. A simple solution to mitigate the problem is caching pre-built dependencies. Rome is a tool that can help with that in combination with Carthage.

Transcript

1. Caching, a simple solution to speeding up build times Rome, a cache for Carthage Tommaso Piazza - @tmpz https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
2. The Problem • A lot of frameworks • Long build times per framework https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
3. Like this much git "ssh://git@stash.feingoldtech.com:7999/cppdsp/feingoldtech-cpp-vadsegmentor" "d702e1e120f1f930abc568508386845b470596d2" git "ssh://git@stash.feingoldtech.com:7999/iossdk/feingoldtech-ios-analytics.git" "306969537850ba16aee481227097c47b82b16813" git "ssh://git@stash.feingoldtech.com:7999/iossdk/feingoldtech-ios-askmd.git" "d884fded5543834b5d40740af447614f4f6d6366" git "ssh://git@stash.feingoldtech.com:7999/iossdk/feingoldtech-ios-auth.git" "v5.0.0" git "ssh://git@stash.feingoldtech.com:7999/iossdk/feingoldtech-ios-core.git" "v1.0.0" git "ssh://git@stash.feingoldtech.com:7999/iossdk/feingoldtech-ios-geonames.git" "a942405b0b692d3062fee3d4a4d283aa41f50ddf" git "ssh://git@stash.feingoldtech.com:7999/iossdk/feingoldtech-ios-lig.git" "97320c62e2e5b0849c463a402d4646ae28c10659" git "ssh://git@stash.feingoldtech.com:7999/iossdk/feingoldtech-ios-realage.git" "65fa4fc13a6384d9ab01575b62671698341c7b75" git "ssh://git@stash.feingoldtech.com:7999/iossdk/feingoldtech-ios-rest.git" "b2e413019e4f0725fc97b0b7e734bbfaf2c8b9e4" git "ssh://git@stash.feingoldtech.com:7999/iossdk/feingoldtech-ios-search.git" "67adc70c1e66b31962162fa73e8cb8cd0fab1842" git "ssh://git@stash.feingoldtech.com:7999/iossdk/feingoldtech-ios-sleep.git" "3b79e4dd252dd69548eb828e3e17e2f0fd763060" git "ssh://git@stash.feingoldtech.com:7999/iossdk/feingoldtech-ios-voice.git" "7cfa3955a02e3889efaa50f01809f97d5d2d3c21" github "Alamofire/Alamofire" "4.4.0" github "IFTTT/RazzleDazzle" "0.1.5" github "JohnEstropia/CoreStore" "3.1.0" github "Marxon13/M13Checkbox" "3.1.3" github "OpenShelter/Tyro" "v0.1.0" github "ResearchKit/ResearchKit" "b04a5aa6a4794bc35c28233d9f182a355b19cb0c" github "SnapKit/SnapKit" "3.2.0" github "bitstadium/HockeySDK-iOS" "3.8.6" github "evgenyneu/SigmaSwiftStatistics" "5.0.1" github "facebook/pop" "1.0.9" github "kishikawakatsumi/KeychainAccess" "v3.0.2" github "marketplacer/Cosmos" "9.0.0" github "netbe/Koloda" "79fbd1ab40545c1996ae20600b74c9c5a1ee905d" github "typelift/Swiftz" "0.6.3" https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
4. More Problems • Frameworks versions change very frequently (every build) • Long app build times (~45m) • Change branch, rebuild at different hash • Sharing https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
5. The obvious solution • Cache • Producer / Consumer flow https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
6. Existing solutions - 1 • Download/Upload entire build folder (> 500 MB) • Cache key computed from entire folder • Changing one library invalidates the cache https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
7. Existing solutions - 2 • Attach build to Github release • Github is not an option for everyone • Tag every build https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
8. Rome Features • Cache just what you need • Per dependency cache keys derived from Cartfile.resolved • Cache redundancy • local folder • Amazon S3 https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
9. More Rome Features • Probe cache for presence of single key • Blacklist some keys • Selective Download/Upload • 1,2,3... all frameworks • Per platform cache • iOS, tvOS, MacOS, watchOS https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
10. Even More Rome Features Cached artifacts • Frameworks • dSYMs • Carthage .version files https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
11. Automation Ready • Use on CI • Use with fastlane • @netbe's fastlane-plugin-rome https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
12. Alone? Useful for a one man team • Change branch • Pull framework version from cache • Enjoy • No rebuilds, no waiting • Productive, not painful experience https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
13. Demo Time https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
14. Rome Commands • Upload rome upload [Framework1, Framework2 ...] [--platform iOS,MacOS,tvOS,watchOS] • Download rome download [Framework1, Framework2 ...] [--platform iOS,MacOS,tvOS,watchOS] • List rome list [--missing | --present] [--platform iOS,MacOS,tvOS,watchOS] https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
15. Workflow example • Download available frameworks • Build only what is missing • Upload what was built • rome list --missing --platform ios | awk '{print $1}' | xargs carthage build --platform ios https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
16. Set up checklist • Cartfile.resolved • Romefile And you're done! • When using S3 • ~/.aws/config or environment variable • ~/.aws/credentials or environment variable https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
17. Romefile [Cache] # Mandatory section S3-Bucket = feingoldtech-development-ios # Optional local = ~/Library/Caches/Rome # Optional, but at least one of the two [RepositoryMap] # Optional section HanekeSwift = Haneke HockeySDK-iOS = HockeySDK feingoldtech-cpp-vadsegmentor = VadSegmentor, SegmentorUtils # Multiple target support feingoldtech-ios-core = FGCore feingoldtech-ios-auth = FGAuth feingoldtech-ios-realage = FGRealAge feingoldtech-ios-voice = FGVoice feingoldtech-ios-lig = FGLiG feingoldtech-ios-geonames = FGGeoNames feingoldtech-ios-search = FGSearch [IgnoreMap] # Optional section GDCWebServer = GDCWebServer https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
18. Minimal Romefile [Cache] local = ~/Library/Caches/Rome https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
19. Why Romefile? - Cache Section To specify cache locations and names [Cache] # Mandatory section S3-Bucket = feingoldtech-development-ios # Optional local = ~/Library/Caches/Rome # Optional, but at least one of the two [RepositoryMap] ... [IgnoreMap] ... https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
20. Why Romefile? - RepositoryMap Section Not needed if dependencies follow Organisation\FrameworkName naming pattern. From Cartfile.resolved git "ssh://...:7999/cppdsp/feingoldtech-cpp-vadsegmentor" "d702...6d2" github "bitstadium/HockeySDK-iOS" "3.8.6" • No Organisation\FrameworkName pattern • Framework name is FGVoice • Organisation\FrameworkName pattern • Framework Name is HockeySDK • Multiple targets per project https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
21. Why Romefile? - IgnoreMap Section Selectively blacklist targets for a Frameworks in Cartfile.resolved github "jspahrsummers/xcconfigs" [Cache] ... [RepositoryMap] ... [IgnoreMap] xcconfigs = xcconfigs https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
22. Roadmap • Auto generation of Romefile • Google Cloud Storage • Upload current framework (like carthage archive) • Suggestions ? https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz
23. Thanks! https://github.com/blender/Rome https://github.com/blender/Rome - CocoaHeads Berlin 17/05/2017 - @tmpz