Standardizing build system using modern Swift Packages in LINE

Mobile DevOps Summit 2022
https://www.mobiledevops.io/summit/agenda/session/1022649

https://github.com/giginet/Scipio

Transcript

1. Standardizing build system using modern Swift Packages in LINE 2022-11-10 Mobile DevOps Summit 2022 Kohki Miki (@giginet) LINE Corp
2. Self Introduction • Kohki Miki 🇯🇵 (@giginet) • Senior iOS Engineer @ LINE Corp (2022/7~) • Developer Experience team • OSS Maintainer : fastlane / Carthage / XcodeGen, etc…
3. Agenda • Our current build system and its issues • Introduce my new SwiftPM-based build tool • Handling Swift Packages as cachable and reusable.
4. State of LINE • LINE • A communication app used by 200 million users • 2,000,000+ lines of Swift Code (iOS)
5. Build Phases of LINE Prebuilt Tasks Building Application Bazel Xcode
6. Prebuilt Frameworks • Regular Swift, Obj-C OSS libraries • C/C++ libraries (e.g. Image processing, encryption) • Binary Frameworks • Separated Features • Code generation from API Schemes Prebuilt Tasks Bazel
7. Prebuilt Frameworks • Regular Swift, Obj-C OSS libraries • C/C++ libraries (e.g. Image processing, encryption) • Binary Frameworks • Separated Features • Code generation from API Schemes Prebuilt Tasks Bazel
8. Prebuilt Frameworks • Regular Swift, Obj-C OSS libraries • C/C++ libraries (e.g. Image processing, encryption) • Binary Frameworks • Separated Features • Code generation from API Schemes Prebuilt Tasks Bazel
9. Prebuilt Frameworks • Regular Swift, Obj-C OSS libraries • C/C++ libraries (e.g. Image processing, encryption) • Binary Frameworks • Separated Features • Code generation from API Schemes Prebuilt Tasks Bazel
10. Prebuilt Frameworks • Regular Swift, Obj-C OSS libraries • C/C++ libraries (e.g. Image processing, encryption) • Binary Frameworks • Separated Features • Code generation from API Schemes Prebuilt Tasks Bazel
11. Build time of each steps 926.42 seconds 503.4 seconds Prebuilt Tasks Building Application Bazel Xcode
12. Problems on maintainability • Tough to maintain Bazel • Implementing tasks in Starlark • No tests • Learning cost for Bazel • Large numbers of submodules • No dependency resolver • Abandoned dependencies Implementations of Bazel rules in Starlark
13. Problems on maintainability • Tough to maintain Bazel • Implementing tasks in Starlark • No tests • Learning cost for Bazel • Large numbers of submodules • No dependency resolver • Abandoned dependencies Submodules hell
14. Problems on maintainability • Tough to maintain Bazel • Implementing tasks in Starlark • No tests • Learning cost for Bazel • Large numbers of submodules • No dependency resolver • Abandoned dependencies http_archive( name = "APIKit", build_file = "//Configuration/repositories:APIKit.BUILD", sha256 = "0d19ebb5769ca464c7b10671ca1df21b8e9edad198f59e2d854c48cc580383 43", strip_prefix = "APIKit-5.0.0", url = "https://github.com/ishkawa/APIKit/archive/ 5.0.0.tar.gz", ) http_archive( name = "APNGKit", build_file = "//Configuration/repositories:APNGKit.BUILD", sha256 = "89245edaabcda2872141ecfcfad6d88fa5f0293af52a4cbcd2dd2fd6e3f931 99", strip_prefix = “APNGKit-2.2.1”, url = “https://github.com/onevcat/APNGKit/archive/ 2.2.1.tar.gz”, ) Dependency de f initions on our WORKSPACE
15. Build speeds •Bazel used to improve build time dramatically •However, Apple Silicon(M1) improves build type than Bazel Need more maintainable build work f low
16. Our requirements • Replacing the prebuilt task • Simple. based on the standard build system • Reusing build artifacts and caching well • Easy to develop and testable • Easy to update dependencies
17. Swift Package with Xcode • Swift Package integration for Xcode is the best solution • Maintained in the future • Most packages are supported
18. Cache Swift Packages with Xcode Xcode stores Swift Package dependencies in Derived Data It can be purged easily!
19. How to cache Swift Packages? Using Swift Package and Swift Package Manager(SwiftPM) is a good choice However, it’s dif f icult to keep build speed by using caches I developed a new tool! Scipio
20. Scipio - An awesome build tool - https://github.com/giginet/Scipio
21. Scipio • Developing new build tool Scipio • Convert Swift Packages to portable binaries • https://github.com/giginet/Scipio • Cache System • Support Static Frameworks • Inspired by Carthage
22. Basic ideas of Scipio • Follow the Apple standard build system • Integrate SwiftPM and XCFramework • Resolving Swift Packages with SwiftPM • Converting to XCFramework Swift Package XCFramework Convert
23. XCFrameworks • Binary distribution format • They can contain multiple frameworks for various architectures and platforms Library.xcframework for arm64 iOS devices for x86_64 iOS Simulators for arm64 Mac devices
24. Using Scipio Prepare a Package.swift describes all app’s dependencies // swift-tools-version: 5.7 import PackageDescription let package = Package( name: "Dependencies", platforms: [.iOS(.v13),], dependencies: [ .package(url: "https://github.com/apple/swift-log.git", from: "1.4.4"), .package(url: "https://github.com/apple/swift-collections.git", .upToNextMinor(from: "1.0.0")), ], targets: [ .target( name: "Dependencies", dependencies: [ .product(name: "Logging", package: "swift-log"), .product(name: "OrderedCollections", package: "swift-collections"), .product(name: "DequeModule", package: "swift-collections"), .product(name: "Collections", package: "swift-collections"), ]), ] )
25. Build all dependencies with Scipio $ scipio prepare /path/to/Dependencies 🔁 Resolving Dependencies... 🗑 Cleaning Dependencies... 📦 Building OrderedCollections for iOS 🚀 Combining into XCFramework... 📦 Building DequeModule for iOS 🚀 Combining into XCFramework... 📦 Building Collections for iOS 🚀 Combining into XCFramework... 📦 Building Logging for iOS 🚀 Combining into XCFramework... ❇ Succeeded.
26. 1. Resolve and checkout package manifest with SwiftPM 2. Scipio generates Xcode Project from each package 3. Build the projects with xcodebuild 4. xcodebuild generates XCFramework How does Scipio work? Resolve manifests Fetch packages Generate Xcode Project Build XCFrameworks
27. Cache mechanism • Scipio supports cache system • Generate matadata within XCFrameworks • Called VersionFile
28. What is VersionFile { "buildOptions" : { "buildConfiguration" : "debug", "isDebugSymbolsEmbedded" : true, "frameworkType" : "static", "sdks" : [ "iOS" ], "isSimulatorSupported" : true }, "targetName" : "AwesomeKit", "clangVersion" : "clang-1400.0.29.102", "pin" : { "version" : "5.0.1", "revision" : "d458564516e5676af9c70b4f4b2a9178294f1bc6" } } • A JSON f ile contains build contexts • Scipio try to check this before building • Building will be skipped if existing framework is valid .AwesomeKit.version
29. Project cache • Just compare with VersionFile in a destination directory • If a valid VersionFile is exist, Scipio skips building AwesomeKit.xcframework .AwesomeKit.version Destination Directory $ scipio prepare /path/to/Dependencies 🔁 Resolving Dependencies... 🗑 Cleaning Dependencies... ✅ Valid OrderedCollections.xcframework is exists. Skip building. ✅ Valid DequeModule.xcframework is exists. Skip building. ✅ Valid Collections.xcframework is exists. Skip building. ✅ Valid Logging.xcframework is exists. Skip building. ❇ Succeeded.
30. Disk cache • Save XCFrameworks to a system cache directory • Compare SHA256 hash of VersionFiles • Restore same frameworks built in the past AwesomeKit.xcframework Destination Directory System Cache Directory AAAAAA.xcframework BBBBBB.xcframework Restore
31. Remote cache • Just replace storage with remotes (e.g. Amazon S3) • Share all caches among developers AwesomeKit.xcframework Destination Directory Remote object storage AAAAAA.xcframework BBBBBB.xcframework Restore 􀇃
32. Kind of Swift Packages • Swift Package can support various use cases • C Family language supports (SE-0038) • Binary Dependencies (SE-0272) • Artifact bundles (SE-0305)
33. SE-0038 C languages supports • Package Manager C Language Target Support • https://github.com/apple/swift-evolution/blob/main/ proposals/0038-swiftpm-c-language-targets.md • Create Swift Package for C/C++ libraries
34. SE-0272 Binary Framework • SE-0272 Package Manager Binary Dependencies • https://github.com/apple/swift-evolution/blob/main/ proposals/0272-swiftpm-binary-dependencies.md • Distribute prebuilt binaries in Swift Package let package = Package( name: "SomePackage", targets: [ .binaryTarget( name: "SomePackageLib", url: "https://github.com/some/package/releases/download/1.0.0/SomePackage-1.0.0.zip", checksum: "839F9F30DC13C30795666DD8F6FB77DD0E097B83D06954073E34FE5154481F7A" ), .binaryTarget( name: "SomeLibOnDisk", path: "artifacts/SomeLibOnDisk.zip" ) ] )
35. SE-0305 Artifact bundles • Package Manager Binary Target Improvements • https://github.com/apple/swift-evolution/ blob/main/proposals/0305-swiftpm-binary- target-improvements.md • Artifact bundle • A directory structure to distribute executables for multiple architectures in Swift Packages <name>.artifactbundle ᵓ info.json ᵓ <artifact> │ ᵓ <variant> │ │ ᵓ <executable> │ │ └ <other f iles> │ └ <variant> │ ᵓ <executable> │ └ <other f iles> ᵓ <artifact> │ └ <variant> │ ᵓ <executable> │ └ <other f iles> │ <artifact> ᴽ └ᴻ
36. Future works • We’re migrating Prebuilt frameworks to Swift Packages • Improve Scipio • Improving stability • Supporting Binary targets
37. Conclusion • Migrating our build system • Using Swift packages is a good solution • Created an OSS, Scipio • Feedbacks and Pull Requests are welcome • https://github.com/giginet/Scipio • Please look forward to our future work!
38. Thank you for listening