Standardizing build system using modern Swift Packages in LINE 2022-11-10 Mobile DevOps Summit 2022 Kohki Miki (@giginet) LINE Corp

Self Introduction • Kohki Miki 🇯🇵 (@giginet) • Senior iOS Engineer @ LINE Corp (2022/7~) • Developer Experience team • OSS Maintainer : fastlane / Carthage / XcodeGen, etc…

Agenda • Our current build system and its issues • Introduce my new SwiftPM-based build tool • Handling Swift Packages as cachable and reusable.

State of LINE • LINE • A communication app used by 200 million users • 2,000,000+ lines of Swift Code (iOS)

Build Phases of LINE Prebuilt Tasks Building Application Bazel Xcode

Prebuilt Frameworks • Regular Swift, Obj-C OSS libraries • C/C++ libraries (e.g. Image processing, encryption) • Binary Frameworks • Separated Features • Code generation from API Schemes Prebuilt Tasks Bazel

Prebuilt Frameworks • Regular Swift, Obj-C OSS libraries • C/C++ libraries (e.g. Image processing, encryption) • Binary Frameworks • Separated Features • Code generation from API Schemes Prebuilt Tasks Bazel

Prebuilt Frameworks • Regular Swift, Obj-C OSS libraries • C/C++ libraries (e.g. Image processing, encryption) • Binary Frameworks • Separated Features • Code generation from API Schemes Prebuilt Tasks Bazel

Prebuilt Frameworks • Regular Swift, Obj-C OSS libraries • C/C++ libraries (e.g. Image processing, encryption) • Binary Frameworks • Separated Features • Code generation from API Schemes Prebuilt Tasks Bazel

Prebuilt Frameworks • Regular Swift, Obj-C OSS libraries • C/C++ libraries (e.g. Image processing, encryption) • Binary Frameworks • Separated Features • Code generation from API Schemes Prebuilt Tasks Bazel

Build time of each steps 926.42 seconds 503.4 seconds Prebuilt Tasks Building Application Bazel Xcode

Problems on maintainability • Tough to maintain Bazel • Implementing tasks in Starlark • No tests • Learning cost for Bazel • Large numbers of submodules • No dependency resolver • Abandoned dependencies Implementations of Bazel rules in Starlark

Problems on maintainability • Tough to maintain Bazel • Implementing tasks in Starlark • No tests • Learning cost for Bazel • Large numbers of submodules • No dependency resolver • Abandoned dependencies Submodules hell

Problems on maintainability • Tough to maintain Bazel • Implementing tasks in Starlark • No tests • Learning cost for Bazel • Large numbers of submodules • No dependency resolver • Abandoned dependencies http_archive( name = "APIKit", build_file = "//Configuration/repositories:APIKit.BUILD", sha256 = "0d19ebb5769ca464c7b10671ca1df21b8e9edad198f59e2d854c48cc580383 43", strip_prefix = "APIKit-5.0.0", url = "https://github.com/ishkawa/APIKit/archive/ 5.0.0.tar.gz", ) http_archive( name = "APNGKit", build_file = "//Configuration/repositories:APNGKit.BUILD", sha256 = "89245edaabcda2872141ecfcfad6d88fa5f0293af52a4cbcd2dd2fd6e3f931 99", strip_prefix = “APNGKit-2.2.1”, url = “https://github.com/onevcat/APNGKit/archive/ 2.2.1.tar.gz”, ) Dependency de f initions on our WORKSPACE

Build speeds •Bazel used to improve build time dramatically •However, Apple Silicon(M1) improves build type than Bazel Need more maintainable build work f low

Our requirements • Replacing the prebuilt task • Simple. based on the standard build system • Reusing build artifacts and caching well • Easy to develop and testable • Easy to update dependencies

Swift Package with Xcode • Swift Package integration for Xcode is the best solution • Maintained in the future • Most packages are supported

Cache Swift Packages with Xcode Xcode stores Swift Package dependencies in Derived Data It can be purged easily!

How to cache Swift Packages? Using Swift Package and Swift Package Manager(SwiftPM) is a good choice However, it’s dif f icult to keep build speed by using caches I developed a new tool! Scipio

Scipio - An awesome build tool - https://github.com/giginet/Scipio

Scipio • Developing new build tool Scipio • Convert Swift Packages to portable binaries • https://github.com/giginet/Scipio • Cache System • Support Static Frameworks • Inspired by Carthage

Basic ideas of Scipio • Follow the Apple standard build system • Integrate SwiftPM and XCFramework • Resolving Swift Packages with SwiftPM • Converting to XCFramework Swift Package XCFramework Convert

XCFrameworks • Binary distribution format • They can contain multiple frameworks for various architectures and platforms Library.xcframework for arm64 iOS devices for x86_64 iOS Simulators for arm64 Mac devices

Using Scipio Prepare a Package.swift describes all app’s dependencies // swift-tools-version: 5.7 import PackageDescription let package = Package( name: "Dependencies", platforms: [.iOS(.v13),], dependencies: [ .package(url: "https://github.com/apple/swift-log.git", from: "1.4.4"), .package(url: "https://github.com/apple/swift-collections.git", .upToNextMinor(from: "1.0.0")), ], targets: [ .target( name: "Dependencies", dependencies: [ .product(name: "Logging", package: "swift-log"), .product(name: "OrderedCollections", package: "swift-collections"), .product(name: "DequeModule", package: "swift-collections"), .product(name: "Collections", package: "swift-collections"), ]), ] )

Build all dependencies with Scipio $ scipio prepare /path/to/Dependencies 🔁 Resolving Dependencies... 🗑 Cleaning Dependencies... 📦 Building OrderedCollections for iOS 🚀 Combining into XCFramework... 📦 Building DequeModule for iOS 🚀 Combining into XCFramework... 📦 Building Collections for iOS 🚀 Combining into XCFramework... 📦 Building Logging for iOS 🚀 Combining into XCFramework... ❇ Succeeded.

1. Resolve and checkout package manifest with SwiftPM 2. Scipio generates Xcode Project from each package 3. Build the projects with xcodebuild 4. xcodebuild generates XCFramework How does Scipio work? Resolve manifests Fetch packages Generate Xcode Project Build XCFrameworks

Cache mechanism • Scipio supports cache system • Generate matadata within XCFrameworks • Called VersionFile

What is VersionFile { "buildOptions" : { "buildConfiguration" : "debug", "isDebugSymbolsEmbedded" : true, "frameworkType" : "static", "sdks" : [ "iOS" ], "isSimulatorSupported" : true }, "targetName" : "AwesomeKit", "clangVersion" : "clang-1400.0.29.102", "pin" : { "version" : "5.0.1", "revision" : "d458564516e5676af9c70b4f4b2a9178294f1bc6" } } • A JSON f ile contains build contexts • Scipio try to check this before building • Building will be skipped if existing framework is valid .AwesomeKit.version

Project cache • Just compare with VersionFile in a destination directory • If a valid VersionFile is exist, Scipio skips building AwesomeKit.xcframework .AwesomeKit.version Destination Directory $ scipio prepare /path/to/Dependencies 🔁 Resolving Dependencies... 🗑 Cleaning Dependencies... ✅ Valid OrderedCollections.xcframework is exists. Skip building. ✅ Valid DequeModule.xcframework is exists. Skip building. ✅ Valid Collections.xcframework is exists. Skip building. ✅ Valid Logging.xcframework is exists. Skip building. ❇ Succeeded.

Disk cache • Save XCFrameworks to a system cache directory • Compare SHA256 hash of VersionFiles • Restore same frameworks built in the past AwesomeKit.xcframework Destination Directory System Cache Directory AAAAAA.xcframework BBBBBB.xcframework Restore

Remote cache • Just replace storage with remotes (e.g. Amazon S3) • Share all caches among developers AwesomeKit.xcframework Destination Directory Remote object storage AAAAAA.xcframework BBBBBB.xcframework Restore 􀇃

Kind of Swift Packages • Swift Package can support various use cases • C Family language supports (SE-0038) • Binary Dependencies (SE-0272) • Artifact bundles (SE-0305)

SE-0038 C languages supports • Package Manager C Language Target Support • https://github.com/apple/swift-evolution/blob/main/ proposals/0038-swiftpm-c-language-targets.md • Create Swift Package for C/C++ libraries

SE-0272 Binary Framework • SE-0272 Package Manager Binary Dependencies • https://github.com/apple/swift-evolution/blob/main/ proposals/0272-swiftpm-binary-dependencies.md • Distribute prebuilt binaries in Swift Package let package = Package( name: "SomePackage", targets: [ .binaryTarget( name: "SomePackageLib", url: "https://github.com/some/package/releases/download/1.0.0/SomePackage-1.0.0.zip", checksum: "839F9F30DC13C30795666DD8F6FB77DD0E097B83D06954073E34FE5154481F7A" ), .binaryTarget( name: "SomeLibOnDisk", path: "artifacts/SomeLibOnDisk.zip" ) ] )

SE-0305 Artifact bundles • Package Manager Binary Target Improvements • https://github.com/apple/swift-evolution/ blob/main/proposals/0305-swiftpm-binary- target-improvements.md • Artifact bundle • A directory structure to distribute executables for multiple architectures in Swift Packages .artifactbundle ᵓ info.json ᵓ │ ᵓ │ │ ᵓ │ │ └ │ └ │ ᵓ │ └ ᵓ │ └ │ ᵓ │ └ │ ᴽ └ᴻ

Future works • We’re migrating Prebuilt frameworks to Swift Packages • Improve Scipio • Improving stability • Supporting Binary targets

Conclusion • Migrating our build system • Using Swift packages is a good solution • Created an OSS, Scipio • Feedbacks and Pull Requests are welcome • https://github.com/giginet/Scipio • Please look forward to our future work!

Thank you for listening