Standardizing build system using modern Swift Packages in LINE

Mobile DevOps Summit 2022
https://www.mobiledevops.io/summit/agenda/session/1022649

https://github.com/giginet/Scipio

Transcript

1. Standardizing build system using
   modern Swift Packages in LINE
   2022-11-10 Mobile DevOps Summit 2022
   Kohki Miki (@giginet)
   
   
   LINE Corp
   

   View Slide

2. Self Introduction
   • Kohki Miki 🇯🇵 (@giginet)
   
   
   • Senior iOS Engineer @ LINE Corp (2022/7~)
   
   
   • Developer Experience team
   
   
   • OSS Maintainer : fastlane / Carthage /
   XcodeGen, etc…
   

   View Slide

3. Agenda
   • Our current build system and its issues
   
   
   • Introduce my new SwiftPM-based build tool
   
   
   • Handling Swift Packages as cachable and reusable.
   

   View Slide

4. State of LINE
   • LINE
   
   
   • A communication app used by 200 million users
   
   
   • 2,000,000+ lines of Swift Code (iOS)
   

   View Slide

5. Build Phases of LINE
   Prebuilt Tasks Building Application
   Bazel Xcode
   

   View Slide

6. Prebuilt Frameworks
   • Regular Swift, Obj-C OSS libraries
   
   
   • C/C++ libraries (e.g. Image processing,
   encryption)
   
   
   • Binary Frameworks
   
   
   • Separated Features
   
   
   • Code generation from API Schemes
   Prebuilt Tasks
   Bazel
   

   View Slide

7. Prebuilt Frameworks
   • Regular Swift, Obj-C OSS libraries
   
   
   • C/C++ libraries (e.g. Image processing,
   encryption)
   
   
   • Binary Frameworks
   
   
   • Separated Features
   
   
   • Code generation from API Schemes
   Prebuilt Tasks
   Bazel
   

   View Slide

8. Prebuilt Frameworks
   • Regular Swift, Obj-C OSS libraries
   
   
   • C/C++ libraries (e.g. Image processing,
   encryption)
   
   
   • Binary Frameworks
   
   
   • Separated Features
   
   
   • Code generation from API Schemes
   Prebuilt Tasks
   Bazel
   

   View Slide

9. Prebuilt Frameworks
   • Regular Swift, Obj-C OSS libraries
   
   
   • C/C++ libraries (e.g. Image processing,
   encryption)
   
   
   • Binary Frameworks
   
   
   • Separated Features
   
   
   • Code generation from API Schemes
   Prebuilt Tasks
   Bazel
   

   View Slide

10. Prebuilt Frameworks
    • Regular Swift, Obj-C OSS libraries
    
    
    • C/C++ libraries (e.g. Image processing,
    encryption)
    
    
    • Binary Frameworks
    
    
    • Separated Features
    
    
    • Code generation from API Schemes
    Prebuilt Tasks
    Bazel
    

    View Slide

11. Build time of each steps
    926.42 seconds 503.4 seconds
    Prebuilt Tasks Building Application
    Bazel Xcode
    

    View Slide

12. Problems on maintainability
    • Tough to maintain Bazel
    
    
    • Implementing tasks in Starlark
    
    
    • No tests
    
    
    • Learning cost for Bazel
    
    
    • Large numbers of submodules
    
    
    • No dependency resolver
    
    
    • Abandoned dependencies
    Implementations of Bazel rules in Starlark
    

    View Slide

13. Problems on maintainability
    • Tough to maintain Bazel
    
    
    • Implementing tasks in Starlark
    
    
    • No tests
    
    
    • Learning cost for Bazel
    
    
    • Large numbers of submodules
    
    
    • No dependency resolver
    
    
    • Abandoned dependencies
    Submodules hell
    

    View Slide

14. Problems on maintainability
    • Tough to maintain Bazel
    
    
    • Implementing tasks in Starlark
    
    
    • No tests
    
    
    • Learning cost for Bazel
    
    
    • Large numbers of submodules
    
    
    • No dependency resolver
    
    
    • Abandoned dependencies
    http_archive(
    name = "APIKit",
    build_file = "//Configuration/repositories:APIKit.BUILD",
    sha256 =
    "0d19ebb5769ca464c7b10671ca1df21b8e9edad198f59e2d854c48cc580383
    43",
    strip_prefix = "APIKit-5.0.0",
    url = "https://github.com/ishkawa/APIKit/archive/
    5.0.0.tar.gz",
    )
    http_archive(
    name = "APNGKit",
    build_file = "//Configuration/repositories:APNGKit.BUILD",
    sha256 =
    "89245edaabcda2872141ecfcfad6d88fa5f0293af52a4cbcd2dd2fd6e3f931
    99",
    strip_prefix = “APNGKit-2.2.1”,
    url = “https://github.com/onevcat/APNGKit/archive/
    2.2.1.tar.gz”,
    )
    Dependency de
    f
    initions on our WORKSPACE
    

    View Slide

15. Build speeds
    •Bazel used to improve build time dramatically
    
    
    •However, Apple Silicon(M1) improves build type than Bazel
    Need more maintainable build work
    f
    low
    

    View Slide

16. Our requirements
    • Replacing the prebuilt task
    
    
    • Simple. based on the standard build system
    
    
    • Reusing build artifacts and caching well
    
    
    • Easy to develop and testable
    
    
    • Easy to update dependencies
    

    View Slide

17. Swift Package with Xcode
    • Swift Package integration for Xcode is the best solution
    
    
    • Maintained in the future
    
    
    • Most packages are supported
    

    View Slide

18. Cache Swift Packages with Xcode
    Xcode stores Swift Package dependencies in Derived Data
    
    
    It can be purged easily!
    

    View Slide

19. How to cache Swift Packages?
    Using Swift Package and Swift Package Manager(SwiftPM) is a
    good choice
    
    
    However, it’s dif
    f
    icult to keep build speed by using caches
    I developed a new tool!
    
    
    Scipio
    

    View Slide

20. Scipio
    
    
    - An awesome build tool -
    
    
    https://github.com/giginet/Scipio
    

    View Slide

21. Scipio
    • Developing new build tool Scipio
    
    
    • Convert Swift Packages to portable binaries
    
    
    • https://github.com/giginet/Scipio
    
    
    • Cache System
    
    
    • Support Static Frameworks
    
    
    • Inspired by Carthage
    

    View Slide

22. Basic ideas of Scipio
    • Follow the Apple standard build system
    
    
    • Integrate SwiftPM and XCFramework
    
    
    • Resolving Swift Packages with SwiftPM
    
    
    • Converting to XCFramework
    Swift Package XCFramework
    Convert
    

    View Slide

23. XCFrameworks
    • Binary distribution format
    
    
    • They can contain multiple frameworks for various
    architectures and platforms
    Library.xcframework
    for arm64 iOS devices
    for x86_64 iOS Simulators
    for arm64 Mac devices
    

    View Slide

24. Using Scipio
    Prepare a Package.swift describes all app’s dependencies
    // swift-tools-version: 5.7
    import PackageDescription
    let package = Package(
    name: "Dependencies",
    platforms: [.iOS(.v13),],
    dependencies: [
    .package(url: "https://github.com/apple/swift-log.git",
    from: "1.4.4"),
    .package(url: "https://github.com/apple/swift-collections.git",
    .upToNextMinor(from: "1.0.0")),
    ],
    targets: [
    .target(
    name: "Dependencies",
    dependencies: [
    .product(name: "Logging", package: "swift-log"),
    .product(name: "OrderedCollections", package: "swift-collections"),
    .product(name: "DequeModule", package: "swift-collections"),
    .product(name: "Collections", package: "swift-collections"),
    ]),
    ]
    )
    

    View Slide

25. Build all dependencies with Scipio
    $ scipio prepare /path/to/Dependencies
    
    
    🔁 Resolving Dependencies...
    
    
    🗑 Cleaning Dependencies...
    
    
    📦 Building OrderedCollections for iOS
    
    
    🚀 Combining into XCFramework...
    
    
    📦 Building DequeModule for iOS
    
    
    🚀 Combining into XCFramework...
    
    
    📦 Building Collections for iOS
    
    
    🚀 Combining into XCFramework...
    
    
    📦 Building Logging for iOS
    
    
    🚀 Combining into XCFramework...
    
    
    ❇ Succeeded.
    

    View Slide

26. 1. Resolve and checkout package
    manifest with SwiftPM
    
    
    2. Scipio generates Xcode Project
    from each package
    
    
    3. Build the projects with xcodebuild
    
    
    4. xcodebuild generates
    XCFramework
    How does Scipio work?
    Resolve manifests
    Fetch packages
    Generate Xcode Project
    Build XCFrameworks
    

    View Slide

27. Cache mechanism
    • Scipio supports cache system
    
    
    • Generate matadata within XCFrameworks
    
    
    • Called VersionFile
    

    View Slide

28. What is VersionFile
    {
    "buildOptions" : {
    "buildConfiguration" : "debug",
    "isDebugSymbolsEmbedded" : true,
    "frameworkType" : "static",
    "sdks" : [
    "iOS"
    ],
    "isSimulatorSupported" : true
    },
    "targetName" : "AwesomeKit",
    "clangVersion" : "clang-1400.0.29.102",
    "pin" : {
    "version" : "5.0.1",
    "revision" :
    "d458564516e5676af9c70b4f4b2a9178294f1bc6"
    }
    }
    • A JSON
    f
    ile contains build
    contexts
    
    
    • Scipio try to check this before
    building
    
    
    • Building will be skipped if
    existing framework is valid
    .AwesomeKit.version
    

    View Slide

29. Project cache
    • Just compare with VersionFile in a
    destination directory
    
    
    • If a valid VersionFile is exist,
    Scipio skips building
    AwesomeKit.xcframework
    .AwesomeKit.version
    Destination Directory
    $ scipio prepare /path/to/Dependencies
    
    
    🔁 Resolving Dependencies...
    
    
    🗑 Cleaning Dependencies...
    
    
    ✅ Valid OrderedCollections.xcframework is exists. Skip building.
    
    
    ✅ Valid DequeModule.xcframework is exists. Skip building.
    
    
    ✅ Valid Collections.xcframework is exists. Skip building.
    
    
    ✅ Valid Logging.xcframework is exists. Skip building.
    
    
    ❇ Succeeded.
    
    
    

    View Slide

30. Disk cache
    • Save XCFrameworks to a system
    cache directory
    
    
    • Compare SHA256 hash of
    VersionFiles
    
    
    • Restore same frameworks built in
    the past
    AwesomeKit.xcframework
    Destination Directory
    System Cache Directory
    AAAAAA.xcframework
    BBBBBB.xcframework
    Restore
    

    View Slide

31. Remote cache
    • Just replace storage with
    remotes (e.g. Amazon S3)
    
    
    • Share all caches among
    developers
    AwesomeKit.xcframework
    Destination Directory
    Remote object storage
    AAAAAA.xcframework
    BBBBBB.xcframework
    Restore
    􀇃
    

    View Slide

32. Kind of Swift Packages
    • Swift Package can support various use cases
    
    
    • C Family language supports (SE-0038)
    
    
    • Binary Dependencies (SE-0272)
    
    
    • Artifact bundles (SE-0305)
    

    View Slide

33. SE-0038 C languages supports
    • Package Manager C Language Target Support
    
    
    • https://github.com/apple/swift-evolution/blob/main/
    proposals/0038-swiftpm-c-language-targets.md
    
    
    • Create Swift Package for C/C++ libraries
    

    View Slide

34. SE-0272 Binary Framework
    • SE-0272 Package Manager Binary Dependencies
    
    
    • https://github.com/apple/swift-evolution/blob/main/
    proposals/0272-swiftpm-binary-dependencies.md
    
    
    • Distribute prebuilt binaries in Swift Package
    let package = Package(
    name: "SomePackage",
    targets: [
    .binaryTarget(
    name: "SomePackageLib",
    url: "https://github.com/some/package/releases/download/1.0.0/SomePackage-1.0.0.zip",
    checksum: "839F9F30DC13C30795666DD8F6FB77DD0E097B83D06954073E34FE5154481F7A"
    ),
    .binaryTarget(
    name: "SomeLibOnDisk",
    path: "artifacts/SomeLibOnDisk.zip"
    )
    ]
    )
    

    View Slide

35. SE-0305 Artifact bundles
    • Package Manager Binary Target Improvements
    
    
    • https://github.com/apple/swift-evolution/
    blob/main/proposals/0305-swiftpm-binary-
    target-improvements.md
    
    
    • Artifact bundle
    
    
    • A directory structure to distribute
    executables for multiple architectures in
    Swift Packages
    .artifactbundle
    
    
    ᵓ info.json
    
    
    ᵓ
    
    
    │ ᵓ
    
    
    │ │ ᵓ
    
    
    │ │ └ f
    iles>
    
    
    │ └
    
    
    │ ᵓ
    
    
    │ └ f
    iles>
    
    
    ᵓ
    
    
    │ └
    
    
    │ ᵓ
    
    
    │ └ f
    iles>
    
    
    │
    
    
    ᴽ └ᴻ
    

    View Slide

36. Future works
    • We’re migrating Prebuilt frameworks to Swift Packages
    
    
    • Improve Scipio
    
    
    • Improving stability
    
    
    • Supporting Binary targets
    

    View Slide

37. Conclusion
    • Migrating our build system
    
    
    • Using Swift packages is a good solution
    
    
    • Created an OSS, Scipio
    
    
    • Feedbacks and Pull Requests are welcome
    
    
    • https://github.com/giginet/Scipio
    
    
    • Please look forward to our future work!
    

    View Slide

38. Thank you for listening
    

    View Slide