Multi-project management tool 「gradle-multi-project-support」

Minseok Oh (LINE Corporation, Blockchain Dev 1 Software Engineer)

line / gradle-multi-project-sup (LINE Corporation, Blockchain Dev 1 Software Engineer)portは、モノプロジェクトでマルチプロジェクトまたはマルチアプリケーションを維持するためのLINEのGradleプラグインのOSSです。 Gradleを使用してモノリポジトリを設定するときに経験した問題とそのアプローチを紹介します。

line/gradle-multi-project-support is LINE's open source for Gradle plugins to maintain multi-projects or multi-applications in mono-repo. The problems we experienced when setting mono-repo up using Gradle and the approach to it will be introduced.

Transcript

1. Minseok Oh / 2020-10-07 gradle-multi-project-support Multi-project Management Tool
2. About me • September 2017, join LINE auth team as a server developer • From June 2020, working for blockchain platform • Interested in development productivity • Common keyboard mania around you
3. Mono-repo • In MSA, multiple source code repositories occur usually. • There can be a task required to write your change across two or more repositories. • Known issues in this case • Work several times for just one task. • It needs to check and manage which version is compatible with each related repository. (a.k.a dependency hell)
5. Mono-repo (Cont'd) • Consider mono-repo as an alternative • Mono-repo's advantages • Easy to reuse code • Room to simplify the management of dependencies • Atomic commits available • Large-scale code refactoring
6. Mono-repo (Cont'd) • Changes after mono-repo • Accelerated changes across multiple services • Relocation of features to change service structure • Extensive code refactoring to improve code quality
7. Mono-repo (Cont'd) • From multi-repo to mono-repo • There are some well-known problems.
 (ex: partial damage to semantic versioning) • However migration to mono-repo is agreed. • We believed that the increase in productivity brought by mono-repo will be significant.
8. Gradle • There was the time when we used two build tools when we migrating to mono-repo. • Maven, Gradle
9. Gradle (Cont'd) • As a result, integrated into Gradle. • Performance and developer experience were considered mainly • Gradle shows relatively better build performance. • Incrementality : Gradle avoids work by tracking input and output of tasks and only running what is necessary, and only processing files that changed when possible. • Build Cache: Reuses the build outputs of any other Gradle build with the same inputs, including between machines. • For better developer experience, • Gradle provides Kotlin-based DSL. • The programming language used for development to match the required for build process
10. Many duplications • After the service structure change and code refactoring, a lot of duplications were still found. • Client settings except for the business interfaces • Common features required by multiple services • Configuration classes, properties, and so on
11. Many duplications (Cont'd) • Extract the duplications into Gradle's sub-projects. • Use other sub-projects as dependencies • In this step, • Code duplication had been greatly reduced. • Reuse in mono-repo was increased.
12. Many duplications (Cont'd) • But, new duplications were found. • As written more sub-projects, the same build scripts appear multiple times. • Some sub-projects with Spring Boot auto-configuration had similar but slightly different settings. • Some sub-projects were omitted from the build script refactoring by human fault. • New approach was required.
14. Gradle's configure (Cont'd) • Order of running scripts in Gradle 1. settings.gradle(.kts) 2. build.gradle(.kts) in project root 3. build.gradle(.kts) in each sub-project
15. Gradle's configure (Cont'd) • Extract the duplications of build script using Gradle's configure. • Almost all build scripts were extracted except for some dependency declarations. • Root build script should know all sub-project's names and those characteristics.
16. Gradle's configure (Cont'd) • Introduce 'type' property • type property represents each sub-project's characteristics.
 (ex: java-lib, kotlin-lib, kotlin- boot-lib, java-boot-lib) • Sub-projects' names were eliminated in root build script by type property.
17. line/gradle-multi-project-support • Retrospective with members • Significant improvement in productivity • All the benefits of mono-repo we want • Easy to create new sub-projects using type property • Low readability of the build script using type property • As the number of sub-projects increases, • Difficult to determine the impact of changes. • Not easy anymore to create a release note.
18. line/gradle-multi-project-support (Cont'd) • Overcome the problems using Gradle's buildSrc. • Add functions to improve readability • Write Gradle plugins to print the affected Git commits for each sub-project.
19. line/gradle-multi-project-support (Cont'd) • And we released these codes as OSS(open source software). • Share approaches only to avoid creating new copy & paste. • The key idea is to allow users to frame mono-repo in their own style.
20. Wrap-up • Migrate multi-repo into mono-repo • Replace Maven with Gradle • Modularize common codes • Extract common build scripts • Improve readability of build script • Write some useful Gradle plugins
21. Give it a try! https://github.com/line/gradle-multi-project-support