Understanding Incremental Processing in the JVM World

Presented at Dutch Android User Group January 2026 Meetup
https://www.meetup.com/dutch-aug/events/312882769/

Transcript

1. Understanding Incremental Processing in the JVM World Subhrajyoti Sen Software Engineer @ Adyen GDE for Android @iamsubhrajyoti
2. XKCD: Compiling
3. The Journey 1. Java 2. Ant 3. Gradle 4. Kotlin
4. Java javac does minimal incremental processing on it’s own Relies on the build tool/IDE to orchestrate incremental processing
5. Ant - <javac> task Compile a .java file only if: • .java doesn’t have a corresponding .class file, or • .java is newer than the .class file - based on timestamps
6. Ant - <javac> task Compile a .java file only if: • .java doesn’t have a corresponding .class file, or • .java is newer than the .class file - based on timestamps This can be further tweaked with include/exclude attributes
7. Ant - <depend> task • Reads the .class files instead of source files • From the information encoded in class files, determines the dependencies of the class files
8. Dependency Graph
9. Dependency Graph - Inverted
10. Remove D.class
11. Without closure set
12. With closure set
13. Gradle
14. Gradle abstract class UppercaseTask : DefaultTask() { @get:InputFile @get:PathSensitive(PathSensitivity.RELATIVE) abstract val inputFile: RegularFileProperty @get:OutputFile abstract val outputFile: RegularFileProperty @TaskAction fun process() {} }
15. Gradle tasks.register<UppercaseTask>("makeUppercase") { // Specify which file to watch inputFile.set(layout.projectDirectory.file("source.txt")) // Specify where to put the result outputFile.set(layout.buildDirectory.file("generated/uppercase.txt")) }
16. Gradle abstract class UppercaseTask : DefaultTask() { @get:Incremental @get:InputFile @get:PathSensitive(PathSensitivity.RELATIVE) abstract val inputFile: RegularFileProperty }
17. How does Gradle remember?
18. How does Gradle remember? For each input and output, Gradle creates a fingerprint
19. How does Gradle remember? For each input and output, Gradle creates a fingerprint The fingerprint has the hash of the input files along with some metadata.
20. How does Gradle remember? For each input and output, Gradle creates a fingerprint The fingerprint has the hash of the input files along with some metadata. Once the task runs, Gradle calculates the fingerprint of the output.
21. How does Gradle remember? For each input and output, Gradle creates a fingerprint The fingerprint has the hash of the input files along with some metadata. Once the task runs, Gradle calculates the fingerprint of the output. When the task is run again, Gradle makes new fingerprints and compares with old fingerprints.
22. How does Gradle remember? For each input and output, Gradle creates a fingerprint The fingerprint has the hash of the input files along with some metadata. Once the task runs, Gradle calculates the fingerprint of the output. When the task is run again, Gradle makes new fingerprints and compares with old fingerprints. If fingerprints don’t match, the task is re-run
23. Compilation Avoidance
24. Compilation Avoidance If a dependent project has changed in an ABI-compatible way (only its private API has changed), then Java compilation tasks will be up-to-date.
25. Compilation Avoidance If a dependent project has changed in an ABI-compatible way (only its private API has changed), then Java compilation tasks will be up-to-date. This means that if project A depends on project B and a class in B is changed in an ABI-compatible way (typically, changing only the body of a method), then Gradle won’t recompile A.
26. What’s an ABI?
27. What’s an ABI? class UserAccount( val username: String, private val securityCode: Int ) { fun login() { println("Logging in $username...") } private fun validateCode(): Boolean = true }
28. ABI public final class UserAccount { public UserAccount(java.lang.String, int); public final java.lang.String getUsername(); public final void login(); }
35. Kotlin Incremental Processing
36. Kotlin Incremental Processing Kotlin assists Gradle in incremental processing. We will focus on Kotlin/JVM implementation
37. Kotlin Incremental Processing Kotlin assists Gradle in incremental processing. We will focus on Kotlin/JVM implementation Kotlin relies on classpath snapshots to determine whether compilation is necessary.
38. Kotlin Incremental Processing Kotlin assists Gradle in incremental processing. We will focus on Kotlin/JVM implementation Kotlin relies on classpath snapshots to determine whether compilation is necessary. Classpath snapshots also rely on the ABI surface.
39. Types of classpath snapshots
40. Types of classpath snapshots Fine-grained snapshots: Includes information about class members. When changes are detected, only classes affected by the changes are recompiled.
41. Types of classpath snapshots Fine-grained snapshots: Includes information about class members. When changes are detected, only classes affected by the changes are recompiled. Coarse snapshots: only contains the ABI hash. When the ABI changes, the compiler recompiles all classes that depend on the changed class. This is useful for libraries and external dependencies. The compiler also keep coarse snapshots of .jar files.
42. Incremental Annotation Processing - Gradle
43. Incremental Annotation Processing - Gradle Gradle supports 2 modes of annotation processing: 1. Aggregating 2. Isolating
44. Incremental Annotation Processing - Gradle Gradle supports 2 modes of annotation processing: 1. Aggregating 2. Isolating A processor can be marked as “dynamic” if it needs to decide the nature at runtime.
45. Isolating
46. Isolating 1-1 or 1-many relationship
47. Isolating 1-1 or 1-many relationship Example: Generating a `<Type>Repository` for each class annotated with `@Entity`
48. Isolating 1-1 or 1-many relationship Example: Generating a `<Type>Repository` for each class annotated with `@Entity` All actions must be based on information reachable via the Abstract Syntax Tree(AST)
49. Isolating 1-1 or 1-many relationship Example: Generating a `<Type>Repository` for each class annotated with `@Entity` All actions must be based on information reachable via the Abstract Syntax Tree(AST) If it’s based on anything outside the AST, it should be aggregating
50. Isolating 1-1 or 1-many relationship Example: Generating a `<Type>Repository` for each class annotated with `@Entity` All actions must be based on information reachable via the Abstract Syntax Tree(AST) If it’s based on anything outside the AST, it should be aggregating Every generated file must be mapped to 1 originating file
51. Aggregating
52. Aggregating Many-1 or Many-many relationship
53. Aggregating Many-1 or Many-many relationship Example: Generating a `StartupRegistry` class for each class annotated with `@Startup`
54. Incremental Annotation Processing - KSP
55. Incremental Annotation Processing - KSP KSP relies on a mapping of input sources and generated output.
56. Incremental Annotation Processing - KSP KSP relies on a mapping of input sources and generated output. Each output needs to be mapped to a corresponding set of KSNode
57. Incremental Annotation Processing - KSP KSP relies on a mapping of input sources and generated output. Each output needs to be mapped to a corresponding set of KSNode By default, KSP tracks the classpath in addition to Java and Kotlin sources.
58. Incremental Annotation Processing - KSP KSP relies on a mapping of input sources and generated output. Each output needs to be mapped to a corresponding set of KSNode By default, KSP tracks the classpath in addition to Java and Kotlin sources. To disable classpath tracking,you can use `ksp.incremental.intermodule=false`
59. Incremental Annotation Processing - KSP You can get a KSNode using the following: • Resolver.getAllFiles • Resolver.getAllSymbolsWithAnnotation • Resolver.getClassDeclarationByName • Resolver.getDeclarationsFromPackage
60. KSP Output Modes
61. KSP Output Modes KSP supports two modes, similar to Gradle annotation processing: • Aggregating • Isolating
62. KSP Output Modes KSP supports two modes, similar to Gradle annotation processing: • Aggregating • Isolating Unlike Gradle annotation processing, KSP marks each output as either isolating or aggregating and not the entire processor.
63. KSP Output Modes KSP supports two modes, similar to Gradle annotation processing: • Aggregating • Isolating Unlike Gradle annotation processing, KSP marks each output as either isolating or aggregating and not the entire processor. Unlike Java, an Isolating output can define multiple source files.
64. Isolating Mode val dependencies = Dependencies( aggregating = false, listOf("ParkingEntity") ) val os = codeGenerator.createNewFile( dependencies = dependencies, packageName = "com.example.generated", fileName = "ParkingRepository" )
65. Aggregating Mode val dependencies = Dependencies( aggregating = true, listOf("StartupWorker1", "StartupWorker2) ) val os = codeGenerator.createNewFile( dependencies = dependencies, packageName = "com.example.generated", fileName = "StartupRegistry" )
66. Dirtiness Propagation 1. If an input file is changed, it will be reprocessed. 2. If an input file is changed and it’s associated with an output, then all other input files associated with that output will also be reprocessed 3. All input files associated with at least one aggregating output will be reprocessed
67. But Code? class EntityProcessor( //.. dependencies ): SymbolProcessor { override fun process(resolver: Resolver): List<Annotated> { val files = resolver.getAllFiles() } }
68. Thank You @iamsubhrajyoti