Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Preparing Large Java project for Kotlin adoption

Preparing Large Java project for Kotlin adoption

Preparing large Java codebase for Kotlin adoption.

How did we prepare ~700k LOC Java project (700+ modules) to adopt Kotlin:

- Achieving compile time null-safety in Java (ErrorProne + NullAway)
- Debugging, benchmarking and profiling ErrorProne Java compiler
- Linting Java and Kotlin code
- Setting up Gradle and Buck build systems
- Automatically formatting Java, Kotlin and other source files

Artem Zinnatullin

October 02, 2018

More Decks by Artem Zinnatullin

Other Decks in Programming


  1. DEFINE “LARGE PROJECT” •19106 text files •13561 Java files •698507

    lines of Java code w/o comments (700K) $ git clone # clean repo state $ cloc repo/
  2. DEFINE “LARGE PROJECT” “Monorepo” (2 big android apps in same

    git repo) Last week: •54 people committed code •264 PRs landed on master •626 commits across different branches •~200 PRs open at the same time
  3. BUILD SETUP 2 build systems: - Gradle -Most of CI

    builds -Release builds -Source of truth for configuration -Buck -Local builds (better performance and resource consumption) -Unit tests on CI -Generates IntelliJ project
  4. PLAN • Build support (both Gradle and Buck) • IDE

    support for Kotlin • Static Analysis for Kotlin • Null safety for Java • Automatic code formatting for Kotlin • Kotlin Annotation Processing?
  5. BUILD SUPPORT: GRADLE We’ve only faced 1 major build issue

    (already fixed) KT-15562 — Kotlin Compiler Daemon protocol violation in massively parallel builds https://youtrack.jetbrains.com/issue/KT-15562
  6. BUILD SUPPORT: GRADLE • 99% OK • Use Kotlin EAPs,

    help JetBrains • Search & report bugs on https://youtrack.jetbrains.com/issues/KT • Kotlin is 100% open source, including Kotlin Gradle Plugin https://github.com/JetBrains/kotlin

    Kotlin support is bundled into IJ CE since 2017. However, in our setup Buck generates IntelliJ project (not perfect) Broken source roots if module uses Kotlin

    should not be root! —> <sourceFolder url="file://$MODULE_DIR$" isTestSource="false" /> <excludeFolder url="file://$MODULE_DIR$/build" isTestSource="false" /> <!-- src/main is missing here! —> <sourceFolder url="file://$MODULE_DIR$/src/test" isTestSource="true" /> </content> Buck generates wrong .iml if module uses Kotlin
  9. So we wrote a tool in Kotlin to post- process

    .iml files for Kotlin modules after Buck IDE SUPPORT: INTELLIJ IDEA COMMUNITY EDITION
  10. Generating project: finished in 25.5 sec Fixing .iml files generated

    by Buck... lyft/buck-iml-fixer finished successfully, patched 697 files, skipped 2 files, took 0.882897147 seconds IDE SUPPORT: INTELLIJ IDEA COMMUNITY EDITION
  11. STATIC ANALYSIS: BEFORE • PMD • Simple checks • Only

    supports Java • Checkstyle • Simple style checker • Only supports Java • Android Lint • Very powerful framework for checks • Rich HTML/XML reports • Uses Universal AST • Supports Java • Supports Kotlin • XML • … • Arbitrary files
  12. • PMD • Checkstyle • Android Lint • Adopt newest

    versions with better Kotlin support • Apply to non-android modules • Rewrite existing custom rules with UAST • New custom rules should only use UAST • New custom Kotlin specific rules • Error Prone • Fix performance issues (we didn’t expect any, but…) • Adopt uber/NullAway — null safety in Java code • Detekt • Apply all of the ones that matter STATIC ANALYSIS: AFTER
  13. ERROR PRONE: NULLAWAY • Checks for null-safety violations in Java

    • null is 99% safe in Kotlin • But we have tons of Java code to interop with • That’s why we need null-safety in Java • Has reasonable defaults similar to Kotlin • Configurable • Runs as Error Prone Check • Can dive into library byte code to infer nullability
  14. NULLAWAY static void log(@Nullable Object x) { System.out.println(x.toString()); } $

    javac … error: [NullAway] dereferenced expression x is @Nullable System.out.println(x.toString()); https://github.com/uber/NullAway
  15. Median build time (Docker, Oracle JDK 8 1.8.0_161, Error Prone

    replaces JDK's javac) • baseline: 4.173 sec • Error Prone with default settings: 14.922 sec, 3.57x slower • Error Prone with most expensive checks disabled: 14.680 sec, 3.51x slower • Error Prone with all checks disabled: 10.358 sec, 2.48x slower ⚠ https://github.com/artem-zinnatullin/error-prone-performance ERROR PRONE: PERFORMANCE ISSUES
  16. Max "Used size" memory (Docker, Oracle JDK 8 1.8.0_161, Error

    Prone replaces JDK's javac) • baseline: 0.89 GB • Error Prone with default settings: 2.3 GB, 2.58x memory used • Error Prone with most expensive checks disabled: 2.08 GB, 2.3x memory used • Error Prone with all checks disabled: N/A (haven't measured yet) Looks like Parallel GC doesn't play well with Error Prone and max memory usage spikes significantly, please see below results with G1 GC on JDK 9. ERROR PRONE: PERFORMANCE ISSUES https://github.com/artem-zinnatullin/error-prone-performance
  17. • Error Prone is designed to run as compiler or

    a plugin to compiler • Build systems with incremental compilation like Gradle can avoid re-checking • Has a loooot checks that run by default, including Google-specific • Do you use Guava? • Do you use Protobuf? • Do you use Thruth? • Some checks are just complicated and slow • etc • etc ERROR PRONE: PERFORMANCE ISSUES
  18. -Disabling all checks still didn’t fix the issue, hmm -What

    do? -Chat with some people first to make sure you’re not insane -Create reproducible, isolated sample (thx Docker) -Run on different machines -Report to upstream! ERROR PRONE: PERFORMANCE ISSUES https://github.com/artem-zinnatullin/error-prone-performance
  19. https://github.com/tbroyer/gradle-errorprone-plugin/pull/68 Sad truth: the problem was in the ErrorProne Gradle

    Plugin It was creating a new ClassLoader to load Error Prone on each compilation thus JIT was unable to reuse optimized code ERROR PRONE: PERFORMANCE ISSUES
  20. • Warnings → Errors, no one looks at warnings •

    Exclude-list setup to enable on per-module basis + all new modules by default • Enable for all core modules to ease adoption • Configure common lateinit field methods ie onCreate(), onBindViews(), etc • Write docs/nullability.md • Small internal talk so everyone is on the same page NULL AWAY: INTEGRATION STRATEGY
  21. • be configurable/flexible • be consistent with IDE • be

    executable from command line • be fast • be isolated from local environment CODE FORMATTING Ideally it should
  22. •Downloads separate version of IntelliJ CE (once) •Removes some default

    plugins to start faster •Installs some plugins to format more languages •Sets custom idea.vmoptions optimized for short start/exit use-case •Sets custom idea.properties to isolate from existing IntellIJs •Uses code style settings from target project https://github.com/lyft/we-will-open-source-it-soon CODE FORMATTING
  23. ➜ instant-android git:(az/test) ✗ git commit -am 'Increase cpu limits'

    Formatting files... IntelliJ IDEA 2018.1.4, build IC-181.5087.20 Formatter Formatting frameworks/work-manager/src/…/java/com/…/workmanager/WorkerData.kt…OK Formatting frameworks/common-result/build.gradle...OK Formatting frameworks/rx/src/main/java/me/lyft/android/rx/RxBinder.java...OK Formatting gradle.properties...OK Formatting Dockerfile...Skipped, not supported. 4 file(s) formatted. Took 12 seconds. [az/test cc6ea59887] Increase cpu limits 5 files changed, 15 insertions(+), 2 deletions(-) CODE FORMATTING
  24. https://github.com/lyft/we-will-open-source-it-soon Problems — Performance •Typically takes ~6 seconds •Actual formatting

    takes less than a second •Potential fix — switch to AOT JVM (Java 10/11) •Potential fix — patch IntelliJ CE to optimize formatting entrypoint CODE FORMATTING
  25. Most of remaining challenges are about humans •Docs docs docs

    •Restrict Kotlin sdtlib with custom Lint rules •KAPT? •Some Buck patches Artem Zinnatullin, @artem_zin TODO