Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Preparing Large Java project for Kotlin adoption

Preparing Large Java project for Kotlin adoption

Preparing large Java codebase for Kotlin adoption.

How did we prepare ~700k LOC Java project (700+ modules) to adopt Kotlin:

- Achieving compile time null-safety in Java (ErrorProne + NullAway)
- Debugging, benchmarking and profiling ErrorProne Java compiler
- Linting Java and Kotlin code
- Setting up Gradle and Buck build systems
- Automatically formatting Java, Kotlin and other source files

Artem Zinnatullin

October 02, 2018
Tweet

More Decks by Artem Zinnatullin

Other Decks in Programming

Transcript

  1. ARTEM ZINNATULLIN, @ARTEM_ZIN
    Preparing Large
    Java project for Kotlin
    adoption

    View Slide

  2. DEFINE “LARGE PROJECT”

    View Slide

  3. DEFINE “LARGE PROJECT”
    •19106 text files
    •13561 Java files
    •698507 lines of Java code w/o comments (700K)
    $ git clone # clean repo state
    $ cloc repo/

    View Slide

  4. DEFINE “LARGE PROJECT”
    •37990 Java files
    •63129167 lines of Java code (63M O_o)
    # After regular build

    View Slide

  5. DEFINE “LARGE PROJECT”
    $ find . -type f -name 'build.gradle' | wc -l
    709 modules

    View Slide

  6. DEFINE “LARGE PROJECT”
    “Monorepo” (2 big android apps in same git repo)
    Last week:
    •54 people committed code
    •264 PRs landed on master
    •626 commits across different branches
    •~200 PRs open at the same time

    View Slide

  7. BUILD SETUP

    View Slide

  8. BUILD SETUP
    2 build systems:
    - Gradle
    -Most of CI builds
    -Release builds
    -Source of truth for configuration
    -Buck
    -Local builds (better performance and resource consumption)
    -Unit tests on CI
    -Generates IntelliJ project

    View Slide

  9. PLAN

    View Slide

  10. PLAN
    • Build support (both Gradle and Buck)
    • IDE support for Kotlin
    • Static Analysis for Kotlin
    • Null safety for Java
    • Automatic code formatting for Kotlin
    • Kotlin Annotation Processing?

    View Slide

  11. BUILD SUPPORT

    View Slide

  12. BUILD SUPPORT: GRADLE
    Everything is good (almost)
    JetBrains maintains Kotlin Gradle
    Plugin

    View Slide

  13. BUILD SUPPORT: GRADLE
    We’ve only faced 1 major build issue
    (already fixed)
    KT-15562 — Kotlin Compiler Daemon
    protocol violation in massively parallel
    builds
    https://youtrack.jetbrains.com/issue/KT-15562

    View Slide

  14. BUILD SUPPORT: GRADLE
    https://youtrack.jetbrains.com/issue/KT-15562

    View Slide

  15. BUILD SUPPORT: GRADLE
    • 99% OK
    • Use Kotlin EAPs, help JetBrains
    • Search & report bugs on https://youtrack.jetbrains.com/issues/KT
    • Kotlin is 100% open source, including Kotlin Gradle Plugin
    https://github.com/JetBrains/kotlin

    View Slide

  16. IDE SUPPORT

    View Slide

  17. IDE SUPPORT: INTELLIJ IDEA COMMUNITY EDITION
    99% works because JetBrains
    Kotlin support is bundled into IJ CE since 2017.
    However, in our setup Buck generates IntelliJ project (not perfect)
    Broken source roots if module uses Kotlin

    View Slide

  18. IDE SUPPORT: INTELLIJ IDEA COMMUNITY EDITION

    View Slide

  19. IDE SUPPORT: INTELLIJ IDEA COMMUNITY EDITION

    View Slide

  20. IDE SUPPORT: INTELLIJ IDEA COMMUNITY EDITION




    View Slide

  21. So we wrote a tool in Kotlin to post-
    process .iml files for Kotlin modules after
    Buck
    IDE SUPPORT: INTELLIJ IDEA COMMUNITY EDITION

    View Slide

  22. Generating project: finished in 25.5 sec
    Fixing .iml files generated by Buck...
    lyft/buck-iml-fixer finished successfully,
    patched 697 files, skipped 2 files, took 0.882897147 seconds
    IDE SUPPORT: INTELLIJ IDEA COMMUNITY EDITION

    View Slide

  23. STATIC ANALYSIS

    View Slide

  24. STATIC ANALYSIS: BEFORE
    • PMD
    • Simple checks
    • Only supports Java
    • Checkstyle
    • Simple style checker
    • Only supports Java
    • Android Lint
    • Very powerful framework for checks
    • Rich HTML/XML reports
    • Uses Universal AST
    • Supports Java
    • Supports Kotlin
    • XML
    • …
    • Arbitrary files

    View Slide

  25. • PMD
    • Checkstyle
    • Android Lint
    • Adopt newest versions with better Kotlin support
    • Apply to non-android modules
    • Rewrite existing custom rules with UAST
    • New custom rules should only use UAST
    • New custom Kotlin specific rules
    • Error Prone
    • Fix performance issues (we didn’t expect any, but…)
    • Adopt uber/NullAway — null safety in Java code
    • Detekt
    • Apply all of the ones that matter
    STATIC ANALYSIS: AFTER

    View Slide

  26. ERROR PRONE: NULLAWAY
    • Checks for null-safety violations in Java
    • null is 99% safe in Kotlin
    • But we have tons of Java code to interop with
    • That’s why we need null-safety in Java
    • Has reasonable defaults similar to Kotlin
    • Configurable
    • Runs as Error Prone Check
    • Can dive into library byte code to infer nullability

    View Slide

  27. NULLAWAY
    static void log(@Nullable Object x) {
    System.out.println(x.toString());
    }
    $ javac …
    error: [NullAway] dereferenced expression x is @Nullable
    System.out.println(x.toString());
    https://github.com/uber/NullAway

    View Slide

  28. Median build time
    (Docker, Oracle JDK 8 1.8.0_161, Error Prone replaces JDK's javac)
    • baseline: 4.173 sec
    • Error Prone with default settings: 14.922 sec, 3.57x slower
    • Error Prone with most expensive checks disabled: 14.680 sec, 3.51x slower
    • Error Prone with all checks disabled: 10.358 sec, 2.48x slower ⚠
    https://github.com/artem-zinnatullin/error-prone-performance
    ERROR PRONE: PERFORMANCE ISSUES

    View Slide

  29. Max "Used size" memory
    (Docker, Oracle JDK 8 1.8.0_161, Error Prone replaces JDK's javac)
    • baseline: 0.89 GB
    • Error Prone with default settings: 2.3 GB, 2.58x memory used
    • Error Prone with most expensive checks disabled: 2.08 GB, 2.3x
    memory used
    • Error Prone with all checks disabled: N/A (haven't measured yet)
    Looks like Parallel GC doesn't play well with Error Prone and max
    memory usage spikes significantly, please see below results with G1
    GC on JDK 9.
    ERROR PRONE: PERFORMANCE ISSUES
    https://github.com/artem-zinnatullin/error-prone-performance

    View Slide

  30. ERROR PRONE: PERFORMANCE ISSUES
    https://github.com/artem-zinnatullin/error-prone-performance

    View Slide

  31. • Error Prone is designed to run as compiler or a plugin to
    compiler
    • Build systems with incremental compilation like Gradle
    can avoid re-checking
    • Has a loooot checks that run by default, including
    Google-specific
    • Do you use Guava?
    • Do you use Protobuf?
    • Do you use Thruth?
    • Some checks are just complicated and slow
    • etc
    • etc
    ERROR PRONE: PERFORMANCE ISSUES

    View Slide

  32. -Disabling all checks still didn’t fix the issue, hmm
    -What do?
    -Chat with some people first to make sure you’re
    not insane
    -Create reproducible, isolated sample (thx Docker)
    -Run on different machines
    -Report to upstream!
    ERROR PRONE: PERFORMANCE ISSUES
    https://github.com/artem-zinnatullin/error-prone-performance

    View Slide

  33. https://github.com/google/error-prone/issues/994
    ERROR PRONE: PERFORMANCE ISSUES

    View Slide

  34. https://github.com/tbroyer/gradle-errorprone-plugin/pull/68
    Sad truth: the problem was in the
    ErrorProne Gradle Plugin
    It was creating a new ClassLoader to load Error Prone on each
    compilation thus JIT was unable to reuse optimized code
    ERROR PRONE: PERFORMANCE ISSUES

    View Slide

  35. https://github.com/tbroyer/gradle-errorprone-plugin/pull/68
    ERROR PRONE: PERFORMANCE ISSUES

    View Slide

  36. https://github.com/tbroyer/gradle-errorprone-plugin/pull/68
    ERROR PRONE: PERFORMANCE ISSUES, ~166% WIN

    View Slide

  37. • Warnings → Errors, no one looks at warnings
    • Exclude-list setup to enable on per-module
    basis + all new modules by default
    • Enable for all core modules to ease adoption
    • Configure common lateinit field methods ie
    onCreate(), onBindViews(), etc
    • Write docs/nullability.md
    • Small internal talk so everyone is on the same
    page
    NULL AWAY: INTEGRATION STRATEGY

    View Slide

  38. CODE
    FOR
    MATTING
    {
    }

    View Slide

  39. • be configurable/flexible
    • be consistent with IDE
    • be executable from command line
    • be fast
    • be isolated from local environment
    CODE FORMATTING
    Ideally it should

    View Slide

  40. So we’ve one!
    CODE FORMATTING
    built*

    View Slide

  41. •Downloads separate version of IntelliJ CE (once)
    •Removes some default plugins to start faster
    •Installs some plugins to format more languages
    •Sets custom idea.vmoptions optimized for short
    start/exit use-case
    •Sets custom idea.properties to isolate from
    existing IntellIJs
    •Uses code style settings from target project
    https://github.com/lyft/we-will-open-source-it-soon
    CODE FORMATTING

    View Slide

  42. https://github.com/lyft/we-will-open-source-it-soon
    Integration:
    •git pre-commit hook
    •only formats files that are about to be committed

    CODE FORMATTING

    View Slide

  43. ➜ instant-android git:(az/test) ✗ git commit -am 'Increase cpu limits'
    Formatting files...
    IntelliJ IDEA 2018.1.4, build IC-181.5087.20 Formatter
    Formatting frameworks/work-manager/src/…/java/com/…/workmanager/WorkerData.kt…OK
    Formatting frameworks/common-result/build.gradle...OK
    Formatting frameworks/rx/src/main/java/me/lyft/android/rx/RxBinder.java...OK
    Formatting gradle.properties...OK
    Formatting Dockerfile...Skipped, not supported.
    4 file(s) formatted.
    Took 12 seconds.
    [az/test cc6ea59887] Increase cpu limits
    5 files changed, 15 insertions(+), 2 deletions(-)
    CODE FORMATTING

    View Slide

  44. https://github.com/lyft/we-will-open-source-it-soon
    Problems — Performance
    •Typically takes ~6 seconds
    •Actual formatting takes less than a second
    •Potential fix — switch to AOT JVM (Java 10/11)
    •Potential fix — patch IntelliJ CE to optimize
    formatting entrypoint
    CODE FORMATTING

    View Slide

  45. RESULTS

    View Slide

  46. RESULTS
    - 303 .kt files
    - 18k LOC
    - 56 modules use Kotlin

    View Slide

  47. TODO

    View Slide

  48. Most of remaining challenges are about humans
    •Docs docs docs
    •Restrict Kotlin sdtlib with custom Lint rules
    •KAPT?
    •Some Buck patches
    Artem Zinnatullin, @artem_zin
    TODO

    View Slide