Computer Says No — Static Analysis and CI in a Kotlin World (Droidcon Italy 2019)

Computer Says No — Static Analysis and CI in a Kotlin World (Droidcon Italy 2019)

Presented with Neil Hutchison — @nnnneeeiil

Even though Android is a mature platform by now, the adoption of Kotlin at Google I/O 2017 brought about a sweeping wave of freshness and enthusiasm amongst developers. Regardless of what the language and design patterns we use when writing an app, there is only one way to ensure correctness and quality: testing, static analysis and continuous integration.
Many still think that setting up a CI for your project is hard, onerous, and not that useful, but we’re going to see how this is not true. Focusing on static analysis and unit testing, we’ll walk through setting up a continuous integration pipeline for a modern open source Android project using Gradle, CircleCI and Kotlin. We’ll see what benefits this brings to a codebase and how with a few tricks we can make sure external contributors adhere to the project code style, how we can prevent subtle bugs to sneak into the codebase, all with very little effort and zero budget.

You can find the Keynote sources here: https://www.dropbox.com/sh/mpmwy60vhph3cbo/AADSXZExFGhkBqEU_8DI88CBa?dl=0

Conference talk link: https://it.droidcon.com/2019/it/talks/415-computer-says-no-static-analysis-and-ci-in-a-kotlin-world

4580c218737149bf44d012a110612010?s=128

Sebastiano Poggi

April 04, 2019
Tweet

Transcript

  1. says COMPUTER SEBASTIANO POGGI @seebrock3r JetBrains NEIL HUTCHISON @nnnneeeiil Toyota

    Connected Europe no
  2. Catch those silly bugs early Static Analysis

  3. Code Quality • It’s important • Consistency • Never fix

    the same bug twice • Have less bugs in the first place • A safety net to catch easy mistakes
  4. Code Quality • How do you achieve a baseline of

    code quality? • Testing • Static Analysis • CI
  5. Testing • DO TEST! • Unit AND Integration • Too

    big of a topic for this talk
  6. Static Analysis • Tools that analyse code statically • Only

    look at code “at rest”, not at runtime • Check for common bugs/oversights • May involve analysing compiled bytecode • Check for code style violations
  7. Static Analysis • We come from Java • Some still

    stuck there • Java has a lot of static analysis tools • Most free/FOSS, but many commercial
  8. Static Analysis • The “holy trinity” • Findbugs (prefer Spotbugs)

    • PMD • Checkstyle • Plenty of others • Error Prone, Infer, Android Lint, …
  9. Static Analysis • And don’t forget your IDE! • Android

    Studio/IDEA inspections • In-editor and on-demand • Code formatter
  10. So what about Kotlin?

  11. Static Analysis • Android Lint works • IDEs still has

    inspections • Not as many/as complex as Java • But it’s getting better all the time The good
  12. Static Analysis • Almost nothing else from the Java world

    is compatible • No first-party tools from JetBrains • Apart from the IDE inspections • IDE Formatter isn’t often strict enough either The bad
  13. Static Analysis The hope — third party tools • ktlint

    • “An anti-bike shedding Kotlin Linter with built-in formatter” • Detekt • “Static code analysis for Kotlin” • SonarQube
  14. Static Analysis • “An anti-bike shedding Kotlin Linter with built-in

    formatter” • “Anti-bike shedding” • Avoids pointless discussions… • …by simply not offering customisation • “Linter” • Verifies code quality by looking for potential errors • Only looks at source code ktlint
  15. Static Analysis • Equivalent to Checkstyle, just not as configurable

    • Follows Kotlin official style guide • Support for Android Kotlin styling with --android • Auto-fix many violations with -F • Minimal configuration via .editorconfig ktlint
  16. Static Analysis • Static Code Analyser • Like PMD/Findbugs •

    Inspects code and AST • Does not inspect compiled bytecode • Limited type resolution capabilities Detekt
  17. Static Analysis • Configurable via YAML file • default-detekt-config.yml for

    defaults • Use :detektGenerateConfig to get started • buildUponDefaultConfig is recommended • Has IDEA/AS plugin • Shows violations in the editor Detekt
  18. Static Analysis • Java support is great • Kotlin is

    kinda supported • Via Android Lint and Detekt • Not nearly as comprehensively as other languages • Good if you are migrating to Kotlin and still have a bunch of Java • Keep an eye on everything from one place SonarQube
  19. Static Analysis • There is some good Kotlin stuff out

    there • No excuses not to use it! • Java has been around almost 24 years • Kotlin is much younger, less widespread (for now) • It needs time for tooling to catch up with Java • Listen to your IDE inspections, use all the tools you have Recap
  20. Empower your Static Analysis CI

  21. CI • Continuous Integration • Not a new concept —

    since the ‘90s • Popularised by XP (eXtreme Programming)
  22. CI • Continuously Integrate all changes • Catch issues ASAP

    • Leverage build system + compiler + tests + static analysis
  23. CI • CI is a practice • Software and services

    to implement it • We generally use “CI servers”
  24. Why adopting CI? • Take out the human factor •

    People forget to run checks • Save devs’ time • Run on-demand or scheduled • Nightlies, PRs, main branch, etc
  25. CI services • Two main kinds • Self-hosted • In-cloud

  26. CI services • Jenkins, TeamCity, etc • More enterprise-friendly and

    customisable • Require setup and maintenance Self-hosted
  27. CI services • Travis, CircleCI, Bitrise, GitLab CI, Google Cloud

    Build, etc • Easier to adopt, better for individuals and small companies • No/little setup and maintenance • Not as enterprise-friendly, and can be costly In-cloud
  28. CI services • Two main kinds • Self-hosted • In-cloud

    some blur the lines
  29. None
  30. No ca$h? No worries. • Most CI services offer a

    free tier • Open source (public) projects • Free build executor(s) • Free CPU time • Free for up to N users
  31. Anatomy of a CI • Nodes • Master • Agents/executors

    Master node • Orchestrates jobs • Delegates actual work • Self-hosted vs In-Cloud
  32. Anatomy of a CI • Nodes • Master • Agents/executors

    Master node • Orchestrates jobs • Delegates actual work • Self-hosted vs In-Cloud
  33. Anatomy of a CI • Nodes • Master • Agents/executors

    Agents/executors • Run actual jobs’ work • Machines or containers • Most commonly Linux …but there’s macOS & Windows too
  34. THE CI BUILD JOB BUILD JOB BUILD JOB The build

    queue
  35. THE CI BUILD JOB BUILD JOB BUILD JOB The build

    queue
  36. THE CI BUILD JOB BUILD JOB BUILD JOB The build

    queue
  37. THE CI BUILD JOB BUILD JOB The build queue

  38. THE CI BUILD JOB The build queue

  39. THE CI The build queue

  40. RELEASE ONLY :app:assemble :app:test :app:lint :app:detekt A typical job :app:ktlintCheck

  41. RELEASE ONLY :app:assemble :app:test :app:lint :app:detekt A typical job :app:ktlintCheck

  42. RELEASE ONLY :app:assemble :app:test :app:lint :app:detekt A typical job :app:ktlintCheck

  43. :app:assemble :app:test :app:lint :app:detekt :app:ktlintCheck :app:build A typical job

  44. :app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease

  45. :app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease

  46. :app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease

  47. :app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease

  48. :app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease

  49. :app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease

  50. :app:assembleRelease :app:testRelease :app:cAT :app:lintRelease

  51. :app:assembleRelease :app:testRelease :app:cAT :app:lintRelease T1

  52. T2 T1 :app:assembleRelease :app:testRelease :app:cAT :app:lintRelease

  53. Pipelines • Group of jobs executed in stages • Can

    be parallelised • Can take less time than monolithic builds • Supported almost everywhere in some form
  54. Pipelines • Group of jobs executed in stages • Can

    be parallelised • Can take less time than monolithic builds • Supported almost everywhere in some form Other names • Workflow • Build chain
  55. Pipelines • Group of jobs executed in stages • Can

    be parallelised • Can take less time than monolithic builds • Supported almost everywhere in some form YES. YES, YOU SHOULD
 USE PIPELINES.
  56. checkout warm up connected test unit test static analysis build

    apk Test different combinations! Pipelines
  57. checkout warm up connected test unit test static analysis build

    apk Test different combinations! Pipelines
  58. A case study CircleCI

  59. Why CircleCI? • Widely adopted in the community • Most

    cloud CIs work similarly • We have an example ready!
  60. CircleCI • Docker-based • Configured with YAML • We’ll use

    2.1 • Free tiers • 1x Linux executor • 4x plan for Open Source
  61. Our example • Squanchy — https://github.com/squanchy-dev/squanchy-android • FOSS conference app

    • 100% Kotlin • Unit tests, but no instrumented tests • Static analysis: Android Lint, Detekt, ktlint • Not very modular
  62. Getting started • Working Gradle build • Identify the tasks

    to execute • Both in Gradle and outside Gradle • Identify the secrets to inject • Use environment variables
  63. ./gradlew build

  64. Getting started • Working Gradle build • • Both in

    Gradle and outside Gradle • Identify the secrets to inject • Use environment variables Identify the tasks to execute
  65. Unit tests Static analysis :testRelease :lintRelease :detekt :ktlintCheck Warm up

    :downloadDependencies Checkout [not Gradle]
  66. Getting started • Working Gradle build • Identify the tasks

    to execute • Both in Gradle and outside Gradle • • Use environment variables Identify the secrets to inject
  67. API keys Play Services JSON Other values APPLICATION_ID FABRIC_API_KEY ALGOLIA_APPLICATION_ID

    ALGOLIA_API_KEY ALGOLIA_INDICES_PREFIX
  68. API keys Play Services JSON Other values APPLICATION_ID FABRIC_API_KEY ALGOLIA_APPLICATION_ID

    ALGOLIA_API_KEY ALGOLIA_INDICES_PREFIX
  69. Play Services JSON Other values APPLICATION_ID API keys FABRIC_API_KEY ALGOLIA_APPLICATION_ID

    ALGOLIA_API_KEY ALGOLIA_INDICES_PREFIX Protip! Use Novoda’s gradle-build-properties-plugin applicationId(applicationProps['applicationId'].or(envVars['APPLICATION_ID']).string) manifestPlaceholders += [ fabricApiKey: secretsProps['fabricApiKey'].or(envVars['FABRIC_API_KEY']).string ] resValueString 'app_name', applicationProps['applicationName'].or("Squanchy") resValueString 'algolia_application_id', applicationProps[‘algoliaId'] .or(envVars['ALGOLIA_APPLICATION_ID']) resValueString 'algolia_api_key', secretsProps[‘algoliaApiKey'] .or(envVars['ALGOLIA_API_KEY']) resValueString 'algolia_indices_prefix', applicationProps[‘algoliaIndicesPrefix'] .or(envVars[‘ALGOLIA_INDICES_PREFIX']) resValueString 'deeplink_scheme', applicationProps['deeplinkScheme'].or("squanchy")
  70. Getting started • Working Gradle build • Identify the tasks

    to execute • Both in Gradle and outside Gradle • Identify the secrets to inject • Use environment variables
  71. Configure CircleCI • YAML file • /.circleci/config.yml • Versioned •

    Currently 2.1
  72. version: 2.1 executors: android: working_directory: ~/squanchy docker: - image: circleci/android:api-28

    environment: ANDROID_HOME: /opt/android/sdk APPLICATION_ID: net.squanchy.example FABRIC_API_KEY: 0000000000000000000000000000000000000000 ALGOLIA_APPLICATION_ID: ABCDEFGH12 ALGOLIA_API_KEY: 00000000000000000000000000000000 ALGOLIA_INDICES_PREFIX: squanchy_dev commands: # Build Tools cache commands restore_build_tools_cache: steps: - restore_cache: name: Restore Android build tools cache keys: - v3-build-tools-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} save_build_tools_cache: steps: - save_cache: name: Save Android build tools cache paths: - /opt/android/sdk/build-tools key: v3-build-tools-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} # Gradle cache commands restore_gradle_cache: steps: - restore_cache: name: Restore Gradle dependencies cache keys: - v1-gradle-dependencies-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} - restore_cache: name: Restore Gradle wrapper cache keys: - v1-gradle-wrapper-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle/wrapper/gradle-wrapper.properties" }} save_gradle_cache: steps: - save_cache: name: Save Gradle dependencies cache paths: - ~/.gradle/caches key: v1-gradle-dependencies-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} - save_cache: name: Save Gradle wrapper cache paths: - ~/.gradle/wrapper key: v1-gradle-wrapper-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle/wrapper/gradle-wrapper.properties" }} # Android Gradle build cache commands restore_android_build_cache: steps: - restore_cache: name: Restore Android Gradle build cache keys: - v3-build-cache-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} save_android_build_cache: steps: - save_cache: name: Save Android Gradle build cache paths: - ~/.android/build-cache key: v3-build-cache-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} ensure_android_sdk_is_ready: steps: - run: name: Ensure Android SDK install is up-to-date command: workspace/repo/.circleci/ci-scripts/ensure-sdkmanager.sh download_gradle_dependencies: steps: - run: name: Download Gradle dependencies command: cd workspace/repo/ && ./gradlew downloadDependencies restore_workspace: steps: - attach_workspace: at: workspace jobs: checkout: executor: android steps: - checkout: path: workspace/repo # Prepare the container for the build - ensure_android_sdk_is_ready - run: name: Create mock Play Services JSON command: workspace/repo/.circleci/ci-scripts/ci-mock-google-services-setup.sh # Persist repo code - persist_to_workspace: root: workspace paths: - repo prepare_for_checks: executor: android steps: - restore_workspace - restore_gradle_cache - restore_android_build_cache - restore_build_tools_cache - download_gradle_dependencies - save_android_build_cache - save_gradle_cache - save_build_tools_cache # Persist built app code - persist_to_workspace: root: workspace paths: - repo/app/build static_analysis: executor: android steps: - restore_workspace - restore_gradle_cache - restore_android_build_cache - restore_build_tools_cache # See https://issuetracker.google.com/issues/62217354 for the parallelism option - run: name: Run static analysis command: cd workspace/repo && ./gradlew lintRelease detekt ktlintCheck -Djava.util.concurrent.ForkJoinPool.common.parallelism=2 # Collect static analysis reports as build artifacts - store_artifacts: path: workspace/repo/app/build/reports destination: reports tests: executor: android steps: - restore_workspace - restore_gradle_cache - restore_android_build_cache - restore_build_tools_cache # See https://issuetracker.google.com/issues/62217354 for the parallelism option - run: name: Run unit tests command: cd workspace/repo && ./gradlew testRelease -Djava.util.concurrent.ForkJoinPool.common.parallelism=2 # Collect JUnit test results - store_test_results: path: workspace/repo/app/build/test-results workflows: version: 2 build_and_test: jobs: - checkout - prepare_for_checks: requires: - checkout - static_analysis: requires: - prepare_for_checks - tests: requires: - prepare_for_checks
  73. version: 2.1 #... commands: jobs: workflows: #... #... #... executors:

  74. version: 2.1 #... commands: jobs: workflows: #... #... #... executors:

  75. workflows: #... jobs: #... commands: #... #... executors: version: 2.1

    android: working_directory: ~/squanchy docker: - image: circleci/android:api-28 environment: ANDROID_HOME: /opt/android/sdk APPLICATION_ID: net.squanchy.example FABRIC_API_KEY: 0000000000000000000000000000000000000000 ALGOLIA_APPLICATION_ID: ABCDEFGH12 ALGOLIA_API_KEY: 00000000000000000000000000000000 ALGOLIA_INDICES_PREFIX: squanchy_dev
  76. commands: #... executors: version: 2.1 android: working_directory: ~/squanchy docker: -

    image: circleci/android:api-28 environment: ANDROID_HOME: /opt/android/sdk APPLICATION_ID: net.squanchy.example FABRIC_API_KEY: 0000000000000000000000000000000000000000 ALGOLIA_APPLICATION_ID: ABCDEFGH12 ALGOLIA_API_KEY: 00000000000000000000000000000000 ALGOLIA_INDICES_PREFIX: squanchy_dev
  77. workflows: #... jobs: #... commands: #... #... executors: version: 2.1

    # Build Tools cache commands restore_build_tools_cache: steps: - restore_cache: name: Restore Android build tools cache keys: - v3-build-tools-{{ checksum "workspace/ repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum
  78. commands: #... # Build Tools cache commands restore_build_tools_cache: steps: -

    restore_cache: name: Restore Android build tools cache keys: - v3-build-tools-{{ checksum "workspace/ repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} save_build_tools_cache: steps: - save_cache: name: Save Android build tools cache paths:
  79. command_name: steps: - step_name: name: Description (shown in UI) other_stuff:

    - {depends on step}
  80. - save_cache: name: Saves path(s) to CircleCI cache paths: -

    /path/to/save keys: - v1-my-cache-{{ checksum "some/thing" }}
  81. - restore_cache: name: Restores from CircleCI cache keys: - v1-my-cache-{{

    checksum "some/thing" }}
  82. - attach_workspace: at: path/to/attach/to

  83. - run: name: Executes something in workspace command: my-script.sh param1

    param2
  84. #... workflows: jobs: #... commands: restore_build_tools_cache save_build_tools_cache restore_gradle_cache save_gradle_cache restore_android_build_cache

    save_android_build_cache ensure_android_sdk_is_ready download_gradle_dependencies restore_workspace
  85. workflows: #... jobs: #... commands: #... #... executors: version: 2.1

    checkout: executor: android steps: - checkout: path: workspace/repo
  86. jobs: #... checkout: executor: android steps: - checkout: path: workspace/repo

    # Prepare the container for the build - ensure_android_sdk_is_ready - run: name: Create mock Play Services JSON command: workspace/repo/.circleci/ci-scripts/ci-mock- google-services-setup.sh # Persist repo code - persist_to_workspace:
  87. job_name: executor: executor_name steps: - step_name: name: Description (shown in

    UI) other_stuff: - {depends on step}
  88. job_name: executor: executor_name steps: - step_name: name: Description (shown in

    UI) other_stuff: - {depends on step} - command_name
  89. workflows: #... jobs: #... commands: #... #... executors: version: 2.1

    checkout prepare_for_checks static_analysis tests
  90. workflows: #... jobs: #... commands: #... #... executors: version: 2.1

    version: 2 build_and_test: jobs:
  91. workflows: #... #... version: 2 build_and_test: jobs: - checkout -

    prepare_for_checks: requires: - checkout - static_analysis: requires: - prepare_for_checks - tests: requires: - prepare_for_checks
  92. workflows: #... #... version: 2 build_and_test: jobs: - checkout -

    prepare_for_checks: requires: - checkout - static_analysis: requires: - prepare_for_checks - tests: requires: - prepare_for_checks checkout static analysis prepare for checks tests Directed Acyclic Graph
  93. A CI run in detail checkout static analysis prepare for

    checks tests
  94. A CI run in detail checkout static analysis prepare for

    checks tests
  95. A CI run in detail checkout setup android sdk create

    mock gms json persist to workspace checkout
  96. A CI run in detail checkout setup android sdk create

    mock gms json persist to workspace checkout
  97. A CI run in detail checkout static analysis prepare for

    checks tests
  98. A CI run in detail restore from workspace restore caches

    download dependencies persist to workspace prepare for checks save caches
  99. A CI run in detail restore from workspace restore caches

    download dependencies persist to workspace prepare for checks save caches
  100. A CI run in detail checkout static analysis prepare for

    checks tests
  101. A CI run in detail restore from workspace restore caches

    run tasks store artefacts static analysis and tests
  102. A CI run in detail restore from workspace restore caches

    run tasks store artefacts static analysis and tests
  103. get stuff

  104. get stuff do stuff save stuff

  105. What are today’s lessons? Conclusions

  106. • Static analysis is good • Use static analysis! •

    Continuous Integration is good • Use a CI! • Static analysis is even better with a CI • Check out Squanchy • https://github.com/squanchy-dev/squanchy-android
  107. THANK YOU bit.ly/Computer-Says-No SEBASTIANO POGGI @seebrock3r JetBrains NEIL HUTCHISON @nnnneeeiil

    Toyota Connected Europe