Computer Says No — Static Analysis and CI in a Kotlin World (App Builders 2019)

Computer Says No — Static Analysis and CI in a Kotlin World (App Builders 2019)

Even though Android is a mature platform by now, the adoption of Kotlin at Google I/O 2017 brought about a sweeping wave of freshness and enthusiasm amongst developers. Regardless of what the language and design patterns we use when writing an app, there is only one way to ensure correctness and quality: testing, static analysis and continuous integration.
Many still think that setting up a CI for your project is hard, onerous, and not that useful, but we’re going to see how this is not true. Focusing on static analysis and unit testing, we’ll walk through setting up a continuous integration pipeline for a modern open source Android project using Gradle, CircleCI and Kotlin. We’ll see what benefits this brings to a codebase and how with a few tricks we can make sure external contributors adhere to the project code style, how we can prevent subtle bugs to sneak into the codebase, all with very little effort and zero budget.

A screen recording is available here: https://youtu.be/pdvBShEectk
You can find the Keynote sources here: https://www.dropbox.com/sh/84m895p4c0lobzt/AABlnA3H2V_kjfoJ-pLl-fQaa?dl=0

Conference talk link: https://appbuilders.ch/#schedule

4580c218737149bf44d012a110612010?s=128

Sebastiano Poggi

April 30, 2019
Tweet

Transcript

  1. 3.

    Code Quality • It’s important • Consistency • Never fix

    the same bug twice • Have less bugs in the first place • A safety net to catch easy mistakes
  2. 4.

    Code Quality • How do you achieve a baseline of

    code quality? • Testing • Static Analysis • CI
  3. 6.

    Static Analysis • Tools that analyse code statically • Only

    look at code “at rest”, not at runtime • Check for common bugs/oversights • May involve analysing compiled bytecode • Check for code style violations
  4. 7.

    Static Analysis • We come from Java • Some still

    stuck there • Java has a lot of static analysis tools • Most free/FOSS, but many commercial
  5. 8.

    Static Analysis • The “holy trinity” • Findbugs (prefer Spotbugs)

    • PMD • Checkstyle • Plenty of others • Error Prone, Infer, Android Lint, …
  6. 9.

    Static Analysis • And don’t forget your IDE! • Android

    Studio/IDEA inspections • In-editor and on-demand • Code formatter
  7. 11.

    Static Analysis • Android Lint works • IDEs still have

    inspections • Not as many/as complex as Java • But it’s getting better all the time The good
  8. 12.

    Static Analysis • Almost nothing else from the Java world

    is compatible • No first-party tools from JetBrains • Apart from the IDE inspections • IDE Formatter isn’t often strict enough either The bad
  9. 13.

    Static Analysis The hope — third party tools • ktlint

    • “An anti-bike shedding Kotlin Linter with built-in formatter” • Detekt • “Static code analysis for Kotlin” • SonarQube
  10. 14.

    Static Analysis • “An anti-bike shedding Kotlin Linter with built-in

    formatter” • “Anti-bike shedding” • Avoids pointless discussions… • …by simply not offering customisation • “Linter” • Verifies code quality by looking for potential errors • Only looks at source code ktlint
  11. 15.

    Static Analysis • Equivalent to Checkstyle, just not as configurable

    • Follows Kotlin official style guide • Support for Android Kotlin styling with --android • Auto-fix many violations with -F • Minimal configuration via .editorconfig ktlint
  12. 16.

    Static Analysis • Static Code Analyser • Like PMD/Findbugs •

    Inspects code and AST • Does not inspect compiled bytecode • Limited type resolution capabilities Detekt
  13. 17.

    Static Analysis • Configurable via YAML file • default-detekt-config.yml for

    defaults • Use :detektGenerateConfig to get started • buildUponDefaultConfig is recommended • Has IDEA/AS plugin • Shows violations in the editor Detekt
  14. 18.

    Static Analysis • Java support is great • Kotlin is

    kinda supported • Via Android Lint and Detekt • Not nearly as comprehensively as other languages • Good if you are migrating to Kotlin and still have a bunch of Java • Keep an eye on everything from one place SonarQube
  15. 19.

    Static Analysis • There is some good Kotlin stuff out

    there • No excuses not to use it! • Java has been around almost 24 years • Kotlin is much younger, less widespread (for now) • It needs time for tooling to catch up with Java • Listen to your IDE inspections, use all the tools you have Recap
  16. 21.

    CI • Continuous Integration • Not a new concept —

    since the ‘90s • Popularised by XP (eXtreme Programming)
  17. 22.

    CI • Continuously Integrate all changes • Catch issues ASAP

    • Leverage build system + compiler + tests + static analysis
  18. 23.

    CI • CI is a practice • Software and services

    to implement it • We generally use “CI servers”
  19. 24.

    Why adopting CI? • Take out the human factor •

    People forget to run checks • Save devs’ time • Run on-demand or scheduled • Nightlies, PRs, main branch, etc
  20. 26.

    CI services • Jenkins, TeamCity, etc • More enterprise-friendly and

    customisable • Require setup and maintenance Self-hosted
  21. 27.

    CI services • Travis CI, CircleCI, Bitrise, GitLab CI, Google

    Cloud Build, etc • Easier to adopt, better for individuals and small companies • No/little setup and maintenance • Not as enterprise-friendly, and can be costly In-cloud
  22. 29.
  23. 30.

    No ca$h? No worries. • Most CI services offer a

    free tier • Open source (public) projects • Free build executor(s) • Free CPU time • Free for up to N users
  24. 31.

    Anatomy of a CI • Nodes • Master • Agents/executors

    Master node • Orchestrates jobs • Delegates actual work • Self-hosted vs In-Cloud
  25. 32.

    Anatomy of a CI • Nodes • Master • Agents/executors

    Master node • Orchestrates jobs • Delegates actual work • Self-hosted vs In-Cloud
  26. 33.

    Anatomy of a CI • Nodes • Master • Agents/executors

    Agents/executors • Run actual jobs’ work • Machines or containers • Most commonly Linux …but there’s macOS & Windows too
  27. 53.

    Pipelines • Group of jobs executed in stages • Can

    be parallelised • Can take less time than monolithic builds • Supported almost everywhere in some form
  28. 54.

    Pipelines • Group of jobs executed in stages • Can

    be parallelised • Can take less time than monolithic builds • Supported almost everywhere in some form Other names • Workflow • Build chain
  29. 55.

    Pipelines • Group of jobs executed in stages • Can

    be parallelised • Can take less time than monolithic builds • Supported almost everywhere in some form YES. YES, YOU SHOULD
 USE PIPELINES.
  30. 56.

    checkout warm up connected test unit test static analysis build

    apk Test different combinations! Pipelines
  31. 57.

    checkout warm up connected test unit test static analysis build

    apk Test different combinations! Pipelines
  32. 59.

    Why CircleCI? • Widely adopted in the community • Most

    cloud CIs work similarly • We have an example ready!
  33. 60.

    CircleCI • Docker-based • Configured with YAML • We’ll use

    2.1 • Free tiers • 1x Linux executor • 4x plan for Open Source
  34. 61.

    Our example • Squanchy — https://github.com/squanchy-dev/squanchy-android • FOSS conference app

    • 100% Kotlin • Unit tests, but no instrumented tests • Static analysis: Android Lint, Detekt, ktlint • Not very modular
  35. 62.

    Getting started • Working Gradle build • Identify the tasks

    to execute • Both in Gradle and outside Gradle • Identify the secrets to inject • Use environment variables
  36. 64.

    Getting started • Working Gradle build • • Both in

    Gradle and outside Gradle • Identify the secrets to inject • Use environment variables Identify the tasks to execute
  37. 66.

    Getting started • Working Gradle build • Identify the tasks

    to execute • Both in Gradle and outside Gradle • • Use environment variables Identify the secrets to inject
  38. 69.

    Play Services JSON Other values APPLICATION_ID API keys FABRIC_API_KEY ALGOLIA_APPLICATION_ID

    ALGOLIA_API_KEY ALGOLIA_INDICES_PREFIX Protip! Use Novoda’s gradle-build-properties-plugin applicationId(applicationProps['applicationId'].or(envVars['APPLICATION_ID']).string) manifestPlaceholders += [ fabricApiKey: secretsProps['fabricApiKey'].or(envVars['FABRIC_API_KEY']).string ] resValueString 'app_name', applicationProps['applicationName'].or("Squanchy") resValueString 'algolia_application_id', applicationProps[‘algoliaId'] .or(envVars['ALGOLIA_APPLICATION_ID']) resValueString 'algolia_api_key', secretsProps[‘algoliaApiKey'] .or(envVars['ALGOLIA_API_KEY']) resValueString 'algolia_indices_prefix', applicationProps[‘algoliaIndicesPrefix'] .or(envVars[‘ALGOLIA_INDICES_PREFIX']) resValueString 'deeplink_scheme', applicationProps['deeplinkScheme'].or("squanchy")
  39. 70.

    Getting started • Working Gradle build • Identify the tasks

    to execute • Both in Gradle and outside Gradle • Identify the secrets to inject • Use environment variables
  40. 72.

    version: 2.1 executors: android: working_directory: ~/squanchy docker: - image: circleci/android:api-28

    environment: ANDROID_HOME: /opt/android/sdk APPLICATION_ID: net.squanchy.example FABRIC_API_KEY: 0000000000000000000000000000000000000000 ALGOLIA_APPLICATION_ID: ABCDEFGH12 ALGOLIA_API_KEY: 00000000000000000000000000000000 ALGOLIA_INDICES_PREFIX: squanchy_dev commands: # Build Tools cache commands restore_build_tools_cache: steps: - restore_cache: name: Restore Android build tools cache keys: - v3-build-tools-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} save_build_tools_cache: steps: - save_cache: name: Save Android build tools cache paths: - /opt/android/sdk/build-tools key: v3-build-tools-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} # Gradle cache commands restore_gradle_cache: steps: - restore_cache: name: Restore Gradle dependencies cache keys: - v1-gradle-dependencies-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} - restore_cache: name: Restore Gradle wrapper cache keys: - v1-gradle-wrapper-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle/wrapper/gradle-wrapper.properties" }} save_gradle_cache: steps: - save_cache: name: Save Gradle dependencies cache paths: - ~/.gradle/caches key: v1-gradle-dependencies-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} - save_cache: name: Save Gradle wrapper cache paths: - ~/.gradle/wrapper key: v1-gradle-wrapper-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle/wrapper/gradle-wrapper.properties" }} # Android Gradle build cache commands restore_android_build_cache: steps: - restore_cache: name: Restore Android Gradle build cache keys: - v3-build-cache-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} save_android_build_cache: steps: - save_cache: name: Save Android Gradle build cache paths: - ~/.android/build-cache key: v3-build-cache-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} ensure_android_sdk_is_ready: steps: - run: name: Ensure Android SDK install is up-to-date command: workspace/repo/.circleci/ci-scripts/ensure-sdkmanager.sh download_gradle_dependencies: steps: - run: name: Download Gradle dependencies command: cd workspace/repo/ && ./gradlew downloadDependencies restore_workspace: steps: - attach_workspace: at: workspace jobs: checkout: executor: android steps: - checkout: path: workspace/repo # Prepare the container for the build - ensure_android_sdk_is_ready - run: name: Create mock Play Services JSON command: workspace/repo/.circleci/ci-scripts/ci-mock-google-services-setup.sh # Persist repo code - persist_to_workspace: root: workspace paths: - repo prepare_for_checks: executor: android steps: - restore_workspace - restore_gradle_cache - restore_android_build_cache - restore_build_tools_cache - download_gradle_dependencies - save_android_build_cache - save_gradle_cache - save_build_tools_cache # Persist built app code - persist_to_workspace: root: workspace paths: - repo/app/build static_analysis: executor: android steps: - restore_workspace - restore_gradle_cache - restore_android_build_cache - restore_build_tools_cache # See https://issuetracker.google.com/issues/62217354 for the parallelism option - run: name: Run static analysis command: cd workspace/repo && ./gradlew lintRelease detekt ktlintCheck -Djava.util.concurrent.ForkJoinPool.common.parallelism=2 # Collect static analysis reports as build artifacts - store_artifacts: path: workspace/repo/app/build/reports destination: reports tests: executor: android steps: - restore_workspace - restore_gradle_cache - restore_android_build_cache - restore_build_tools_cache # See https://issuetracker.google.com/issues/62217354 for the parallelism option - run: name: Run unit tests command: cd workspace/repo && ./gradlew testRelease -Djava.util.concurrent.ForkJoinPool.common.parallelism=2 # Collect JUnit test results - store_test_results: path: workspace/repo/app/build/test-results workflows: version: 2 build_and_test: jobs: - checkout - prepare_for_checks: requires: - checkout - static_analysis: requires: - prepare_for_checks - tests: requires: - prepare_for_checks
  41. 75.

    workflows: #... jobs: #... commands: #... #... executors: version: 2.1

    android: working_directory: ~/squanchy docker: - image: circleci/android:api-28 environment: ANDROID_HOME: /opt/android/sdk APPLICATION_ID: net.squanchy.example FABRIC_API_KEY: 0000000000000000000000000000000000000000 ALGOLIA_APPLICATION_ID: ABCDEFGH12 ALGOLIA_API_KEY: 00000000000000000000000000000000 ALGOLIA_INDICES_PREFIX: squanchy_dev
  42. 76.

    commands: #... executors: version: 2.1 android: working_directory: ~/squanchy docker: -

    image: circleci/android:api-28 environment: ANDROID_HOME: /opt/android/sdk APPLICATION_ID: net.squanchy.example FABRIC_API_KEY: 0000000000000000000000000000000000000000 ALGOLIA_APPLICATION_ID: ABCDEFGH12 ALGOLIA_API_KEY: 00000000000000000000000000000000 ALGOLIA_INDICES_PREFIX: squanchy_dev
  43. 77.

    workflows: #... jobs: #... commands: #... #... executors: version: 2.1

    # Build Tools cache commands restore_build_tools_cache: steps: - restore_cache: name: Restore Android build tools cache keys: - v3-build-tools-{{ checksum "workspace/ repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum
  44. 78.

    commands: #... # Build Tools cache commands restore_build_tools_cache: steps: -

    restore_cache: name: Restore Android build tools cache keys: - v3-build-tools-{{ checksum "workspace/ repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} save_build_tools_cache: steps: - save_cache: name: Save Android build tools cache paths:
  45. 81.

    - save_cache: name: Saves path(s) to CircleCI cache paths: -

    /path/to/save keys: - v1-my-cache-{{ checksum "some/thing" }}
  46. 84.

    #... workflows: jobs: #... commands: restore_build_tools_cache save_build_tools_cache restore_gradle_cache save_gradle_cache restore_android_build_cache

    save_android_build_cache ensure_android_sdk_is_ready download_gradle_dependencies restore_workspace
  47. 85.

    workflows: #... jobs: #... commands: #... #... executors: version: 2.1

    checkout: executor: android steps: - checkout: path: workspace/repo
  48. 86.

    jobs: #... checkout: executor: android steps: - checkout: path: workspace/repo

    # Prepare the container for the build - ensure_android_sdk_is_ready - run: name: Create mock Play Services JSON command: workspace/repo/.circleci/ci-scripts/ci-mock- google-services-setup.sh # Persist repo code - persist_to_workspace:
  49. 87.

    job_name: executor: executor_name steps: - step_name: name: Description (shown in

    UI) other_stuff: - {depends on step} - command_name
  50. 88.

    workflows: #... jobs: #... commands: #... #... executors: version: 2.1

    checkout prepare_for_checks static_analysis tests
  51. 90.

    workflows: #... #... version: 2 build_and_test: jobs: - checkout -

    prepare_for_checks: requires: - checkout - static_analysis: requires: - prepare_for_checks - tests: requires: - prepare_for_checks
  52. 91.

    workflows: #... #... version: 2 build_and_test: jobs: - checkout -

    prepare_for_checks: requires: - checkout - static_analysis: requires: - prepare_for_checks - tests: requires: - prepare_for_checks checkout static analysis prepare for checks tests Directed Acyclic Graph
  53. 94.

    A CI run in detail checkout setup android sdk create

    mock gms json persist to workspace checkout
  54. 95.

    A CI run in detail checkout setup android sdk create

    mock gms json persist to workspace checkout
  55. 97.

    A CI run in detail restore from workspace restore caches

    download dependencies persist to workspace prepare for checks save caches
  56. 98.

    A CI run in detail restore from workspace restore caches

    download dependencies persist to workspace prepare for checks save caches
  57. 100.

    A CI run in detail restore from workspace restore caches

    run tasks store artefacts static analysis and tests
  58. 101.

    A CI run in detail restore from workspace restore caches

    run tasks store artefacts static analysis and tests
  59. 102.
  60. 105.

    • Static analysis is good • Use static analysis! •

    Continuous Integration is good • Use a CI! • Static analysis is even better with a CI • Check out Squanchy • https://github.com/squanchy-dev/squanchy-android