Computer Says No — Static Analysis and CI in a Kotlin World (Droidcon Italy 2019)

Presented with Neil Hutchison — @nnnneeeiil

Even though Android is a mature platform by now, the adoption of Kotlin at Google I/O 2017 brought about a sweeping wave of freshness and enthusiasm amongst developers. Regardless of what the language and design patterns we use when writing an app, there is only one way to ensure correctness and quality: testing, static analysis and continuous integration.
Many still think that setting up a CI for your project is hard, onerous, and not that useful, but we’re going to see how this is not true. Focusing on static analysis and unit testing, we’ll walk through setting up a continuous integration pipeline for a modern open source Android project using Gradle, CircleCI and Kotlin. We’ll see what benefits this brings to a codebase and how with a few tricks we can make sure external contributors adhere to the project code style, how we can prevent subtle bugs to sneak into the codebase, all with very little effort and zero budget.

—

You can find the Keynote sources here: https://www.dropbox.com/sh/mpmwy60vhph3cbo/AADSXZExFGhkBqEU_8DI88CBa?dl=0

—

Conference talk link: https://it.droidcon.com/2019/it/talks/415-computer-says-no-static-analysis-and-ci-in-a-kotlin-world

Transcript

1. says COMPUTER SEBASTIANO POGGI @seebrock3r JetBrains NEIL HUTCHISON @nnnneeeiil Toyota Connected Europe no
2. Catch those silly bugs early Static Analysis
3. Code Quality • It’s important • Consistency • Never fix the same bug twice • Have less bugs in the first place • A safety net to catch easy mistakes
4. Code Quality • How do you achieve a baseline of code quality? • Testing • Static Analysis • CI
5. Testing • DO TEST! • Unit AND Integration • Too big of a topic for this talk
6. Static Analysis • Tools that analyse code statically • Only look at code “at rest”, not at runtime • Check for common bugs/oversights • May involve analysing compiled bytecode • Check for code style violations
7. Static Analysis • We come from Java • Some still stuck there • Java has a lot of static analysis tools • Most free/FOSS, but many commercial
8. Static Analysis • The “holy trinity” • Findbugs (prefer Spotbugs) • PMD • Checkstyle • Plenty of others • Error Prone, Infer, Android Lint, …
9. Static Analysis • And don’t forget your IDE! • Android Studio/IDEA inspections • In-editor and on-demand • Code formatter
10. So what about Kotlin?
11. Static Analysis • Android Lint works • IDEs still has inspections • Not as many/as complex as Java • But it’s getting better all the time The good
12. Static Analysis • Almost nothing else from the Java world is compatible • No first-party tools from JetBrains • Apart from the IDE inspections • IDE Formatter isn’t often strict enough either The bad
13. Static Analysis The hope — third party tools • ktlint • “An anti-bike shedding Kotlin Linter with built-in formatter” • Detekt • “Static code analysis for Kotlin” • SonarQube
14. Static Analysis • “An anti-bike shedding Kotlin Linter with built-in formatter” • “Anti-bike shedding” • Avoids pointless discussions… • …by simply not offering customisation • “Linter” • Verifies code quality by looking for potential errors • Only looks at source code ktlint
15. Static Analysis • Equivalent to Checkstyle, just not as configurable • Follows Kotlin official style guide • Support for Android Kotlin styling with --android • Auto-fix many violations with -F • Minimal configuration via .editorconfig ktlint
16. Static Analysis • Static Code Analyser • Like PMD/Findbugs • Inspects code and AST • Does not inspect compiled bytecode • Limited type resolution capabilities Detekt
17. Static Analysis • Configurable via YAML file • default-detekt-config.yml for defaults • Use :detektGenerateConfig to get started • buildUponDefaultConfig is recommended • Has IDEA/AS plugin • Shows violations in the editor Detekt
18. Static Analysis • Java support is great • Kotlin is kinda supported • Via Android Lint and Detekt • Not nearly as comprehensively as other languages • Good if you are migrating to Kotlin and still have a bunch of Java • Keep an eye on everything from one place SonarQube
19. Static Analysis • There is some good Kotlin stuff out there • No excuses not to use it! • Java has been around almost 24 years • Kotlin is much younger, less widespread (for now) • It needs time for tooling to catch up with Java • Listen to your IDE inspections, use all the tools you have Recap
20. Empower your Static Analysis CI
21. CI • Continuous Integration • Not a new concept — since the ‘90s • Popularised by XP (eXtreme Programming)
22. CI • Continuously Integrate all changes • Catch issues ASAP • Leverage build system + compiler + tests + static analysis
23. CI • CI is a practice • Software and services to implement it • We generally use “CI servers”
24. Why adopting CI? • Take out the human factor • People forget to run checks • Save devs’ time • Run on-demand or scheduled • Nightlies, PRs, main branch, etc
25. CI services • Two main kinds • Self-hosted • In-cloud
26. CI services • Jenkins, TeamCity, etc • More enterprise-friendly and customisable • Require setup and maintenance Self-hosted
27. CI services • Travis, CircleCI, Bitrise, GitLab CI, Google Cloud Build, etc • Easier to adopt, better for individuals and small companies • No/little setup and maintenance • Not as enterprise-friendly, and can be costly In-cloud
28. CI services • Two main kinds • Self-hosted • In-cloud some blur the lines
30. No ca$h? No worries. • Most CI services offer a free tier • Open source (public) projects • Free build executor(s) • Free CPU time • Free for up to N users
31. Anatomy of a CI • Nodes • Master • Agents/executors Master node • Orchestrates jobs • Delegates actual work • Self-hosted vs In-Cloud
32. Anatomy of a CI • Nodes • Master • Agents/executors Master node • Orchestrates jobs • Delegates actual work • Self-hosted vs In-Cloud
33. Anatomy of a CI • Nodes • Master • Agents/executors Agents/executors • Run actual jobs’ work • Machines or containers • Most commonly Linux …but there’s macOS & Windows too
34. THE CI BUILD JOB BUILD JOB BUILD JOB The build queue
35. THE CI BUILD JOB BUILD JOB BUILD JOB The build queue
36. THE CI BUILD JOB BUILD JOB BUILD JOB The build queue
37. THE CI BUILD JOB BUILD JOB The build queue
38. THE CI BUILD JOB The build queue
39. THE CI The build queue
40. RELEASE ONLY :app:assemble :app:test :app:lint :app:detekt A typical job :app:ktlintCheck
41. RELEASE ONLY :app:assemble :app:test :app:lint :app:detekt A typical job :app:ktlintCheck
42. RELEASE ONLY :app:assemble :app:test :app:lint :app:detekt A typical job :app:ktlintCheck
43. :app:assemble :app:test :app:lint :app:detekt :app:ktlintCheck :app:build A typical job
44. :app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease
45. :app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease
46. :app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease
47. :app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease
48. :app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease
49. :app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease
50. :app:assembleRelease :app:testRelease :app:cAT :app:lintRelease
51. :app:assembleRelease :app:testRelease :app:cAT :app:lintRelease T1
52. T2 T1 :app:assembleRelease :app:testRelease :app:cAT :app:lintRelease
53. Pipelines • Group of jobs executed in stages • Can be parallelised • Can take less time than monolithic builds • Supported almost everywhere in some form
54. Pipelines • Group of jobs executed in stages • Can be parallelised • Can take less time than monolithic builds • Supported almost everywhere in some form Other names • Workflow • Build chain
55. Pipelines • Group of jobs executed in stages • Can be parallelised • Can take less time than monolithic builds • Supported almost everywhere in some form YES. YES, YOU SHOULD
 USE PIPELINES.
56. checkout warm up connected test unit test static analysis build apk Test different combinations! Pipelines
57. checkout warm up connected test unit test static analysis build apk Test different combinations! Pipelines
58. A case study CircleCI
59. Why CircleCI? • Widely adopted in the community • Most cloud CIs work similarly • We have an example ready!
60. CircleCI • Docker-based • Configured with YAML • We’ll use 2.1 • Free tiers • 1x Linux executor • 4x plan for Open Source
61. Our example • Squanchy — https://github.com/squanchy-dev/squanchy-android • FOSS conference app • 100% Kotlin • Unit tests, but no instrumented tests • Static analysis: Android Lint, Detekt, ktlint • Not very modular
62. Getting started • Working Gradle build • Identify the tasks to execute • Both in Gradle and outside Gradle • Identify the secrets to inject • Use environment variables
63. ./gradlew build
64. Getting started • Working Gradle build • • Both in Gradle and outside Gradle • Identify the secrets to inject • Use environment variables Identify the tasks to execute
65. Unit tests Static analysis :testRelease :lintRelease :detekt :ktlintCheck Warm up :downloadDependencies Checkout [not Gradle]
66. Getting started • Working Gradle build • Identify the tasks to execute • Both in Gradle and outside Gradle • • Use environment variables Identify the secrets to inject
67. API keys Play Services JSON Other values APPLICATION_ID FABRIC_API_KEY ALGOLIA_APPLICATION_ID ALGOLIA_API_KEY ALGOLIA_INDICES_PREFIX
68. API keys Play Services JSON Other values APPLICATION_ID FABRIC_API_KEY ALGOLIA_APPLICATION_ID ALGOLIA_API_KEY ALGOLIA_INDICES_PREFIX
69. Play Services JSON Other values APPLICATION_ID API keys FABRIC_API_KEY ALGOLIA_APPLICATION_ID ALGOLIA_API_KEY ALGOLIA_INDICES_PREFIX Protip! Use Novoda’s gradle-build-properties-plugin applicationId(applicationProps['applicationId'].or(envVars['APPLICATION_ID']).string) manifestPlaceholders += [ fabricApiKey: secretsProps['fabricApiKey'].or(envVars['FABRIC_API_KEY']).string ] resValueString 'app_name', applicationProps['applicationName'].or("Squanchy") resValueString 'algolia_application_id', applicationProps[‘algoliaId'] .or(envVars['ALGOLIA_APPLICATION_ID']) resValueString 'algolia_api_key', secretsProps[‘algoliaApiKey'] .or(envVars['ALGOLIA_API_KEY']) resValueString 'algolia_indices_prefix', applicationProps[‘algoliaIndicesPrefix'] .or(envVars[‘ALGOLIA_INDICES_PREFIX']) resValueString 'deeplink_scheme', applicationProps['deeplinkScheme'].or("squanchy")
70. Getting started • Working Gradle build • Identify the tasks to execute • Both in Gradle and outside Gradle • Identify the secrets to inject • Use environment variables
71. Configure CircleCI • YAML file • /.circleci/config.yml • Versioned • Currently 2.1
72. version: 2.1 executors: android: working_directory: ~/squanchy docker: - image: circleci/android:api-28 environment: ANDROID_HOME: /opt/android/sdk APPLICATION_ID: net.squanchy.example FABRIC_API_KEY: 0000000000000000000000000000000000000000 ALGOLIA_APPLICATION_ID: ABCDEFGH12 ALGOLIA_API_KEY: 00000000000000000000000000000000 ALGOLIA_INDICES_PREFIX: squanchy_dev commands: # Build Tools cache commands restore_build_tools_cache: steps: - restore_cache: name: Restore Android build tools cache keys: - v3-build-tools-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} save_build_tools_cache: steps: - save_cache: name: Save Android build tools cache paths: - /opt/android/sdk/build-tools key: v3-build-tools-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} # Gradle cache commands restore_gradle_cache: steps: - restore_cache: name: Restore Gradle dependencies cache keys: - v1-gradle-dependencies-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} - restore_cache: name: Restore Gradle wrapper cache keys: - v1-gradle-wrapper-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle/wrapper/gradle-wrapper.properties" }} save_gradle_cache: steps: - save_cache: name: Save Gradle dependencies cache paths: - ~/.gradle/caches key: v1-gradle-dependencies-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} - save_cache: name: Save Gradle wrapper cache paths: - ~/.gradle/wrapper key: v1-gradle-wrapper-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle/wrapper/gradle-wrapper.properties" }} # Android Gradle build cache commands restore_android_build_cache: steps: - restore_cache: name: Restore Android Gradle build cache keys: - v3-build-cache-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} save_android_build_cache: steps: - save_cache: name: Save Android Gradle build cache paths: - ~/.android/build-cache key: v3-build-cache-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} ensure_android_sdk_is_ready: steps: - run: name: Ensure Android SDK install is up-to-date command: workspace/repo/.circleci/ci-scripts/ensure-sdkmanager.sh download_gradle_dependencies: steps: - run: name: Download Gradle dependencies command: cd workspace/repo/ && ./gradlew downloadDependencies restore_workspace: steps: - attach_workspace: at: workspace jobs: checkout: executor: android steps: - checkout: path: workspace/repo # Prepare the container for the build - ensure_android_sdk_is_ready - run: name: Create mock Play Services JSON command: workspace/repo/.circleci/ci-scripts/ci-mock-google-services-setup.sh # Persist repo code - persist_to_workspace: root: workspace paths: - repo prepare_for_checks: executor: android steps: - restore_workspace - restore_gradle_cache - restore_android_build_cache - restore_build_tools_cache - download_gradle_dependencies - save_android_build_cache - save_gradle_cache - save_build_tools_cache # Persist built app code - persist_to_workspace: root: workspace paths: - repo/app/build static_analysis: executor: android steps: - restore_workspace - restore_gradle_cache - restore_android_build_cache - restore_build_tools_cache # See https://issuetracker.google.com/issues/62217354 for the parallelism option - run: name: Run static analysis command: cd workspace/repo && ./gradlew lintRelease detekt ktlintCheck -Djava.util.concurrent.ForkJoinPool.common.parallelism=2 # Collect static analysis reports as build artifacts - store_artifacts: path: workspace/repo/app/build/reports destination: reports tests: executor: android steps: - restore_workspace - restore_gradle_cache - restore_android_build_cache - restore_build_tools_cache # See https://issuetracker.google.com/issues/62217354 for the parallelism option - run: name: Run unit tests command: cd workspace/repo && ./gradlew testRelease -Djava.util.concurrent.ForkJoinPool.common.parallelism=2 # Collect JUnit test results - store_test_results: path: workspace/repo/app/build/test-results workflows: version: 2 build_and_test: jobs: - checkout - prepare_for_checks: requires: - checkout - static_analysis: requires: - prepare_for_checks - tests: requires: - prepare_for_checks
73. version: 2.1 #... commands: jobs: workflows: #... #... #... executors:
74. version: 2.1 #... commands: jobs: workflows: #... #... #... executors:
75. workflows: #... jobs: #... commands: #... #... executors: version: 2.1 android: working_directory: ~/squanchy docker: - image: circleci/android:api-28 environment: ANDROID_HOME: /opt/android/sdk APPLICATION_ID: net.squanchy.example FABRIC_API_KEY: 0000000000000000000000000000000000000000 ALGOLIA_APPLICATION_ID: ABCDEFGH12 ALGOLIA_API_KEY: 00000000000000000000000000000000 ALGOLIA_INDICES_PREFIX: squanchy_dev
76. commands: #... executors: version: 2.1 android: working_directory: ~/squanchy docker: - image: circleci/android:api-28 environment: ANDROID_HOME: /opt/android/sdk APPLICATION_ID: net.squanchy.example FABRIC_API_KEY: 0000000000000000000000000000000000000000 ALGOLIA_APPLICATION_ID: ABCDEFGH12 ALGOLIA_API_KEY: 00000000000000000000000000000000 ALGOLIA_INDICES_PREFIX: squanchy_dev
77. workflows: #... jobs: #... commands: #... #... executors: version: 2.1 # Build Tools cache commands restore_build_tools_cache: steps: - restore_cache: name: Restore Android build tools cache keys: - v3-build-tools-{{ checksum "workspace/ repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum
78. commands: #... # Build Tools cache commands restore_build_tools_cache: steps: - restore_cache: name: Restore Android build tools cache keys: - v3-build-tools-{{ checksum "workspace/ repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} save_build_tools_cache: steps: - save_cache: name: Save Android build tools cache paths:
79. command_name: steps: - step_name: name: Description (shown in UI) other_stuff: - {depends on step}
80. - save_cache: name: Saves path(s) to CircleCI cache paths: - /path/to/save keys: - v1-my-cache-{{ checksum "some/thing" }}
81. - restore_cache: name: Restores from CircleCI cache keys: - v1-my-cache-{{ checksum "some/thing" }}
82. - attach_workspace: at: path/to/attach/to
83. - run: name: Executes something in workspace command: my-script.sh param1 param2
84. #... workflows: jobs: #... commands: restore_build_tools_cache save_build_tools_cache restore_gradle_cache save_gradle_cache restore_android_build_cache save_android_build_cache ensure_android_sdk_is_ready download_gradle_dependencies restore_workspace
85. workflows: #... jobs: #... commands: #... #... executors: version: 2.1 checkout: executor: android steps: - checkout: path: workspace/repo
86. jobs: #... checkout: executor: android steps: - checkout: path: workspace/repo # Prepare the container for the build - ensure_android_sdk_is_ready - run: name: Create mock Play Services JSON command: workspace/repo/.circleci/ci-scripts/ci-mock- google-services-setup.sh # Persist repo code - persist_to_workspace:
87. job_name: executor: executor_name steps: - step_name: name: Description (shown in UI) other_stuff: - {depends on step}
88. job_name: executor: executor_name steps: - step_name: name: Description (shown in UI) other_stuff: - {depends on step} - command_name
89. workflows: #... jobs: #... commands: #... #... executors: version: 2.1 checkout prepare_for_checks static_analysis tests
90. workflows: #... jobs: #... commands: #... #... executors: version: 2.1 version: 2 build_and_test: jobs:
91. workflows: #... #... version: 2 build_and_test: jobs: - checkout - prepare_for_checks: requires: - checkout - static_analysis: requires: - prepare_for_checks - tests: requires: - prepare_for_checks
92. workflows: #... #... version: 2 build_and_test: jobs: - checkout - prepare_for_checks: requires: - checkout - static_analysis: requires: - prepare_for_checks - tests: requires: - prepare_for_checks checkout static analysis prepare for checks tests Directed Acyclic Graph
93. A CI run in detail checkout static analysis prepare for checks tests
94. A CI run in detail checkout static analysis prepare for checks tests
95. A CI run in detail checkout setup android sdk create mock gms json persist to workspace checkout
96. A CI run in detail checkout setup android sdk create mock gms json persist to workspace checkout
97. A CI run in detail checkout static analysis prepare for checks tests
98. A CI run in detail restore from workspace restore caches download dependencies persist to workspace prepare for checks save caches
99. A CI run in detail restore from workspace restore caches download dependencies persist to workspace prepare for checks save caches
100. A CI run in detail checkout static analysis prepare for checks tests
101. A CI run in detail restore from workspace restore caches run tasks store artefacts static analysis and tests
102. A CI run in detail restore from workspace restore caches run tasks store artefacts static analysis and tests
103. get stuff
104. get stuff do stuff save stuff
105. What are today’s lessons? Conclusions
106. • Static analysis is good • Use static analysis! • Continuous Integration is good • Use a CI! • Static analysis is even better with a CI • Check out Squanchy • https://github.com/squanchy-dev/squanchy-android
107. THANK YOU bit.ly/Computer-Says-No SEBASTIANO POGGI @seebrock3r JetBrains NEIL HUTCHISON @nnnneeeiil Toyota Connected Europe