Tweet
Tweet

Slide 1

Slide 1 text

says COMPUTER SEBASTIANO POGGI @seebrock3r JetBrains NEIL HUTCHISON @nnnneeeiil Toyota Connected Europe no

Slide 2

Slide 2 text

Catch those silly bugs early Static Analysis

Slide 3

Slide 3 text

Code Quality • It’s important • Consistency • Never fix the same bug twice • Have less bugs in the first place • A safety net to catch easy mistakes

Slide 4

Slide 4 text

Code Quality • How do you achieve a baseline of code quality? • Testing • Static Analysis • CI

Slide 5

Slide 5 text

Testing • DO TEST! • Unit AND Integration • Too big of a topic for this talk

Slide 6

Slide 6 text

Static Analysis • Tools that analyse code statically • Only look at code “at rest”, not at runtime • Check for common bugs/oversights • May involve analysing compiled bytecode • Check for code style violations

Slide 7

Slide 7 text

Static Analysis • We come from Java • Some still stuck there • Java has a lot of static analysis tools • Most free/FOSS, but many commercial

Slide 8

Slide 8 text

Static Analysis • The “holy trinity” • Findbugs (prefer Spotbugs) • PMD • Checkstyle • Plenty of others • Error Prone, Infer, Android Lint, …

Slide 9

Slide 9 text

Static Analysis • And don’t forget your IDE! • Android Studio/IDEA inspections • In-editor and on-demand • Code formatter

Slide 10

Slide 10 text

So what about Kotlin?

Slide 11

Slide 11 text

Static Analysis • Android Lint works • IDEs still has inspections • Not as many/as complex as Java • But it’s getting better all the time The good

Slide 12

Slide 12 text

Static Analysis • Almost nothing else from the Java world is compatible • No first-party tools from JetBrains • Apart from the IDE inspections • IDE Formatter isn’t often strict enough either The bad

Slide 13

Slide 13 text

Static Analysis The hope — third party tools • ktlint • “An anti-bike shedding Kotlin Linter with built-in formatter” • Detekt • “Static code analysis for Kotlin” • SonarQube

Slide 14

Slide 14 text

Static Analysis • “An anti-bike shedding Kotlin Linter with built-in formatter” • “Anti-bike shedding” • Avoids pointless discussions… • …by simply not offering customisation • “Linter” • Verifies code quality by looking for potential errors • Only looks at source code ktlint

Slide 15

Slide 15 text

Static Analysis • Equivalent to Checkstyle, just not as configurable • Follows Kotlin official style guide • Support for Android Kotlin styling with --android • Auto-fix many violations with -F • Minimal configuration via .editorconfig ktlint

Slide 16

Slide 16 text

Static Analysis • Static Code Analyser • Like PMD/Findbugs • Inspects code and AST • Does not inspect compiled bytecode • Limited type resolution capabilities Detekt

Slide 17

Slide 17 text

Static Analysis • Configurable via YAML file • default-detekt-config.yml for defaults • Use :detektGenerateConfig to get started • buildUponDefaultConfig is recommended • Has IDEA/AS plugin • Shows violations in the editor Detekt

Slide 18

Slide 18 text

Static Analysis • Java support is great • Kotlin is kinda supported • Via Android Lint and Detekt • Not nearly as comprehensively as other languages • Good if you are migrating to Kotlin and still have a bunch of Java • Keep an eye on everything from one place SonarQube

Slide 19

Slide 19 text

Static Analysis • There is some good Kotlin stuff out there • No excuses not to use it! • Java has been around almost 24 years • Kotlin is much younger, less widespread (for now) • It needs time for tooling to catch up with Java • Listen to your IDE inspections, use all the tools you have Recap

Slide 20

Slide 20 text

Empower your Static Analysis CI

Slide 21

Slide 21 text

CI • Continuous Integration • Not a new concept — since the ‘90s • Popularised by XP (eXtreme Programming)

Slide 22

Slide 22 text

CI • Continuously Integrate all changes • Catch issues ASAP • Leverage build system + compiler + tests + static analysis

Slide 23

Slide 23 text

CI • CI is a practice • Software and services to implement it • We generally use “CI servers”

Slide 24

Slide 24 text

Why adopting CI? • Take out the human factor • People forget to run checks • Save devs’ time • Run on-demand or scheduled • Nightlies, PRs, main branch, etc

Slide 25

Slide 25 text

CI services • Two main kinds • Self-hosted • In-cloud

Slide 26

Slide 26 text

CI services • Jenkins, TeamCity, etc • More enterprise-friendly and customisable • Require setup and maintenance Self-hosted

Slide 27

Slide 27 text

CI services • Travis, CircleCI, Bitrise, GitLab CI, Google Cloud Build, etc • Easier to adopt, better for individuals and small companies • No/little setup and maintenance • Not as enterprise-friendly, and can be costly In-cloud

Slide 28

Slide 28 text

CI services • Two main kinds • Self-hosted • In-cloud some blur the lines

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

No ca$h? No worries. • Most CI services offer a free tier • Open source (public) projects • Free build executor(s) • Free CPU time • Free for up to N users

Slide 31

Slide 31 text

Anatomy of a CI • Nodes • Master • Agents/executors Master node • Orchestrates jobs • Delegates actual work • Self-hosted vs In-Cloud

Slide 32

Slide 32 text

Anatomy of a CI • Nodes • Master • Agents/executors Master node • Orchestrates jobs • Delegates actual work • Self-hosted vs In-Cloud

Slide 33

Slide 33 text

Anatomy of a CI • Nodes • Master • Agents/executors Agents/executors • Run actual jobs’ work • Machines or containers • Most commonly Linux …but there’s macOS & Windows too

Slide 34

Slide 34 text

THE CI BUILD JOB BUILD JOB BUILD JOB The build queue

Slide 35

Slide 35 text

THE CI BUILD JOB BUILD JOB BUILD JOB The build queue

Slide 36

Slide 36 text

THE CI BUILD JOB BUILD JOB BUILD JOB The build queue

Slide 37

Slide 37 text

THE CI BUILD JOB BUILD JOB The build queue

Slide 38

Slide 38 text

THE CI BUILD JOB The build queue

Slide 39

Slide 39 text

THE CI The build queue

Slide 40

Slide 40 text

RELEASE ONLY :app:assemble :app:test :app:lint :app:detekt A typical job :app:ktlintCheck

Slide 41

Slide 41 text

RELEASE ONLY :app:assemble :app:test :app:lint :app:detekt A typical job :app:ktlintCheck

Slide 42

Slide 42 text

RELEASE ONLY :app:assemble :app:test :app:lint :app:detekt A typical job :app:ktlintCheck

Slide 43

Slide 43 text

:app:assemble :app:test :app:lint :app:detekt :app:ktlintCheck :app:build A typical job

Slide 44

Slide 44 text

:app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease

Slide 45

Slide 45 text

:app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease

Slide 46

Slide 46 text

:app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease

Slide 47

Slide 47 text

:app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease

Slide 48

Slide 48 text

:app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease

Slide 49

Slide 49 text

:app:assembleRelease A better job :app:testRelease :app:cAT :app:lintRelease

Slide 50

Slide 50 text

:app:assembleRelease :app:testRelease :app:cAT :app:lintRelease

Slide 51

Slide 51 text

:app:assembleRelease :app:testRelease :app:cAT :app:lintRelease T1

Slide 52

Slide 52 text

T2 T1 :app:assembleRelease :app:testRelease :app:cAT :app:lintRelease

Slide 53

Slide 53 text

Pipelines • Group of jobs executed in stages • Can be parallelised • Can take less time than monolithic builds • Supported almost everywhere in some form

Slide 54

Slide 54 text

Pipelines • Group of jobs executed in stages • Can be parallelised • Can take less time than monolithic builds • Supported almost everywhere in some form Other names • Workflow • Build chain

Slide 55

Slide 55 text

Pipelines • Group of jobs executed in stages • Can be parallelised • Can take less time than monolithic builds • Supported almost everywhere in some form YES. YES, YOU SHOULD
 USE PIPELINES.

Slide 56

Slide 56 text

checkout warm up connected test unit test static analysis build apk Test different combinations! Pipelines

Slide 57

Slide 57 text

checkout warm up connected test unit test static analysis build apk Test different combinations! Pipelines

Slide 58

Slide 58 text

A case study CircleCI

Slide 59

Slide 59 text

Why CircleCI? • Widely adopted in the community • Most cloud CIs work similarly • We have an example ready!

Slide 60

Slide 60 text

CircleCI • Docker-based • Configured with YAML • We’ll use 2.1 • Free tiers • 1x Linux executor • 4x plan for Open Source

Slide 61

Slide 61 text

Our example • Squanchy — https://github.com/squanchy-dev/squanchy-android • FOSS conference app • 100% Kotlin • Unit tests, but no instrumented tests • Static analysis: Android Lint, Detekt, ktlint • Not very modular

Slide 62

Slide 62 text

Getting started • Working Gradle build • Identify the tasks to execute • Both in Gradle and outside Gradle • Identify the secrets to inject • Use environment variables

Slide 63

Slide 63 text

./gradlew build

Slide 64

Slide 64 text

Getting started • Working Gradle build • • Both in Gradle and outside Gradle • Identify the secrets to inject • Use environment variables Identify the tasks to execute

Slide 65

Slide 65 text

Unit tests Static analysis :testRelease :lintRelease :detekt :ktlintCheck Warm up :downloadDependencies Checkout [not Gradle]

Slide 66

Slide 66 text

Getting started • Working Gradle build • Identify the tasks to execute • Both in Gradle and outside Gradle • • Use environment variables Identify the secrets to inject

Slide 67

Slide 67 text

API keys Play Services JSON Other values APPLICATION_ID FABRIC_API_KEY ALGOLIA_APPLICATION_ID ALGOLIA_API_KEY ALGOLIA_INDICES_PREFIX

Slide 68

Slide 68 text

API keys Play Services JSON Other values APPLICATION_ID FABRIC_API_KEY ALGOLIA_APPLICATION_ID ALGOLIA_API_KEY ALGOLIA_INDICES_PREFIX

Slide 69

Slide 69 text

Play Services JSON Other values APPLICATION_ID API keys FABRIC_API_KEY ALGOLIA_APPLICATION_ID ALGOLIA_API_KEY ALGOLIA_INDICES_PREFIX Protip! Use Novoda’s gradle-build-properties-plugin applicationId(applicationProps['applicationId'].or(envVars['APPLICATION_ID']).string) manifestPlaceholders += [ fabricApiKey: secretsProps['fabricApiKey'].or(envVars['FABRIC_API_KEY']).string ] resValueString 'app_name', applicationProps['applicationName'].or("Squanchy") resValueString 'algolia_application_id', applicationProps[‘algoliaId'] .or(envVars['ALGOLIA_APPLICATION_ID']) resValueString 'algolia_api_key', secretsProps[‘algoliaApiKey'] .or(envVars['ALGOLIA_API_KEY']) resValueString 'algolia_indices_prefix', applicationProps[‘algoliaIndicesPrefix'] .or(envVars[‘ALGOLIA_INDICES_PREFIX']) resValueString 'deeplink_scheme', applicationProps['deeplinkScheme'].or("squanchy")

Slide 70

Slide 70 text

Getting started • Working Gradle build • Identify the tasks to execute • Both in Gradle and outside Gradle • Identify the secrets to inject • Use environment variables

Slide 71

Slide 71 text

Configure CircleCI • YAML file • /.circleci/config.yml • Versioned • Currently 2.1

Slide 72

Slide 72 text

version: 2.1 executors: android: working_directory: ~/squanchy docker: - image: circleci/android:api-28 environment: ANDROID_HOME: /opt/android/sdk APPLICATION_ID: net.squanchy.example FABRIC_API_KEY: 0000000000000000000000000000000000000000 ALGOLIA_APPLICATION_ID: ABCDEFGH12 ALGOLIA_API_KEY: 00000000000000000000000000000000 ALGOLIA_INDICES_PREFIX: squanchy_dev commands: # Build Tools cache commands restore_build_tools_cache: steps: - restore_cache: name: Restore Android build tools cache keys: - v3-build-tools-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} save_build_tools_cache: steps: - save_cache: name: Save Android build tools cache paths: - /opt/android/sdk/build-tools key: v3-build-tools-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} # Gradle cache commands restore_gradle_cache: steps: - restore_cache: name: Restore Gradle dependencies cache keys: - v1-gradle-dependencies-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} - restore_cache: name: Restore Gradle wrapper cache keys: - v1-gradle-wrapper-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle/wrapper/gradle-wrapper.properties" }} save_gradle_cache: steps: - save_cache: name: Save Gradle dependencies cache paths: - ~/.gradle/caches key: v1-gradle-dependencies-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} - save_cache: name: Save Gradle wrapper cache paths: - ~/.gradle/wrapper key: v1-gradle-wrapper-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle/wrapper/gradle-wrapper.properties" }} # Android Gradle build cache commands restore_android_build_cache: steps: - restore_cache: name: Restore Android Gradle build cache keys: - v3-build-cache-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} save_android_build_cache: steps: - save_cache: name: Save Android Gradle build cache paths: - ~/.android/build-cache key: v3-build-cache-{{ checksum "workspace/repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} ensure_android_sdk_is_ready: steps: - run: name: Ensure Android SDK install is up-to-date command: workspace/repo/.circleci/ci-scripts/ensure-sdkmanager.sh download_gradle_dependencies: steps: - run: name: Download Gradle dependencies command: cd workspace/repo/ && ./gradlew downloadDependencies restore_workspace: steps: - attach_workspace: at: workspace jobs: checkout: executor: android steps: - checkout: path: workspace/repo # Prepare the container for the build - ensure_android_sdk_is_ready - run: name: Create mock Play Services JSON command: workspace/repo/.circleci/ci-scripts/ci-mock-google-services-setup.sh # Persist repo code - persist_to_workspace: root: workspace paths: - repo prepare_for_checks: executor: android steps: - restore_workspace - restore_gradle_cache - restore_android_build_cache - restore_build_tools_cache - download_gradle_dependencies - save_android_build_cache - save_gradle_cache - save_build_tools_cache # Persist built app code - persist_to_workspace: root: workspace paths: - repo/app/build static_analysis: executor: android steps: - restore_workspace - restore_gradle_cache - restore_android_build_cache - restore_build_tools_cache # See https://issuetracker.google.com/issues/62217354 for the parallelism option - run: name: Run static analysis command: cd workspace/repo && ./gradlew lintRelease detekt ktlintCheck -Djava.util.concurrent.ForkJoinPool.common.parallelism=2 # Collect static analysis reports as build artifacts - store_artifacts: path: workspace/repo/app/build/reports destination: reports tests: executor: android steps: - restore_workspace - restore_gradle_cache - restore_android_build_cache - restore_build_tools_cache # See https://issuetracker.google.com/issues/62217354 for the parallelism option - run: name: Run unit tests command: cd workspace/repo && ./gradlew testRelease -Djava.util.concurrent.ForkJoinPool.common.parallelism=2 # Collect JUnit test results - store_test_results: path: workspace/repo/app/build/test-results workflows: version: 2 build_and_test: jobs: - checkout - prepare_for_checks: requires: - checkout - static_analysis: requires: - prepare_for_checks - tests: requires: - prepare_for_checks

Slide 73

Slide 73 text

version: 2.1 #... commands: jobs: workflows: #... #... #... executors:

Slide 74

Slide 74 text

version: 2.1 #... commands: jobs: workflows: #... #... #... executors:

Slide 75

Slide 75 text

workflows: #... jobs: #... commands: #... #... executors: version: 2.1 android: working_directory: ~/squanchy docker: - image: circleci/android:api-28 environment: ANDROID_HOME: /opt/android/sdk APPLICATION_ID: net.squanchy.example FABRIC_API_KEY: 0000000000000000000000000000000000000000 ALGOLIA_APPLICATION_ID: ABCDEFGH12 ALGOLIA_API_KEY: 00000000000000000000000000000000 ALGOLIA_INDICES_PREFIX: squanchy_dev

Slide 76

Slide 76 text

commands: #... executors: version: 2.1 android: working_directory: ~/squanchy docker: - image: circleci/android:api-28 environment: ANDROID_HOME: /opt/android/sdk APPLICATION_ID: net.squanchy.example FABRIC_API_KEY: 0000000000000000000000000000000000000000 ALGOLIA_APPLICATION_ID: ABCDEFGH12 ALGOLIA_API_KEY: 00000000000000000000000000000000 ALGOLIA_INDICES_PREFIX: squanchy_dev

Slide 77

Slide 77 text

workflows: #... jobs: #... commands: #... #... executors: version: 2.1 # Build Tools cache commands restore_build_tools_cache: steps: - restore_cache: name: Restore Android build tools cache keys: - v3-build-tools-{{ checksum "workspace/ repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum

Slide 78

Slide 78 text

commands: #... # Build Tools cache commands restore_build_tools_cache: steps: - restore_cache: name: Restore Android build tools cache keys: - v3-build-tools-{{ checksum "workspace/ repo/.circleci/config.yml" }}-{{ checksum "workspace/repo/gradle.properties" }}-{{ checksum "workspace/repo/dependencies.gradle" }} save_build_tools_cache: steps: - save_cache: name: Save Android build tools cache paths:

Slide 79

Slide 79 text

command_name: steps: - step_name: name: Description (shown in UI) other_stuff: - {depends on step}

Slide 80

Slide 80 text

- save_cache: name: Saves path(s) to CircleCI cache paths: - /path/to/save keys: - v1-my-cache-{{ checksum "some/thing" }}

Slide 81

Slide 81 text

- restore_cache: name: Restores from CircleCI cache keys: - v1-my-cache-{{ checksum "some/thing" }}

Slide 82

Slide 82 text

- attach_workspace: at: path/to/attach/to

Slide 83

Slide 83 text

- run: name: Executes something in workspace command: my-script.sh param1 param2

Slide 84

Slide 84 text

#... workflows: jobs: #... commands: restore_build_tools_cache save_build_tools_cache restore_gradle_cache save_gradle_cache restore_android_build_cache save_android_build_cache ensure_android_sdk_is_ready download_gradle_dependencies restore_workspace

Slide 85

Slide 85 text

workflows: #... jobs: #... commands: #... #... executors: version: 2.1 checkout: executor: android steps: - checkout: path: workspace/repo

Slide 86

Slide 86 text

jobs: #... checkout: executor: android steps: - checkout: path: workspace/repo # Prepare the container for the build - ensure_android_sdk_is_ready - run: name: Create mock Play Services JSON command: workspace/repo/.circleci/ci-scripts/ci-mock- google-services-setup.sh # Persist repo code - persist_to_workspace:

Slide 87

Slide 87 text

job_name: executor: executor_name steps: - step_name: name: Description (shown in UI) other_stuff: - {depends on step}

Slide 88

Slide 88 text

job_name: executor: executor_name steps: - step_name: name: Description (shown in UI) other_stuff: - {depends on step} - command_name

Slide 89

Slide 89 text

workflows: #... jobs: #... commands: #... #... executors: version: 2.1 checkout prepare_for_checks static_analysis tests

Slide 90

Slide 90 text

workflows: #... jobs: #... commands: #... #... executors: version: 2.1 version: 2 build_and_test: jobs:

Slide 91

Slide 91 text

workflows: #... #... version: 2 build_and_test: jobs: - checkout - prepare_for_checks: requires: - checkout - static_analysis: requires: - prepare_for_checks - tests: requires: - prepare_for_checks

Slide 92

Slide 92 text

workflows: #... #... version: 2 build_and_test: jobs: - checkout - prepare_for_checks: requires: - checkout - static_analysis: requires: - prepare_for_checks - tests: requires: - prepare_for_checks checkout static analysis prepare for checks tests Directed Acyclic Graph

Slide 93

Slide 93 text

A CI run in detail checkout static analysis prepare for checks tests

Slide 94

Slide 94 text

A CI run in detail checkout static analysis prepare for checks tests

Slide 95

Slide 95 text

A CI run in detail checkout setup android sdk create mock gms json persist to workspace checkout

Slide 96

Slide 96 text

A CI run in detail checkout setup android sdk create mock gms json persist to workspace checkout

Slide 97

Slide 97 text

A CI run in detail checkout static analysis prepare for checks tests

Slide 98

Slide 98 text

A CI run in detail restore from workspace restore caches download dependencies persist to workspace prepare for checks save caches

Slide 99

Slide 99 text

A CI run in detail restore from workspace restore caches download dependencies persist to workspace prepare for checks save caches

Slide 100

Slide 100 text

A CI run in detail checkout static analysis prepare for checks tests

Slide 101

Slide 101 text

A CI run in detail restore from workspace restore caches run tasks store artefacts static analysis and tests

Slide 102

Slide 102 text

A CI run in detail restore from workspace restore caches run tasks store artefacts static analysis and tests

Slide 103

Slide 103 text

get stuff

Slide 104

Slide 104 text

get stuff do stuff save stuff

Slide 105

Slide 105 text

What are today’s lessons? Conclusions

Slide 106

Slide 106 text

• Static analysis is good • Use static analysis! • Continuous Integration is good • Use a CI! • Static analysis is even better with a CI • Check out Squanchy • https://github.com/squanchy-dev/squanchy-android

Slide 107

Slide 107 text

THANK YOU bit.ly/Computer-Says-No SEBASTIANO POGGI @seebrock3r JetBrains NEIL HUTCHISON @nnnneeeiil Toyota Connected Europe