Automating Android Workflows with Github Actions

Automating Android Workflows with Github Actions

Companion slides for my talk about Android CI/CD powered by Github Actions, including platform models, CI topologies and unique challenges we can find when architecting Android pipelines.

Delivered in the following events

- GDG Curitiba Android Meetup #08 (online) - August / 2020
- TDC São Paulo Trilha Android (online) - August/2020

D4b7a3e2ed10f86e0b52498713ba2601?s=128

Ubiratan Soares

August 26, 2020
Tweet

Transcript

  1. AUTOMATING YOUR ANDROID WORKFLOWS WITH Ubiratan Soares August / 2020

    GITHUB ACTIONS
  2. Code Changes Tests Build Deployable Deploy Tests Versioning Release Continuous

    Integration Continuous Delivery
  3. None
  4. None
  5. PLATFORM MODEL

  6. Step Step Step Job . . . Workflow Runner Event

  7. Step Step Step Job . . . Workflow Runner Push

    Pull Request Release REST API Tag . . .
  8. Step Step Step Static Analysis . . . Pull Request

    Runner Runner Step Step Step . . . Tests
  9. Step Step Step Static Analysis . . . PR Runner

    Github-hosted Runner vCPUs RAM SSD Ubuntu 2 7 GB 14 GB MacOS Catalina 2 7 GB 14 GB Windows Server 2019 2 7 GB 14 GB
  10. Step Step Step PR . . . Runner Shell Script

    Github Action Github Runner JS Action Docker Action Ubuntu YES YES MacOS YES NO Windows Server YES NO
  11. Code action.yml java -jar step.jar action.yml ./gradlew build Code main.js

  12. main.js action.yml java -jar step.jar action.yml ./gradlew build Code Code

  13. jobs: build: # Something before steps: - uses: actions/cache@v1

  14. name: 'Cache' [. . .] inputs: path: description: 'A list

    of files, directories, and wildcard patterns to cache and restore' required: true key: description: 'An explicit key for restoring and saving the cache' required: true restore-keys: description: 'An ordered list of keys to use for restoring the cache if no cache hit occurred' required: false outputs: cache-hit: description: 'A boolean value to indicate an exact match was found for the primary key' https://github.com/actions/cache/blob/master/action.yml
  15. name: 'Cache' [. . .] inputs: path: description: 'A list

    of files, directories, and wildcard patterns to cache and restore' required: true key: description: 'An explicit key for restoring and saving the cache' required: true restore-keys: description: 'An ordered list of keys to use for restoring the cache if no cache hit occurred' required: false outputs: cache-hit: description: 'A boolean value to indicate an exact match was found for the primary key' https://github.com/actions/cache/blob/master/action.yml
  16. Some Job Step 02 Step 01 Step 03

  17. Some Job Step 02 Step 01 Step 03 Output =

    "Value A”
  18. Some Job Step 02 Step 01 Step 03 ENV =

    “Value B” Output = "Value A”
  19. Some Job Step 02 Step 01 Step 03 ENV =

    “Value B” Output = "Value A” (mutable) (immutable)
  20. DIVING IN THE CONFIGURATION

  21. Name Events Jobs pr.yml Context Step Step Name Context

  22. Name Jobs pr.yml Context Step Step Name Context name: pr-pipeline

    Events
  23. Events Jobs pr.yml Context Step Step Name Context on: push

    Name
  24. Events Jobs pr.yml Context Step Step Name Context on: [push,

    pull_request] Name
  25. Events Jobs pr.yml Context Step Step Name Context on: push:

    branches: - master pull_request: branches: - master release: types: - created Name
  26. Name Events Jobs pr.yml Context Step Step Name Context env:

    GRADLE_OPTS: -Dorg.gradle.daemon=false defaults: run: shell: bash working-directory: temp
  27. Name Events Jobs pr.yml Context Step Step Name Context jobs:

    static_analysis: # Detekt and Ktlint assemble: # Build debug APK unit_tests: # Run JVM tests acceptance_tests: # Run Espresso tests quality_checks: # Reports to quality gate system (eg Sonarqube)
  28. Name Events Jobs pr.yml Context Step Step Name Context jobs:

    unit_tests: runs-on: ubuntu-latest steps: - name: Checkout sources uses: actions/checkout@v2 - name: Run tests run: ./gradlew test
  29. LOCAL TESTING

  30. None
  31. ➜ github-actions-playground ✗ act -l pull_request ╭───────╮ │ hello │

    ╰───────╯ ➜ github-actions-playground ✗ act pull_request [Simple Workflow/hello] Start image=node:12.6-buster-s [Simple Workflow/hello] docker run image=node:12.6- [Simple Workflow/hello] ⭐ Run echo "Hello World" | Hello World [Simple Workflow/hello] ✅ Success - echo "Hello World” ➜ github-actions-playground ✗ name: Simple Workflow on: pull_request jobs: hello: runs-on: ubuntu-latest steps: - name: run: echo "Hello World"
  32. ➜ github-actions-playground ✗ act -l release ╭───────────╮ │ first_job │

    ╰───────────╯ ‑ ╭────────────╮ │ second_job │ ╰────────────╯ name: Two Jobs on: [release] jobs: first_job: runs-on: ubuntu-latest steps: - name: First step run: echo "Job = 1 | Step = 1" - name: Second step run: echo "Job = 1 | Step = 2" second_job: runs-on: ubuntu-latest needs: first_job steps: - name: First step run: echo "Job = 2 | Step = 1" - name: Second step run: echo "Job = 2 | Step = 2"
  33. ➜ github-actions-playground ✗ act -l release *DRYRUN* [Two Jobs/first_job ]

    Start image=node:12.6-buster-slim *DRYRUN* [Two Jobs/first_job ] docker run image=node:12.6-buster-slim entrypoint=["/usr/bin/tail" "-f" "/dev *DRYRUN* [Two Jobs/first_job ] ⭐ Run First step *DRYRUN* [Two Jobs/first_job ] ✅ Success - First step *DRYRUN* [Two Jobs/first_job ] ⭐ Run Second step *DRYRUN* [Two Jobs/first_job ] ✅ Success - Second step *DRYRUN* [Two Jobs/second_job] Start image=node:12.6-buster-slim *DRYRUN* [Two Jobs/second_job] docker run image=node:12.6-buster-slim entrypoint=["/usr/bin/tail" "-f" "/dev *DRYRUN* [Two Jobs/second_job] ⭐ Run First step *DRYRUN* [Two Jobs/second_job] ✅ Success - First step *DRYRUN* [Two Jobs/second_job] ⭐ Run Second step *DRYRUN* [Two Jobs/second_job] ✅ Success - Second step
  34. A COMPLETE PIPELINE

  35. name: Main on: [push] jobs: build: runs-on: macOS-latest steps: -

    name: Project Checkout uses: actions/checkout@v2.3.2 - name: Compute key for CI cache run: ./compute-ci-cache-key.sh key.txt - name: Setup cache uses: actions/cache@v2.1.0 continue-on-error: true with: path: ~/.gradle/caches key: ${{ runner.os }}-gradle-${{ hashFiles('key.txt') }} restore-keys: ${{ runner.os }}-gradle- https://github.com/dotanuki-labs/norris
  36. - name: Setup JDK 1.8 uses: actions/setup-java@v1.4.1 with: java-version: 1.8

    - name: Copy CI gradle.properties run: | mkdir -p ~/.gradle cp .github/githubci-gradle.properties ~/.gradle/gradle.properties - name: Resolve dependencies run: ./gradlew androidDependencies --no-daemon --stacktrace - name: Check code formatting run: ./gradlew ktlintCheck --no-daemon --stacktrace - name: Check code smells run: ./gradlew detekt --no-daemon --stacktrace - name: Run unit tests and capture code coverage run: ./gradlew clean jacocoTestReport jacocoTestReportDebug --no-daemon - name: Assemble APK run: ./gradlew assembleDebug -xlint --no-daemon --stacktrace https://github.com/dotanuki-labs/norris
  37. - name: Assemble APK run: ./gradlew assembleDebug -xlint --no-daemon --stacktrace

    - name: Run functional acceptance tests with Espresso uses: reactivecircus/android-emulator-runner@v2.11.0 with: api-level: 29 arch: x86_64 script: ./gradlew :app:connectedDebugAndroidTest --stacktrace - name: Archive build reports if: always() uses: actions/upload-artifact@v2.1.4 with: name: build-reports path: app/build/reports - name: Archive build outputs if: always() uses: actions/upload-artifact@v2.1.4 with: name: build-outputs path: app/build/outputs https://github.com/dotanuki-labs/norris
  38. - name: Collect all test results from all modules if:

    always() run: | mkdir -p junit find . -type f -regex ".*/build/test-results/.*xml" -exec cp {} junit\; - name: Archive test results if: always() uses: actions/upload-artifact@v2.1.4 with: name: junit-results path: junit - name: Share test reports with Codecov uses: codecov/codecov-action@v1.0.12 https://github.com/dotanuki-labs/norris
  39. Push Checkout Restore Cache Setup JDK Cache Key Ktlint Check

    Detekt Check Run Unit Tests Collect reports Archive reports Codecov Assemble Debug Espresso Tests Collect outputs Archive outputs •1 Jobs •14 Steps •1 VM
  40. PIPELINES AND TOPOLOGIES

  41. jobs: first: steps: # Your steps here second: needs: first

    steps: # Your steps here Step first Step Step second Step Step Sequential Jobs
  42. jobs: first: steps: # Steps for first job second: steps:

    # Steps for second job third: needs: [first, second] steps: # Steps for third job Step first Step Step second Step third Step Step Fan-in to Job
  43. Step first Step Step second Step third Step Step jobs:

    first: steps: # Steps for first job second: needs: first steps: # Steps for second job third: needs: first steps: # Steps for third job Fan-out from Job
  44. Step first Step Step second Step Step Step third Step

    Artifacts Upload Upload Download
  45. None
  46. None
  47. None
  48. None
  49. None
  50. None
  51. None
  52. None
  53. None
  54. None
  55. None
  56. None
  57. Push Checkout Static Analysis Restore Cache Setup JDK Cache Key

    Ktlint Check Detekt Check Checkout Unit Tests Restore Cache Cache Key Setup JDK Run Unit Tests Collect reports Codecov Checkout Assemble APK Restore Cache Setup JDK Cache Key Assemble Debug Checkout Acceptance Tests Restore Cache Setup JDK Cache Key Espresso Tests Collect outputs Archive outputs •4 Jobs •25 Steps •Up to 3 VMs
  58. BUILDING WITH MATRICES

  59. None
  60. None
  61. name: Pre Merge Checks on: push: branches: - master jobs:

    gradle: if: ${{ !contains(github.event.head_commit.message, 'ci skip') }} strategy: fail-fast: false matrix: os: [ubuntu-latest, macos-latest, windows-latest] jdk: [8, 11, 14] runs-on: ${{ matrix.os }} env: JDK_VERSION: ${{ matrix.jdk }} GRADLE_OPTS: -Dorg.gradle.daemon=false
  62. name: Pre Merge Checks on: push: branches: - master jobs:

    gradle: if: ${{ !contains(github.event.head_commit.message, 'ci skip') }} strategy: fail-fast: false matrix: os: [ubuntu-latest, macos-latest, windows-latest] jdk: [8, 11, 14] runs-on: ${{ matrix.os }} env: JDK_VERSION: ${{ matrix.jdk }} GRADLE_OPTS: -Dorg.gradle.daemon=false
  63. steps: - name: Checkout Repo uses: actions/checkout@v2 # Let's cleanup

    the gradle cache folders to make sure # we don't accidentally cache stale files. - name: Cleanup Gradle Folders shell: bash run: | rm -rf ~/.gradle/caches/ && \ rm -rf ~/.gradle/wrapper/ - name: Cache Gradle Folders uses: actions/cache@v2 with: path: | ~/.gradle/caches/ ~/.gradle/wrapper/ key: cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}-${{ hashFiles(‘...’) }} restore-keys: | cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}- cache-gradle-${{ matrix.os }}- cache-gradle-
  64. steps: - name: Checkout Repo uses: actions/checkout@v2 # Let's cleanup

    the gradle cache folders to make sure # we don't accidentally cache stale files. - name: Cleanup Gradle Folders shell: bash run: | rm -rf ~/.gradle/caches/ && \ rm -rf ~/.gradle/wrapper/ - name: Cache Gradle Folders uses: actions/cache@v2 with: path: | ~/.gradle/caches/ ~/.gradle/wrapper/ key: cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}-${{ hashFiles(‘...’) }} restore-keys: | cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}- cache-gradle-${{ matrix.os }}- cache-gradle-
  65. - name: Setup Java uses: actions/setup-java@v1 with: java-version: ${{ matrix.jdk

    }} - name: Build detekt run: ./gradlew build :detekt-cli:shadowJarExecutable --parallel - name: Run detekt-cli --help run: java -jar ./detekt-cli/build/run/detekt --help - name: Run detekt-cli with argsfile run: java -jar ./detekt-cli/build/run/detekt "@./config/detekt/argsfile"
  66. - name: Setup Java uses: actions/setup-java@v1 with: java-version: ${{ matrix.jdk

    }} - name: Build detekt run: ./gradlew build :detekt-cli:shadowJarExecutable --parallel - name: Run detekt-cli --help run: java -jar ./detekt-cli/build/run/detekt --help - name: Run detekt-cli with argsfile run: java -jar ./detekt-cli/build/run/detekt "@./config/detekt/argsfile"
  67. Matrix (A, B) Step Step Job . . . Step

    Step Job . . . Step Step Job . . . Step Step Job . . . A0, B0 A1, B0 A2, B0 An, Bm . . .
  68. None
  69. None
  70. DISCOVERING PIPELINE BLOCKS

  71. None
  72. None
  73. None
  74. None
  75. CHALENGES OF ANDROID

  76. Code Changes Tests Build Deployable CD ??? CI

  77. Decision Matrix Merged PR Push Cherry pick Tag Nightly Static

    Analysis ✓ ✓ ✓ Unit tests ✓ ✓ ✓ ✓ Espresso tests ✓ ✓ ✓ ✓ Smoke Tests ✓ ✓ ✓ ✓ Security Analysis ✓ ✓ ✓ ✓ Build ✓ ✓ ✓ ✓ Artifacts generation ✓ ✓ ✓ ✓ Artifact distribution ✓ ✓
  78. Decision Matrix Merged PR Push Cherry pick Tag Nightly Static

    Analysis ✓ ✓ ✓ Unit tests ✓ ✓ ✓ ✓ Espresso tests ✓ ✓ ✓ Smoke Tests ✓ ✓ Security Analysis ✓ Build ✓ ✓ ✓ ✓ Artifacts generation ✓ ✓ ✓ ✓ Artifact distribution ✓ ✓
  79. None
  80. None
  81. None
  82. None
  83. JOB JOB JOB - name: Setup cache uses: actions/cache@v2.1.1 with:

    path: ~/.gradle/caches key: ${{ runner.os }}-gradle-${{ hashFiles(‘...’) }} restore-keys: ${{ runner.os }}-gradle- put(key) get(key) put(key) get(key) time
  84. JOB JOB JOB - name: Setup cache uses: actions/cache@v2.1.1 with:

    path: ~/.gradle/caches key: ${{ runner.os }}-gradle-${{ hashFiles(‘...’) }} restore-keys: ${{ runner.os }}-gradle- put(key) get(key) put(key) get(key) time
  85. JOB JOB JOB - name: Setup cache uses: actions/cache@v2.1.1 with:

    path: ~/.gradle/caches key: ${{ runner.os }}-gradle-${{ hashFiles(‘key’) }} restore-keys: ${{ runner.os }}-gradle- put(key) get(key) put(key) get(key) time
  86. ➜ wireguard-android git:(master) ✗ tree -L 4 . ├── COPYING

    ├── README.md ├── app │ ├── app.iml │ ├── build │ ├── build.gradle │ ├── nonnull.gradle │ ├── proguard-rules.pro │ ├── src │ └── tools ├── build.gradle ├── gradle │ └── wrapper │ ├── gradle-wrapper.jar │ └── gradle-wrapper.properties ├── gradle.properties ├── gradlew └── settings.gradle
  87. ➜ wireguard-android git:(master) ✗ tree -L 4 . ├── COPYING

    ├── README.md ├── app │ ├── app.iml │ ├── build │ ├── build.gradle │ ├── nonnull.gradle │ ├── proguard-rules.pro │ ├── src │ └── tools ├── build.gradle ├── gradle │ └── wrapper │ ├── gradle-wrapper.jar │ └── gradle-wrapper.properties ├── gradle.properties ├── gradlew └── settings.gradle key hash
  88. ➜ plaid git:(master) ✗ tree -L 4 . ├── about

    ├── app │ └── build.gradle ├── build ├── build.gradle ├── core ├── designernews ├── dribbble ├── gradle │ └── wrapper │ ├── gradle-wrapper.jar │ └── gradle-wrapper.properties ├── gradle.properties ├── gradlew ├── repositories.gradle ├── screenshots ├── search ├── settings.gradle ├── shared_dependencies.gradle ├── test_dependencies.gradle ├── test_shared └── third_party
  89. ➜ plaid git:(master) ✗ tree -L 4 . ├── about

    ├── app │ └── build.gradle ├── build ├── build.gradle ├── core ├── designernews ├── dribbble ├── gradle │ └── wrapper │ ├── gradle-wrapper.jar │ └── gradle-wrapper.properties ├── gradle.properties ├── gradlew ├── repositories.gradle ├── screenshots ├── search ├── settings.gradle ├── shared_dependencies.gradle ├── test_dependencies.gradle ├── test_shared └── third_party key hash
  90. ➜ norris git:(master) tree -L 4 . ├── app ├──

    build.gradle.kts ├── buildSrc │ ├── build.gradle.kts │ └── src │ └── main │ └── kotlin │ ├── BuildPlugins.kt │ ├── Libraries.kt │ ├── configs │ ├── dependencies │ ├── plugins │ └── versioning.kt ├── features ├── gradle │ └── wrapper │ ├── gradle-wrapper.jar │ └── gradle-wrapper.properties ├── gradle.properties ├── gradlew ├── platform └── settings.gradle.kts
  91. ➜ norris git:(master) tree -L 4 . ├── app ├──

    build.gradle.kts ├── buildSrc │ ├── build.gradle.kts │ └── src │ └── main │ └── kotlin │ ├── BuildPlugins.kt │ ├── Libraries.kt │ ├── configs │ ├── dependencies │ ├── plugins │ └── versioning.kt ├── features ├── gradle │ └── wrapper │ ├── gradle-wrapper.jar │ └── gradle-wrapper.properties ├── gradle.properties ├── gradlew ├── platform └── settings.gradle.kts key hash
  92. #!/bin/bash RESULT_FILE=$1 if [ -f $RESULT_FILE ]; then rm $RESULT_FILE

    fi touch $RESULT_FILE checksum_file() { echo $(openssl md5 $1 | awk '{print $2}') } https://github.com/chrisbanes/tivi/blob/main/checksum.sh
  93. FILES=() while read -r -d ''; do FILES+=("$REPLY") done <

    <(find . -type f \( -name "build.gradle*" -o -name "dependencies.kt" \ -o -name "gradle-wrapper.properties" \) -print0) # Loop through files and append MD5 to result file for FILE in ${FILES[@]}; do echo $(checksum_file $FILE) >> $RESULT_FILE done # Now sort the file so that it is sort $RESULT_FILE -o $RESULT_FILE https://github.com/chrisbanes/tivi/blob/main/checksum.sh
  94. FILES=() while read -r -d ''; do FILES+=("$REPLY") done <

    <(find . -type f \( -name "build.gradle*" -o -name "dependencies.kt" \ -o -name "gradle-wrapper.properties" \) -print0) # Loop through files and append MD5 to result file for FILE in ${FILES[@]}; do echo $(checksum_file $FILE) >> $RESULT_FILE done # Now sort the file so that it is sort $RESULT_FILE -o $RESULT_FILE https://github.com/chrisbanes/tivi/blob/main/checksum.sh
  95. None
  96. THE END IS NEAR

  97. Final Remarks • Open-source model for reusable Actions is quite

    powerful • Mixed execution model (Cloud + Self-hosted Runners) is handy! • Lacking some features, but with a great release peace • Very promissig solution with a great Developer Experience
  98. UBIRATAN SOARES Brazilian Computer Scientist Senior Software Engineer @ N26

    GDE for Android and Kotlin @ubiratanfsoares ubiratansoares.dev
  99. https://speakerdeck.com/ubiratansoares

  100. THANKS