Automating Android Workflows with Github Actions

Companion slides for my talk about Android CI/CD powered by Github Actions, including platform models, CI topologies and unique challenges we can find when automating Android pipelines withing this platform.

Delivered in the following events :

- GDG Curitiba Android Meetup #08 (online) - August / 2020
- TDC São Paulo Trilha Android (online) - August/2020
- DevCommunity Summit (online) - October/2021
- Droidcon Berlin - October/2021

Transcript

1. AUTOMATING YOUR ANDROID WORKFLOWS Ubiratan Soares October / 2021 GITHUB ACTIONS
2. CRASHING COURSE !!!
3. Step Step Step Job . . . Workflow Runner Event
4. Job . . . Workflow Push Pull Request Release REST API Tag . . . Job . . . Job . . .
5. Pull Request Build . . . Internal Library • 1 Event • 1 Job triggered
6. Tag Build . . . Internal Library Publish . . . • 1 Event • 1 Job triggered • 1 Job chained
7. <project-dir>/ .github/workflows compliance-checks.yaml ci.yaml etc.yaml
8. What are your thoughts about YAML? Software Engineer
9. Job Runner
10. Step Job Runner Shell Script Command
11. Step Job Runner Shell Script Command Step Composite Action (Bash)
12. Step Job Runner Shell Script Command Step Composite Action (Bash) Step Nested Composite Action
13. Step Job Runner Shell Script Command Step Composite Action (Bash) Step Nested Composite Action Step Code-defined
14. Action Type | Runner Ubuntu MacOS Windows JS YES YES YES Docker YES NO NO Composite/Simple YES YES YES Composite/Nested YES 👀 👀 Step Job Step Runner . . . by
19. name: Android CI on: pull_request: push: branches: - 'master' - '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
20. name: Android CI on: pull_request: push: branches: - 'master' - '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
21. name: Android CI on: pull_request: push: branches: - 'master' - '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
22. name: Android CI on: pull_request: push: branches: - 'master' - '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
23. name: Android CI on: pull_request: push: branches: - 'master' - '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 signalapp/Signal-Android/blob/master/.github/workflows/android.yml
24. steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
25. steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
26. steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
27. steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
28. name: Android CI on: pull_request: push: branches: - 'master' - '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' • Multi-triggers • 1 Job • 5 mandatory Steps • 1 conditional Step github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
29. jobs: build: # Something before steps: - uses: actions/cache@v1
30. name: 'Cache' [. . .] inputs: path: description: 'A list of files, directories, and wildcard patterns to cache and restore' required: true key: description: 'An explicit key for restoring and saving the cache' required: true restore-keys: description: 'An ordered list of keys to use for restoring the cache if no cache hit occurred' required: false outputs: cache-hit: description: 'A boolean value to indicate an exact match was found for the primary key' https://github.com/actions/cache/blob/master/action.yml
31. name: 'Cache' [. . .] inputs: path: description: 'A list of files, directories, and wildcard patterns to cache and restore' required: true key: description: 'An explicit key for restoring and saving the cache' required: true restore-keys: description: 'An ordered list of keys to use for restoring the cache if no cache hit occurred' required: false outputs: cache-hit: description: 'A boolean value to indicate an exact match was found for the primary key' https://github.com/actions/cache/blob/master/action.yml
33. SCALING PIPELINES
34. Step Step Job . . . Workflow Runner Github-hosted Runner vCPUs RAM SSD Ubuntu 2 7 GB 14 GB MacOS 3 14 GB 14 GB Windows 2 7 GB 14 GB by
35. Github-hosted Runner Self-hosted Runner
36. Step Job 1 Step Step Job 2 Step Step Job N Step . . . RUNNER RUNNER RUNNER Step Single Job . . . RUNNER vs Step Step Step Step Step 🤔
37. Build Matrix
41. name: Pre Merge Checks on: push: branches: - main pull_request: branches: - '**' jobs: gradle: if: ${{ !contains(github.event.head_commit.message, 'ci skip') }} strategy: fail-fast: false matrix: os: [ubuntu-latest, macos-latest, windows-latest] jdk: [8, 11, 17] runs-on: ${{ matrix.os }} env: JDK_VERSION: ${{ matrix.jdk }} GRADLE_OPTS: -Dorg.gradle.daemon=false
42. name: Pre Merge Checks on: push: branches: - main pull_request: branches: - '**' jobs: gradle: if: ${{ !contains(github.event.head_commit.message, 'ci skip') }} strategy: fail-fast: false matrix: os: [ubuntu-latest, macos-latest, windows-latest] jdk: [8, 11, 17] runs-on: ${{ matrix.os }} env: JDK_VERSION: ${{ matrix.jdk }} GRADLE_OPTS: -Dorg.gradle.daemon=false
43. steps: - name: Checkout Repo uses: actions/checkout@v2 - name: Cache Gradle Folders uses: actions/cache@v2 with: path: | ~/.gradle/caches/ ~/.gradle/wrapper/ key: cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}-${{ hashFiles('gradle/libs.versions.toml') restore-keys: | cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}- cache-gradle-${{ matrix.os }}- cache-gradle- - name: Setup Java uses: actions/setup-java@v2 with: java-version: ${{ matrix.jdk }} distribution: 'adopt' - name: Build detekt run: ./gradlew build moveJarForIntegrationTest -x detekt
44. steps: - name: Checkout Repo uses: actions/checkout@v2 - name: Cache Gradle Folders uses: actions/cache@v2 with: path: | ~/.gradle/caches/ ~/.gradle/wrapper/ key: cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}-${{ hashFiles('gradle/libs.versions.toml') restore-keys: | cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}- cache-gradle-${{ matrix.os }}- cache-gradle- - name: Setup Java uses: actions/setup-java@v2 with: java-version: ${{ matrix.jdk }} distribution: 'adopt' - name: Build detekt run: ./gradlew build moveJarForIntegrationTest -x detekt
45. Matrix (A, B) Step Step Job . . . Step Step Job . . . Step Step Job . . . Step Step Job . . . A0, B0 A1, B0 A2, B0 An, Bm . . .
48. Custom Topologies
49. Event Job 2 Job 3 Job 5 Job 6 Job 7 Job 4 Job 1
50. jobs: first: steps: # Your steps here second: needs: first steps: # Your steps here Step first Step Step second Step Step Sequential Jobs
51. jobs: first: steps: # Steps for first job second: steps: # Steps for second job third: needs: [first, second] steps: # Steps for third job Step first Step Step second Step third Step Step Fan-in to Job
52. Step first Step Step second Step third Step Step jobs: first: steps: # Steps for first job second: needs: first steps: # Steps for second job third: needs: first steps: # Steps for third job Fan-out from Job
55. https://github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml acceptance_tests: runs-on: ubuntu-20.04 timeout-minutes: 15 needs: [assemble_apk, espresso_prepare, unit_tests, static_analysis] steps: - name: Project Checkout uses: actions/checkout@v2.3.5 - name: Fetch Instrumentation artefacts uses: actions/download-artifact@v2.0.10 - name: Run Espresso tests over emulator.wtf run: ./scripts/emulator-wtf.sh ${{ secrets.EMULATOR_WTF_TOKEN }} - name: Archive execution results if: success() uses: actions/upload-artifact@v2.2.4 with: name: emulator-wtf-results path: emulator-wtf-results
56. https://github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml acceptance_tests: runs-on: ubuntu-20.04 timeout-minutes: 15 needs: [assemble_apk, espresso_prepare, unit_tests, static_analysis] steps: - name: Project Checkout uses: actions/checkout@v2.3.5 - name: Fetch Instrumentation artefacts uses: actions/download-artifact@v2.0.10 - name: Run Espresso tests over emulator.wtf run: ./scripts/emulator-wtf.sh ${{ secrets.EMULATOR_WTF_TOKEN }} - name: Archive execution results if: success() uses: actions/upload-artifact@v2.2.4 with: name: emulator-wtf-results path: emulator-wtf-results
57. STUDY CASE
58. https://github.com/dotanuki-labs/norris + 📸 📸 📸
60. <module-dir>/ .gitignore src/androidTest/<package>/ScreenshotTests.kt screenshots/debug your.nice.package.SomeTestName.png your.nice.package.AnotherTestName.png your.nice.package.OneMoreTest.png . . .
61. <module-dir>/ .gitignore src/androidTest/<package>/ScreenshotTests.kt screenshots/debug your.nice.package.SomeTestName.png your.nice.package.AnotherTestName.png your.nice.package.OneMoreTest.png . . .
62. <module-dir>/ .gitignore src/androidTest/<package>/ScreenshotTests.kt screenshots/debug your.nice.package.SomeTestName.png your.nice.package.AnotherTestName.png your.nice.package.OneMoreTest.png . . .
63. Hooking over 3rd party Gradle plugins / tasks
64. What if I told you … there is another way !!!
65. 🔥 • Have the 2 sets of images available beforehand • Define target device at pipeline time • Copy images into convention folder according device spec • Spawn emulator according device spec • Invoke target Gradle task and run tests !!!
68. name: Main jobs: # Other jobs screenshot_tests: runs-on: macOS-10.15 needs: [acceptance_tests] timeout-minutes: 20 strategy: fail-fast: true matrix: device: ['nexus4', 'pixel'] github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125
69. name: Main jobs: # Other jobs screenshot_tests: runs-on: macOS-10.15 needs: [acceptance_tests] timeout-minutes: 20 strategy: fail-fast: true matrix: device: ['nexus4', 'pixel'] github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125
70. name: Main jobs: # Other jobs screenshot_tests: runs-on: macOS-10.15 needs: [acceptance_tests] timeout-minutes: 20 strategy: fail-fast: true matrix: device: ['nexus4', 'pixel'] github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125
71. steps: - name: Project Checkout uses: actions/checkout@v2.3.4 - name: Synchronize screenshots run: ./scripts/screenshots-sync.sh ${{ matrix.device }} - name: Assign emulator profile id: emulator-profile uses: ./.github/actions/assign-emulator-profile with: device: ${{ matrix.device }} - name: Run Screenshot tests uses: reactivecircus/android-emulator-runner@v2.20.0 with: api-level: 28 target: 'google_apis' profile: ${{ steps.emulator-profile.outputs.assigned }} script: ./gradlew clean executeScreenTests --no-daemon --stacktrace github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125
72. steps: - name: Project Checkout uses: actions/checkout@v2.3.4 - name: Synchronize screenshots run: ./scripts/screenshots-sync.sh ${{ matrix.device }} - name: Assign emulator profile id: emulator-profile uses: ./.github/actions/assign-emulator-profile with: device: ${{ matrix.device }} - name: Run Screenshot tests uses: reactivecircus/android-emulator-runner@v2.20.0 with: api-level: 28 target: 'google_apis' profile: ${{ steps.emulator-profile.outputs.assigned }} script: ./gradlew clean executeScreenTests --no-daemon --stacktrace github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125
73. steps: - name: Project Checkout uses: actions/checkout@v2.3.4 - name: Synchronize screenshots run: ./scripts/screenshots-sync.sh ${{ matrix.device }} - name: Assign emulator profile id: emulator-profile uses: ./.github/actions/assign-emulator-profile with: device: ${{ matrix.device }} - name: Run Screenshot tests uses: reactivecircus/android-emulator-runner@v2.20.0 with: api-level: 28 target: 'google_apis' profile: ${{ steps.emulator-profile.outputs.assigned }} script: ./gradlew clean executeScreenTests --no-daemon --stacktrace github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125 Internal GHA “nexus4” “Nexus 4”
74. steps: - name: Project Checkout uses: actions/checkout@v2.3.4 - name: Synchronize screenshots run: ./scripts/screenshots-sync.sh ${{ matrix.device }} - name: Assign emulator profile id: emulator-profile uses: ./.github/actions/assign-emulator-profile with: device: ${{ matrix.device }} - name: Run Screenshot tests uses: reactivecircus/android-emulator-runner@v2.20.0 with: api-level: 28 target: 'google_apis' profile: ${{ steps.emulator-profile.outputs.assigned }} script: ./gradlew clean executeScreenTests --no-daemon --stacktrace github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125 Internal GHA “nexus4” “Nexus 4”
75. #! /usr/bin/env bash set -e readonly features=( "search" "facts" ) readonly device="$1" sync() { echo -e "Syncing screenshots for device profile : $device" for feature in "${features[@]}" do origin="features/$feature/screenshots/$device" destination="features/$feature/screenshots/debug" rm -rf $destination echo -e "✔ Cleaned contents at $destination" cp -R $origin $destination echo -e "✔ Copied images from $origin$ to $destination" done } github.com/dotanuki-labs/norris/blob/master/scripts/screenshots-sync.sh
76. #! /usr/bin/env bash set -e readonly features=( "search" "facts" ) readonly device="$1" sync() { echo -e "Syncing screenshots for device profile : $device" for feature in "${features[@]}" do origin="features/$feature/screenshots/$device" destination="features/$feature/screenshots/debug" rm -rf $destination echo -e "✔ Cleaned contents at $destination" cp -R $origin $destination echo -e "✔ Copied images from $origin$ to $destination" done } github.com/dotanuki-labs/norris/blob/master/scripts/screenshots-sync.sh
79. ANDROID TIPS++
80. Emulators on CI
84. Definining Timeouts
85. name: Main on: pull_request: push: branches: - master jobs: static_analysis: runs-on: ubuntu-20.04 timeout-minutes: 10 steps: # Previous steps - name: Check code formatting run: ./gradlew ktlintCheck detekt —stacktrace https://github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml
86. name: Main on: pull_request: push: branches: - master jobs: static_analysis: runs-on: ubuntu-20.04 timeout-minutes: 10 steps: # Previous steps - name: Check code formatting run: ./gradlew ktlintCheck detekt —stacktrace https://github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml
89. Gradle Caching
90. JOB Save Restore Cloud Storage JOB Save Restore Cloud Storage JOB Workflow Run Workflow Run Workflow Run
91. Hard questions around CI caching • How do we compute entry-keys that maximize cache hits? • What should be cached per execution? • Should we segregate cached files per Job? • Etc 🤔
92. Github Actions Required Configuration Layered Caching Tasks Caching Distribution Caching Segregation per Job actions/setup-java (built-in) Easiest (Zero conf) No Yes Yes No actions/cache Hard (All custom) No Opt-in per Configuration Opt-in per Configuration Possible gradle/gradle-build-action Easy (Sensible Defaults) No Yes Yes Yes burrunan/gradle-cache-action Easy (Sensible Defaults) Yes Yes Yes Yes
93. Food for thought
94. Task | Condition Merged PR Push Cherry pick Tag Nightly Static Analysis ✓ ✓ ✓ Unit tests ✓ ✓ ✓ ✓ Espresso tests ✓ ✓ ✓ ✓ Smoke Tests ✓ ✓ ✓ ✓ Security Analysis ✓ ✓ ✓ ✓ Build ✓ ✓ ✓ ✓ Artifacts generation ✓ ✓ ✓ ✓ Artifact distribution ✓ ✓
95. Decision Matrix Merged PR Push Cherry pick Tag Nightly Static Analysis ✓ ✓ ✓ Unit tests ✓ ✓ ✓ ✓ Espresso tests ✓ 🤔 ✓ ✓ 🤔 Smoke Tests 🤔 🤔 ✓ ✓ 🤔 Security Analysis 🤔 🤔 🤔 ✓ Build ✓ ✓ 🤔 ✓ ✓ Artifacts generation ✓ ✓ ✓ ✓ Artifact distribution ✓ ✓
96. Code Changes Tests Build Deployable Deploy Release Continuous Integration Continuous Delivery Monitoring E2E Tests
97. Code Changes Tests Build Deployable CD ??? CI
98. FINAL REMARKS
99. TL;DR • Open-source model for reusable Actions is quite powerful • Pipelines can scale both horizontally and vertically • Awesome option for open-source projects • Unique challenges for big Android projects
100. UBIRATAN SOARES Brazilian Computer Scientist Senior Software Engineer @ N26 GDE for Android and Kotlin @ubiratanfsoares ubiratansoares.dev
101. THANKS