Save 37% off PRO during our Black Friday Sale! »

Automating Android Workflows with Github Actions

Automating Android Workflows with Github Actions

Companion slides for my talk about Android CI/CD powered by Github Actions, including platform models, CI topologies and unique challenges we can find when automating Android pipelines withing this platform.

Delivered in the following events :

- GDG Curitiba Android Meetup #08 (online) - August / 2020
- TDC São Paulo Trilha Android (online) - August/2020
- DevCommunity Summit (online) - October/2021
- Droidcon Berlin - October/2021

D4b7a3e2ed10f86e0b52498713ba2601?s=128

Ubiratan Soares
PRO

August 26, 2020
Tweet

Transcript

  1. AUTOMATING YOUR ANDROID WORKFLOWS Ubiratan Soares October / 2021 GITHUB

    ACTIONS
  2. CRASHING COURSE !!!

  3. Step Step Step Job . . . Workflow Runner Event

  4. Job . . . Workflow Push Pull Request Release REST

    API Tag . . . Job . . . Job . . .
  5. Pull Request Build . . . Internal Library • 1

    Event • 1 Job triggered
  6. Tag Build . . . Internal Library Publish . .

    . • 1 Event • 1 Job triggered • 1 Job chained
  7. <project-dir>/ .github/workflows compliance-checks.yaml ci.yaml etc.yaml

  8. What are your thoughts about YAML? Software Engineer

  9. Job Runner

  10. Step Job Runner Shell Script Command

  11. Step Job Runner Shell Script Command Step Composite Action (Bash)

  12. Step Job Runner Shell Script Command Step Composite Action (Bash)

    Step Nested Composite Action
  13. Step Job Runner Shell Script Command Step Composite Action (Bash)

    Step Nested Composite Action Step Code-defined
  14. Action Type | Runner Ubuntu MacOS Windows JS YES YES

    YES Docker YES NO NO Composite/Simple YES YES YES Composite/Nested YES 👀 👀 Step Job Step Runner . . . by
  15. None
  16. None
  17. None
  18. None
  19. name: Android CI on: pull_request: push: branches: - 'master' -

    '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  20. name: Android CI on: pull_request: push: branches: - 'master' -

    '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  21. name: Android CI on: pull_request: push: branches: - 'master' -

    '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  22. name: Android CI on: pull_request: push: branches: - 'master' -

    '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  23. name: Android CI on: pull_request: push: branches: - 'master' -

    '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  24. steps: - uses: actions/checkout@v2 - name: Set up JDK 11

    uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  25. steps: - uses: actions/checkout@v2 - name: Set up JDK 11

    uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  26. steps: - uses: actions/checkout@v2 - name: Set up JDK 11

    uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  27. steps: - uses: actions/checkout@v2 - name: Set up JDK 11

    uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  28. name: Android CI on: pull_request: push: branches: - 'master' -

    '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' • Multi-triggers • 1 Job • 5 mandatory Steps • 1 conditional Step github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  29. jobs: build: # Something before steps: - uses: actions/cache@v1

  30. name: 'Cache' [. . .] inputs: path: description: 'A list

    of files, directories, and wildcard patterns to cache and restore' required: true key: description: 'An explicit key for restoring and saving the cache' required: true restore-keys: description: 'An ordered list of keys to use for restoring the cache if no cache hit occurred' required: false outputs: cache-hit: description: 'A boolean value to indicate an exact match was found for the primary key' https://github.com/actions/cache/blob/master/action.yml
  31. name: 'Cache' [. . .] inputs: path: description: 'A list

    of files, directories, and wildcard patterns to cache and restore' required: true key: description: 'An explicit key for restoring and saving the cache' required: true restore-keys: description: 'An ordered list of keys to use for restoring the cache if no cache hit occurred' required: false outputs: cache-hit: description: 'A boolean value to indicate an exact match was found for the primary key' https://github.com/actions/cache/blob/master/action.yml
  32. None
  33. SCALING PIPELINES

  34. Step Step Job . . . Workflow Runner Github-hosted Runner

    vCPUs RAM SSD Ubuntu 2 7 GB 14 GB MacOS 3 14 GB 14 GB Windows 2 7 GB 14 GB by
  35. Github-hosted Runner Self-hosted Runner

  36. Step Job 1 Step Step Job 2 Step Step Job

    N Step . . . RUNNER RUNNER RUNNER Step Single Job . . . RUNNER vs Step Step Step Step Step 🤔
  37. Build Matrix

  38. None
  39. None
  40. None
  41. name: Pre Merge Checks on: push: branches: - main pull_request:

    branches: - '**' jobs: gradle: if: ${{ !contains(github.event.head_commit.message, 'ci skip') }} strategy: fail-fast: false matrix: os: [ubuntu-latest, macos-latest, windows-latest] jdk: [8, 11, 17] runs-on: ${{ matrix.os }} env: JDK_VERSION: ${{ matrix.jdk }} GRADLE_OPTS: -Dorg.gradle.daemon=false
  42. name: Pre Merge Checks on: push: branches: - main pull_request:

    branches: - '**' jobs: gradle: if: ${{ !contains(github.event.head_commit.message, 'ci skip') }} strategy: fail-fast: false matrix: os: [ubuntu-latest, macos-latest, windows-latest] jdk: [8, 11, 17] runs-on: ${{ matrix.os }} env: JDK_VERSION: ${{ matrix.jdk }} GRADLE_OPTS: -Dorg.gradle.daemon=false
  43. steps: - name: Checkout Repo uses: actions/checkout@v2 - name: Cache

    Gradle Folders uses: actions/cache@v2 with: path: | ~/.gradle/caches/ ~/.gradle/wrapper/ key: cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}-${{ hashFiles('gradle/libs.versions.toml') restore-keys: | cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}- cache-gradle-${{ matrix.os }}- cache-gradle- - name: Setup Java uses: actions/setup-java@v2 with: java-version: ${{ matrix.jdk }} distribution: 'adopt' - name: Build detekt run: ./gradlew build moveJarForIntegrationTest -x detekt
  44. steps: - name: Checkout Repo uses: actions/checkout@v2 - name: Cache

    Gradle Folders uses: actions/cache@v2 with: path: | ~/.gradle/caches/ ~/.gradle/wrapper/ key: cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}-${{ hashFiles('gradle/libs.versions.toml') restore-keys: | cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}- cache-gradle-${{ matrix.os }}- cache-gradle- - name: Setup Java uses: actions/setup-java@v2 with: java-version: ${{ matrix.jdk }} distribution: 'adopt' - name: Build detekt run: ./gradlew build moveJarForIntegrationTest -x detekt
  45. Matrix (A, B) Step Step Job . . . Step

    Step Job . . . Step Step Job . . . Step Step Job . . . A0, B0 A1, B0 A2, B0 An, Bm . . .
  46. None
  47. None
  48. Custom Topologies

  49. Event Job 2 Job 3 Job 5 Job 6 Job

    7 Job 4 Job 1
  50. jobs: first: steps: # Your steps here second: needs: first

    steps: # Your steps here Step first Step Step second Step Step Sequential Jobs
  51. jobs: first: steps: # Steps for first job second: steps:

    # Steps for second job third: needs: [first, second] steps: # Steps for third job Step first Step Step second Step third Step Step Fan-in to Job
  52. Step first Step Step second Step third Step Step jobs:

    first: steps: # Steps for first job second: needs: first steps: # Steps for second job third: needs: first steps: # Steps for third job Fan-out from Job
  53. None
  54. None
  55. https://github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml acceptance_tests: runs-on: ubuntu-20.04 timeout-minutes: 15 needs: [assemble_apk, espresso_prepare, unit_tests,

    static_analysis] steps: - name: Project Checkout uses: actions/checkout@v2.3.5 - name: Fetch Instrumentation artefacts uses: actions/download-artifact@v2.0.10 - name: Run Espresso tests over emulator.wtf run: ./scripts/emulator-wtf.sh ${{ secrets.EMULATOR_WTF_TOKEN }} - name: Archive execution results if: success() uses: actions/upload-artifact@v2.2.4 with: name: emulator-wtf-results path: emulator-wtf-results
  56. https://github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml acceptance_tests: runs-on: ubuntu-20.04 timeout-minutes: 15 needs: [assemble_apk, espresso_prepare, unit_tests,

    static_analysis] steps: - name: Project Checkout uses: actions/checkout@v2.3.5 - name: Fetch Instrumentation artefacts uses: actions/download-artifact@v2.0.10 - name: Run Espresso tests over emulator.wtf run: ./scripts/emulator-wtf.sh ${{ secrets.EMULATOR_WTF_TOKEN }} - name: Archive execution results if: success() uses: actions/upload-artifact@v2.2.4 with: name: emulator-wtf-results path: emulator-wtf-results
  57. STUDY CASE

  58. https://github.com/dotanuki-labs/norris + 📸 📸 📸

  59. None
  60. <module-dir>/ .gitignore src/androidTest/<package>/ScreenshotTests.kt screenshots/debug your.nice.package.SomeTestName.png your.nice.package.AnotherTestName.png your.nice.package.OneMoreTest.png . . .

  61. <module-dir>/ .gitignore src/androidTest/<package>/ScreenshotTests.kt screenshots/debug your.nice.package.SomeTestName.png your.nice.package.AnotherTestName.png your.nice.package.OneMoreTest.png . . .

  62. <module-dir>/ .gitignore src/androidTest/<package>/ScreenshotTests.kt screenshots/debug your.nice.package.SomeTestName.png your.nice.package.AnotherTestName.png your.nice.package.OneMoreTest.png . . .

  63. Hooking over 3rd party Gradle plugins / tasks

  64. What if I told you … there is another way

    !!!
  65. 🔥 • Have the 2 sets of images available beforehand

    • Define target device at pipeline time • Copy images into convention folder according device spec • Spawn emulator according device spec • Invoke target Gradle task and run tests !!!
  66. None
  67. None
  68. name: Main jobs: # Other jobs screenshot_tests: runs-on: macOS-10.15 needs:

    [acceptance_tests] timeout-minutes: 20 strategy: fail-fast: true matrix: device: ['nexus4', 'pixel'] github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125
  69. name: Main jobs: # Other jobs screenshot_tests: runs-on: macOS-10.15 needs:

    [acceptance_tests] timeout-minutes: 20 strategy: fail-fast: true matrix: device: ['nexus4', 'pixel'] github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125
  70. name: Main jobs: # Other jobs screenshot_tests: runs-on: macOS-10.15 needs:

    [acceptance_tests] timeout-minutes: 20 strategy: fail-fast: true matrix: device: ['nexus4', 'pixel'] github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125
  71. steps: - name: Project Checkout uses: actions/checkout@v2.3.4 - name: Synchronize

    screenshots run: ./scripts/screenshots-sync.sh ${{ matrix.device }} - name: Assign emulator profile id: emulator-profile uses: ./.github/actions/assign-emulator-profile with: device: ${{ matrix.device }} - name: Run Screenshot tests uses: reactivecircus/android-emulator-runner@v2.20.0 with: api-level: 28 target: 'google_apis' profile: ${{ steps.emulator-profile.outputs.assigned }} script: ./gradlew clean executeScreenTests --no-daemon --stacktrace github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125
  72. steps: - name: Project Checkout uses: actions/checkout@v2.3.4 - name: Synchronize

    screenshots run: ./scripts/screenshots-sync.sh ${{ matrix.device }} - name: Assign emulator profile id: emulator-profile uses: ./.github/actions/assign-emulator-profile with: device: ${{ matrix.device }} - name: Run Screenshot tests uses: reactivecircus/android-emulator-runner@v2.20.0 with: api-level: 28 target: 'google_apis' profile: ${{ steps.emulator-profile.outputs.assigned }} script: ./gradlew clean executeScreenTests --no-daemon --stacktrace github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125
  73. steps: - name: Project Checkout uses: actions/checkout@v2.3.4 - name: Synchronize

    screenshots run: ./scripts/screenshots-sync.sh ${{ matrix.device }} - name: Assign emulator profile id: emulator-profile uses: ./.github/actions/assign-emulator-profile with: device: ${{ matrix.device }} - name: Run Screenshot tests uses: reactivecircus/android-emulator-runner@v2.20.0 with: api-level: 28 target: 'google_apis' profile: ${{ steps.emulator-profile.outputs.assigned }} script: ./gradlew clean executeScreenTests --no-daemon --stacktrace github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125 Internal GHA “nexus4” “Nexus 4”
  74. steps: - name: Project Checkout uses: actions/checkout@v2.3.4 - name: Synchronize

    screenshots run: ./scripts/screenshots-sync.sh ${{ matrix.device }} - name: Assign emulator profile id: emulator-profile uses: ./.github/actions/assign-emulator-profile with: device: ${{ matrix.device }} - name: Run Screenshot tests uses: reactivecircus/android-emulator-runner@v2.20.0 with: api-level: 28 target: 'google_apis' profile: ${{ steps.emulator-profile.outputs.assigned }} script: ./gradlew clean executeScreenTests --no-daemon --stacktrace github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125 Internal GHA “nexus4” “Nexus 4”
  75. #! /usr/bin/env bash set -e readonly features=( "search" "facts" )

    readonly device="$1" sync() { echo -e "Syncing screenshots for device profile : $device" for feature in "${features[@]}" do origin="features/$feature/screenshots/$device" destination="features/$feature/screenshots/debug" rm -rf $destination echo -e "✔ Cleaned contents at $destination" cp -R $origin $destination echo -e "✔ Copied images from $origin$ to $destination" done } github.com/dotanuki-labs/norris/blob/master/scripts/screenshots-sync.sh
  76. #! /usr/bin/env bash set -e readonly features=( "search" "facts" )

    readonly device="$1" sync() { echo -e "Syncing screenshots for device profile : $device" for feature in "${features[@]}" do origin="features/$feature/screenshots/$device" destination="features/$feature/screenshots/debug" rm -rf $destination echo -e "✔ Cleaned contents at $destination" cp -R $origin $destination echo -e "✔ Copied images from $origin$ to $destination" done } github.com/dotanuki-labs/norris/blob/master/scripts/screenshots-sync.sh
  77. None
  78. None
  79. ANDROID TIPS++

  80. Emulators on CI

  81. None
  82. None
  83. None
  84. Definining Timeouts

  85. name: Main on: pull_request: push: branches: - master jobs: static_analysis:

    runs-on: ubuntu-20.04 timeout-minutes: 10 steps: # Previous steps - name: Check code formatting run: ./gradlew ktlintCheck detekt —stacktrace https://github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml
  86. name: Main on: pull_request: push: branches: - master jobs: static_analysis:

    runs-on: ubuntu-20.04 timeout-minutes: 10 steps: # Previous steps - name: Check code formatting run: ./gradlew ktlintCheck detekt —stacktrace https://github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml
  87. None
  88. None
  89. Gradle Caching

  90. JOB Save Restore Cloud Storage JOB Save Restore Cloud Storage

    JOB Workflow Run Workflow Run Workflow Run
  91. Hard questions around CI caching • How do we compute

    entry-keys that maximize cache hits? • What should be cached per execution? • Should we segregate cached files per Job? • Etc 🤔
  92. Github Actions Required Configuration Layered Caching Tasks Caching Distribution Caching

    Segregation per Job actions/setup-java (built-in) Easiest (Zero conf) No Yes Yes No actions/cache Hard (All custom) No Opt-in per Configuration Opt-in per Configuration Possible gradle/gradle-build-action Easy (Sensible Defaults) No Yes Yes Yes burrunan/gradle-cache-action Easy (Sensible Defaults) Yes Yes Yes Yes
  93. Food for thought

  94. Task | Condition Merged PR Push Cherry pick Tag Nightly

    Static Analysis ✓ ✓ ✓ Unit tests ✓ ✓ ✓ ✓ Espresso tests ✓ ✓ ✓ ✓ Smoke Tests ✓ ✓ ✓ ✓ Security Analysis ✓ ✓ ✓ ✓ Build ✓ ✓ ✓ ✓ Artifacts generation ✓ ✓ ✓ ✓ Artifact distribution ✓ ✓
  95. Decision Matrix Merged PR Push Cherry pick Tag Nightly Static

    Analysis ✓ ✓ ✓ Unit tests ✓ ✓ ✓ ✓ Espresso tests ✓ 🤔 ✓ ✓ 🤔 Smoke Tests 🤔 🤔 ✓ ✓ 🤔 Security Analysis 🤔 🤔 🤔 ✓ Build ✓ ✓ 🤔 ✓ ✓ Artifacts generation ✓ ✓ ✓ ✓ Artifact distribution ✓ ✓
  96. Code Changes Tests Build Deployable Deploy Release Continuous Integration Continuous

    Delivery Monitoring E2E Tests
  97. Code Changes Tests Build Deployable CD ??? CI

  98. FINAL REMARKS

  99. TL;DR • Open-source model for reusable Actions is quite powerful

    • Pipelines can scale both horizontally and vertically • Awesome option for open-source projects • Unique challenges for big Android projects
  100. UBIRATAN SOARES Brazilian Computer Scientist Senior Software Engineer @ N26

    GDE for Android and Kotlin @ubiratanfsoares ubiratansoares.dev
  101. THANKS