Tweet
Tweet

Slide 1

Slide 1 text

AUTOMATING YOUR ANDROID WORKFLOWS Ubiratan Soares October / 2021 GITHUB ACTIONS

Slide 2

Slide 2 text

CRASHING COURSE !!!

Slide 3

Slide 3 text

Step Step Step Job . . . Workflow Runner Event

Slide 4

Slide 4 text

Job . . . Workflow Push Pull Request Release REST API Tag . . . Job . . . Job . . .

Slide 5

Slide 5 text

Pull Request Build . . . Internal Library • 1 Event • 1 Job triggered

Slide 6

Slide 6 text

Tag Build . . . Internal Library Publish . . . • 1 Event • 1 Job triggered • 1 Job chained

Slide 7

Slide 7 text

/ .github/workflows compliance-checks.yaml ci.yaml etc.yaml

Slide 8

Slide 8 text

What are your thoughts about YAML? Software Engineer

Slide 9

Slide 9 text

Job Runner

Slide 10

Slide 10 text

Step Job Runner Shell Script Command

Slide 11

Slide 11 text

Step Job Runner Shell Script Command Step Composite Action (Bash)

Slide 12

Slide 12 text

Step Job Runner Shell Script Command Step Composite Action (Bash) Step Nested Composite Action

Slide 13

Slide 13 text

Step Job Runner Shell Script Command Step Composite Action (Bash) Step Nested Composite Action Step Code-defined

Slide 14

Slide 14 text

Action Type | Runner Ubuntu MacOS Windows JS YES YES YES Docker YES NO NO Composite/Simple YES YES YES Composite/Nested YES 👀 👀 Step Job Step Runner . . . by

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

name: Android CI on: pull_request: push: branches: - 'master' - '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml

Slide 20

Slide 20 text

name: Android CI on: pull_request: push: branches: - 'master' - '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml

Slide 21

Slide 21 text

name: Android CI on: pull_request: push: branches: - 'master' - '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml

Slide 22

Slide 22 text

name: Android CI on: pull_request: push: branches: - 'master' - '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml

Slide 23

Slide 23 text

name: Android CI on: pull_request: push: branches: - 'master' - '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 signalapp/Signal-Android/blob/master/.github/workflows/android.yml

Slide 24

Slide 24 text

steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml

Slide 25

Slide 25 text

steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml

Slide 26

Slide 26 text

steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml

Slide 27

Slide 27 text

steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml

Slide 28

Slide 28 text

name: Android CI on: pull_request: push: branches: - 'master' - '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' • Multi-triggers • 1 Job • 5 mandatory Steps • 1 conditional Step github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml

Slide 29

Slide 29 text

jobs: build: # Something before steps: - uses: actions/cache@v1

Slide 30

Slide 30 text

name: 'Cache' [. . .] inputs: path: description: 'A list of files, directories, and wildcard patterns to cache and restore' required: true key: description: 'An explicit key for restoring and saving the cache' required: true restore-keys: description: 'An ordered list of keys to use for restoring the cache if no cache hit occurred' required: false outputs: cache-hit: description: 'A boolean value to indicate an exact match was found for the primary key' https://github.com/actions/cache/blob/master/action.yml

Slide 31

Slide 31 text

name: 'Cache' [. . .] inputs: path: description: 'A list of files, directories, and wildcard patterns to cache and restore' required: true key: description: 'An explicit key for restoring and saving the cache' required: true restore-keys: description: 'An ordered list of keys to use for restoring the cache if no cache hit occurred' required: false outputs: cache-hit: description: 'A boolean value to indicate an exact match was found for the primary key' https://github.com/actions/cache/blob/master/action.yml

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

SCALING PIPELINES

Slide 34

Slide 34 text

Step Step Job . . . Workflow Runner Github-hosted Runner vCPUs RAM SSD Ubuntu 2 7 GB 14 GB MacOS 3 14 GB 14 GB Windows 2 7 GB 14 GB by

Slide 35

Slide 35 text

Github-hosted Runner Self-hosted Runner

Slide 36

Slide 36 text

Step Job 1 Step Step Job 2 Step Step Job N Step . . . RUNNER RUNNER RUNNER Step Single Job . . . RUNNER vs Step Step Step Step Step 🤔

Slide 37

Slide 37 text

Build Matrix

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

name: Pre Merge Checks on: push: branches: - main pull_request: branches: - '**' jobs: gradle: if: ${{ !contains(github.event.head_commit.message, 'ci skip') }} strategy: fail-fast: false matrix: os: [ubuntu-latest, macos-latest, windows-latest] jdk: [8, 11, 17] runs-on: ${{ matrix.os }} env: JDK_VERSION: ${{ matrix.jdk }} GRADLE_OPTS: -Dorg.gradle.daemon=false

Slide 42

Slide 42 text

name: Pre Merge Checks on: push: branches: - main pull_request: branches: - '**' jobs: gradle: if: ${{ !contains(github.event.head_commit.message, 'ci skip') }} strategy: fail-fast: false matrix: os: [ubuntu-latest, macos-latest, windows-latest] jdk: [8, 11, 17] runs-on: ${{ matrix.os }} env: JDK_VERSION: ${{ matrix.jdk }} GRADLE_OPTS: -Dorg.gradle.daemon=false

Slide 43

Slide 43 text

steps: - name: Checkout Repo uses: actions/checkout@v2 - name: Cache Gradle Folders uses: actions/cache@v2 with: path: | ~/.gradle/caches/ ~/.gradle/wrapper/ key: cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}-${{ hashFiles('gradle/libs.versions.toml') restore-keys: | cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}- cache-gradle-${{ matrix.os }}- cache-gradle- - name: Setup Java uses: actions/setup-java@v2 with: java-version: ${{ matrix.jdk }} distribution: 'adopt' - name: Build detekt run: ./gradlew build moveJarForIntegrationTest -x detekt

Slide 44

Slide 44 text

steps: - name: Checkout Repo uses: actions/checkout@v2 - name: Cache Gradle Folders uses: actions/cache@v2 with: path: | ~/.gradle/caches/ ~/.gradle/wrapper/ key: cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}-${{ hashFiles('gradle/libs.versions.toml') restore-keys: | cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}- cache-gradle-${{ matrix.os }}- cache-gradle- - name: Setup Java uses: actions/setup-java@v2 with: java-version: ${{ matrix.jdk }} distribution: 'adopt' - name: Build detekt run: ./gradlew build moveJarForIntegrationTest -x detekt

Slide 45

Slide 45 text

Matrix (A, B) Step Step Job . . . Step Step Job . . . Step Step Job . . . Step Step Job . . . A0, B0 A1, B0 A2, B0 An, Bm . . .

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

Custom Topologies

Slide 49

Slide 49 text

Event Job 2 Job 3 Job 5 Job 6 Job 7 Job 4 Job 1

Slide 50

Slide 50 text

jobs: first: steps: # Your steps here second: needs: first steps: # Your steps here Step first Step Step second Step Step Sequential Jobs

Slide 51

Slide 51 text

jobs: first: steps: # Steps for first job second: steps: # Steps for second job third: needs: [first, second] steps: # Steps for third job Step first Step Step second Step third Step Step Fan-in to Job

Slide 52

Slide 52 text

Step first Step Step second Step third Step Step jobs: first: steps: # Steps for first job second: needs: first steps: # Steps for second job third: needs: first steps: # Steps for third job Fan-out from Job

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

https://github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml acceptance_tests: runs-on: ubuntu-20.04 timeout-minutes: 15 needs: [assemble_apk, espresso_prepare, unit_tests, static_analysis] steps: - name: Project Checkout uses: actions/[email protected] - name: Fetch Instrumentation artefacts uses: actions/[email protected] - name: Run Espresso tests over emulator.wtf run: ./scripts/emulator-wtf.sh ${{ secrets.EMULATOR_WTF_TOKEN }} - name: Archive execution results if: success() uses: actions/[email protected] with: name: emulator-wtf-results path: emulator-wtf-results

Slide 56

Slide 56 text

https://github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml acceptance_tests: runs-on: ubuntu-20.04 timeout-minutes: 15 needs: [assemble_apk, espresso_prepare, unit_tests, static_analysis] steps: - name: Project Checkout uses: actions/[email protected] - name: Fetch Instrumentation artefacts uses: actions/[email protected] - name: Run Espresso tests over emulator.wtf run: ./scripts/emulator-wtf.sh ${{ secrets.EMULATOR_WTF_TOKEN }} - name: Archive execution results if: success() uses: actions/[email protected] with: name: emulator-wtf-results path: emulator-wtf-results

Slide 57

Slide 57 text

STUDY CASE

Slide 58

Slide 58 text

https://github.com/dotanuki-labs/norris + 📸 📸 📸

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

/ .gitignore src/androidTest//ScreenshotTests.kt screenshots/debug your.nice.package.SomeTestName.png your.nice.package.AnotherTestName.png your.nice.package.OneMoreTest.png . . .

Slide 61

Slide 61 text

/ .gitignore src/androidTest//ScreenshotTests.kt screenshots/debug your.nice.package.SomeTestName.png your.nice.package.AnotherTestName.png your.nice.package.OneMoreTest.png . . .

Slide 62

Slide 62 text

/ .gitignore src/androidTest//ScreenshotTests.kt screenshots/debug your.nice.package.SomeTestName.png your.nice.package.AnotherTestName.png your.nice.package.OneMoreTest.png . . .

Slide 63

Slide 63 text

Hooking over 3rd party Gradle plugins / tasks

Slide 64

Slide 64 text

What if I told you … there is another way !!!

Slide 65

Slide 65 text

🔥 • Have the 2 sets of images available beforehand • Define target device at pipeline time • Copy images into convention folder according device spec • Spawn emulator according device spec • Invoke target Gradle task and run tests !!!

Slide 66

Slide 66 text

No content

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

name: Main jobs: # Other jobs screenshot_tests: runs-on: macOS-10.15 needs: [acceptance_tests] timeout-minutes: 20 strategy: fail-fast: true matrix: device: ['nexus4', 'pixel'] github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125

Slide 69

Slide 69 text

name: Main jobs: # Other jobs screenshot_tests: runs-on: macOS-10.15 needs: [acceptance_tests] timeout-minutes: 20 strategy: fail-fast: true matrix: device: ['nexus4', 'pixel'] github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125

Slide 70

Slide 70 text

name: Main jobs: # Other jobs screenshot_tests: runs-on: macOS-10.15 needs: [acceptance_tests] timeout-minutes: 20 strategy: fail-fast: true matrix: device: ['nexus4', 'pixel'] github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125

Slide 71

Slide 71 text

steps: - name: Project Checkout uses: actions/[email protected] - name: Synchronize screenshots run: ./scripts/screenshots-sync.sh ${{ matrix.device }} - name: Assign emulator profile id: emulator-profile uses: ./.github/actions/assign-emulator-profile with: device: ${{ matrix.device }} - name: Run Screenshot tests uses: reactivecircus/[email protected] with: api-level: 28 target: 'google_apis' profile: ${{ steps.emulator-profile.outputs.assigned }} script: ./gradlew clean executeScreenTests --no-daemon --stacktrace github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125

Slide 72

Slide 72 text

steps: - name: Project Checkout uses: actions/[email protected] - name: Synchronize screenshots run: ./scripts/screenshots-sync.sh ${{ matrix.device }} - name: Assign emulator profile id: emulator-profile uses: ./.github/actions/assign-emulator-profile with: device: ${{ matrix.device }} - name: Run Screenshot tests uses: reactivecircus/[email protected] with: api-level: 28 target: 'google_apis' profile: ${{ steps.emulator-profile.outputs.assigned }} script: ./gradlew clean executeScreenTests --no-daemon --stacktrace github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125

Slide 73

Slide 73 text

steps: - name: Project Checkout uses: actions/[email protected] - name: Synchronize screenshots run: ./scripts/screenshots-sync.sh ${{ matrix.device }} - name: Assign emulator profile id: emulator-profile uses: ./.github/actions/assign-emulator-profile with: device: ${{ matrix.device }} - name: Run Screenshot tests uses: reactivecircus/[email protected] with: api-level: 28 target: 'google_apis' profile: ${{ steps.emulator-profile.outputs.assigned }} script: ./gradlew clean executeScreenTests --no-daemon --stacktrace github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125 Internal GHA “nexus4” “Nexus 4”

Slide 74

Slide 74 text

steps: - name: Project Checkout uses: actions/[email protected] - name: Synchronize screenshots run: ./scripts/screenshots-sync.sh ${{ matrix.device }} - name: Assign emulator profile id: emulator-profile uses: ./.github/actions/assign-emulator-profile with: device: ${{ matrix.device }} - name: Run Screenshot tests uses: reactivecircus/[email protected] with: api-level: 28 target: 'google_apis' profile: ${{ steps.emulator-profile.outputs.assigned }} script: ./gradlew clean executeScreenTests --no-daemon --stacktrace github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125 Internal GHA “nexus4” “Nexus 4”

Slide 75

Slide 75 text

#! /usr/bin/env bash set -e readonly features=( "search" "facts" ) readonly device="$1" sync() { echo -e "Syncing screenshots for device profile : $device" for feature in "${features[@]}" do origin="features/$feature/screenshots/$device" destination="features/$feature/screenshots/debug" rm -rf $destination echo -e "✔ Cleaned contents at $destination" cp -R $origin $destination echo -e "✔ Copied images from $origin$ to $destination" done } github.com/dotanuki-labs/norris/blob/master/scripts/screenshots-sync.sh

Slide 76

Slide 76 text

#! /usr/bin/env bash set -e readonly features=( "search" "facts" ) readonly device="$1" sync() { echo -e "Syncing screenshots for device profile : $device" for feature in "${features[@]}" do origin="features/$feature/screenshots/$device" destination="features/$feature/screenshots/debug" rm -rf $destination echo -e "✔ Cleaned contents at $destination" cp -R $origin $destination echo -e "✔ Copied images from $origin$ to $destination" done } github.com/dotanuki-labs/norris/blob/master/scripts/screenshots-sync.sh

Slide 77

Slide 77 text

No content

Slide 78

Slide 78 text

No content

Slide 79

Slide 79 text

ANDROID TIPS++

Slide 80

Slide 80 text

Emulators on CI

Slide 81

Slide 81 text

No content

Slide 82

Slide 82 text

No content

Slide 83

Slide 83 text

No content

Slide 84

Slide 84 text

Definining Timeouts

Slide 85

Slide 85 text

name: Main on: pull_request: push: branches: - master jobs: static_analysis: runs-on: ubuntu-20.04 timeout-minutes: 10 steps: # Previous steps - name: Check code formatting run: ./gradlew ktlintCheck detekt —stacktrace https://github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml

Slide 86

Slide 86 text

name: Main on: pull_request: push: branches: - master jobs: static_analysis: runs-on: ubuntu-20.04 timeout-minutes: 10 steps: # Previous steps - name: Check code formatting run: ./gradlew ktlintCheck detekt —stacktrace https://github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml

Slide 87

Slide 87 text

No content

Slide 88

Slide 88 text

No content

Slide 89

Slide 89 text

Gradle Caching

Slide 90

Slide 90 text

JOB Save Restore Cloud Storage JOB Save Restore Cloud Storage JOB Workflow Run Workflow Run Workflow Run

Slide 91

Slide 91 text

Hard questions around CI caching • How do we compute entry-keys that maximize cache hits? • What should be cached per execution? • Should we segregate cached files per Job? • Etc 🤔

Slide 92

Slide 92 text

Github Actions Required Configuration Layered Caching Tasks Caching Distribution Caching Segregation per Job actions/setup-java (built-in) Easiest (Zero conf) No Yes Yes No actions/cache Hard (All custom) No Opt-in per Configuration Opt-in per Configuration Possible gradle/gradle-build-action Easy (Sensible Defaults) No Yes Yes Yes burrunan/gradle-cache-action Easy (Sensible Defaults) Yes Yes Yes Yes

Slide 93

Slide 93 text

Food for thought

Slide 94

Slide 94 text

Task | Condition Merged PR Push Cherry pick Tag Nightly Static Analysis ✓ ✓ ✓ Unit tests ✓ ✓ ✓ ✓ Espresso tests ✓ ✓ ✓ ✓ Smoke Tests ✓ ✓ ✓ ✓ Security Analysis ✓ ✓ ✓ ✓ Build ✓ ✓ ✓ ✓ Artifacts generation ✓ ✓ ✓ ✓ Artifact distribution ✓ ✓

Slide 95

Slide 95 text

Decision Matrix Merged PR Push Cherry pick Tag Nightly Static Analysis ✓ ✓ ✓ Unit tests ✓ ✓ ✓ ✓ Espresso tests ✓ 🤔 ✓ ✓ 🤔 Smoke Tests 🤔 🤔 ✓ ✓ 🤔 Security Analysis 🤔 🤔 🤔 ✓ Build ✓ ✓ 🤔 ✓ ✓ Artifacts generation ✓ ✓ ✓ ✓ Artifact distribution ✓ ✓

Slide 96

Slide 96 text

Code Changes Tests Build Deployable Deploy Release Continuous Integration Continuous Delivery Monitoring E2E Tests

Slide 97

Slide 97 text

Code Changes Tests Build Deployable CD ??? CI

Slide 98

Slide 98 text

FINAL REMARKS

Slide 99

Slide 99 text

TL;DR • Open-source model for reusable Actions is quite powerful • Pipelines can scale both horizontally and vertically • Awesome option for open-source projects • Unique challenges for big Android projects

Slide 100

Slide 100 text

UBIRATAN SOARES Brazilian Computer Scientist Senior Software Engineer @ N26 GDE for Android and Kotlin @ubiratanfsoares ubiratansoares.dev

Slide 101

Slide 101 text

THANKS