Tweet
Tweet

Slide 1

Slide 1 text

EMULATORS IN ACTION HOW TO RUN UI TESTS ON YOUR CI droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 2

Slide 2 text

ABOUT ME 2 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 3

Slide 3 text

ABOUT ME ▸ Daniel Hartwich ▸ Android Developer at ▸ Twitter: KiLLyA_ ▸ GitHub: dhartwich1991 3 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 4

Slide 4 text

LOOKING FOR NEW COLLEAGUES ▸ 1 (Senior) Android Developer - Platform Team ▸ 1 Automation Android Developer - Mobile Releases Team 4 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 5

Slide 5 text

WHAT IS IN THIS? ▸ UI Tests in Android ▸ Spoon ! ▸ Working setup at ▸ UI Tests on CI (Jenkins) ▸ Fastlane ▸ The solution (TM) 5 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 6

Slide 6 text

UI TESTS IN ANDROID 6 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 7

Slide 7 text

▸ UI Tests ▸ Espresso 7 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 8

Slide 8 text

@Test fun changesTextWhenClickingButton() { onView(withId(R.id.press_me_button)) .perform(click()) onView(withId(R.id.change_text_text)).check(matches(withText("First"))) onView(withId(R.id.press_me_button)) .perform(click()) onView(withId(R.id.change_text_text)).check(matches(withText("Second"))) } ▸ Simple view assertion ▸ When I click one button should show different text 8 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 9

Slide 9 text

9 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 10

Slide 10 text

✅ 10 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 11

Slide 11 text

UI TESTS ▸ (usually) quick to execute ▸ test UI of specific screen ▸ Run them with ./gradlew cAT ▸ Is this enough? ▸ Yes...but! 11 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 12

Slide 12 text

SPOON ! 12 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 13

Slide 13 text

SPOON ▸ Distribute tests to different devices ▸ possibility to specify those devices ▸ Run on different versions of Android ▸ Take screenshots during critical parts of your tests ▸ Save important files like DBs ▸ Test sharding! 13 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 14

Slide 14 text

SOUNDS GOOD, SOUNDS FUN, BUT HOW? ▸ Gradle Spoon plugin (recommended) ▸ classpath 'com.stanfy.spoon:spoon-gradle- plugin:1.2.2' ▸ ⚠ some problems with Android Studio 3.0 ▸ Guys are working on it. There is a snapshot (2.0) available at the moment ▸ spoonDebugAndroidTest 14 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 15

Slide 15 text

HOW WE WORK AT 15 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 16

Slide 16 text

▸ ~25 developers ▸ Split in independent feature teams ▸ Release Trains ! ▸ Code Freeze every 2 weeks ▸ Followed by a Release + Rollout (20% -> 50% -> 100%) ▸ Work on feature branches, merge to master ▸ Current # of open PRs: ~40 16 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 17

Slide 17 text

▸ We use Jenkins CI ▸ Running Unit Tests, Static Analyzers, assemble different build types etc. ▸ On Every PR ▸ Run UI Tests on CI 17 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 18

Slide 18 text

▸ Huge load on Jenkins ▸ Waiting for long running UI tests ▸ Jenkins without UI (only raw metal - Linux machines) ▸ Tests were failing / slow / flaky ▸ shell scripts magicians? ! 18 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 19

Slide 19 text

INTEGRATIONWITHSPOON.SH #!/usr/bin/env bash # Configure pre-conditions PACKAGE_NAME="com.xing.android" AVD_NAME="integration-tests" PORT=${1-6000} ############################################################################## # Ensure the Test APK is built already. TEST_APK_FILE="core-app/build/outputs/apk/core-app-debug-androidTest.apk" if [ ! -f "${TEST_APK_FILE}" ] then echo "Test APK doesn't exist, aborting. Make sure you run ./gradlew :core-app:assembleDebug :core-app:assembleDebugAndroidTest" exit else echo "androidTest APK Exists, continuing" fi # Calculate the Serial Number of the emulator instance SERIAL=emulator-${PORT} echo "Creating (forceful) AVD with name ${AVD_NAME}" # We have to echo "no" because it will ask us if we want to use a custom hardware profile, and we don't. echo "no" | android create avd \ -n "${AVD_NAME}" \ -k "system-images;android-22;default;x86_64" \ -f echo "AVD ${AVD_NAME} created." # Start the Android Emulator # "2>&1" combines stderr and stdout into the stdout stream START_EMULATOR="/opt/android-sdk-linux/tools/emulator \ -avd ${AVD_NAME} \ -netspeed full \ -netdelay none \ -no-skin \ -no-window \ -gpu guest \ -port ${PORT}" echo $START_EMULATOR $START_EMULATOR 2>&1 & # Ensure Android Emulator has booted successfully before continuing EMU_BOOTED='unknown' MAX_RETRY_COUNT=27 while [[ ${EMU_BOOTED} != *"stopped"* ]]; do sleep 7 EMU_BOOTED=`adb -s ${SERIAL} shell getprop init.svc.bootanim || echo unknown` # Exit if the emulator didin't start in 140 seconds. MAX_RETRY_COUNT=$(($MAX_RETRY_COUNT - 1)) if [[ $MAX_RETRY_COUNT -eq 0 ]]; then echo "Emulator startup timeout. Aborting" exit 1 fi done duration=$(( SECONDS - start )) echo "Android Emulator started after $duration seconds." # Use the Spoon utility as a test runner SPOON_COMMAND="./gradlew --no-daemon spoonDebugAndroidTest -PspoonDevice=emulator-${PORT}" echo "Running: ${SPOON_COMMAND}" ${SPOON_COMMAND} 19 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 20

Slide 20 text

KILL-EMULATOR.SH #!/usr/bin/env bash ############################################################## # # KILL-EMULATOR # # Kills an Android emulator which requires authentication. # It works by opening a telnet session and authenticates, before issuing the # kill command. # # Usage: `kill-emulator.sh ` # where is optional (defaults to 6000) # # Since SDK Tools 25.1.6, the Android emulator has required authentication # before any commands can be run on it. This breaks commands such as # `adb emu kill`. # # References: # - https://developer.android.com/studio/run/emulator-commandline.html#console-session # - https://code.google.com/p/android/issues/detail?id=21021# # ############################################################## # Read port form the console PORT=${1-6000} # Read token for emulator TOKEN=$(<$HOME/.emulator_console_auth_token) # Notify user that everything is going to be OK echo "Killing emulator on port $PORT with auth token $TOKEN" # Start telnet and pray that it will work TELNET=`( echo "auth $TOKEN"; sleep 1; echo "kill"; sleep 1 ) | telnet localhost $PORT | grep "OK: killing emulator, bye bye"` if [ "$?" -ne 0 ]; then echo "Couldn't kill emulator $PORT. Aborting" exit 1 else echo "Emulator dead" exit 0 fi 20 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 21

Slide 21 text

"# Start telnet and pray that it will work" - kill-emulator.sh - 21 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 22

Slide 22 text

! 22 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 23

Slide 23 text

Why? - Unmaintainable - What if you die? - What if one of the scripts fails - Too many cases you can't handle - Flaky / Slow - People will not trust in tests - And bother you a lot... 23 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 24

Slide 24 text

PROBLEM: HOW DO YOU CREATE/MANAGE EMULATORS ETC. ON JENKINS? 24 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 25

Slide 25 text

FASTLANE (TO THE RESCUE) 25 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 26

Slide 26 text

FASTLANE ▸ ruby tool to handle tedious tasks ▸ mainly focussed on releasing applications ▸ super cool ▸ has lots of plugins ▸ huge community ▸ "over 10,391,703 Developer Hours Saved" 26 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 27

Slide 27 text

▸ sudo gem install fastlane -NV ▸ fastlane init inside your existing project ▸ Ready to go! ▸ Create your 'lanes' (definitions of tasks) inside Fastfile 27 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 28

Slide 28 text

Espresso lane desc "Run UI tests using default test runner" lane :espresso_test do gradle(task: "cAT") end Run command: fastlane espresso_test spoon lane desc "Run UI tests using spoon" lane :espresso_spoon_test do gradle(task: "spoonDebugAndroidTest") end Run command: fastlane espresso_spoon_test 28 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 29

Slide 29 text

▸ How does this help us? ▸ it doesn't ▸ we still have the same problem with emulators ▸ what to do? ▸ Plugin magic 29 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 30

Slide 30 text

FASTLANE-PLUGIN-AUTOMATED-TEST- EMULATOR-RUN 30 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 31

Slide 31 text

▸ Wraps gradle/shell tasks ▸ Creates and manages emulators ▸ easy to configure ▸ start multiple emulators ▸ fastlane add_plugin automated_test_emulator_run ▸ create AVD(emulator) config using JSON 31 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 32

Slide 32 text

{ "avd_list": [ { "avd_name": "Test-Emulator-API23-Nexus-5-1", "create_avd_package": "system-images;android-23;google_apis;x86_64", "create_avd_device": "Nexus 5X", "create_avd_tag": "google_apis", "create_avd_abi": "x86_64", "create_avd_additional_options": "", "create_avd_hardware_config_filepath": "~/Android/AVD_Snapshots/Nexus_5X_API_23/Test-Emulator-API23-Nexus-5-1.ini", "launch_avd_port": "", "launch_avd_snapshot_filepath": "~/Android/AVD_Snapshots/Nexus_5X_API_23/Nexus_5X_API_23_SNAPSHOT.img", "launch_avd_launch_binary_name": "emulator", "launch_avd_additional_options": "-gpu on" }, { "avd_name": "Test-Emulator-API23-Nexus-5-2", "create_avd_package": "system-images;android-26;google_apis;x86_64", "create_avd_device": "Nexus 5X", "create_avd_tag": "google_apis", "create_avd_abi": "x86_64", "create_avd_additional_options": "", "create_avd_hardware_config_filepath": "~/Android/AVD_Snapshots/Nexus_5X_API_26/Test-Emulator-API26-Nexus-5-2.ini", "launch_avd_port": "", "launch_avd_snapshot_filepath": "~/Android/AVD_Snapshots/Nexus_5X_API_23/Nexus_5X_API_26_SNAPSHOT.img", "launch_avd_launch_binary_name": "emulator", "launch_avd_additional_options": "-gpu on" } ] } 32 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 33

Slide 33 text

▸ You can configure everything from here you would normally need to do from command line ▸ It is easy to read ▸ Other people can maintain/tweak it ▸ It scales (why not use 3,4 or 5 emulators?) 33 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 34

Slide 34 text

▸ Now we can create lanes to use this plugin ▸ desc "Runs tests with AVD setup according to JSON file config with usage of spoon." lane :Automation_EmulatorRun_Spoon do automated_test_emulator_run( AVD_setup_path: "fastlane/avdconfig/AVD_setup.json", gradle_task:"spoonDebugAndroidTest" ) end ▸ Now let's run it. ▸ fastlane Automation_EmulatorRun_Spoon 34 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 35

Slide 35 text

▸ Yayyyy! ▸ This handles emulator creation ▸ We can run multiple emulators with different versions ▸ easy to maintain ▸ are we done? 35 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 36

Slide 36 text

36 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 37

Slide 37 text

▸ We are still ~25 developers ▸ We have 4 Jenkins slaves ▸ 1 slave = 1 computer ▸ 8 nodes per slave (same computer) ▸ multiple ui test jobs can run on the same jenkins slave 37 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 38

Slide 38 text

▸ spoon tests are executed on all connected devices ▸ Tests on the same node interfere with each other ▸ Tests get flaky again. ▸ Tests are still slow 38 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 39

Slide 39 text

REMEMBER!!! - YOU CAN SPECIFY TARGET DEVICES ON SPOON 39 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 40

Slide 40 text

▸ We don't know what are the names of the emulators that get created by the plugin ▸ Something with emulator-${portNumber} ▸ How do we connect our spoon test run with the created emulators? 40 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 41

Slide 41 text

Spoon configuration (build.gradle) spoon { debug = true shard = true devices = ['emulator-5556', 'emulator-5558'] } ▸ You can specify target devices using the devices array 41 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 42

Slide 42 text

MHHHH ! 42 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 43

Slide 43 text

▸ This is it! ▸ We need a way to set devices from our emulator plugin ▸ It knows which ports it assigned ▸ And thus can pass the names to the gradle task it executes 43 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 44

Slide 44 text

▸ The current plugin does not support this. ▸ But it's open source ▸ So we can tweak it! 44 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 45

Slide 45 text

# Launching tests shell_task = "#{params[:shell_task]}" unless params[:shell_task].nil? gradle_task = "#{params[:gradle_task]}" unless params[:gradle_task].nil? spoon_task = "#{params[:spoon_task]}" unless params[:spoon_task].nil? ▸ Created spoon_task 45 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 46

Slide 46 text

TWEAKING THE PLUGIN ports = Array.new spoon_devices = " -PspoonDevice=" spoon_devices = spoon_devices + "emulator-" + avd_schemes[0].launch_avd_port.to_s for i in 1...avd_schemes.length ports << avd_schemes[i].launch_avd_port spoon_devices = spoon_devices + ",emulator-" + avd_schemes[i].launch_avd_port.to_s end gradle_spoon_task = params[:spoon_task] gradle_spoon_task = gradle_spoon_task + spoon_devices 46 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 47

Slide 47 text

▸ Create a new lane inside your Fastfile ▸ use spoon_task instead of gradle_task ▸ -pSpoonDevice will be passed to the task that is executed ▸ We need to read this value in build.gradle ▸ And configure our spoon to run on the passed emulators 47 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 48

Slide 48 text

./gradlew spoonDebugAndroidTest - pSpoonDevice=emulator-5556, emulator-5558 spoon { if (project.hasProperty('spoonDevice')) { devices = [] project.properties['spoonDevice'].split(',').each { devices += [it] } } ▸ This reads -pSpoonDevice flag ▸ and propagates devices[] with emulator names 48 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 49

Slide 49 text

▸ Now the tests will execute only on the devices created for the fastlane task we execute ▸ Tasks are now independent from each other ▸ Less failure due to interference ▸ Test runs get way faster ▸ Developers like if they don't need to retry to run the tests multiple times 49 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 50

Slide 50 text

✅ MISSION ACCOMPLISHED ✅ ▸ ? 50 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 51

Slide 51 text

REMAINING PROBLEMS ▸ Sometimes can still be flaky (ADB issues) ▸ Emulators still don't get shut down properly ▸ Too much load on jenkins ▸ No retrying of flaky tests ▸ Still not perfect, but improving the current situation 51 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 52

Slide 52 text

THANK YOU ❤ 52 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI

Slide 53

Slide 53 text

QUESTIONS? 53 — droidConPL Krakow 2017 - Emulators in Action - How to run UI tests on your CI