and commands which might be helpful in automating AVD run. - Introduce you to problems that you will need to face in order to create such automation on you own. - Present how to solve those problems. - Give you brief idea how you interact with command line from programming language.
execute_shell(cmd, display_cmd=True, display_output=True): if display_cmd: print("- Executing shell command: {}".format(cmd)) with subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, bufsize=1, universal_newlines=True) as p: output = "" for line in p.stdout: if display_output: print(line.strip()) output += line if p.returncode != 0: raise Exception(p.returncode, p.args) return output
execute_shell(cmd, display_cmd=True, display_output=True): if display_cmd: print("- Executing shell command: {}".format(cmd)) with subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, bufsize=1, universal_newlines=True) as p: output = "" for line in p.stdout: if display_output: print(line.strip()) output += line if p.returncode != 0: raise Exception(p.returncode, p.args) return output Blocking
order to avoid potential ADB errors: error: protocol fault (couldn't read status): Connection reset by peer Kill ADB Server Start ADB Server Diagram: Any operation that is in grey box, is using execute_shell function
currently running AVD instances. - Free memory allocated by HAXM to allow starting new AVD instances. Get Visible Devices Diagram: List of (device name, status) Loop Next item starts with “emulator” Kill AVD Instance continue No Yes
of devices attached emulator-5554 offline - Emulator instance init case $ adb devices List of devices attached emulator-5554 device - Emulator system booting case $ adb devices List of devices emulator-5554 unauthorized - Device unauthorized case
-s AVD_NAME_IN_ADB emu kill Where: -s - specific device on which command should be executed AVD_NAME_IN_ADB - serial name of AVD which is visible after in ADB e.g after calling `adb devices` command Native AVD are always named in the same pattern: ‘emulator-` + four digit number which is a port to which AVD was assigned (in example emulator-5556)
no longer forcefully killed. There is a save state process and we need to synchronise with that before we make any other AVD related operations. Diagram: Loop Get Visible Devices “emulator” in list No Yes List of (device name, status) sleep(5) break Timeout start time current time - start time >= timeout Yes No
not only to machine RAM capacity but also to RAM limit set in Intel® Hardware Accelerated Execution Manager (Intel® HAXM). - In the past you had to download HAXM from Intel official website. Now you can do it from Android Studio. RAM limit is set during installation.
not only to machine RAM capacity but also to RAM limit set in Intel® Hardware Accelerated Execution Manager (Intel® HAXM). - In the past you had to download HAXM from Intel official website. Now you can do it from Android Studio. RAM limit is set during installation. - First statement is not really true. If you start all AVD in the same moment RAM limit won’t kick in. e.g. you can start even 10 AVD at the same time with 2GB RAM limit. Don’t do it, your machine will simply restart from OOM.
not only to machine RAM capacity but also to RAM limit set in Intel® Hardware Accelerated Execution Manager (Intel® HAXM). - In the past you had to download HAXM from Intel official website. Now you can do it from Android Studio. RAM limit is set during installation. - First statement is not really true. If you start all AVD in the same moment RAM limit won’t kick in. e.g. you can start even 10 AVD at the same time with 2GB RAM limit. Don’t do it, your machine will simply restart from OOM. - Pick number of AVD according to your machine power. If you have enough RAM but not enough computing power AVD will lag and that will affect your tests.
create avd —-name AVD_NAME —-abi AVD_ABI —-package AVD_PACKAGE Required parameters: AVD_NAME - unique alias under which all settings and properties of AVD will be connected to. Can take any value you like. AVD_ABI - application binary interface e.g. x86, x86_64, google_apis/x86_64 (https://developer.android.com/ndk/guides/abis.html) AVD_PACKAGE - system image to be launched. You can pick Android SDK version to be used and if it should support Google Apis or Google Play Store. Example: • system-images;android-23;google_apis;x86_64
sdkmanager —-list —-verbose Where: --list - lists all available packages --verbose - took me a while before I have found this parameter, it unwraps all package names
v7a System Image system-images;a...-10;default;x86 | 4 | Intel x86 Atom System Image system-images;a...pis;armeabi-v7a | 5 | Google APIs ARM EABI v7a Syste... system-images;a...google_apis;x86 | 5 | Google APIs Intel x86 Atom Sys… $ sdkmanager —-list VS system-images;android-10;default;armeabi-v7a Description: ARM EABI v7a System Image Version: 4 system-images;android-10;default;x86 Description: Intel x86 Atom System Image Version: 4 system-images;android-10;google_apis;armeabi-v7a Description: Google APIs ARM EABI v7a System Image Version: 5 system-images;android-10;google_apis;x86 Description: Google APIs Intel x86 Atom System Image Version: 5 $ sdkmanager —-list —-verbose
any config we like and make sure that script will fetch all dependencies and files that we are lacking. Diagram: List system- image packages Download Package Next item Loop
you answer `yes` then you will be asked 52 low level questions about where to store cache, where to allocate memory during AVD initial boot, networking, CPU properties and so on… $ avdmanager create avd --name "testAVD" --package "system-images;android-26;google_apis;x86" --abi "google_apis/x86" Do you wish to create a custom hardware profile? [no]
| avdmanager create avd --name "testAVD" --package "system-images;android-26;google_apis;x86" --abi "google_apis/x86" But if you are scripting, then you can make it simpler in my opinion. First of all you don’t need to bother with those answers. How to handle AVD config? Let’s try creating AVD: If you answer `yes` then you will be asked 52 low level questions about where to store cache, where to allocate memory during AVD initial boot, networking, CPU properties and so on… $ avdmanager create avd --name "testAVD" --package "system-images;android-26;google_apis;x86" --abi "google_apis/x86" Do you wish to create a custom hardware profile? [no]
AVD and refuse to provide custom hardware config profile: $ echo "no" | avdmanager create avd --name "AVDfromTerminal" --package "system-images;android-23;google_apis;x86" --abi "google_apis/x86" Do you wish to create a custom hardware profile? [no] no
1. Set new environment variable named ANDROID_SDK_HOME and put path to which `~/.android/avd/` file structure will be written. 2. Set new environment variable name ANDROID_AVD_HOME to specify directory only for AVD files. 3. Add parameter --path AVD_PATH to your `avdmanager create avd` command. Default: ~/.android/avd/
hw.camera.back=emulated hw.camera.front=emulated hw.cpu.arch={} hw.cpu.ncore=4 hw.dPad=no hw.device.hash2={} hw.device.manufacturer=Google hw.device.name={} hw.gps=yes hw.gpu.enabled=yes hw.gpu.mode=auto hw.initialOrientation=Portrait hw.keyboard=yes hw.lcd.density={} hw.mainKeys=no hw.ramSize=1536 hw.sdCard=yes hw.sensors.orientation=yes hw.sensors.proximity=yes hw.trackBall=no image.sysdir.1={} runtime.network.latency=none runtime.network.speed=full sdcard.size={} showDeviceFrame=yes skin.dynamic=yes skin.name={} skin.path={} tag.display={} tag.id={} vm.heapSize=256 """ 1. Make string template from it. 2. Every time you create new AVD simply overwrite it’s config.ini. based on AVD_NAME based on AVD_NAME based on AVD_NAME your choice your choice your choice based on AVD_PACKAGE delete it based on AVD_ABI your choice based on AVD_ABI based on AVD_ABI based on AVD_SD_CARD based on AVD_ABI
with ‘emulator-port’ name format. - When you create new AVD you want to be able to interact with it and monitor it’s behaviour. If you know port of AVD you have started then you know how to interact with it. Port is a reference to your AVD in ADB. - Google recommends to start AVD on port range 5554 - 5586 so you want to see if those ports are free.
is available store Check Port Availability No Yes Next item Number of needed ports continue List of available ports len of ‘List of available ports’ > ‘Number of needed ports’ exit() No Yes
not running case on port 5554 $ lsof -i:5554 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME qemu-syst 46043 F1sherKK 45u IPv4 0x78c5e65ae41cd8ff 0t0 TCP localhost:sgi-esphttp (LISTEN) - AVD launched on port 5554 case Seems like no output - but in reality it returns code 1 which means error! Error because it couldn’t find any resource on port 5554.
USER FD TYPE DEVICE SIZE/OFF NODE NAME qemu-syst 46043 F1sherKK 43u IPv4 0x78c5e65ae9a40e17 0t0 TCP localhost:personal-agent (LISTEN) qemu-syst 46043 F1sherKK 44u IPv6 0x78c5e65ae4824a5f 0t0 TCP localhost:personal-agent (LISTEN) qemu-syst 46043 F1sherKK 62u IPv4 0x78c5e65ae5543aef 0t0 TCP localhost:personal-agent-> localhost:57472 (ESTABLISHED) adb 46046 F1sherKK 9u IPv4 0x78c5e65af299f1f7 0t0 TCP localhost:57472-> localhost:personal-agent (ESTABLISHED) - Hint Every time you reserve port for AVD - additional one is taken by ADB. AVD has to be started only on ports with even number! Uneven ports are reserved for ADB interaction with AVD.
AVD_NAME -port AVD_PORT Optional but very helpful parameters: AVD_PORT - port on which AVD will run. Only way to have reference to created AVD from code. Takes 4 digit int value e.g. 5554.
AVD_NAME -port AVD_PORT -no-boot-anim Optional but very helpful parameters: AVD_PORT - port on which AVD will run. Only way to have reference to created AVD from code. Takes 4 digit int value e.g. 5554 -no-boot-anim - disables boot animation during emulator startup for faster booting
AVD_NAME -port AVD_PORT -no-boot-anim -gpu on Optional but very helpful parameters: AVD_PORT - port on which AVD will run. Only way to have reference to created AVD from code. Takes 4 digit int value e.g. 5554 -no-boot-anim - disables boot animation during emulator startup for faster booting -gpu on - makes use of machine GPU to render graphics - speed ups AVD performance
AVD_NAME -port AVD_PORT -no-boot-anim -gpu on -memory AVD_MEMORY Optional but very helpful parameters: AVD_PORT - port on which AVD will run. Only way to have reference to created AVD from code. Takes 4 digit int value e.g. 5554 -no-boot-anim - disables boot animation during emulator startup for faster booting -gpu on - makes use of machine GPU to render graphics - speed ups AVD performance AVD_MEMORY - possibility to extend or decrease memory usage by AVD e.g. -memory 2048
AVD_NAME -port AVD_PORT -no-boot-anim -gpu on -memory AVD_MEMORY -wipe-data -initdata AVD_MEMORY_IMAGE Optional but very helpful parameters: -wipe-data - resets emulator memory -> it resets userdata-qemu.img (current memory state) file to userdata.img (initial memory state) state -initdata - Allows to specify new path to memory image. File under path becomes new userdata.img. But userdata.img is used only on first AVD boot or after using -wipe-data. Therefore -initdata should be always used with -wipe-data. AVD_MEMORY_IMAGE - absolute path to AVD memory image
You can start AVD, change any option you like e.g language, set VPN, disable animations, setup pin or fingerprint (if you need to test that) etc. 2. Turn off AVD and copy userdata-qemu to create reusable AVD memory state.
22914 Problems: 1. Process which starts take over command line until AVD turns off. 2. No output. Add “&” to delegate process to different thread. We want delegate process to different thread because it will end when device is turned off. We need to avoid blocking our script main thread after starting AVD.
emulator:Android emulator version 26.0.3.0 (build_id 3965150) emulator:Found AVD name 'Nexus_5X_API_26' emulator:Found AVD target architecture: x86 emulator:argv[0]: 'emulator'; program directory: '/Users/F1sherKK/Library/Android/sdk/ tools' emulator: Found directory: /Users/F1sherKK/Library/Android/sdk/system-images/ android-26/google_apis/x86/ emulator:Probing for /Users/F1sherKK/Library/Android/sdk/system-images/android-26/ google_apis/x86//kernel-ranchu: file exists emulator:Auto-config: -engine qemu2 (based on configuration) emulator: Found directory: /Users/F1sherKK/Library/Android/sdk/system-images/ android-26/google_apis/x86/ emulator:try dir /Users/F1sherKK/Library/Android/sdk/tools emulator:try dir . (…) Problems: 1. Process which starts take over command line until AVD turns off. 2. No output. 3. Too much output. Command line: Add “-verbose” to receive output from AVD.
[1] 24475 Problems: 1. Process which starts take over command line until AVD turns off. 2. No output. 3. Too much output. Command line: Add “&> AVD_LOG_PATH” to dump AVD output to file.
is called after AVD has finished launching to prevent machine going OOM. Diagram: Loop Yes sleep(5) exit() Timeout per AVD current time - start time >= timeout No List with AVD data Start AVD Get AVD boot status launched Next item No continue Yes Starting AVD sequentially start time
devices 1. Print currently visible devices and their statuses in ADB. <SDK_location>/platform-tools/adb 2. Get specific property from specific device $ adb -s ADB_DEVICE shell getprop DEVICE_PROPERTY
.apk files for test -Inspecting your .apk data -Retrieving instrumentation runner and app package from .apk -Checking if app is installed on device -Installing and uninstalling .apk on/from device Actions:
Gradle tasks: clean - removes whole buildDir GRADLE_BUILD_TASK - task name depends on project, it builds specific type/flavour of android application Build app .apk and test .apk files
Gradle tasks: clean - removes whole buildDir GRADLE_BUILD_TASK - task name depends on project, it builds specific type/flavour of android application Build app .apk and test .apk files If you have specific flavour name, build tasks will be camel cased result of: assemble + BUILD_TYPE_NAME + AndroidTest assembleStaging assembleStagingAndroidTest Example:
be able to pick .apk file by itself then it won’t be able to run without programmers help. Diagram: .apk name part Pick .apk dir where .apk is stored List with paths to app and test .apk
…/app-staging-401.apk …/app-staging-402.apk …/app-staging-403.apk And then inspect .apk files and find pair of two newest ones based on version code. 400002 400003 400001 400002 400003 Names might not contain versioning.
comparison. - Get instrumentation runner package for further test running. - Get app package and test package for checking if it’s currently on device. Diagram: .apk file path Dump Badging .apk package .apk file path Instrumentation runner package List contents of .zip archive Parse output version code Parse output
Asset Packaging Tool Explanation: Command: $ aapt dump badging PATH_TO_APK Where: dump badging - prints the label and icon for the app declared in .apk PATH_TO_APK - path to .apk file, absolute or relative to your current directory
Asset Packaging Tool Explanation: Command: $ aapt l -a PATH_TO_APK Where: l -a - list contents of Zip-compatible archive PATH_TO_APK - path to .apk file, absolute or relative to your current directory
app package is currently on device then we can remove it before next install attempt to avoid: Diagram: Installation error: INSTALL_FAILED_VERSION_DOWNGRADE Get Packages Installed On Device Next item Loop app and test packages is in list of device packages continue Device Packages List No Yes Uninstall Package
<SDK_location>/platform-tools/adb Command: $ adb -s AVD_NAME_IN_ADB shell pm list packages Where: AVD_NAME_IN_ADB - serial name of AVD which is visible after in ADB e.g after calling `adb devices` command shell - interaction with shell module of adb pm - interaction with adb shell module named package manager list packages - listing all installed packages on device
AVD_NAME_IN_ADB shell pm uninstall APP_PACKAGE Where: AVD_NAME_IN_ADB - serial name of AVD which is visible after in ADB e.g after calling `adb devices` command shell - interaction with shell module of adb pm - interaction with adb shell module named package manager APP_PACKAGE - package of Android application or Android application under test
AVD_NAME_IN_ADB install APK_FILE_PATH Where: AVD_NAME_IN_ADB - serial name of AVD which is visible after in ADB e.g after calling `adb devices` command APK_FILE_PATH - path to .apk file that should be installed
apk/staging/app-staging-403.apk Success Takes more time - depending on .apk size even up to 20 seconds. You will need to install two .apk files - app and test. Command line: $ adb -s emulator-5554 install /Users/F1sherKK/MyApp/build/outputs/ apk/staging/app-staging-403.apk | adb -s emulator-5554 install /Users/ F1sherKK/MyApp/build/outputs/apk/androidTest/staging/app-staging-403- androidTest.apk Success Two .apk installs single process.
AVD_NAME_IN_ADB shell am instrument -e package JAVA_PACKAGE TEST_APPLICATION_PACKAGE/INSTRUMENTATION_RUNNER_PACKAGE Where: AVD_NAME_IN_ADB - serial name of AVD which is visible after in ADB e.g after calling `adb devices` command shell - interaction with shell module of adb am - interaction with adb shell module named activity manager instrument - interaction with adb shell am module responsible for instrumentation process in which tests run Running tests by java package will allow you to test parts of your app rather than whole flavour. If you have long tests it will allow you to avoid memory problems.
code annotated with @Test are stored package com.example.labs.azimo.note.tests.function.login; import … public class Login_Function_Tests { @Rule public ActivityTestRule<LoginActivity> activityRule = new ActivityTestRule<>(LoginActivity.class, true, false); @Before public void before() {…} @After public void after() {…} @Test public void testLogin_insertWrongEmail_shouldDisplayError() {…} @Test public void testLogin_whenEmailFieldIsEmpty_shouldDisplayError() {…} @Test public void testLogin_whenPasswordFieldIsEmpty_shouldDisplayError() {…} @Test public void testLogin_whenSwitchingFromLoginToRegister_shouldKeepData() {…} all login tests tests below belongs to this package
AVD_NAME_IN_ADB shell am instrument -e package JAVA_PACKAGE TEST_APPLICATION_PACKAGE/INSTRUMENTATION_RUNNER_PACKAGE Where: TEST_APPLICATION_PACKAGE - package of your application with ‘.test’ added at the end INSTRUMENTATION_RUNNER_PACKAGE - java package to Instrumentation Runner class which should be used during test session. Running tests by java package will allow you to test parts of your app rather than whole flavour. If you have long tests it will allow you to avoid memory problems. Remember that both values can be automatically scrapped with usage of aapt binary. It will decrease complexity of your input config to script.
package com.example.labs.azimo.note.tests.function.login com.example.labs.azimo.note.test/ com.example.labs.azimo.note.config.AzimoTestRunner Running Tests Problems: 1. Script cannot tell when tests has finished. 2. No info about which tests have passed and which have failed.
AVD_NAME_IN_ADB shell am instrument -w -r -e package JAVA_PACKAGE TEST_APPLICATION_PACKAGE/INSTRUMENTATION_RUNNER_PACKAGE Where: -w - instead of starting process in different thread, current thread will be blocked and you will receive basic output -r - turns basic output into raw output so there is more information about tests and their status Running tests by java package will allow you to test parts of your app rather than whole flavour. If you have long tests it will allow you to avoid memory problems.
-r -e package com.example.labs.azimo.note.tests.function.login com.example.labs.azimo.note.test/ com.example.labs.azimo.note.config.AzimoTestRunner Running Tests Test Case Output Test Case Output Test Case Output … Session Result Code + Time
numtests=4 INSTRUMENTATION_STATUS: stream= INSTRUMENTATION_STATUS: id=AndroidJUnitRunner INSTRUMENTATION_STATUS: test=testLogin_whenPasswordFieldIsEmpty_shouldDisplayError INSTRUMENTATION_STATUS: class=com.example.labs.azimo.note.tests.function.login.Login_Function_Tests INSTRUMENTATION_STATUS: current=1 INSTRUMENTATION_STATUS_CODE: 1 INSTRUMENTATION_STATUS: numtests=4 INSTRUMENTATION_STATUS: stream=. INSTRUMENTATION_STATUS: id=AndroidJUnitRunner INSTRUMENTATION_STATUS: test=testLogin_whenPasswordFieldIsEmpty_shouldDisplayError INSTRUMENTATION_STATUS: class=com.example.labs.azimo.note.tests.function.login.Login_Function_Tests INSTRUMENTATION_STATUS: current=1 INSTRUMENTATION_STATUS_CODE: 0 Index of current test (can be used to determine how much tests are left): INSTRUMENTATION_STATUS: current=1 Test case output
numtests=4 INSTRUMENTATION_STATUS: stream= INSTRUMENTATION_STATUS: id=AndroidJUnitRunner INSTRUMENTATION_STATUS: test=testLogin_whenPasswordFieldIsEmpty_shouldDisplayError INSTRUMENTATION_STATUS: class=com.example.labs.azimo.note.tests.function.login.Login_Function_Tests INSTRUMENTATION_STATUS: current=1 INSTRUMENTATION_STATUS_CODE: 1 INSTRUMENTATION_STATUS: numtests=4 INSTRUMENTATION_STATUS: stream=. INSTRUMENTATION_STATUS: id=AndroidJUnitRunner INSTRUMENTATION_STATUS: test=testLogin_whenPasswordFieldIsEmpty_shouldDisplayError INSTRUMENTATION_STATUS: class=com.example.labs.azimo.note.tests.function.login.Login_Function_Tests INSTRUMENTATION_STATUS: current=1 INSTRUMENTATION_STATUS_CODE: 0 STATUS CODE with value 1 means that test has started: INSTRUMENTATION_STATUS_CODE: 1 Test case output
numtests=4 INSTRUMENTATION_STATUS: stream= INSTRUMENTATION_STATUS: id=AndroidJUnitRunner INSTRUMENTATION_STATUS: test=testLogin_whenPasswordFieldIsEmpty_shouldDisplayError INSTRUMENTATION_STATUS: class=com.example.labs.azimo.note.tests.function.login.Login_Function_Tests INSTRUMENTATION_STATUS: current=1 INSTRUMENTATION_STATUS_CODE: 1 INSTRUMENTATION_STATUS: numtests=4 INSTRUMENTATION_STATUS: stream=. INSTRUMENTATION_STATUS: id=AndroidJUnitRunner INSTRUMENTATION_STATUS: test=testLogin_whenPasswordFieldIsEmpty_shouldDisplayError INSTRUMENTATION_STATUS: class=com.example.labs.azimo.note.tests.function.login.Login_Function_Tests INSTRUMENTATION_STATUS: current=1 INSTRUMENTATION_STATUS_CODE: 0 STATUS CODE with value 0 means that test has PASSED: INSTRUMENTATION_STATUS_CODE: 0 Test case output
numtests=4 INSTRUMENTATION_STATUS: stream= com.example.labs.azimo.note.tests.function.login.Login_Function_Tests: INSTRUMENTATION_STATUS: id=AndroidJUnitRunner INSTRUMENTATION_STATUS: test=testLogin_whenPasswordFieldIsEmpty_shouldDisplayError INSTRUMENTATION_STATUS: class=com.example.labs.azimo.note.tests.function.login.Login_Function_Tests INSTRUMENTATION_STATUS: current=1 INSTRUMENTATION_STATUS_CODE: 1 INSTRUMENTATION_STATUS: numtests=4 INSTRUMENTATION_STATUS: stream= Error in testLogin_whenPasswordFieldIsEmpty_shouldDisplayError(com.example.labs.azimo.note.tests.function.login.Login_Function_Tests): org.mockito.exceptions.misusing.InvalidUseOfMatchersException: No matchers found for additional matcher Not(?) -> at com.example.labs.azimo.note.tests.function.login.Login_Function_Tests.testLogin_whenPasswordFieldIsEmpty_shouldDisplayError(Login_Function_Tests.java:96) at com.example.labs.azimo.note.tests.function.login.Login_Function_Tests.testLogin_whenPasswordFieldIsEmpty_shouldDisplayError(Login_Function_Tests.java:96) at java.lang.reflect.Method.invoke(Native Method) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at android.support.test.internal.runner.junit4.statement.RunBefores.evaluate(RunBefores.java:80) at android.support.test.internal.runner.junit4.statement.RunAfters.evaluate(RunAfters.java:61) at android.support.test.rule.ActivityTestRule$ActivityStatement.evaluate(ActivityTestRule.java:433) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at org.junit.runner.JUnitCore.run(JUnitCore.java:115) at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:58) at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:369) at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2074) INSTRUMENTATION_STATUS: id=AndroidJUnitRunner INSTRUMENTATION_STATUS: test=testLogin_whenPasswordFieldIsEmpty_shouldDisplayError INSTRUMENTATION_STATUS: class=com.example.labs.azimo.note.tests.function.login.Login_Function_Tests INSTRUMENTATION_STATUS: stack= org.mockito.exceptions.misusing.InvalidUseOfMatchersException: No matchers found for additional matcher Not(?) -> at com.example.labs.azimo.note.tests.function.login.Login_Function_Tests.testLogin_whenPasswordFieldIsEmpty_shouldDisplayError(Login_Function_Tests.java:96) at com.example.labs.azimo.note.tests.function.login.Login_Function_Tests.testLogin_whenPasswordFieldIsEmpty_shouldDisplayError(Login_Function_Tests.java:96) at java.lang.reflect.Method.invoke(Native Method) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at android.support.test.internal.runner.junit4.statement.RunBefores.evaluate(RunBefores.java:80) at android.support.test.internal.runner.junit4.statement.RunAfters.evaluate(RunAfters.java:61) at android.support.test.rule.ActivityTestRule$ActivityStatement.evaluate(ActivityTestRule.java:433) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at org.junit.runner.JUnitCore.run(JUnitCore.java:115) at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:58) at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:369) at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2074) INSTRUMENTATION_STATUS: current=1 INSTRUMENTATION_STATUS_CODE: -2 Full error stack trace Full error stack trace Test case output
numtests=4 INSTRUMENTATION_STATUS: stream= com.example.labs.azimo.note.tests.function.login.Login_Function_Tests: INSTRUMENTATION_STATUS: id=AndroidJUnitRunner INSTRUMENTATION_STATUS: test=testLogin_whenPasswordFieldIsEmpty_shouldDisplayError INSTRUMENTATION_STATUS: class=com.example.labs.azimo.note.tests.function.login.Login_Function_Tests INSTRUMENTATION_STATUS: current=1 INSTRUMENTATION_STATUS_CODE: 1 INSTRUMENTATION_STATUS: numtests=4 INSTRUMENTATION_STATUS: stream= Error in testLogin_whenPasswordFieldIsEmpty_shouldDisplayError(com.example.labs.azimo.note.tests.function.login.Login_Function_Tests): org.mockito.exceptions.misusing.InvalidUseOfMatchersException: No matchers found for additional matcher Not(?) -> at com.example.labs.azimo.note.tests.function.login.Login_Function_Tests.testLogin_whenPasswordFieldIsEmpty_shouldDisplayError(Login_Function_Tests.java:96) at com.example.labs.azimo.note.tests.function.login.Login_Function_Tests.testLogin_whenPasswordFieldIsEmpty_shouldDisplayError(Login_Function_Tests.java:96) at java.lang.reflect.Method.invoke(Native Method) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at android.support.test.internal.runner.junit4.statement.RunBefores.evaluate(RunBefores.java:80) at android.support.test.internal.runner.junit4.statement.RunAfters.evaluate(RunAfters.java:61) at android.support.test.rule.ActivityTestRule$ActivityStatement.evaluate(ActivityTestRule.java:433) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at org.junit.runner.JUnitCore.run(JUnitCore.java:115) at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:58) at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:369) at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2074) INSTRUMENTATION_STATUS: id=AndroidJUnitRunner INSTRUMENTATION_STATUS: test=testLogin_whenPasswordFieldIsEmpty_shouldDisplayError INSTRUMENTATION_STATUS: class=com.example.labs.azimo.note.tests.function.login.Login_Function_Tests INSTRUMENTATION_STATUS: stack= org.mockito.exceptions.misusing.InvalidUseOfMatchersException: No matchers found for additional matcher Not(?) -> at com.example.labs.azimo.note.tests.function.login.Login_Function_Tests.testLogin_whenPasswordFieldIsEmpty_shouldDisplayError(Login_Function_Tests.java:96) at com.example.labs.azimo.note.tests.function.login.Login_Function_Tests.testLogin_whenPasswordFieldIsEmpty_shouldDisplayError(Login_Function_Tests.java:96) at java.lang.reflect.Method.invoke(Native Method) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at android.support.test.internal.runner.junit4.statement.RunBefores.evaluate(RunBefores.java:80) at android.support.test.internal.runner.junit4.statement.RunAfters.evaluate(RunAfters.java:61) at android.support.test.rule.ActivityTestRule$ActivityStatement.evaluate(ActivityTestRule.java:433) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at org.junit.runner.JUnitCore.run(JUnitCore.java:115) at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:58) at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:369) at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2074) INSTRUMENTATION_STATUS: current=1 INSTRUMENTATION_STATUS_CODE: -2 status code -2 means test has failed Test case output
devices No n-th Device Thread Run Test Package Test Results Loop Next Line Parse Line n-th Logging Thread Write to .json Yes Thread References List Start Test Thread for next package Next item Check Test Threads Statues Loop is there idle thread with unsaved logs No Yes Start Test Logging Thread join Packages are distributed between all devices. Devices List Package Package
All_Login_Tests com.example.labs.azimo.note.tests.function.login Test Case - 10 seconds Test Case - 10 seconds Test Case - 10 seconds com.example.labs.azimo.note.tests.endtoend.login Test Case - 1 minute Test Case - 2 minute Test Case - 1 minute Test Case - 2 minute Test Case - 1 minute Test Case - 2 minute Device 1 - received package function.login Device 2 - received package endtoend.login Execution time took 30 seconds. Execution time took 9 minutes. Device 1 had to wait for Device 2 - 8 minutes 30 seconds while it could take half of tests.
Test Case - 2 minute Test Case - 1 minute Test Case - 2 minute Test Case - 1 minute Test Case - 2 minute No shards: Shards Number - 3 What do you expect to happen: Test Case - 1 minute Test Case - 2 minute Device 1: endtoend.login shard 1 of 3 Test Case - 1 minute Test Case - 2 minute Device 2: endtoend.login shard 2 of 3 Test Case - 1 minute Test Case - 2 minute Device 3: endtoend.login shard 3 of 3 Execution time 3 min. Execution time 3 min. Execution time 3 min.
in different variation): Device 1: endtoend.login shard 1 of 3 Test Case - 1 minute Test Case - 2 minute Device 2: endtoend.login shard 2 of 3 Test Case - 1 minute Test Case - 1 minute Test Case - 2 minute Device 3: endtoend.login shard 3 of 3 Test Case - 2 minute Execution time 0 min. Execution time 3 min. Execution time 6 min. 6 min wasted. 3 min wasted. Google - Test Sharding com.example.labs.azimo.note.tests.endtoend.login Test Case - 1 minute Test Case - 2 minute Test Case - 1 minute Test Case - 2 minute Test Case - 1 minute Test Case - 2 minute No shards:
adb -s AVD_NAME_IN_ADB shell am instrument -w -r -e package JAVA_PACKAGE -e numShards NUM_SHARDS -e shardIndex SHARD_INDEX TEST_APPLICATION_PACKAGE/INSTRUMENTATION_RUNNER_PACKAGE Where: NUM_SHARDS - number of parts to which inserted package should be divided SHARD_INDEX - index of package part
No n-th Device Thread Run Test Package Test Results Loop Next Line Parse Line n-th Logging Thread Write to .json Yes Thread References List Start Test Thread for next package Next item Check Test Threads Statues Loop is there idle thread with unsaved logs No Yes Start Test Logging Thread join With this structure the more shards you use the less idle devices you have. Each device receives new launch command instantly after process created with previous one has ended. Devices List Package Launch Cmd … Package Launch Cmd Package Launch Cmd
tests with ability to: • maintaining AVD instances according to provided .json or working with currently connected devices • building and installing .apk files • specifying test sets according to your liking • speeding up test run (when using packages) by making each device independent • generating HTML dashboard with output, device LogCats and more https://github.com/AzimoLabs/AzimoNote Example project: Tool:
f1sherkk
0nihajim
safie_recruit
shirayanagiryuji
tsukuboshi
.jpg){kind=avatar}
izzii
sansan_randd
biatunky
databricksjapan
.jpg){kind=avatar}
mdalmijn
azukiazusa1
tagomoris
tiina
aarron
rasmusluckow
addyosmani
morganepeng
smashingmag
sstephenson
soudai
matthewcrist
mclloyd
swwweet
thekraken
chrisshort
![[email protected] @F1sherKK AzimoLabs FisherKK/F1sherKK-MyRoadToAI Kamil Krzyk Android Platform Engineer at](https://files.speakerdeck.com/presentations/1d8190b2d4e848d7bc9dd70be9dfa554/slide_1.jpg){kind=link}
![Starting AVD Command line: $ emulator -avd Nexus_5X_API_26 & [1]](https://files.speakerdeck.com/presentations/1d8190b2d4e848d7bc9dd70be9dfa554/slide_58.jpg){kind=link}
![Starting AVD $ emulator -avd Nexus_5X_API_26 -verbose & [1] 22914](https://files.speakerdeck.com/presentations/1d8190b2d4e848d7bc9dd70be9dfa554/slide_59.jpg){kind=link}
