Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Where and how to run UI tests (Droidcon London, 2021)

Where and how to run UI tests (Droidcon London, 2021)

UI tests have never been easy.
Yes, with frameworks like Kaspresso and Kakao it gets easier year by year.

However, running ui tests still raises a lot of questions, like:
- Which test runner do I need to use?
- How to deal with unstable network?
- Emulator or Real Device?
- How to clear the state and why?
- How to make tests less flakiness?
- How to organise good reports?

We gonna talk about all of these topics and deep dive into test runners, emulators, why adb is not the best option and find the most pragmatic options for your team no matter how big is it

E20836b1ea89956e0b96362046e2b564?s=128

Alexey Bykov

April 26, 2022
Tweet

More Decks by Alexey Bykov

Other Decks in Programming

Transcript

  1. 1 Where and how to run UI tests Alexey Bykov

  2. 2 Alexey Bykov Android Software Engineer @Revolut • 5 years

    in Android • EX Programm commette: AppsConf, Podlodka • EX: Android Academy MSK @nonewsss
  3. 3 1. Why UI testing?

  4. 4 Design:

  5. 5 Constitutes 1. Wheels 2. Seat 3. Handlebars 4. Pedal

    system 5. Frame
  6. 6 
 unit and integration tests All tested! 
 unit

    and integration tests 
 unit and integration tests 
 unit and integration tests 
 unit and integration tests
  7. 7 First implementation

  8. 8 2h later…

  9. 9 Finally!

  10. 10 Main question Is it ridable?

  11. 11 Check the junctions

  12. 12 Expected result — bike can ride

  13. 13 Road

  14. 14 com.bike.app

  15. 15 What if bicycle has been broken?

  16. 16 Call to repair

  17. 17 Hand it over to the repair

  18. 18 Wait

  19. 19 Each bug takes a long time

  20. 20 Mobile: Bug found 1. Fix/open PR, wait for the

    review, merge
  21. 21 Mobile: Bug found 1. Fix/open PR, wait for the

    review, merge 2. Test it
  22. 22 Mobile: Bug found 1. Fix/open PR, wait for the

    review, merge 2. Test it 3. Upload to the Google play
  23. 23 Mobile: Bug found 1. Fix/open PR, wait for the

    review, merge 2. Test it 3. Upload to the Google play 4. Wait for Google play review…
  24. 24 Mobile: Bug found 1. Fix/open PR, wait for the

    review, merge 2. Test it 3. Upload to the Google play 4. Wait for Google play review… 5. Wait for rolling out
  25. 25 Mobile: Bug found 1. Fix/open PR, wait for the

    review, merge 2. Test it 3. Upload to the Google play 4. Wait for Google play review… 5. Wait for rolling out 6. Wait for the users to download it
  26. 26 What affects quality?

  27. 27 What affects quality? 1. More engineers == More features

  28. 28 What affects quality? 1. More engineers == More features

    2. Release frequency
  29. 29 UI testing — our need

  30. 30 UI testing — complicated

  31. 31

  32. 32 Writing Espresso UiAutomator Google provided Community KAutomator Kakao Kaspresso

  33. 33 Writing Running AndroidJUnitRunner Orchestrator Google provided Espresso UiAutomator Google

    provided Community KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Community Flank
  34. 34 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community

    KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Real network Mock network Device Device condition
  35. 35 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community

    KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes
  36. 36 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community

    KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release pipeline Validation on PR When to write?
  37. 37 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community

    KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure Influx db Grafana Found bugs dynamics
  38. 38 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community

    KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure Influx db Grafana Found bugs dynamics
  39. 39 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community

    KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Own Server Firebase GCloud Amazon Kubernetes Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure Influx db Grafana Found bugs dynamics Cloud Should work fine
  40. 40 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community

    KUiAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure Influx db Grafana Found bugs dynamics This talk about
  41. 41 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community

    KUiAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure Influx db Grafana Found bugs dynamics This talk about
  42. 42 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community

    KUiAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure Influx db Grafana Found bugs dynamics This talk about
  43. 43 2. Basics of UI tests running

  44. 44 Pff, I just need to press a green arrow

  45. 45 Let’s take a look

  46. 46 app.apk ./gradlew assembleDebug

  47. 47 app.apk ./gradlew assembleDebug instrumented.apk ./gradlew assembleDebugAndroidTest @Test

  48. 48 app.apk adb install instrumented.apk @Test

  49. 49 app.apk adb install instrumented.apk @Test launch tests

  50. 50 app.apk adb install instrumented.apk @Test launch tests PASSED

  51. 51 app.apk adb install instrumented.apk @Test launch tests PASSED Generate

    report Let’s focus
  52. 52 Android JUnit Runner

  53. 53 Android JUnit Runner Input: tests output: test result TestMethod#testName

    PASSED
  54. 54 adb shell am instrument -w -m -e debug false

    -e class 'com.alexbykov.myapplication.ExampleInstrumentedTest#myTest' com.alexbykov.myapplication.test/ androidx.test.runner.AndroidJUnitRunner To launch tests:
  55. 55 adb shell am instrument -w -m -e debug false

    -e class 'com.alexbykov.myapplication.ExampleInstrumentedTest#myTest' com.alexbykov.myapplication.test/ androidx.test.runner.AndroidJUnitRunner To launch tests:
  56. 56 Full picture ./gradlew assembleDebug ./gradlew assembleDebugAndroidTest // build

  57. 57 ./gradlew assembleDebug ./gradlew assembleDebugAndroidTest Full picture adb install debug.apk

    adb install instrumented.apk // build // install
  58. 58 adb shell am instrument -w -m -e debug false

    -e package 'com.alexbykov.myapplication' com.alexbykov.myapplication.test/ androidx.test.runner.AndroidJUnitRunner ./gradlew assembleDebug ./gradlew assembleDebugAndroidTest adb install debug.apk adb install instrumented.apk Full picture // install // build // run
  59. 59 2.1 How to deal with state?

  60. 60 Test1

  61. 61 Test1

  62. 62 PASSED Test1

  63. 63 Test1 data db, cache, files PASSED Test1

  64. 64 PASSED Test1 data db, cache, files Test1 Test2

  65. 65 PASSED Test1 data db, cache, files Test1 Test2 Test2

    FAILED
  66. 66 Solution: Isolate each test

  67. 67 Clearing 1. Within process 2. Clear package data

  68. 68 Within process clearing 1. LogoutComponent from the code @Before

    fun setUp() { DI.provideLogoutInteractor().logout() }
  69. 69 It’s bottleneck @Before fun setUp() { DI.provideLogoutInteractor().logout() } May

    be broken
  70. 70 All tests will be failed ¯\_(ツ)_/¯

  71. 71 Within process clearing 1. LogoutComponent from the code 2.

    Internal storage clearing
  72. 72 Internal storage clearing All db saved: /data/data/packagename/databases/

  73. 73 Internal storage clearing All db saved: /data/data/packagename/databases/ @get:Rule val

    clearDatabasesRule = ClearDatabasesRule() @get:Rule val clearFilesRule = ClearFilesRule() @get:Rule val clearPreferencesRule = ClearPreferencesRule()
  74. 74 Internal storage clearing All db saved: /data/data/packagename/databases/ @get:Rule val

    clearDatabasesRule = ClearDatabasesRule() @get:Rule val clearFilesRule = ClearFilesRule() @get:Rule val clearPreferencesRule = ClearPreferencesRule() https://github.com/AdevintaSpain/Barista
  75. 75 It doesn’t solve all issues !

  76. 76 It doesn’t solve all issues 1. You may have

    Runtime cache
  77. 77 It doesn’t solve all issues 1. You may have

    Runtime cache 2. App/Test process may be crashed ¯\_(ツ)_/¯
  78. 78 Within process: Conclusions + Fast implementation + Fast execution

  79. 79 Within process: Conclusions - Bottleneck - Don’t have any

    guarantee about clearing - App/Test process might be crashed + Fast implementation + Fast execution
  80. 80 Clear package data

  81. 81 Idea

  82. 82 First thought adb shell am instrument -w -m -e

    debug false -e class ‘….ExampleInstrumentedTest#test1, ….ExampleInstrumentedTest#test2’ com.alexbykov.myapplication.test/ androidx.test.runner.AndroidJUnitRunner
  83. 83 Launch first test adb shell am instrument …. ExampleInstrumentedTest#test1

  84. 84 Clear adb shell am instrument …. ExampleInstrumentedTest#test1 adb shell

    pm clear …
  85. 85 Launch second test adb shell am instrument …. ExampleInstrumentedTest#test1

    adb shell pm clear … adb shell am instrument …. ExampleInstrumentedTest#test2
  86. 86 Clear again adb shell am instrument …. ExampleInstrumentedTest#test1 adb

    shell pm clear … adb shell am instrument …. ExampleInstrumentedTest#test2 adb shell pm clear …
  87. 87 Orchestrator

  88. 88 app.apk instrumented.apk

  89. 89 app.apk instrumented.apk +1 apk orchestrator.apk

  90. 90 app.apk instrumented.apk Start test orchestrator.apk AndroidJunitRunner am instrument

  91. 91 app.apk instrumented.apk Clear orchestrator.apk AndroidJunitRunner am instrument pm clear

    data data
  92. 92 app.apk instrumented.apk Clear orchestrator.apk AndroidJunitRunner am instrument pm clear

    am instrument
  93. 93 How does orchestrator execute commands?

  94. 94 app.apk instrumented.apk Test-services orchestrator.apk test-services.apk shell commands

  95. 95 Package clear: Conclusions + Isolated environment

  96. 96 Package clear: Conclusions - It takes time + Isolated

    environment
  97. 97 It takes time: 300 tests

  98. 98 It takes time: 300 tests Each gap is 7-12

    seconds
  99. 99 Package clear: Conclusions - It takes time - It’s

    complicated + Isolated environment
  100. 100 It’s complicated 2 more apk you need to install

    orchestrator.apk services.apk
  101. 101 It’s complicated 2 more apk you need to install

    orchestrator.apk services.apk Mobile device less reliable than host PC
  102. 102 3. Problems with UI tests run

  103. 103 Problems 1. Flakiness

  104. 104 Flakiness tests will always be

  105. 105 1. Reason of flakiness Production code // toasts, custom

    views, http-requests, databases, animations
  106. 106 Production code Test code // Simulate user actions 1.

    Reason of flakiness
  107. 107 Production code Test code Device problems // Connection, notifications,

    or it’s just stuck 1. Reason of flakiness
  108. 108 Production code Test code Device problems Infrastructure problems //

    I/O, CPU, Memory 1. Reason of flakiness
  109. 109 Problems 1. Flakiness

  110. 110 Problems 1. Flakiness 2. Speed

  111. 111 2. Speed problems 1. Auto-scalling

  112. 112 Auto-scalling Test Run

  113. 113 Auto-scalling Test Run Test Run 2

  114. 114 Test Run: Finished Test Run Test Run 2

  115. 115 Connect all emulators to 2 Test Run 2

  116. 116 1. Auto-scalling 2. Build agents overloading 2. Speed problems

  117. 117 Problems 1. Flakiness 2. Speed

  118. 118 1. Flakiness 2. Speed 3. Tooling Problems

  119. 119 3. Tooling 1. Impossible to validate flakiness

  120. 120 3. Tooling 1. Impossible to validate flakiness 2. Poor

    test report
  121. 121 AndroidJUnitRunner Can’t help us with that

  122. 122 4. Choose: Test runner

  123. 123 Flakiness protection levels Test code

  124. 124 Example1: Kaspresso fun <T> invokeFlakySafely( params: FlakySafetyParams, failureMessage: String?

    = null, action: () -> T ): T { var cachedError: Throwable val startTime = System.currentTimeMillis() do { try { return action.invoke() } catch (error: Throwable) { if (error.isAllowed(params.allowedExceptions)) { cachedError = error lock.withLock { Timer().schedule(params.intervalMs) { lock.withLock { condition.signalAll() } } condition.await() } } else { throw error } } } while (System.currentTimeMillis() - startTime <= params.timeoutMs) throw cachedError.withMessage(failureMessage) }
  125. 125 fun <T> invokeFlakySafely( params: FlakySafetyParams, failureMessage: String? = null,

    action: () -> T ): T { var cachedError: Throwable val startTime = System.currentTimeMillis() do { try { return action.invoke() } catch (error: Throwable) { … } } while (System.currentTimeMillis() - startTime <= params.timeoutMs) throw cachedError.withMessage(failureMessage) } Trying execute actions
  126. 126 fun <T> invokeFlakySafely( params: FlakySafetyParams, failureMessage: String? = null,

    action: () -> T ): T { var cachedError: Throwable val startTime = System.currentTimeMillis() do { try { return action.invoke() } catch (error: Throwable) { … } } while (System.currentTimeMillis() - startTime <= params.timeoutMs) throw cachedError.withMessage(failureMessage) } In a loop
  127. 127 Example2: Notification

  128. 128 Example2: Notification class CloseNotificationsRule : ExternalResource() { override fun

    before() { UiDevice .getInstance(InstrumentationRegistry.getInstrumentation()) .pressHome() } }
  129. 129 Example2: Notification class CloseNotificationsRule : ExternalResource() { override fun

    before() { UiDevice .getInstance(InstrumentationRegistry.getInstrumentation()) .pressHome() } } @get:Rule val closeNotification = CloseNotificationsRule()
  130. 130 Flakiness protection levels Test code

  131. 131 Flakiness protection levels Test code Device

  132. 132 Example: Disabling animations adb shell "settings put global window_animation_scale

    0.0" adb shell "settings put global transition_animation_scale 0.0" adb shell "settings put global animator_duration_scale 0.0"
  133. 133 Example: Other adb shell "settings put global window_animation_scale 0.0"

    adb shell "settings put global transition_animation_scale 0.0" adb shell "settings put global animator_duration_scale 0.0" adb shell "settings put system screen_off_timeout 2147483647" adb shell "settings put secure long_press_timeout 1500"
  134. 134 Flakiness protection levels Test code Device

  135. 135 Flakiness protection levels Test code Device Test Runner

  136. 136 Test runner — last layer of flakiness protection ONE

    OF THE GOALS
  137. 137 Open source test runners Marathon Avito Runner Spoon Fork

    Composer Flank
  138. 138 Open source test runners Marathon Avito Runner Spoon Fork

    Composer Deprecated Flank
  139. 139 Marathon vision Just connect Devices to adb, I’ll do

    the job https://github.com/MarathonLabs/marathon
  140. 140 Marathon + Retries

  141. 141 Marathon + Retries retryStrategy: type: "fixed-quota" totalAllowedRetryQuota: 100 retryPerTestQuota:

    3
  142. 142 Retry Test1 Test2

  143. 143 Passed! Test1 Test2

  144. 144 Failed :( Test1 Test2

  145. 145 Retry on another device Test1 Test2

  146. 146 Marathon + Retries + Flakiness strategy

  147. 147 Idea: Flakiness strategy — Calculate probability of passing —

    Launch flakiness test in parallel before it’s failed
  148. 148 Flakiness strategy Test1: Flaky in 50% //0.5 — probability

    of passing
  149. 149 Flakiness strategy Test1: Flaky in 50% //0.5 — probability

    of passing We want minimum: 80% //0.8 — probability of passing
  150. 150 Flakiness strategy Test1: Flaky in 50% //0.5 — probability

    of passing We want minimum: 80% //0.8 — probability of passing Max parallel run: 3 //because more can affect speed of run
  151. 151 Probability 0.5 = 0.125 Test1: Flaky in 50% We

    want minimum: 80% Max parallel run: 3 //probability of all failed 3
  152. 152 Probability 0.5 = 0.125 1 - 0.125 = 0.875

    > 0.8 Test1: Flaky in 50% We want minimum: 80% Max parallel run: 3 //probability of all failed //Test will be executed 3 times in parallel 3
  153. 153 Flaky test (50% —> 80%) Test Test Test

  154. 154 analyticsConfiguration: graphite: host: "influx.svc.cluster.local" port: "8080" prefix: "prf" To

    make it work
  155. 155 Marathon + Retries + Flakiness strategy

  156. 156 Marathon + Retries + Flakiness strategy + Good report

    and allure support
  157. 157

  158. 158 Failed tests: Video/Logs

  159. 159 However default report doesn’t support retries

  160. 160 But we can do that via Allure allure generate

    report marathon_output/allure-results -o marathon_output/ allure-report --clean
  161. 161 Allure retries

  162. 162 Marathon + Retries + Flakiness strategy + Good report

    and allure support
  163. 163 Marathon + Retries + Flakiness strategy + Good report

    and allure support + Isolated environment applicationPmClear: true testApplicationPmClear: true
  164. 164 Marathon + Retries + Flakiness strategy + Good report

    and allure support + Isolated environment + ADB replacement
  165. 165 Replace ADB? WAT?

  166. 166 Where do we use ADB?

  167. 167 ADB in instrumented tests Install apps Clearing Pulling files

    Record video
  168. 168 Problem Most of the commands create a new thread

    under the hood
  169. 169

  170. 170 Solution: ADAM

  171. 171 Solution: ADAM Uses Kotlin coroutines under the hood

  172. 172 Adam vs ADB ADB: 300 tests T: 32 min

    I/O: 22% CPU: 48% Adam: 300 tests T: 26 min I/O: 19% CPU: 47 %
  173. 173 Marathon + Retries + Flakiness strategy + Good report

    and allure support + Isolated environment + ADB replacement
  174. 174 Marathon + Retries + Flakiness strategy + Good report

    and allure support + Isolated environment + ADB replacement + Flakiness validation strictMode: true shardingStrategy: type: "count" count: 100
  175. 175 Marathon + Retries + Flakiness strategy + Good report

    and allure support + Isolated environment + ADB replacement + Flakiness validation + Cross platform (iOS support)
  176. 176 Open source test runners Marathon

  177. 177 Open source test runners Marathon Avito Runner

  178. 178 Avito runner vision Just connect Kubernetes to me, I’ll

    do the job https://avito-tech.github.io/avito-android/test_runner/TestRunner/
  179. 179 Test Run: Finished Test Run Test Run 2

  180. 180 Connect all emulators to 2 Test Run 2

  181. 181 Marathon Possible, but you need 
 to connect devices

    to adb by yourself Autoscalling
  182. 182 Device farm

  183. 183 Device state idle busy Device farm

  184. 184 Build agent

  185. 185 ADB connect Build agent Device farm

  186. 186 Avito runner I’ll do it for you! Autoscalling

  187. 187 Difference Marathon Avito Test Runner - Auto-scalling - Adam

    - Flakiness strategy - CLI - Doesn’t support iOS
  188. 188 Difference Marathon Avito Test Runner - Auto-scalling - Adam

    - Flakiness strategy - CLI - Doesn’t support iOS - It’s not ready for open source
  189. 189 Open source test runners Marathon Avito Runner

  190. 190 Open source test runners Marathon Avito Runner Flank

  191. 191 Flank vision https://flank.github.io/flank/ I’ll make your life easier to

    work with Firebase Test Lab https://firebase.google.com/products/test-lab
  192. 192 Flank + Don’t need to care about where to

    run — Firebase Test Lab — paid service + Crossplatform
  193. 193 Which one do I need to choose?

  194. 194 It depends :)

  195. 195 It depends :) Product? Team size? Company? Scalability? How

    often do you need to run? How many tests do you have? How often your codebase changed?
  196. 196 That’s on you Marathon, Avito Runner: + invest your

    time to build infrastructure + invest your time to support
  197. 197 That’s on you Marathon, Avito Runner: + invest your

    time to support + invest your time to build infrastructure Flank + Firebase test lab: + You have to pay always
  198. 198 Marathon — the most powerful

  199. 199 Marathon — the most powerful For any team.

  200. 200 Real device vs Emulator

  201. 201 Real device

  202. 202 Real device + Real condition

  203. 203 Real device + Real condition + Doesn’t consume CI

    resources
  204. 204 Real device + Real condition + Doesn’t consume CI

    resources — Room with special condition
  205. 205 Real device + Real condition + Doesn’t consume CI

    resources — Room with special condition — Breaks often
  206. 206 Emulator

  207. 207 + Easy manageable Emulator

  208. 208 + Easy manageable + Easy configurable Emulator

  209. 209 + Easy manageable + Easy configurable + Work faster

    Emulator
  210. 210 Works faster? Before: Real devices

  211. 211 Works faster? Before: Real devices Average: 50 seconds

  212. 212 Works faster? Before: Real devices Average: 50 seconds After:

    Emulators Average: 30 seconds
  213. 213 Why Fresh emulator instance per each test suit

  214. 214 Why Fresh emulator instance per each test suit Only

    your application installed
  215. 215 Why Fresh emulator instance per each test suit Only

    your application installed Special configuration
  216. 216 PlayStore.enabled=false abi.type=x86_64 avd.ini.encoding=UTF-8 hw.cpu.arch=x86_64 hw.cpu.ncore=2 hw.ramSize=2048 hw.lcd.density=120 hw.lcd.width=320 hw.lcd.height=480

    hw.audioInput=no hw.audioOutput=no hw.accelerometer=no hw.gyroscope=no hw.dPad=no hw.mainKeys=yes hw.keyboard=no hw.sensors.proximity=no hw.sensors.magnetic_field=no hw.sensors.orientation=no hw.sensors.temperature=no hw.sensors.light=no hw.sensors.pressure=no hw.sensors.humidity=no hw.sensors.magnetic_field_uncalibrated=no hw.sensors.gyroscope_uncalibrated=no image.sysdir.1=system-images/android-29/google_apis/x86_64/ tag.display=Google APIs tag.id=google_apis skin.dynamic=yes skin.name=320x480 disk.dataPartition.size=8G Special configuration
  217. 217 PlayStore.enabled=false abi.type=x86_64 avd.ini.encoding=UTF-8 hw.cpu.arch=x86_64 hw.cpu.ncore=2 hw.ramSize=2048 hw.lcd.density=120 hw.lcd.width=320 hw.lcd.height=480

    hw.audioInput=no hw.audioOutput=no hw.accelerometer=no hw.gyroscope=no hw.dPad=no hw.mainKeys=yes hw.keyboard=no hw.sensors.proximity=no hw.sensors.magnetic_field=no hw.sensors.orientation=no hw.sensors.temperature=no hw.sensors.light=no hw.sensors.pressure=no hw.sensors.humidity=no hw.sensors.magnetic_field_uncalibrated=no hw.sensors.gyroscope_uncalibrated=no image.sysdir.1=system-images/android-29/google_apis/x86_64/ tag.display=Google APIs tag.id=google_apis skin.dynamic=yes skin.name=320x480 disk.dataPartition.size=8G Special configuration
  218. 218 PlayStore.enabled=false abi.type=x86_64 avd.ini.encoding=UTF-8 hw.cpu.arch=x86_64 hw.cpu.ncore=2 hw.ramSize=2048 hw.lcd.density=120 hw.lcd.width=320 hw.lcd.height=480

    hw.audioInput=no hw.audioOutput=no hw.accelerometer=no hw.gyroscope=no hw.dPad=no hw.mainKeys=yes hw.keyboard=no hw.sensors.proximity=no hw.sensors.magnetic_field=no hw.sensors.orientation=no hw.sensors.temperature=no hw.sensors.light=no hw.sensors.pressure=no hw.sensors.humidity=no Special configuration
  219. 219 Emulator Emulator + Easy manageable + Easy configurable +

    Work faster
  220. 220 Emulator Emulator + Easy manageable + Easy configurable +

    Work faster — Consume CI resources
  221. 221 Emulator Emulator + Easy manageable + Easy configurable +

    Work faster — Consume CI resources — It’s not a real device
  222. 222 How to build emulator?

  223. 223 How to build emulator? Programatically (AVD Manager)

  224. 224 Programatically sample shorturl.at/gkEY1 46 lines of code :(

  225. 225 How to build emulator? Programatically (AVD Manager) Docker image

  226. 226 Docker emulators Google (official) shorturl.at/azNY4 Avito shorturl.at/fmHKN Agoda shorturl.at/xGJKY

  227. 227 docker run -d -p 5555:5555 -p 5554:5554 -p 8554:8554

    
 --privileged avitotech/android-emulator-29:915c1f20be //Run emulator (headless instance)
  228. 228 docker run -d -p 5555:5555 -p 5554:5554 -p 8554:8554

    
 --privileged avitotech/android-emulator-29:915c1f20be //Run emulator (headless instance) //Connect to adb adb connect localhost:5555
  229. 229 docker run -d -p 5555:5555 -p 5554:5554 -p 8554:8554

    
 --privileged avitotech/android-emulator-29:915c1f20be //Run emulator (headless instance) //Connect to adb adb connect localhost:5555 //Kill emulators (after tests finished) docker kill $(docker ps -q)
 docker rm $(docker ps -a -q)
  230. 230 Conclusion Headless emulators — pragmatic choice (faster and easy)

  231. 231 Conclusion Headless emulators — pragmatic choice (faster and easy)

    It’s fine to start with real device (If your build agent is not powerful)
  232. 232 How to deal with network

  233. 233 Options Do nothing

  234. 234 Do nothing (Real network) + E2E testing

  235. 235 Do nothing (Real network) + E2E testing — Server

    is down (applies to development)
  236. 236 Do nothing (Real network) + E2E testing — Server

    is down (applies to development) — Network issue
  237. 237 Options Do nothing

  238. 238 Options Dedicated stage server Do nothing

  239. 239 Dedicated stage + A bit stablier

  240. 240 Dedicated stage + A bit stablier — Network issue

  241. 241 Dedicated stage + A bit stablier — Network issue

    — Not as same as a real server
  242. 242 Dedicated stage + A bit stablier — Network issue

    — Not as same as a real server — It takes time to support
  243. 243 Options Dedicated stage server Do nothing

  244. 244 Options Mock/Stub it Dedicated stage server Do nothing

  245. 245 Mock it / Stub it + Stable

  246. 246 Mock it / Stub it + Stable + Still

    can catch the bugs
  247. 247 Mock it / Stub it + Stable + Still

    can catch the bugs + Pragmatic
  248. 248 Mock it / Stub it + Stable + Still

    can catch the bugs — Not fully E2E + Pragmatic — Requires some effort from engineers
  249. 249 How to mock/stub?

  250. 250 Network: How to mock/stub? MockWebServer

  251. 251 Network: How to mock/stub? MockWebServer Custom Interceptor for testing

  252. 252 Network: How to mock/stub? MockWebServer Custom Interceptor for testing

    DI: Dagger2/Hilt, Toothpick, Koin/Kodein
  253. 253 Network: How to mock/stub? MockWebServer Custom Interceptor for testing

    DI: Dagger2/Hilt, Toothpick, Koin/Kodein Service Locator / different src
  254. 254 MockWebServer Custom Interceptor for testing DI: Dagger2/Hilt, Toothpick, Koin/Kodein

    Service Locator Didn’t work properly
  255. 255 Why: — A lot of requests

  256. 256 Why: — A lot of requests //100+ in some

    features
  257. 257 Why: — A lot of requests //100+ in some

    features — Long hours to mock
  258. 258 Solution

  259. 259 Solution 2 network modes: Record and Replay

  260. 260 Idea: Record Build

  261. 261 Build Run test Idea: Record

  262. 262 Build Run test Real network Idea: Record

  263. 263 Build Run test Real network Output ClassName#testName.yaml Idea: Record

  264. 264 How does yaml file look like?

  265. 265 - recorded: 2021-10-13T14:30:23.507Z request: method: GET uri: https://api.server.com/inbox headers:

    response: status: 200 headers: date: Wed, 13 Oct 2021 14:30:26 GMT request-id: 9JAJH8Y9JMRU server: MyServer body: '{"messages":[]}' Example
  266. 266 Build Idea: Replay

  267. 267 Build ClassName#testName.yaml Idea: Replay

  268. 268 Build ClassName#testName.yaml Idea: Replay Run test

  269. 269 Build ClassName#testName.yaml Idea: Replay Run test

  270. 270 Modes usage Local machine: Playback, Record CI: Playback

  271. 271 How to implement this?

  272. 272 Solutions Wiremock https://github.com/wiremock/wiremock

  273. 273 Solutions Wiremock + MITM under the hood

  274. 274 Solutions Wiremock + MITM under the hood + Requests

    indexing
  275. 275 Solutions Wiremock + MITM under the hood + Requests

    indexing — Support only 1 host
  276. 276 Solutions Wiremock + MITM under the hood + Requests

    indexing — Support only 1 host — Not Android Friendly
  277. 277 Solutions Wiremock OkReplay https://github.com/airbnb/okreplay

  278. 278 Solutions Wiremock OkReplay + Readable report

  279. 279 Solutions Wiremock OkReplay + Readable report + Interceptor-based solution

  280. 280 Solutions Wiremock OkReplay + Readable report + Interceptor-based solution

    — Lack of indexing
  281. 281 Solutions Wiremock OkReplay + Readable report + Interceptor-based solution

    — Lack of indexing — Constant execution time per each request
  282. 282 Solutions Wiremock OkReplay + Readable report + Interceptor-based solution

    — Lack of indexing — Constant execution time per each request — Abandoned :(
  283. 283 We’re implementing our own open-source solution for it

  284. 284 Network: Conclusion — Mocked/Stubbed network it — the only

    way to have stable UI tests
  285. 285 Network: Conclusion — Mocked/Stubbed network it — the only

    way to have stable UI tests — Small project: MockWebServer
  286. 286 Network: Conclusion — Mocked/Stubbed network it — the only

    way to have stable UI tests — Small project: MockWebServer — Have a lot of requests to mock? — OkReplay
  287. 287 Network: Conclusion — Mocked/Stubbed network it — the only

    way to have stable UI tests — Small project: MockWebServer — Have a lot of requests to mock? — OkReplay — Didn’t use OkHttp? — Wiremock
  288. 288 Conclusions

  289. 289 Android UI testing

  290. 290 Android UI testing Expectations Reality

  291. 291 Before you get started Do not even start if

    you want to try it just for fun
  292. 292 Before you get started Do not even start if

    you want to try it just for fun You should have clear goal
  293. 293 Before you get started You should have clear goal

    Which solves real problems Do not even start if you want to try it just for fun
  294. 294 UI testing blog https://github.com/android-ui-testing/Cookbook/

  295. 295 Thanks to Eugene Matsyuk Alexey Tvorogov Dmitriy Voronin Anton

    Malinskiy Dmitry Movchan Kaspresso adb server Kaspresso creator Marathon creator Avito runner creator Kaspresso adb server & a lot of researches
  296. 296 Resources 1. https://github.com/KasperskyLab/Kaspresso 3. https://github.com/MarathonLabs/marathon 2. https://proandroiddev.com/autotests-on-android-the-entire-picture-51c07995fc2f ( 4.

    https://github.com/avito-tech/avito-android 5. https://github.com/Flank/flank 6. https://github.com/airbnb/okreplay 7. https://github.com/wiremock/wiremock
  297. 297 Alexey Bykov, Android Software Engineer Presentation @nonewsss Thank you!