Where and how to run UI tests (Droidcon Lisbon & Android Makers, Paris)

UI tests have never been easy.
Yes, with frameworks like Kaspresso and Kakao it gets easier year by year.

However, running ui tests still raises a lot of questions, like:
- Which test runner do I need to use?
- How to deal with unstable network?
- Emulator or Real Device?
- How to clear the state and why?
- How to make tests less flakiness?
- How to organise good reports?

We gonna talk about all of these topics and deep dive into test runners, emulators, why adb is not the best option and find the most pragmatic options for your team no matter how big is it

Transcript

1. 1 Where and how to run UI tests Alexey Bykov
2. 2 Alexey Bykov Android Software Engineer @Revolut • 5 years in Android • EX Programm commette: AppsConf, Podlodka • EX: Android Academy MSK @nonewsss
3. 3 1. Why UI testing?
4. 4 Design:
5. 5 Constitutes 1. Wheels 2. Seat 3. Handlebars 4. Pedal system 5. Frame
6. 6 
 unit and integration tests All tested! 
 unit and integration tests 
 unit and integration tests 
 unit and integration tests 
 unit and integration tests
7. 7 First implementation
8. 8 2h later…
9. 9 Finally!
10. 10 Main question Is it ridable?
11. 11 Check the junctions
12. 12 Expected result — bike can ride
13. 13 Road
14. 14 com.bike.app
15. 15 What if bicycle has been broken?
16. 16 Call to repair
17. 17 Hand it over to the repair
18. 18 Wait
19. 19 Each bug takes a long time
20. 20 Mobile: Bug found 1. Fix/open PR, wait for the review, merge
21. 21 Mobile: Bug found 1. Fix/open PR, wait for the review, merge 2. Test it
22. 22 Mobile: Bug found 1. Fix/open PR, wait for the review, merge 2. Test it 3. Upload to the Google play
23. 23 Mobile: Bug found 1. Fix/open PR, wait for the review, merge 2. Test it 3. Upload to the Google play 4. Wait for Google play review…
24. 24 Mobile: Bug found 1. Fix/open PR, wait for the review, merge 2. Test it 3. Upload to the Google play 4. Wait for Google play review… 5. Wait for rolling out
25. 25 Mobile: Bug found 1. Fix/open PR, wait for the review, merge 2. Test it 3. Upload to the Google play 4. Wait for Google play review… 5. Wait for rolling out 6. Wait for the users to download it
26. 26 What affects quality?
27. 27 What affects quality? 1. More engineers == More features
28. 28 What affects quality? 1. More engineers == More features 2. Release frequency
29. 29 UI testing — our need
30. 30 UI testing — complicated
32. 32 Writing Espresso UiAutomator Google provided Community KAutomator Kakao Kaspresso
33. 33 Writing Running AndroidJUnitRunner Orchestrator Google provided Espresso UiAutomator Google provided Community KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Community Flank
34. 34 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Real network Mock network Device Device condition
35. 35 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes
36. 36 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release pipeline Validation on PR When to write?
37. 37 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure In fl ux db Grafana Found bugs dynamics
38. 38 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure In fl ux db Grafana Found bugs dynamics
39. 39 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Own Server Firebase GCloud Amazon Kubernetes Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure In fl ux db Grafana Found bugs dynamics Cloud Should work ne
40. 40 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KUiAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure In fl ux db Grafana Found bugs dynamics This talk about
41. 41 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KUiAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure In fl ux db Grafana Found bugs dynamics This talk about
42. 42 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KUiAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure In fl ux db Grafana Found bugs dynamics This talk about
43. 43 2. Basics of UI tests running
44. 44 Pff, I just need to press a green arrow
45. 45 Let’s take a look
46. 46 app.apk ./gradlew assembleDebug
47. 47 app.apk ./gradlew assembleDebug instrumented.apk ./gradlew assembleDebugAndroidTest @Test
48. 48 app.apk adb install instrumented.apk @Test
49. 49 app.apk adb install instrumented.apk @Test launch tests
50. 50 app.apk adb install instrumented.apk @Test launch tests PASSED
51. 51 app.apk adb install instrumented.apk @Test launch tests PASSED Generate report Let’s focus
52. 52 Android JUnit Runner
53. 53 Android JUnit Runner Input: tests output: test result TestMethod#testName PASSED
54. 54 adb shell am instrument -w -m -e debug false -e class 'com.alexbykov.myapplication.ExampleInstrumentedTest#myTest' com.alexbykov.myapplication.test/ androidx.test.runner.AndroidJUnitRunner To launch tests:
55. 55 adb shell am instrument -w -m -e debug false -e class 'com.alexbykov.myapplication.ExampleInstrumentedTest#myTest' com.alexbykov.myapplication.test/ androidx.test.runner.AndroidJUnitRunner To launch tests:
56. 56 Full picture ./gradlew assembleDebug ./gradlew assembleDebugAndroidTest // build
57. 57 ./gradlew assembleDebug ./gradlew assembleDebugAndroidTest Full picture adb install debug.apk adb install instrumented.apk // build // install
58. 58 adb shell am instrument -w -m -e debug false -e package 'com.alexbykov.myapplication' com.alexbykov.myapplication.test/ androidx.test.runner.AndroidJUnitRunner ./gradlew assembleDebug ./gradlew assembleDebugAndroidTest adb install debug.apk adb install instrumented.apk Full picture // install // build // run
59. 59 2.1 How to deal with state?
60. 60 Test1
61. 61 Test1
62. 62 PASSED Test1
63. 63 Test1 data db, cache, files PASSED Test1
64. 64 PASSED Test1 data db, cache, files Test1 Test2
65. 65 PASSED Test1 data db, cache, files Test1 Test2 Test2 FAILED
66. 66 Solution: Isolate each test
67. 67 Clearing 1. Within process 2. Clear package data
68. 68 Within process clearing 1. LogoutComponent from the code @Before fun setUp() { DI.provideLogoutInteractor().logout() }
69. 69 It’s bottleneck @Before fun setUp() { DI.provideLogoutInteractor().logout() } May be broken
70. 70 All tests will be failed ¯\_(ツ)_/¯
71. 71 Within process clearing 1. LogoutComponent from the code 2. Internal storage clearing
72. 72 Internal storage clearing All db saved: /data/data/packagename/databases/
73. 73 Internal storage clearing All db saved: /data/data/packagename/databases/ @get:Rule val clearDatabasesRule = ClearDatabasesRule() @get:Rule val clearFilesRule = ClearFilesRule() @get:Rule val clearPreferencesRule = ClearPreferencesRule()
74. 74 Internal storage clearing All db saved: /data/data/packagename/databases/ @get:Rule val clearDatabasesRule = ClearDatabasesRule() @get:Rule val clearFilesRule = ClearFilesRule() @get:Rule val clearPreferencesRule = ClearPreferencesRule() https://github.com/AdevintaSpain/Barista
75. 75 It doesn’t solve all issues !
76. 76 It doesn’t solve all issues 1. You may have Runtime cache
77. 77 It doesn’t solve all issues 1. You may have Runtime cache 2. App/Test process may be crashed ¯\_(ツ)_/¯
78. 78 Within process: Conclusions + Fast implementation + Fast execution
79. 79 Within process: Conclusions - Bottleneck - Don’t have any guarantee about clearing - App/Test process might be crashed + Fast implementation + Fast execution
80. 80 Clear package data
81. 81 Idea
82. 82 First thought adb shell am instrument -w -m -e debug false -e class ‘….ExampleInstrumentedTest#test1,….ExampleInstrumentedTest#tes t2’ com.alexbykov.myapplication.test/ androidx.test.runner.AndroidJUnitRunner
83. 83 Launch rst test adb shell am instrument …. ExampleInstrumentedTest#test1
84. 84 Clear adb shell am instrument …. ExampleInstrumentedTest#test1 adb shell pm clear …
85. 85 Launch second test adb shell am instrument …. ExampleInstrumentedTest#test1 adb shell pm clear … adb shell am instrument …. ExampleInstrumentedTest#test2
86. 86 Clear again adb shell am instrument …. ExampleInstrumentedTest#test1 adb shell pm clear … adb shell am instrument …. ExampleInstrumentedTest#test2 adb shell pm clear …
87. 87 Orchestrator
88. 88 app.apk instrumented.apk
89. 89 app.apk instrumented.apk +1 apk orchestrator.apk
90. 90 app.apk instrumented.apk Start test orchestrator.apk AndroidJunitRunner am instrument
91. 91 app.apk instrumented.apk Clear orchestrator.apk AndroidJunitRunner am instrument pm clear data data
92. 92 app.apk instrumented.apk Clear orchestrator.apk AndroidJunitRunner am instrument pm clear am instrument
93. 93 How does orchestrator execute commands?
94. 94 app.apk instrumented.apk Test-services orchestrator.apk test-services.apk shell commands
95. 95 Package clear: Conclusions + Isolated environment
96. 96 Package clear: Conclusions - It takes time + Isolated environment
97. 97 It takes time: 300 tests
98. 98 It takes time: 300 tests Each gap is 7-12 seconds
99. 99 Package clear: Conclusions - It takes time - It’s complicated + Isolated environment
100. 100 It’s complicated 2 more apk you need to install orchestrator.apk services.apk
101. 101 It’s complicated 2 more apk you need to install orchestrator.apk services.apk Mobile device less reliable than host PC
102. 102 3. Problems with UI tests run
103. 103 Problems 1. Flakiness
104. 104 Flakiness tests will always be
105. 105 1. Reason of akiness Production code // toasts, custom views, http-requests, databases, animations
106. 106 Production code Test code // Simulate user actions 1. Reason of akiness
107. 107 Production code Test code Device problems // Connection, notifications, or it’s just stuck 1. Reason of akiness
108. 108 Production code Test code Device problems Infrastructure problems // I/O, CPU, Memory 1. Reason of akiness
109. 109 Problems 1. Flakiness
110. 110 Problems 1. Flakiness 2. Speed
111. 111 2. Speed problems 1. Auto-scalling
112. 112 Auto-scalling Test Run
113. 113 Auto-scalling Test Run Test Run 2
114. 114 Test Run: Finished Test Run Test Run 2
115. 115 Connect all emulators to 2 Test Run 2
116. 116 1. Auto-scalling 2. Build agents overloading 2. Speed problems
117. 117 Problems 1. Flakiness 2. Speed
118. 118 1. Flakiness 2. Speed 3. Tooling Problems
119. 119 3. Tooling 1. Impossible to validate flakiness
120. 120 3. Tooling 1. Impossible to validate flakiness 2. Poor test report
121. 121 AndroidJUnitRunner Can’t help us with that
122. 122 4. Choose: Test runner
123. 123 Flakiness protection levels Test code
124. 124 Example1: Kaspresso fun <T> invokeFlakySafely( params: FlakySafetyParams, failureMessage: String? = null, action: () -> T ): T { var cachedError: Throwable val startTime = System.currentTimeMillis() do { try { return action.invoke() } catch (error: Throwable) { if (error.isAllowed(params.allowedExceptions)) { cachedError = error lock.withLock { Timer().schedule(params.intervalMs) { lock.withLock { condition.signalAll() } } condition.await() } } else { throw error } } } while (System.currentTimeMillis() - startTime <= params.timeoutMs) throw cachedError.withMessage(failureMessage) }
125. 125 fun <T> invokeFlakySafely( params: FlakySafetyParams, failureMessage: String? = null, action: () -> T ): T { var cachedError: Throwable val startTime = System.currentTimeMillis() do { try { return action.invoke() } catch (error: Throwable) { … } } while (System.currentTimeMillis() - startTime <= params.timeoutMs) throw cachedError.withMessage(failureMessage) } Trying execute actions
126. 126 fun <T> invokeFlakySafely( params: FlakySafetyParams, failureMessage: String? = null, action: () -> T ): T { var cachedError: Throwable val startTime = System.currentTimeMillis() do { try { return action.invoke() } catch (error: Throwable) { … } } while (System.currentTimeMillis() - startTime <= params.timeoutMs) throw cachedError.withMessage(failureMessage) } In a loop
127. 127 Example2: Noti cation
128. 128 Example2: Noti cation class CloseNotificationsRule : ExternalResource() { override fun before() { UiDevice .getInstance(InstrumentationRegistry.getInstrumentation()) .pressHome() } }
129. 129 Example2: Noti cation class CloseNotificationsRule : ExternalResource() { override fun before() { UiDevice .getInstance(InstrumentationRegistry.getInstrumentation()) .pressHome() } } @get:Rule val closeNotification = CloseNotificationsRule()
130. 130 Flakiness protection levels Test code
131. 131 Flakiness protection levels Test code Device
132. 132 Example: Disabling animations adb shell "settings put global window_animation_scale 0.0" adb shell "settings put global transition_animation_scale 0.0" adb shell "settings put global animator_duration_scale 0.0"
133. 133 Example: Other adb shell "settings put global window_animation_scale 0.0" adb shell "settings put global transition_animation_scale 0.0" adb shell "settings put global animator_duration_scale 0.0" adb shell "settings put system screen_off_timeout 2147483647" adb shell "settings put secure long_press_timeout 1500"
134. 134 Flakiness protection levels Test code Device
135. 135 Flakiness protection levels Test code Device Test Runner
136. 136 Test runner — last layer of flakiness protection ONE OF THE GOALS
137. 137 Open source test runners Marathon Avito Runner Spoon Fork Composer Flank
138. 138 Open source test runners Marathon Avito Runner Spoon Fork Composer Deprecated Flank
139. 139 Marathon vision Just connect Devices to adb, I’ll do the job https://github.com/MarathonLabs/marathon
140. 140 Marathon + Retries
141. 141 Marathon + Retries retryStrategy: type: "fixed-quota" totalAllowedRetryQuota: 100 retryPerTestQuota: 3
142. 142 Retry Test1 Test2
143. 143 Passed! Test1 Test2
144. 144 Failed :( Test1 Test2
145. 145 Retry on another device Test1 Test2
146. 146 Marathon + Retries + Flakiness strategy
147. 147 Idea: Flakiness strategy — Calculate probability of passing — Launch flakiness test in parallel before it’s failed
148. 148 Flakiness strategy Test1: Flaky in 50% //0.5 — probability of passing
149. 149 Flakiness strategy Test1: Flaky in 50% //0.5 — probability of passing We want minimum: 80% //0.8 — probability of passing
150. 150 Flakiness strategy Test1: Flaky in 50% //0.5 — probability of passing We want minimum: 80% //0.8 — probability of passing Max parallel run: 3 //because more can affect speed of run
151. 151 Probability 0.5 = 0.125 Test1: Flaky in 50% We want minimum: 80% Max parallel run: 3 //probability of all failed 3
152. 152 Probability 0.5 = 0.125 1 - 0.125 = 0.875 > 0.8 Test1: Flaky in 50% We want minimum: 80% Max parallel run: 3 //probability of all failed //Test will be executed 3 times in parallel 3
153. 153 Flaky test (50% —> 80%) Test Test Test
154. 154 analyticsConfiguration: graphite: host: "influx.svc.cluster.local" port: "8080" prefix: "prf" To make it work
155. 155 Marathon + Retries + Flakiness strategy
156. 156 Marathon + Retries + Flakiness strategy + Good report and allure support
158. 158 Failed tests: Video/Logs
159. 159 However default report doesn’t support retries
160. 160 But we can do that via Allure allure generate report marathon_output/allure-results -o marathon_output/ allure-report --clean
161. 161 Allure retries
162. 162 Marathon + Retries + Flakiness strategy + Good report and allure support
163. 163 Marathon + Retries + Flakiness strategy + Good report and allure support + Isolated environment applicationPmClear: true testApplicationPmClear: true
164. 164 Marathon + Retries + Flakiness strategy + Good report and allure support + Isolated environment + ADB replacement
165. 165 Replace ADB? WAT?
166. 166 Where do we use ADB?
167. 167 ADB in instrumented tests Install apps Clearing Pulling files Record video
168. 168 Problem Most of the commands create a new thread under the hood
170. 170 Solution: ADAM
171. 171 Solution: ADAM Uses Kotlin coroutines under the hood
172. 172 Adam vs ADB ADB: 300 tests T: 32 min I/O: 22% CPU: 48% Adam: 300 tests T: 26 min I/O: 19% CPU: 47 %
173. 173 Marathon + Retries + Flakiness strategy + Good report and allure support + Isolated environment + ADB replacement
174. 174 Marathon + Retries + Flakiness strategy + Good report and allure support + Isolated environment + ADB replacement + Flakiness validation strictMode: true shardingStrategy: type: "count" count: 100
175. 175 Marathon + Retries + Flakiness strategy + Good report and allure support + Isolated environment + ADB replacement + Flakiness validation + Cross platform (iOS support)
176. 176 Open source test runners Marathon
177. 177 Open source test runners Marathon Avito Runner
178. 178 Avito runner vision Just connect Kubernetes to me, I’ll do the job https://avito-tech.github.io/avito-android/test_runner/TestRunner/
179. 179 Test Run: Finished Test Run Test Run 2
180. 180 Connect all emulators to 2 Test Run 2
181. 181 Marathon Possible, but you need 
 to connect devices to adb by yourself Autoscalling
182. 182 Device farm
183. 183 Device state idle busy Device farm
184. 184 Build agent
185. 185 ADB connect Build agent Device farm
186. 186 Avito runner I’ll do it for you! Autoscalling
187. 187 Difference Marathon Avito Test Runner - Auto-scalling - Adam - Flakiness strategy - CLI - Doesn’t support iOS
188. 188 Difference Marathon Avito Test Runner - Auto-scalling - Adam - Flakiness strategy - CLI - Doesn’t support iOS - It’s not ready for open source
189. 189 Open source test runners Marathon Avito Runner
190. 190 Open source test runners Marathon Avito Runner Flank
191. 191 Flank vision https://flank.github.io/flank/ I’ll make your life easier to work with Firebase Test Lab https://firebase.google.com/products/test-lab
192. 192 Flank + Don’t need to care about where to run — Firebase Test Lab — paid service + Crossplatform
193. 193 Which one do I need to choose?
194. 194 It depends :)
195. 195 It depends :) Product? Team size? Company? Scalability? How often do you need to run? How many tests do you have? How often your codebase changed?
196. 196 That’s on you Marathon, Avito Runner: + invest your time to build infrastructure + invest your time to support
197. 197 That’s on you Marathon, Avito Runner: + invest your time to support + invest your time to build infrastructure Flank + Firebase test lab: + You have to pay always
198. 198 Marathon — the most powerful
199. 199 Marathon — the most powerful For any team.
200. 200 Real device vs Emulator
201. 201 Real device
202. 202 Real device + Real condition
203. 203 Real device + Real condition + Doesn’t consume CI resources
204. 204 Real device + Real condition + Doesn’t consume CI resources — Room with special condition
205. 205 Real device + Real condition + Doesn’t consume CI resources — Room with special condition — Breaks often
206. 206 Emulator
207. 207 + Easy manageable Emulator
208. 208 + Easy manageable + Easy configurable Emulator
209. 209 + Easy manageable + Easy configurable + Work faster Emulator
210. 210 Works faster? Before: Real devices
211. 211 Works faster? Before: Real devices Average: 50 seconds
212. 212 Works faster? Before: Real devices Average: 50 seconds After: Emulators Average: 30 seconds
213. 213 Why Fresh emulator instance per each test suit
214. 214 Why Fresh emulator instance per each test suit Only your application installed
215. 215 Why Fresh emulator instance per each test suit Only your application installed Special configuration
216. 216 PlayStore.enabled=false abi.type=x86_64 avd.ini.encoding=UTF-8 hw.cpu.arch=x86_64 hw.cpu.ncore=2 hw.ramSize=2048 hw.lcd.density=120 hw.lcd.width=320 hw.lcd.height=480 hw.audioInput=no hw.audioOutput=no hw.accelerometer=no hw.gyroscope=no hw.dPad=no hw.mainKeys=yes hw.keyboard=no hw.sensors.proximity=no hw.sensors.magnetic_field=no hw.sensors.orientation=no hw.sensors.temperature=no hw.sensors.light=no hw.sensors.pressure=no hw.sensors.humidity=no hw.sensors.magnetic_field_uncalibrated=no hw.sensors.gyroscope_uncalibrated=no image.sysdir.1=system-images/android-29/google_apis/x86_64/ tag.display=Google APIs tag.id=google_apis skin.dynamic=yes skin.name=320x480 disk.dataPartition.size=8G Special configuration
217. 217 PlayStore.enabled=false abi.type=x86_64 avd.ini.encoding=UTF-8 hw.cpu.arch=x86_64 hw.cpu.ncore=2 hw.ramSize=2048 hw.lcd.density=120 hw.lcd.width=320 hw.lcd.height=480 hw.audioInput=no hw.audioOutput=no hw.accelerometer=no hw.gyroscope=no hw.dPad=no hw.mainKeys=yes hw.keyboard=no hw.sensors.proximity=no hw.sensors.magnetic_field=no hw.sensors.orientation=no hw.sensors.temperature=no hw.sensors.light=no hw.sensors.pressure=no hw.sensors.humidity=no hw.sensors.magnetic_field_uncalibrated=no hw.sensors.gyroscope_uncalibrated=no image.sysdir.1=system-images/android-29/google_apis/x86_64/ tag.display=Google APIs tag.id=google_apis skin.dynamic=yes skin.name=320x480 disk.dataPartition.size=8G Special configuration
218. 218 PlayStore.enabled=false abi.type=x86_64 avd.ini.encoding=UTF-8 hw.cpu.arch=x86_64 hw.cpu.ncore=2 hw.ramSize=2048 hw.lcd.density=120 hw.lcd.width=320 hw.lcd.height=480 hw.audioInput=no hw.audioOutput=no hw.accelerometer=no hw.gyroscope=no hw.dPad=no hw.mainKeys=yes hw.keyboard=no hw.sensors.proximity=no hw.sensors.magnetic_field=no hw.sensors.orientation=no hw.sensors.temperature=no hw.sensors.light=no hw.sensors.pressure=no hw.sensors.humidity=no Special configuration
219. 219 Emulator Emulator + Easy manageable + Easy configurable + Work faster
220. 220 Emulator Emulator + Easy manageable + Easy configurable + Work faster — Consume CI resources
221. 221 Emulator Emulator + Easy manageable + Easy configurable + Work faster — Consume CI resources — It’s not a real device
222. 222 How to build emulator?
223. 223 How to build emulator? Programatically (AVD Manager)
224. 224 Programatically sample shorturl.at/gkEY1 46 lines of code :(
225. 225 How to build emulator? Programatically (AVD Manager) Docker image
226. 226 Docker emulators Google (official) shorturl.at/azNY4 Avito shorturl.at/fmHKN Agoda shorturl.at/xGJKY
227. 227 docker run -d -p 5555:5555 -p 5554:5554 -p 8554:8554 
 --privileged avitotech/android-emulator-29:915c1f20be //Run emulator (headless instance)
228. 228 docker run -d -p 5555:5555 -p 5554:5554 -p 8554:8554 
 --privileged avitotech/android-emulator-29:915c1f20be //Run emulator (headless instance) //Connect to adb adb connect localhost:5555
229. 229 docker run -d -p 5555:5555 -p 5554:5554 -p 8554:8554 
 --privileged avitotech/android-emulator-29:915c1f20be //Run emulator (headless instance) //Connect to adb adb connect localhost:5555 //Kill emulators (after tests finished) docker kill $(docker ps -q)
 docker rm $(docker ps -a -q)
230. 230 Conclusion Headless emulators — pragmatic choice (faster and easy)
231. 231 Conclusion Headless emulators — pragmatic choice (faster and easy) It’s fine to start with real device (If your build agent is not powerful)
232. 232 How to deal with network
233. 233 Options Do nothing
234. 234 Do nothing (Real network) + E2E testing
235. 235 Do nothing (Real network) + E2E testing — Server is down (applies to development)
236. 236 Do nothing (Real network) + E2E testing — Server is down (applies to development) — Network issue
237. 237 Options Do nothing
238. 238 Options Dedicated stage server Do nothing
239. 239 Dedicated stage + A bit stablier
240. 240 Dedicated stage + A bit stablier — Network issue
241. 241 Dedicated stage + A bit stablier — Network issue — Not as same as a real server
242. 242 Dedicated stage + A bit stablier — Network issue — Not as same as a real server — It takes time to support
243. 243 Options Dedicated stage server Do nothing
244. 244 Options Mock/Stub it Dedicated stage server Do nothing
245. 245 Mock it / Stub it + Stable
246. 246 Mock it / Stub it + Stable + Still can catch the bugs
247. 247 Mock it / Stub it + Stable + Still can catch the bugs + Pragmatic
248. 248 Mock it / Stub it + Stable + Still can catch the bugs — Not fully E2E + Pragmatic — Requires some effort from engineers
249. 249 How to mock/stub?
250. 250 Network: How to mock/stub? MockWebServer
251. 251 Network: How to mock/stub? MockWebServer Custom Interceptor for testing
252. 252 Network: How to mock/stub? MockWebServer Custom Interceptor for testing DI: Dagger2/Hilt, Toothpick, Koin/Kodein
253. 253 Network: How to mock/stub? MockWebServer Custom Interceptor for testing DI: Dagger2/Hilt, Toothpick, Koin/Kodein Service Locator / different src
254. 254 MockWebServer Custom Interceptor for testing DI: Dagger2/Hilt, Toothpick, Koin/Kodein Service Locator Didn’t work properly
255. 255 Why: — A lot of requests
256. 256 Why: — A lot of requests //100+ in some features
257. 257 Why: — A lot of requests //100+ in some features — Long hours to mock
258. 258 Solution
259. 259 Solution 2 network modes: Record and Replay
260. 260 Idea: Record Build
261. 261 Build Run test Idea: Record
262. 262 Build Run test Real network Idea: Record
263. 263 Build Run test Real network Output ClassName#testName.yaml Idea: Record
264. 264 How does yaml le look like?
265. 265 - recorded: 2021-10-13T14:30:23.507Z request: method: GET uri: https://api.server.com/inbox headers: response: status: 200 headers: date: Wed, 13 Oct 2021 14:30:26 GMT request-id: 9JAJH8Y9JMRU server: MyServer body: '{"messages":[]}' Example
266. 266 Build Idea: Replay
267. 267 Build ClassName#testName.yaml Idea: Replay
268. 268 Build ClassName#testName.yaml Idea: Replay Run test
269. 269 Build ClassName#testName.yaml Idea: Replay Run test
270. 270 Modes usage Local machine: Playback, Record CI: Playback
271. 271 How to implement this?
272. 272 Solutions Wiremock https://github.com/wiremock/wiremock
273. 273 Solutions Wiremock + MITM under the hood
274. 274 Solutions Wiremock + MITM under the hood + Requests indexing
275. 275 Solutions Wiremock + MITM under the hood + Requests indexing — Support only 1 host
276. 276 Solutions Wiremock + MITM under the hood + Requests indexing — Support only 1 host — Not Android Friendly
277. 277 Solutions Wiremock OkReplay https://github.com/airbnb/okreplay
278. 278 Solutions Wiremock OkReplay + Readable report
279. 279 Solutions Wiremock OkReplay + Readable report + Interceptor-based solution
280. 280 Solutions Wiremock OkReplay + Readable report + Interceptor-based solution — Lack of indexing
281. 281 Solutions Wiremock OkReplay + Readable report + Interceptor-based solution — Lack of indexing — Constant execution time per each request
282. 282 Solutions Wiremock OkReplay + Readable report + Interceptor-based solution — Lack of indexing — Constant execution time per each request — Abandoned :(
283. 283 We’re implementing our own open-source solution for it
284. 284 Network: Conclusion — Mocked/Stubbed network it — the only way to have stable UI tests
285. 285 Network: Conclusion — Mocked/Stubbed network it — the only way to have stable UI tests — Small project: MockWebServer
286. 286 Network: Conclusion — Mocked/Stubbed network it — the only way to have stable UI tests — Small project: MockWebServer — Have a lot of requests to mock? — OkReplay
287. 287 Network: Conclusion — Mocked/Stubbed network it — the only way to have stable UI tests — Small project: MockWebServer — Have a lot of requests to mock? — OkReplay — Didn’t use OkHttp? — Wiremock
288. 288 Conclusions
289. 289 Android UI testing
290. 290 Android UI testing Expectations Reality
291. 291 Before you get started Do not even start if you want to try it just for fun
292. 292 Before you get started Do not even start if you want to try it just for fun You should have clear goal
293. 293 Before you get started You should have clear goal Which solves real problems Do not even start if you want to try it just for fun
294. 294 UI testing blog https://github.com/android-ui-testing/Cookbook/
295. 295 Thanks to Eugene Matsyuk Alexey Tvorogov Dmitriy Voronin Anton Malinskiy Dmitry Movchan Kaspresso adb server Kaspresso creator Marathon creator Avito runner creator Kaspresso adb server & a lot of researches
296. 296 Resources 1. https://github.com/KasperskyLab/Kaspresso 3. https://github.com/MarathonLabs/marathon 2. https://proandroiddev.com/autotests-on-android-the-entire-picture-51c07995fc2f ( 4. https://github.com/avito-tech/avito-android 5. https://github.com/Flank/flank 6. https://github.com/airbnb/okreplay 7. https://github.com/wiremock/wiremock
297. 297 Alexey Bykov, Android Software Engineer Presentation @nonewsss Thank you!