Tweet
Tweet

Slide 1

Slide 1 text

1 Where and how to run UI tests Alexey Bykov

Slide 2

Slide 2 text

2 Alexey Bykov Android Software Engineer @Revolut • 5 years in Android • EX Programm commette: AppsConf, Podlodka • EX: Android Academy MSK @nonewsss

Slide 3

Slide 3 text

3 1. Why UI testing?

Slide 4

Slide 4 text

4 Design:

Slide 5

Slide 5 text

5 Constitutes 1. Wheels 2. Seat 3. Handlebars 4. Pedal system 5. Frame

Slide 6

Slide 6 text

6 
 unit and integration tests All tested! 
 unit and integration tests 
 unit and integration tests 
 unit and integration tests 
 unit and integration tests

Slide 7

Slide 7 text

7 First implementation

Slide 8

Slide 8 text

8 2h later…

Slide 9

Slide 9 text

9 Finally!

Slide 10

Slide 10 text

10 Main question Is it ridable?

Slide 11

Slide 11 text

11 Check the junctions

Slide 12

Slide 12 text

12 Expected result — bike can ride

Slide 13

Slide 13 text

13 Road

Slide 14

Slide 14 text

14 com.bike.app

Slide 15

Slide 15 text

15 What if bicycle has been broken?

Slide 16

Slide 16 text

16 Call to repair

Slide 17

Slide 17 text

17 Hand it over to the repair

Slide 18

Slide 18 text

18 Wait

Slide 19

Slide 19 text

19 Each bug takes a long time

Slide 20

Slide 20 text

20 Mobile: Bug found 1. Fix/open PR, wait for the review, merge

Slide 21

Slide 21 text

21 Mobile: Bug found 1. Fix/open PR, wait for the review, merge 2. Test it

Slide 22

Slide 22 text

22 Mobile: Bug found 1. Fix/open PR, wait for the review, merge 2. Test it 3. Upload to the Google play

Slide 23

Slide 23 text

23 Mobile: Bug found 1. Fix/open PR, wait for the review, merge 2. Test it 3. Upload to the Google play 4. Wait for Google play review…

Slide 24

Slide 24 text

24 Mobile: Bug found 1. Fix/open PR, wait for the review, merge 2. Test it 3. Upload to the Google play 4. Wait for Google play review… 5. Wait for rolling out

Slide 25

Slide 25 text

25 Mobile: Bug found 1. Fix/open PR, wait for the review, merge 2. Test it 3. Upload to the Google play 4. Wait for Google play review… 5. Wait for rolling out 6. Wait for the users to download it

Slide 26

Slide 26 text

26 What affects quality?

Slide 27

Slide 27 text

27 What affects quality? 1. More engineers == More features

Slide 28

Slide 28 text

28 What affects quality? 1. More engineers == More features 2. Release frequency

Slide 29

Slide 29 text

29 UI testing — our need

Slide 30

Slide 30 text

30 UI testing — complicated

Slide 31

Slide 31 text

31

Slide 32

Slide 32 text

32 Writing Espresso UiAutomator Google provided Community KAutomator Kakao Kaspresso

Slide 33

Slide 33 text

33 Writing Running AndroidJUnitRunner Orchestrator Google provided Espresso UiAutomator Google provided Community KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Community Flank

Slide 34

Slide 34 text

34 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Real network Mock network Device Device condition

Slide 35

Slide 35 text

35 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes

Slide 36

Slide 36 text

36 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release pipeline Validation on PR When to write?

Slide 37

Slide 37 text

37 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure In fl ux db Grafana Found bugs dynamics

Slide 38

Slide 38 text

38 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure In fl ux db Grafana Found bugs dynamics

Slide 39

Slide 39 text

39 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Community Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Own Server Firebase GCloud Amazon Kubernetes Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure In fl ux db Grafana Found bugs dynamics Cloud Should work ne

Slide 40

Slide 40 text

40 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KUiAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure In fl ux db Grafana Found bugs dynamics This talk about

Slide 41

Slide 41 text

41 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KUiAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure In fl ux db Grafana Found bugs dynamics This talk about

Slide 42

Slide 42 text

42 Writing Running AndroidJUnitRunner Orchestrator Espresso UiAutomator Google provided Community KUiAutomator Kakao Kaspresso Spoon Marathon AvitoRunner Composer Fork Flank Google provided Whereto run Emulator Real device Docker container Device Real network Mock network Device condition Infrastructure Cloud Own Server Firebase GCloud Amazon Kubernetes Cloud Processes CI/CD Release Validation on PR When to write? Analytics Test report Allure In fl ux db Grafana Found bugs dynamics This talk about

Slide 43

Slide 43 text

43 2. Basics of UI tests running

Slide 44

Slide 44 text

44 Pff, I just need to press a green arrow

Slide 45

Slide 45 text

45 Let’s take a look

Slide 46

Slide 46 text

46 app.apk ./gradlew assembleDebug

Slide 47

Slide 47 text

47 app.apk ./gradlew assembleDebug instrumented.apk ./gradlew assembleDebugAndroidTest @Test

Slide 48

Slide 48 text

48 app.apk adb install instrumented.apk @Test

Slide 49

Slide 49 text

49 app.apk adb install instrumented.apk @Test launch tests

Slide 50

Slide 50 text

50 app.apk adb install instrumented.apk @Test launch tests PASSED

Slide 51

Slide 51 text

51 app.apk adb install instrumented.apk @Test launch tests PASSED Generate report Let’s focus

Slide 52

Slide 52 text

52 Android JUnit Runner

Slide 53

Slide 53 text

53 Android JUnit Runner Input: tests output: test result TestMethod#testName PASSED

Slide 54

Slide 54 text

54 adb shell am instrument -w -m -e debug false -e class 'com.alexbykov.myapplication.ExampleInstrumentedTest#myTest' com.alexbykov.myapplication.test/ androidx.test.runner.AndroidJUnitRunner To launch tests:

Slide 55

Slide 55 text

55 adb shell am instrument -w -m -e debug false -e class 'com.alexbykov.myapplication.ExampleInstrumentedTest#myTest' com.alexbykov.myapplication.test/ androidx.test.runner.AndroidJUnitRunner To launch tests:

Slide 56

Slide 56 text

56 Full picture ./gradlew assembleDebug ./gradlew assembleDebugAndroidTest // build

Slide 57

Slide 57 text

57 ./gradlew assembleDebug ./gradlew assembleDebugAndroidTest Full picture adb install debug.apk adb install instrumented.apk // build // install

Slide 58

Slide 58 text

58 adb shell am instrument -w -m -e debug false -e package 'com.alexbykov.myapplication' com.alexbykov.myapplication.test/ androidx.test.runner.AndroidJUnitRunner ./gradlew assembleDebug ./gradlew assembleDebugAndroidTest adb install debug.apk adb install instrumented.apk Full picture // install // build // run

Slide 59

Slide 59 text

59 2.1 How to deal with state?

Slide 60

Slide 60 text

60 Test1

Slide 61

Slide 61 text

61 Test1

Slide 62

Slide 62 text

62 PASSED Test1

Slide 63

Slide 63 text

63 Test1 data db, cache, files PASSED Test1

Slide 64

Slide 64 text

64 PASSED Test1 data db, cache, files Test1 Test2

Slide 65

Slide 65 text

65 PASSED Test1 data db, cache, files Test1 Test2 Test2 FAILED

Slide 66

Slide 66 text

66 Solution: Isolate each test

Slide 67

Slide 67 text

67 Clearing 1. Within process 2. Clear package data

Slide 68

Slide 68 text

68 Within process clearing 1. LogoutComponent from the code @Before fun setUp() { DI.provideLogoutInteractor().logout() }

Slide 69

Slide 69 text

69 It’s bottleneck @Before fun setUp() { DI.provideLogoutInteractor().logout() } May be broken

Slide 70

Slide 70 text

70 All tests will be failed ¯\_(ツ)_/¯

Slide 71

Slide 71 text

71 Within process clearing 1. LogoutComponent from the code 2. Internal storage clearing

Slide 72

Slide 72 text

72 Internal storage clearing All db saved: /data/data/packagename/databases/

Slide 73

Slide 73 text

73 Internal storage clearing All db saved: /data/data/packagename/databases/ @get:Rule val clearDatabasesRule = ClearDatabasesRule() @get:Rule val clearFilesRule = ClearFilesRule() @get:Rule val clearPreferencesRule = ClearPreferencesRule()

Slide 74

Slide 74 text

74 Internal storage clearing All db saved: /data/data/packagename/databases/ @get:Rule val clearDatabasesRule = ClearDatabasesRule() @get:Rule val clearFilesRule = ClearFilesRule() @get:Rule val clearPreferencesRule = ClearPreferencesRule() https://github.com/AdevintaSpain/Barista

Slide 75

Slide 75 text

75 It doesn’t solve all issues !

Slide 76

Slide 76 text

76 It doesn’t solve all issues 1. You may have Runtime cache

Slide 77

Slide 77 text

77 It doesn’t solve all issues 1. You may have Runtime cache 2. App/Test process may be crashed ¯\_(ツ)_/¯

Slide 78

Slide 78 text

78 Within process: Conclusions + Fast implementation + Fast execution

Slide 79

Slide 79 text

79 Within process: Conclusions - Bottleneck - Don’t have any guarantee about clearing - App/Test process might be crashed + Fast implementation + Fast execution

Slide 80

Slide 80 text

80 Clear package data

Slide 81

Slide 81 text

81 Idea

Slide 82

Slide 82 text

82 First thought adb shell am instrument -w -m -e debug false -e class ‘….ExampleInstrumentedTest#test1,….ExampleInstrumentedTest#tes t2’ com.alexbykov.myapplication.test/ androidx.test.runner.AndroidJUnitRunner

Slide 83

Slide 83 text

83 Launch rst test adb shell am instrument …. ExampleInstrumentedTest#test1

Slide 84

Slide 84 text

84 Clear adb shell am instrument …. ExampleInstrumentedTest#test1 adb shell pm clear …

Slide 85

Slide 85 text

85 Launch second test adb shell am instrument …. ExampleInstrumentedTest#test1 adb shell pm clear … adb shell am instrument …. ExampleInstrumentedTest#test2

Slide 86

Slide 86 text

86 Clear again adb shell am instrument …. ExampleInstrumentedTest#test1 adb shell pm clear … adb shell am instrument …. ExampleInstrumentedTest#test2 adb shell pm clear …

Slide 87

Slide 87 text

87 Orchestrator

Slide 88

Slide 88 text

88 app.apk instrumented.apk

Slide 89

Slide 89 text

89 app.apk instrumented.apk +1 apk orchestrator.apk

Slide 90

Slide 90 text

90 app.apk instrumented.apk Start test orchestrator.apk AndroidJunitRunner am instrument

Slide 91

Slide 91 text

91 app.apk instrumented.apk Clear orchestrator.apk AndroidJunitRunner am instrument pm clear data data

Slide 92

Slide 92 text

92 app.apk instrumented.apk Clear orchestrator.apk AndroidJunitRunner am instrument pm clear am instrument

Slide 93

Slide 93 text

93 How does orchestrator execute commands?

Slide 94

Slide 94 text

94 app.apk instrumented.apk Test-services orchestrator.apk test-services.apk shell commands

Slide 95

Slide 95 text

95 Package clear: Conclusions + Isolated environment

Slide 96

Slide 96 text

96 Package clear: Conclusions - It takes time + Isolated environment

Slide 97

Slide 97 text

97 It takes time: 300 tests

Slide 98

Slide 98 text

98 It takes time: 300 tests Each gap is 7-12 seconds

Slide 99

Slide 99 text

99 Package clear: Conclusions - It takes time - It’s complicated + Isolated environment

Slide 100

Slide 100 text

100 It’s complicated 2 more apk you need to install orchestrator.apk services.apk

Slide 101

Slide 101 text

101 It’s complicated 2 more apk you need to install orchestrator.apk services.apk Mobile device less reliable than host PC

Slide 102

Slide 102 text

102 3. Problems with UI tests run

Slide 103

Slide 103 text

103 Problems 1. Flakiness

Slide 104

Slide 104 text

104 Flakiness tests will always be

Slide 105

Slide 105 text

105 1. Reason of akiness Production code // toasts, custom views, http-requests, databases, animations

Slide 106

Slide 106 text

106 Production code Test code // Simulate user actions 1. Reason of akiness

Slide 107

Slide 107 text

107 Production code Test code Device problems // Connection, notifications, or it’s just stuck 1. Reason of akiness

Slide 108

Slide 108 text

108 Production code Test code Device problems Infrastructure problems // I/O, CPU, Memory 1. Reason of akiness

Slide 109

Slide 109 text

109 Problems 1. Flakiness

Slide 110

Slide 110 text

110 Problems 1. Flakiness 2. Speed

Slide 111

Slide 111 text

111 2. Speed problems 1. Auto-scalling

Slide 112

Slide 112 text

112 Auto-scalling Test Run

Slide 113

Slide 113 text

113 Auto-scalling Test Run Test Run 2

Slide 114

Slide 114 text

114 Test Run: Finished Test Run Test Run 2

Slide 115

Slide 115 text

115 Connect all emulators to 2 Test Run 2

Slide 116

Slide 116 text

116 1. Auto-scalling 2. Build agents overloading 2. Speed problems

Slide 117

Slide 117 text

117 Problems 1. Flakiness 2. Speed

Slide 118

Slide 118 text

118 1. Flakiness 2. Speed 3. Tooling Problems

Slide 119

Slide 119 text

119 3. Tooling 1. Impossible to validate flakiness

Slide 120

Slide 120 text

120 3. Tooling 1. Impossible to validate flakiness 2. Poor test report

Slide 121

Slide 121 text

121 AndroidJUnitRunner Can’t help us with that

Slide 122

Slide 122 text

122 4. Choose: Test runner

Slide 123

Slide 123 text

123 Flakiness protection levels Test code

Slide 124

Slide 124 text

124 Example1: Kaspresso fun invokeFlakySafely( params: FlakySafetyParams, failureMessage: String? = null, action: () -> T ): T { var cachedError: Throwable val startTime = System.currentTimeMillis() do { try { return action.invoke() } catch (error: Throwable) { if (error.isAllowed(params.allowedExceptions)) { cachedError = error lock.withLock { Timer().schedule(params.intervalMs) { lock.withLock { condition.signalAll() } } condition.await() } } else { throw error } } } while (System.currentTimeMillis() - startTime <= params.timeoutMs) throw cachedError.withMessage(failureMessage) }

Slide 125

Slide 125 text

125 fun invokeFlakySafely( params: FlakySafetyParams, failureMessage: String? = null, action: () -> T ): T { var cachedError: Throwable val startTime = System.currentTimeMillis() do { try { return action.invoke() } catch (error: Throwable) { … } } while (System.currentTimeMillis() - startTime <= params.timeoutMs) throw cachedError.withMessage(failureMessage) } Trying execute actions

Slide 126

Slide 126 text

126 fun invokeFlakySafely( params: FlakySafetyParams, failureMessage: String? = null, action: () -> T ): T { var cachedError: Throwable val startTime = System.currentTimeMillis() do { try { return action.invoke() } catch (error: Throwable) { … } } while (System.currentTimeMillis() - startTime <= params.timeoutMs) throw cachedError.withMessage(failureMessage) } In a loop

Slide 127

Slide 127 text

127 Example2: Noti cation

Slide 128

Slide 128 text

128 Example2: Noti cation class CloseNotificationsRule : ExternalResource() { override fun before() { UiDevice .getInstance(InstrumentationRegistry.getInstrumentation()) .pressHome() } }

Slide 129

Slide 129 text

129 Example2: Noti cation class CloseNotificationsRule : ExternalResource() { override fun before() { UiDevice .getInstance(InstrumentationRegistry.getInstrumentation()) .pressHome() } } @get:Rule val closeNotification = CloseNotificationsRule()

Slide 130

Slide 130 text

130 Flakiness protection levels Test code

Slide 131

Slide 131 text

131 Flakiness protection levels Test code Device

Slide 132

Slide 132 text

132 Example: Disabling animations adb shell "settings put global window_animation_scale 0.0" adb shell "settings put global transition_animation_scale 0.0" adb shell "settings put global animator_duration_scale 0.0"

Slide 133

Slide 133 text

133 Example: Other adb shell "settings put global window_animation_scale 0.0" adb shell "settings put global transition_animation_scale 0.0" adb shell "settings put global animator_duration_scale 0.0" adb shell "settings put system screen_off_timeout 2147483647" adb shell "settings put secure long_press_timeout 1500"

Slide 134

Slide 134 text

134 Flakiness protection levels Test code Device

Slide 135

Slide 135 text

135 Flakiness protection levels Test code Device Test Runner

Slide 136

Slide 136 text

136 Test runner — last layer of flakiness protection ONE OF THE GOALS

Slide 137

Slide 137 text

137 Open source test runners Marathon Avito Runner Spoon Fork Composer Flank

Slide 138

Slide 138 text

138 Open source test runners Marathon Avito Runner Spoon Fork Composer Deprecated Flank

Slide 139

Slide 139 text

139 Marathon vision Just connect Devices to adb, I’ll do the job https://github.com/MarathonLabs/marathon

Slide 140

Slide 140 text

140 Marathon + Retries

Slide 141

Slide 141 text

141 Marathon + Retries retryStrategy: type: "fixed-quota" totalAllowedRetryQuota: 100 retryPerTestQuota: 3

Slide 142

Slide 142 text

142 Retry Test1 Test2

Slide 143

Slide 143 text

143 Passed! Test1 Test2

Slide 144

Slide 144 text

144 Failed :( Test1 Test2

Slide 145

Slide 145 text

145 Retry on another device Test1 Test2

Slide 146

Slide 146 text

146 Marathon + Retries + Flakiness strategy

Slide 147

Slide 147 text

147 Idea: Flakiness strategy — Calculate probability of passing — Launch flakiness test in parallel before it’s failed

Slide 148

Slide 148 text

148 Flakiness strategy Test1: Flaky in 50% //0.5 — probability of passing

Slide 149

Slide 149 text

149 Flakiness strategy Test1: Flaky in 50% //0.5 — probability of passing We want minimum: 80% //0.8 — probability of passing

Slide 150

Slide 150 text

150 Flakiness strategy Test1: Flaky in 50% //0.5 — probability of passing We want minimum: 80% //0.8 — probability of passing Max parallel run: 3 //because more can affect speed of run

Slide 151

Slide 151 text

151 Probability 0.5 = 0.125 Test1: Flaky in 50% We want minimum: 80% Max parallel run: 3 //probability of all failed 3

Slide 152

Slide 152 text

152 Probability 0.5 = 0.125 1 - 0.125 = 0.875 > 0.8 Test1: Flaky in 50% We want minimum: 80% Max parallel run: 3 //probability of all failed //Test will be executed 3 times in parallel 3

Slide 153

Slide 153 text

153 Flaky test (50% —> 80%) Test Test Test

Slide 154

Slide 154 text

154 analyticsConfiguration: graphite: host: "influx.svc.cluster.local" port: "8080" prefix: "prf" To make it work

Slide 155

Slide 155 text

155 Marathon + Retries + Flakiness strategy

Slide 156

Slide 156 text

156 Marathon + Retries + Flakiness strategy + Good report and allure support

Slide 157

Slide 157 text

157

Slide 158

Slide 158 text

158 Failed tests: Video/Logs

Slide 159

Slide 159 text

159 However default report doesn’t support retries

Slide 160

Slide 160 text

160 But we can do that via Allure allure generate report marathon_output/allure-results -o marathon_output/ allure-report --clean

Slide 161

Slide 161 text

161 Allure retries

Slide 162

Slide 162 text

162 Marathon + Retries + Flakiness strategy + Good report and allure support

Slide 163

Slide 163 text

163 Marathon + Retries + Flakiness strategy + Good report and allure support + Isolated environment applicationPmClear: true testApplicationPmClear: true

Slide 164

Slide 164 text

164 Marathon + Retries + Flakiness strategy + Good report and allure support + Isolated environment + ADB replacement

Slide 165

Slide 165 text

165 Replace ADB? WAT?

Slide 166

Slide 166 text

166 Where do we use ADB?

Slide 167

Slide 167 text

167 ADB in instrumented tests Install apps Clearing Pulling files Record video

Slide 168

Slide 168 text

168 Problem Most of the commands create a new thread under the hood

Slide 169

Slide 169 text

169

Slide 170

Slide 170 text

170 Solution: ADAM

Slide 171

Slide 171 text

171 Solution: ADAM Uses Kotlin coroutines under the hood

Slide 172

Slide 172 text

172 Adam vs ADB ADB: 300 tests T: 32 min I/O: 22% CPU: 48% Adam: 300 tests T: 26 min I/O: 19% CPU: 47 %

Slide 173

Slide 173 text

173 Marathon + Retries + Flakiness strategy + Good report and allure support + Isolated environment + ADB replacement

Slide 174

Slide 174 text

174 Marathon + Retries + Flakiness strategy + Good report and allure support + Isolated environment + ADB replacement + Flakiness validation strictMode: true shardingStrategy: type: "count" count: 100

Slide 175

Slide 175 text

175 Marathon + Retries + Flakiness strategy + Good report and allure support + Isolated environment + ADB replacement + Flakiness validation + Cross platform (iOS support)

Slide 176

Slide 176 text

176 Open source test runners Marathon

Slide 177

Slide 177 text

177 Open source test runners Marathon Avito Runner

Slide 178

Slide 178 text

178 Avito runner vision Just connect Kubernetes to me, I’ll do the job https://avito-tech.github.io/avito-android/test_runner/TestRunner/

Slide 179

Slide 179 text

179 Test Run: Finished Test Run Test Run 2

Slide 180

Slide 180 text

180 Connect all emulators to 2 Test Run 2

Slide 181

Slide 181 text

181 Marathon Possible, but you need 
 to connect devices to adb by yourself Autoscalling

Slide 182

Slide 182 text

182 Device farm

Slide 183

Slide 183 text

183 Device state idle busy Device farm

Slide 184

Slide 184 text

184 Build agent

Slide 185

Slide 185 text

185 ADB connect Build agent Device farm

Slide 186

Slide 186 text

186 Avito runner I’ll do it for you! Autoscalling

Slide 187

Slide 187 text

187 Difference Marathon Avito Test Runner - Auto-scalling - Adam - Flakiness strategy - CLI - Doesn’t support iOS

Slide 188

Slide 188 text

188 Difference Marathon Avito Test Runner - Auto-scalling - Adam - Flakiness strategy - CLI - Doesn’t support iOS - It’s not ready for open source

Slide 189

Slide 189 text

189 Open source test runners Marathon Avito Runner

Slide 190

Slide 190 text

190 Open source test runners Marathon Avito Runner Flank

Slide 191

Slide 191 text

191 Flank vision https://flank.github.io/flank/ I’ll make your life easier to work with Firebase Test Lab https://firebase.google.com/products/test-lab

Slide 192

Slide 192 text

192 Flank + Don’t need to care about where to run — Firebase Test Lab — paid service + Crossplatform

Slide 193

Slide 193 text

193 Which one do I need to choose?

Slide 194

Slide 194 text

194 It depends :)

Slide 195

Slide 195 text

195 It depends :) Product? Team size? Company? Scalability? How often do you need to run? How many tests do you have? How often your codebase changed?

Slide 196

Slide 196 text

196 That’s on you Marathon, Avito Runner: + invest your time to build infrastructure + invest your time to support

Slide 197

Slide 197 text

197 That’s on you Marathon, Avito Runner: + invest your time to support + invest your time to build infrastructure Flank + Firebase test lab: + You have to pay always

Slide 198

Slide 198 text

198 Marathon — the most powerful

Slide 199

Slide 199 text

199 Marathon — the most powerful For any team.

Slide 200

Slide 200 text

200 Real device vs Emulator

Slide 201

Slide 201 text

201 Real device

Slide 202

Slide 202 text

202 Real device + Real condition

Slide 203

Slide 203 text

203 Real device + Real condition + Doesn’t consume CI resources

Slide 204

Slide 204 text

204 Real device + Real condition + Doesn’t consume CI resources — Room with special condition

Slide 205

Slide 205 text

205 Real device + Real condition + Doesn’t consume CI resources — Room with special condition — Breaks often

Slide 206

Slide 206 text

206 Emulator

Slide 207

Slide 207 text

207 + Easy manageable Emulator

Slide 208

Slide 208 text

208 + Easy manageable + Easy configurable Emulator

Slide 209

Slide 209 text

209 + Easy manageable + Easy configurable + Work faster Emulator

Slide 210

Slide 210 text

210 Works faster? Before: Real devices

Slide 211

Slide 211 text

211 Works faster? Before: Real devices Average: 50 seconds

Slide 212

Slide 212 text

212 Works faster? Before: Real devices Average: 50 seconds After: Emulators Average: 30 seconds

Slide 213

Slide 213 text

213 Why Fresh emulator instance per each test suit

Slide 214

Slide 214 text

214 Why Fresh emulator instance per each test suit Only your application installed

Slide 215

Slide 215 text

215 Why Fresh emulator instance per each test suit Only your application installed Special configuration

Slide 216

Slide 216 text

216 PlayStore.enabled=false abi.type=x86_64 avd.ini.encoding=UTF-8 hw.cpu.arch=x86_64 hw.cpu.ncore=2 hw.ramSize=2048 hw.lcd.density=120 hw.lcd.width=320 hw.lcd.height=480 hw.audioInput=no hw.audioOutput=no hw.accelerometer=no hw.gyroscope=no hw.dPad=no hw.mainKeys=yes hw.keyboard=no hw.sensors.proximity=no hw.sensors.magnetic_field=no hw.sensors.orientation=no hw.sensors.temperature=no hw.sensors.light=no hw.sensors.pressure=no hw.sensors.humidity=no hw.sensors.magnetic_field_uncalibrated=no hw.sensors.gyroscope_uncalibrated=no image.sysdir.1=system-images/android-29/google_apis/x86_64/ tag.display=Google APIs tag.id=google_apis skin.dynamic=yes skin.name=320x480 disk.dataPartition.size=8G Special configuration

Slide 217

Slide 217 text

217 PlayStore.enabled=false abi.type=x86_64 avd.ini.encoding=UTF-8 hw.cpu.arch=x86_64 hw.cpu.ncore=2 hw.ramSize=2048 hw.lcd.density=120 hw.lcd.width=320 hw.lcd.height=480 hw.audioInput=no hw.audioOutput=no hw.accelerometer=no hw.gyroscope=no hw.dPad=no hw.mainKeys=yes hw.keyboard=no hw.sensors.proximity=no hw.sensors.magnetic_field=no hw.sensors.orientation=no hw.sensors.temperature=no hw.sensors.light=no hw.sensors.pressure=no hw.sensors.humidity=no hw.sensors.magnetic_field_uncalibrated=no hw.sensors.gyroscope_uncalibrated=no image.sysdir.1=system-images/android-29/google_apis/x86_64/ tag.display=Google APIs tag.id=google_apis skin.dynamic=yes skin.name=320x480 disk.dataPartition.size=8G Special configuration

Slide 218

Slide 218 text

218 PlayStore.enabled=false abi.type=x86_64 avd.ini.encoding=UTF-8 hw.cpu.arch=x86_64 hw.cpu.ncore=2 hw.ramSize=2048 hw.lcd.density=120 hw.lcd.width=320 hw.lcd.height=480 hw.audioInput=no hw.audioOutput=no hw.accelerometer=no hw.gyroscope=no hw.dPad=no hw.mainKeys=yes hw.keyboard=no hw.sensors.proximity=no hw.sensors.magnetic_field=no hw.sensors.orientation=no hw.sensors.temperature=no hw.sensors.light=no hw.sensors.pressure=no hw.sensors.humidity=no Special configuration

Slide 219

Slide 219 text

219 Emulator Emulator + Easy manageable + Easy configurable + Work faster

Slide 220

Slide 220 text

220 Emulator Emulator + Easy manageable + Easy configurable + Work faster — Consume CI resources

Slide 221

Slide 221 text

221 Emulator Emulator + Easy manageable + Easy configurable + Work faster — Consume CI resources — It’s not a real device

Slide 222

Slide 222 text

222 How to build emulator?

Slide 223

Slide 223 text

223 How to build emulator? Programatically (AVD Manager)

Slide 224

Slide 224 text

224 Programatically sample shorturl.at/gkEY1 46 lines of code :(

Slide 225

Slide 225 text

225 How to build emulator? Programatically (AVD Manager) Docker image

Slide 226

Slide 226 text

226 Docker emulators Google (official) shorturl.at/azNY4 Avito shorturl.at/fmHKN Agoda shorturl.at/xGJKY

Slide 227

Slide 227 text

227 docker run -d -p 5555:5555 -p 5554:5554 -p 8554:8554 
 --privileged avitotech/android-emulator-29:915c1f20be //Run emulator (headless instance)

Slide 228

Slide 228 text

228 docker run -d -p 5555:5555 -p 5554:5554 -p 8554:8554 
 --privileged avitotech/android-emulator-29:915c1f20be //Run emulator (headless instance) //Connect to adb adb connect localhost:5555

Slide 229

Slide 229 text

229 docker run -d -p 5555:5555 -p 5554:5554 -p 8554:8554 
 --privileged avitotech/android-emulator-29:915c1f20be //Run emulator (headless instance) //Connect to adb adb connect localhost:5555 //Kill emulators (after tests finished) docker kill $(docker ps -q)
 docker rm $(docker ps -a -q)

Slide 230

Slide 230 text

230 Conclusion Headless emulators — pragmatic choice (faster and easy)

Slide 231

Slide 231 text

231 Conclusion Headless emulators — pragmatic choice (faster and easy) It’s fine to start with real device (If your build agent is not powerful)

Slide 232

Slide 232 text

232 How to deal with network

Slide 233

Slide 233 text

233 Options Do nothing

Slide 234

Slide 234 text

234 Do nothing (Real network) + E2E testing

Slide 235

Slide 235 text

235 Do nothing (Real network) + E2E testing — Server is down (applies to development)

Slide 236

Slide 236 text

236 Do nothing (Real network) + E2E testing — Server is down (applies to development) — Network issue

Slide 237

Slide 237 text

237 Options Do nothing

Slide 238

Slide 238 text

238 Options Dedicated stage server Do nothing

Slide 239

Slide 239 text

239 Dedicated stage + A bit stablier

Slide 240

Slide 240 text

240 Dedicated stage + A bit stablier — Network issue

Slide 241

Slide 241 text

241 Dedicated stage + A bit stablier — Network issue — Not as same as a real server

Slide 242

Slide 242 text

242 Dedicated stage + A bit stablier — Network issue — Not as same as a real server — It takes time to support

Slide 243

Slide 243 text

243 Options Dedicated stage server Do nothing

Slide 244

Slide 244 text

244 Options Mock/Stub it Dedicated stage server Do nothing

Slide 245

Slide 245 text

245 Mock it / Stub it + Stable

Slide 246

Slide 246 text

246 Mock it / Stub it + Stable + Still can catch the bugs

Slide 247

Slide 247 text

247 Mock it / Stub it + Stable + Still can catch the bugs + Pragmatic

Slide 248

Slide 248 text

248 Mock it / Stub it + Stable + Still can catch the bugs — Not fully E2E + Pragmatic — Requires some effort from engineers

Slide 249

Slide 249 text

249 How to mock/stub?

Slide 250

Slide 250 text

250 Network: How to mock/stub? MockWebServer

Slide 251

Slide 251 text

251 Network: How to mock/stub? MockWebServer Custom Interceptor for testing

Slide 252

Slide 252 text

252 Network: How to mock/stub? MockWebServer Custom Interceptor for testing DI: Dagger2/Hilt, Toothpick, Koin/Kodein

Slide 253

Slide 253 text

253 Network: How to mock/stub? MockWebServer Custom Interceptor for testing DI: Dagger2/Hilt, Toothpick, Koin/Kodein Service Locator / different src

Slide 254

Slide 254 text

254 MockWebServer Custom Interceptor for testing DI: Dagger2/Hilt, Toothpick, Koin/Kodein Service Locator Didn’t work properly

Slide 255

Slide 255 text

255 Why: — A lot of requests

Slide 256

Slide 256 text

256 Why: — A lot of requests //100+ in some features

Slide 257

Slide 257 text

257 Why: — A lot of requests //100+ in some features — Long hours to mock

Slide 258

Slide 258 text

258 Solution

Slide 259

Slide 259 text

259 Solution 2 network modes: Record and Replay

Slide 260

Slide 260 text

260 Idea: Record Build

Slide 261

Slide 261 text

261 Build Run test Idea: Record

Slide 262

Slide 262 text

262 Build Run test Real network Idea: Record

Slide 263

Slide 263 text

263 Build Run test Real network Output ClassName#testName.yaml Idea: Record

Slide 264

Slide 264 text

264 How does yaml le look like?

Slide 265

Slide 265 text

265 - recorded: 2021-10-13T14:30:23.507Z request: method: GET uri: https://api.server.com/inbox headers: response: status: 200 headers: date: Wed, 13 Oct 2021 14:30:26 GMT request-id: 9JAJH8Y9JMRU server: MyServer body: '{"messages":[]}' Example

Slide 266

Slide 266 text

266 Build Idea: Replay

Slide 267

Slide 267 text

267 Build ClassName#testName.yaml Idea: Replay

Slide 268

Slide 268 text

268 Build ClassName#testName.yaml Idea: Replay Run test

Slide 269

Slide 269 text

269 Build ClassName#testName.yaml Idea: Replay Run test

Slide 270

Slide 270 text

270 Modes usage Local machine: Playback, Record CI: Playback

Slide 271

Slide 271 text

271 How to implement this?

Slide 272

Slide 272 text

272 Solutions Wiremock https://github.com/wiremock/wiremock

Slide 273

Slide 273 text

273 Solutions Wiremock + MITM under the hood

Slide 274

Slide 274 text

274 Solutions Wiremock + MITM under the hood + Requests indexing

Slide 275

Slide 275 text

275 Solutions Wiremock + MITM under the hood + Requests indexing — Support only 1 host

Slide 276

Slide 276 text

276 Solutions Wiremock + MITM under the hood + Requests indexing — Support only 1 host — Not Android Friendly

Slide 277

Slide 277 text

277 Solutions Wiremock OkReplay https://github.com/airbnb/okreplay

Slide 278

Slide 278 text

278 Solutions Wiremock OkReplay + Readable report

Slide 279

Slide 279 text

279 Solutions Wiremock OkReplay + Readable report + Interceptor-based solution

Slide 280

Slide 280 text

280 Solutions Wiremock OkReplay + Readable report + Interceptor-based solution — Lack of indexing

Slide 281

Slide 281 text

281 Solutions Wiremock OkReplay + Readable report + Interceptor-based solution — Lack of indexing — Constant execution time per each request

Slide 282

Slide 282 text

282 Solutions Wiremock OkReplay + Readable report + Interceptor-based solution — Lack of indexing — Constant execution time per each request — Abandoned :(

Slide 283

Slide 283 text

283 We’re implementing our own open-source solution for it

Slide 284

Slide 284 text

284 Network: Conclusion — Mocked/Stubbed network it — the only way to have stable UI tests

Slide 285

Slide 285 text

285 Network: Conclusion — Mocked/Stubbed network it — the only way to have stable UI tests — Small project: MockWebServer

Slide 286

Slide 286 text

286 Network: Conclusion — Mocked/Stubbed network it — the only way to have stable UI tests — Small project: MockWebServer — Have a lot of requests to mock? — OkReplay

Slide 287

Slide 287 text

287 Network: Conclusion — Mocked/Stubbed network it — the only way to have stable UI tests — Small project: MockWebServer — Have a lot of requests to mock? — OkReplay — Didn’t use OkHttp? — Wiremock

Slide 288

Slide 288 text

288 Conclusions

Slide 289

Slide 289 text

289 Android UI testing

Slide 290

Slide 290 text

290 Android UI testing Expectations Reality

Slide 291

Slide 291 text

291 Before you get started Do not even start if you want to try it just for fun

Slide 292

Slide 292 text

292 Before you get started Do not even start if you want to try it just for fun You should have clear goal

Slide 293

Slide 293 text

293 Before you get started You should have clear goal Which solves real problems Do not even start if you want to try it just for fun

Slide 294

Slide 294 text

294 UI testing blog https://github.com/android-ui-testing/Cookbook/

Slide 295

Slide 295 text

295 Thanks to Eugene Matsyuk Alexey Tvorogov Dmitriy Voronin Anton Malinskiy Dmitry Movchan Kaspresso adb server Kaspresso creator Marathon creator Avito runner creator Kaspresso adb server & a lot of researches

Slide 296

Slide 296 text

296 Resources 1. https://github.com/KasperskyLab/Kaspresso 3. https://github.com/MarathonLabs/marathon 2. https://proandroiddev.com/autotests-on-android-the-entire-picture-51c07995fc2f ( 4. https://github.com/avito-tech/avito-android 5. https://github.com/Flank/flank 6. https://github.com/airbnb/okreplay 7. https://github.com/wiremock/wiremock

Slide 297

Slide 297 text

297 Alexey Bykov, Android Software Engineer Presentation @nonewsss Thank you!