[Mobius] Layers of testing in iOS applications

We want to take you on the journey through different layers of tests in an iOS app that’s used by more than half million unique users per month. Get to know the way we test it, step by step, layer by layer. We start with advanced code static analysis, continue with mutation, snapshot and functional tests. We carry on with continuous delivery, A/B tests and usability tests. We want to share with you which tools we are using and what testing practices we believe in. This is a true story driven by problems we experience at work, presented from tester’s and developer’s perspective.

Transcript

1. Layers of testing in iOS applications Ewa Ludwiczak & Aleksander Grzyb Mobius 2018 Piter
2. Ewa Ludwiczak Senior Software Test Engineer ewa.ludwiczak@allegro.pl @ewabielskapoz Aleksander Grzyb iOS Developer aleksander.grzyb@allegro.pl @aleksandergrzyb 3
3. 4 Allegro application in Poland
4. 5 Allegro application in Poland
5. 6 Allegro application in Poland
6. 7 Static analysis Mutation testing Snapshot tests Functional tests Continuous delivery Phased release A/B tests Usability tests Problem - test solution
7. 8 1. Complex code Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
8. 9 1. Complex code Static analysis Problem - test solution
9. 10 Problem 1:
10. 11 Problem 1: Complex code
12. 13 Problem 1: Complex code Solution: SwiftLint https://github.com/realm/SwiftLint
13. 14 Problem 1: Complex code Solution: SwiftLint https://github.com/realm/SwiftLint
14. 15 Problem 1: Complex code Solution: SwiftLint OFFER LISTING SEARCH CART ... HOME ALLEGRO
15. 16 Problem 1: Complex code Solution: SwiftLint SEARCH
16. 17 Problem 1: Complex code Solution: SwiftLint too short constant name
17. 18 Problem 1: Complex code Solution: SwiftLint too long variable name too short constant name
18. 19 Problem 1: Complex code Solution: SwiftLint too long variable name too short constant name
19. 20 Problem 1: Complex code Solution: SwiftLint SEARCH
20. 21 Problem 1: Complex code SwiftFormat + SwiftLint https://github.com/nicklockwood/SwiftFormat
21. 22 Problem 1: Complex code SwiftFormat + SwiftLint OBSERVED LISTING OFFER CART ... HOME ALLEGRO
22. 23 Problem 1: Complex code SwiftFormat + SwiftLint
23. respect modular architecture 24 Problem 1: Complex code Solution: SwiftLint remarks
24. respect modular architecture run locally after each build 25 Problem 1: Complex code Solution: SwiftLint remarks
25. respect modular architecture format obvious rules, lint less trivial run locally after each build 26 Problem 1: Complex code Solution: SwiftLint remarks
26. respect modular architecture format obvious rules, lint less trivial run locally after each build 27 Problem 1: Complex code Solution: SwiftLint remarks apply to production code and tests
27. respect modular architecture format obvious rules, lint less trivial use the same versions of formatter & linter run locally after each build apply to production code and tests 28 Problem 1: Complex code Solution: SwiftLint remarks
28. 29 1. Complex code Static analysis 2. Evergreen unit test Mutation testing Problem - test solution
29. 30 http://cleanswifter.com/asynchronous-ios-unit-test-tutorial/
30. 31 http://cleanswifter.com/asynchronous-ios-unit-test-tutorial/
33. 34 Problem 2: Evergreen unit tests
34. 35 Problem 2: Evergreen unit tests
35. 36 Problem 2: Evergreen unit tests Solution: Mutation testing function
36. 37 Problem 2: Evergreen unit tests Solution: Mutation testing function mutant mutation
37. 38 Problem 2: Evergreen unit tests Solution: Mutation testing function mutant mutation
38. 39 Problem 2: Evergreen unit tests Solution: Mutation testing change function logic
39. 40 Problem 2: Evergreen unit tests Solution: Mutation testing change function logic
40. 41 Problem 2: Evergreen unit tests Solution: Mutation testing negate conditions
41. 42 Problem 2: Evergreen unit tests Solution: Mutation testing change mathematical operators
42. 43 Problem 2: Evergreen unit tests Solution: Mutation testing remarks code coverage says little about test quality
43. 44 Problem 2: Evergreen unit tests Solution: Mutation testing remarks code coverage says little about test quality use manual mutation in non-trivial test cases
44. code coverage says little about test quality not all mutations add value use manual mutation in non-trivial test cases 45 Problem 2: Evergreen unit tests Solution: Mutation testing remarks
45. code coverage says little about test quality not all mutations add value use manual mutation in non-trivial test cases 46 Problem 2: Evergreen unit tests Solution: Mutation testing remarks automated mutation testing Problem 2: Evergreen unit tests Solution: Mutation testing https://github.com/mull-project/mull
47. 48 1. Complex code Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests Problem - test solution
48. 49 Problem 3: Testing views
49. 50 Problem 3: Testing views Solution: Snapshot testing via iOSSnapshotTestCase https://github.com/uber/ios-snapshot-test-case
50. 51 reference image January, 2017 January, 2018 Problem 3: Testing views
51. 52 reference image recorded image January, 2018 January, 2017 Problem 3: Testing views
52. 53 reference image recorded image diff January, 2017 January, 2018 Problem 3: Testing views
53. 54 reference image recorded image January, 2018 January, 2017 Problem 3: Testing views
54. 55 Problem 3: Testing views
55. 56 Problem 3: Testing views
56. 57 Problem 3: Testing views
57. 58 Problem 3: Testing views
58. 59 Problem 3: Testing views
59. 60 Problem 3: Testing views
60. 61 Problem 3: Testing views Mocking size classes
61. 62 Problem 3: Testing views Mocking size classes
62. 63 Problem 3: Testing views Mocking size classes
63. 64 Problem 3: Testing views Mocking dynamic type font size
64. 65 Problem 3: Testing views Mocking dynamic type font size http://ocmock.org
65. 66 Problem 3: Testing views Mocking dynamic type font size http://ocmock.org
66. 67 Problem 3: Testing views Mocking dynamic type font size http://ocmock.org
67. 68 Problem 3: Testing views
68. 69 Problem 3: Testing views Snapshot tests overhead snapshot tests all tests
69. and for a bigger view, one test can take over 1 second 70 Problem 3: Testing views Snapshot tests overhead
70. 71 Problem 3: Testing views Snapshot tests overhead OBSERVED MODULE LISTING MODULE OFFER MODULE CART MODULE ... HOME MODULE ALLEGRO APP
71. quickly verify view’s layout 72 Problem 3: Testing views Snapshot tests remarks
72. quickly verify view’s layout test accessibility 73 Problem 3: Testing views Snapshot tests remarks
73. quickly verify view’s layout be careful when testing large views test accessibility 74 Problem 3: Testing views Snapshot tests remarks
74. quickly verify view’s layout be careful when testing large views test accessibility 75 measure your tests Problem 3: Testing views Snapshot tests remarks
75. 76 1. Complex code Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests Problem - test solution
76. 77 Problem 4: Predicting consequences of change
77. 78 Problem 4: Predicting consequences of change
78. 79 Problem 4: Predicting consequences of change
79. 80 Problem 4: Predicting consequences of change Solution: End-to-end UI tests LISTING OFFER HOME ... CART ALLEGRO SEARCH
80. 81 Problem 4: Predicting consequences of change Solution: End-to-end UI tests remarks KIF framework https://github.com/kif-framework/KIF
81. 82 Problem 4: Predicting consequences of change Solution: End-to-end UI tests remarks KIF framework https://github.com/kif-framework/KIF
82. 83 Problem 4: Predicting consequences of change Solution: End-to-end UI tests remarks KIF framework separate test target
83. 84 Problem 4: Predicting consequences of change Solution: End-to-end UI tests remarks KIF framework separate test target flaky tests
84. 85 Problem 4: Predicting consequences of change Solution: UI tests with mocked API LISTING OFFER HOME ... CART ALLEGRO SEARCH
85. Problem 4: Predicting consequences of change Solution: UI tests with mocked API 86 JSON responses Backend
86. Problem 4: Predicting consequences of change Solution: Scheme validation 87 JSON files JSON scheme
87. Problem 4: Predicting consequences of change Solution: Scheme validation 88 JSON files JSON scheme Configuration file
88. Problem 4: Predicting consequences of change Solution: Scheme validation 89 JSON files JSON scheme Configuration file Backend request
89. Problem 4: Predicting consequences of change Solution: Scheme validation 90 JSON files JSON scheme Configuration file Backend request Remote validation
90. Problem 4: Predicting consequences of change Solution: Scheme validation 91 Configuration file Remote validation JSON scheme API response JSON file
91. Problem 4: Predicting consequences of change Solution: Scheme validation 92 Configuration file Remote validation JSON scheme API response JSON file
92. Problem 4: Predicting consequences of change Solution: Scheme validation 93 Configuration file Remote validation JSON file JSON scheme API response
93. 94 Problem 4: Predicting consequences of change Solution: UI tests with mocked API remarks faster, more reliable UI tests
94. 95 Problem 4: Predicting consequences of change Solution: UI tests with mocked API remarks faster, more reliable UI tests relatively slow test coverage progress
95. 96 Problem 4: Predicting consequences of change Solution: UI tests with mocked API remarks faster, more reliable UI tests relatively slow test coverage progress tests not critical to merge
96. 97 Problem 4: Predicting consequences of change Solution: UI tests improvements More bugs detected 10% comparison of 10 real API and mocked API UI test runs Less flaky tests 70% comparison of 10 real API and mocked API UI test runs
97. 98 1. Complex code Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery Problem - test solution
98. 99 Problem 5: Is our release candidate working?
99. 10 0 Problem 5: Is our release candidate working? Solution: Dogfooding
100. 10 1 Problem 5: Is our release candidate working? Solution: Dogfooding
101. 10 2 Problem 5: Is our release candidate working? Solution: Dogfooding
102. 10 3 Problem 5: Is our release candidate working? Deployment process
103. 10 4 Problem 5: Is our release candidate working? Making our deployment reliable
104. 10 5 1. Crash on production Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release Problem - test solution
105. 10 6 Problem 6: Testing in laboratory condition
106. 10 7 Problem 6: Testing in laboratory condition ENG “After latest release application crashes - soft 9.0” Elen Musk
107. 10 8 Problem 6: Testing in laboratory condition Elen Musk ENG “After latest release application crashes - soft 9.0”
108. 10 9 Problem 6: Testing in laboratory condition Solution: Phased release
109. 11 0 Problem 6: Testing in laboratory condition Solution: Phased release https://www.smithsonianmag.com/smart-news/story-real-canary-coal-mine-180961570/
110. 11 1 1. Complex code Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests Problem - test solution
111. 112 Problem 7: Decision making
112. 113 Problem 7: Decision making 100% recommends this seller
113. 114 Problem 7: Decision making
114. 115 Problem 7: Decision making see all reviews
115. 116 Problem 7: Decision making
116. 117 Problem 7: Decision making Solution: A/B testing https://www.optimizely.com
117. 118 Problem 7: Decision making OFFER LISTING SEARCH OFFER COMPARISON ... HOME ALLEGRO
118. 119 Problem 7: Decision making A/B test runs in Offer Comparison module OFFER LISTING SEARCH OFFER COMPARISON ... HOME ALLEGRO
119. 120 Problem 7: Decision making Optimizely is linked to main app OFFER LISTING SEARCH OFFER COMPARISON ... HOME ALLEGRO A/B test runs in Offer Comparison module
120. 121 Problem 7: Decision making OFFER COMPARISON
121. 122 Problem 7: Decision making OFFER COMPARISON
122. 123 Problem 7: Decision making OFFER COMPARISON
123. 124 Problem 7: Decision making ALLEGRO
124. 125 Problem 7: Decision making ALLEGRO
125. 126 Problem 7: Decision making ALLEGRO
126. 127 1. Complex code Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
127. 128 Problem 8: Do we have the right product? “This app is a toy. I have nothing to say...” “Very bad. After updates it’s only worse not better...” Elton Johny Tejlor Swift
128. 129 Problem 8: Does the user know how to use our app? Solution: Usability testing One-way mirror
129. 130 Problem 8: Does the user know how to use our app? One-way mirror
130. 131 Problem 8: Does the user know how to use our app? Solution: Usability testing One-way mirror
131. 132 Problem 8: Does the user know how to use our app? Solution: Usability testing One-way mirror
132. 133 1. Complex code Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
133. 134 Thank you for your attention! Ewa Ludwiczak Senior Software Test Engineer ewa.ludwiczak@allegro.pl @ewabielskapoz Aleksander Grzyb iOS Developer aleksander.grzyb@allegro.pl @aleksandergrzyb