[AppDevCon] Ogres, onions and layers – story about tests at scale

We want to take you on the journey. Journey through different layers of tests in an iOS app that’s used by more than half million unique users per month. Get to know the way we test it, step by step, layer by layer. We start with advanced code static analysis, continue with mutation, snapshot and functional tests. We carry on with continuous delivery, A/B tests and usability tests. We want to share with you which tools we are using and what testing practices we believe in. This is a true story driven by problems we experience at work, presented from tester’s and developer’s perspective.

Transcript

2. Ogres, onions and layers – story about tests at scale Ewa Ludwiczak & Aleksander Grzyb AppDevCon 2018 Amsterdam
3. Ewa Ludwiczak Senior Software Test Engineer ewa.ludwiczak@allegro.pl @ewabielskapoz Aleksander Grzyb iOS Engineer aleksander.grzyb@allegro.pl @aleksandergrzyb 3
4. 4 Allegro application in Poland
5. 5 Allegro application in Poland
6. 6 Allegro application in Poland
7. 7 Static analysis Mutation testing Snapshot tests Functional tests Continuous delivery Phased release A/B tests Usability tests Problem - test solution
8. 8 1. Crash on production Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
9. 9 1. Crash on production Static analysis Problem - test solution
10. 10 Problem 1: Crash on production
11. 11 Problem 1: Crash on production caused by force unwrapping
12. 12 Problem 1: Crash on production caused by force unwrapping Solution: SwiftLint https://github.com/realm/SwiftLint
13. 13 Problem 1: Crash on production caused by force unwrapping Solution: SwiftLint OFFER LISTING SEARCH CART ... HOME ALLEGRO
14. 14 Problem 1: Crash on production caused by force unwrapping Solution: SwiftLint SEARCH
15. 15 Problem 1: Crash on production caused by force unwrapping Solution: SwiftLint too short constant name
16. 16 Problem 1: Crash on production caused by force unwrapping Solution: SwiftLint too long variable name too short constant name
17. 17 Problem 1: Crash on production caused by force unwrapping Solution: SwiftLint too long variable name too short constant name
18. 18 Problem 1: Crash on production caused by force unwrapping SwiftFormat + SwiftLint https://github.com/nicklockwood/SwiftFormat
19. 19 Problem 1: Crash on production caused by force unwrapping SwiftFormat + SwiftLint OBSERVED LISTING OFFER CART ... HOME ALLEGRO
20. 20 Problem 1: Crash on production caused by force unwrapping SwiftFormat + SwiftLint
21. respect modular architecture 21 Problem 1: Crash on production caused by force unwrapping Solution: SwiftLint remarks
22. respect modular architecture run locally after each build 22 Problem 1: Crash on production caused by force unwrapping Solution: SwiftLint remarks
23. respect modular architecture format obvious rules, lint less trivial run locally after each build 23 Problem 1: Crash on production caused by force unwrapping Solution: SwiftLint remarks
24. respect modular architecture format obvious rules, lint less trivial run locally after each build 24 Problem 1: Crash on production caused by force unwrapping Solution: SwiftLint remarks apply to production code and tests
25. respect modular architecture format obvious rules, lint less trivial use the same versions of formatter & linter run locally after each build apply to production code and tests 25 Problem 1: Crash on production caused by force unwrapping Solution: SwiftLint remarks
26. 26 1. Crash on production Static analysis 2. Evergreen unit test Mutation testing Problem - test solution
27. 27 http://cleanswifter.com/asynchronous-ios-unit-test-tutorial/
28. 28 http://cleanswifter.com/asynchronous-ios-unit-test-tutorial/
31. 31 Problem 2: Evergreen unit tests
32. 32 Problem 2: Evergreen unit tests
33. 33 Problem 2: Evergreen unit tests Solution: Mutation testing function
34. 34 Problem 2: Evergreen unit tests Solution: Mutation testing function mutant mutation
35. 35 Problem 2: Evergreen unit tests Solution: Mutation testing function mutant mutation
36. 36 Problem 2: Evergreen unit tests Solution: Mutation testing change function logic
37. 37 Problem 2: Evergreen unit tests Solution: Mutation testing change function logic
38. 38 Problem 2: Evergreen unit tests Solution: Mutation testing negate conditions
39. 39 Problem 2: Evergreen unit tests Solution: Mutation testing change mathematical operators
40. 40 Problem 2: Evergreen unit tests Solution: Mutation testing AND <-> OR replacement
41. 41 Problem 2: Evergreen unit tests Solution: Mutation testing remarks code coverage says little about test quality
42. 42 Problem 2: Evergreen unit tests Solution: Mutation testing remarks code coverage says little about test quality use manual mutation in non-trivial test cases
43. code coverage says little about test quality not all mutations add value use manual mutation in non-trivial test cases 43 Problem 2: Evergreen unit tests Solution: Mutation testing remarks
44. code coverage says little about test quality not all mutations add value use manual mutation in non-trivial test cases 44 Problem 2: Evergreen unit tests Solution: Mutation testing remarks automated mutation testing Problem 2: Evergreen unit tests Solution: Mutation testing https://github.com/mull-project/mull
46. 46 1. Crash on production Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests Problem - test solution
47. 47 Problem 3: Testing views
48. 48 Problem 3: Testing views Solution: Snapshot testing via iOSSnapshotTestCase https://github.com/uber/ios-snapshot-test-case
49. 49 reference image January, 2017 January, 2018
50. 50 reference image recorded image January, 2018 January, 2017
51. 51 reference image recorded image diff January, 2017 January, 2018
52. 52 reference image recorded image January, 2018 January, 2017
53. 53 Compact Width
54. 54 Compact Width Regular Width
55. 55 xSmall
56. 56 xSmall Large (Default)
57. 57 xSmall xxxLarge Large (Default)
58. 58 xSmall xxxLarge Large (Default)
59. 59 Problem 3: Testing views Mocking size classes
60. 60 Problem 3: Testing views Mocking size classes
61. 61 Problem 3: Testing views Mocking size classes
62. 62 Problem 3: Testing views Mocking dynamic type font size
63. 63 Problem 3: Testing views Mocking dynamic type font size http://ocmock.org
64. 64 Problem 3: Testing views Mocking dynamic type font size http://ocmock.org
65. 65 Problem 3: Testing views Mocking dynamic type font size http://ocmock.org
66. 66 Problem 3: Testing views
67. 67 Problem 3: Testing views Snapshot tests overhead snapshot tests all tests
68. and for a bigger view, one test can take over 1 second 68 Problem 3: Testing views Snapshot tests overhead
69. 69 Problem 3: Testing views Snapshot tests overhead OBSERVED MODULE LISTING MODULE OFFER MODULE CART MODULE ... HOME MODULE ALLEGRO APP
70. quickly verify view’s layout 70 Problem 3: Testing views Snapshot tests remarks
71. quickly verify view’s layout enables to test accessibility 71 Problem 3: Testing views Snapshot tests remarks
72. quickly verify view’s layout be careful when testing large views enables to test accessibility 72 Problem 3: Testing views Snapshot tests remarks
73. quickly verify view’s layout be careful when testing large views enables to test accessibility 73 measure your tests Problem 3: Testing views Snapshot tests remarks
74. 74 1. Crash on production Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests Problem - test solution
75. 75 Problem 4: Predicting consequences of change
76. 76 Problem 4: Predicting consequences of change
77. 77 Problem 4: Predicting consequences of change
78. 78 Problem 4: Predicting consequences of change Solution: End-to-end UI tests LISTING OFFER HOME ... CART ALLEGRO SEARCH https://github.com/kif-framework/KIF
79. 79 Problem 4: Predicting consequences of change Solution: End-to-end UI tests remarks separate test target
80. 80 Problem 4: Predicting consequences of change Solution: End-to-end UI tests remarks separate test target flaky tests
81. 81 Problem 4: Predicting consequences of change Solution: UI tests with mocked API LISTING OFFER HOME ... CART ALLEGRO SEARCH https://github.com/kif-framework/KIF
82. 82 Problem 4: Predicting consequences of change Solution: UI tests with mocked API JSON files JSON scheme configuration file validation
83. 83 Problem 4: Predicting consequences of change Solution: UI tests with mocked API remarks faster, more reliable UI tests
84. 84 Problem 4: Predicting consequences of change Solution: UI tests with mocked API remarks faster, more reliable UI tests relatively slow test coverage progress
85. 85 Problem 4: Predicting consequences of change Solution: UI tests with mocked API remarks faster, more reliable UI tests relatively slow test coverage progress tests not critical to merge
86. 86 Problem 4: Predicting consequences of change Solution: UI tests improvements More bugs detected 10% comparison of 10 real API and mocked API UI test runs Less flaky tests 70% comparison of 10 real API and mocked API UI test runs
87. 87 1. Crash on production Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery Problem - test solution
88. 88 Problem 5: Is our release candidate working?
89. 89 Problem 5: Is our release candidate working? Solution: Dogfooding
90. 90 Problem 5: Is our release candidate working? Solution: Dogfooding
91. 91 Problem 5: Is our release candidate working? Deployment process
92. 92 Problem 5: Is our release candidate working? Making our deployment reliable
93. 93 1. Crash on production Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release Problem - test solution
94. 94 Problem 6: Testing in laboratory condition
95. 95 Problem 6: Testing in laboratory condition ENG “After latest release application crashes - soft 9.0” Elen Musk
96. 96 Problem 6: Testing in laboratory condition Elen Musk ENG “After latest release application crashes - soft 9.0”
97. 97 Problem 6: Testing in laboratory condition Solution: Phased release
98. 98 Problem 6: Testing in laboratory condition Solution: Phased release https://www.smithsonianmag.com/smart-news/story-real-canary-coal-mine-180961570/
99. 99 1. Crash on production Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests Problem - test solution
100. 100 Problem 7: Decision making
101. 101 Problem 7: Decision making 100% recommends this seller
102. 102 Problem 7: Decision making
103. 103 Problem 7: Decision making see all reviews
104. 104 Problem 7: Decision making
105. 105 Problem 7: Decision making Solution: A/B testing https://www.optimizely.com
106. 106 Problem 7: Decision making OFFER LISTING SEARCH OFFER COMPARISON ... HOME ALLEGRO
107. 107 Problem 7: Decision making A/B test runs in Offer Comparison module OFFER LISTING SEARCH OFFER COMPARISON ... HOME ALLEGRO
108. 108 Problem 7: Decision making Optimizely is linked to main app OFFER LISTING SEARCH OFFER COMPARISON ... HOME ALLEGRO A/B test runs in Offer Comparison module
109. 109 Problem 7: Decision making OFFER COMPARISON
110. 110 Problem 7: Decision making OFFER COMPARISON
111. 111 Problem 7: Decision making OFFER COMPARISON
112. 112 Problem 7: Decision making ALLEGRO
113. 113 Problem 7: Decision making ALLEGRO
114. 114 Problem 7: Decision making ALLEGRO
115. 115 1. Crash on production Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
116. 116 Problem 8: Do we have the right product? “This app is a toy. I have nothing to say...” “Very bad. After updates, only worse not better...” Elton Johny Tejlor Swift
117. 117 Problem 8: Does the user know how to use our app? Solution: Usability testing One-way mirror
118. 118 Problem 8: Does the user know how to use our app? One-way mirror
119. 119 Problem 8: Does the user know how to use our app? Solution: Usability testing One-way mirror
120. 120 Problem 8: Does the user know how to use our app? Solution: Usability testing One-way mirror
121. 121 1. Crash on production Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
122. 122 Thank you for your attention! Ewa Ludwiczak Senior Software Test Engineer ewa.ludwiczak@allegro.pl @ewabielskapoz Aleksander Grzyb iOS Engineer aleksander.grzyb@allegro.pl @aleksandergrzyb
123. Q&A 123 Thank you for your attention!