Ogres, onions, layers - story about test at scale

Transcript

1. Ogres, onions and layers story about tests at scale Ewa Ludwiczak & Aleksander Grzyb iOSDevUK 2018
2. Ewa Ludwiczak Senior Software Test Engineer ewa.ludwiczak@allegro.pl @ewabielskapoz Aleksander Grzyb iOS Developer aleksander.grzyb@allegro.pl @aleksandergrzyb 2
3. 3 Allegro application in Poland
4. 4 1. Code formatting Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
5. 5 1. Code formatting Static analysis Problem - test solution
6. 6 Problem 1: Code formatting
7. 7 Problem 1: Code formatting
8. 8 Problem 1: Code formatting SwiftFormat https://github.com/nicklockwood/SwiftFormat
9. 9 Problem 1: Code formatting SwiftFormat OBSERVED LISTING OFFER CART ... HOME ALLEGRO
10. 10 Problem 1: Code formatting SwiftFormat
11. 11 Problem 1: Code formatting SwiftLint https://github.com/realm/SwiftLint
12. 12 Problem 1: Code formatting SwiftLint OFFER LISTING CART ... ALLEGRO SEARCH HOME
13. 13 Problem 1: Code formatting SwiftLint SEARCH
14. 14 Problem 1: Code formatting SwiftLint too long variable name too short constant name
15. 15 Problem 1: Code formatting SwiftLint SEARCH
16. respect modular architecture format obvious rules, lint less trivial use the same versions of formatter & linter run locally after each build apply to production code and tests 16 Problem 1: Code formatting SwiftFormat + SwiftLint
17. 17 1. Code formatting Static analysis 2. Evergreen unit test Mutation testing Problem - test solution
18. 18 http://cleanswifter.com/asynchronous-ios-unit-test-tutorial/
19. 19 Problem 2: Evergreen unit tests
20. 20 Problem 2: Evergreen unit tests
21. 21 Problem 2: Evergreen unit tests Solution: Mutation testing function mutant mutation
22. 22 Problem 2: Evergreen unit tests Solution: Mutation testing change function logic
23. 23 Problem 2: Evergreen unit tests Solution: Mutation testing negate conditions
24. 24 Problem 2: Evergreen unit tests Solution: Mutation testing change mathematical operators
25. code coverage says little about test quality not all mutations add value use manual mutation in non-trivial test cases 25 Problem 2: Evergreen unit tests Solution: Mutation testing remarks automated mutation testing Problem 2: Evergreen unit tests Solution: Mutation testing https://github.com/mull-project/mull
27. 27 1. Code formatting Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests Problem - test solution
28. 28 Problem 3: Testing views
29. 29 Problem 3: Testing views Solution: Snapshot testing via iOSSnapshotTestCase https://github.com/uber/ios-snapshot-test-case
30. 30 reference image September, 2017 September, 2018 Problem 3: Testing views
31. 31 reference image recorded image September, 2018 September, 2017 Problem 3: Testing views
32. 32 reference image recorded image diff September, 2017 September, 2018 Problem 3: Testing views
33. 33 reference image recorded image September, 2018 September, 2017 Problem 3: Testing views
34. 34 Problem 3: Testing views
35. 35 Problem 3: Testing views
36. 36 Problem 3: Testing views
37. 37 Problem 3: Testing views Mocking size classes
38. 38 Problem 3: Testing views Mocking dynamic type font size
39. 39 Problem 3: Testing views Mocking dynamic type font size http://ocmock.org
40. 40 Problem 3: Testing views Snapshot tests overhead snapshot tests all tests
41. and for a bigger view, one test can take over 1 second 41 Problem 3: Testing views Snapshot tests overhead
42. 42 Problem 3: Testing views Snapshot tests overhead OBSERVED LISTING OFFER CART ... HOME ALLEGRO
43. 43 Problem 3: Testing views Snapshot tests remarks longer test execution test dynamic type quickly verify view layout preview of changes during code review
44. 44 1. Code formatting Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests Problem - test solution
45. 45 Problem 4: Predicting consequences of change
46. 46 Problem 4: Predicting consequences of change Solution: End-to-end UI tests LISTING OFFER HOME ... CART ALLEGRO SEARCH
47. 47 Problem 4: Predicting consequences of change Solution: End-to-end UI tests remarks KIF framework separate test target flaky tests
48. 48 Problem 4: Predicting consequences of change Solution: UI tests with mocked API LISTING OFFER HOME ... CART ALLEGRO SEARCH
49. Problem 4: Predicting consequences of change Solution: UI tests with mocked API 49 JSON responses Backend
50. Problem 4: Predicting consequences of change Solution: Scheme validation 50 JSON files JSON scheme Configuration file Backend request Remote validation
51. Problem 4: Predicting consequences of change Solution: Scheme validation 51 Configuration file Remote validation JSON file JSON scheme API response
52. 52 Problem 4: Predicting consequences of change Solution: UI tests with mocked API remarks faster, more reliable UI tests relatively slow test coverage progress tests not critical to merge
53. 53 Problem 4: Predicting consequences of change Solution: UI tests with mocked API remarks
54. 54 1. Code formatting Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery Problem - test solution
55. 55 Problem 5: Is our release candidate working?
56. 56 Problem 5: Is our release candidate working?
57. 57 Problem 5: Is our release candidate working? Release branch is created Release branch is created App deployed to App Store
58. 58 Problem 5: Is our release candidate working?
59. 59 Problem 5: Is our release candidate working? main branch is always deployable deploy habit user trust
60. 60 Problem 5: Is our release candidate working?
61. 61 Problem 5: Is our release candidate working? Solution: Dogfooding
62. 62 Problem 5: Is our release candidate working? Solution: Dogfooding
63. 63 Problem 5: Is our release candidate working? Deployment process
64. 64 Problem 5: Is our release candidate working? Making our deployment reliable
65. 65 1. Code formatting Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release Problem - test solution
66. 66 Problem 6: Testing in laboratory condition
67. 67 Problem 6: Testing in laboratory condition
68. 68 Problem 6: Testing in laboratory condition ENG “After latest release application crashes” “Nid yw'r cais yn gweithio”
69. 69 Problem 6: Testing in laboratory condition Solution: Phased release
70. 70 Problem 6: Testing in laboratory condition Solution: Phased release https://www.smithsonianmag.com/smart-news/story-real-canary-coal-mine-180961570/
71. 71 1. Code formatting Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests Problem - test solution
72. 72 Problem 7: Decision making
73. 73 Problem 7: Decision making
74. 74 Problem 7: Decision making
75. 75 Problem 7: Decision making share offer rhannu cynnig
76. 76 Problem 7: Decision making https://firebase.google.com/docs/ab-testing/
77. 77 Problem 7: Decision making Firebase is linked to main app OFFER COMPARISON LISTING SEARCH OFFER ... HOME ALLEGRO A/B test runs in Offer module
78. 78 1. Code formatting Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
79. 79 Problem 8: Do we have the right product? “This app is a toy. I have nothing to say...” Elton Johny Tejlor Swift “Tegan yw hwn. Nid oes gennyf ddim i'w ddweud...” “Very bad. After updates it’s only worse not better...” “Drwg iawn. Ar ôl diweddariadau, dim ond yn waeth na fydd yn well...”
80. 80 Problem 8: Does the user know how to use our app? Solution: Usability testing One-way mirror
81. 81 Problem 8: Does the user know how to use our app? One-way mirror
82. 82 Problem 8: Does the user know how to use our app? Solution: Usability testing One-way mirror
83. 83 Problem 8: Does the user know how to use our app? Solution: Usability testing One-way mirror One-way mirror
84. 84 Problem 8: Does the user know how to use our app? Solution: Usability testing One-way mirror
85. 85 Problem 8: Does the user know how to use our app? Solution: Usability testing One-way mirror One-way mirror
86. 86 1. Code formatting Static analysis 2. Evergreen unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
87. 87 Thank you for your attention! Ewa Ludwiczak Senior Software Test Engineer ewa.ludwiczak@allegro.pl @ewabielskapoz Aleksander Grzyb iOS Developer aleksander.grzyb@allegro.pl @aleksandergrzyb
88. 88 Thank you for your attention!