Ogres, onions and layers - story about tests at scale

Presentation for TestCon Moscow 2018

Transcript

2. Ogres, onions and layers story about tests at scale Ewa Ludwiczak & Kamil Pyć TestCon Moscow 2018
3. 3 Ewa Ludwiczak Senior Software Test Engineer ewa.ludwiczak@allegro.pl @ewabielskapoz Kamil Pyć Senior iOS Developer kamil.pyc@allegro.pl @KamilPyc
4. 4 Allegro application in Poland
5. 5 Tests are like onions
6. 6 1. Static analysis 2. Mutation testing 3. Snapshot tests 4. Functional tests Tests are like onions 5. Continuous integration 6. Phased release 7. AB Tests 8. Application analytics
7. Web Mobile 7 instant updates ~2 days ~7 days Browsers App Store Review Devices Backend Why we bother?
8. 8 1. Static analysis 2. Mutation testing 3. Snapshot tests 4. Functional tests Tests are like onions 5. Continuous integration 6. Phased release 7. AB Tests 8. Application analytics
9. 9 Problem 1: Cyclomatic complexity
11. 11 Problem 1: Cyclomatic complexity Solution: SwiftLint https://github.com/realm/SwiftLint
12. 12 Problem 1: Cyclomatic complexity Solution: SwiftLint
13. 13 Problem 1: Cyclomatic complexity SwiftFormat + SwiftLint https://github.com/nicklockwood/SwiftFormat
14. 14 Problem 1: Cyclomatic complexity SwiftFormat + SwiftLint
15. format obvious rules, lint less trivial run locally after each build 15 Problem 1: Cyclomatic complexity Solution: SwiftLint remarks apply to production code and tests use the same versions of formatter & linter
16. 16 1. Static analysis 2. Mutation testing 3. Snapshot tests 4. Functional tests Tests are like onions 5. Continuous integration 6. Phased release 7. AB Tests 8. Application analytics
17. 17 Problem 2: Evergreen unit tests
18. 18 Problem 2: Evergreen unit tests Solution: Mutation testing function mutant mutation
19. 19 Problem 2: Evergreen unit tests Solution: Mutation testing change function logic
20. 20 Problem 2: Evergreen unit tests Solution: Mutation testing negate conditions
21. 21 Problem 2: Evergreen unit tests Solution: Mutation testing change mathematical operators
22. code coverage says little about test quality not all mutations add value use manual mutation in non-trivial test cases 22 Problem 2: Evergreen unit tests Solution: Mutation testing remarks automated mutation testing Problem 2: Evergreen unit tests Solution: Mutation testing https://github.com/mull-project/mull
24. 24 1. Static analysis 2. Mutation testing 3. Snapshot tests 4. Functional tests Tests are like onions 5. Continuous integration 6. Phased release 7. AB Tests 8. Application analytics
25. 25 Problem 3: Simple mistakes Solution: Code review
26. 26 Problem 3: Simple mistakes Solution: Snapshot tests in code review
27. 27 Problem 3: Simple mistakes Solution: Snapshot tests in code review reference image recorded image diff
28. 28 Problem 3: Simple mistakes Solution: Snapshot tests in code review
29. value your time involve testers and designers into review 29 Problem 3: Simple mistakes Solution: Snapshot tests in code review spot visual mistakes on code review level
30. 30 1. Static analysis 2. Mutation testing 3. Snapshot tests 4. Functional tests Tests are like onions 5. Continuous integration 6. Phased release 7. AB Tests 8. Application analytics
31. 31 Problem 4: Predicting consequences of change
32. 32 Problem 4: Predicting consequences of change Solution: End-to-end UI tests https://github.com/kif-framework/KIF
33. 33 Problem 4: Predicting consequences of change Solution: End-to-end UI tests separate test target flaky test
34. Problem 4: Predicting consequences of change Solution: Functional UI tests with mocked API 34 JSON responses Backend
35. Problem 4: Predicting consequences of change Solution: Scheme validation 35 JSON files JSON scheme Configuration file Backend request Remote validation
36. Problem 4: Predicting consequences of change Solution: Scheme validation 36 Configuration file Remote validation JSON file JSON scheme API response
37. 37 Problem 4: Predicting consequences of change Solution: Functional UI tests with mocked API faster, more reliable UI tests relatively slow test coverage progress tests not critical to merge
38. 38 Problem 4: Predicting consequences of change Solution: UI tests improvements More bugs detected 10% comparison of 10 real API vs mocked API UI test runs Less flaky tests 70% comparison of 10 real API vs mocked API UI test runs
39. 39 1. Static analysis 2. Mutation testing 3. Snapshot tests 4. Functional tests Tests are like onions 5. Continuous integration 6. Phased release 7. AB Tests 8. Application analytics
40. 40 Problem 5: Late feedback Solution: Continuous integration
41. Problem 5: Late feedback Solution: Continuous integration 41
42. Problem 5: Late feedback Solution: Continuous delivery 42
43. Get as many tester as possible Automate your beta releases Highlight what should be checked 43 Problem 5: Late feedback Solution: Continuous delivery
44. 44 1. Static analysis 2. Mutation testing 3. Snapshot tests 4. Functional tests Tests are like onions 5. Continuous integration 6. Phased release 7. AB Tests 8. Application analytics
45. 45 Problem 6: Testing in laboratory condition XXX “Application doesn’t start after the update”
46. 46 Problem 6: Testing in laboratory condition Solution: Phased release
47. There are situations you cannot predict Prepare for failure 47 Remote configuration Problem 6: Testing in laboratory condition Solution: Phased release
48. 48 1. Static analysis 2. Mutation testing 3. Snapshot tests 4. Functional tests Tests are like onions 5. Continuous integration 6. Phased release 7. AB Tests 8. Application analytics
49. 49 Problem 7: Testing effectiveness ADD TO BASKET ADD TO BASKET
50. 50 Problem 7: Testing effectiveness Solution: AB Tests
51. 51 Problem 7: Testing effectiveness Solution: AB Tests
53. 53 1. Static analysis 2. Mutation testing 3. Snapshot tests 4. Functional tests Tests are like onions 5. Continuous integration 6. Phased release 7. AB Tests 8. Application analytics
54. 54 Problem 8: 1 Application release Stable version 2 Comment on App Store 3 https://fabric.io/kits/ios/crashlytics
55. 55 Problem 8: Decision making “Where is categories button? I can’t find it!”
56. 56 Problem 8: Decision making Solution: Make use of app analytical data
57. 57 Problem 8: Decision making Solution: Make use of app analytical data
58. Data-based decisions Bug priority and severity 58 Analyze and plan your reaction Problem 8: Decision making Solution: Make use of app analytical data
59. 59 1. Static analysis 2. Mutation testing 3. Snapshot tests 4. Functional tests Tests are like onions link_to_presentation 5. Continuous integration 6. Phased release 7. AB Tests 8. Application analytics
60. 60 Thank you for your attention We will be happy to answear your questions
61. 61 Ewa Ludwiczak Senior Software Test Engineer ewa.ludwiczak@allegro.pl @ewabielskapoz Kamil Pyć Senior iOS Developer kamil.pyc@allegro.pl @KamilPyc