Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[Mobius] Layers of testing in iOS applications

[Mobius] Layers of testing in iOS applications

We want to take you on the journey through different layers of tests in an iOS app that’s used by more than half million unique users per month. Get to know the way we test it, step by step, layer by layer. We start with advanced code static analysis, continue with mutation, snapshot and functional tests. We carry on with continuous delivery, A/B tests and usability tests. We want to share with you which tools we are using and what testing practices we believe in. This is a true story driven by problems we experience at work, presented from tester’s and developer’s perspective.

Aleksander Grzyb

April 20, 2018
Tweet

More Decks by Aleksander Grzyb

Other Decks in Programming

Transcript

  1. 7 Static analysis Mutation testing Snapshot tests Functional tests Continuous

    delivery Phased release A/B tests Usability tests Problem - test solution
  2. 8 1. Complex code Static analysis 2. Evergreen unit test

    Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
  3. 12

  4. respect modular architecture run locally after each build 25 Problem

    1: Complex code Solution: SwiftLint remarks
  5. respect modular architecture format obvious rules, lint less trivial run

    locally after each build 26 Problem 1: Complex code Solution: SwiftLint remarks
  6. respect modular architecture format obvious rules, lint less trivial run

    locally after each build 27 Problem 1: Complex code Solution: SwiftLint remarks apply to production code and tests
  7. respect modular architecture format obvious rules, lint less trivial use

    the same versions of formatter & linter run locally after each build apply to production code and tests 28 Problem 1: Complex code Solution: SwiftLint remarks
  8. 29 1. Complex code Static analysis 2. Evergreen unit test

    Mutation testing Problem - test solution
  9. 32

  10. 33

  11. 43 Problem 2: Evergreen unit tests Solution: Mutation testing remarks

    code coverage says little about test quality
  12. 44 Problem 2: Evergreen unit tests Solution: Mutation testing remarks

    code coverage says little about test quality use manual mutation in non-trivial test cases
  13. code coverage says little about test quality not all mutations

    add value use manual mutation in non-trivial test cases 45 Problem 2: Evergreen unit tests Solution: Mutation testing remarks
  14. code coverage says little about test quality not all mutations

    add value use manual mutation in non-trivial test cases 46 Problem 2: Evergreen unit tests Solution: Mutation testing remarks automated mutation testing Problem 2: Evergreen unit tests Solution: Mutation testing https://github.com/mull-project/mull
  15. 47

  16. 48 1. Complex code Static analysis 2. Evergreen unit test

    Mutation testing 3. Testing views Snapshot tests Problem - test solution
  17. and for a bigger view, one test can take over

    1 second 70 Problem 3: Testing views Snapshot tests overhead
  18. 71 Problem 3: Testing views Snapshot tests overhead OBSERVED MODULE

    LISTING MODULE OFFER MODULE CART MODULE ... HOME MODULE ALLEGRO APP
  19. quickly verify view’s layout be careful when testing large views

    test accessibility 74 Problem 3: Testing views Snapshot tests remarks
  20. quickly verify view’s layout be careful when testing large views

    test accessibility 75 measure your tests Problem 3: Testing views Snapshot tests remarks
  21. 76 1. Complex code Static analysis 2. Evergreen unit test

    Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests Problem - test solution
  22. 80 Problem 4: Predicting consequences of change Solution: End-to-end UI

    tests LISTING OFFER HOME ... CART ALLEGRO SEARCH
  23. 81 Problem 4: Predicting consequences of change Solution: End-to-end UI

    tests remarks KIF framework https://github.com/kif-framework/KIF
  24. 82 Problem 4: Predicting consequences of change Solution: End-to-end UI

    tests remarks KIF framework https://github.com/kif-framework/KIF
  25. 83 Problem 4: Predicting consequences of change Solution: End-to-end UI

    tests remarks KIF framework separate test target
  26. 84 Problem 4: Predicting consequences of change Solution: End-to-end UI

    tests remarks KIF framework separate test target flaky tests
  27. 85 Problem 4: Predicting consequences of change Solution: UI tests

    with mocked API LISTING OFFER HOME ... CART ALLEGRO SEARCH
  28. Problem 4: Predicting consequences of change Solution: Scheme validation 89

    JSON files JSON scheme Configuration file Backend request
  29. Problem 4: Predicting consequences of change Solution: Scheme validation 90

    JSON files JSON scheme Configuration file Backend request Remote validation
  30. Problem 4: Predicting consequences of change Solution: Scheme validation 91

    Configuration file Remote validation JSON scheme API response JSON file
  31. Problem 4: Predicting consequences of change Solution: Scheme validation 92

    Configuration file Remote validation JSON scheme API response JSON file
  32. Problem 4: Predicting consequences of change Solution: Scheme validation 93

    Configuration file Remote validation JSON file JSON scheme API response
  33. 94 Problem 4: Predicting consequences of change Solution: UI tests

    with mocked API remarks faster, more reliable UI tests
  34. 95 Problem 4: Predicting consequences of change Solution: UI tests

    with mocked API remarks faster, more reliable UI tests relatively slow test coverage progress
  35. 96 Problem 4: Predicting consequences of change Solution: UI tests

    with mocked API remarks faster, more reliable UI tests relatively slow test coverage progress tests not critical to merge
  36. 97 Problem 4: Predicting consequences of change Solution: UI tests

    improvements More bugs detected 10% comparison of 10 real API and mocked API UI test runs Less flaky tests 70% comparison of 10 real API and mocked API UI test runs
  37. 98 1. Complex code Static analysis 2. Evergreen unit test

    Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery Problem - test solution
  38. 10 5 1. Crash on production Static analysis 2. Evergreen

    unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release Problem - test solution
  39. 10 7 Problem 6: Testing in laboratory condition ENG “After

    latest release application crashes - soft 9.0” Elen Musk
  40. 10 8 Problem 6: Testing in laboratory condition Elen Musk

    ENG “After latest release application crashes - soft 9.0”
  41. 11 0 Problem 6: Testing in laboratory condition Solution: Phased

    release https://www.smithsonianmag.com/smart-news/story-real-canary-coal-mine-180961570/
  42. 11 1 1. Complex code Static analysis 2. Evergreen unit

    test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests Problem - test solution
  43. 119 Problem 7: Decision making A/B test runs in Offer

    Comparison module OFFER LISTING SEARCH OFFER COMPARISON ... HOME ALLEGRO
  44. 120 Problem 7: Decision making Optimizely is linked to main

    app OFFER LISTING SEARCH OFFER COMPARISON ... HOME ALLEGRO A/B test runs in Offer Comparison module
  45. 127 1. Complex code Static analysis 2. Evergreen unit test

    Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
  46. 128 Problem 8: Do we have the right product? “This

    app is a toy. I have nothing to say...” “Very bad. After updates it’s only worse not better...” Elton Johny Tejlor Swift
  47. 129 Problem 8: Does the user know how to use

    our app? Solution: Usability testing One-way mirror
  48. 131 Problem 8: Does the user know how to use

    our app? Solution: Usability testing One-way mirror
  49. 132 Problem 8: Does the user know how to use

    our app? Solution: Usability testing One-way mirror
  50. 133 1. Complex code Static analysis 2. Evergreen unit test

    Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
  51. 134 Thank you for your attention! Ewa Ludwiczak Senior Software

    Test Engineer [email protected] @ewabielskapoz Aleksander Grzyb iOS Developer [email protected] @aleksandergrzyb