$30 off During Our Annual Pro Sale. View Details »

[Mobius] Layers of testing in iOS applications

[Mobius] Layers of testing in iOS applications

We want to take you on the journey through different layers of tests in an iOS app that’s used by more than half million unique users per month. Get to know the way we test it, step by step, layer by layer. We start with advanced code static analysis, continue with mutation, snapshot and functional tests. We carry on with continuous delivery, A/B tests and usability tests. We want to share with you which tools we are using and what testing practices we believe in. This is a true story driven by problems we experience at work, presented from tester’s and developer’s perspective.

Aleksander Grzyb

April 20, 2018
Tweet

More Decks by Aleksander Grzyb

Other Decks in Programming

Transcript

  1. Layers of testing in iOS applications Ewa Ludwiczak & Aleksander

    Grzyb Mobius 2018 Piter
  2. Ewa Ludwiczak Senior Software Test Engineer ewa.ludwiczak@allegro.pl @ewabielskapoz Aleksander Grzyb

    iOS Developer aleksander.grzyb@allegro.pl @aleksandergrzyb 3
  3. 4 Allegro application in Poland

  4. 5 Allegro application in Poland

  5. 6 Allegro application in Poland

  6. 7 Static analysis Mutation testing Snapshot tests Functional tests Continuous

    delivery Phased release A/B tests Usability tests Problem - test solution
  7. 8 1. Complex code Static analysis 2. Evergreen unit test

    Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
  8. 9 1. Complex code Static analysis Problem - test solution

  9. 10 Problem 1:

  10. 11 Problem 1: Complex code

  11. 12

  12. 13 Problem 1: Complex code Solution: SwiftLint https://github.com/realm/SwiftLint

  13. 14 Problem 1: Complex code Solution: SwiftLint https://github.com/realm/SwiftLint

  14. 15 Problem 1: Complex code Solution: SwiftLint OFFER LISTING SEARCH

    CART ... HOME ALLEGRO
  15. 16 Problem 1: Complex code Solution: SwiftLint SEARCH

  16. 17 Problem 1: Complex code Solution: SwiftLint too short constant

    name
  17. 18 Problem 1: Complex code Solution: SwiftLint too long variable

    name too short constant name
  18. 19 Problem 1: Complex code Solution: SwiftLint too long variable

    name too short constant name
  19. 20 Problem 1: Complex code Solution: SwiftLint SEARCH

  20. 21 Problem 1: Complex code SwiftFormat + SwiftLint https://github.com/nicklockwood/SwiftFormat

  21. 22 Problem 1: Complex code SwiftFormat + SwiftLint OBSERVED LISTING

    OFFER CART ... HOME ALLEGRO
  22. 23 Problem 1: Complex code SwiftFormat + SwiftLint

  23. respect modular architecture 24 Problem 1: Complex code Solution: SwiftLint

    remarks
  24. respect modular architecture run locally after each build 25 Problem

    1: Complex code Solution: SwiftLint remarks
  25. respect modular architecture format obvious rules, lint less trivial run

    locally after each build 26 Problem 1: Complex code Solution: SwiftLint remarks
  26. respect modular architecture format obvious rules, lint less trivial run

    locally after each build 27 Problem 1: Complex code Solution: SwiftLint remarks apply to production code and tests
  27. respect modular architecture format obvious rules, lint less trivial use

    the same versions of formatter & linter run locally after each build apply to production code and tests 28 Problem 1: Complex code Solution: SwiftLint remarks
  28. 29 1. Complex code Static analysis 2. Evergreen unit test

    Mutation testing Problem - test solution
  29. 30 http://cleanswifter.com/asynchronous-ios-unit-test-tutorial/

  30. 31 http://cleanswifter.com/asynchronous-ios-unit-test-tutorial/

  31. 32

  32. 33

  33. 34 Problem 2: Evergreen unit tests

  34. 35 Problem 2: Evergreen unit tests

  35. 36 Problem 2: Evergreen unit tests Solution: Mutation testing function

  36. 37 Problem 2: Evergreen unit tests Solution: Mutation testing function

    mutant mutation
  37. 38 Problem 2: Evergreen unit tests Solution: Mutation testing function

    mutant mutation
  38. 39 Problem 2: Evergreen unit tests Solution: Mutation testing change

    function logic
  39. 40 Problem 2: Evergreen unit tests Solution: Mutation testing change

    function logic
  40. 41 Problem 2: Evergreen unit tests Solution: Mutation testing negate

    conditions
  41. 42 Problem 2: Evergreen unit tests Solution: Mutation testing change

    mathematical operators
  42. 43 Problem 2: Evergreen unit tests Solution: Mutation testing remarks

    code coverage says little about test quality
  43. 44 Problem 2: Evergreen unit tests Solution: Mutation testing remarks

    code coverage says little about test quality use manual mutation in non-trivial test cases
  44. code coverage says little about test quality not all mutations

    add value use manual mutation in non-trivial test cases 45 Problem 2: Evergreen unit tests Solution: Mutation testing remarks
  45. code coverage says little about test quality not all mutations

    add value use manual mutation in non-trivial test cases 46 Problem 2: Evergreen unit tests Solution: Mutation testing remarks automated mutation testing Problem 2: Evergreen unit tests Solution: Mutation testing https://github.com/mull-project/mull
  46. 47

  47. 48 1. Complex code Static analysis 2. Evergreen unit test

    Mutation testing 3. Testing views Snapshot tests Problem - test solution
  48. 49 Problem 3: Testing views

  49. 50 Problem 3: Testing views Solution: Snapshot testing via iOSSnapshotTestCase

    https://github.com/uber/ios-snapshot-test-case
  50. 51 reference image January, 2017 January, 2018 Problem 3: Testing

    views
  51. 52 reference image recorded image January, 2018 January, 2017 Problem

    3: Testing views
  52. 53 reference image recorded image diff January, 2017 January, 2018

    Problem 3: Testing views
  53. 54 reference image recorded image January, 2018 January, 2017 Problem

    3: Testing views
  54. 55 Problem 3: Testing views

  55. 56 Problem 3: Testing views

  56. 57 Problem 3: Testing views

  57. 58 Problem 3: Testing views

  58. 59 Problem 3: Testing views

  59. 60 Problem 3: Testing views

  60. 61 Problem 3: Testing views Mocking size classes

  61. 62 Problem 3: Testing views Mocking size classes

  62. 63 Problem 3: Testing views Mocking size classes

  63. 64 Problem 3: Testing views Mocking dynamic type font size

  64. 65 Problem 3: Testing views Mocking dynamic type font size

    http://ocmock.org
  65. 66 Problem 3: Testing views Mocking dynamic type font size

    http://ocmock.org
  66. 67 Problem 3: Testing views Mocking dynamic type font size

    http://ocmock.org
  67. 68 Problem 3: Testing views

  68. 69 Problem 3: Testing views Snapshot tests overhead snapshot tests

    all tests
  69. and for a bigger view, one test can take over

    1 second 70 Problem 3: Testing views Snapshot tests overhead
  70. 71 Problem 3: Testing views Snapshot tests overhead OBSERVED MODULE

    LISTING MODULE OFFER MODULE CART MODULE ... HOME MODULE ALLEGRO APP
  71. quickly verify view’s layout 72 Problem 3: Testing views Snapshot

    tests remarks
  72. quickly verify view’s layout test accessibility 73 Problem 3: Testing

    views Snapshot tests remarks
  73. quickly verify view’s layout be careful when testing large views

    test accessibility 74 Problem 3: Testing views Snapshot tests remarks
  74. quickly verify view’s layout be careful when testing large views

    test accessibility 75 measure your tests Problem 3: Testing views Snapshot tests remarks
  75. 76 1. Complex code Static analysis 2. Evergreen unit test

    Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests Problem - test solution
  76. 77 Problem 4: Predicting consequences of change

  77. 78 Problem 4: Predicting consequences of change

  78. 79 Problem 4: Predicting consequences of change

  79. 80 Problem 4: Predicting consequences of change Solution: End-to-end UI

    tests LISTING OFFER HOME ... CART ALLEGRO SEARCH
  80. 81 Problem 4: Predicting consequences of change Solution: End-to-end UI

    tests remarks KIF framework https://github.com/kif-framework/KIF
  81. 82 Problem 4: Predicting consequences of change Solution: End-to-end UI

    tests remarks KIF framework https://github.com/kif-framework/KIF
  82. 83 Problem 4: Predicting consequences of change Solution: End-to-end UI

    tests remarks KIF framework separate test target
  83. 84 Problem 4: Predicting consequences of change Solution: End-to-end UI

    tests remarks KIF framework separate test target flaky tests
  84. 85 Problem 4: Predicting consequences of change Solution: UI tests

    with mocked API LISTING OFFER HOME ... CART ALLEGRO SEARCH
  85. Problem 4: Predicting consequences of change Solution: UI tests with

    mocked API 86 JSON responses Backend
  86. Problem 4: Predicting consequences of change Solution: Scheme validation 87

    JSON files JSON scheme
  87. Problem 4: Predicting consequences of change Solution: Scheme validation 88

    JSON files JSON scheme Configuration file
  88. Problem 4: Predicting consequences of change Solution: Scheme validation 89

    JSON files JSON scheme Configuration file Backend request
  89. Problem 4: Predicting consequences of change Solution: Scheme validation 90

    JSON files JSON scheme Configuration file Backend request Remote validation
  90. Problem 4: Predicting consequences of change Solution: Scheme validation 91

    Configuration file Remote validation JSON scheme API response JSON file
  91. Problem 4: Predicting consequences of change Solution: Scheme validation 92

    Configuration file Remote validation JSON scheme API response JSON file
  92. Problem 4: Predicting consequences of change Solution: Scheme validation 93

    Configuration file Remote validation JSON file JSON scheme API response
  93. 94 Problem 4: Predicting consequences of change Solution: UI tests

    with mocked API remarks faster, more reliable UI tests
  94. 95 Problem 4: Predicting consequences of change Solution: UI tests

    with mocked API remarks faster, more reliable UI tests relatively slow test coverage progress
  95. 96 Problem 4: Predicting consequences of change Solution: UI tests

    with mocked API remarks faster, more reliable UI tests relatively slow test coverage progress tests not critical to merge
  96. 97 Problem 4: Predicting consequences of change Solution: UI tests

    improvements More bugs detected 10% comparison of 10 real API and mocked API UI test runs Less flaky tests 70% comparison of 10 real API and mocked API UI test runs
  97. 98 1. Complex code Static analysis 2. Evergreen unit test

    Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery Problem - test solution
  98. 99 Problem 5: Is our release candidate working?

  99. 10 0 Problem 5: Is our release candidate working? Solution:

    Dogfooding
  100. 10 1 Problem 5: Is our release candidate working? Solution:

    Dogfooding
  101. 10 2 Problem 5: Is our release candidate working? Solution:

    Dogfooding
  102. 10 3 Problem 5: Is our release candidate working? Deployment

    process
  103. 10 4 Problem 5: Is our release candidate working? Making

    our deployment reliable
  104. 10 5 1. Crash on production Static analysis 2. Evergreen

    unit test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release Problem - test solution
  105. 10 6 Problem 6: Testing in laboratory condition

  106. 10 7 Problem 6: Testing in laboratory condition ENG “After

    latest release application crashes - soft 9.0” Elen Musk
  107. 10 8 Problem 6: Testing in laboratory condition Elen Musk

    ENG “After latest release application crashes - soft 9.0”
  108. 10 9 Problem 6: Testing in laboratory condition Solution: Phased

    release
  109. 11 0 Problem 6: Testing in laboratory condition Solution: Phased

    release https://www.smithsonianmag.com/smart-news/story-real-canary-coal-mine-180961570/
  110. 11 1 1. Complex code Static analysis 2. Evergreen unit

    test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests Problem - test solution
  111. 112 Problem 7: Decision making

  112. 113 Problem 7: Decision making 100% recommends this seller

  113. 114 Problem 7: Decision making

  114. 115 Problem 7: Decision making see all reviews

  115. 116 Problem 7: Decision making

  116. 117 Problem 7: Decision making Solution: A/B testing https://www.optimizely.com

  117. 118 Problem 7: Decision making OFFER LISTING SEARCH OFFER COMPARISON

    ... HOME ALLEGRO
  118. 119 Problem 7: Decision making A/B test runs in Offer

    Comparison module OFFER LISTING SEARCH OFFER COMPARISON ... HOME ALLEGRO
  119. 120 Problem 7: Decision making Optimizely is linked to main

    app OFFER LISTING SEARCH OFFER COMPARISON ... HOME ALLEGRO A/B test runs in Offer Comparison module
  120. 121 Problem 7: Decision making OFFER COMPARISON

  121. 122 Problem 7: Decision making OFFER COMPARISON

  122. 123 Problem 7: Decision making OFFER COMPARISON

  123. 124 Problem 7: Decision making ALLEGRO

  124. 125 Problem 7: Decision making ALLEGRO

  125. 126 Problem 7: Decision making ALLEGRO

  126. 127 1. Complex code Static analysis 2. Evergreen unit test

    Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
  127. 128 Problem 8: Do we have the right product? “This

    app is a toy. I have nothing to say...” “Very bad. After updates it’s only worse not better...” Elton Johny Tejlor Swift
  128. 129 Problem 8: Does the user know how to use

    our app? Solution: Usability testing One-way mirror
  129. 130 Problem 8: Does the user know how to use

    our app? One-way mirror
  130. 131 Problem 8: Does the user know how to use

    our app? Solution: Usability testing One-way mirror
  131. 132 Problem 8: Does the user know how to use

    our app? Solution: Usability testing One-way mirror
  132. 133 1. Complex code Static analysis 2. Evergreen unit test

    Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
  133. 134 Thank you for your attention! Ewa Ludwiczak Senior Software

    Test Engineer ewa.ludwiczak@allegro.pl @ewabielskapoz Aleksander Grzyb iOS Developer aleksander.grzyb@allegro.pl @aleksandergrzyb