Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[AppDevCon] Ogres, onions and layers – story about tests at scale

[AppDevCon] Ogres, onions and layers – story about tests at scale

We want to take you on the journey. Journey through different layers of tests in an iOS app that’s used by more than half million unique users per month. Get to know the way we test it, step by step, layer by layer. We start with advanced code static analysis, continue with mutation, snapshot and functional tests. We carry on with continuous delivery, A/B tests and usability tests. We want to share with you which tools we are using and what testing practices we believe in. This is a true story driven by problems we experience at work, presented from tester’s and developer’s perspective.

F993dbeef6dcba513da6f1acc04098ee?s=128

Aleksander Grzyb

March 16, 2018
Tweet

More Decks by Aleksander Grzyb

Other Decks in Programming

Transcript

  1. 1

  2. Ogres, onions and layers – story about tests at scale

    Ewa Ludwiczak & Aleksander Grzyb AppDevCon 2018 Amsterdam
  3. Ewa Ludwiczak Senior Software Test Engineer ewa.ludwiczak@allegro.pl @ewabielskapoz Aleksander Grzyb

    iOS Engineer aleksander.grzyb@allegro.pl @aleksandergrzyb 3
  4. 4 Allegro application in Poland

  5. 5 Allegro application in Poland

  6. 6 Allegro application in Poland

  7. 7 Static analysis Mutation testing Snapshot tests Functional tests Continuous

    delivery Phased release A/B tests Usability tests Problem - test solution
  8. 8 1. Crash on production Static analysis 2. Evergreen unit

    test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
  9. 9 1. Crash on production Static analysis Problem - test

    solution
  10. 10 Problem 1: Crash on production

  11. 11 Problem 1: Crash on production caused by force unwrapping

  12. 12 Problem 1: Crash on production caused by force unwrapping

    Solution: SwiftLint https://github.com/realm/SwiftLint
  13. 13 Problem 1: Crash on production caused by force unwrapping

    Solution: SwiftLint OFFER LISTING SEARCH CART ... HOME ALLEGRO
  14. 14 Problem 1: Crash on production caused by force unwrapping

    Solution: SwiftLint SEARCH
  15. 15 Problem 1: Crash on production caused by force unwrapping

    Solution: SwiftLint too short constant name
  16. 16 Problem 1: Crash on production caused by force unwrapping

    Solution: SwiftLint too long variable name too short constant name
  17. 17 Problem 1: Crash on production caused by force unwrapping

    Solution: SwiftLint too long variable name too short constant name
  18. 18 Problem 1: Crash on production caused by force unwrapping

    SwiftFormat + SwiftLint https://github.com/nicklockwood/SwiftFormat
  19. 19 Problem 1: Crash on production caused by force unwrapping

    SwiftFormat + SwiftLint OBSERVED LISTING OFFER CART ... HOME ALLEGRO
  20. 20 Problem 1: Crash on production caused by force unwrapping

    SwiftFormat + SwiftLint
  21. respect modular architecture 21 Problem 1: Crash on production caused

    by force unwrapping Solution: SwiftLint remarks
  22. respect modular architecture run locally after each build 22 Problem

    1: Crash on production caused by force unwrapping Solution: SwiftLint remarks
  23. respect modular architecture format obvious rules, lint less trivial run

    locally after each build 23 Problem 1: Crash on production caused by force unwrapping Solution: SwiftLint remarks
  24. respect modular architecture format obvious rules, lint less trivial run

    locally after each build 24 Problem 1: Crash on production caused by force unwrapping Solution: SwiftLint remarks apply to production code and tests
  25. respect modular architecture format obvious rules, lint less trivial use

    the same versions of formatter & linter run locally after each build apply to production code and tests 25 Problem 1: Crash on production caused by force unwrapping Solution: SwiftLint remarks
  26. 26 1. Crash on production Static analysis 2. Evergreen unit

    test Mutation testing Problem - test solution
  27. 27 http://cleanswifter.com/asynchronous-ios-unit-test-tutorial/

  28. 28 http://cleanswifter.com/asynchronous-ios-unit-test-tutorial/

  29. 29

  30. 30

  31. 31 Problem 2: Evergreen unit tests

  32. 32 Problem 2: Evergreen unit tests

  33. 33 Problem 2: Evergreen unit tests Solution: Mutation testing function

  34. 34 Problem 2: Evergreen unit tests Solution: Mutation testing function

    mutant mutation
  35. 35 Problem 2: Evergreen unit tests Solution: Mutation testing function

    mutant mutation
  36. 36 Problem 2: Evergreen unit tests Solution: Mutation testing change

    function logic
  37. 37 Problem 2: Evergreen unit tests Solution: Mutation testing change

    function logic
  38. 38 Problem 2: Evergreen unit tests Solution: Mutation testing negate

    conditions
  39. 39 Problem 2: Evergreen unit tests Solution: Mutation testing change

    mathematical operators
  40. 40 Problem 2: Evergreen unit tests Solution: Mutation testing AND

    <-> OR replacement
  41. 41 Problem 2: Evergreen unit tests Solution: Mutation testing remarks

    code coverage says little about test quality
  42. 42 Problem 2: Evergreen unit tests Solution: Mutation testing remarks

    code coverage says little about test quality use manual mutation in non-trivial test cases
  43. code coverage says little about test quality not all mutations

    add value use manual mutation in non-trivial test cases 43 Problem 2: Evergreen unit tests Solution: Mutation testing remarks
  44. code coverage says little about test quality not all mutations

    add value use manual mutation in non-trivial test cases 44 Problem 2: Evergreen unit tests Solution: Mutation testing remarks automated mutation testing Problem 2: Evergreen unit tests Solution: Mutation testing https://github.com/mull-project/mull
  45. 45

  46. 46 1. Crash on production Static analysis 2. Evergreen unit

    test Mutation testing 3. Testing views Snapshot tests Problem - test solution
  47. 47 Problem 3: Testing views

  48. 48 Problem 3: Testing views Solution: Snapshot testing via iOSSnapshotTestCase

    https://github.com/uber/ios-snapshot-test-case
  49. 49 reference image January, 2017 January, 2018

  50. 50 reference image recorded image January, 2018 January, 2017

  51. 51 reference image recorded image diff January, 2017 January, 2018

  52. 52 reference image recorded image January, 2018 January, 2017

  53. 53 Compact Width

  54. 54 Compact Width Regular Width

  55. 55 xSmall

  56. 56 xSmall Large (Default)

  57. 57 xSmall xxxLarge Large (Default)

  58. 58 xSmall xxxLarge Large (Default)

  59. 59 Problem 3: Testing views Mocking size classes

  60. 60 Problem 3: Testing views Mocking size classes

  61. 61 Problem 3: Testing views Mocking size classes

  62. 62 Problem 3: Testing views Mocking dynamic type font size

  63. 63 Problem 3: Testing views Mocking dynamic type font size

    http://ocmock.org
  64. 64 Problem 3: Testing views Mocking dynamic type font size

    http://ocmock.org
  65. 65 Problem 3: Testing views Mocking dynamic type font size

    http://ocmock.org
  66. 66 Problem 3: Testing views

  67. 67 Problem 3: Testing views Snapshot tests overhead snapshot tests

    all tests
  68. and for a bigger view, one test can take over

    1 second 68 Problem 3: Testing views Snapshot tests overhead
  69. 69 Problem 3: Testing views Snapshot tests overhead OBSERVED MODULE

    LISTING MODULE OFFER MODULE CART MODULE ... HOME MODULE ALLEGRO APP
  70. quickly verify view’s layout 70 Problem 3: Testing views Snapshot

    tests remarks
  71. quickly verify view’s layout enables to test accessibility 71 Problem

    3: Testing views Snapshot tests remarks
  72. quickly verify view’s layout be careful when testing large views

    enables to test accessibility 72 Problem 3: Testing views Snapshot tests remarks
  73. quickly verify view’s layout be careful when testing large views

    enables to test accessibility 73 measure your tests Problem 3: Testing views Snapshot tests remarks
  74. 74 1. Crash on production Static analysis 2. Evergreen unit

    test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests Problem - test solution
  75. 75 Problem 4: Predicting consequences of change

  76. 76 Problem 4: Predicting consequences of change

  77. 77 Problem 4: Predicting consequences of change

  78. 78 Problem 4: Predicting consequences of change Solution: End-to-end UI

    tests LISTING OFFER HOME ... CART ALLEGRO SEARCH https://github.com/kif-framework/KIF
  79. 79 Problem 4: Predicting consequences of change Solution: End-to-end UI

    tests remarks separate test target
  80. 80 Problem 4: Predicting consequences of change Solution: End-to-end UI

    tests remarks separate test target flaky tests
  81. 81 Problem 4: Predicting consequences of change Solution: UI tests

    with mocked API LISTING OFFER HOME ... CART ALLEGRO SEARCH https://github.com/kif-framework/KIF
  82. 82 Problem 4: Predicting consequences of change Solution: UI tests

    with mocked API JSON files JSON scheme configuration file validation
  83. 83 Problem 4: Predicting consequences of change Solution: UI tests

    with mocked API remarks faster, more reliable UI tests
  84. 84 Problem 4: Predicting consequences of change Solution: UI tests

    with mocked API remarks faster, more reliable UI tests relatively slow test coverage progress
  85. 85 Problem 4: Predicting consequences of change Solution: UI tests

    with mocked API remarks faster, more reliable UI tests relatively slow test coverage progress tests not critical to merge
  86. 86 Problem 4: Predicting consequences of change Solution: UI tests

    improvements More bugs detected 10% comparison of 10 real API and mocked API UI test runs Less flaky tests 70% comparison of 10 real API and mocked API UI test runs
  87. 87 1. Crash on production Static analysis 2. Evergreen unit

    test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery Problem - test solution
  88. 88 Problem 5: Is our release candidate working?

  89. 89 Problem 5: Is our release candidate working? Solution: Dogfooding

  90. 90 Problem 5: Is our release candidate working? Solution: Dogfooding

  91. 91 Problem 5: Is our release candidate working? Deployment process

  92. 92 Problem 5: Is our release candidate working? Making our

    deployment reliable
  93. 93 1. Crash on production Static analysis 2. Evergreen unit

    test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release Problem - test solution
  94. 94 Problem 6: Testing in laboratory condition

  95. 95 Problem 6: Testing in laboratory condition ENG “After latest

    release application crashes - soft 9.0” Elen Musk
  96. 96 Problem 6: Testing in laboratory condition Elen Musk ENG

    “After latest release application crashes - soft 9.0”
  97. 97 Problem 6: Testing in laboratory condition Solution: Phased release

  98. 98 Problem 6: Testing in laboratory condition Solution: Phased release

    https://www.smithsonianmag.com/smart-news/story-real-canary-coal-mine-180961570/
  99. 99 1. Crash on production Static analysis 2. Evergreen unit

    test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests Problem - test solution
  100. 100 Problem 7: Decision making

  101. 101 Problem 7: Decision making 100% recommends this seller

  102. 102 Problem 7: Decision making

  103. 103 Problem 7: Decision making see all reviews

  104. 104 Problem 7: Decision making

  105. 105 Problem 7: Decision making Solution: A/B testing https://www.optimizely.com

  106. 106 Problem 7: Decision making OFFER LISTING SEARCH OFFER COMPARISON

    ... HOME ALLEGRO
  107. 107 Problem 7: Decision making A/B test runs in Offer

    Comparison module OFFER LISTING SEARCH OFFER COMPARISON ... HOME ALLEGRO
  108. 108 Problem 7: Decision making Optimizely is linked to main

    app OFFER LISTING SEARCH OFFER COMPARISON ... HOME ALLEGRO A/B test runs in Offer Comparison module
  109. 109 Problem 7: Decision making OFFER COMPARISON

  110. 110 Problem 7: Decision making OFFER COMPARISON

  111. 111 Problem 7: Decision making OFFER COMPARISON

  112. 112 Problem 7: Decision making ALLEGRO

  113. 113 Problem 7: Decision making ALLEGRO

  114. 114 Problem 7: Decision making ALLEGRO

  115. 115 1. Crash on production Static analysis 2. Evergreen unit

    test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
  116. 116 Problem 8: Do we have the right product? “This

    app is a toy. I have nothing to say...” “Very bad. After updates, only worse not better...” Elton Johny Tejlor Swift
  117. 117 Problem 8: Does the user know how to use

    our app? Solution: Usability testing One-way mirror
  118. 118 Problem 8: Does the user know how to use

    our app? One-way mirror
  119. 119 Problem 8: Does the user know how to use

    our app? Solution: Usability testing One-way mirror
  120. 120 Problem 8: Does the user know how to use

    our app? Solution: Usability testing One-way mirror
  121. 121 1. Crash on production Static analysis 2. Evergreen unit

    test Mutation testing 3. Testing views Snapshot tests 4. Change management Functional tests 5. Is RC is working? Continuous delivery 6. Laboratory conditions Phased release 7. Decision making A/B tests 8. How users use our app? Usability tests Problem - test solution
  122. 122 Thank you for your attention! Ewa Ludwiczak Senior Software

    Test Engineer ewa.ludwiczak@allegro.pl @ewabielskapoz Aleksander Grzyb iOS Engineer aleksander.grzyb@allegro.pl @aleksandergrzyb
  123. Q&A 123 Thank you for your attention!