[Mobius] Layers of testing in iOS applications

We want to take you on the journey through different layers of tests in an iOS app that’s used by more than half million unique users per month. Get to know the way we test it, step by step, layer by layer. We start with advanced code static analysis, continue with mutation, snapshot and functional tests. We carry on with continuous delivery, A/B tests and usability tests. We want to share with you which tools we are using and what testing practices we believe in. This is a true story driven by problems we experience at work, presented from tester’s and developer’s perspective.

Transcript

1. Layers of testing
   in iOS applications
   Ewa Ludwiczak & Aleksander Grzyb
   Mobius 2018 Piter
   

   View Slide

2. Ewa Ludwiczak
   Senior Software Test Engineer
   ewa.ludwiczak@allegro.pl
   @ewabielskapoz
   Aleksander Grzyb
   iOS Developer
   aleksander.grzyb@allegro.pl
   @aleksandergrzyb
   3
   

   View Slide

3. 4
   Allegro application in Poland
   

   View Slide

4. 5
   Allegro application in Poland
   

   View Slide

5. 6
   Allegro application in Poland
   

   View Slide

6. 7
   Static analysis
   Mutation testing
   Snapshot tests
   Functional tests
   Continuous delivery
   Phased release
   A/B tests
   Usability tests
   Problem - test solution
   

   View Slide

7. 8
   1. Complex code
   Static analysis
   2. Evergreen unit test
   Mutation testing
   3. Testing views
   Snapshot tests
   4. Change management
   Functional tests
   5. Is RC is working?
   Continuous delivery
   6. Laboratory conditions
   Phased release
   7. Decision making
   A/B tests
   8. How users use our app?
   Usability tests
   Problem - test solution
   

   View Slide

8. 9
   1. Complex code
   Static analysis
   Problem - test solution
   

   View Slide

9. 10
   Problem 1:
   

   View Slide

10. 11
    Problem 1: Complex code
    

    View Slide

11. View Slide

12. 13
    Problem 1: Complex code
    Solution: SwiftLint
    https://github.com/realm/SwiftLint
    

    View Slide

13. 14
    Problem 1: Complex code
    Solution: SwiftLint
    https://github.com/realm/SwiftLint
    

    View Slide

14. 15
    Problem 1: Complex code
    Solution: SwiftLint
    OFFER
    LISTING
    SEARCH CART
    ...
    HOME
    ALLEGRO
    

    View Slide

15. 16
    Problem 1: Complex code
    Solution: SwiftLint
    SEARCH
    

    View Slide

16. 17
    Problem 1: Complex code
    Solution: SwiftLint
    too short constant name
    

    View Slide

17. 18
    Problem 1: Complex code
    Solution: SwiftLint
    too long variable name
    too short constant name
    

    View Slide

18. 19
    Problem 1: Complex code
    Solution: SwiftLint
    too long variable name
    too short constant name
    

    View Slide

19. 20
    Problem 1: Complex code
    Solution: SwiftLint
    SEARCH
    

    View Slide

20. 21
    Problem 1: Complex code
    SwiftFormat + SwiftLint
    https://github.com/nicklockwood/SwiftFormat
    

    View Slide

21. 22
    Problem 1: Complex code
    SwiftFormat + SwiftLint
    OBSERVED
    LISTING
    OFFER CART
    ...
    HOME
    ALLEGRO
    

    View Slide

22. 23
    Problem 1: Complex code
    SwiftFormat + SwiftLint
    

    View Slide

23. respect modular
    architecture
    24
    Problem 1: Complex code
    Solution: SwiftLint remarks
    

    View Slide

24. respect modular
    architecture
    run locally after each
    build
    25
    Problem 1: Complex code
    Solution: SwiftLint remarks
    

    View Slide

25. respect modular
    architecture
    format obvious rules,
    lint less trivial
    run locally after each
    build
    26
    Problem 1: Complex code
    Solution: SwiftLint remarks
    

    View Slide

26. respect modular
    architecture
    format obvious rules,
    lint less trivial
    run locally after each
    build
    27
    Problem 1: Complex code
    Solution: SwiftLint remarks
    apply to production
    code and tests
    

    View Slide

27. respect modular
    architecture
    format obvious rules,
    lint less trivial
    use the same versions
    of formatter & linter
    run locally after each
    build
    apply to production
    code and tests
    28
    Problem 1: Complex code
    Solution: SwiftLint remarks
    

    View Slide

28. 29
    1. Complex code
    Static analysis
    2. Evergreen unit test
    Mutation testing
    Problem - test solution
    

    View Slide

29. 30
    http://cleanswifter.com/asynchronous-ios-unit-test-tutorial/
    

    View Slide

30. 31
    http://cleanswifter.com/asynchronous-ios-unit-test-tutorial/
    

    View Slide

31. View Slide

32. View Slide

33. 34
    Problem 2: Evergreen unit tests
    

    View Slide

34. 35
    Problem 2: Evergreen unit tests
    

    View Slide

35. 36
    Problem 2: Evergreen unit tests
    Solution: Mutation testing
    function
    

    View Slide

36. 37
    Problem 2: Evergreen unit tests
    Solution: Mutation testing
    function mutant
    mutation
    

    View Slide

37. 38
    Problem 2: Evergreen unit tests
    Solution: Mutation testing
    function mutant
    mutation
    

    View Slide

38. 39
    Problem 2: Evergreen unit tests
    Solution: Mutation testing
    change function logic
    

    View Slide

39. 40
    Problem 2: Evergreen unit tests
    Solution: Mutation testing
    change function logic
    

    View Slide

40. 41
    Problem 2: Evergreen unit tests
    Solution: Mutation testing
    negate conditions
    

    View Slide

41. 42
    Problem 2: Evergreen unit tests
    Solution: Mutation testing
    change mathematical operators
    

    View Slide

42. 43
    Problem 2: Evergreen unit tests
    Solution: Mutation testing remarks
    code coverage says little
    about test quality
    

    View Slide

43. 44
    Problem 2: Evergreen unit tests
    Solution: Mutation testing remarks
    code coverage says little
    about test quality
    use manual mutation in
    non-trivial test cases
    

    View Slide

44. code coverage says little
    about test quality
    not all mutations
    add value
    use manual mutation in
    non-trivial test cases
    45
    Problem 2: Evergreen unit tests
    Solution: Mutation testing remarks
    

    View Slide

45. code coverage says little
    about test quality
    not all mutations
    add value
    use manual mutation in
    non-trivial test cases
    46
    Problem 2: Evergreen unit tests
    Solution: Mutation testing remarks
    automated mutation
    testing
    Problem 2: Evergreen unit tests
    Solution: Mutation testing
    https://github.com/mull-project/mull
    

    View Slide

46. View Slide

47. 48
    1. Complex code
    Static analysis
    2. Evergreen unit test
    Mutation testing
    3. Testing views
    Snapshot tests
    Problem - test solution
    

    View Slide

48. 49
    Problem 3: Testing views
    

    View Slide

49. 50
    Problem 3: Testing views
    Solution: Snapshot testing via iOSSnapshotTestCase
    https://github.com/uber/ios-snapshot-test-case
    

    View Slide

50. 51
    reference image
    January, 2017 January, 2018
    Problem 3: Testing views
    

    View Slide

51. 52
    reference image recorded image
    January, 2018
    January, 2017
    Problem 3: Testing views
    

    View Slide

52. 53
    reference image recorded image
    diff
    January, 2017 January, 2018
    Problem 3: Testing views
    

    View Slide

53. 54
    reference image recorded image
    January, 2018
    January, 2017
    Problem 3: Testing views
    

    View Slide

54. 55
    Problem 3: Testing views
    

    View Slide

55. 56
    Problem 3: Testing views
    

    View Slide

56. 57
    Problem 3: Testing views
    

    View Slide

57. 58
    Problem 3: Testing views
    

    View Slide

58. 59
    Problem 3: Testing views
    

    View Slide

59. 60
    Problem 3: Testing views
    

    View Slide

60. 61
    Problem 3: Testing views
    Mocking size classes
    

    View Slide

61. 62
    Problem 3: Testing views
    Mocking size classes
    

    View Slide

62. 63
    Problem 3: Testing views
    Mocking size classes
    

    View Slide

63. 64
    Problem 3: Testing views
    Mocking dynamic type font size
    

    View Slide

64. 65
    Problem 3: Testing views
    Mocking dynamic type font size
    http://ocmock.org
    

    View Slide

65. 66
    Problem 3: Testing views
    Mocking dynamic type font size
    http://ocmock.org
    

    View Slide

66. 67
    Problem 3: Testing views
    Mocking dynamic type font size
    http://ocmock.org
    

    View Slide

67. 68
    Problem 3: Testing views
    

    View Slide

68. 69
    Problem 3: Testing views
    Snapshot tests overhead
    snapshot tests
    all tests
    

    View Slide

69. and for a bigger view, one test can take over 1 second
    70
    Problem 3: Testing views
    Snapshot tests overhead
    

    View Slide

70. 71
    Problem 3: Testing views
    Snapshot tests overhead
    OBSERVED MODULE
    LISTING MODULE
    OFFER MODULE CART MODULE
    ...
    HOME MODULE
    ALLEGRO APP
    

    View Slide

71. quickly verify view’s
    layout
    72
    Problem 3: Testing views
    Snapshot tests remarks
    

    View Slide

72. quickly verify view’s
    layout
    test accessibility
    73
    Problem 3: Testing views
    Snapshot tests remarks
    

    View Slide

73. quickly verify view’s
    layout
    be careful when
    testing large views
    test accessibility
    74
    Problem 3: Testing views
    Snapshot tests remarks
    

    View Slide

74. quickly verify view’s
    layout
    be careful when
    testing large views
    test accessibility
    75
    measure your tests
    Problem 3: Testing views
    Snapshot tests remarks
    

    View Slide

75. 76
    1. Complex code
    Static analysis
    2. Evergreen unit test
    Mutation testing
    3. Testing views
    Snapshot tests
    4. Change management
    Functional tests
    Problem - test solution
    

    View Slide

76. 77
    Problem 4: Predicting consequences of change
    

    View Slide

77. 78
    Problem 4: Predicting consequences of change
    

    View Slide

78. 79
    Problem 4: Predicting consequences of change
    

    View Slide

79. 80
    Problem 4: Predicting consequences of change
    Solution: End-to-end UI tests
    LISTING OFFER
    HOME
    ...
    CART
    ALLEGRO
    SEARCH
    

    View Slide

80. 81
    Problem 4: Predicting consequences of change
    Solution: End-to-end UI tests remarks
    KIF framework
    https://github.com/kif-framework/KIF
    

    View Slide

81. 82
    Problem 4: Predicting consequences of change
    Solution: End-to-end UI tests remarks
    KIF framework
    https://github.com/kif-framework/KIF
    

    View Slide

82. 83
    Problem 4: Predicting consequences of change
    Solution: End-to-end UI tests remarks
    KIF framework
    separate test target
    

    View Slide

83. 84
    Problem 4: Predicting consequences of change
    Solution: End-to-end UI tests remarks
    KIF framework
    separate test target
    flaky tests
    

    View Slide

84. 85
    Problem 4: Predicting consequences of change
    Solution: UI tests with mocked API
    LISTING OFFER
    HOME
    ...
    CART
    ALLEGRO
    SEARCH
    

    View Slide

85. Problem 4: Predicting consequences of change
    Solution: UI tests with mocked API
    86
    JSON
    responses
    Backend
    

    View Slide

86. Problem 4: Predicting consequences of change
    Solution: Scheme validation
    87
    JSON files
    JSON scheme
    

    View Slide

87. Problem 4: Predicting consequences of change
    Solution: Scheme validation
    88
    JSON files
    JSON scheme
    Configuration
    file
    

    View Slide

88. Problem 4: Predicting consequences of change
    Solution: Scheme validation
    89
    JSON files
    JSON scheme
    Configuration
    file
    Backend
    request
    

    View Slide

89. Problem 4: Predicting consequences of change
    Solution: Scheme validation
    90
    JSON files
    JSON scheme
    Configuration
    file
    Backend
    request
    Remote
    validation
    

    View Slide

90. Problem 4: Predicting consequences of change
    Solution: Scheme validation
    91
    Configuration
    file
    Remote
    validation
    JSON scheme
    API response
    JSON file
    

    View Slide

91. Problem 4: Predicting consequences of change
    Solution: Scheme validation
    92
    Configuration
    file
    Remote
    validation
    JSON scheme
    API response
    JSON file
    

    View Slide

92. Problem 4: Predicting consequences of change
    Solution: Scheme validation
    93
    Configuration
    file
    Remote
    validation
    JSON file
    JSON scheme
    API response
    

    View Slide

93. 94
    Problem 4: Predicting consequences of change
    Solution: UI tests with mocked API remarks
    faster, more reliable
    UI tests
    

    View Slide

94. 95
    Problem 4: Predicting consequences of change
    Solution: UI tests with mocked API remarks
    faster, more reliable
    UI tests
    relatively slow test
    coverage progress
    

    View Slide

95. 96
    Problem 4: Predicting consequences of change
    Solution: UI tests with mocked API remarks
    faster, more reliable
    UI tests
    relatively slow test
    coverage progress
    tests not critical
    to merge
    

    View Slide

96. 97
    Problem 4: Predicting consequences of change
    Solution: UI tests improvements
    More bugs detected
    10%
    comparison of 10 real API
    and mocked API UI test
    runs
    Less flaky tests
    70%
    comparison of 10 real API
    and mocked API UI test
    runs
    

    View Slide

97. 98
    1. Complex code
    Static analysis
    2. Evergreen unit test
    Mutation testing
    3. Testing views
    Snapshot tests
    4. Change management
    Functional tests
    5. Is RC is working?
    Continuous delivery
    Problem - test solution
    

    View Slide

98. 99
    Problem 5: Is our release candidate working?
    

    View Slide

99. 10
    0
    Problem 5: Is our release candidate working?
    Solution: Dogfooding
    

    View Slide

100. 10
     1
     Problem 5: Is our release candidate working?
     Solution: Dogfooding
     

     View Slide

101. 10
     2
     Problem 5: Is our release candidate working?
     Solution: Dogfooding
     

     View Slide

102. 10
     3
     Problem 5: Is our release candidate working?
     Deployment process
     

     View Slide

103. 10
     4
     Problem 5: Is our release candidate working?
     Making our deployment reliable
     

     View Slide

104. 10
     5
     1. Crash on production
     Static analysis
     2. Evergreen unit test
     Mutation testing
     3. Testing views
     Snapshot tests
     4. Change management
     Functional tests
     5. Is RC is working?
     Continuous delivery
     6. Laboratory conditions
     Phased release
     Problem - test solution
     

     View Slide

105. 10
     6
     Problem 6: Testing in laboratory condition
     

     View Slide

106. 10
     7
     Problem 6: Testing in laboratory condition
     ENG “After latest release
     application crashes - soft 9.0”
     Elen Musk
     

     View Slide

107. 10
     8
     Problem 6: Testing in laboratory condition
     Elen Musk
     ENG “After latest release
     application crashes - soft 9.0”
     

     View Slide

108. 10
     9
     Problem 6: Testing in laboratory condition
     Solution: Phased release
     

     View Slide

109. 11
     0
     Problem 6: Testing in laboratory condition
     Solution: Phased release
     https://www.smithsonianmag.com/smart-news/story-real-canary-coal-mine-180961570/
     

     View Slide

110. 11
     1
     1. Complex code
     Static analysis
     2. Evergreen unit test
     Mutation testing
     3. Testing views
     Snapshot tests
     4. Change management
     Functional tests
     5. Is RC is working?
     Continuous delivery
     6. Laboratory conditions
     Phased release
     7. Decision making
     A/B tests
     Problem - test solution
     

     View Slide

111. 112
     Problem 7: Decision making
     

     View Slide

112. 113
     Problem 7: Decision making
     100% recommends this seller
     

     View Slide

113. 114
     Problem 7: Decision making
     

     View Slide

114. 115
     Problem 7: Decision making
     see all reviews
     

     View Slide

115. 116
     Problem 7: Decision making
     

     View Slide

116. 117
     Problem 7: Decision making
     Solution: A/B testing
     https://www.optimizely.com
     

     View Slide

117. 118
     Problem 7: Decision making
     OFFER
     LISTING
     SEARCH
     OFFER
     COMPARISON
     ...
     HOME
     ALLEGRO
     

     View Slide

118. 119
     Problem 7: Decision making
     A/B test runs in Offer
     Comparison module
     OFFER
     LISTING
     SEARCH
     OFFER
     COMPARISON
     ...
     HOME
     ALLEGRO
     

     View Slide

119. 120
     Problem 7: Decision making
     Optimizely is linked to
     main app
     OFFER
     LISTING
     SEARCH
     OFFER
     COMPARISON
     ...
     HOME
     ALLEGRO
     A/B test runs in Offer
     Comparison module
     

     View Slide

120. 121
     Problem 7: Decision making
     OFFER
     COMPARISON
     

     View Slide

121. 122
     Problem 7: Decision making
     OFFER
     COMPARISON
     

     View Slide

122. 123
     Problem 7: Decision making
     OFFER
     COMPARISON
     

     View Slide

123. 124
     Problem 7: Decision making
     ALLEGRO
     

     View Slide

124. 125
     Problem 7: Decision making
     ALLEGRO
     

     View Slide

125. 126
     Problem 7: Decision making
     ALLEGRO
     

     View Slide

126. 127
     1. Complex code
     Static analysis
     2. Evergreen unit test
     Mutation testing
     3. Testing views
     Snapshot tests
     4. Change management
     Functional tests
     5. Is RC is working?
     Continuous delivery
     6. Laboratory conditions
     Phased release
     7. Decision making
     A/B tests
     8. How users use our app?
     Usability tests
     Problem - test solution
     

     View Slide

127. 128
     Problem 8: Do we have the right product?
     “This app is a toy.
     I have nothing to
     say...”
     “Very bad. After updates it’s
     only worse not better...”
     Elton Johny Tejlor Swift
     

     View Slide

128. 129
     Problem 8: Does the user know how to use our app?
     Solution: Usability testing
     One-way mirror
     

     View Slide

129. 130
     Problem 8: Does the user know how to use our app?
     One-way mirror
     

     View Slide

130. 131
     Problem 8: Does the user know how to use our app?
     Solution: Usability testing
     One-way mirror
     

     View Slide

131. 132
     Problem 8: Does the user know how to use our app?
     Solution: Usability testing
     One-way mirror
     

     View Slide

132. 133
     1. Complex code
     Static analysis
     2. Evergreen unit test
     Mutation testing
     3. Testing views
     Snapshot tests
     4. Change management
     Functional tests
     5. Is RC is working?
     Continuous delivery
     6. Laboratory conditions
     Phased release
     7. Decision making
     A/B tests
     8. How users use our app?
     Usability tests
     Problem - test solution
     

     View Slide

133. 134
     Thank you for your attention!
     Ewa Ludwiczak
     Senior Software Test Engineer
     ewa.ludwiczak@allegro.pl
     @ewabielskapoz
     Aleksander Grzyb
     iOS Developer
     aleksander.grzyb@allegro.pl
     @aleksandergrzyb
     

     View Slide