$30 off During Our Annual Pro Sale. View Details »

[Mobius] Layers of testing in iOS applications

[Mobius] Layers of testing in iOS applications

We want to take you on the journey through different layers of tests in an iOS app that’s used by more than half million unique users per month. Get to know the way we test it, step by step, layer by layer. We start with advanced code static analysis, continue with mutation, snapshot and functional tests. We carry on with continuous delivery, A/B tests and usability tests. We want to share with you which tools we are using and what testing practices we believe in. This is a true story driven by problems we experience at work, presented from tester’s and developer’s perspective.

Aleksander Grzyb

April 20, 2018
Tweet

More Decks by Aleksander Grzyb

Other Decks in Programming

Transcript

  1. Layers of testing
    in iOS applications
    Ewa Ludwiczak & Aleksander Grzyb
    Mobius 2018 Piter

    View Slide

  2. Ewa Ludwiczak
    Senior Software Test Engineer
    [email protected]
    @ewabielskapoz
    Aleksander Grzyb
    iOS Developer
    [email protected]
    @aleksandergrzyb
    3

    View Slide

  3. 4
    Allegro application in Poland

    View Slide

  4. 5
    Allegro application in Poland

    View Slide

  5. 6
    Allegro application in Poland

    View Slide

  6. 7
    Static analysis
    Mutation testing
    Snapshot tests
    Functional tests
    Continuous delivery
    Phased release
    A/B tests
    Usability tests
    Problem - test solution

    View Slide

  7. 8
    1. Complex code
    Static analysis
    2. Evergreen unit test
    Mutation testing
    3. Testing views
    Snapshot tests
    4. Change management
    Functional tests
    5. Is RC is working?
    Continuous delivery
    6. Laboratory conditions
    Phased release
    7. Decision making
    A/B tests
    8. How users use our app?
    Usability tests
    Problem - test solution

    View Slide

  8. 9
    1. Complex code
    Static analysis
    Problem - test solution

    View Slide

  9. 10
    Problem 1:

    View Slide

  10. 11
    Problem 1: Complex code

    View Slide

  11. 12

    View Slide

  12. 13
    Problem 1: Complex code
    Solution: SwiftLint
    https://github.com/realm/SwiftLint

    View Slide

  13. 14
    Problem 1: Complex code
    Solution: SwiftLint
    https://github.com/realm/SwiftLint

    View Slide

  14. 15
    Problem 1: Complex code
    Solution: SwiftLint
    OFFER
    LISTING
    SEARCH CART
    ...
    HOME
    ALLEGRO

    View Slide

  15. 16
    Problem 1: Complex code
    Solution: SwiftLint
    SEARCH

    View Slide

  16. 17
    Problem 1: Complex code
    Solution: SwiftLint
    too short constant name

    View Slide

  17. 18
    Problem 1: Complex code
    Solution: SwiftLint
    too long variable name
    too short constant name

    View Slide

  18. 19
    Problem 1: Complex code
    Solution: SwiftLint
    too long variable name
    too short constant name

    View Slide

  19. 20
    Problem 1: Complex code
    Solution: SwiftLint
    SEARCH

    View Slide

  20. 21
    Problem 1: Complex code
    SwiftFormat + SwiftLint
    https://github.com/nicklockwood/SwiftFormat

    View Slide

  21. 22
    Problem 1: Complex code
    SwiftFormat + SwiftLint
    OBSERVED
    LISTING
    OFFER CART
    ...
    HOME
    ALLEGRO

    View Slide

  22. 23
    Problem 1: Complex code
    SwiftFormat + SwiftLint

    View Slide

  23. respect modular
    architecture
    24
    Problem 1: Complex code
    Solution: SwiftLint remarks

    View Slide

  24. respect modular
    architecture
    run locally after each
    build
    25
    Problem 1: Complex code
    Solution: SwiftLint remarks

    View Slide

  25. respect modular
    architecture
    format obvious rules,
    lint less trivial
    run locally after each
    build
    26
    Problem 1: Complex code
    Solution: SwiftLint remarks

    View Slide

  26. respect modular
    architecture
    format obvious rules,
    lint less trivial
    run locally after each
    build
    27
    Problem 1: Complex code
    Solution: SwiftLint remarks
    apply to production
    code and tests

    View Slide

  27. respect modular
    architecture
    format obvious rules,
    lint less trivial
    use the same versions
    of formatter & linter
    run locally after each
    build
    apply to production
    code and tests
    28
    Problem 1: Complex code
    Solution: SwiftLint remarks

    View Slide

  28. 29
    1. Complex code
    Static analysis
    2. Evergreen unit test
    Mutation testing
    Problem - test solution

    View Slide

  29. 30
    http://cleanswifter.com/asynchronous-ios-unit-test-tutorial/

    View Slide

  30. 31
    http://cleanswifter.com/asynchronous-ios-unit-test-tutorial/

    View Slide

  31. 32

    View Slide

  32. 33

    View Slide

  33. 34
    Problem 2: Evergreen unit tests

    View Slide

  34. 35
    Problem 2: Evergreen unit tests

    View Slide

  35. 36
    Problem 2: Evergreen unit tests
    Solution: Mutation testing
    function

    View Slide

  36. 37
    Problem 2: Evergreen unit tests
    Solution: Mutation testing
    function mutant
    mutation

    View Slide

  37. 38
    Problem 2: Evergreen unit tests
    Solution: Mutation testing
    function mutant
    mutation

    View Slide

  38. 39
    Problem 2: Evergreen unit tests
    Solution: Mutation testing
    change function logic

    View Slide

  39. 40
    Problem 2: Evergreen unit tests
    Solution: Mutation testing
    change function logic

    View Slide

  40. 41
    Problem 2: Evergreen unit tests
    Solution: Mutation testing
    negate conditions

    View Slide

  41. 42
    Problem 2: Evergreen unit tests
    Solution: Mutation testing
    change mathematical operators

    View Slide

  42. 43
    Problem 2: Evergreen unit tests
    Solution: Mutation testing remarks
    code coverage says little
    about test quality

    View Slide

  43. 44
    Problem 2: Evergreen unit tests
    Solution: Mutation testing remarks
    code coverage says little
    about test quality
    use manual mutation in
    non-trivial test cases

    View Slide

  44. code coverage says little
    about test quality
    not all mutations
    add value
    use manual mutation in
    non-trivial test cases
    45
    Problem 2: Evergreen unit tests
    Solution: Mutation testing remarks

    View Slide

  45. code coverage says little
    about test quality
    not all mutations
    add value
    use manual mutation in
    non-trivial test cases
    46
    Problem 2: Evergreen unit tests
    Solution: Mutation testing remarks
    automated mutation
    testing
    Problem 2: Evergreen unit tests
    Solution: Mutation testing
    https://github.com/mull-project/mull

    View Slide

  46. 47

    View Slide

  47. 48
    1. Complex code
    Static analysis
    2. Evergreen unit test
    Mutation testing
    3. Testing views
    Snapshot tests
    Problem - test solution

    View Slide

  48. 49
    Problem 3: Testing views

    View Slide

  49. 50
    Problem 3: Testing views
    Solution: Snapshot testing via iOSSnapshotTestCase
    https://github.com/uber/ios-snapshot-test-case

    View Slide

  50. 51
    reference image
    January, 2017 January, 2018
    Problem 3: Testing views

    View Slide

  51. 52
    reference image recorded image
    January, 2018
    January, 2017
    Problem 3: Testing views

    View Slide

  52. 53
    reference image recorded image
    diff
    January, 2017 January, 2018
    Problem 3: Testing views

    View Slide

  53. 54
    reference image recorded image
    January, 2018
    January, 2017
    Problem 3: Testing views

    View Slide

  54. 55
    Problem 3: Testing views

    View Slide

  55. 56
    Problem 3: Testing views

    View Slide

  56. 57
    Problem 3: Testing views

    View Slide

  57. 58
    Problem 3: Testing views

    View Slide

  58. 59
    Problem 3: Testing views

    View Slide

  59. 60
    Problem 3: Testing views

    View Slide

  60. 61
    Problem 3: Testing views
    Mocking size classes

    View Slide

  61. 62
    Problem 3: Testing views
    Mocking size classes

    View Slide

  62. 63
    Problem 3: Testing views
    Mocking size classes

    View Slide

  63. 64
    Problem 3: Testing views
    Mocking dynamic type font size

    View Slide

  64. 65
    Problem 3: Testing views
    Mocking dynamic type font size
    http://ocmock.org

    View Slide

  65. 66
    Problem 3: Testing views
    Mocking dynamic type font size
    http://ocmock.org

    View Slide

  66. 67
    Problem 3: Testing views
    Mocking dynamic type font size
    http://ocmock.org

    View Slide

  67. 68
    Problem 3: Testing views

    View Slide

  68. 69
    Problem 3: Testing views
    Snapshot tests overhead
    snapshot tests
    all tests

    View Slide

  69. and for a bigger view, one test can take over 1 second
    70
    Problem 3: Testing views
    Snapshot tests overhead

    View Slide

  70. 71
    Problem 3: Testing views
    Snapshot tests overhead
    OBSERVED MODULE
    LISTING MODULE
    OFFER MODULE CART MODULE
    ...
    HOME MODULE
    ALLEGRO APP

    View Slide

  71. quickly verify view’s
    layout
    72
    Problem 3: Testing views
    Snapshot tests remarks

    View Slide

  72. quickly verify view’s
    layout
    test accessibility
    73
    Problem 3: Testing views
    Snapshot tests remarks

    View Slide

  73. quickly verify view’s
    layout
    be careful when
    testing large views
    test accessibility
    74
    Problem 3: Testing views
    Snapshot tests remarks

    View Slide

  74. quickly verify view’s
    layout
    be careful when
    testing large views
    test accessibility
    75
    measure your tests
    Problem 3: Testing views
    Snapshot tests remarks

    View Slide

  75. 76
    1. Complex code
    Static analysis
    2. Evergreen unit test
    Mutation testing
    3. Testing views
    Snapshot tests
    4. Change management
    Functional tests
    Problem - test solution

    View Slide

  76. 77
    Problem 4: Predicting consequences of change

    View Slide

  77. 78
    Problem 4: Predicting consequences of change

    View Slide

  78. 79
    Problem 4: Predicting consequences of change

    View Slide

  79. 80
    Problem 4: Predicting consequences of change
    Solution: End-to-end UI tests
    LISTING OFFER
    HOME
    ...
    CART
    ALLEGRO
    SEARCH

    View Slide

  80. 81
    Problem 4: Predicting consequences of change
    Solution: End-to-end UI tests remarks
    KIF framework
    https://github.com/kif-framework/KIF

    View Slide

  81. 82
    Problem 4: Predicting consequences of change
    Solution: End-to-end UI tests remarks
    KIF framework
    https://github.com/kif-framework/KIF

    View Slide

  82. 83
    Problem 4: Predicting consequences of change
    Solution: End-to-end UI tests remarks
    KIF framework
    separate test target

    View Slide

  83. 84
    Problem 4: Predicting consequences of change
    Solution: End-to-end UI tests remarks
    KIF framework
    separate test target
    flaky tests

    View Slide

  84. 85
    Problem 4: Predicting consequences of change
    Solution: UI tests with mocked API
    LISTING OFFER
    HOME
    ...
    CART
    ALLEGRO
    SEARCH

    View Slide

  85. Problem 4: Predicting consequences of change
    Solution: UI tests with mocked API
    86
    JSON
    responses
    Backend

    View Slide

  86. Problem 4: Predicting consequences of change
    Solution: Scheme validation
    87
    JSON files
    JSON scheme

    View Slide

  87. Problem 4: Predicting consequences of change
    Solution: Scheme validation
    88
    JSON files
    JSON scheme
    Configuration
    file

    View Slide

  88. Problem 4: Predicting consequences of change
    Solution: Scheme validation
    89
    JSON files
    JSON scheme
    Configuration
    file
    Backend
    request

    View Slide

  89. Problem 4: Predicting consequences of change
    Solution: Scheme validation
    90
    JSON files
    JSON scheme
    Configuration
    file
    Backend
    request
    Remote
    validation

    View Slide

  90. Problem 4: Predicting consequences of change
    Solution: Scheme validation
    91
    Configuration
    file
    Remote
    validation
    JSON scheme
    API response
    JSON file

    View Slide

  91. Problem 4: Predicting consequences of change
    Solution: Scheme validation
    92
    Configuration
    file
    Remote
    validation
    JSON scheme
    API response
    JSON file

    View Slide

  92. Problem 4: Predicting consequences of change
    Solution: Scheme validation
    93
    Configuration
    file
    Remote
    validation
    JSON file
    JSON scheme
    API response

    View Slide

  93. 94
    Problem 4: Predicting consequences of change
    Solution: UI tests with mocked API remarks
    faster, more reliable
    UI tests

    View Slide

  94. 95
    Problem 4: Predicting consequences of change
    Solution: UI tests with mocked API remarks
    faster, more reliable
    UI tests
    relatively slow test
    coverage progress

    View Slide

  95. 96
    Problem 4: Predicting consequences of change
    Solution: UI tests with mocked API remarks
    faster, more reliable
    UI tests
    relatively slow test
    coverage progress
    tests not critical
    to merge

    View Slide

  96. 97
    Problem 4: Predicting consequences of change
    Solution: UI tests improvements
    More bugs detected
    10%
    comparison of 10 real API
    and mocked API UI test
    runs
    Less flaky tests
    70%
    comparison of 10 real API
    and mocked API UI test
    runs

    View Slide

  97. 98
    1. Complex code
    Static analysis
    2. Evergreen unit test
    Mutation testing
    3. Testing views
    Snapshot tests
    4. Change management
    Functional tests
    5. Is RC is working?
    Continuous delivery
    Problem - test solution

    View Slide

  98. 99
    Problem 5: Is our release candidate working?

    View Slide

  99. 10
    0
    Problem 5: Is our release candidate working?
    Solution: Dogfooding

    View Slide

  100. 10
    1
    Problem 5: Is our release candidate working?
    Solution: Dogfooding

    View Slide

  101. 10
    2
    Problem 5: Is our release candidate working?
    Solution: Dogfooding

    View Slide

  102. 10
    3
    Problem 5: Is our release candidate working?
    Deployment process

    View Slide

  103. 10
    4
    Problem 5: Is our release candidate working?
    Making our deployment reliable

    View Slide

  104. 10
    5
    1. Crash on production
    Static analysis
    2. Evergreen unit test
    Mutation testing
    3. Testing views
    Snapshot tests
    4. Change management
    Functional tests
    5. Is RC is working?
    Continuous delivery
    6. Laboratory conditions
    Phased release
    Problem - test solution

    View Slide

  105. 10
    6
    Problem 6: Testing in laboratory condition

    View Slide

  106. 10
    7
    Problem 6: Testing in laboratory condition
    ENG “After latest release
    application crashes - soft 9.0”
    Elen Musk

    View Slide

  107. 10
    8
    Problem 6: Testing in laboratory condition
    Elen Musk
    ENG “After latest release
    application crashes - soft 9.0”

    View Slide

  108. 10
    9
    Problem 6: Testing in laboratory condition
    Solution: Phased release

    View Slide

  109. 11
    0
    Problem 6: Testing in laboratory condition
    Solution: Phased release
    https://www.smithsonianmag.com/smart-news/story-real-canary-coal-mine-180961570/

    View Slide

  110. 11
    1
    1. Complex code
    Static analysis
    2. Evergreen unit test
    Mutation testing
    3. Testing views
    Snapshot tests
    4. Change management
    Functional tests
    5. Is RC is working?
    Continuous delivery
    6. Laboratory conditions
    Phased release
    7. Decision making
    A/B tests
    Problem - test solution

    View Slide

  111. 112
    Problem 7: Decision making

    View Slide

  112. 113
    Problem 7: Decision making
    100% recommends this seller

    View Slide

  113. 114
    Problem 7: Decision making

    View Slide

  114. 115
    Problem 7: Decision making
    see all reviews

    View Slide

  115. 116
    Problem 7: Decision making

    View Slide

  116. 117
    Problem 7: Decision making
    Solution: A/B testing
    https://www.optimizely.com

    View Slide

  117. 118
    Problem 7: Decision making
    OFFER
    LISTING
    SEARCH
    OFFER
    COMPARISON
    ...
    HOME
    ALLEGRO

    View Slide

  118. 119
    Problem 7: Decision making
    A/B test runs in Offer
    Comparison module
    OFFER
    LISTING
    SEARCH
    OFFER
    COMPARISON
    ...
    HOME
    ALLEGRO

    View Slide

  119. 120
    Problem 7: Decision making
    Optimizely is linked to
    main app
    OFFER
    LISTING
    SEARCH
    OFFER
    COMPARISON
    ...
    HOME
    ALLEGRO
    A/B test runs in Offer
    Comparison module

    View Slide

  120. 121
    Problem 7: Decision making
    OFFER
    COMPARISON

    View Slide

  121. 122
    Problem 7: Decision making
    OFFER
    COMPARISON

    View Slide

  122. 123
    Problem 7: Decision making
    OFFER
    COMPARISON

    View Slide

  123. 124
    Problem 7: Decision making
    ALLEGRO

    View Slide

  124. 125
    Problem 7: Decision making
    ALLEGRO

    View Slide

  125. 126
    Problem 7: Decision making
    ALLEGRO

    View Slide

  126. 127
    1. Complex code
    Static analysis
    2. Evergreen unit test
    Mutation testing
    3. Testing views
    Snapshot tests
    4. Change management
    Functional tests
    5. Is RC is working?
    Continuous delivery
    6. Laboratory conditions
    Phased release
    7. Decision making
    A/B tests
    8. How users use our app?
    Usability tests
    Problem - test solution

    View Slide

  127. 128
    Problem 8: Do we have the right product?
    “This app is a toy.
    I have nothing to
    say...”
    “Very bad. After updates it’s
    only worse not better...”
    Elton Johny Tejlor Swift

    View Slide

  128. 129
    Problem 8: Does the user know how to use our app?
    Solution: Usability testing
    One-way mirror

    View Slide

  129. 130
    Problem 8: Does the user know how to use our app?
    One-way mirror

    View Slide

  130. 131
    Problem 8: Does the user know how to use our app?
    Solution: Usability testing
    One-way mirror

    View Slide

  131. 132
    Problem 8: Does the user know how to use our app?
    Solution: Usability testing
    One-way mirror

    View Slide

  132. 133
    1. Complex code
    Static analysis
    2. Evergreen unit test
    Mutation testing
    3. Testing views
    Snapshot tests
    4. Change management
    Functional tests
    5. Is RC is working?
    Continuous delivery
    6. Laboratory conditions
    Phased release
    7. Decision making
    A/B tests
    8. How users use our app?
    Usability tests
    Problem - test solution

    View Slide

  133. 134
    Thank you for your attention!
    Ewa Ludwiczak
    Senior Software Test Engineer
    [email protected]
    @ewabielskapoz
    Aleksander Grzyb
    iOS Developer
    [email protected]
    @aleksandergrzyb

    View Slide