UI Testing over the years

UI Testing over the years

UI testing has been around since iOS 4, but it never really caught on. I've asked various members of the iOS community about their experience and tips.
Join me on a trip of different frameworks, strategies and tools to find a solution that fits your app. You will hear about KIF, EarlGrey, XCU and various open source projects to help you write better tests, to parallelize them or even to remind you to write tests.

This talk was hand-crafted and recorded for App Builders 2020. Video: https://vimeo.com/416980950

832ece085bfe2c7c5b0ed6be62d7e675?s=128

Peter Steinberger

May 11, 2020
Tweet

Transcript

  1. UI Testing over the years App Builders 2020 Peter Steinberger

    https://twitter.com/steipete
  2. PSPDFKit https://pspdfkit.com - The Complete PDF Solution You Can Rely

    On
  3. Scope — Manual Tests — UI Tests — Snapshot Tests

    — Integration Tests — Unit Tests UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  4. Source: iOS Dev Survey, https://iosdevsurvey.com/2019/13-testing-and-ci-cd/

  5. UI Testing over the years — App Builders 2020 |

    Peter Steinberger — @steipete
  6. What framework(s) do you use for UI testing? — XCUITest

    — KIF — EarlGrey (v1/v2) — Xamarin.UITest — Appium — Calabash — Other UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  7. How reliable are your UI tests? — ! — "

    — # — $ — % https://twitter.com/steipete/status/1255021187064893440
  8. Do you distribute your tests? — No (One test at

    a time) — Parallel Simulators — Distribute across Machines — Distribute across machines + Parallel Simulators UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  9. At Funding Circle we have been writing UI tests since

    the beginning. We have a few hundreds currently and they focus on the frequent paths. They take about 30 min to run (not parallelised yet). We still use KIF We suffered the usual pain points (hard to maintain, flaky, etc), but we have always managed to keep them under control. Every PR has to come with the relevant automated tests in order to be merged. — Edu Caselles | Funding Circle Group https://twitter.com/CasellesEdu/status/1255185884045082624
  10. KIF: Keep It Functional — Written in 2011 by Square

    — Lots of private API — Busy waiting — Uses accessiblityLabel UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  11. Leena Snidate / Codenomicon - http://heartbleed.com/heartbleed.svg

  12. KIF: Keep It Functional — Written in 2011 by Square

    — Lots of private API — Busy waiting — Uses accessiblityLabel to query views UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  13. ! Create your own UITouch extension UITouch { public init!(in

    view: UIView!) public init!(at point: CGPoint, in view: UIView!) open func setLocationInWindow(_ location: CGPoint) open func setPhaseAndUpdateTimestamp(_ phase: UITouch.Phase) } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  14. None
  15. https://developer.apple.com/documentation/uikit/uitouch/phase

  16. https://developer.apple.com/documentation/uikit/uitouch/phase/regionentered

  17. None
  18. None
  19. None
  20. ! Accessibility 101 — accessiblityLabel — accessiblityIdentifier - value, traits,

    frame, hint - container, containerSpace UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  21. UI Testing over the years — App Builders 2020 |

    Peter Steinberger — @steipete
  22. UIAutomation — Added in iOS 4 (2010) — Removed in

    iOS 10 (2016) — JavaScript-based ! UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  23. http://blog.manbolo.com/2012/04/08/ios-automated-tests-with-uiautomation

  24. var testName = "Test 1"; var target = UIATarget.localTarget(); var

    app = target.frontMostApp(); var window = app.mainWindow(); UIALogger.logStart(testName); // select the elements UIALogger.logMessage("Select the first tab"); var tabBar = app.tabBar(); var selectedTabName = tabBar.selectedButton().name(); if (selectedTabName != "First") { tabBar.buttons()["First"].tap(); } // tap on the text fiels UIALogger.logMessage("Tap on the text field now"); var recipeName = "Unusually Long Name for a Recipe"; window.textFields()[0].setValue(recipeName); target.delay(2); // tap on the text fiels UIALogger.logMessage("Dismiss the keyboard"); app.keyboard().buttons()["return"].tap(); var textValue = window.staticTexts()["RecipeName"].value(); if (textValue === recipeName){ UIALogger.logPass(testName); } else { app.logElementTree(); UIALogger.logFail(testName); } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  25. var testName = "Test 1" var target = UIATarget.localTarget() var

    app = target.frontMostApp() var window = app.mainWindow() UIALogger.logStart(testName) // select the elements UIALogger.logMessage("Select the first tab") var tabBar = app.tabBar() var selectedTabName = tabBar.selectedButton().name() if (selectedTabName != "First") { tabBar.buttons()["First"].tap() } // tap on the text fiels UIALogger.logMessage("Tap on the text field now") var recipeName = "Unusually Long Name for a Recipe" window.textFields()[0].setValue(recipeName) target.delay(2) // tap on the text fiels UIALogger.logMessage("Dismiss the keyboard") app.keyboard().buttons()["return"].tap() var textValue = window.staticTexts()["RecipeName"].value() if (textValue == recipeName) { UIALogger.logPass(testName) } else { app.logElementTree() UIALogger.logFail(testName) } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  26. https://developer.apple.com/documentation/uikit/uiaccessibilityidentification/1623132-accessibilityidentifier

  27. https://www.shutterstock.com/image-photo/plain-simple-black-white-boxes-copyspace-47099215

  28. None
  29. func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplicationLaunchOptionsKey: Any]?) -> Bool

    { UIApplication.shared.keyWindow?.layer.speed = 100 return true } https://pspdfkit.com/blog/2016/running-ui-tests-with-ludicrous-speed/
  30. Busy Waiting testWithViewController(pdfController) { pdfController.tapOnAnnotation(fileLinkAnnotation) // Verify that our web

    view controller is visible waitForCondition(pdfController.isControllerClassVisible(PSPDFWebViewController.self)) pdfController.dismissViewControllerAnimated(false, completion: nil) if #available(iOS 9, *) { pdfController.tapOnAnnotation(webLinkAnnotation) waitForCondition(pdfController.isControllerClassVisible(SFSafariViewController.self)) pdfController.dismissViewControllerAnimated(false, completion: nil) } } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  31. https://pspdfkit.com/blog/2019/open-links-in-safari-not-safari-view-controller/

  32. Boolean CTTRunLoopRunUntil(Boolean(^fulfilled_)(), Boolean polling_, CFTimeInterval timeout_) { // Loop Observer

    Callback __block Boolean fulfilled = NO; void (^beforeWaiting) (CFRunLoopObserverRef observer, CFRunLoopActivity activity) = ^(CFRunLoopObserverRef observer, CFRunLoopActivity activity) { assert(!fulfilled); //RunLoop should be stopped after condition is fulfilled. // Check Condition fulfilled = fulfilled_(); if(fulfilled) { // Condition fulfilled: stop RunLoop now. CFRunLoopStop(CFRunLoopGetCurrent()); } else if(polling_) { // Condition not fulfilled, and we are polling: prevent RunLoop from waiting and continue looping. CFRunLoopWakeUp(CFRunLoopGetCurrent()); } }; CFRunLoopObserverRef observer = CFRunLoopObserverCreateWithHandler(NULL, kCFRunLoopBeforeWaiting, true, 0, beforeWaiting); CFRunLoopAddObserver(CFRunLoopGetCurrent(), observer, kCFRunLoopDefaultMode); // Run! CFRunLoopRunInMode(kCFRunLoopDefaultMode, timeout_, false); CFRunLoopRemoveObserver(CFRunLoopGetCurrent(), observer, kCFRunLoopDefaultMode); CFRelease(observer); return fulfilled; } https://bou.io/CTTRunLoopRunUntil.html
  33. func CTTRunLoopRunUntil(_ fulfilled_: @escaping () -> Bool, _ polling_: Bool,

    _ timeout_: CFTimeInterval) -> Bool { // Loop Observer Callback var fulfilled = false let beforeWaiting: ((_ observer: CFRunLoopObserver?, _ activity: CFRunLoopActivity) -> Void)? = { observer, activity in assert(!fulfilled) //RunLoop should be stopped after condition is fulfilled. // Check Condition fulfilled = fulfilled_() if fulfilled { // Condition fulfilled: stop RunLoop now. CFRunLoopStop(CFRunLoopGetCurrent()) } else if polling_ { // Condition not fulfilled, and we are polling: prevent RunLoop from waiting and continue looping. CFRunLoopWakeUp(CFRunLoopGetCurrent()) } } let observer = CFRunLoopObserverCreateWithHandler(nil, CFOptionFlags(CFRunLoopActivity.beforeWaiting.rawValue), true, CFIndex(0), beforeWaiting) CFRunLoopAddObserver(CFRunLoopGetCurrent(), observer, CFRunLoopMode.defaultMode) // Run! CFRunLoopRunInMode(CFRunLoopMode.defaultMode, timeout_, false) CFRunLoopRemoveObserver(CFRunLoopGetCurrent(), observer, CFRunLoopMode.defaultMode) return fulfilled } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  34. https://stackoverflow.com/questions/56154827/nsattributedstring-from-html-on-main-thread-behaves-as-if-multithreading

  35. ! Square UI Testing over the years — App Builders

    2020 | Peter Steinberger — @steipete
  36. We have over 8000 tests consisting of a mixture of

    Unit and UI Tests in our codebase that run on devices housed in an on-site lab. A test suite of around 200 tests run with every PR. If a test fails, it is automatically rerun to account for flakiness. We're still doing manual testing to account for hard to automate paths: I doubt those devices take out their airpods while watching Tiger King. — Carola Nitz | Netflix https://twitter.com/CaroN
  37. At Twitch we have about 350 iOS UI tests, taking

    ~29 min with parallel Simulators on Jenkins. Running them on devices has been both flaky and slower every time we tried. We retry failed tests because we’ve always struggled with flakiness. Our UI tests are for the most part end-to-end integration tests that hit the server. Just because the UI tests are passing doesn’t mean the app has no bugs. But it’s been really valuable as a “first line of defense” to catch crashes earlier and keep master relatively healthy, both important in a team of 18 engineers. — Javier Soto | Twitch https://twitter.com/Javi/status/1255123189690068993
  38. https://developer.apple.com/videos/play/wwdc2016/409/

  39. https://github.com/lyndsey-ferguson/fastlane-plugin-test_center

  40. https://heartbeat.fritz.ai/the-fast-furious-and-flaky-continuous-integration-with-xcode-10-parallel-tests-2da71fd48cb1

  41. https://github.com/linkedin/bluepill

  42. None
  43. https://twitter.com/steipete/status/410078113205592065

  44. https://ortastuff.s3.amazonaws.com/gifs/marin-ok.gif

  45. https://gist.github.com/KrauseFx/d8cb973ea2cab3f4f835

  46. https://danger.systems/swift/

  47. UI Testing over the years — App Builders 2020 |

    Peter Steinberger — @steipete
  48. https://pspdfkit.com/blog/2016/converting-xcode-test-results-the-fast-way/

  49. https://github.com/fastlane-community/trainer

  50. https://github.com/thii/xcbeautify

  51. https://github.com/facebook/xctool

  52. I love this one tweet calling them indeterministic; croissants are

    "flaky" Our UI tests also cover a lot of networking, which doesn't help. But on the other hand we implicitly test a lot more with that — a blessing and a curse. A pull request from me yesterday ran 854 "features". 1 failure. Took 35 minutes. Tests were split across 13 nodes. — Bas Broek | XING https://twitter.com/basthomas/status/1255044115512782848
  53. https://github.com/Subito-it/Mendoza

  54. We have a very large UI test suite. Over time

    this allowed us to cut our release cycle down to 1 week. Besides reducing manual verification, it proved to be extremely helpful on large under the hood refactorings. — Tomas Camin | Subito https://twitter.com/tomascamin/status/1255048867378343937
  55. https://github.com/Subito-it/SBTUITestTunnel

  56. https://github.com/depoon/SwiftLocalhost

  57. https://github.com/google/eDistantObject

  58. We've tried many different implementations of UI tests, including writing

    custom frameworks. Our suite would take from 4 to 6 hours. Even the most basic UI test would sometimes fail because it could not find a text field, or a label. — Eneko Alonso | Mindbody https://twitter.com/eneko/status/1255499202400002064
  59. XCUI Best Practices // use accessibility identifiers let resultView =

    app.otherElements["view_result"] // always wait before asserting let viewExists = resultView.waitForExistence(timeout: 10) XCTAssert(viewExists) // write helpers (Page Object Pattern) class SearchPageObject { func type(query: String) -> Self func tapSend() -> ResultPage } https://medium.com/@danielcarlosce/some-good-practices-for-xcuitest-807bfe6b720d, https://medium.com/capital-one-tech/robot-pattern-testing-for-xcuitest-4c2f0c40b4ad, https:// blog.novoda.com/ui-testing-part-3/
  60. https://github.com/danielCarlosCE/XCUITestExample

  61. https://github.com/google/EarlGrey/issues/633

  62. Prevent Keyboard Tutorials // rdar://34634970 if let defaults = UserDefaults(suiteName:

    "com.apple.Preferences") { // Since iOS 11 defaults.set(true, forKey:"DidShowGestureKeyboardIntroduction") // iPhone since iOS 13 defaults.set(true, forKey:"DidShowContinuousPathIntroduction") } https://github.com/google/EarlGrey/issues/633
  63. ? https://twitter.com/steipete

  64. UI Testing over the years — App Builders 2020 |

    Peter Steinberger — @steipete
  65. Tracking Busy Resources — Grand Central Dispatch — NSTimer /

    performSelector:afterDelay: — Layout Pass — View Appear/Disappear — Keyboard Transitions — CA/UI Animations — View Controller Transitions — WebView/Network Requests — Gesture Recognition — UIScrollView — Systemwide User Interaction UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  66. UI Testing over the years — App Builders 2020 |

    Peter Steinberger — @steipete
  67. None
  68. UI Testing over the years — App Builders 2020 |

    Peter Steinberger — @steipete
  69. https://opensource.apple.com

  70. https://opensource.apple.com/source/dyld/dyld-733.8/include/mach-o/dyld_priv.h.auto.html

  71. Interposing C functions struct dyld_interpose_tuple { const void* replacement; const

    void* replacee; }; extern void dyld_dynamic_interpose(const struct mach_header* mh, const struct dyld_interpose_tuple array[], size_t count); https://github.com/opensource-apple/dyld/blob/master/include/mach-o/dyld_priv.h
  72. https://chromium.googlesource.com/chromium/src/+/18a4f63fd5dc592a6b31f2a832de145b151adbde/media/audio/mac/coreaudiodispatchoverride.cc

  73. Interposing Grand Central Dispatch __attribute__((used)) __attribute__((section("__DATA,__interpose"))) static const dyld_interpose_tuple grey_static_interpose_tuples[]

    = { { grey_dispatch_after, dispatch_after }, { grey_dispatch_async, dispatch_async }, { grey_dispatch_sync, dispatch_sync }, { grey_dispatch_after_f, dispatch_after_f }, { grey_dispatch_async_f, dispatch_async_f }, { grey_dispatch_sync_f, dispatch_sync_f } }; UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  74. https://github.com/facebook/fishhook

  75. ✅ SwiftUI UI Testing over the years — App Builders

    2020 | Peter Steinberger — @steipete
  76. https://github.com/Dimillian/ACHNBrowserUI/pull/89/checks

  77. https://github.com/Dimillian/ACHNBrowserUI/pull/89/checks

  78. Recap — KIF — accessibilityIdentifier + Inspector — UIAutomation —

    Accelerating Animations layer.speed — Busy Waiting — Running Single Tests -only-testing — Parallel Testing — EarlGrey 1/2 — dyld_dynamic_interpose — SwiftUI UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  79. ! Tooling UI Testing over the years — App Builders

    2020 | Peter Steinberger — @steipete
  80. ! Thanks UI Testing over the years — App Builders

    2020 | Peter Steinberger — @steipete