UI Testing over the years

UI testing has been around since iOS 4, but it never really caught on. I've asked various members of the iOS community about their experience and tips.
Join me on a trip of different frameworks, strategies and tools to find a solution that fits your app. You will hear about KIF, EarlGrey, XCU and various open source projects to help you write better tests, to parallelize them or even to remind you to write tests.

This talk was hand-crafted and recorded for App Builders 2020. Video: https://vimeo.com/416980950

Transcript

1. UI Testing over the years App Builders 2020 Peter Steinberger https://twitter.com/steipete
2. PSPDFKit https://pspdfkit.com - The Complete PDF Solution You Can Rely On
3. Scope — Manual Tests — UI Tests — Snapshot Tests — Integration Tests — Unit Tests UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
4. Source: iOS Dev Survey, https://iosdevsurvey.com/2019/13-testing-and-ci-cd/
5. UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
6. What framework(s) do you use for UI testing? — XCUITest — KIF — EarlGrey (v1/v2) — Xamarin.UITest — Appium — Calabash — Other UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
7. How reliable are your UI tests? — ! — " — # — $ — % https://twitter.com/steipete/status/1255021187064893440
8. Do you distribute your tests? — No (One test at a time) — Parallel Simulators — Distribute across Machines — Distribute across machines + Parallel Simulators UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
9. At Funding Circle we have been writing UI tests since the beginning. We have a few hundreds currently and they focus on the frequent paths. They take about 30 min to run (not parallelised yet). We still use KIF We suffered the usual pain points (hard to maintain, flaky, etc), but we have always managed to keep them under control. Every PR has to come with the relevant automated tests in order to be merged. — Edu Caselles | Funding Circle Group https://twitter.com/CasellesEdu/status/1255185884045082624
10. KIF: Keep It Functional — Written in 2011 by Square — Lots of private API — Busy waiting — Uses accessiblityLabel UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
11. Leena Snidate / Codenomicon - http://heartbleed.com/heartbleed.svg
12. KIF: Keep It Functional — Written in 2011 by Square — Lots of private API — Busy waiting — Uses accessiblityLabel to query views UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
13. ! Create your own UITouch extension UITouch { public init!(in view: UIView!) public init!(at point: CGPoint, in view: UIView!) open func setLocationInWindow(_ location: CGPoint) open func setPhaseAndUpdateTimestamp(_ phase: UITouch.Phase) } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
15. https://developer.apple.com/documentation/uikit/uitouch/phase
16. https://developer.apple.com/documentation/uikit/uitouch/phase/regionentered
20. ! Accessibility 101 — accessiblityLabel — accessiblityIdentifier - value, traits, frame, hint - container, containerSpace UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
21. UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
22. UIAutomation — Added in iOS 4 (2010) — Removed in iOS 10 (2016) — JavaScript-based ! UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
23. http://blog.manbolo.com/2012/04/08/ios-automated-tests-with-uiautomation
24. var testName = "Test 1"; var target = UIATarget.localTarget(); var app = target.frontMostApp(); var window = app.mainWindow(); UIALogger.logStart(testName); // select the elements UIALogger.logMessage("Select the first tab"); var tabBar = app.tabBar(); var selectedTabName = tabBar.selectedButton().name(); if (selectedTabName != "First") { tabBar.buttons()["First"].tap(); } // tap on the text fiels UIALogger.logMessage("Tap on the text field now"); var recipeName = "Unusually Long Name for a Recipe"; window.textFields()[0].setValue(recipeName); target.delay(2); // tap on the text fiels UIALogger.logMessage("Dismiss the keyboard"); app.keyboard().buttons()["return"].tap(); var textValue = window.staticTexts()["RecipeName"].value(); if (textValue === recipeName){ UIALogger.logPass(testName); } else { app.logElementTree(); UIALogger.logFail(testName); } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
25. var testName = "Test 1" var target = UIATarget.localTarget() var app = target.frontMostApp() var window = app.mainWindow() UIALogger.logStart(testName) // select the elements UIALogger.logMessage("Select the first tab") var tabBar = app.tabBar() var selectedTabName = tabBar.selectedButton().name() if (selectedTabName != "First") { tabBar.buttons()["First"].tap() } // tap on the text fiels UIALogger.logMessage("Tap on the text field now") var recipeName = "Unusually Long Name for a Recipe" window.textFields()[0].setValue(recipeName) target.delay(2) // tap on the text fiels UIALogger.logMessage("Dismiss the keyboard") app.keyboard().buttons()["return"].tap() var textValue = window.staticTexts()["RecipeName"].value() if (textValue == recipeName) { UIALogger.logPass(testName) } else { app.logElementTree() UIALogger.logFail(testName) } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
26. https://developer.apple.com/documentation/uikit/uiaccessibilityidentification/1623132-accessibilityidentifier
27. https://www.shutterstock.com/image-photo/plain-simple-black-white-boxes-copyspace-47099215
29. func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplicationLaunchOptionsKey: Any]?) -> Bool { UIApplication.shared.keyWindow?.layer.speed = 100 return true } https://pspdfkit.com/blog/2016/running-ui-tests-with-ludicrous-speed/
30. Busy Waiting testWithViewController(pdfController) { pdfController.tapOnAnnotation(fileLinkAnnotation) // Verify that our web view controller is visible waitForCondition(pdfController.isControllerClassVisible(PSPDFWebViewController.self)) pdfController.dismissViewControllerAnimated(false, completion: nil) if #available(iOS 9, *) { pdfController.tapOnAnnotation(webLinkAnnotation) waitForCondition(pdfController.isControllerClassVisible(SFSafariViewController.self)) pdfController.dismissViewControllerAnimated(false, completion: nil) } } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
31. https://pspdfkit.com/blog/2019/open-links-in-safari-not-safari-view-controller/
32. Boolean CTTRunLoopRunUntil(Boolean(^fulfilled_)(), Boolean polling_, CFTimeInterval timeout_) { // Loop Observer Callback __block Boolean fulfilled = NO; void (^beforeWaiting) (CFRunLoopObserverRef observer, CFRunLoopActivity activity) = ^(CFRunLoopObserverRef observer, CFRunLoopActivity activity) { assert(!fulfilled); //RunLoop should be stopped after condition is fulfilled. // Check Condition fulfilled = fulfilled_(); if(fulfilled) { // Condition fulfilled: stop RunLoop now. CFRunLoopStop(CFRunLoopGetCurrent()); } else if(polling_) { // Condition not fulfilled, and we are polling: prevent RunLoop from waiting and continue looping. CFRunLoopWakeUp(CFRunLoopGetCurrent()); } }; CFRunLoopObserverRef observer = CFRunLoopObserverCreateWithHandler(NULL, kCFRunLoopBeforeWaiting, true, 0, beforeWaiting); CFRunLoopAddObserver(CFRunLoopGetCurrent(), observer, kCFRunLoopDefaultMode); // Run! CFRunLoopRunInMode(kCFRunLoopDefaultMode, timeout_, false); CFRunLoopRemoveObserver(CFRunLoopGetCurrent(), observer, kCFRunLoopDefaultMode); CFRelease(observer); return fulfilled; } https://bou.io/CTTRunLoopRunUntil.html
33. func CTTRunLoopRunUntil(_ fulfilled_: @escaping () -> Bool, _ polling_: Bool, _ timeout_: CFTimeInterval) -> Bool { // Loop Observer Callback var fulfilled = false let beforeWaiting: ((_ observer: CFRunLoopObserver?, _ activity: CFRunLoopActivity) -> Void)? = { observer, activity in assert(!fulfilled) //RunLoop should be stopped after condition is fulfilled. // Check Condition fulfilled = fulfilled_() if fulfilled { // Condition fulfilled: stop RunLoop now. CFRunLoopStop(CFRunLoopGetCurrent()) } else if polling_ { // Condition not fulfilled, and we are polling: prevent RunLoop from waiting and continue looping. CFRunLoopWakeUp(CFRunLoopGetCurrent()) } } let observer = CFRunLoopObserverCreateWithHandler(nil, CFOptionFlags(CFRunLoopActivity.beforeWaiting.rawValue), true, CFIndex(0), beforeWaiting) CFRunLoopAddObserver(CFRunLoopGetCurrent(), observer, CFRunLoopMode.defaultMode) // Run! CFRunLoopRunInMode(CFRunLoopMode.defaultMode, timeout_, false) CFRunLoopRemoveObserver(CFRunLoopGetCurrent(), observer, CFRunLoopMode.defaultMode) return fulfilled } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
34. https://stackoverflow.com/questions/56154827/nsattributedstring-from-html-on-main-thread-behaves-as-if-multithreading
35. ! Square UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
36. We have over 8000 tests consisting of a mixture of Unit and UI Tests in our codebase that run on devices housed in an on-site lab. A test suite of around 200 tests run with every PR. If a test fails, it is automatically rerun to account for flakiness. We're still doing manual testing to account for hard to automate paths: I doubt those devices take out their airpods while watching Tiger King. — Carola Nitz | Netflix https://twitter.com/CaroN
37. At Twitch we have about 350 iOS UI tests, taking ~29 min with parallel Simulators on Jenkins. Running them on devices has been both flaky and slower every time we tried. We retry failed tests because we’ve always struggled with flakiness. Our UI tests are for the most part end-to-end integration tests that hit the server. Just because the UI tests are passing doesn’t mean the app has no bugs. But it’s been really valuable as a “first line of defense” to catch crashes earlier and keep master relatively healthy, both important in a team of 18 engineers. — Javier Soto | Twitch https://twitter.com/Javi/status/1255123189690068993
38. https://developer.apple.com/videos/play/wwdc2016/409/
39. https://github.com/lyndsey-ferguson/fastlane-plugin-test_center
40. https://heartbeat.fritz.ai/the-fast-furious-and-flaky-continuous-integration-with-xcode-10-parallel-tests-2da71fd48cb1
41. https://github.com/linkedin/bluepill
43. https://twitter.com/steipete/status/410078113205592065
44. https://ortastuff.s3.amazonaws.com/gifs/marin-ok.gif
45. https://gist.github.com/KrauseFx/d8cb973ea2cab3f4f835
46. https://danger.systems/swift/
47. UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
48. https://pspdfkit.com/blog/2016/converting-xcode-test-results-the-fast-way/
49. https://github.com/fastlane-community/trainer
50. https://github.com/thii/xcbeautify
51. https://github.com/facebook/xctool
52. I love this one tweet calling them indeterministic; croissants are "flaky" Our UI tests also cover a lot of networking, which doesn't help. But on the other hand we implicitly test a lot more with that — a blessing and a curse. A pull request from me yesterday ran 854 "features". 1 failure. Took 35 minutes. Tests were split across 13 nodes. — Bas Broek | XING https://twitter.com/basthomas/status/1255044115512782848
53. https://github.com/Subito-it/Mendoza
54. We have a very large UI test suite. Over time this allowed us to cut our release cycle down to 1 week. Besides reducing manual verification, it proved to be extremely helpful on large under the hood refactorings. — Tomas Camin | Subito https://twitter.com/tomascamin/status/1255048867378343937
55. https://github.com/Subito-it/SBTUITestTunnel
56. https://github.com/depoon/SwiftLocalhost
57. https://github.com/google/eDistantObject
58. We've tried many different implementations of UI tests, including writing custom frameworks. Our suite would take from 4 to 6 hours. Even the most basic UI test would sometimes fail because it could not find a text field, or a label. — Eneko Alonso | Mindbody https://twitter.com/eneko/status/1255499202400002064
59. XCUI Best Practices // use accessibility identifiers let resultView = app.otherElements["view_result"] // always wait before asserting let viewExists = resultView.waitForExistence(timeout: 10) XCTAssert(viewExists) // write helpers (Page Object Pattern) class SearchPageObject { func type(query: String) -> Self func tapSend() -> ResultPage } https://medium.com/@danielcarlosce/some-good-practices-for-xcuitest-807bfe6b720d, https://medium.com/capital-one-tech/robot-pattern-testing-for-xcuitest-4c2f0c40b4ad, https:// blog.novoda.com/ui-testing-part-3/
60. https://github.com/danielCarlosCE/XCUITestExample
61. https://github.com/google/EarlGrey/issues/633
62. Prevent Keyboard Tutorials // rdar://34634970 if let defaults = UserDefaults(suiteName: "com.apple.Preferences") { // Since iOS 11 defaults.set(true, forKey:"DidShowGestureKeyboardIntroduction") // iPhone since iOS 13 defaults.set(true, forKey:"DidShowContinuousPathIntroduction") } https://github.com/google/EarlGrey/issues/633
63. ? https://twitter.com/steipete
64. UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
65. Tracking Busy Resources — Grand Central Dispatch — NSTimer / performSelector:afterDelay: — Layout Pass — View Appear/Disappear — Keyboard Transitions — CA/UI Animations — View Controller Transitions — WebView/Network Requests — Gesture Recognition — UIScrollView — Systemwide User Interaction UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
66. UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
68. UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
69. https://opensource.apple.com
70. https://opensource.apple.com/source/dyld/dyld-733.8/include/mach-o/dyld_priv.h.auto.html
71. Interposing C functions struct dyld_interpose_tuple { const void* replacement; const void* replacee; }; extern void dyld_dynamic_interpose(const struct mach_header* mh, const struct dyld_interpose_tuple array[], size_t count); https://github.com/opensource-apple/dyld/blob/master/include/mach-o/dyld_priv.h
72. https://chromium.googlesource.com/chromium/src/+/18a4f63fd5dc592a6b31f2a832de145b151adbde/media/audio/mac/coreaudiodispatchoverride.cc
73. Interposing Grand Central Dispatch __attribute__((used)) __attribute__((section("__DATA,__interpose"))) static const dyld_interpose_tuple grey_static_interpose_tuples[] = { { grey_dispatch_after, dispatch_after }, { grey_dispatch_async, dispatch_async }, { grey_dispatch_sync, dispatch_sync }, { grey_dispatch_after_f, dispatch_after_f }, { grey_dispatch_async_f, dispatch_async_f }, { grey_dispatch_sync_f, dispatch_sync_f } }; UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
74. https://github.com/facebook/fishhook
75. ✅ SwiftUI UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
76. https://github.com/Dimillian/ACHNBrowserUI/pull/89/checks
77. https://github.com/Dimillian/ACHNBrowserUI/pull/89/checks
78. Recap — KIF — accessibilityIdentifier + Inspector — UIAutomation — Accelerating Animations layer.speed — Busy Waiting — Running Single Tests -only-testing — Parallel Testing — EarlGrey 1/2 — dyld_dynamic_interpose — SwiftUI UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
79. ! Tooling UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
80. ! Thanks UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete