Share
Embed
UI Testing over the years
UI testing has been around since iOS 4, but it never really caught on. I've asked various members of the iOS community about their experience and tips.
Join me on a trip of different frameworks, strategies and tools to find a solution that fits your app. You will hear about KIF, EarlGrey, XCU and various open source projects to help you write better tests, to parallelize them or even to remind you to write tests.
This talk was hand-crafted and recorded for App Builders 2020. Video: https://vimeo.com/416980950
Peter Steinberger
May 11, 2020
Transcript
80 slides
-
UI Testing over the years App Builders 2020 Peter Steinberger
https://twitter.com/steipete -
Scope — Manual Tests — UI Tests — Snapshot Tests
— Integration Tests — Unit Tests UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete -
UI Testing over the years — App Builders 2020 |
Peter Steinberger — @steipete -
What framework(s) do you use for UI testing? — XCUITest
— KIF — EarlGrey (v1/v2) — Xamarin.UITest — Appium — Calabash — Other UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete -
How reliable are your UI tests? — ! — "
— # — $ — % https://twitter.com/steipete/status/1255021187064893440 -
Do you distribute your tests? — No (One test at
a time) — Parallel Simulators — Distribute across Machines — Distribute across machines + Parallel Simulators UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete -
At Funding Circle we have been writing UI tests since
the beginning. We have a few hundreds currently and they focus on the frequent paths. They take about 30 min to run (not parallelised yet). We still use KIF We suffered the usual pain points (hard to maintain, flaky, etc), but we have always managed to keep them under control. Every PR has to come with the relevant automated tests in order to be merged. — Edu Caselles | Funding Circle Group https://twitter.com/CasellesEdu/status/1255185884045082624 -
KIF: Keep It Functional — Written in 2011 by Square
— Lots of private API — Busy waiting — Uses accessiblityLabel UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete -
KIF: Keep It Functional — Written in 2011 by Square
— Lots of private API — Busy waiting — Uses accessiblityLabel to query views UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete -
! Create your own UITouch extension UITouch { public init!(in
view: UIView!) public init!(at point: CGPoint, in view: UIView!) open func setLocationInWindow(_ location: CGPoint) open func setPhaseAndUpdateTimestamp(_ phase: UITouch.Phase) } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete -
! Accessibility 101 — accessiblityLabel — accessiblityIdentifier - value, traits,
frame, hint - container, containerSpace UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete -
UI Testing over the years — App Builders 2020 |
Peter Steinberger — @steipete -
UIAutomation — Added in iOS 4 (2010) — Removed in
iOS 10 (2016) — JavaScript-based ! UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete -
var testName = "Test 1"; var target = UIATarget.localTarget(); var
app = target.frontMostApp(); var window = app.mainWindow(); UIALogger.logStart(testName); // select the elements UIALogger.logMessage("Select the first tab"); var tabBar = app.tabBar(); var selectedTabName = tabBar.selectedButton().name(); if (selectedTabName != "First") { tabBar.buttons()["First"].tap(); } // tap on the text fiels UIALogger.logMessage("Tap on the text field now"); var recipeName = "Unusually Long Name for a Recipe"; window.textFields()[0].setValue(recipeName); target.delay(2); // tap on the text fiels UIALogger.logMessage("Dismiss the keyboard"); app.keyboard().buttons()["return"].tap(); var textValue = window.staticTexts()["RecipeName"].value(); if (textValue === recipeName){ UIALogger.logPass(testName); } else { app.logElementTree(); UIALogger.logFail(testName); } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete -
var testName = "Test 1" var target = UIATarget.localTarget() var
app = target.frontMostApp() var window = app.mainWindow() UIALogger.logStart(testName) // select the elements UIALogger.logMessage("Select the first tab") var tabBar = app.tabBar() var selectedTabName = tabBar.selectedButton().name() if (selectedTabName != "First") { tabBar.buttons()["First"].tap() } // tap on the text fiels UIALogger.logMessage("Tap on the text field now") var recipeName = "Unusually Long Name for a Recipe" window.textFields()[0].setValue(recipeName) target.delay(2) // tap on the text fiels UIALogger.logMessage("Dismiss the keyboard") app.keyboard().buttons()["return"].tap() var textValue = window.staticTexts()["RecipeName"].value() if (textValue == recipeName) { UIALogger.logPass(testName) } else { app.logElementTree() UIALogger.logFail(testName) } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete -
func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplicationLaunchOptionsKey: Any]?) -> Bool
{ UIApplication.shared.keyWindow?.layer.speed = 100 return true } https://pspdfkit.com/blog/2016/running-ui-tests-with-ludicrous-speed/ -
Busy Waiting testWithViewController(pdfController) { pdfController.tapOnAnnotation(fileLinkAnnotation) // Verify that our web
view controller is visible waitForCondition(pdfController.isControllerClassVisible(PSPDFWebViewController.self)) pdfController.dismissViewControllerAnimated(false, completion: nil) if #available(iOS 9, *) { pdfController.tapOnAnnotation(webLinkAnnotation) waitForCondition(pdfController.isControllerClassVisible(SFSafariViewController.self)) pdfController.dismissViewControllerAnimated(false, completion: nil) } } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete -
Boolean CTTRunLoopRunUntil(Boolean(^fulfilled_)(), Boolean polling_, CFTimeInterval timeout_) { // Loop Observer
Callback __block Boolean fulfilled = NO; void (^beforeWaiting) (CFRunLoopObserverRef observer, CFRunLoopActivity activity) = ^(CFRunLoopObserverRef observer, CFRunLoopActivity activity) { assert(!fulfilled); //RunLoop should be stopped after condition is fulfilled. // Check Condition fulfilled = fulfilled_(); if(fulfilled) { // Condition fulfilled: stop RunLoop now. CFRunLoopStop(CFRunLoopGetCurrent()); } else if(polling_) { // Condition not fulfilled, and we are polling: prevent RunLoop from waiting and continue looping. CFRunLoopWakeUp(CFRunLoopGetCurrent()); } }; CFRunLoopObserverRef observer = CFRunLoopObserverCreateWithHandler(NULL, kCFRunLoopBeforeWaiting, true, 0, beforeWaiting); CFRunLoopAddObserver(CFRunLoopGetCurrent(), observer, kCFRunLoopDefaultMode); // Run! CFRunLoopRunInMode(kCFRunLoopDefaultMode, timeout_, false); CFRunLoopRemoveObserver(CFRunLoopGetCurrent(), observer, kCFRunLoopDefaultMode); CFRelease(observer); return fulfilled; } https://bou.io/CTTRunLoopRunUntil.html -
func CTTRunLoopRunUntil(_ fulfilled_: @escaping () -> Bool, _ polling_: Bool,
_ timeout_: CFTimeInterval) -> Bool { // Loop Observer Callback var fulfilled = false let beforeWaiting: ((_ observer: CFRunLoopObserver?, _ activity: CFRunLoopActivity) -> Void)? = { observer, activity in assert(!fulfilled) //RunLoop should be stopped after condition is fulfilled. // Check Condition fulfilled = fulfilled_() if fulfilled { // Condition fulfilled: stop RunLoop now. CFRunLoopStop(CFRunLoopGetCurrent()) } else if polling_ { // Condition not fulfilled, and we are polling: prevent RunLoop from waiting and continue looping. CFRunLoopWakeUp(CFRunLoopGetCurrent()) } } let observer = CFRunLoopObserverCreateWithHandler(nil, CFOptionFlags(CFRunLoopActivity.beforeWaiting.rawValue), true, CFIndex(0), beforeWaiting) CFRunLoopAddObserver(CFRunLoopGetCurrent(), observer, CFRunLoopMode.defaultMode) // Run! CFRunLoopRunInMode(CFRunLoopMode.defaultMode, timeout_, false) CFRunLoopRemoveObserver(CFRunLoopGetCurrent(), observer, CFRunLoopMode.defaultMode) return fulfilled } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete -
! Square UI Testing over the years — App Builders
2020 | Peter Steinberger — @steipete -
We have over 8000 tests consisting of a mixture of
Unit and UI Tests in our codebase that run on devices housed in an on-site lab. A test suite of around 200 tests run with every PR. If a test fails, it is automatically rerun to account for flakiness. We're still doing manual testing to account for hard to automate paths: I doubt those devices take out their airpods while watching Tiger King. — Carola Nitz | Netflix https://twitter.com/CaroN -
At Twitch we have about 350 iOS UI tests, taking
~29 min with parallel Simulators on Jenkins. Running them on devices has been both flaky and slower every time we tried. We retry failed tests because we’ve always struggled with flakiness. Our UI tests are for the most part end-to-end integration tests that hit the server. Just because the UI tests are passing doesn’t mean the app has no bugs. But it’s been really valuable as a “first line of defense” to catch crashes earlier and keep master relatively healthy, both important in a team of 18 engineers. — Javier Soto | Twitch https://twitter.com/Javi/status/1255123189690068993 -
UI Testing over the years — App Builders 2020 |
Peter Steinberger — @steipete -
I love this one tweet calling them indeterministic; croissants are
"flaky" Our UI tests also cover a lot of networking, which doesn't help. But on the other hand we implicitly test a lot more with that — a blessing and a curse. A pull request from me yesterday ran 854 "features". 1 failure. Took 35 minutes. Tests were split across 13 nodes. — Bas Broek | XING https://twitter.com/basthomas/status/1255044115512782848 -
We have a very large UI test suite. Over time
this allowed us to cut our release cycle down to 1 week. Besides reducing manual verification, it proved to be extremely helpful on large under the hood refactorings. — Tomas Camin | Subito https://twitter.com/tomascamin/status/1255048867378343937 -
We've tried many different implementations of UI tests, including writing
custom frameworks. Our suite would take from 4 to 6 hours. Even the most basic UI test would sometimes fail because it could not find a text field, or a label. — Eneko Alonso | Mindbody https://twitter.com/eneko/status/1255499202400002064 -
XCUI Best Practices // use accessibility identifiers let resultView =
app.otherElements["view_result"] // always wait before asserting let viewExists = resultView.waitForExistence(timeout: 10) XCTAssert(viewExists) // write helpers (Page Object Pattern) class SearchPageObject { func type(query: String) -> Self func tapSend() -> ResultPage } https://medium.com/@danielcarlosce/some-good-practices-for-xcuitest-807bfe6b720d, https://medium.com/capital-one-tech/robot-pattern-testing-for-xcuitest-4c2f0c40b4ad, https:// blog.novoda.com/ui-testing-part-3/ -
Prevent Keyboard Tutorials // rdar://34634970 if let defaults = UserDefaults(suiteName:
"com.apple.Preferences") { // Since iOS 11 defaults.set(true, forKey:"DidShowGestureKeyboardIntroduction") // iPhone since iOS 13 defaults.set(true, forKey:"DidShowContinuousPathIntroduction") } https://github.com/google/EarlGrey/issues/633 -
UI Testing over the years — App Builders 2020 |
Peter Steinberger — @steipete -
Tracking Busy Resources — Grand Central Dispatch — NSTimer /
performSelector:afterDelay: — Layout Pass — View Appear/Disappear — Keyboard Transitions — CA/UI Animations — View Controller Transitions — WebView/Network Requests — Gesture Recognition — UIScrollView — Systemwide User Interaction UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete -
UI Testing over the years — App Builders 2020 |
Peter Steinberger — @steipete -
UI Testing over the years — App Builders 2020 |
Peter Steinberger — @steipete -
Interposing C functions struct dyld_interpose_tuple { const void* replacement; const
void* replacee; }; extern void dyld_dynamic_interpose(const struct mach_header* mh, const struct dyld_interpose_tuple array[], size_t count); https://github.com/opensource-apple/dyld/blob/master/include/mach-o/dyld_priv.h -
Interposing Grand Central Dispatch __attribute__((used)) __attribute__((section("__DATA,__interpose"))) static const dyld_interpose_tuple grey_static_interpose_tuples[]
= { { grey_dispatch_after, dispatch_after }, { grey_dispatch_async, dispatch_async }, { grey_dispatch_sync, dispatch_sync }, { grey_dispatch_after_f, dispatch_after_f }, { grey_dispatch_async_f, dispatch_async_f }, { grey_dispatch_sync_f, dispatch_sync_f } }; UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete -
✅ SwiftUI UI Testing over the years — App Builders
2020 | Peter Steinberger — @steipete -
Recap — KIF — accessibilityIdentifier + Inspector — UIAutomation —
Accelerating Animations layer.speed — Busy Waiting — Running Single Tests -only-testing — Parallel Testing — EarlGrey 1/2 — dyld_dynamic_interpose — SwiftUI UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete -
! Tooling UI Testing over the years — App Builders
2020 | Peter Steinberger — @steipete -
! Thanks UI Testing over the years — App Builders
2020 | Peter Steinberger — @steipete