Slide 1

Slide 1 text

UI Testing over the years App Builders 2020 Peter Steinberger https://twitter.com/steipete

Slide 2

Slide 2 text

PSPDFKit https://pspdfkit.com - The Complete PDF Solution You Can Rely On

Slide 3

Slide 3 text

Scope — Manual Tests — UI Tests — Snapshot Tests — Integration Tests — Unit Tests UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 4

Slide 4 text

Source: iOS Dev Survey, https://iosdevsurvey.com/2019/13-testing-and-ci-cd/

Slide 5

Slide 5 text

UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 6

Slide 6 text

What framework(s) do you use for UI testing? — XCUITest — KIF — EarlGrey (v1/v2) — Xamarin.UITest — Appium — Calabash — Other UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 7

Slide 7 text

How reliable are your UI tests? — ! — " — # — $ — % https://twitter.com/steipete/status/1255021187064893440

Slide 8

Slide 8 text

Do you distribute your tests? — No (One test at a time) — Parallel Simulators — Distribute across Machines — Distribute across machines + Parallel Simulators UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 9

Slide 9 text

At Funding Circle we have been writing UI tests since the beginning. We have a few hundreds currently and they focus on the frequent paths. They take about 30 min to run (not parallelised yet). We still use KIF We suffered the usual pain points (hard to maintain, flaky, etc), but we have always managed to keep them under control. Every PR has to come with the relevant automated tests in order to be merged. — Edu Caselles | Funding Circle Group https://twitter.com/CasellesEdu/status/1255185884045082624

Slide 10

Slide 10 text

KIF: Keep It Functional — Written in 2011 by Square — Lots of private API — Busy waiting — Uses accessiblityLabel UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 11

Slide 11 text

Leena Snidate / Codenomicon - http://heartbleed.com/heartbleed.svg

Slide 12

Slide 12 text

KIF: Keep It Functional — Written in 2011 by Square — Lots of private API — Busy waiting — Uses accessiblityLabel to query views UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 13

Slide 13 text

! Create your own UITouch extension UITouch { public init!(in view: UIView!) public init!(at point: CGPoint, in view: UIView!) open func setLocationInWindow(_ location: CGPoint) open func setPhaseAndUpdateTimestamp(_ phase: UITouch.Phase) } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

https://developer.apple.com/documentation/uikit/uitouch/phase

Slide 16

Slide 16 text

https://developer.apple.com/documentation/uikit/uitouch/phase/regionentered

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

! Accessibility 101 — accessiblityLabel — accessiblityIdentifier - value, traits, frame, hint - container, containerSpace UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 21

Slide 21 text

UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 22

Slide 22 text

UIAutomation — Added in iOS 4 (2010) — Removed in iOS 10 (2016) — JavaScript-based ! UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 23

Slide 23 text

http://blog.manbolo.com/2012/04/08/ios-automated-tests-with-uiautomation

Slide 24

Slide 24 text

var testName = "Test 1"; var target = UIATarget.localTarget(); var app = target.frontMostApp(); var window = app.mainWindow(); UIALogger.logStart(testName); // select the elements UIALogger.logMessage("Select the first tab"); var tabBar = app.tabBar(); var selectedTabName = tabBar.selectedButton().name(); if (selectedTabName != "First") { tabBar.buttons()["First"].tap(); } // tap on the text fiels UIALogger.logMessage("Tap on the text field now"); var recipeName = "Unusually Long Name for a Recipe"; window.textFields()[0].setValue(recipeName); target.delay(2); // tap on the text fiels UIALogger.logMessage("Dismiss the keyboard"); app.keyboard().buttons()["return"].tap(); var textValue = window.staticTexts()["RecipeName"].value(); if (textValue === recipeName){ UIALogger.logPass(testName); } else { app.logElementTree(); UIALogger.logFail(testName); } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 25

Slide 25 text

var testName = "Test 1" var target = UIATarget.localTarget() var app = target.frontMostApp() var window = app.mainWindow() UIALogger.logStart(testName) // select the elements UIALogger.logMessage("Select the first tab") var tabBar = app.tabBar() var selectedTabName = tabBar.selectedButton().name() if (selectedTabName != "First") { tabBar.buttons()["First"].tap() } // tap on the text fiels UIALogger.logMessage("Tap on the text field now") var recipeName = "Unusually Long Name for a Recipe" window.textFields()[0].setValue(recipeName) target.delay(2) // tap on the text fiels UIALogger.logMessage("Dismiss the keyboard") app.keyboard().buttons()["return"].tap() var textValue = window.staticTexts()["RecipeName"].value() if (textValue == recipeName) { UIALogger.logPass(testName) } else { app.logElementTree() UIALogger.logFail(testName) } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 26

Slide 26 text

https://developer.apple.com/documentation/uikit/uiaccessibilityidentification/1623132-accessibilityidentifier

Slide 27

Slide 27 text

https://www.shutterstock.com/image-photo/plain-simple-black-white-boxes-copyspace-47099215

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplicationLaunchOptionsKey: Any]?) -> Bool { UIApplication.shared.keyWindow?.layer.speed = 100 return true } https://pspdfkit.com/blog/2016/running-ui-tests-with-ludicrous-speed/

Slide 30

Slide 30 text

Busy Waiting testWithViewController(pdfController) { pdfController.tapOnAnnotation(fileLinkAnnotation) // Verify that our web view controller is visible waitForCondition(pdfController.isControllerClassVisible(PSPDFWebViewController.self)) pdfController.dismissViewControllerAnimated(false, completion: nil) if #available(iOS 9, *) { pdfController.tapOnAnnotation(webLinkAnnotation) waitForCondition(pdfController.isControllerClassVisible(SFSafariViewController.self)) pdfController.dismissViewControllerAnimated(false, completion: nil) } } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 31

Slide 31 text

https://pspdfkit.com/blog/2019/open-links-in-safari-not-safari-view-controller/

Slide 32

Slide 32 text

Boolean CTTRunLoopRunUntil(Boolean(^fulfilled_)(), Boolean polling_, CFTimeInterval timeout_) { // Loop Observer Callback __block Boolean fulfilled = NO; void (^beforeWaiting) (CFRunLoopObserverRef observer, CFRunLoopActivity activity) = ^(CFRunLoopObserverRef observer, CFRunLoopActivity activity) { assert(!fulfilled); //RunLoop should be stopped after condition is fulfilled. // Check Condition fulfilled = fulfilled_(); if(fulfilled) { // Condition fulfilled: stop RunLoop now. CFRunLoopStop(CFRunLoopGetCurrent()); } else if(polling_) { // Condition not fulfilled, and we are polling: prevent RunLoop from waiting and continue looping. CFRunLoopWakeUp(CFRunLoopGetCurrent()); } }; CFRunLoopObserverRef observer = CFRunLoopObserverCreateWithHandler(NULL, kCFRunLoopBeforeWaiting, true, 0, beforeWaiting); CFRunLoopAddObserver(CFRunLoopGetCurrent(), observer, kCFRunLoopDefaultMode); // Run! CFRunLoopRunInMode(kCFRunLoopDefaultMode, timeout_, false); CFRunLoopRemoveObserver(CFRunLoopGetCurrent(), observer, kCFRunLoopDefaultMode); CFRelease(observer); return fulfilled; } https://bou.io/CTTRunLoopRunUntil.html

Slide 33

Slide 33 text

func CTTRunLoopRunUntil(_ fulfilled_: @escaping () -> Bool, _ polling_: Bool, _ timeout_: CFTimeInterval) -> Bool { // Loop Observer Callback var fulfilled = false let beforeWaiting: ((_ observer: CFRunLoopObserver?, _ activity: CFRunLoopActivity) -> Void)? = { observer, activity in assert(!fulfilled) //RunLoop should be stopped after condition is fulfilled. // Check Condition fulfilled = fulfilled_() if fulfilled { // Condition fulfilled: stop RunLoop now. CFRunLoopStop(CFRunLoopGetCurrent()) } else if polling_ { // Condition not fulfilled, and we are polling: prevent RunLoop from waiting and continue looping. CFRunLoopWakeUp(CFRunLoopGetCurrent()) } } let observer = CFRunLoopObserverCreateWithHandler(nil, CFOptionFlags(CFRunLoopActivity.beforeWaiting.rawValue), true, CFIndex(0), beforeWaiting) CFRunLoopAddObserver(CFRunLoopGetCurrent(), observer, CFRunLoopMode.defaultMode) // Run! CFRunLoopRunInMode(CFRunLoopMode.defaultMode, timeout_, false) CFRunLoopRemoveObserver(CFRunLoopGetCurrent(), observer, CFRunLoopMode.defaultMode) return fulfilled } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 34

Slide 34 text

https://stackoverflow.com/questions/56154827/nsattributedstring-from-html-on-main-thread-behaves-as-if-multithreading

Slide 35

Slide 35 text

! Square UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 36

Slide 36 text

We have over 8000 tests consisting of a mixture of Unit and UI Tests in our codebase that run on devices housed in an on-site lab. A test suite of around 200 tests run with every PR. If a test fails, it is automatically rerun to account for flakiness. We're still doing manual testing to account for hard to automate paths: I doubt those devices take out their airpods while watching Tiger King. — Carola Nitz | Netflix https://twitter.com/CaroN

Slide 37

Slide 37 text

At Twitch we have about 350 iOS UI tests, taking ~29 min with parallel Simulators on Jenkins. Running them on devices has been both flaky and slower every time we tried. We retry failed tests because we’ve always struggled with flakiness. Our UI tests are for the most part end-to-end integration tests that hit the server. Just because the UI tests are passing doesn’t mean the app has no bugs. But it’s been really valuable as a “first line of defense” to catch crashes earlier and keep master relatively healthy, both important in a team of 18 engineers. — Javier Soto | Twitch https://twitter.com/Javi/status/1255123189690068993

Slide 38

Slide 38 text

https://developer.apple.com/videos/play/wwdc2016/409/

Slide 39

Slide 39 text

https://github.com/lyndsey-ferguson/fastlane-plugin-test_center

Slide 40

Slide 40 text

https://heartbeat.fritz.ai/the-fast-furious-and-flaky-continuous-integration-with-xcode-10-parallel-tests-2da71fd48cb1

Slide 41

Slide 41 text

https://github.com/linkedin/bluepill

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

https://twitter.com/steipete/status/410078113205592065

Slide 44

Slide 44 text

https://ortastuff.s3.amazonaws.com/gifs/marin-ok.gif

Slide 45

Slide 45 text

https://gist.github.com/KrauseFx/d8cb973ea2cab3f4f835

Slide 46

Slide 46 text

https://danger.systems/swift/

Slide 47

Slide 47 text

UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 48

Slide 48 text

https://pspdfkit.com/blog/2016/converting-xcode-test-results-the-fast-way/

Slide 49

Slide 49 text

https://github.com/fastlane-community/trainer

Slide 50

Slide 50 text

https://github.com/thii/xcbeautify

Slide 51

Slide 51 text

https://github.com/facebook/xctool

Slide 52

Slide 52 text

I love this one tweet calling them indeterministic; croissants are "flaky" Our UI tests also cover a lot of networking, which doesn't help. But on the other hand we implicitly test a lot more with that — a blessing and a curse. A pull request from me yesterday ran 854 "features". 1 failure. Took 35 minutes. Tests were split across 13 nodes. — Bas Broek | XING https://twitter.com/basthomas/status/1255044115512782848

Slide 53

Slide 53 text

https://github.com/Subito-it/Mendoza

Slide 54

Slide 54 text

We have a very large UI test suite. Over time this allowed us to cut our release cycle down to 1 week. Besides reducing manual verification, it proved to be extremely helpful on large under the hood refactorings. — Tomas Camin | Subito https://twitter.com/tomascamin/status/1255048867378343937

Slide 55

Slide 55 text

https://github.com/Subito-it/SBTUITestTunnel

Slide 56

Slide 56 text

https://github.com/depoon/SwiftLocalhost

Slide 57

Slide 57 text

https://github.com/google/eDistantObject

Slide 58

Slide 58 text

We've tried many different implementations of UI tests, including writing custom frameworks. Our suite would take from 4 to 6 hours. Even the most basic UI test would sometimes fail because it could not find a text field, or a label. — Eneko Alonso | Mindbody https://twitter.com/eneko/status/1255499202400002064

Slide 59

Slide 59 text

XCUI Best Practices // use accessibility identifiers let resultView = app.otherElements["view_result"] // always wait before asserting let viewExists = resultView.waitForExistence(timeout: 10) XCTAssert(viewExists) // write helpers (Page Object Pattern) class SearchPageObject { func type(query: String) -> Self func tapSend() -> ResultPage } https://medium.com/@danielcarlosce/some-good-practices-for-xcuitest-807bfe6b720d, https://medium.com/capital-one-tech/robot-pattern-testing-for-xcuitest-4c2f0c40b4ad, https:// blog.novoda.com/ui-testing-part-3/

Slide 60

Slide 60 text

https://github.com/danielCarlosCE/XCUITestExample

Slide 61

Slide 61 text

https://github.com/google/EarlGrey/issues/633

Slide 62

Slide 62 text

Prevent Keyboard Tutorials // rdar://34634970 if let defaults = UserDefaults(suiteName: "com.apple.Preferences") { // Since iOS 11 defaults.set(true, forKey:"DidShowGestureKeyboardIntroduction") // iPhone since iOS 13 defaults.set(true, forKey:"DidShowContinuousPathIntroduction") } https://github.com/google/EarlGrey/issues/633

Slide 63

Slide 63 text

? https://twitter.com/steipete

Slide 64

Slide 64 text

UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 65

Slide 65 text

Tracking Busy Resources — Grand Central Dispatch — NSTimer / performSelector:afterDelay: — Layout Pass — View Appear/Disappear — Keyboard Transitions — CA/UI Animations — View Controller Transitions — WebView/Network Requests — Gesture Recognition — UIScrollView — Systemwide User Interaction UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 66

Slide 66 text

UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 69

Slide 69 text

https://opensource.apple.com

Slide 70

Slide 70 text

https://opensource.apple.com/source/dyld/dyld-733.8/include/mach-o/dyld_priv.h.auto.html

Slide 71

Slide 71 text

Interposing C functions struct dyld_interpose_tuple { const void* replacement; const void* replacee; }; extern void dyld_dynamic_interpose(const struct mach_header* mh, const struct dyld_interpose_tuple array[], size_t count); https://github.com/opensource-apple/dyld/blob/master/include/mach-o/dyld_priv.h

Slide 72

Slide 72 text

https://chromium.googlesource.com/chromium/src/+/18a4f63fd5dc592a6b31f2a832de145b151adbde/media/audio/mac/coreaudiodispatchoverride.cc

Slide 73

Slide 73 text

Interposing Grand Central Dispatch __attribute__((used)) __attribute__((section("__DATA,__interpose"))) static const dyld_interpose_tuple grey_static_interpose_tuples[] = { { grey_dispatch_after, dispatch_after }, { grey_dispatch_async, dispatch_async }, { grey_dispatch_sync, dispatch_sync }, { grey_dispatch_after_f, dispatch_after_f }, { grey_dispatch_async_f, dispatch_async_f }, { grey_dispatch_sync_f, dispatch_sync_f } }; UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 74

Slide 74 text

https://github.com/facebook/fishhook

Slide 75

Slide 75 text

✅ SwiftUI UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 76

Slide 76 text

https://github.com/Dimillian/ACHNBrowserUI/pull/89/checks

Slide 77

Slide 77 text

https://github.com/Dimillian/ACHNBrowserUI/pull/89/checks

Slide 78

Slide 78 text

Recap — KIF — accessibilityIdentifier + Inspector — UIAutomation — Accelerating Animations layer.speed — Busy Waiting — Running Single Tests -only-testing — Parallel Testing — EarlGrey 1/2 — dyld_dynamic_interpose — SwiftUI UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 79

Slide 79 text

! Tooling UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete

Slide 80

Slide 80 text

! Thanks UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete