Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Shifting (UI) tests to the far left

Shifting (UI) tests to the far left

Writing specs is a hard problem, with insufficient/hidden specs and people communication flaws being the two most common problems in requirements engineering. In this talk, we will examine how, and under what conditions, tests can be considered as specs and how shifting them to the far left could remove a lot of the complexity mobile teams face.

gsoti

June 26, 2022
Tweet

More Decks by gsoti

Other Decks in Programming

Transcript

  1. Georgios Sotiropoulos, he a d of mobile @ xm.com Shifting

    (UI) tests to the far left C a n UI tests be used to write specs?
  2. from functional requirements → functional speci fi cations → tests

    as specs → Use UI tests to write specs → Shift UI tests to the left
  3. Functional requirements • Provide a description of the required behaviour

    • Usually live in requirements documents (PRD)
  4. Functional requirements • Incomplete and/or hidden requirements • Communication f

    laws between project team and customer The two most common problems http://napire.org/#/explore
  5. find the max example • Given an array of integers,

    f ind the maximum element Function a l requirement
  6. platform migration example • When logging in, we should be

    authenticated at the same time to the old and the new API, allowing us to access endpoints in both systems Function a l requirement
  7. Functional specifications • More technical • Often speci f ies

    the behaviour in terms of inputs and outputs • What for? 1. to let the developers know what to build 2. to let the testers know what tests to run
  8. find the max example • Given an array of integers,

    f ind the maximum element Function a l requirement Function a l speci f ic a tion • ∀x, set: (Max(set) = x) 㱺 x ∈ s ⋀ ∀y ∈ s: y ≤ x Hillel Wayne- What is a speci f ication?
  9. platform migration example • When logging in, we should be

    authenticated at the same time to the old and the new API, allowing us to access endpoints in both systems Function a l requirement Function a l speci f ic a tion Scenario: User logs in Given that an existing user launches the app When he f ills in his credentials in the login page and taps on the login button Then he should login to both systems and be presented with the home page
  10. Multiple ways to write specifications • Natural language • Mathematical

    equations • Inputs and outputs examples • BDD scenarios • Decision tables, state diagrams, f low charts, Harel statecharts, Formal speci f ications
  11. if 1. they do not ch a nge a fter

    a ref a ctor Tests can be specs Hillel Wayne- What is a speci f ication?
  12. Is this test a specification? assertEquals(max([1,2,3,4,5]), 5) f ind the

    m a x ex a mple It is an insu ff icient spec!!
  13. How would a test that is a specification would look

    like? array = generate a random array of integers max = max(array) sorted = sort(array) assertEquals(max == sorted.lastElement) f ind the m a x ex a mple 100x testProperty(“The max element in an array should be last element of the sorted version of the array”) { }
  14. Property-based testing f ind the m a x ex a

    mple https://github.com/typelift/SwiftCheck
  15. if 1. they do not ch a nge a fter

    a ref a ctor 2. they c a n a lso be suf f icient specs Tests can be specs Hillel Wayne- What is a speci f ication? Can UI tests be specs?
  16. func testLoginSuccessFlow() { runTests(withFlag: .on(.newLogin), stubs: []) { XCTAssert(app.staticTexts[“usernameTextfield"].exists) app.textFields[“usernameTextField"].tap()

    app.textFields["usernameTextField"].typeText("username") app.secureTextFields["passwordTextfield"].tap() app.secureTextFields["passwordTextfield"].typeText("pass") app.buttons["loginButton"].tap() XCTAssert(app.otherElements[“homeView”].exists) } } pl a tform migr a tion ex a mple
  17. Can UI tests be specs? ✅ if 1. they do

    not ch a nge a fter a ref a ctor 2. they c a n a lso be suf f icient specs
  18. func testLoginSuccessFlow() { runTests(withFlag: .on(.newLogin), stubs: []) { // assert

    that login screen is loaded XCTAssert(app.staticTexts[“usernameTextfield"].exists) // fill in username app.textFields[“usernameTextField"].tap() app.textFields["usernameTextField"].typeText("username") // fill in password app.secureTextFields["passwordTextfield"].tap() app.secureTextFields["passwordTextfield"].typeText("pass") // tap login button app.buttons["loginButton"].tap() // assert that home screen is loaded XCTAssert(app.otherElements[“homeView”].exists) } } Adding comments
  19. if 1. they do not ch a nge a fter

    a ref a ctor ✅ 2. they c a n a lso be suf f icient specs 3. if they a re written in a cle a r hum a n re a d a ble form Can UI tests be specs?
  20. func testLoginSuccessFlow() { runTests(withFlag: .on(.newLogin), stubs: []) { LoginScreen(app) .assertLoginScreenIsLoaded()

    .fill(username: "username", password: "password") .tapLoginButton() .assertHomeScreenIsLoaded() } } https://martinfowler.com/bliki/PageObject.html P a ge Object p a ttern Please avoid fancy code!
  21. Can UI tests be specs? if 1. they do not

    ch a nge a fter a ref a ctor ✅ 2. they c a n a lso be suf f icient specs 3. if they a re written in a cle a r hum a n re a d a ble form ✅
  22. Insu ffi cient/hidden requirements Wh a t speci f ic

    a tions rem a in hidden in our ex a mple?
  23. func testLoginSuccessFlow() { runTests(withFlag: .on(.newLogin), stubs: []) { LoginScreen(app) .assertLoginScreenIsLoaded()

    .fill(username: "username", password: "password") .tapLoginButton() .assertFullScreenSpinnerIsLoaded() .assertHomeScreenIsLoaded() } } Lo a ding st a tes
  24. func testLogin_AppVersionNoLongerSupported_FailureFlow() { let stubs = stub login with error:

    “noLongerSupported" runTests(withFlag: .on(.newLogin), stubs: stubs) { LoginScreen(app) .assertLoginScreenIsLoaded() .fill(username: "username", password: "password") .tapLoginButton() .assertFullScreenSpinnerIsLoaded() .assertForceUpdateAlertIsLoaded() } } Error st a tes / non-h a ppy p a th f lows
  25. HTTP c a lls func testLoginSuccessFlow() { runTests(withFlag: .on(.newLogin), stubs:

    []) { LoginScreen(app) .assertLoginScreenIsLoaded() .fill(username: "username", password: "password") .tapLoginButton() .assertFullScreenSpinnerIsLoaded() .assertHomeScreenIsLoaded() } }
  26. HTTP c a lls func testLoginSuccessFlow() { runTests(withFlag: .on(.newLogin), stubs:

    []) { LoginScreen(app) .assertLoginScreenIsLoaded() .fill(username: "username", password: "password") .tapLoginButton() .assertFullScreenSpinnerIsLoaded() .assertHomeScreenIsLoaded() .flushMonitoredRequests(with: """ /authentication/login/mt5 ✅ /GetServerUri ✅ /Home ✅ /LoginWithToken ✅ /PushNotifications/Device/Login ✅ /Account ✅ """) } } NEW API OLD API
  27. • Is this really a spec or an implementation detail?

    • Yes, and it is a speci f ication that is also enforced upon the implementation over time (being a test) • HTTP c a lls
  28. • Is this really a spec or an implementation detail?

    • Yes, and it is a speci f ication that is also enforced upon the implementation over time (being a test) • Prevents unintended addition or deletion of API calls • Perfect documentation of the interaction with the server HTTP c a lls
  29. Audit logs / a n a lytics events func testLoginSuccessFlow()

    { runTests(withFlag: .on(.newLogin), stubs: []) { LoginScreen(app) .assertLoginScreenIsLoaded() .fill(username: "username", password: "password") .tapLoginButton() .assertFullScreenSpinnerIsLoaded() .assertHomeScreenIsLoaded() .flushAuditLogs(with: """ Login button pressed (user: 3412) Login successful (user: 3412) """) } }
  30. if they do not ch a nge a fter a

    ref a ctor ✅ a nd if they c a n a lso be suf f icient specs a nd if they a re written in a cle a r hum a n re a d a ble form ✅ Can UI tests be specs? ✅
  31. The specs MR Shift UI tests to the left 1.

    Open the very f irst MR writing the specs a s UI tests
  32. Shift UI tests to the left 2. G a ther

    a ll UI tests under the s a me folder a nd a pply code ownership Under code ownership
  33. Shift UI tests to the left 2. G a ther

    a ll UI tests under the s a me folder a nd a pply code ownership Approval required Change code UI tests fail Update UI tests
  34. Shift UI tests to the left 3. M a ke

    the set up of the network stubs explicit
  35. Shift UI tests to the left 3. M a ke

    the set up of the network stubs explicit
  36. Shift UI tests to the left 4. Write the a

    ctu a l scen a rio using empty methods
  37. Shift UI tests to the left 5. Option a l:

    Specify API c a lls involved
  38. Reducing communication flaws • Leaves no ambiguity to the developers

    • Specs that never go out of sync with the implementation • Under source control and code ownership, keep track when a requirement is changed
  39. Conclusion • You can use (UI) tests to write speci

    f ications • They are actually quite powerful to reveal hidden speci f ications • You can also shift them to the left (and apply code ownership)
  40. 2nd example: search • Results should include all symbols that

    contain the search term in the symbol name in a case insensitive way 
 • Symbols that have a priority should be sorted by the priority and included in the top results 
 • All other symbols should be sorted alphabetically under the priority symbols Function a l requirement
  41. 2nd example: search • Results should include all symbols that

    contain the search term in the symbol name in a case insensitive way Function a l requirement Function a l speci f ic a tion • EU → EURUSD, EURGBP, Deutsche Bank
  42. 2nd example: search • Symbols that have a priority should

    be sorted by the priority and included in the top results Function a l requirement Function a l speci f ic a tion • priority: Int? (priority = 1 being the highest) • EU → EURUSD (1), Deutsche Bank (2), EURGBP (3) • EU → Deutsche Bank (1), EURGBP (1), EURUSD (1)
  43. 2nd example: search • All other symbols should be sorted

    alphabetically under the priority symbols Function a l requirement Function a l speci f ic a tion • EU → EURUSD (1), Deutsche Bank (2), EURGBP (3), Amadeus, EURCHF, Eurobank, Thomson Reuters
  44. 2nd example: search • Results should include all symbols that

    contain the search term in the symbol name in a case insensitive way Function a l requirement Test a s speci f ic a tion 
 term = gener a te a r a ndom term randomCaseTerm = fl ip term from upper to lower case (and vice versa) in random characters matchedNames = generate names that contain randomCaseTerm notMatchedNames = generate names that do not contain randomCaseTerm matchedSymbols = generate symbols from matchedNames (and random priority) notMatchedSymbols = generate symbols from notMatchedNames (and random priority) allSymbols = matchedSymbols + notMatchedSymbols results = search(in: allSymbols, term: term) assertEquals(Set(results), Set(matchedSymbols))
  45. 2nd example: search • Symbols that have a priority should

    be sorted by the priority and included in the top results Function a l requirement Test a s speci f ic a tion 
 a scendingPriorities = gener a te a r a ndom list with a scending priorities matchedSymbols = generate symbols with ascendingPriorities (and a matchedName) shuf fl edSymbols = matchedSymbols.shuf fl ed results = search(in: shuf fl edSymbols, term: term) assertEquals(results, matchedSymbols)