Snapshot testing. Testing the UI and beyond

In this talk, Georgios will explain what is Snapshot Testing, how it helps develop and test the UI and will present a complete strategy for covering all extreme UI cases. He will also explore how this actually changed the team's workflow when developing the UI, paving the way to the "perfect pull-request" and how Snapshot Testing can be so much more than a visual regression testing tool.

Transcript

1. SNAPSHOT TESTING GEORGE SOTIROPOULOS MOBILE TEAM @XM TESTING THE UI AND BEYOND
2. SOFTWARE VERIFICATION • Assure that software fully satisfies all the expected requirements, equivalent to software testing • Static and dynamic verification
3. STATIC VERIFICATION • Type system (Swift, Kotlin) • Static analysis • Code review
4. DYNAMIC VERIFICATION • Unit tests • Snapshot tests • Characterisation tests • Integration tests • UI functional tests • Manual
5. WHAT IS SNAPSHOT TESTING? XCTAssertEqual(add(3,5), 8) assertSnapshot(add(3,5))
6. HOW DOES IT WORK? func testAddition() { assertSnapshot(add(3,5)) } • Record mode Saves a snapshot of the output as an artifact on disk e.g. testAddition1.txt • Verify mode Compares produced snapshot with saved reference snapshot
7. VISUAL TESTING METHOD SCREENSHOT TESTING
8. COMPARING SCREENSHOTS SCREENSHOT TESTING Record Verify Diff
9. WHY DO THIS? SCREENSHOT TESTING • They provide far more coverage than a unit test normally allows • A fast way to see how our view looks while developing • Plug them in the CI and immediately detect UI breaking changes and avoid regression bugs
10. MULTIPLE UI CASES OF THE DOMAIN SCREENSHOT TESTING STRATEGY
11. TESTING LANGUAGES SCREENSHOT TESTING STRATEGY
12. TESTING DEVICES SIZES SCREENSHOT TESTING STRATEGY
13. TESTING THEMES SCREENSHOT TESTING STRATEGY
14. ALL COMBINATIONS SCREENSHOT TESTING STRATEGY
15. APPLYING THE STRATEGY AN EXAMPLE
16. TESTING THEMES, SIZES AND EXTREME VALUES THE SCREENSHOTS
17. TESTING THEMES, SIZES AND EXTREME VALUES func testSizesThemesAndExtremeValues() { SnapshotTestStrategy.VC.all { (device, theme) in applyTheme(theme) let accountsViewModel = ActiveAccountViewModel.generateForExtremeValues() let vc = ActiveAccountViewController(viewModel: accountsViewModel) _ = traitControllers(device: device, orientation: .portrait, child: vc) FBSnapshotVerifyView(vc.view, identifier: "_device_\(device)_theme_\ (theme)") } } THE CODE
18. DIFFERENT UI CASES OF THE DOMAIN THE SCREENSHOTS
19. DIFFERENT UI CASES OF THE DOMAIN func testRealAccount() { SnapshotTestStrategy.VC.basic { (device, theme) in THE CODE applyTheme(theme) let accountsViewModel = ActiveAccountViewModel.generate(type: .real) let vc = ActiveAccountViewController(viewModel: accountsViewModel) _ = traitControllers(device: device, orientation: .portrait, child: vc) FBSnapshotVerifyView(vc.view, identifier: "_device_\(device)_theme_\ (theme)") } }
20. UNCOVERING MORE UI CASES CODE COVERAGE IS OUR FRIEND
21. UNCOVERING MORE UI CASES THE SCREENSHOTS
22. SCREENSHOT TESTING STRATEGY • Start with a test case for all possible theme/size combinations. Use extreme values to stress the UI with larger texts. • Cover all different UI cases (different domain cases) with basic theme/size combination per case. Use code coverage to discover them. • Testing individual views instead of view controllers usually requires two theme/size combinations (one for small width and light theme and one for large width and dark theme) A SUMMARY
23. THE PERFECT PULL REQUEST
24. TEXT INSTEAD OF IMAGE SNAPSHOT TESTING THE UI • Could also use a textual representation of the UI as a snapshot
25. COMPLEX DATA STRUCTURES SNAPSHOT TESTING let request = URLRequest(endpoint: .user) assertSnapshot(request) -POST https://xm.com/user +GET https://xm.com/user Accept: application/json { "name" : "George", "email" : “george@xm.com", +"type": “demo", "password" : "abc123" }
26. COMPLEX DATA STRUCTURES SNAPSHOT TESTING curl -X POST -H "Content-Type: application/json" -d '{"name":"George", "email":"george@xm.com", "password": #some_encrypted_password#}' http://xm.com/user
27. COMPLEX DATA STRUCTURES SNAPSHOT TESTING struct URLProvider: Codable { let webTraderHomeURL: String let loginURL: String let newsURL: String let serverTimeURL: String let userURL: String ... ... ... let watchlistURL: String }
28. COMPLEX DATA STRUCTURES SNAPSHOT TESTING _assertInlineSnapshot(matching: provider, as: .json, with: """ { "loginURL" : "https://xm.com/login", "newsURL" : "https://xm.com/news", "serverTimeURL" : "https://xm.com/serverTime", "userURL" : "https://xm.com/user/{userid}", "watchlistURL" : "https://xm.com/watchlist/ {watchlistid}", ... "webTraderHomeURL" : “https://xm.com/home" } """)
29. COMPLEX DATA FORMATS SNAPSHOT TESTING • PDF • XML • HTML • Code generation
30. AS A CHANGE DETECTOR SNAPSHOT TESTING • Snapshot tests are essentially change detectors • Ideal when using external systems that we do not control • ..or when using external libraries that we want to update from time-to-time V2 V1 ✅ ✅ ✅ ✅
31. CONS SNAPSHOT TESTING • Effortless to update but easier to accept new snapshots that have problems • The context of what you test is lost • Might lead to fragile tests
32. TOOLS • https://github.com/pointfreeco/swift-snapshot-testing • https://github.com/uber/ios-snapshot-test-case • https://github.com/Karumi/Shot • https://github.com/Karumi/KotlinSnapshot • https://jestjs.io
33. REFERENCES • https://speakerdeck.com/sotiropoulos/snapshot-testing-testing-the-ui-and-beyond • https://medium.com/xmglobal • https://github.com/trading-point/screenshot-testing-android-demo • https://github.com/trading-point/screenshot-testing-ios-demo • https://www.pointfree.co/episodes/ep41-a-tour-of-snapshot-testing • https://jestjs.io/docs/en/snapshot-testing • https://codeclimate.com/blog/gold-master-testing/
34. WE ARE HIRING https://www.xm.com/careers hr@xm.com Currently 4 Android / 4 iOS
35. THANK YOU QUESTIONS?