Building a screenshot testing pipeline that scales

Scaling screenshot tests has been a difficult topic on Android for a while. When rigorously visual regression testing design system components or large scale applications, we face issues of scale such as long CI/CD run times, increased costs or a reduction of productivity due to conflicting pull requests.

This talk will go through some of the tradeoffs being made and proposes a screenshot testing pipeline that combines paparazzi to generate screenshots with reg-suit to create a diffing report in a GCS bucket.

Transcript

1. 1 Building a screenshot testing pipeline that scales Lukas Appelhans
2. 2 Building a screenshot testing pipeline Motivation
3. 3 Motivation: Feature development
4. 4 Motivation: Feature development
5. 5 Motivation: Design system
6. 6 Motivation: Manual review
7. 7 Motivation: Manual review
8. 8 Motivation: Ideal state (iOS)
9. 9 Choosing which frameworks to use. Structure of this talk Building a screenshot testing pipeline. Using the screenshot testing pipeline. Adoption at Mercari and what we’re missing. 02 03 04 01
10. 10 What is the purpose of screenshot testing? Verifying that UI code is rendering correctly. (As opposed to: Verifying that our code’s logic is working correctly.)
11. 11 How many tests do we expect to have? A lot of them. E2E Tests Integration tests Unit tests Screenshot tests
12. 12 How does screenshot testing differ from other tests? Unit/Integration/E2E tests • Given, when, then is clearly specified. • Post-condition can be verified automatically.
13. 13 How does screenshot testing differ from other tests? Unit/Integration/E2E tests • Given, when, then is clearly specified. • Post-condition can be verified automatically. Screenshot tests • Given (UI code with data), then. • Post-condition cannot be verified automatically. (Whether UI code renders “correctly” is up to the viewer.) • Use diffing to find which parts of the UI changed to make manual review easier.
14. 14 Test cases that generate screenshots. Screenshot testing A report of visual differences. 02 01
15. 15 JVM-based (Paparazzi) Test cases that generate screenshots Instrumented (on device/emulator) (Shot) 02 01
16. 16 Test cases: Framework choices Paparazzi • Fast and cheap. • Easy to set up. • There are some differences between UI rendered using Paparazzi and on-device. @Test fun launchComposable() { paparazzi.snapshot { MyComposable() } }
17. 17 Test cases: Framework choices Shot • As close to the “real world” as possible. • More difficult setup, especially on CI. (Instrumented tests.) • Takes significantly longer to run. • More costly. (You’ll probably run this on Firebase test lab.) @Test fun rendersMyComposable() { composeRule.setContent { MyComposable() } compareScreenshot(composeRule) }
18. 18 Reporting visual differences: Diffing tools Compare two sets of screenshots, make visual differences easy to review. Built-in reporting (e.g. Paparazzi) Reg-suit 02 01
19. 19 Reporting visual differences: Where to store screenshot files? Store directly in git (use git-lfs) Store externally (e.g. GCS bucket) 02 01
20. 20 Reporting visual differences: What to compare to? In a pull request, we want a report of the visual differences between the current state of the branch and when the branch was started. 1 2 3 4 Base commit master HEAD branch HEAD
21. 21 Summary of choices Paparazzi to generate screenshots. Reg-suit to diff them. 02 01
22. 22 Generate screenshots 1. Generate screenshots in all modules. ./gradlew recordPaparazzi[Debug/Dev/Release] 2. Copy screenshots of each module into a separate directory. cp ./*/src/test/snapshots/images/*.png ./screenshots Or create further subdirectories per module, package, etc.
23. 23 Generate diff report 3. Use reg-suit to generate report and store screenshots in the cloud. export EXPECTED_KEY=$(git merge-base HEAD origin/master) export ACTUAL_KEY=$GITHUB_SHA npx reg-suit run
24. 24 Generate diff report 3. Use reg-suit to generate report and store screenshots in the cloud. export EXPECTED_KEY=$(git merge-base HEAD origin/master) export ACTUAL_KEY=$GITHUB_SHA npx reg-suit run
25. 25 Configuring reg-suit { "core": { "workingDir": ".reg", "actualDir": "screenshots", }, "plugins": { … } } ./regconfig.json
26. 26 Configuring reg-suit { …, "plugins": { "reg-simple-keygen-plugin": { "expectedKey": "${EXPECTED_KEY}", "actualKey": "${ACTUAL_KEY}" }, … } } ./regconfig.json
27. 27 Configuring reg-suit { …, "plugins": { …, "reg-publish-gcs-plugin": { "bucketName": "gcs-bucket-name" }, … } } ./regconfig.json
28. 28 Configuring reg-suit { …, "plugins": { …, "reg-notify-github-plugin": { "clientId": "your-client-id", "setCommitStatus": false, "shortDescription": true } } } ./regconfig.json
29. 29 Configuring reg-suit { …, "plugins": { …, "reg-notify-github-plugin": { "clientId": "your-client-id", "setCommitStatus": false, "shortDescription": true } } } ./regconfig.json
30. 30 Configuring reg-suit { …, "plugins": { …, "reg-notify-github-plugin": { "clientId": "your-client-id", "setCommitStatus": false, "shortDescription": true } } } ./regconfig.json
31. 31 Using screenshot tests We know how to build the infrastructure, but how do we use it?
32. 32 How to write tests class ChipScreenshotTest { @get:Rule val paparazzi = MercariPaparazzi() // Test cases }
33. 33 How to write test cases class ChipScreenshotTest { … @Test fun shortLabel() = paparazzi.snapshot { Chip( label = "Foo", selected = false, onSelectionChanged = {} ) } }
34. 34 Pre-merge screenshot test report
35. 35 Pre-merge screenshot test report
36. 36 Pre-merge screenshot test report
37. 37 Pre-merge screenshot test report (2)
38. 39 Current scale ~30 secs ~700 Time it takes to generate the screenshots Amount of screenshots
39. 40 Caveats 1. Code that requires multiple compositions will not render correctly. val styles = listOf(Large, Medium, Small) var index by remember { mutableStateOf(0) } Text( maxLines = 1, style = styles[index], onTextLayout = { textLayoutResult -> if (textLayoutResult.hasVisualOverflow) { index = index.plus(1).coerceAtMost(styles.size - 1) } } )
40. 41 Caveats 1. Code that requires multiple compositions will not render correctly. 2. Supporting multiple densities for each test case. In addition, there was a bug in Paparazzi that prevented it to work properly with our build cache. Fixed in version 1.3.
41. 42 What are we missing? Full screens Adoption Testing full screens Adoption outside of Design System Components
42. 43 Thank you!