Mastering Screenshot Testing for Android Apps

Screenshot testing helps in verifying that your UI renders correctly across various devices and configurations. Also, can directly display the UI difference after any code changes.

In this talk, we will dive into the world of screenshot testing using popular tools like paparazzi, roborazzi and Google’s experimental plugin: Compose Preview Screenshot Testing.

Starting with the basics, we’ll explore the implementation of screenshot tests, how to set up and configure these tools, and integrate them into CI pipeline. I’ll share experience about weird issues our team faced and how to address them, and best practices for testing adaptive layouts, ensuring your app looks great on any screen size.

As Compose Preview Screenshot Testing is still in Alpha phase, I will catch up with the issues/pain point under disucssing, and share an ideal world of screenshot testing.

Transcript

1. Mastering Screenshot Testing for Android Apps @bing 2024.9
2. Slide QR Code Sample App QR Code Sample app link: https://github.com/bingningO/screenshot- testing-sample-app 2
3. Self-Intro Bing X: Bing76869213 Github: bingningO Senior Android Engineer at From China! Favorites: Anime, Snowboarding, Tennis, Hiking 3
4. How confident are you when your changed UI? 4
5. 5 Still relying on QA to manually test the UI changes ?
6. UI Change in different devices? 6 Pictures from google.store Pictures from amazon.store
7. UI Change in different devices? 7 Pictures from unext.news
8. UI Change in shared components? 8
9. UI Change in big refactoring? 9 • Multi-Activities -> Single Activity yurihondo: Road to Single Activity at Hedgehog 9/13 14:20~ • Material Design M2 -> M3 Planning…
10. Why not use Screenshot Testing? 10
11. How it works Your UI 11 Take Screenshots Reference New Compare
12. 3-ways-diff Report 12 Pictures from google.doc
13. Screenshot Testing with other Testing Unit Testing UI Testing E2E Testing Deploy Instrumentation Testing Performance Testing refer: https://developer.android.com/training/testing 13
14. Screenshot Testing with other Testing Unit Testing UI Testing E2E Testing Screenshot Testing Deploy Instrumentation Testing Performance Testing refer: https://developer.android.com/training/testing Ensure visual consistency 14
15. Key Benefits 2 Automation of UI Verification 1 Visual Regression Detection 3 Consistency Across Devices 4 Enhanced Confidence in Refactoring 15
16. Agenda 01 | Implementation of Screenshot Testing ◦ Roborazzi ◦ Compose Preview Screenshot Testing 02 | Experience in My Project 03 | More Challenges ◦ Test for Adaptive Layout ◦ Test for Animations 16
17. 01 | Implementation of Screenshot Testing 17
18. Library List 18 Paparazzi Roborazzi Shot Screengrab Compose Preview Screenshot Testing Screenshot Tests for Android
19. Library List 19 Paparazzi Roborazzi Shot Screengrab Compose Preview Screenshot Testing Screenshot Tests for Android
20. Roborazzi 20
21. Roborazzi • An open-source library • Introduced by nowinandroid, droidkaigiApp2023, droidkaigiApp2024 • Works with Robolectric • Render previews: by Robolectric Native Graphics 21
22. Roborazzi — Basic Code 22
23. Roborazzi – Single Test @RunWith(AndroidJUnit4::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) @Config( qualifiers = RobolectricDeviceQualifiers.Pixel7, ) class SettingScreenTest { @get:Rule val composeTestRule = createComposeRule() @Test fun settingScreenTest() { composeTestRule.setContent { SettingScreenSuccess(...) } composeTestRule.onRoot().captureRoboImage() } } 23
24. Roborazzi – Single Test @RunWith(AndroidJUnit4::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) @Config( qualifiers = RobolectricDeviceQualifiers.Pixel7, ) class SettingScreenTest { @get:Rule val composeTestRule = createComposeRule() @Test fun settingScreenTest() { composeTestRule.setContent { SettingScreenSuccess(...) } composeTestRule.onRoot().captureRoboImage() } } Configure 24
25. Roborazzi – Single Test @RunWith(AndroidJUnit4::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) @Config( qualifiers = RobolectricDeviceQualifiers.Pixel7, ) class SettingScreenTest { @get:Rule val composeTestRule = createComposeRule() @Test fun settingScreenTest() { composeTestRule.setContent { SettingScreenSuccess(...) } composeTestRule.onRoot().captureRoboImage() } } 25 Test Rule
26. Roborazzi – Single Test @RunWith(AndroidJUnit4::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) @Config( qualifiers = RobolectricDeviceQualifiers.Pixel7, ) class SettingScreenTest { @get:Rule val composeTestRule = createComposeRule() @Test fun settingScreenTest() { composeTestRule.setContent { SettingScreenSuccess(...) } composeTestRule.onRoot().captureRoboImage() } } 26 Capture screenshot
27. Roborazzi – Single Test 27 Record Record current screenshots as reference images Compare Record current screenshots as new images, and compare with reference ones
28. Roborazzi – Single Test Result 28
29. What if, we want to run screenshot testing for all composables? 29
30. What if, we want to run screenshot testing for all composables? Don't try to write all tests by hands 30
31. Roborazzi — Automatic Testing 31
32. Roborazzi – Automatic Testing • Get all @Preview list ◦ Showkase • Take screenshot for them ◦ Roborazzi 32
33. @RunWith(ParameterizedRobolectricTestRunner::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ScreenshotTestForPreviews( private val showkaseBrowserComponent: ShowkaseBrowserComponent, ) { @get:Rule val composeTestRule = createComposeRule() @Test @Category(ScreenshotTestCategory::class) fun previewScreenshot() { captureRoboImage { showkaseBrowserComponent.component() } } companion object { @ParameterizedRobolectricTestRunner.Parameters @JvmStatic fun components(): Iterable<Array<Any?>> { return Showkase.getMetadata().componentList.map { component -> arrayOf(component) } } } } 33
34. @RunWith(ParameterizedRobolectricTestRunner::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ScreenshotTestForPreviews( private val showkaseBrowserComponent: ShowkaseBrowserComponent, ) { @get:Rule val composeTestRule = createComposeRule() @Test @Category(ScreenshotTestCategory::class) fun previewScreenshot() { captureRoboImage { showkaseBrowserComponent.component() } } companion object { @ParameterizedRobolectricTestRunner.Parameters @JvmStatic fun components(): Iterable<Array<Any?>> { return Showkase.getMetadata().componentList.map { component -> arrayOf(component) } } } } 34 Configure
35. @RunWith(ParameterizedRobolectricTestRunner::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ScreenshotTestForPreviews( private val showkaseBrowserComponent: ShowkaseBrowserComponent, ) { @get:Rule val composeTestRule = createComposeRule() @Test @Category(ScreenshotTestCategory::class) fun previewScreenshot() { captureRoboImage { showkaseBrowserComponent.component() } } companion object { @ParameterizedRobolectricTestRunner.Parameters @JvmStatic fun components(): Iterable<Array<Any?>> { return Showkase.getMetadata().componentList.map { component -> arrayOf(component) } } } } 35 Showkase: Get component list Showkase: Provide component list
36. @RunWith(ParameterizedRobolectricTestRunner::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ScreenshotTestForPreviews( private val showkaseBrowserComponent: ShowkaseBrowserComponent, ) { @get:Rule val composeTestRule = createComposeRule() @Test @Category(ScreenshotTestCategory::class) fun previewScreenshot() { captureRoboImage { showkaseBrowserComponent.component() } } companion object { @ParameterizedRobolectricTestRunner.Parameters @JvmStatic fun components(): Iterable<Array<Any?>> { return Showkase.getMetadata().componentList.map { component -> arrayOf(component) } } } } 36 Capture screenshot
37. Roborazzi – Automatic Testing Result 37
38. Compose Preview Screenshot Testing — Simplify Automatic Testing 38 CPST as abbreviated in this slides
39. • By Google, Experimental, mentioned in Google IO 2024 • Works with Compose, automatically test all previews • Works as easy as composable previews CPST refer: google android developer document 39
40. • By Google, Experimental, mentioned in Google IO 2024 • Works with Compose, automatically test all previews • Works as easy as composable previews CPST refer: google android developer document 40
41. 41 Junit Paparazzi Compose testing Robolectric UI Automator Truth Espresso Mocks Hamcrest Junit 5 Instrumentation
42. 42 Junit Paparazzi Compose testing Robolectric UI Automator Truth Espresso Mocks Hamcrest Junit 5 Instrumentation
43. 43 Junit Paparazzi Compose testing Robolectric UI Automator Truth Espresso Mocks Hamcrest Junit 5 Instrumentation No More Test Code
44. CPST - Setup refer: google android developer document • Ensure Environment • Configure gradle • Add @Preview under ./src/screenshotTest/ 44
45. CPST - Setup refer: google android developer document • Ensure Environment • Configure gradle • Add @Preview under ./src/screenshotTest/ 45 DONE！
46. CPST - Result 46
47. CPST - More Discussion refer: issuetracker.google • Feature request ◦ Support @Preview in main source set, link ◦ Customize configuration for screenshot report ◦ Support for more complex composables ◦ Support for Enhanced integration with other testing frameworks • Other ◦ The future of Compose preview screenshot testing v.s. Current solutions (e.g. Paparazzi, Roborazzi…), link1 , link2 47
48. Compare Common Feature Strengths Limitations Roborazzi • Support for Jetpack Compose • Capture and validate the UI difference including 1-dp change Flexiable • Both Compose and traditional Views • Support customize • Requires setup and integration • May have a steeper learning curve Compose Preview Screenshots Testing Fast • Integrated with the IDE • Compatible with Compose • Experimental • Limited to static @Preview composables 48
49. 02 | Experience in our project 49
50. 50 Automatic Screenshot Testing in Large Projects Roborazzi Showkase Multi-module Github Actions
51. 51 Configure for Multi-Module app ui domain network local database Data module
52. Screenshot Gradle Plugin 52 Configure for Multi-Module app ui domain network local database Data module Screenshot Testing ShowKase @Preview
53. 53 Configure for CI New pull request New merge into main Record reference images refer: the companion branch approach Compare new changes Comment in the PR
54. 54 Automation Result
55. 55 Running Status • Started over 4 months • 520 images, CI jobs take around 10 min in busy days
56. 56 Feedbacks
57. 57 Feedbacks • Good ◦ Easy to see UI change ◦ Give confidence to PR owners ◦ Good preparation for big refactoring • Encountered Issues ◦ Please check my speech in Shibuya.apk
58. 58 03 | More Challenges
59. 59 Long List Screen Foldable Screen Tablet
60. 60 @Preview for different screen sizes
61. 61 Challenges and Solutions Animations Adaptive Layout
62. 62 Screenshot Testing for Adaptive Layout
63. Test for Adaptive Layout 63 • Multiple device types • Orientations • Themes • Fonts • … Saiki Iijima: 使って知るCustomLayout. vs DailyScheduler at Hedgehog 9/13 15:20~ Custom Adaptive Layout Speech:
64. //@PreviewScreenSizes @PreviewFontScale @PreviewLightDark //@Preview(device = Devices.TABLET) //@Preview(device = Devices.PIXEL_7, backgroundColor = 0xFFFFFFFF) //@Preview(device = Devices.PIXEL_4) @Preview(widthDp = 360, heightDp = 640) @Preview(widthDp = 1920, heightDp = 1080) @Preview(widthDp = 1080, heightDp = 1920) @Composable fun AnimateScreenPreview() { AppTheme { AnimateScreen(modifier = Modifier.fillMaxSize()) } } CPST: natively supports it @Preview Code: Just add annotations 64
65. 65 CPST: Results
66. 66 CPST: Results
67. //@PreviewScreenSizes @PreviewFontScale @PreviewLightDark //@Preview(device = Devices.TABLET) //@Preview(device = Devices.PIXEL_7, backgroundColor = 0xFFFFFFFF) //@Preview(device = Devices.PIXEL_4) @Preview(widthDp = 360, heightDp = 640) @Preview(widthDp = 1920, heightDp = 1080) @Preview(widthDp = 1080, heightDp = 1920) @Composable fun AnimateScreenPreview() { AppTheme { AnimateScreen(modifier = Modifier.fillMaxSize()) } } The results nearly the same by @PreviewScreenS izes issue link 😫 67 CPST: issues
68. //@PreviewScreenSizes @PreviewFontScale @PreviewLightDark //@Preview(device = Devices.TABLET) //@Preview(device = Devices.PIXEL_7, backgroundColor = 0xFFFFFFFF) //@Preview(device = Devices.PIXEL_4) @Preview(widthDp = 360, heightDp = 640) @Preview(widthDp = 1920, heightDp = 1080) @Preview(widthDp = 1080, heightDp = 1920) @Composable fun AnimateScreenPreview() { AppTheme { AnimateScreen(modifier = Modifier.fillMaxSize()) } } Got build error by setting device type 😫, issue link 68 CPST: issues
69. @Test @Category(ScreenshotTestCategory::class) fun previewScreenshot_tablet() { val componentKey = showkaseBrowserComponent.componentKey val filePath = "$DEFAULT_ROBORAZZI_OUTPUT_DIR_PATH/tablet/$componentKey.png" println("componentKey tablet: $componentKey") RuntimeEnvironment.setQualifiers("w1280dp-h800dp-240dpi") captureRoboImage(filePath) } 69 Roborazzi: set different test environments refer: robolectric#device configuration
70. @Test @Category(ScreenshotTestCategory::class) fun previewScreenshot_tablet() { val componentKey = showkaseBrowserComponent.componentKey val filePath = "$DEFAULT_ROBORAZZI_OUTPUT_DIR_PATH/tablet/$componentKey.png" println("componentKey tablet: $componentKey") RuntimeEnvironment.setQualifiers("w1280dp-h800dp-240dpi") captureRoboImage(filePath) } 70 Roborazzi: set different test environments More Details
71. 71 Roborazzi: results
72. 72 Roborazzi: results
73. Test for Adaptive Layout Summary • Compose Preview Screenshot Testing ◦ Good: Natively supports adaptive layouts with simple annotations ◦ Pain: Still experimental, with unresolved bugs 73
74. Test for Adaptive Layout Summary • Roborazzi ◦ Good: Highly configurable for different test environments ◦ Pain: Requires manual setup and has some limitations 74
75. 75 Screenshot Testing for Animations
76. Animation Type • Android build-in animations ◦ e.g. progressbar, shared element transitions 76
77. Animation Type • Value-based animations ◦ e.g. translating, rotating, alpha changing 77
78. Animation Type • Animated vector images ◦ e.g. Lottie Animation 78
79. Animation Type • Android build-in animations ◦ e.g. progressbar, shared element transitions • Value-based animations ◦ e.g. translating, rotating, alpha changing • Animated vector images ◦ e.g. Lottie Animation 79
80. • Control the animation state manually • Capture screenshots at specific points ——————————————————————————— • Roborazzi ◦ Set state in single test code ◦ Or add state in @Preview • Compose Preview Testing ◦ add state in @Preview Challenge 1: Capture Animation States 80
81. @Test fun loadingContentTest_start() { composeTestRule.setContent { // set different progress state LoadingContent(progress = 0.0f) } composeTestRule.onRoot().captureRoboImage() } 81 Challenge 1: Test Code
82. 82 Challenge 1: Results
83. • Lottie Compose Code: set iterations as 1 • Test Code: ◦ wait until the composition is idle ◦ take screenshot ——————————————————————————— • Only works for Roborazzi • Compose Screenshot Testing issue link 83 Challenge 2: Capture Lottie Animation
84. val progress by animateLottieCompositionAsState( ... iterations = if (Build.FINGERPRINT == "robolectric") 1 else LottieConstants.IterateForever, ) LottieAnimation( ... composition = composition, progress = { progress }, ) 84 Challenge 2: Lottie Compose Code
85. @Test fun animateLottieTest() { composeTestRule.setContent { AnimateLottie() } // wait until the composition is idle composeTestRule.waitForIdle() testDispatcher.scheduler.advanceUntilIdle() // take a screenshot composeTestRule.onRoot().captureRoboImage() } 85 Challenge 2: Test Code
86. Screenshot result ⇨ ⇦ @Preview 86 Challenge 2: Results
87. • Simulate user interactions, by compose-ui-test • Capture the sequence of screenshots as a GIF ——————————————————————————— • Works as a feature of Roborazzi, refer ◦ Can’t be compared Challenge 3: Capture as a GIF 87
88. 88 Challenge 3: Test Code @get:Rule val roborazziRule = RoborazziRule( captureRoot = onView(ViewMatchers.isRoot()), options = RoborazziRule.Options( captureType = RoborazziRule.CaptureType.Gif(), ) ) @Test fun animateLottieTest() { composeTestRule.setContent { AppTheme { AnimateTab() } } waitUntilIdle(testRule = composeTestRule, testDispatcher = testDispatcher) composeTestRule.onNodeWithText("WORK").performClick() waitUntilIdle(testRule = composeTestRule, testDispatcher = testDispatcher) }
89. 89 Challenge 3: Value-based Animation Result
90. 90 Challenge 3: Lottie Animation Result
91. • Almost There: ◦ The solutions are mostly sufficient • Ideal Goal: ◦ A simpler, more streamlined approach. ◦ Smoother GIF production. ◦ Support for custom duration, frame rates, and other animation parameters. 91 Test for Animations Summary
92. A more complex use case 92
93. Use Case: Material3#ListDetailPaneScaffold onClick 93 Tablet-Size Phone-Size onClick refer: M3 design guide
94. • Roborazzi ◦ Set different screen sizes ◦ Take screenshot after clicking one item • CPST ◦ Does not work because of mentioned issues 94 Use Case: Material3#ListDetailPaneScaffold
95. 95 Use Case: Test Code @Test fun contactsListScreenTest_tablet() { RuntimeEnvironment.setQualifiers("w1280dp-h800dp-240dpi") ... composeTestRule.onNodeWithText("Google Express").performClick() composeTestRule.onAllNodes(isRoot()).onFirst().captureRoboImage() } @Test fun contactsListScreenTest_phone() { RuntimeEnvironment.setQualifiers("w411dp-h891dp-port") ... composeTestRule.onNodeWithText("Google Express").performClick() composeTestRule.onAllNodes(isRoot()).onFirst().captureRoboImage() }
96. phone-size tablet-size 96 Use Case: Results
97. Recap 01 | Implementation of Screenshot Testing ◦ Roborazzi ◦ Compose Preview Screenshot Testing 02 | Experience in My Project 03 | More Challenges ◦ Test for Adaptive Layout ◦ Test for Animations 97
98. 98 Feel free to check today’s examples in the Sample App bingningO/screenshot-testing-sample-app
99. Happy Testing! Thanks ご清聴ありがとうございました 99
100. Additional Data 100
101. Roborazzi – Summary • Good Points ◦ Fast and Light ◦ Free to do customize • Pain Points ◦ Hard to configure gradle ◦ Hard to analyse bugs ◦ Showkase has several limitations
102. CPST – Summary • Good Points ◦ Google official support ◦ Super easy to use ◦ Super good compatibility with compose @Preview 102
103. CPST – Summary • Pain Points ◦ Have to put all @Preview you want to test into ./screenshotTest source set ◦ Report always fail if having difference ◦ No customize settings at present ◦ Still experimental with some bugs 103