Mastering Screenshot Testing for Android Apps

Slide 1

Slide 1 text

Mastering Screenshot Testing for Android Apps @bing 2024.9

Slide 2

Slide 2 text

Slide QR Code Sample App QR Code Sample app link: https://github.com/bingningO/screenshot- testing-sample-app 2

Slide 3

Slide 3 text

Self-Intro Bing X: Bing76869213 Github: bingningO Senior Android Engineer at From China! Favorites: Anime, Snowboarding, Tennis, Hiking 3

Slide 4

Slide 4 text

How conﬁdent are you when your changed UI? 4

Slide 5

Slide 5 text

5 Still relying on QA to manually test the UI changes ?

Slide 6

Slide 6 text

UI Change in different devices? 6 Pictures from google.store Pictures from amazon.store

Slide 7

Slide 7 text

UI Change in different devices? 7 Pictures from unext.news

Slide 8

Slide 8 text

UI Change in shared components? 8

Slide 9

Slide 9 text

UI Change in big refactoring? 9 ● Multi-Activities -> Single Activity yurihondo: Road to Single Activity at Hedgehog 9/13 14:20~ ● Material Design M2 -> M3 Planning…

Slide 10

Slide 10 text

Why not use Screenshot Testing? 10

Slide 11

Slide 11 text

How it works Your UI 11 Take Screenshots Reference New Compare

Slide 12

Slide 12 text

3-ways-diff Report 12 Pictures from google.doc

Slide 13

Slide 13 text

Screenshot Testing with other Testing Unit Testing UI Testing E2E Testing Deploy Instrumentation Testing Performance Testing refer: https://developer.android.com/training/testing 13

Slide 14

Slide 14 text

Screenshot Testing with other Testing Unit Testing UI Testing E2E Testing Screenshot Testing Deploy Instrumentation Testing Performance Testing refer: https://developer.android.com/training/testing Ensure visual consistency 14

Slide 15

Slide 15 text

Key Benefits 2 Automation of UI Verification 1 Visual Regression Detection 3 Consistency Across Devices 4 Enhanced Confidence in Refactoring 15

Slide 16

Slide 16 text

Agenda 01 | Implementation of Screenshot Testing ○ Roborazzi ○ Compose Preview Screenshot Testing 02 | Experience in My Project 03 | More Challenges ○ Test for Adaptive Layout ○ Test for Animations 16

Slide 17

Slide 17 text

01 | Implementation of Screenshot Testing 17

Slide 18

Slide 18 text

Library List 18 Paparazzi Roborazzi Shot Screengrab Compose Preview Screenshot Testing Screenshot Tests for Android

Slide 19

Slide 19 text

Library List 19 Paparazzi Roborazzi Shot Screengrab Compose Preview Screenshot Testing Screenshot Tests for Android

Slide 20

Slide 20 text

Roborazzi 20

Slide 21

Slide 21 text

Roborazzi ● An open-source library ● Introduced by nowinandroid, droidkaigiApp2023, droidkaigiApp2024 ● Works with Robolectric ● Render previews: by Robolectric Native Graphics 21

Slide 22

Slide 22 text

Roborazzi — Basic Code 22

Slide 23

Slide 23 text

Roborazzi – Single Test @RunWith(AndroidJUnit4::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) @Config( qualifiers = RobolectricDeviceQualifiers.Pixel7, ) class SettingScreenTest { @get:Rule val composeTestRule = createComposeRule() @Test fun settingScreenTest() { composeTestRule.setContent { SettingScreenSuccess(...) } composeTestRule.onRoot().captureRoboImage() } } 23

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Roborazzi – Single Test 27 Record Record current screenshots as reference images Compare Record current screenshots as new images, and compare with reference ones

Slide 28

Slide 28 text

Roborazzi – Single Test Result 28

Slide 29

Slide 29 text

What if, we want to run screenshot testing for all composables? 29

Slide 30

Slide 30 text

What if, we want to run screenshot testing for all composables? Don't try to write all tests by hands 30

Slide 31

Slide 31 text

Roborazzi — Automatic Testing 31

Slide 32

Slide 32 text

Roborazzi – Automatic Testing ● Get all @Preview list ○ Showkase ● Take screenshot for them ○ Roborazzi 32

Slide 33

Slide 33 text

@RunWith(ParameterizedRobolectricTestRunner::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ScreenshotTestForPreviews( private val showkaseBrowserComponent: ShowkaseBrowserComponent, ) { @get:Rule val composeTestRule = createComposeRule() @Test @Category(ScreenshotTestCategory::class) fun previewScreenshot() { captureRoboImage { showkaseBrowserComponent.component() } } companion object { @ParameterizedRobolectricTestRunner.Parameters @JvmStatic fun components(): Iterable> { return Showkase.getMetadata().componentList.map { component -> arrayOf(component) } } } } 33

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Roborazzi – Automatic Testing Result 37

Slide 38

Slide 38 text

Compose Preview Screenshot Testing — Simplify Automatic Testing 38 CPST as abbreviated in this slides

Slide 39

Slide 39 text

● By Google, Experimental, mentioned in Google IO 2024 ● Works with Compose, automatically test all previews ● Works as easy as composable previews CPST refer: google android developer document 39

Slide 40

Slide 40 text

Slide 41

Slide 41 text

41 Junit Paparazzi Compose testing Robolectric UI Automator Truth Espresso Mocks Hamcrest Junit 5 Instrumentation

Slide 42

Slide 42 text

42 Junit Paparazzi Compose testing Robolectric UI Automator Truth Espresso Mocks Hamcrest Junit 5 Instrumentation

Slide 43

Slide 43 text

43 Junit Paparazzi Compose testing Robolectric UI Automator Truth Espresso Mocks Hamcrest Junit 5 Instrumentation No More Test Code

Slide 44

Slide 44 text

CPST - Setup refer: google android developer document ● Ensure Environment ● Conﬁgure gradle ● Add @Preview under ./src/screenshotTest/ 44

Slide 45

Slide 45 text

CPST - Setup refer: google android developer document ● Ensure Environment ● Conﬁgure gradle ● Add @Preview under ./src/screenshotTest/ 45 DONE！

Slide 46

Slide 46 text

CPST - Result 46

Slide 47

Slide 47 text

CPST - More Discussion refer: issuetracker.google ● Feature request ○ Support @Preview in main source set, link ○ Customize conﬁguration for screenshot report ○ Support for more complex composables ○ Support for Enhanced integration with other testing frameworks ● Other ○ The future of Compose preview screenshot testing v.s. Current solutions (e.g. Paparazzi, Roborazzi…), link1 , link2 47

Slide 48

Slide 48 text

Compare Common Feature Strengths Limitations Roborazzi ● Support for Jetpack Compose ● Capture and validate the UI difference including 1-dp change Flexiable ● Both Compose and traditional Views ● Support customize ● Requires setup and integration ● May have a steeper learning curve Compose Preview Screenshots Testing Fast ● Integrated with the IDE ● Compatible with Compose ● Experimental ● Limited to static @Preview composables 48

Slide 49

Slide 49 text

02 | Experience in our project 49

Slide 50

Slide 50 text

50 Automatic Screenshot Testing in Large Projects Roborazzi Showkase Multi-module Github Actions

Slide 51

Slide 51 text

51 Conﬁgure for Multi-Module app ui domain network local database Data module

Slide 52

Slide 52 text

Screenshot Gradle Plugin 52 Conﬁgure for Multi-Module app ui domain network local database Data module Screenshot Testing ShowKase @Preview

Slide 53

Slide 53 text

53 Conﬁgure for CI New pull request New merge into main Record reference images refer: the companion branch approach Compare new changes Comment in the PR

Slide 54

Slide 54 text

54 Automation Result

Slide 55

Slide 55 text

55 Running Status ● Started over 4 months ● 520 images, CI jobs take around 10 min in busy days

Slide 56

Slide 56 text

56 Feedbacks

Slide 57

Slide 57 text

57 Feedbacks ● Good ○ Easy to see UI change ○ Give conﬁdence to PR owners ○ Good preparation for big refactoring ● Encountered Issues ○ Please check my speech in Shibuya.apk

Slide 58

Slide 58 text

58 03 | More Challenges

Slide 59

Slide 59 text

59 Long List Screen Foldable Screen Tablet

Slide 60

Slide 60 text

60 @Preview for different screen sizes

Slide 61

Slide 61 text

61 Challenges and Solutions Animations Adaptive Layout

Slide 62

Slide 62 text

62 Screenshot Testing for Adaptive Layout

Slide 63

Slide 63 text

Test for Adaptive Layout 63 ● Multiple device types ● Orientations ● Themes ● Fonts ● … Saiki Iijima: 使って知るCustomLayout. vs DailyScheduler at Hedgehog 9/13 15:20~ Custom Adaptive Layout Speech:

Slide 64

Slide 64 text

//@PreviewScreenSizes @PreviewFontScale @PreviewLightDark //@Preview(device = Devices.TABLET) //@Preview(device = Devices.PIXEL_7, backgroundColor = 0xFFFFFFFF) //@Preview(device = Devices.PIXEL_4) @Preview(widthDp = 360, heightDp = 640) @Preview(widthDp = 1920, heightDp = 1080) @Preview(widthDp = 1080, heightDp = 1920) @Composable fun AnimateScreenPreview() { AppTheme { AnimateScreen(modifier = Modifier.fillMaxSize()) } } CPST: natively supports it @Preview Code: Just add annotations 64

Slide 65

Slide 65 text

65 CPST: Results

Slide 66

Slide 66 text

66 CPST: Results

Slide 67

Slide 67 text

//@PreviewScreenSizes @PreviewFontScale @PreviewLightDark //@Preview(device = Devices.TABLET) //@Preview(device = Devices.PIXEL_7, backgroundColor = 0xFFFFFFFF) //@Preview(device = Devices.PIXEL_4) @Preview(widthDp = 360, heightDp = 640) @Preview(widthDp = 1920, heightDp = 1080) @Preview(widthDp = 1080, heightDp = 1920) @Composable fun AnimateScreenPreview() { AppTheme { AnimateScreen(modifier = Modifier.fillMaxSize()) } } The results nearly the same by @PreviewScreenS izes issue link 😫 67 CPST: issues

Slide 68

Slide 68 text

//@PreviewScreenSizes @PreviewFontScale @PreviewLightDark //@Preview(device = Devices.TABLET) //@Preview(device = Devices.PIXEL_7, backgroundColor = 0xFFFFFFFF) //@Preview(device = Devices.PIXEL_4) @Preview(widthDp = 360, heightDp = 640) @Preview(widthDp = 1920, heightDp = 1080) @Preview(widthDp = 1080, heightDp = 1920) @Composable fun AnimateScreenPreview() { AppTheme { AnimateScreen(modifier = Modifier.fillMaxSize()) } } Got build error by setting device type 😫, issue link 68 CPST: issues

Slide 69

Slide 69 text

@Test @Category(ScreenshotTestCategory::class) fun previewScreenshot_tablet() { val componentKey = showkaseBrowserComponent.componentKey val filePath = "$DEFAULT_ROBORAZZI_OUTPUT_DIR_PATH/tablet/$componentKey.png" println("componentKey tablet: $componentKey") RuntimeEnvironment.setQualifiers("w1280dp-h800dp-240dpi") captureRoboImage(filePath) } 69 Roborazzi: set different test environments refer: robolectric#device conﬁguration

Slide 70

Slide 70 text

@Test @Category(ScreenshotTestCategory::class) fun previewScreenshot_tablet() { val componentKey = showkaseBrowserComponent.componentKey val filePath = "$DEFAULT_ROBORAZZI_OUTPUT_DIR_PATH/tablet/$componentKey.png" println("componentKey tablet: $componentKey") RuntimeEnvironment.setQualifiers("w1280dp-h800dp-240dpi") captureRoboImage(filePath) } 70 Roborazzi: set different test environments More Details

Slide 71

Slide 71 text

71 Roborazzi: results

Slide 72

Slide 72 text

72 Roborazzi: results

Slide 73

Slide 73 text

Test for Adaptive Layout Summary ● Compose Preview Screenshot Testing ○ Good: Natively supports adaptive layouts with simple annotations ○ Pain: Still experimental, with unresolved bugs 73

Slide 74

Slide 74 text

Test for Adaptive Layout Summary ● Roborazzi ○ Good: Highly conﬁgurable for different test environments ○ Pain: Requires manual setup and has some limitations 74

Slide 75

Slide 75 text

75 Screenshot Testing for Animations

Slide 76

Slide 76 text

Animation Type ● Android build-in animations ○ e.g. progressbar, shared element transitions 76

Slide 77

Slide 77 text

Animation Type ● Value-based animations ○ e.g. translating, rotating, alpha changing 77

Slide 78

Slide 78 text

Animation Type ● Animated vector images ○ e.g. Lottie Animation 78

Slide 79

Slide 79 text

Animation Type ● Android build-in animations ○ e.g. progressbar, shared element transitions ● Value-based animations ○ e.g. translating, rotating, alpha changing ● Animated vector images ○ e.g. Lottie Animation 79

Slide 80

Slide 80 text

● Control the animation state manually ● Capture screenshots at speciﬁc points ——————————————————————————— ● Roborazzi ○ Set state in single test code ○ Or add state in @Preview ● Compose Preview Testing ○ add state in @Preview Challenge 1: Capture Animation States 80

Slide 81

Slide 81 text

@Test fun loadingContentTest_start() { composeTestRule.setContent { // set different progress state LoadingContent(progress = 0.0f) } composeTestRule.onRoot().captureRoboImage() } 81 Challenge 1: Test Code

Slide 82

Slide 82 text

82 Challenge 1: Results

Slide 83

Slide 83 text

● Lottie Compose Code: set iterations as 1 ● Test Code: ○ wait until the composition is idle ○ take screenshot ——————————————————————————— ● Only works for Roborazzi ● Compose Screenshot Testing issue link 83 Challenge 2: Capture Lottie Animation

Slide 84

Slide 84 text

val progress by animateLottieCompositionAsState( ... iterations = if (Build.FINGERPRINT == "robolectric") 1 else LottieConstants.IterateForever, ) LottieAnimation( ... composition = composition, progress = { progress }, ) 84 Challenge 2: Lottie Compose Code

Slide 85

Slide 85 text

@Test fun animateLottieTest() { composeTestRule.setContent { AnimateLottie() } // wait until the composition is idle composeTestRule.waitForIdle() testDispatcher.scheduler.advanceUntilIdle() // take a screenshot composeTestRule.onRoot().captureRoboImage() } 85 Challenge 2: Test Code

Slide 86

Slide 86 text

Screenshot result ⇨ ⇦ @Preview 86 Challenge 2: Results

Slide 87

Slide 87 text

● Simulate user interactions, by compose-ui-test ● Capture the sequence of screenshots as a GIF ——————————————————————————— ● Works as a feature of Roborazzi, refer ○ Can’t be compared Challenge 3: Capture as a GIF 87

Slide 88

Slide 88 text

88 Challenge 3: Test Code @get:Rule val roborazziRule = RoborazziRule( captureRoot = onView(ViewMatchers.isRoot()), options = RoborazziRule.Options( captureType = RoborazziRule.CaptureType.Gif(), ) ) @Test fun animateLottieTest() { composeTestRule.setContent { AppTheme { AnimateTab() } } waitUntilIdle(testRule = composeTestRule, testDispatcher = testDispatcher) composeTestRule.onNodeWithText("WORK").performClick() waitUntilIdle(testRule = composeTestRule, testDispatcher = testDispatcher) }

Slide 89

Slide 89 text

89 Challenge 3: Value-based Animation Result

Slide 90

Slide 90 text

90 Challenge 3: Lottie Animation Result

Slide 91

Slide 91 text

● Almost There: ○ The solutions are mostly suﬃcient ● Ideal Goal: ○ A simpler, more streamlined approach. ○ Smoother GIF production. ○ Support for custom duration, frame rates, and other animation parameters. 91 Test for Animations Summary

Slide 92

Slide 92 text

A more complex use case 92

Slide 93

Slide 93 text

Use Case: Material3#ListDetailPaneScaffold onClick 93 Tablet-Size Phone-Size onClick refer: M3 design guide

Slide 94

Slide 94 text

● Roborazzi ○ Set different screen sizes ○ Take screenshot after clicking one item ● CPST ○ Does not work because of mentioned issues 94 Use Case: Material3#ListDetailPaneScaffold

Slide 95

Slide 95 text

95 Use Case: Test Code @Test fun contactsListScreenTest_tablet() { RuntimeEnvironment.setQualifiers("w1280dp-h800dp-240dpi") ... composeTestRule.onNodeWithText("Google Express").performClick() composeTestRule.onAllNodes(isRoot()).onFirst().captureRoboImage() } @Test fun contactsListScreenTest_phone() { RuntimeEnvironment.setQualifiers("w411dp-h891dp-port") ... composeTestRule.onNodeWithText("Google Express").performClick() composeTestRule.onAllNodes(isRoot()).onFirst().captureRoboImage() }

Slide 96

Slide 96 text

phone-size tablet-size 96 Use Case: Results

Slide 97

Slide 97 text

Recap 01 | Implementation of Screenshot Testing ○ Roborazzi ○ Compose Preview Screenshot Testing 02 | Experience in My Project 03 | More Challenges ○ Test for Adaptive Layout ○ Test for Animations 97

Slide 98

Slide 98 text

98 Feel free to check today’s examples in the Sample App bingningO/screenshot-testing-sample-app

Slide 99

Slide 99 text

Happy Testing! Thanks ご清聴ありがとうございました 99

Slide 100

Slide 100 text

Additional Data 100

Slide 101

Slide 101 text

Roborazzi – Summary ● Good Points ○ Fast and Light ○ Free to do customize ● Pain Points ○ Hard to conﬁgure gradle ○ Hard to analyse bugs ○ Showkase has several limitations

Slide 102

Slide 102 text

CPST – Summary ● Good Points ○ Google oﬃcial support ○ Super easy to use ○ Super good compatibility with compose @Preview 102

Slide 103

Slide 103 text

CPST – Summary ● Pain Points ○ Have to put all @Preview you want to test into ./screenshotTest source set ○ Report always fail if having difference ○ No customize settings at present ○ Still experimental with some bugs 103