A to Z of snapshot testing in Android

Transcript

1. A to Z of snapshot testing in Android
2. ≠ ≠ = Snapshot
3. Snapshot Test UI Test ~ 1000 ~ 1600 ~ 2300 ~30,000 handful ~ 500 ~ 20 ? Ref: “Building Mobile App At scale: 39 Engineering Challenge”
4. When Card Render, 1. Text should show 2. Icon that looks like location pin with plus icon should show 3. Icon should be on left and Text on right 4. Content have padding of 64dp 5. It should have a background color of #XXXXXX and content tint color of #XXXXXX
5. When Card Render, Text should show fun shouldShowText() { composeTestRule.setContent { ClickableCard() } composeTestRule .onNodeWithText("Click Me!") .assertIsDisplayed() }
6. When Card Render, 1. Text should show 2. Icon that looks like location pin with plus icon should show 3. Icon should be on left and Text on right 4. Content have padding of 64dp 5. It should have a background color of #XXXXXX and content tint color of #XXXXXX
7. When Card Render, 1. Text should show 2. Icon that looks like location pin with plus icon should show 3. Icon should be on left and Text on right 4. Content have padding of 64dp 5. It should have a background color of #XXXXXX and content tint color of #XXXXXX
8. When Card Render, 1. Text should show 2. Icon that looks like location pin with plus icon should show 3. Icon should be on left and Text on right 4. Content have padding of 64dp 5. It should have a background color of #XXXXXX and content tint color of #XXXXXX
9. When Card Render, 1. Text should show 2. Icon that looks like location pin with plus icon should show 3. Icon should be on left and Text on right 4. Content have padding of 64dp 5. It should have a background color of #XXXXXX and content tint color of #XXXXXX
10. Photo by Erica Magugliani on Unsplash UI Test • Cannot test visual aspect of a component • Slow to execute • Require complex setup
11. Snapshot Test UI Test Interaction bugs (Scroll, clicks, etc) Visual bugs (Styling, spacing, theme etc)
12. Visual Bugs 1. Changes from libraries 2. Styling, Padding, Margins, Colors, Theme etc 3. Layout edge cases a. Long texts b. Constraints c. Internationalization (RTL Language, fonts) d. Accessibility (Bigger Font Size, Color Contrast)
13. Visual Bugs 1. Changes from libraries 2. Styling, Padding, Margins, Colors, Theme etc 3. Layout edge cases a. Long texts b. Constraints c. Internationalization (RTL Language, fonts) d. Accessibility (Bigger Font Size, Color Contrast)
14. Visual Bugs 1. Changes from libraries 2. Styling, Padding, Margins, Colors, Theme etc 3. Layout edge cases a. Long texts b. Constraints c. Internationalization (RTL Language, fonts) d. Accessibility (Bigger Font Size, Color Contrast)
15. Visual Bugs 1. Changes from libraries 2. Styling, Padding, Margins, Colors, Theme etc 3. Layout edge cases a. Long texts b. Constraints c. Internationalization (RTL Language, fonts) d. Accessibility (Bigger Font Size, Color Contrast)
16. Visual Bugs 1. Changes from libraries 2. Styling, Padding, Margins, Colors, Theme etc 3. Layout edge cases a. Long texts b. Constraints c. Internationalization (RTL Language, fonts) d. Accessibility (Bigger Font Size, Color Contrast)
17. Visual Bugs 1. Changes from libraries 2. Styling, Padding, Margins, Colors, Theme etc 3. Layout edge cases a. Long texts b. Constraints c. Internationalization (RTL Language, fonts) d. Accessibility (Bigger Font Size, Color Contrast)
21. Tools are great but manual validation is tedious and hard
22. Tools are great but manual validation is tedious and hard and they don’t catch regressions
23. Snapshot Testing Validate app’s appearance by using a screenshot or a snapshot as a reference to compare against.
24. Take a snapshot of the component and store it as a reference for verification Record Verify Compare the last snapshot against the current changes to see if it matches
25. Record Process Dev make changes and record a new snapshot PR submitted
26. Record Process Dev make changes and record a new snapshot LGTM! PR submitted
27. Record Process Dev make changes and record a new snapshot LGTM! PR submitted Store snapshot for verification for future changes
28. Verify Process PR submitted
29. Verify Process PR submitted
30. Verify Process PR submitted
31. Verify Process PR submitted
32. Verify Process PR submitted • Forgot to Record • Regression
33. Verify Process PR submitted • Forgot to Record • Regression
34. Verify Process PR submitted • Forgot to Record • Regression
35. Configuration Matters!
36. Where to start Snapshot Testing Photo by Shalaka Gamage on Unsplash
37. What go with? 1. Screen level snapshot testing 2. Component level snapshot testing?
38. Screen level snapshot testing 1. A lot of dependencies
39. Screen level snapshot testing 1. A lot of dependencies 2. Too many states
40. Screen level snapshot testing 1. A lot of dependencies 2. Too many states 3. More components = More flakiness
41. Screen level snapshot testing 1. A lot of dependencies 2. Too many states 3. More components = More flakiness 4. Hard to trace which changes breaks
43. App Bar are likely not to change, not priority to test
44. App Bar are likely not to change, not priority to test Using Library component, not priority to test
45. • Main feature of the app • Has different states
46. • Main feature of the app • Has different states Important text
47. App Bar are likely not to change, not priority to test • Main feature of the app • Has different states Important text Using Library component, not priority to test
48. 1. Most used features 2. Components that are being iterated quickly 3. Important text bodies and messages
49. Tooling Photo by Brendan Steeves on Unsplash
50. Tooling - pedrovgs/Shot - shopify/android-testify* - dropbox/dropshots - takahirom/roborazzi - QuickBirdEng/kotlin-snapshot-testing - facebook/screenshot-tests-for-android
51. Roborazzi A snapshot testing framework built upon square’s paparazzi to make it work with Robolectric
52. • Main feature of the app • Has different states
53. @Composable fun ProjectCard() { val project = loadData() Column { ProjectIcon(project) Row { Text(project.name) ... } } }
54. @Composable fun ProjectCard() { val project = loadData() Column { ProjectIcon(project) Row { Text(project.name) ... } } }
55. Use a stateless composable @Composable fun ProjectCard() { val project = loadData() ProjectCardContent(project) } @Composable fun ProjectCardContent( val project: Project ) { // Render here } Snapshot test UI Test
56. @RunWith(AndroidJUnit4::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ProjectCardScreenshotTest { @get:Rule val composeTestRule = createComposeRule() @Test fun screenshotTest() { composeTestRule.setContent { ProjectCard(...) } composeTestRule.onRoot().captureRoboImage() } }
57. @RunWith(AndroidJUnit4::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ProjectCardScreenshotTest { @get:Rule val composeTestRule = createComposeRule() @Test fun screenshotTest() { composeTestRule.setContent { ProjectCard(...) } composeTestRule.onRoot().captureRoboImage() } }
58. @RunWith(AndroidJUnit4::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ProjectCardScreenshotTest { @get:Rule val composeTestRule = createComposeRule() @Test fun screenshotTest() { composeTestRule.setContent { ProjectCard(...) } composeTestRule.onRoot().captureRoboImage() } }
59. @RunWith(AndroidJUnit4::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ProjectCardScreenshotTest { @get:Rule val composeTestRule = createComposeRule() @Test fun screenshotTest() { composeTestRule.setContent { ProjectCard(...) } composeTestRule.onRoot().captureRoboImage() } }
60. @RunWith(AndroidJUnit4::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ProjectCardScreenshotTest { @get:Rule val composeTestRule = createComposeRule() @Test fun screenshotTest() { composeTestRule.setContent { ProjectCard(...) } composeTestRule.onRoot().captureRoboImage() } }
61. ./gradlew recordRoborazziDebug ./gradlew testDebugUnitTest -Proborazzi.test.record=true ./gradlew verifyRoborazziDebug ./gradlew testDebugUnitTest -Proborazzi.test.verify=true
62. ./gradlew recordRoborazziDebug ./gradlew testDebugUnitTest -Proborazzi.test.record=true ./gradlew verifyRoborazziDebug ./gradlew testDebugUnitTest -Proborazzi.test.verify=true
63. ./gradlew recordRoborazziDebug ./gradlew testDebugUnitTest -Proborazzi.test.record=true ./gradlew verifyRoborazziDebug ./gradlew testDebugUnitTest -Proborazzi.test.verify=true
64. ./gradlew compareRoborazziDebug ./gradlew testDebugUnitTest -Proborazzi.test.compare=true
65. @RunWith(AndroidJUnit4::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ProjectCardScreenshotTest { @get:Rule val composeTestRule = createComposeRule() @Test fun screenshotTest() { composeTestRule.setContent { ProjectCard(...) } composeTestRule.onRoot().captureRoboImage() } }
66. @RunWith(ParameterizedRobolectricTestRunner::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ProjectCardScreenshotTest( private val config: DeviceConfig ) { companion object { @JvmStatic @Parameters fun testParamsProvider() = listOf( DeviceConfig("Small_Phone_Font_Small", RobolectricDeviceQualifiers.SmallPhone, 0.5f), DeviceConfig("Small_Phone_Font_Normal", RobolectricDeviceQualifiers.SmallPhone, 1.0f), DeviceConfig("Medium_Tablet_Font_Large", RobolectricDeviceQualifiers.MediumTablet, 2.0f), ) } @Test fun screenshotTest() { RuntimeEnvironment.setFontScale(config.fontScale) RuntimeEnvironment.setQualifiers(config.qualifier) ... } } Use Parameterized Runner for testing multiple device configuration
67. @RunWith(ParameterizedRobolectricTestRunner::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ProjectCardScreenshotTest( private val config: DeviceConfig ) { companion object { @JvmStatic @Parameters fun testParamsProvider() = listOf( DeviceConfig("Small_Phone_Font_Small", RobolectricDeviceQualifiers.SmallPhone, 0.5f), DeviceConfig("Small_Phone_Font_Normal", RobolectricDeviceQualifiers.SmallPhone, 1.0f), DeviceConfig("Medium_Tablet_Font_Large", RobolectricDeviceQualifiers.MediumTablet, 2.0f), ) } @Test fun screenshotTest() { RuntimeEnvironment.setFontScale(config.fontScale) RuntimeEnvironment.setQualifiers(config.qualifier) ... } } Use Parameterized Runner for testing multiple device configuration
68. @RunWith(ParameterizedRobolectricTestRunner::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ProjectCardScreenshotTest( private val config: DeviceConfig ) { companion object { @JvmStatic @Parameters fun testParamsProvider() = listOf( DeviceConfig("Small_Phone_Font_Small", RobolectricDeviceQualifiers.SmallPhone, 0.5f), DeviceConfig("Small_Phone_Font_Normal", RobolectricDeviceQualifiers.SmallPhone, 1.0f), DeviceConfig("Medium_Tablet_Font_Large", RobolectricDeviceQualifiers.MediumTablet, 2.0f), ) } @Test fun screenshotTest() { RuntimeEnvironment.setFontScale(config.fontScale) RuntimeEnvironment.setQualifiers(config.qualifier) ... } } Use Parameterized Runner for testing multiple device configuration
69. @RunWith(ParameterizedRobolectricTestRunner::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ProjectCardScreenshotTest( private val config: DeviceConfig ) { companion object { @JvmStatic @Parameters fun testParamsProvider() = listOf( DeviceConfig("Small_Phone_Font_Small", RobolectricDeviceQualifiers.SmallPhone, 0.5f), DeviceConfig("Small_Phone_Font_Normal", RobolectricDeviceQualifiers.SmallPhone, 1.0f), DeviceConfig("Medium_Tablet_Font_Large", RobolectricDeviceQualifiers.MediumTablet, 2.0f), ) } @Test fun screenshotTest() { RuntimeEnvironment.setFontScale(config.fontScale) RuntimeEnvironment.setQualifiers(config.qualifier) ... } } Use Parameterized Runner for testing multiple device configuration
70. Small Phone with Font Scale 0.5 Small Phone with Font Scale 1.0 Medium Tablet with Font Scale 2.0
71. Road to Snapshot Testing 1. Analyze which are to test, start with small & critical components 2. Scale up to different configurations 3. Integrate into your development workflow 4. Happy Snapshot Testing!
72. Good read - Introduction to snapshot testing blog series - Android screenshot testing playground - AndroidUITesting Utils - NowinAndroid bit.ly/droid-snap-read
73. A to Z of snapshot testing in Android aungkyawpaing.dev https://github.com/vincent-paing/