Slide 1
Slide 1 text
A to Z of
snapshot testing
in Android
Slide 2
Slide 2 text
≠
≠
=
Snapshot
Slide 3
Slide 3 text
Snapshot Test UI Test
~ 1000
~ 1600
~ 2300
~30,000
handful
~ 500
~ 20
?
Ref: “Building Mobile App At scale: 39 Engineering Challenge”
Slide 4
Slide 4 text
When Card Render,
1. Text should show
2. Icon that looks like location pin with
plus icon should show
3. Icon should be on left and Text on right
4. Content have padding of 64dp
5. It should have a background color of
#XXXXXX and content tint color of
#XXXXXX
Slide 5
Slide 5 text
When Card Render, Text
should show
fun shouldShowText() {
composeTestRule.setContent {
ClickableCard()
}
composeTestRule
.onNodeWithText("Click Me!")
.assertIsDisplayed()
}
Slide 6
Slide 6 text
When Card Render,
1. Text should show
2. Icon that looks like location pin with
plus icon should show
3. Icon should be on left and Text on right
4. Content have padding of 64dp
5. It should have a background color of
#XXXXXX and content tint color of
#XXXXXX
Slide 7
Slide 7 text
When Card Render,
1. Text should show
2. Icon that looks like location pin with
plus icon should show
3. Icon should be on left and Text on right
4. Content have padding of 64dp
5. It should have a background color of
#XXXXXX and content tint color of
#XXXXXX
Slide 8
Slide 8 text
When Card Render,
1. Text should show
2. Icon that looks like location pin with
plus icon should show
3. Icon should be on left and Text on right
4. Content have padding of 64dp
5. It should have a background color of
#XXXXXX and content tint color of
#XXXXXX
Slide 9
Slide 9 text
When Card Render,
1. Text should show
2. Icon that looks like location pin with
plus icon should show
3. Icon should be on left and Text on right
4. Content have padding of 64dp
5. It should have a background color of
#XXXXXX and content tint color of
#XXXXXX
Slide 10
Slide 10 text
Photo by Erica Magugliani on Unsplash
UI Test
● Cannot test visual aspect of a
component
● Slow to execute
● Require complex setup
Slide 11
Slide 11 text
Snapshot Test
UI Test
Interaction bugs
(Scroll, clicks, etc)
Visual bugs
(Styling, spacing, theme etc)
Slide 12
Slide 12 text
Visual Bugs
1. Changes from libraries
2. Styling, Padding, Margins, Colors, Theme etc
3. Layout edge cases
a. Long texts
b. Constraints
c. Internationalization (RTL Language, fonts)
d. Accessibility (Bigger Font Size, Color Contrast)
Slide 13
Slide 13 text
Visual Bugs
1. Changes from libraries
2. Styling, Padding, Margins, Colors, Theme etc
3. Layout edge cases
a. Long texts
b. Constraints
c. Internationalization (RTL Language, fonts)
d. Accessibility (Bigger Font Size, Color Contrast)
Slide 14
Slide 14 text
Visual Bugs
1. Changes from libraries
2. Styling, Padding, Margins, Colors, Theme etc
3. Layout edge cases
a. Long texts
b. Constraints
c. Internationalization (RTL Language, fonts)
d. Accessibility (Bigger Font Size, Color Contrast)
Slide 15
Slide 15 text
Visual Bugs
1. Changes from libraries
2. Styling, Padding, Margins, Colors, Theme etc
3. Layout edge cases
a. Long texts
b. Constraints
c. Internationalization (RTL Language, fonts)
d. Accessibility (Bigger Font Size, Color Contrast)
Slide 16
Slide 16 text
Visual Bugs
1. Changes from libraries
2. Styling, Padding, Margins, Colors, Theme etc
3. Layout edge cases
a. Long texts
b. Constraints
c. Internationalization (RTL Language, fonts)
d. Accessibility (Bigger Font Size, Color Contrast)
Slide 17
Slide 17 text
Visual Bugs
1. Changes from libraries
2. Styling, Padding, Margins, Colors, Theme etc
3. Layout edge cases
a. Long texts
b. Constraints
c. Internationalization (RTL Language, fonts)
d. Accessibility (Bigger Font Size, Color Contrast)
Slide 18
Slide 18 text
No content
Slide 19
Slide 19 text
No content
Slide 20
Slide 20 text
No content
Slide 21
Slide 21 text
Tools are great but manual
validation is tedious and
hard
Slide 22
Slide 22 text
Tools are great but manual
validation is tedious and
hard and they don’t catch
regressions
Slide 23
Slide 23 text
Snapshot Testing
Validate app’s appearance by using a
screenshot or a snapshot as a reference to
compare against.
Slide 24
Slide 24 text
Take a snapshot of the component and
store it as a reference for verification
Record
Verify Compare the last snapshot against the
current changes to see if it matches
Slide 25
Slide 25 text
Record Process
Dev make changes and
record a new snapshot
PR
submitted
Slide 26
Slide 26 text
Record Process
Dev make changes and
record a new snapshot
LGTM!
PR
submitted
Slide 27
Slide 27 text
Record Process
Dev make changes and
record a new snapshot
LGTM!
PR
submitted
Store snapshot for
verification for future
changes
Slide 28
Slide 28 text
Verify Process
PR
submitted
Slide 29
Slide 29 text
Verify Process
PR
submitted
Slide 30
Slide 30 text
Verify Process
PR
submitted
Slide 31
Slide 31 text
Verify Process
PR
submitted
Slide 32
Slide 32 text
Verify Process
PR
submitted
● Forgot to Record
● Regression
Slide 33
Slide 33 text
Verify Process
PR
submitted
● Forgot to Record
● Regression
Slide 34
Slide 34 text
Verify Process
PR
submitted
● Forgot to Record
● Regression
Slide 35
Slide 35 text
Configuration Matters!
Slide 36
Slide 36 text
Where to start
Snapshot Testing
Photo by Shalaka Gamage on Unsplash
Slide 37
Slide 37 text
What go with?
1. Screen level snapshot testing
2. Component level snapshot testing?
Slide 38
Slide 38 text
Screen level snapshot testing
1. A lot of dependencies
Slide 39
Slide 39 text
Screen level snapshot testing
1. A lot of dependencies
2. Too many states
Slide 40
Slide 40 text
Screen level snapshot testing
1. A lot of dependencies
2. Too many states
3. More components = More flakiness
Slide 41
Slide 41 text
Screen level snapshot testing
1. A lot of dependencies
2. Too many states
3. More components = More flakiness
4. Hard to trace which changes breaks
Slide 42
Slide 42 text
No content
Slide 43
Slide 43 text
App Bar are likely not to change, not priority to test
Slide 44
Slide 44 text
App Bar are likely not to change, not priority to test
Using Library component, not priority to test
Slide 45
Slide 45 text
● Main feature of the app
● Has different states
Slide 46
Slide 46 text
● Main feature of the app
● Has different states
Important text
Slide 47
Slide 47 text
App Bar are likely not to change, not priority to test
● Main feature of the app
● Has different states
Important text
Using Library component, not priority to test
Slide 48
Slide 48 text
1. Most used features
2. Components that are being iterated quickly
3. Important text bodies and messages
Slide 49
Slide 49 text
Tooling
Photo by Brendan Steeves on Unsplash
Slide 50
Slide 50 text
Tooling
- pedrovgs/Shot
- shopify/android-testify*
- dropbox/dropshots
- takahirom/roborazzi
- QuickBirdEng/kotlin-snapshot-testing
- facebook/screenshot-tests-for-android
Slide 51
Slide 51 text
Roborazzi
A snapshot testing framework built upon
square’s paparazzi to make it work with
Robolectric
Slide 52
Slide 52 text
● Main feature of the app
● Has different states
Slide 53
Slide 53 text
@Composable
fun ProjectCard() {
val project = loadData()
Column {
ProjectIcon(project)
Row {
Text(project.name)
...
}
}
}
Slide 54
Slide 54 text
@Composable
fun ProjectCard() {
val project = loadData()
Column {
ProjectIcon(project)
Row {
Text(project.name)
...
}
}
}
Slide 55
Slide 55 text
Use a stateless composable
@Composable
fun ProjectCard() {
val project = loadData()
ProjectCardContent(project)
}
@Composable
fun ProjectCardContent(
val project: Project
) {
// Render here
}
Snapshot test
UI Test
Slide 56
Slide 56 text
@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ProjectCardScreenshotTest {
@get:Rule
val composeTestRule = createComposeRule()
@Test
fun screenshotTest() {
composeTestRule.setContent {
ProjectCard(...)
}
composeTestRule.onRoot().captureRoboImage()
}
}
Slide 57
Slide 57 text
@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ProjectCardScreenshotTest {
@get:Rule
val composeTestRule = createComposeRule()
@Test
fun screenshotTest() {
composeTestRule.setContent {
ProjectCard(...)
}
composeTestRule.onRoot().captureRoboImage()
}
}
Slide 58
Slide 58 text
@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ProjectCardScreenshotTest {
@get:Rule
val composeTestRule = createComposeRule()
@Test
fun screenshotTest() {
composeTestRule.setContent {
ProjectCard(...)
}
composeTestRule.onRoot().captureRoboImage()
}
}
Slide 59
Slide 59 text
@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ProjectCardScreenshotTest {
@get:Rule
val composeTestRule = createComposeRule()
@Test
fun screenshotTest() {
composeTestRule.setContent {
ProjectCard(...)
}
composeTestRule.onRoot().captureRoboImage()
}
}
Slide 60
Slide 60 text
@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ProjectCardScreenshotTest {
@get:Rule
val composeTestRule = createComposeRule()
@Test
fun screenshotTest() {
composeTestRule.setContent {
ProjectCard(...)
}
composeTestRule.onRoot().captureRoboImage()
}
}
Slide 61
Slide 61 text
./gradlew recordRoborazziDebug
./gradlew testDebugUnitTest -Proborazzi.test.record=true
./gradlew verifyRoborazziDebug
./gradlew testDebugUnitTest -Proborazzi.test.verify=true
Slide 62
Slide 62 text
./gradlew recordRoborazziDebug
./gradlew testDebugUnitTest -Proborazzi.test.record=true
./gradlew verifyRoborazziDebug
./gradlew testDebugUnitTest -Proborazzi.test.verify=true
Slide 63
Slide 63 text
./gradlew recordRoborazziDebug
./gradlew testDebugUnitTest -Proborazzi.test.record=true
./gradlew verifyRoborazziDebug
./gradlew testDebugUnitTest -Proborazzi.test.verify=true
Slide 64
Slide 64 text
./gradlew compareRoborazziDebug
./gradlew testDebugUnitTest -Proborazzi.test.compare=true
Slide 65
Slide 65 text
@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ProjectCardScreenshotTest {
@get:Rule
val composeTestRule = createComposeRule()
@Test
fun screenshotTest() {
composeTestRule.setContent {
ProjectCard(...)
}
composeTestRule.onRoot().captureRoboImage()
}
}
Slide 66
Slide 66 text
@RunWith(ParameterizedRobolectricTestRunner::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ProjectCardScreenshotTest(
private val config: DeviceConfig
) {
companion object {
@JvmStatic
@Parameters
fun testParamsProvider() = listOf(
DeviceConfig("Small_Phone_Font_Small", RobolectricDeviceQualifiers.SmallPhone, 0.5f),
DeviceConfig("Small_Phone_Font_Normal", RobolectricDeviceQualifiers.SmallPhone, 1.0f),
DeviceConfig("Medium_Tablet_Font_Large", RobolectricDeviceQualifiers.MediumTablet, 2.0f),
)
}
@Test
fun screenshotTest() {
RuntimeEnvironment.setFontScale(config.fontScale)
RuntimeEnvironment.setQualifiers(config.qualifier)
...
}
}
Use Parameterized Runner for testing
multiple device configuration
Slide 67
Slide 67 text
@RunWith(ParameterizedRobolectricTestRunner::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ProjectCardScreenshotTest(
private val config: DeviceConfig
) {
companion object {
@JvmStatic
@Parameters
fun testParamsProvider() = listOf(
DeviceConfig("Small_Phone_Font_Small", RobolectricDeviceQualifiers.SmallPhone, 0.5f),
DeviceConfig("Small_Phone_Font_Normal", RobolectricDeviceQualifiers.SmallPhone, 1.0f),
DeviceConfig("Medium_Tablet_Font_Large", RobolectricDeviceQualifiers.MediumTablet, 2.0f),
)
}
@Test
fun screenshotTest() {
RuntimeEnvironment.setFontScale(config.fontScale)
RuntimeEnvironment.setQualifiers(config.qualifier)
...
}
}
Use Parameterized Runner for testing
multiple device configuration
Slide 68
Slide 68 text
@RunWith(ParameterizedRobolectricTestRunner::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ProjectCardScreenshotTest(
private val config: DeviceConfig
) {
companion object {
@JvmStatic
@Parameters
fun testParamsProvider() = listOf(
DeviceConfig("Small_Phone_Font_Small", RobolectricDeviceQualifiers.SmallPhone, 0.5f),
DeviceConfig("Small_Phone_Font_Normal", RobolectricDeviceQualifiers.SmallPhone, 1.0f),
DeviceConfig("Medium_Tablet_Font_Large", RobolectricDeviceQualifiers.MediumTablet, 2.0f),
)
}
@Test
fun screenshotTest() {
RuntimeEnvironment.setFontScale(config.fontScale)
RuntimeEnvironment.setQualifiers(config.qualifier)
...
}
}
Use Parameterized Runner for testing
multiple device configuration
Slide 69
Slide 69 text
@RunWith(ParameterizedRobolectricTestRunner::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ProjectCardScreenshotTest(
private val config: DeviceConfig
) {
companion object {
@JvmStatic
@Parameters
fun testParamsProvider() = listOf(
DeviceConfig("Small_Phone_Font_Small", RobolectricDeviceQualifiers.SmallPhone, 0.5f),
DeviceConfig("Small_Phone_Font_Normal", RobolectricDeviceQualifiers.SmallPhone, 1.0f),
DeviceConfig("Medium_Tablet_Font_Large", RobolectricDeviceQualifiers.MediumTablet, 2.0f),
)
}
@Test
fun screenshotTest() {
RuntimeEnvironment.setFontScale(config.fontScale)
RuntimeEnvironment.setQualifiers(config.qualifier)
...
}
}
Use Parameterized Runner for testing
multiple device configuration
Slide 70
Slide 70 text
Small Phone with Font Scale 0.5
Small Phone with Font Scale 1.0
Medium Tablet with Font Scale 2.0
Slide 71
Slide 71 text
Road to Snapshot Testing
1. Analyze which are to test, start with small & critical components
2. Scale up to different configurations
3. Integrate into your development workflow
4. Happy Snapshot Testing!
Slide 72
Slide 72 text
Good read
- Introduction to snapshot testing blog series
- Android screenshot testing playground
- AndroidUITesting Utils
- NowinAndroid
bit.ly/droid-snap-read
Slide 73
Slide 73 text
A to Z of snapshot
testing in Android
aungkyawpaing.dev
https://github.com/vincent-paing/