Robolectric Native Graphics and Roborazzi

Transcript

1. Robolectric Native Graphics and Roborazzi takahirom
2. 1. Why now? 2. What exactly is Robolectric? 3. What is Robolectric Native Graphics? 4. Roborazzi a. What is Roborazzi? b. How to use Roborazzi c. How to Leverage Roborazzi Effectively i. Component-based screenshot testing ii. Screen-based screenshot testing iii. UI testing debugging 5. What to test 6. Practices
3. Why now? Tokyo
4. Number of detectable bugs Maintenance cost
5. Number of detectable bugs Maintenance cost This maintenance cost includes: • Performance issues (such as slowness) • False positives (false alerts, flakiness) • Time consumed by human intervention • The difficulty of running the test
6. Number of detectable bugs Maintenance cost A higher maintenance cost means fewer tests can be maintained.
7. Number of detectable bugs Good value for cost Maintenance cost
8. Number of detectable bugs Good value for cost Maintenance cost Manual Production test
9. Number of detectable bugs End to End test (E2E) Good value for cost Maintenance cost Manual Production test
10. Number of detectable bugs Manual Production test End to End test (E2E) Duration: A few hours to a few days 💰 Is there a way to make this more cost-effective? Good value for cost Maintenance cost
11. Number of detectable bugs End to End test (E2E) The new feature Robolectric Native Graphics feature may reduce the need for these tests ???? Good value for cost Maintenance cost Manual Production test
12. What exactly is Robolectric? Tokyo
13. What is Robolectric? > Running tests on an Android emulator or device is slow! Building, deploying, and launching the app often takes a minute or more. That’s no way to do TDD. There must be a better way. > Robolectric is a framework that brings fast and reliable unit tests to Android. Tests run inside the JVM on your workstation in seconds. https://robolectric.org/ A Test Framework for Android Apps. You can use Android classes in local test. Faster and More Reliable tests than Emulators or Physical Devices tests.
14. How fast is Robolectric? Emulator Robolectric Robolectric takes a few seconds to start up, but once it does, it is fast. Code: https://github.com/takahirom/roborazzi-usage-examples/pull/5 Robolectric: 4.11.1 Emulator: Pixel 6 API 32 CPU: Apple M1 Max
15. How to use Robolectric Tokyo
16. How to use Robolectric: Setup app/build.gradle.kts android { testOptions { unitTests { isIncludeAndroidResources = true } } Otherwise, you'll get an 'Unable to resolve activity for Intent' error. All changes from the new project template https://github.com/takahirom/roborazzi-usage-examples/ compare/b697...5c12
17. How to use Robolectric: Setup Add dependencies app/build.gradle.kts dependencies { ... testImplementation(libs.robolectric) testImplementation(libs.espresso.core) testImplementation(libs.ui.test.junit4) } All changes from the new project template https://github.com/takahirom/roborazzi-usage-examples/ compare/b697...5c12 gradle/libs.versions.toml robolectric = { group = "org.robolectric", name = "robolectric", version.ref = "robolectric" } espresso-core = { group = "androidx.test.espresso", name = "espresso-core", version.ref = "espresso-core" } ui-test-junit4 = { group = "androidx.compose.ui", name = "ui-test-junit4" } ... Robolectric is compatible with the Espresso testing library
18. How to use Robolectric: Setup Add dependencies app/build.gradle.kts dependencies { ... testImplementation(libs.robolectric) testImplementation(libs.espresso.core) testImplementation(libs.ui.test.junit4) } All changes from the new project template https://github.com/takahirom/roborazzi-usage-examples/ compare/b697...5c12 gradle/libs.versions.toml robolectric = { group = "org.robolectric", name = "robolectric", version.ref = "robolectric" } espresso-core = { group = "androidx.test.espresso", name = "espresso-core", version.ref = "espresso-core" } ui-test-junit4 = { group = "androidx.compose.ui", name = "ui-test-junit4" } ... Robolectric is compatible with the Espresso testing library Robolectric is compatible with the Compose testing library.
19. How to use Robolectric: Test for Compose @RunWith(RobolectricTestRunner::class) class FirstRobolectricComposeTest { @get:Rule val composeRule = createComposeRule() @Test fun test() { composeRule.setContent {
20. How to use Robolectric: Test for Compose @RunWith(RobolectricTestRunner::class) class FirstRobolectricComposeTest { @get:Rule val composeRule = createComposeRule() @Test fun test() { composeRule.setContent { Use RobolectricTestRunner to Replace Android Classes with Robolectric
21. @get:Rule val composeRule = createComposeRule() @Test fun test() { composeRule.setContent { Greeting(name = "Robolectric") } composeRule .onNode(hasText("Hello Robolectric!")) .assertExists() Setup compose test with createComposeRule()
22. @Test fun test() { composeRule.setContent { Greeting(name = "Robolectric") } composeRule .onNode(hasText("Hello Robolectric!")) .assertExists() } } Call a Composable function to test Assert what you want to check
23. What is Robolectric Native Graphics? Tokyo
24. What is Robolectric Native Graphics? > Robolectric 4.10 adds support for native Android graphics. It is currently disabled by default and can be enabled using @GraphicsMode(NATIVE). > When native graphics is enabled, interactions with Android graphics classes use real native Android graphics code and are much higher fidelity. https://github.com/robolectric/robole ctric/releases/tag/robolectric-4.10 Robolectric Native Graphics is a feature of Robolectric. You can use real native Android graphics code and are much higher fidelity
25. Try Robolectric Native Graphics @GraphicsMode(NATIVE) @Test fun test() { val bitmap = activity.findViewById<View>(R.id.button_first) .drawToBitmap() println(bitmap) }
26. Verifying the Use of Actual Native Android Graphics Code Android's native implementation uses Skia. Therefore, I checked whether Robolectric Native Graphics also uses Skia, utilizing lldb for this purpose. https://twitter.com/new_runnable/st atus/1718191787163172986 Robolectric Native Graphics Utilize Real Native Android Graphics Classes.
27. What is Roborazzi? Tokyo
28. What is Roborazzi? Roborazzi is an Android Library that allows you to execute screenshot tests on your local machine without the need for real devices or emulators. It leverages Robolectric Native Graphics for rendering views. I, Takahirom, am the author of this library. It is utilized in Google's 'Now in Android' app and by companies such as Uber. https://twitter.com/ppvi/status/1694741201457418450 https://twitter.com/tsmith/status/1695095257044852823
29. What is Roborazzi? Roborazzi is an Android Library that allows you to execute screenshot tests on your local machine without the need for real devices or emulators. It leverages Robolectric Native Graphics for rendering views. I, Takahirom, am the author of this library. It is utilized in Google's 'Now in Android' app and by companies such as Uber. https://twitter.com/ppvi/status/1694741201457418450 https://twitter.com/tsmith/status/1695095257044852823 Screenshot library using Robolectric Native Graphics
30. What is Roborazzi? Robolectric Robolectric is a test Framework
31. What is Roborazzi? Robolectric Robolectric Native Graphics New feature Robolectric Native Graphics
32. What is Roborazzi? Robolectric Robolectric Native Graphics Roborazzi The new Roborazzi library uses Robolectric Native Graphics for screenshot testing
33. Roborazzi vs. Other Tools like Paparazzi: A Comparative Overview Paparazzi: Paparazzi is an Android library that enables you to take screenshots in a JVM environment, differing from Roborazzi which relies on Robolectric Native Graphics. It is built on Android Studio's Layout Lib
34. Roborazzi vs. Other Tools like Paparazzi: A Comparative Overview Roborazzi Android Studio Preview based tool Rendering Android's native graphics classes Android's native graphics classes Pros Robolectric support • Dagger Hilt support • Espresso support • User interaction (click,scroll) support • Animation, SideEffect support Shadow rendering support Cons Lack of Windows support You can check Sergio Sastre Flórez’s slide: https://speakerdeck.com/gio_sastre/screenshot-testing-for-android-which-library-is-right-for-you
35. Roborazzi vs. Other Tools like Paparazzi: A Comparative Overview Roborazzi Android Studio Preview based tool Rendering Android's native graphics classes Android's native graphics classes Pros Robolectric support • Dagger Hilt support • Espresso support • User interaction (click,scroll) support • Animation, SideEffect support Shadow rendering support Cons Lack of Windows support You can check Sergio Sastre Flórez’s slide: https://speakerdeck.com/gio_sastre/screenshot-testing-for-android-which-library-is-right-for-you Both use the same graphics classes
36. Roborazzi vs. Other Tools like Paparazzi: A Comparative Overview Roborazzi Android Studio Preview based tool Rendering Android's native graphics classes Android's native graphics classes Pros Robolectric support • Dagger Hilt support • Espresso support • User interaction (click,scroll) support • Animation, SideEffect support Shadow rendering support Cons Lack of Windows support You can check Sergio Sastre Flórez’s slide: https://speakerdeck.com/gio_sastre/screenshot-testing-for-android-which-library-is-right-for-you Roborazzi can use Robolectric’s testing features
37. Roborazzi vs. Other Tools like Paparazzi: A Comparative Overview Roborazzi Android Studio Preview based tool Rendering Android's native graphics classes Android's native graphics classes Pros Robolectric support • Dagger Hilt support • Espresso support • User interaction (click,scroll) support • Animation, SideEffect support Shadow rendering support Cons Lack of Windows support You can check Sergio Sastre Flórez’s slide: https://speakerdeck.com/gio_sastre/screenshot-testing-for-android-which-library-is-right-for-you Currently, Robolectric Native Graphics doesn’t support Windows You can run Roborazzi on CI
38. How to use Roborazzi Tokyo
39. How to use Roborazzi: Setup All changes: https://github.com/takahirom/roborazzi-usage-examples/ commit/3a02 Just add a Gradle Plugin and dependencies
40. How to use Roborazzi @GraphicsMode(GraphicsMode.Mode.NATIVE) @RunWith(RobolectricTestRunner::class) class FirstRobolectricComposeTest { ... @Test fun roborazziTest() { composeRule.setContent { Greeting(name = "Roborazzi") All changes: https://github.com/takahirom/ roborazzi-usage-examples/ commit/3a02
41. How to use Roborazzi @GraphicsMode(GraphicsMode.Mode.NATIVE) @RunWith(RobolectricTestRunner::class) class FirstRobolectricComposeTest { ... @Test fun roborazziTest() { composeRule.setContent { Greeting(name = "Roborazzi") All changes: https://github.com/takahirom/ roborazzi-usage-examples/ commit/3a02 This test is nearly identical to the one used in Robolectric.
42. How to use Roborazzi @GraphicsMode(GraphicsMode.Mode.NATIVE) @RunWith(RobolectricTestRunner::class) class FirstRobolectricComposeTest { ... @Test fun roborazziTest() { composeRule.setContent { Greeting(name = "Roborazzi") All changes: https://github.com/takahirom/ roborazzi-usage-examples/ commit/3a02 Enable Robolectric Native Graphics to take screenshot
43. composeRule .onNode(hasText("Hello Robolectric!")) .captureRoboImage() composeRule .onRoot() .captureRoboImage() } All changes: https://github.com/takahirom/ roborazzi-usage-examples/ commit/3a02 Just call .captureRoboImage() method in your test code.
44. composeRule .onNode(hasText("Hello Robolectric!")) .captureRoboImage() composeRule .onRoot() .captureRoboImage() } All changes: https://github.com/takahirom/ roborazzi-usage-examples/ commit/3a02 You can specify the compose node of screenshot Or you can take the root node screenshot.
45. You can take screenshots of anything you want in your app’s test https://github.com/takahirom/roborazzi
46. How to use Roborazzi build/output/roborazzi/test.png From JetNews Record the reference image ./gradlew recordRoborazziDebug Or You can use gradle.properties options roborazzi.test.record=true and ./gradlew testDebugUnitTest
47. How to use Roborazzi build/output/roborazzi/test_compare.png From JetNews Compare the new image with the reference image ./gradlew compareRoborazziDebug Or You can use gradle.properties roborazzi.test.compare=true and ./gradlew testDebugUnitTest
48. How to use Roborazzi Ensure that the new image remains unchanged compared to the reference image. ./gradlew verifyRoborazziDebug Or You can use gradle.properties roborazzi.test.verify=true and ./gradlew testDebugUnitTest Roborazzi throws an AssertionError if the image changes.
49. How to Leverage Roborazzi Effectively Tokyo
50. How to Leverage Roborazzi Effectively Roborazzi excels in three types of tests: 1) Component-based screenshot testing 2) Screen-based screenshot testing 3) UI test debugging
51. Component-based screenshot testing with Roborazzi • What: Screenshot tests for component design like a Button component • Why: Ensuring Consistency, Detecting Visual Regression, Cross-Setting and Cross-Form-Factor Consistency, Documentation, Enhancing User Experience • How: You can utilize Jetpack Compose Preview for component-based screenshot testing. There are two methods: ・Utilizing Showkase, which employs Kotlin Symbol Processing to gather Preview functions. ・Using Reflection, with libraries such as ClassGraph Full changes https://github.com/takahirom/ro borazzi-usage-examples/pull/1
52. Component-based screenshot testing with Roborazzi Full changes https://github.com/takahirom/roborazzi-usage- examples/pull/1 @GraphicsMode(GraphicsMode.Mode.NATIVE) @RunWith(ParameterizedRobolectricTestRunner::class) class ComponentScreenshotTest( private val composablePreview: ComposablePreviewFunction, ) { @get:Rule val composeRule = createComposeRule() @Test fun test() { composeRule.setContent { composablePreview() } composeRule.onRoot().captureRoboImage( roboOutputName() + "_" + composablePreview.toString() + ".png" ) } ... Automated Collection of Composable Functions via Compose Preview — No More Writing Individual Tests
53. Component-based screenshot testing with Roborazzi https://github.com/takahirom/roborazzi Recommended article: Efficient Testing with Robolectric & Roborazzi Across Many UI States, Devices and Configurations You can use predefined device settings and configurations, such as dark mode.
54. Screen-based screenshot testing with Roborazzi What: Screenshot tests for screen How: Robolectric can launch your Activity using either Compose test's createAndroidComposeRule<ArticleActivity>() or Espresso's ActivityScenario.launch() Why: ? Full changes https://github.com/takahirom/roborazzi-usage- examples/pull/4
55. Why do we do Screen-based Screenshot Testing Why: Identify and solve the problems faced by users, Ensuring Comprehensive Coverage, Detecting Layout and Integration Issues, Enhancing Test Fidelity Production
56. Why do we do Screen-based Screenshot Testing Why: Identify and solve the problems faced by users, Ensuring Comprehensive Coverage, Detecting Layout and Integration Issues, Enhancing Test Fidelity Production Test
57. Obstacles in Implementing Screen-Based Tests • The app will have dependencies that are difficult to test, such as network requests to the server. This makes the tests flaky because they can't be controlled. • Additionally, there will be threads in the app that aren't controlled during testing, which also contributes to the tests' flakiness. Full changes https://github.com/takahir om/roborazzi-usage-exa mples/pull/4 Dependencies (Server APIs, Threads)
58. Obstacles in Implementing Screen-Based Tests • The app will have dependencies that are difficult to test, such as network requests to the server. This makes the tests flaky because they can't be controlled. • Additionally, there will be threads in the app that aren't controlled during testing, which also contributes to the tests' flakiness. Full changes https://github.com/takahirom/roborazzi-usage- examples/pull/4 We need to replace dependencies in tests.
59. How to replace dependencies in tests We can use Dagger Hilt to replace dependencies in tests. Full changes to introduce Dagger Hilt https://github.com/takahirom/roborazzi-usage- examples/pull/2 Fake Dependencies production test Dependencies (Server APIs, Threads)
60. How to replace dependencies in tests Full changes https://github.com/takahirom/roborazzi-usage- examples/pull/4 @Module @TestInstallIn( components = [SingletonComponent::class], replaces = [ArticleApiModule::class] ) object FakeArticleApiModule { @Provides fun provideArticleApi(): ArticleApi { return FakeArticleApi() }
61. How to replace dependencies in tests Full changes https://github.com/takahirom/roborazzi-usage- examples/pull/4 @Module @TestInstallIn( components = [SingletonComponent::class], replaces = [ArticleApiModule::class] ) object FakeArticleApiModule { @Provides fun provideArticleApi(): ArticleApi { return FakeArticleApi() } You can replace dependencies in your test
62. Screen-based screenshot testing Full changes https://github.com/takahirom/roborazzi- usage-examples/pull/4 @HiltAndroidTest @RunWith(RobolectricTestRunner::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ArticleScreenTest { private val composeTestRule = createAndroidComposeRule<ArticleActivity>() @get:Rule val ruleChain = RuleChain .outerRule(HiltAndroidRule(this)) .around(composeTestRule) @Test fun checkScreenShot() { composeTestRule .onRoot() .captureRoboImage() }
63. Screen-based screenshot testing Full changes https://github.com/takahirom/roborazzi- usage-examples/pull/4 @HiltAndroidTest @RunWith(RobolectricTestRunner::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ArticleScreenTest { private val composeTestRule = createAndroidComposeRule<ArticleActivity>() @get:Rule val ruleChain = RuleChain .outerRule(HiltAndroidRule(this)) .around(composeTestRule) @Test fun checkScreenShot() { composeTestRule .onRoot() .captureRoboImage() } Add @HiltAndroidTest and HiltAndroidRule for Hilt
64. Full changes https://github.com/takahirom/roborazzi- usage-examples/pull/4 @RunWith(RobolectricTestRunner::class) @GraphicsMode(GraphicsMode.Mode.NATIVE) class ArticleScreenTest { private val composeTestRule = createAndroidComposeRule<ArticleActivity>() @get:Rule val ruleChain = RuleChain .outerRule(HiltAndroidRule(this)) .around(composeTestRule) @Test fun checkScreenShot() { composeTestRule .onRoot() .captureRoboImage() } } You can use captureRoboImage() extension function to take screenshot tests
65. UI test debugging What: Roborazzi can be used as a supplemental tool for debugging UI tests, such as verifying button visibility. Why: Writing tests becomes challenging when unable to visually assess the layout. How: It serves as an auxiliary tool used in conjunction with standard UI testing methods to enhance debugging capabilities.
66. UI test debugging composeTestRule .onNode(hasText("article title 1")) .assertExists() composeTestRule .onRoot() .captureRoboImage( roborazziOptions = RoborazziOptions( taskType = roborazziSystemPropertyTaskType() .convertVerifyingToComparing() ) )
67. UI test debugging composeTestRule .onNode(hasText("article title 1")) .assertExists() composeTestRule .onRoot() .captureRoboImage( roborazziOptions = RoborazziOptions( taskType = roborazziSystemPropertyTaskType() .convertVerifyingToComparing() ) ) Coming Soon: You can just take a screenshots during Roborazzi verify task Tell me your opinion! https://github.com/takahirom/ roborazzi/issues/215
68. What to test Tokyo
69. Number of detectable bugs Maintenance cost
70. Number of detectable bugs Good value for cost Maintenance cost
71. Number of detectable bugs End to End test (E2E) Good value for cost Maintenance cost Manual Production test
72. Number of detectable bugs Component-based Screenshot tests End to End test (E2E) You can automatically collect Preview functions and check the appearance of the components. Good value for cost Maintenance cost Manual Production test
73. Number of detectable bugs Component-based Screenshot tests End to End test (E2E) Robolectric UI tests Robolectric is faster and more reliable, but it is not the actual Android Framework Good value for cost Maintenance cost Manual Production test
74. Component-based Screenshot tests End to End test (E2E) Screenshot tests can capture what users see; however, they require checking after every change, which increases maintenance costs Robolectric Screen-based Screenshot tests Good value for cost Robolectric UI tests Number of detectable bugs Maintenance cost Manual Production test
75. Component-based Screenshot tests End to End test (E2E) Good value for cost Roborazzi Robolectric Screen-based Screenshot tests Robolectric UI tests Number of detectable bugs Maintenance cost Manual Production test
76. My recommendations for Testing fewer tests fewer tests
77. E2E My recommendations for Testing fewer tests fewer tests
78. E2E Robolectric Screen-based Screenshot tests My recommendations for Testing fewer tests fewer tests
79. E2E Robolectric Screen-based Screenshot tests My recommendations for Testing fewer tests fewer tests Maintenance costs may limit how many screen-based screenshot tests you can run, but starting them is easy.
80. E2E Robolectric Screen-based Screenshot tests Robolectric UI tests My recommendations for Testing fewer tests fewer tests
81. E2E Robolectric Screen-based Screenshot tests Robolectric UI tests My recommendations for Testing fewer tests fewer tests Screen-based screenshot tests and UI tests have almost identical code, so they can be migrated.
82. E2E Robolectric Screen-based Screenshot tests Robolectric UI tests Component-based Screenshot tests My recommendations for Testing fewer tests fewer tests
83. E2E Robolectric Screen-based Screenshot tests Robolectric UI tests Component-based Screenshot tests Unit tests for essential domain classes not covered by higher-level tests My recommendations for Testing fewer tests fewer tests
84. Practices Tokyo
85. Practices 1. Development Process a. Adopt a test-driven approach, beginning with screenshot tests of a blank screen. b. Set Roborazzi to be enabled by default. 2. Write Tests Effectively a. Employ Robot testing pattern or page object pattern for testing. For more information please read this article https://medium.com/@takahirom/roborazzi-elevating-android-visual-te sting-to-the-next-level-46ec56b24055 3. Debug Effectively a. Utilize dump mode for debugging. b. Implement code coverage analysis in debugging. 4. CI Effectively c. Define image storage locations in CI pipeline.
86. Summary 1. Robolectric: Streamlines Android unit tests within the JVM. 2. Roborazzi: Enables screenshot testing without real devices. 3. Benefits: Lowers maintenance costs and improves bug detection. 4. Usage: Integrates with Espresso and Dagger Hilt. 5. Strategy: Combine unit, component, and screen-based tests for optimal coverage
87. Thank you Tokyo