How to write and automate performance tests in Android

Performance testing has always been neglected by developers. We write unit tests and integration tests but for some reason we never test the performance of our apps and when we do is already too late because our users are complaining about it. It has not been historically easy to write these tests and we usually rely on manual testing, which is not really accurate, or complicated tools that we don’t even know how to use or we don’t understand. But, is there an easy way to write and automate performance tests? Yes there is, and I’m going to show you how you can do it.

In this talk you will learn how you can write and automate performance tests for your Android apps. We will explore different ways of writing performance tests and we will see how we can take advantage of a few not well-known APIs that the Android platform offer us to help us in our journey towards the perfect automation testing suite.

Transcript

1. How to write and automate performance tests @Orbycius Marcos Holgado
2. @orbycius Slow & clunky Some user “ ”
3. @orbycius Slow & clunky Some user “ ” “ ” Laggs massively Some other user
4. @orbycius Slow & clunky Some user “ ” “ ” Laggs massively Some other user “ ” So laggy it makes me sad Another user
5. It doesn’t matter how much value your app gives to users if they can’t use it @orbycius
6. Testing is hard @orbycius
7. We didn't write tests because… {any excuse here} @orbycius
8. We didn't write tests because… lack of time @orbycius
9. We didn't write tests because… testing is hard @orbycius
10. What’s performance testing? @orbycius
11. frames @orbycius
12. 60 frames per second @orbycius
13. ≈ 16ms 1000ms 60frames @orbycius
14. What else? @orbycius
15. What else? Battery usage @orbycius
16. What else? Battery usage Memory usage @orbycius
17. Pre-requisite @orbycius
18. Pre-requisite SDK 23/24 @orbycius
19. Pre-requisite SDK 23/24 Don’t use emulators @orbycius
20. @orbycius https://github.com/marcosholgado/performance-test https://github.com/marcosholgado/dagger-playground
21. @orbycius Startup time
22. @RunWith(AndroidJUnit4::class) @LargeTest class FirstTest { @get:Rule val activityRule = ActivityTestRule(MainActivity::class.java) @Before fun setUp() { // do something } @After fun after() { // do something } @Test fun testFirst() { // do something } } @orbycius @RunWith(AndroidJUnit4::class) @LargeTest class FirstTest { @get:Rule val activityRule = ActivityTestRule(MainActivity::class.java) @Before fun setUp() { // do something } @After fun after() { // do something } @Test fun testFirst() { // do something } }
23. @RunWith(AndroidJUnit4::class) @LargeTest class FirstTest { @get:Rule val activityRule = ActivityTestRule(MainActivity::class.java) @Before fun setUp() { // do something } @After fun after() { // do something } @Test fun testFirst() { // do something } } @RunWith(AndroidJUnit4::class) @LargeTest class FirstTest { @get:Rule val activityRule = ActivityTestRule(MainActivity::class.java) @Before fun setUp() { // do something } @After fun after() { // do something } @Test fun testFirst() { // do something } } @orbycius
24. @RunWith(AndroidJUnit4::class) @LargeTest class FirstTest { @get:Rule val activityRule = ActivityTestRule(MainActivity::class.java) @Before fun setUp() { // do something } @After fun after() { // do something } @Test fun testFirst() { // do something } } @RunWith(AndroidJUnit4::class) @LargeTest class FirstTest { @get:Rule val activityRule = ActivityTestRule(MainActivity::class.java) @Before fun setUp() { // do something } @After fun after() { // do something } @Test fun testFirst() { // do something } } @orbycius
25. @RunWith(AndroidJUnit4::class) @LargeTest class FirstTest { @get:Rule val activityRule = ActivityTestRule(MainActivity::class.java) @Before fun setUp() { // do something } @After fun after() { // do something } @Test fun testFirst() { // do something } } @orbycius
26. @orbycius @get:Rule val activityRule = ActivityTestRule(MainActivity::class.java)
27. @orbycius class MyActivityTestRule<T: Activity>(activityClass: Class<T>): ActivityTestRule<T>(activityClass) { }
28. class MyActivityTestRule<T: Activity>(activityClass: Class<T>): ActivityTestRule<T>(activityClass) { var timestamp: Long = 0 @orbycius }
29. class MyActivityTestRule<T: Activity>(activityClass: Class<T>): ActivityTestRule<T>(activityClass) { var timestamp: Long = 0 @orbycius } class MyActivityTestRule<T: Activity>(activityClass: Class<T>): ActivityTestRule<T>(activityClass) { var timestamp: Long = 0 override fun beforeActivityLaunched() { timestamp = System.currentTimeMillis() super.beforeActivityLaunched() }
30. class MyActivityTestRule<T: Activity>(activityClass: Class<T>): ActivityTestRule<T>(activityClass) { var timestamp: Long = 0 @orbycius } class MyActivityTestRule<T: Activity>(activityClass: Class<T>): ActivityTestRule<T>(activityClass) { var timestamp: Long = 0 override fun beforeActivityLaunched() { timestamp = System.currentTimeMillis() super.beforeActivityLaunched() } class MyActivityTestRule<T: Activity>(activityClass: Class<T>): ActivityTestRule<T>(activityClass) { var timestamp: Long = 0 override fun beforeActivityLaunched() { timestamp = System.currentTimeMillis() super.beforeActivityLaunched() } override fun afterActivityLaunched() { super.afterActivityLaunched() timestamp = System.currentTimeMillis() - timestamp }
31. @orbycius @RunWith(AndroidJUnit4::class) class FirstTest { @get:Rule val activityRule = MyActivityTestRule(MainActivity::class.java) @Test fun startUpTest() { assertWithMessage("Start up time: ${activityRule.timestamp}") .that(activityRule.timestamp) .isLessThan(300) } }
32. @orbycius FrameMetrics
33. @orbycius FrameMetrics Animation Duration Command Issue Duration Draw Duration First Draw Frame Input Handling Duration Intended Vsync Timestamp Layout Measure Duration Swap Buffers Duration Sync Duration Total Duration Unknown Delay Duration Vsync Timestamp
34. @orbycius class MainAdapter : RecyclerView.Adapter<MainViewHolder>() { override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): MainViewHolder { val view = LayoutInflater .from(parent.context) .inflate(R.layout.item, parent, false) return MainViewHolder(view) } override fun onBindViewHolder(holder: MainViewHolder, pos: Int) { (holder.itemView as TextView).text = "This is item $pos" // Add sleep to recreate work on the main thread Thread.sleep(150) } override fun getItemCount(): Int = 200 }
35. class MainAdapter : RecyclerView.Adapter<MainViewHolder>() { override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): MainViewHolder { val view = LayoutInflater .from(parent.context) .inflate(R.layout.item, parent, false) return MainViewHolder(view) } override fun onBindViewHolder(holder: MainViewHolder, pos: Int) { (holder.itemView as TextView).text = "This is item $pos" // Add sleep to recreate work on the main thread Thread.sleep(150) } override fun getItemCount(): Int = 200 } @orbycius
36. @orbycius
37. @orbycius var percJankyFrames = 0f
38. @orbycius @Test fun testFirst() { for (i in 0..30) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition<RecyclerView.ViewHolder>(i)) } for (i in 30.downTo(1)) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition<RecyclerView.ViewHolder>(i)) } assertWithMessage("Janky frames over $PERCENTAGE% value was $percJankyFrames%") .that(percJankyFrames) .isLessThan(PERCENTAGE) } var percJankyFrames = 0f
39. @Test fun testFirst() { @orbycius for (i in 0..30) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition<RecyclerView.ViewHolder>(i)) val handler = Handler(Looper.getMainLooper()) activityRule.activity.window .addOnFrameMetricsAvailableListener(object : Window.OnFrameMetricsAvailableListener { private var totalFrames = 0 private var jankyFrames = 0 override fun onFrameMetricsAvailable( window: Window, frameMetrics: FrameMetrics, drop: Int ) { totalFrames++ val duration = (0.000001 * frameMetrics.getMetric(FrameMetrics.TOTAL_DURATION)).toFloat() if (duration > 16f) { jankyFrames++ percJankyFrames = jankyFrames.toFloat() / totalFrames * 100 } } }, handler) var percJankyFrames = 0f
40. val handler = Handler(Looper.getMainLooper()) activityRule.activity.window .addOnFrameMetricsAvailableListener(object : Window.OnFrameMetricsAvailableListener { private var totalFrames = 0 private var jankyFrames = 0 override fun onFrameMetricsAvailable( window: Window, frameMetrics: FrameMetrics, drop: Int ) { totalFrames++ val duration = (0.000001 * frameMetrics.getMetric(FrameMetrics.TOTAL_DURATION)).toFloat() if (duration > 16f) { jankyFrames++ percJankyFrames = jankyFrames.toFloat() / totalFrames * 100 } } }, handler) val handler = Handler(Looper.getMainLooper()) activityRule.activity.window .addOnFrameMetricsAvailableListener(object : Window.OnFrameMetricsAvailableListener { private var totalFrames = 0 private var jankyFrames = 0 override fun onFrameMetricsAvailable( window: Window, frameMetrics: FrameMetrics, drop: Int ) { totalFrames++ val duration = (0.000001 * frameMetrics.getMetric(FrameMetrics.TOTAL_DURATION)).toFloat() if (duration > 16f) { jankyFrames++ percJankyFrames = jankyFrames.toFloat() / totalFrames * 100 } } }, handler) @orbycius var percJankyFrames = 0f @Test fun testFirst() { for (i in 0..30) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition<RecyclerView.ViewHolder>(i))
41. val handler = Handler(Looper.getMainLooper()) activityRule.activity.window .addOnFrameMetricsAvailableListener(object : Window.OnFrameMetricsAvailableListener { private var totalFrames = 0 private var jankyFrames = 0 override fun onFrameMetricsAvailable( window: Window, frameMetrics: FrameMetrics, drop: Int ) { totalFrames++ val duration = (0.000001 * frameMetrics.getMetric(FrameMetrics.TOTAL_DURATION)).toFloat() if (duration > 16f) { jankyFrames++ percJankyFrames = jankyFrames.toFloat() / totalFrames * 100 } } }, handler) val handler = Handler(Looper.getMainLooper()) activityRule.activity.window .addOnFrameMetricsAvailableListener(object : Window.OnFrameMetricsAvailableListener { private var totalFrames = 0 private var jankyFrames = 0 override fun onFrameMetricsAvailable( window: Window, frameMetrics: FrameMetrics, drop: Int ) { totalFrames++ val duration = (0.000001 * frameMetrics.getMetric(FrameMetrics.TOTAL_DURATION)).toFloat() if (duration > 16f) { jankyFrames++ percJankyFrames = jankyFrames.toFloat() / totalFrames * 100 } } }, handler) @orbycius var percJankyFrames = 0f @Test fun testFirst() { for (i in 0..30) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition<RecyclerView.ViewHolder>(i))
42. val handler = Handler(Looper.getMainLooper()) activityRule.activity.window .addOnFrameMetricsAvailableListener(object : Window.OnFrameMetricsAvailableListener { private var totalFrames = 0 private var jankyFrames = 0 override fun onFrameMetricsAvailable( window: Window, frameMetrics: FrameMetrics, drop: Int ) { totalFrames++ val duration = (0.000001 * frameMetrics.getMetric(FrameMetrics.TOTAL_DURATION)).toFloat() if (duration > 16f) { jankyFrames++ percJankyFrames = jankyFrames.toFloat() / totalFrames * 100 } } }, handler) val handler = Handler(Looper.getMainLooper()) activityRule.activity.window .addOnFrameMetricsAvailableListener(object : Window.OnFrameMetricsAvailableListener { private var totalFrames = 0 private var jankyFrames = 0 override fun onFrameMetricsAvailable( window: Window, frameMetrics: FrameMetrics, drop: Int ) { totalFrames++ val duration = (0.000001 * frameMetrics.getMetric(FrameMetrics.TOTAL_DURATION)).toFloat() if (duration > 16f) { jankyFrames++ percJankyFrames = jankyFrames.toFloat() / totalFrames * 100 } } }, handler) @orbycius var percJankyFrames = 0f @Test fun testFirst() { for (i in 0..30) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition<RecyclerView.ViewHolder>(i))
43. val handler = Handler(Looper.getMainLooper()) activityRule.activity.window .addOnFrameMetricsAvailableListener(object : Window.OnFrameMetricsAvailableListener { private var totalFrames = 0 private var jankyFrames = 0 override fun onFrameMetricsAvailable( window: Window, frameMetrics: FrameMetrics, drop: Int ) { totalFrames++ val duration = (0.000001 * frameMetrics.getMetric(FrameMetrics.TOTAL_DURATION)).toFloat() if (duration > 16f) { jankyFrames++ percJankyFrames = jankyFrames.toFloat() / totalFrames * 100 } } }, handler) val handler = Handler(Looper.getMainLooper()) activityRule.activity.window .addOnFrameMetricsAvailableListener(object : Window.OnFrameMetricsAvailableListener { private var totalFrames = 0 private var jankyFrames = 0 override fun onFrameMetricsAvailable( window: Window, frameMetrics: FrameMetrics, drop: Int ) { totalFrames++ val duration = (0.000001 * frameMetrics.getMetric(FrameMetrics.TOTAL_DURATION)).toFloat() if (duration > 16f) { jankyFrames++ percJankyFrames = jankyFrames.toFloat() / totalFrames * 100 } } }, handler) @orbycius var percJankyFrames = 0f @Test fun testFirst() { for (i in 0..30) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition<RecyclerView.ViewHolder>(i))
44. val handler = Handler(Looper.getMainLooper()) activityRule.activity.window .addOnFrameMetricsAvailableListener(object : Window.OnFrameMetricsAvailableListener { private var totalFrames = 0 private var jankyFrames = 0 override fun onFrameMetricsAvailable( window: Window, frameMetrics: FrameMetrics, drop: Int ) { totalFrames++ val duration = (0.000001 * frameMetrics.getMetric(FrameMetrics.TOTAL_DURATION)).toFloat() if (duration > 16f) { jankyFrames++ percJankyFrames = jankyFrames.toFloat() / totalFrames * 100 } } }, handler) val handler = Handler(Looper.getMainLooper()) activityRule.activity.window .addOnFrameMetricsAvailableListener(object : Window.OnFrameMetricsAvailableListener { private var totalFrames = 0 private var jankyFrames = 0 override fun onFrameMetricsAvailable( window: Window, frameMetrics: FrameMetrics, drop: Int ) { totalFrames++ val duration = (0.000001 * frameMetrics.getMetric(FrameMetrics.TOTAL_DURATION)).toFloat() if (duration > 16f) { jankyFrames++ percJankyFrames = jankyFrames.toFloat() / totalFrames * 100 } } }, handler) @orbycius var percJankyFrames = 0f @Test fun testFirst() { for (i in 0..30) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition<RecyclerView.ViewHolder>(i))
45. val handler = Handler(Looper.getMainLooper()) activityRule.activity.window .addOnFrameMetricsAvailableListener(object : Window.OnFrameMetricsAvailableListener { private var totalFrames = 0 private var jankyFrames = 0 override fun onFrameMetricsAvailable( window: Window, frameMetrics: FrameMetrics, drop: Int ) { totalFrames++ val duration = (0.000001 * frameMetrics.getMetric(FrameMetrics.TOTAL_DURATION)).toFloat() if (duration > 16f) { jankyFrames++ percJankyFrames = jankyFrames.toFloat() / totalFrames * 100 } } }, handler) @orbycius var percJankyFrames = 0f @Test fun testFirst() { for (i in 0..30) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition<RecyclerView.ViewHolder>(i))
46. @orbycius Results companion object { private const val PERCENTAGE = 20f } // with Thread.sleep() Percentage of janky frames was 81.08% // without Thread.sleep() Percentage of janky frames was 0%
47. @orbycius Espresso vs UIAutomator
48. @orbycius for (i in 0..30) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition<RecyclerView.ViewHolder>(i)) } for (i in 30.downTo(1)) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition<RecyclerView.ViewHolder>(i)) } Espresso UIAutomator
49. @orbycius for (i in 0..30) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition<RecyclerView.ViewHolder>(i)) } for (i in 30.downTo(1)) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition<RecyclerView.ViewHolder>(i)) } Espresso @Before fun setUp() { device = UiDevice.getInstance(InstrumentationRegistry.getInstrumentation()) } val appViews = UiScrollable(UiSelector().scrollable(true)) appViews.setAsVerticalList() appViews.scrollTextIntoView("This is item 24") appViews.scrollTextIntoView("This is item 1") UIAutomator
50. @orbycius // with Thread.sleep() Percentage of janky frames was 81.08% // without Thread.sleep() Percentage of janky frames was 0% Espresso UIAutomator // with Thread.sleep() Percentage of janky frames was 9.89% // without Thread.sleep() Percentage of janky frames was 0%
51. @orbycius 81.08% 9.89% Espresso vs UIAutomator
52. @orbycius 81.08% 9.89% 71.19% Espresso vs UIAutomator
53. @orbycius adb shell dumpsys gfxinfo
54. @orbycius
55. @orbycius
56. androidx.test.jank androidx.test.jank.annotations @orbycius androidx.test.jank androidx.test.jank.annotations
57. @UseMonitorFactory @GfxFrameStatsMonitor @GfxMonitor @JankTest androidx.test.jank androidx.test.jank.annotations @orbycius androidx.test.jank androidx.test.jank.annotations @GfxFrameStatsMonitor @GfxMonitor @JankTest @UseMonitorFactory
58. @UseMonitorFactory interface IMonitor interface IMonitorFactory @GfxFrameStatsMonitor @GfxMonitor @JankTest androidx.test.jank androidx.test.jank.annotations @orbycius androidx.test.jank androidx.test.jank.annotations @GfxFrameStatsMonitor @GfxMonitor @JankTest @UseMonitorFactory interface IMonitor interface IMonitorFactory
59. @UseMonitorFactory interface IMonitor interface IMonitorFactory @GfxFrameStatsMonitor @GfxMonitor @JankTest androidx.test.jank androidx.test.jank.annotations @orbycius androidx.test.jank androidx.test.jank.annotations @GfxFrameStatsMonitor @GfxMonitor @JankTest @UseMonitorFactory interface IMonitor interface IMonitorFactory class JankTestBase class JankTestBase
60. @UseMonitorFactory interface IMonitor interface IMonitorFactory @GfxFrameStatsMonitor @GfxMonitor @JankTest androidx.test.jank androidx.test.jank.annotations @orbycius class JankTestBase class JankTestBase
61. @orbycius JankTestBase
62. @orbycius Monitor JankTestBase
63. @orbycius Monitor JankTestBase monitor.startIteration()
64. @orbycius Monitor JankTestBase monitor.startIteration() test.run()
65. @orbycius Monitor JankTestBase monitor.startIteration() test.run() monitor.stopIteration()
66. @orbycius Monitor JankTestBase get results and compare monitor.startIteration() test.run() monitor.stopIteration()
67. @orbycius Monitor JankTestBase get results and compare monitor.startIteration() test.run() monitor.stopIteration() FAILS PASS
68. class SecondTest : JankTestBase() { private lateinit var device: UiDevice @Throws(Exception::class) public override fun setUp() { super.setUp() device = UiDevice.getInstance(instrumentation) } private fun launchApp(packageName: String) { val pm = instrumentation.context.packageManager val appIntent = pm.getLaunchIntentForPackage(packageName) appIntent!!.flags = Intent.FLAG_ACTIVITY_NEW_TASK instrumentation.context.startActivity(appIntent) } @Throws(OperationApplicationException::class, RemoteException::class) fun launchApp() { launchApp(PACKAGE_NAME) device.waitForIdle() } } @orbycius class SecondTest : JankTestBase() { private lateinit var device: UiDevice @Throws(Exception::class) public override fun setUp() { super.setUp() device = UiDevice.getInstance(instrumentation) } private fun launchApp(packageName: String) { val pm = instrumentation.context.packageManager val appIntent = pm.getLaunchIntentForPackage(packageName) appIntent!!.flags = Intent.FLAG_ACTIVITY_NEW_TASK instrumentation.context.startActivity(appIntent) } @Throws(OperationApplicationException::class, RemoteException::class) fun launchApp() { launchApp(PACKAGE_NAME) device.waitForIdle() } }
69. class SecondTest : JankTestBase() { private lateinit var device: UiDevice @Throws(Exception::class) public override fun setUp() { super.setUp() device = UiDevice.getInstance(instrumentation) } private fun launchApp(packageName: String) { val pm = instrumentation.context.packageManager val appIntent = pm.getLaunchIntentForPackage(packageName) appIntent!!.flags = Intent.FLAG_ACTIVITY_NEW_TASK instrumentation.context.startActivity(appIntent) } @Throws(OperationApplicationException::class, RemoteException::class) fun launchApp() { launchApp(PACKAGE_NAME) device.waitForIdle() } } class SecondTest : JankTestBase() { private lateinit var device: UiDevice @Throws(Exception::class) public override fun setUp() { super.setUp() device = UiDevice.getInstance(instrumentation) } private fun launchApp(packageName: String) { val pm = instrumentation.context.packageManager val appIntent = pm.getLaunchIntentForPackage(packageName) appIntent!!.flags = Intent.FLAG_ACTIVITY_NEW_TASK instrumentation.context.startActivity(appIntent) } @Throws(OperationApplicationException::class, RemoteException::class) fun launchApp() { launchApp(PACKAGE_NAME) device.waitForIdle() } } @orbycius
70. class SecondTest : JankTestBase() { private lateinit var device: UiDevice @Throws(Exception::class) public override fun setUp() { super.setUp() device = UiDevice.getInstance(instrumentation) } private fun launchApp(packageName: String) { val pm = instrumentation.context.packageManager val appIntent = pm.getLaunchIntentForPackage(packageName) appIntent!!.flags = Intent.FLAG_ACTIVITY_NEW_TASK instrumentation.context.startActivity(appIntent) } @Throws(OperationApplicationException::class, RemoteException::class) fun launchApp() { launchApp(PACKAGE_NAME) device.waitForIdle() } } @orbycius
71. @orbycius class SecondTest : JankTestBase() { @JankTest( beforeTest = "launchApp", expectedFrames = EXPECTED_FRAMES, defaultIterationCount = 1 ) @GfxMonitor(processName = PACKAGE_NAME) fun testSecond() { for (i in 0 until INNER_LOOP) { val appViews = UiScrollable(UiSelector().scrollable(true)) appViews.setAsVerticalList() appViews.scrollTextIntoView("This is item 24") appViews.scrollTextIntoView("This is item 1") } SystemClock.sleep(200) } companion object { private const val INNER_LOOP = 2 private const val EXPECTED_FRAMES = 450 private const val PACKAGE_NAME = "com.marcosholgado.performancetest" }
72. @JankTest( beforeTest = "launchApp", expectedFrames = EXPECTED_FRAMES, defaultIterationCount = 1 ) @GfxMonitor(processName = PACKAGE_NAME) fun testSecond() { for (i in 0 until INNER_LOOP) { val appViews = UiScrollable(UiSelector().scrollable(true)) appViews.setAsVerticalList() appViews.scrollTextIntoView("This is item 24") appViews.scrollTextIntoView("This is item 1") } SystemClock.sleep(200) } companion object { private const val INNER_LOOP = 2 private const val EXPECTED_FRAMES = 450 private const val PACKAGE_NAME = "com.marcosholgado.performancetest" } @JankTest( beforeTest = "launchApp", expectedFrames = EXPECTED_FRAMES, defaultIterationCount = 1 ) @GfxMonitor(processName = PACKAGE_NAME) fun testSecond() { for (i in 0 until INNER_LOOP) { val appViews = UiScrollable(UiSelector().scrollable(true)) appViews.setAsVerticalList() appViews.scrollTextIntoView("This is item 24") appViews.scrollTextIntoView("This is item 1") } SystemClock.sleep(200) } companion object { private const val INNER_LOOP = 2 private const val EXPECTED_FRAMES = 450 private const val PACKAGE_NAME = "com.marcosholgado.performancetest" } @orbycius class SecondTest : JankTestBase() {
73. @JankTest( beforeTest = "launchApp", expectedFrames = EXPECTED_FRAMES, defaultIterationCount = 1 ) @GfxMonitor(processName = PACKAGE_NAME) fun testSecond() { for (i in 0 until INNER_LOOP) { val appViews = UiScrollable(UiSelector().scrollable(true)) appViews.setAsVerticalList() appViews.scrollTextIntoView("This is item 24") appViews.scrollTextIntoView("This is item 1") } SystemClock.sleep(200) } companion object { private const val INNER_LOOP = 2 private const val EXPECTED_FRAMES = 450 private const val PACKAGE_NAME = "com.marcosholgado.performancetest" } @JankTest( beforeTest = "launchApp", expectedFrames = EXPECTED_FRAMES, defaultIterationCount = 1 ) @GfxMonitor(processName = PACKAGE_NAME) fun testSecond() { for (i in 0 until INNER_LOOP) { val appViews = UiScrollable(UiSelector().scrollable(true)) appViews.setAsVerticalList() appViews.scrollTextIntoView("This is item 24") appViews.scrollTextIntoView("This is item 1") } SystemClock.sleep(200) } companion object { private const val INNER_LOOP = 2 private const val EXPECTED_FRAMES = 450 private const val PACKAGE_NAME = "com.marcosholgado.performancetest" } @orbycius class SecondTest : JankTestBase() {
74. @JankTest( beforeTest = "launchApp", expectedFrames = EXPECTED_FRAMES, defaultIterationCount = 1 ) @GfxMonitor(processName = PACKAGE_NAME) fun testSecond() { for (i in 0 until INNER_LOOP) { val appViews = UiScrollable(UiSelector().scrollable(true)) appViews.setAsVerticalList() appViews.scrollTextIntoView("This is item 24") appViews.scrollTextIntoView("This is item 1") } SystemClock.sleep(200) } companion object { private const val INNER_LOOP = 2 private const val EXPECTED_FRAMES = 450 private const val PACKAGE_NAME = "com.marcosholgado.performancetest" } @orbycius class SecondTest : JankTestBase() {
75. @orbycius // with Thread.sleep() Expected frames was 450. Received frames is 420
76. @orbycius // with Thread.sleep() Expected frames was 450. Received frames is 420 // without Thread.sleep() Expected frames was 450. Received frames is 476
77. @orbycius Problems with GfxMonitor
78. @orbycius How to calculate totalFrames (threshold) Problems with GfxMonitor
79. @orbycius How to calculate totalFrames (threshold) Problems with GfxMonitor Frame deadline errors
80. @orbycius How to calculate totalFrames (threshold) Problems with GfxMonitor Frame deadline errors junit.framework.AssertionFailedError: Failed to parse NUM_FRAME_DEADLINE_MISSED
81. androidx.test.jank androidx.test.jank.annotations @GfxFrameStatsMonitor @GfxMonitor @JankTest @UseMonitorFactory interface IMonitor interface IMonitorFactory class JankTestBase @UseMonitorFactory @orbycius androidx.test.jank androidx.test.jank.annotations @GfxFrameStatsMonitor @GfxMonitor @JankTest @UseMonitorFactory interface IMonitor interface IMonitorFactory class JankTestBase @UseMonitorFactory
82. androidx.test.jank androidx.test.jank.annotations @GfxFrameStatsMonitor @GfxMonitor @JankTest @UseMonitorFactory interface IMonitor interface IMonitorFactory class JankTestBase @UseMonitorFactory @orbycius
83. @orbycius class ThirdTest : JankTestBase() { ... } class ThirdTest : JankTestBase() { ... }
84. @orbycius class ThirdTest : JankTestBase() { ... } @UseMonitorFactory(MyFactory::class)
85. } class MyFactory(private val instrumentation: Instrumentation): IMonitorFactory { @orbycius class MyFactory(private val instrumentation: Instrumentation): IMonitorFactory { }
86. override fun getMonitors(testMethod: Method?, instance: Any?): MutableList<IMonitor> { } } class MyFactory(private val instrumentation: Instrumentation): IMonitorFactory { @orbycius override fun getMonitors(testMethod: Method?, instance: Any?): MutableList<IMonitor> { }
87. val monitors = mutableListOf<IMonitor>() override fun getMonitors(testMethod: Method?, instance: Any?): MutableList<IMonitor> { } } class MyFactory(private val instrumentation: Instrumentation): IMonitorFactory { @orbycius val monitors = mutableListOf<IMonitor>()
88. val gfxMonitorArgs = testMethod?.getAnnotation(GfxMonitor::class.java) val monitors = mutableListOf<IMonitor>() override fun getMonitors(testMethod: Method?, instance: Any?): MutableList<IMonitor> { } } class MyFactory(private val instrumentation: Instrumentation): IMonitorFactory { @orbycius val gfxMonitorArgs = testMethod?.getAnnotation(GfxMonitor::class.java)
89. val gfxMonitorArgs = testMethod?.getAnnotation(GfxMonitor::class.java) val monitors = mutableListOf<IMonitor>() override fun getMonitors(testMethod: Method?, instance: Any?): MutableList<IMonitor> { } } class MyFactory(private val instrumentation: Instrumentation): IMonitorFactory { @orbycius if (gfxMonitorArgs != null) { // GfxMonitor only works on M+ if (API_LEVEL_ACTUAL <= 22) { Log.w("PerfTest", "Skipping GfxMonitor. Not supported”) } else { monitors.add(MyMonitor( instrumentation, gfxMonitorArgs.processName)) } } return monitors
90. class GfxMonitorImpl( private val instrumentation: Instrumentation, private val process: String ) : IMonitor { } @orbycius class GfxMonitorImpl( private val instrumentation: Instrumentation, private val process: String ) : IMonitor { }
91. class GfxMonitorImpl( private val instrumentation: Instrumentation, private val process: String ) : IMonitor { } @orbycius override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { }
92. @orbycius override fun startIteration() { // Clear out any previous data val stdout = executeShellCommand( String.format("dumpsys gfxinfo %s reset", process) ) val reader = BufferedReader(InputStreamReader(stdout)) reader.use { reader -> // Read the output, but don't do anything with it while (reader.readLine() != null) { } } } override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { }
93. override fun stopIteration(): Bundle { } @orbycius override fun stopIteration(): Bundle { } override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { }
94. // Dump the latest stats val stdout = executeShellCommand( String.format("dumpsys gfxinfo %s", process) ) override fun stopIteration(): Bundle { } @orbycius override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { } // Dump the latest stats val stdout = executeShellCommand( String.format("dumpsys gfxinfo %s", process) )
95. val reader = BufferedReader(InputStreamReader(stdout)) reader.use { reader -> var line: String? = reader.readLine() do { // Parse line as frame stat value } } // Dump the latest stats val stdout = executeShellCommand( String.format("dumpsys gfxinfo %s", process) ) override fun stopIteration(): Bundle { } @orbycius override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { } val reader = BufferedReader(InputStreamReader(stdout)) reader.use { reader -> var line: String? = reader.readLine() do { // Parse line as frame stat value } }
96. val reader = BufferedReader(InputStreamReader(stdout)) reader.use { reader -> var line: String? = reader.readLine() do { // Parse line as frame stat value } } // Dump the latest stats val stdout = executeShellCommand( String.format("dumpsys gfxinfo %s", process) ) override fun stopIteration(): Bundle { } @orbycius override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { } // Make sure we found all the stats ... val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret
97. @orbycius // Parse line as frame stat value // Make sure we found all the stats
98. @orbycius // Parse line as frame stat value enum class JankStat private constructor( private val parsePattern: Pattern, private val groupIndex: Int, internal val type: Class<*>, optional: Boolean = false ) { TOTAL_FRAMES( Pattern.compile("\\s*Total frames rendered: (\\d+)"), 1, Int::class.java ) }
99. @orbycius // Parse line as frame stat value enum class JankStat private constructor( private val parsePattern: Pattern, private val groupIndex: Int, internal val type: Class<*>, optional: Boolean = false ) { TOTAL_FRAMES( Pattern.compile("\\s*Total frames rendered: (\\d+)"), 1, Int::class.java ) } Total frames rendered: ### Janky frames: ### (##.##%) 50th percentile: ##ms 90th percentile: ##ms 95th percentile: ##ms 99th percentile: ##ms Number Missed Vsync: # Number High input latency: # Number Slow UI thread: # Number Slow bitmap uploads: # Number Slow draw: # Number Frame deadline missed: #
100. @orbycius // Parse line as frame stat value for (stat in JankStat.values()) { val part: String? = stat.parse(line!!) if (part != null) { when { stat.type == Int::class.java -> { val stats = accumulatedStats[stat] as MutableList<Int> stats.add(Integer.valueOf(part)) } stat.type == Double::class.java -> { val stats = accumulatedStats[stat] as MutableList<Double> stats.add(java.lang.Double.valueOf(part)) } else -> // Shouldn't get here throw IllegalStateException("Unsupported JankStat type") } break } }
101. for (stat in JankStat.values()) { val part: String? = stat.parse(line!!) if (part != null) { when { stat.type == Int::class.java -> { val stats = accumulatedStats[stat] as MutableList<Int> stats.add(Integer.valueOf(part)) } stat.type == Double::class.java -> { val stats = accumulatedStats[stat] as MutableList<Double> stats.add(java.lang.Double.valueOf(part)) } else -> // Shouldn't get here throw IllegalStateException("Unsupported JankStat type") } break } } @orbycius private val accumulatedStats = EnumMap<JankStat, List<Number>>(JankStat::class.java) // Parse line as frame stat value
102. @orbycius // Make sure we found all the stats
103. for (stat in JankStat.values()) { if (!stat.wasParsedSuccessfully() && !stat.isOptional) { Assert.fail(String.format("Failed to parse %s", stat.name)) } stat.reset() } val jankyFreeFrames = accumulatedStats[JankStat.TOTAL_FRAMES] as List<Int> @orbycius // Make sure we found all the stats for (stat in JankStat.values()) { if (!stat.wasParsedSuccessfully() && !stat.isOptional) { Assert.fail(String.format("Failed to parse %s", stat.name)) } stat.reset() } val jankyFreeFrames = accumulatedStats[JankStat.TOTAL_FRAMES] as List<Int>
104. for (stat in JankStat.values()) { if (!stat.wasParsedSuccessfully() && !stat.isOptional) { Assert.fail(String.format("Failed to parse %s", stat.name)) } stat.reset() } val jankyFreeFrames = accumulatedStats[JankStat.TOTAL_FRAMES] as List<Int> @orbycius // Make sure we found all the stats val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret
105. override fun getMetrics(): Bundle { } @orbycius override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { } override fun getMetrics(): Bundle { }
106. override fun getMetrics(): Bundle { } @orbycius override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { } val metrics = Bundle() // Retrieve the total number of frames val totals = accumulatedStats[JankStat.TOTAL_FRAMES] as List<Int> // Store avg, min and max of total frames metrics.putInt(GfxMonitor.KEY_AVG_TOTAL_FRAMES, computeAverage(totals)) metrics.putInt(GfxMonitor.KEY_MAX_TOTAL_FRAMES, Collections.max(totals)) metrics.putInt(GfxMonitor.KEY_MIN_TOTAL_FRAMES, Collections.min(totals)) ... return metrics
107. @orbycius class GfxMonitorImpl( private val instrumentation: Instrumentation, private val process: String ) : IMonitor { override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { } }
108. @orbycius class GfxMonitorImpl( private val instrumentation: Instrumentation, private val process: String ) : IMonitor { override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { } } junit.framework.AssertionFailedError: Failed to parse NUM_FRAME_DEADLINE_MISSED
109. junit.framework.AssertionFailedError: Failed to parse NUM_FRAME_DEADLINE_MISSED for (stat in JankStat.values()) { if (!stat.wasParsedSuccessfully() && !stat.isOptional) { Assert.fail(String.format("Failed to parse %s", stat.name)) } stat.reset() } @orbycius for (stat in JankStat.values()) { if (!stat.wasParsedSuccessfully() && !stat.isOptional) { Assert.fail(String.format("Failed to parse %s", stat.name)) } stat.reset() } junit.framework.AssertionFailedError: Failed to parse NUM_FRAME_DEADLINE_MISSED
110. for (stat in JankStat.values()) { if (!stat.wasParsedSuccessfully() && !stat.isOptional) { Assert.fail(String.format("Failed to parse %s", stat.name)) } stat.reset() } junit.framework.AssertionFailedError: Failed to parse NUM_FRAME_DEADLINE_MISSED for (stat in JankStat.values()) { if (!stat.wasParsedSuccessfully() && !stat.isOptional) { Assert.fail(String.format("Failed to parse %s", stat.name)) } stat.reset() } @orbycius
111. enum class JankStat private constructor( private val parsePattern: Pattern, private val groupIndex: Int, internal val type: Class<*>, optional: Boolean = false ) { TOTAL_FRAMES( Pattern.compile("\\s*Total frames rendered: (\\d+)"), 1, Int::class.java ), NUM_FRAME_DEADLINE_MISSED( Pattern.compile("\\s*Number Frame deadline missed: (\\d+)"), 1, Int::class.java junit.framework.AssertionFailedError: Failed to parse NUM_FRAME_DEADLINE_MISSED @orbycius junit.framework.AssertionFailedError: Failed to parse NUM_FRAME_DEADLINE_MISSED enum class JankStat private constructor( private val parsePattern: Pattern, private val groupIndex: Int, internal val type: Class<*>, optional: Boolean = false ) { TOTAL_FRAMES( Pattern.compile("\\s*Total frames rendered: (\\d+)"), 1, Int::class.java ), NUM_FRAME_DEADLINE_MISSED( Pattern.compile("\\s*Number Frame deadline missed: (\\d+)"), 1, Int::class.java ); );
112. enum class JankStat private constructor( private val parsePattern: Pattern, private val groupIndex: Int, internal val type: Class<*>, optional: Boolean = false ) { TOTAL_FRAMES( Pattern.compile("\\s*Total frames rendered: (\\d+)"), 1, Int::class.java ), NUM_FRAME_DEADLINE_MISSED( Pattern.compile("\\s*Number Frame deadline missed: (\\d+)"), 1, Int::class.java junit.framework.AssertionFailedError: Failed to parse NUM_FRAME_DEADLINE_MISSED @orbycius junit.framework.AssertionFailedError: Failed to parse NUM_FRAME_DEADLINE_MISSED );
113. enum class JankStat private constructor( private val parsePattern: Pattern, private val groupIndex: Int, internal val type: Class<*>, optional: Boolean = false ) { TOTAL_FRAMES( Pattern.compile("\\s*Total frames rendered: (\\d+)"), 1, Int::class.java ), NUM_FRAME_DEADLINE_MISSED( Pattern.compile("\\s*Number Frame deadline missed: (\\d+)"), 1, Int::class.java junit.framework.AssertionFailedError: Failed to parse NUM_FRAME_DEADLINE_MISSED @orbycius junit.framework.AssertionFailedError: Failed to parse NUM_FRAME_DEADLINE_MISSED ); , true
114. @orbycius class MyMonitor( private val instrumentation: Instrumentation, private val process: String ) : } GfxMonitorImpl {
115. @orbycius class MyMonitor( private val instrumentation: Instrumentation, private val process: String ) : } GfxMonitorImpl {
116. @orbycius class MyMonitor( private val instrumentation: Instrumentation, private val process: String ) : } IMonitor {
117. @orbycius How can we get other stats?
118. val jankyFreeFrames = accumulatedStats[JankStat.TOTAL_FRAMES] as List<Int> val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret @orbycius val jankyFreeFrames = accumulatedStats[JankStat.TOTAL_FRAMES] as List<Int> val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret
119. val jankyFreeFrames = accumulatedStats[JankStat.TOTAL_FRAMES] as List<Int> val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret @orbycius
120. val jankyFreeFrames = accumulatedStats[JankStat.TOTAL_FRAMES] as List<Int> val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret @orbycius Total frames rendered: ### Janky frames: ### (##.##%) 50th percentile: ##ms 90th percentile: ##ms 95th percentile: ##ms 99th percentile: ##ms Number Missed Vsync: # Number High input latency: # Number Slow UI thread: # Number Slow bitmap uploads: # Number Slow draw: # Number Frame deadline missed: #
121. val jankyFreeFrames = accumulatedStats[JankStat.TOTAL_FRAMES] as List<Int> val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret @orbycius Total frames rendered: ### Janky frames: ### (##.##%) 50th percentile: ##ms 90th percentile: ##ms 95th percentile: ##ms 99th percentile: ##ms Number Missed Vsync: # Number High input latency: # Number Slow UI thread: # Number Slow bitmap uploads: # Number Slow draw: # Number Frame deadline missed: # JankStat.TOTAL_FRAMES JankStat.NUM_JANKY JankStat.FRAME_TIME_50TH JankStat.FRAME_TIME_90TH JankStat.FRAME_TIME_95TH JankStat.FRAME_TIME_99TH JankStat.NUM_MISSED_VSYNC JankStat.NUM_HIGH_INPUT_LATENCY JankStat.NUM_SLOW_UI_THREAD JankStat.NUM_SLOW_BITMAP_UPLOADS JankStat.NUM_SLOW_DRAW JankStat.NUM_FRAME_DEADLINE_MISSED
122. @orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List<Int> val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret
123. public class JankTestBase extends InstrumentationTestCase { protected void runTest() throws Throwable { ... Bundle results = monitor.stopIteration(); int numFrames = results.getInt("num-frames", 0); // Fail the test if we didn't get enough frames assertTrue(String.format( "Too few frames received. Monitor: %s, Expected: %d, Received: %d.", monitor.getClass().getSimpleName(), annotation.expectedFrames(), numFrames), numFrames >= annotation.expectedFrames()); ... } } @orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List<Int> val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret public class JankTestBase extends InstrumentationTestCase { protected void runTest() throws Throwable { ... Bundle results = monitor.stopIteration(); int numFrames = results.getInt("num-frames", 0); // Fail the test if we didn't get enough frames assertTrue(String.format( "Too few frames received. Monitor: %s, Expected: %d, Received: %d.", monitor.getClass().getSimpleName(), annotation.expectedFrames(), numFrames), numFrames >= annotation.expectedFrames()); ... } }
124. public class JankTestBase extends InstrumentationTestCase { protected void runTest() throws Throwable { ... Bundle results = monitor.stopIteration(); int numFrames = results.getInt("num-frames", 0); // Fail the test if we didn't get enough frames assertTrue(String.format( "Too few frames received. Monitor: %s, Expected: %d, Received: %d.", monitor.getClass().getSimpleName(), annotation.expectedFrames(), numFrames), numFrames >= annotation.expectedFrames()); ... } } public class JankTestBase extends InstrumentationTestCase { protected void runTest() throws Throwable { ... Bundle results = monitor.stopIteration(); int numFrames = results.getInt("num-frames", 0); // Fail the test if we didn't get enough frames assertTrue(String.format( "Too few frames received. Monitor: %s, Expected: %d, Received: %d.", monitor.getClass().getSimpleName(), annotation.expectedFrames(), numFrames), numFrames >= annotation.expectedFrames()); ... } } @orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List<Int> val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret
125. public class JankTestBase extends InstrumentationTestCase { protected void runTest() throws Throwable { ... Bundle results = monitor.stopIteration(); int numFrames = results.getInt("num-frames", 0); // Fail the test if we didn't get enough frames assertTrue(String.format( "Too few frames received. Monitor: %s, Expected: %d, Received: %d.", monitor.getClass().getSimpleName(), annotation.expectedFrames(), numFrames), numFrames >= annotation.expectedFrames()); ... } } @orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List<Int> val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret
126. @orbycius Returns just one stat
127. @orbycius Returns just one stat Always checks >=
128. @orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List<Int> val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret monitor
129. @orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List<Int> val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret Janky frames: ### (##.##%) numFrames >= annotation.expectedFrames() monitor JankTestBase
130. @orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List<Int> val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret Janky frames: ### (##.##%) 10.50% numFrames >= annotation.expectedFrames() numFrames monitor JankTestBase
131. @orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List<Int> val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret Janky frames: ### (##.##%) 10.50% numFrames >= annotation.expectedFrames() 5.00% numFrames expectedFrames monitor JankTestBase
132. @orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List<Int> val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret Janky frames: ### (##.##%) 10.50% numFrames >= annotation.expectedFrames() >= 5.00% numFrames expectedFrames monitor JankTestBase
133. @orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List<Int> val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret Janky frames: ### (##.##%) 10.50% numFrames >= annotation.expectedFrames() >= 5.00% PASS numFrames expectedFrames monitor JankTestBase
134. @orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List<Int> val ret = Bundle() ret.putInt("num-frames", return ret Janky frames: ### (##.##%) 10.50% numFrames >= annotation.expectedFrames() >= 5.00% numFrames expectedFrames monitor JankTestBase jankyFreeFrames[jankyFreeFrames.size - 1]) 100 -
135. @orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List<Int> val ret = Bundle() ret.putInt("num-frames", return ret Janky frames: ### (##.##%) numFrames >= annotation.expectedFrames() >= 5.00% numFrames expectedFrames monitor JankTestBase jankyFreeFrames[jankyFreeFrames.size - 1]) 100 - 89.50%
136. @orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List<Int> val ret = Bundle() ret.putInt("num-frames", return ret Janky frames: ### (##.##%) numFrames >= annotation.expectedFrames() >= numFrames expectedFrames monitor JankTestBase jankyFreeFrames[jankyFreeFrames.size - 1]) 100 - 89.50% 95.00%
137. @orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List<Int> val ret = Bundle() ret.putInt("num-frames", return ret Janky frames: ### (##.##%) numFrames >= annotation.expectedFrames() >= numFrames expectedFrames monitor JankTestBase jankyFreeFrames[jankyFreeFrames.size - 1]) 100 - 89.50% 95.00% FAIL
138. @orbycius Percentiles
139. @orbycius 0 40 80 120 160 0 40 80 120 160 16.299 Average
140. @orbycius avg 99th perc 95th per 90th per 50th per
141. @orbycius avg 99th perc 95th per 90th per 50th per No sleep 4ms 7ms 5ms 5ms 5ms
142. @orbycius avg 99th perc 95th per 90th per 50th per No sleep 4ms 7ms 5ms 5ms 5ms Sleep 150ms 16ms 150ms 150ms 12ms 5ms
143. @orbycius avg 99th perc 95th per 90th per 50th per No sleep 4ms 7ms 5ms 5ms 5ms Sleep 150ms 16ms 150ms 150ms 12ms 5ms Sleep 150ms %20 4.5ms 150ms 5ms 5ms 5ms
144. @orbycius JankStat.TOTAL_FRAMES JankStat.NUM_JANKY JankStat.FRAME_TIME_50TH JankStat.FRAME_TIME_90TH JankStat.FRAME_TIME_95TH JankStat.FRAME_TIME_99TH JankStat.NUM_MISSED_VSYNC JankStat.NUM_HIGH_INPUT_LATENCY JankStat.NUM_SLOW_UI_THREAD JankStat.NUM_SLOW_BITMAP_UPLOADS JankStat.NUM_SLOW_DRAW JankStat.NUM_FRAME_DEADLINE_MISSED
145. @orbycius public class JankTestBase extends InstrumentationTestCase { }
146. @orbycius public class JankTestBase extends InstrumentationTestCase { }
147. @RunWith(AndroidJUnit4::class) class FifthTest { @orbycius @RunWith(AndroidJUnit4::class) class FifthTest {
148. private var monitor: IMonitor init { if (API_LEVEL_ACTUAL <= 22) { error("Not supported by current platform.") } else { monitor = PercentileMonitor( InstrumentationRegistry.getInstrumentation(), PACKAGE_NAME ) } } @RunWith(AndroidJUnit4::class) class FifthTest { @orbycius private var monitor: IMonitor init { if (API_LEVEL_ACTUAL <= 22) { error("Not supported by current platform.") } else { monitor = PercentileMonitor( InstrumentationRegistry.getInstrumentation(), PACKAGE_NAME ) } }
149. private var monitor: IMonitor init { if (API_LEVEL_ACTUAL <= 22) { error("Not supported by current platform.") } else { monitor = PercentileMonitor( InstrumentationRegistry.getInstrumentation(), PACKAGE_NAME ) } } @RunWith(AndroidJUnit4::class) class FifthTest { @orbycius @get:Rule val activityRule = ActivityTestRule(MainActivity::class.java)
150. @get:Rule var activityRule: ActivityTestRule<MainActivity> = object : ActivityTestRule<MainActivity>(MainActivity::class.java) { override fun beforeActivityLaunched() { monitor.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { val results = monitor.stopIteration() val percentile = results.getInt("percentilesValue", Integer.MAX_VALUE) TestCase.assertTrue( String.format( "Monitor: %s, Expected: %d, Received: %d.", monitor::class.java.simpleName, EXPECTED_MSECS, percentile ), percentile < EXPECTED_MSECS ) super.afterActivityFinished() } } @orbycius @get:Rule var activityRule: ActivityTestRule<MainActivity> = object : ActivityTestRule<MainActivity>(MainActivity::class.java) { override fun beforeActivityLaunched() { monitor.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { val results = monitor.stopIteration() val percentile = results.getInt("percentilesValue", Integer.MAX_VALUE) TestCase.assertTrue( String.format( "Monitor: %s, Expected: %d, Received: %d.", monitor::class.java.simpleName, EXPECTED_MSECS, percentile ), percentile < EXPECTED_MSECS ) super.afterActivityFinished() } }
151. @get:Rule var activityRule: ActivityTestRule<MainActivity> = object : ActivityTestRule<MainActivity>(MainActivity::class.java) { override fun beforeActivityLaunched() { monitor.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { val results = monitor.stopIteration() val percentile = results.getInt("percentilesValue", Integer.MAX_VALUE) TestCase.assertTrue( String.format( "Monitor: %s, Expected: %d, Received: %d.", monitor::class.java.simpleName, EXPECTED_MSECS, percentile ), percentile < EXPECTED_MSECS ) super.afterActivityFinished() } } @get:Rule var activityRule: ActivityTestRule<MainActivity> = object : ActivityTestRule<MainActivity>(MainActivity::class.java) { override fun beforeActivityLaunched() { monitor.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { val results = monitor.stopIteration() val percentile = results.getInt("percentilesValue", Integer.MAX_VALUE) TestCase.assertTrue( String.format( "Monitor: %s, Expected: %d, Received: %d.", monitor::class.java.simpleName, EXPECTED_MSECS, percentile ), percentile < EXPECTED_MSECS ) super.afterActivityFinished() } } @orbycius
152. @get:Rule var activityRule: ActivityTestRule<MainActivity> = object : ActivityTestRule<MainActivity>(MainActivity::class.java) { override fun beforeActivityLaunched() { monitor.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { val results = monitor.stopIteration() val percentile = results.getInt("percentilesValue", Integer.MAX_VALUE) TestCase.assertTrue( String.format( "Monitor: %s, Expected: %d, Received: %d.", monitor::class.java.simpleName, EXPECTED_MSECS, percentile ), percentile < EXPECTED_MSECS ) super.afterActivityFinished() } } @get:Rule var activityRule: ActivityTestRule<MainActivity> = object : ActivityTestRule<MainActivity>(MainActivity::class.java) { override fun beforeActivityLaunched() { monitor.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { val results = monitor.stopIteration() val percentile = results.getInt("percentilesValue", Integer.MAX_VALUE) TestCase.assertTrue( String.format( "Monitor: %s, Expected: %d, Received: %d.", monitor::class.java.simpleName, EXPECTED_MSECS, percentile ), percentile < EXPECTED_MSECS ) super.afterActivityFinished() } } @orbycius
153. @get:Rule var activityRule: ActivityTestRule<MainActivity> = object : ActivityTestRule<MainActivity>(MainActivity::class.java) { override fun beforeActivityLaunched() { monitor.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { val results = monitor.stopIteration() val percentile = results.getInt("percentilesValue", Integer.MAX_VALUE) TestCase.assertTrue( String.format( "Monitor: %s, Expected: %d, Received: %d.", monitor::class.java.simpleName, EXPECTED_MSECS, percentile ), percentile < EXPECTED_MSECS ) super.afterActivityFinished() } } @orbycius
154. @orbycius @Test @PerformanceTest( processName = PACKAGE_NAME, perfType = PerformanceTest.PerfType.AVG_FRAME_TIME_95TH, threshold = 18, assertionType = PerformanceTest.AssertionType.LESS_OR_EQUAL ) fun testSixth() { for (i in 0 until INNER_LOOP) { val appViews = UiScrollable(UiSelector().scrollable(true)) appViews.setAsVerticalList() appViews.scrollTextIntoView("This is item 24") appViews.scrollTextIntoView("This is item 1") } }
155. @orbycius val jankyFreeFrames = accumulatedStats[JankStat.FRAME_TIME_99TH] as List<Int> val ret = Bundle() ret.putInt("percentilesValue", percentileTimeInMs[percentileTimeInMs.size - 1]) return ret monitor
156. @orbycius override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { } override fun getMetrics(): Bundle { } val metrics = Bundle() // Retrieve the total number of frames val totals = accumulatedStats[JankStat.TOTAL_FRAMES] as List<Int> // Store avg, min and max of total frames metrics.putInt(GfxMonitor.KEY_AVG_TOTAL_FRAMES, computeAverage(totals)) metrics.putInt(GfxMonitor.KEY_MAX_TOTAL_FRAMES, Collections.max(totals)) metrics.putInt(GfxMonitor.KEY_MIN_TOTAL_FRAMES, Collections.min(totals)) ... return metrics val jankyFreeFrames = accumulatedStats[JankStat.FRAME_TIME_99TH] as List<Int> val ret = Bundle() ret.putInt("percentilesValue", percentileTimeInMs[percentileTimeInMs.size - 1]) return ret monitor
157. @orbycius override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { } override fun getMetrics(): Bundle { } val metrics = Bundle() // Retrieve the total number of frames val totals = accumulatedStats[JankStat.TOTAL_FRAMES] as List<Int> // Store avg, min and max of total frames metrics.putInt(GfxMonitor.KEY_AVG_TOTAL_FRAMES, computeAverage(totals)) metrics.putInt(GfxMonitor.KEY_MAX_TOTAL_FRAMES, Collections.max(totals)) metrics.putInt(GfxMonitor.KEY_MIN_TOTAL_FRAMES, Collections.min(totals)) ... return metrics monitor return getMetrics()
158. } @Retention(AnnotationRetention.RUNTIME) @Target(AnnotationTarget.FUNCTION) annotation class PerformanceTest( val processName: String = "", val perfType: PerfType val threshold: Int = Int.MAX_VALUE, val assertionType: AssertionType ) { @orbycius @Retention(AnnotationRetention.RUNTIME) @Target(AnnotationTarget.FUNCTION) annotation class PerformanceTest( val processName: String = "", val perfType: PerfType val threshold: Int = Int.MAX_VALUE, val assertionType: AssertionType ) { } ,
159. } @Retention(AnnotationRetention.RUNTIME) @Target(AnnotationTarget.FUNCTION) annotation class PerformanceTest( val processName: String = "", val perfType: PerfType val threshold: Int = Int.MAX_VALUE, val assertionType: AssertionType ) { @orbycius , enum class PerfType(val type: String) { TOTAL_FRAMES("gfx-avg-total-frames"), MAX_TOTAL_FRAMES("gfx-max-total-frames"), MIN_TOTAL_FRAMES("gfx-min-total-frames"), … } = PerfType.TOTAL_FRAMES
160. @Retention(AnnotationRetention.RUNTIME) @Target(AnnotationTarget.FUNCTION) annotation class PerformanceTest( val processName: String = "", val perfType: PerfType = PerfType.TOTAL_FRAMES, val threshold: Int = Int.MAX_VALUE, val assertionType: AssertionType ) { } @orbycius enum class PerfType(val type: String) { TOTAL_FRAMES("gfx-avg-total-frames"), MAX_TOTAL_FRAMES("gfx-max-total-frames"), MIN_TOTAL_FRAMES("gfx-min-total-frames"), … } enum class AssertionType(val type: Int) { LESS(0), LESS_OR_EQUAL(1), GREATER(2), GREATER_OR_EQUAL(3), EQUAL(4) } = AssertionType.LESS_OR_EQUAL
161. } open class ActivityPerfTestRule<T: Activity>(activityClass: Class<T>): ActivityTestRule<T>(activityClass) { @orbycius open class ActivityPerfTestRule<T: Activity>(activityClass: Class<T>): ActivityTestRule<T>(activityClass) { }
162. } open class ActivityPerfTestRule<T: Activity>(activityClass: Class<T>): ActivityTestRule<T>(activityClass) { @orbycius private var monitor: IMonitor? = null private var annotation: PerformanceTest? = null init { if (API_LEVEL_ACTUAL <= 22) { error("Not supported by current platform.") } }
163. } open class ActivityPerfTestRule<T: Activity>(activityClass: Class<T>): ActivityTestRule<T>(activityClass) { @orbycius private var monitor: IMonitor? = null private var annotation: PerformanceTest? = null init { if (API_LEVEL_ACTUAL <= 22) { error("Not supported by current platform.") } } override fun apply(base: Statement?, description: Description?): Statement { annotation = description?.getAnnotation(PerformanceTest::class.java) annotation?.let { monitor = PerfMonitor( InstrumentationRegistry.getInstrumentation(), it.processName ) } return super.apply(base, description) }
164. @orbycius open class ActivityPerfTestRule<T: Activity>(activityClass: Class<T>): ActivityTestRule<T>(activityClass) { // ... }
165. override fun beforeActivityLaunched() { monitor?.startIteration() super.beforeActivityLaunched() } @orbycius open class ActivityPerfTestRule<T: Activity>(activityClass: Class<T>): ActivityTestRule<T>(activityClass) { // ... } override fun beforeActivityLaunched() { monitor?.startIteration() super.beforeActivityLaunched() }
166. override fun beforeActivityLaunched() { monitor?.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { monitor?.let { val results = it.stopIteration() val res: Double = results?.get(annotation?.perfType?.type) as Double val assertion = when(annotation?.assertionType) { PerformanceTest.AssertionType.LESS -> res < annotation!!.threshold PerformanceTest.AssertionType.GREATER -> res > annotation!!.threshold … } TestCase.assertTrue("Error msg here.", assertion) } super.afterActivityFinished() } @orbycius open class ActivityPerfTestRule<T: Activity>(activityClass: Class<T>): ActivityTestRule<T>(activityClass) { // ... } override fun beforeActivityLaunched() { monitor?.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { monitor?.let { val results = it.stopIteration() val res: Double = results?.get(annotation?.perfType?.type) as Double val assertion = when(annotation?.assertionType) { PerformanceTest.AssertionType.LESS -> res < annotation!!.threshold PerformanceTest.AssertionType.GREATER -> res > annotation!!.threshold … } TestCase.assertTrue("Error msg here.", assertion) } super.afterActivityFinished() }
167. override fun beforeActivityLaunched() { monitor?.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { monitor?.let { val results = it.stopIteration() val res: Double = results?.get(annotation?.perfType?.type) as Double val assertion = when(annotation?.assertionType) { PerformanceTest.AssertionType.LESS -> res < annotation!!.threshold PerformanceTest.AssertionType.GREATER -> res > annotation!!.threshold … } TestCase.assertTrue("Error msg here.", assertion) } super.afterActivityFinished() } override fun afterActivityFinished() { monitor?.let { val results = it.stopIteration() val res: Double = results?.get(annotation?.perfType?.type) as Double val assertion = when(annotation?.assertionType) { PerformanceTest.AssertionType.LESS -> res < annotation!!.threshold PerformanceTest.AssertionType.GREATER -> res > annotation!!.threshold … } TestCase.assertTrue("Error msg here.", assertion) } super.afterActivityFinished() } @orbycius open class ActivityPerfTestRule<T: Activity>(activityClass: Class<T>): ActivityTestRule<T>(activityClass) { // ... }
168. override fun beforeActivityLaunched() { monitor?.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { monitor?.let { val results = it.stopIteration() val res: Double = results?.get(annotation?.perfType?.type) as Double val assertion = when(annotation?.assertionType) { PerformanceTest.AssertionType.LESS -> res < annotation!!.threshold PerformanceTest.AssertionType.GREATER -> res > annotation!!.threshold … } TestCase.assertTrue("Error msg here.", assertion) } super.afterActivityFinished() } @orbycius open class ActivityPerfTestRule<T: Activity>(activityClass: Class<T>): ActivityTestRule<T>(activityClass) { // ... }
169. @orbycius @Test @PerformanceTest( processName = PACKAGE_NAME, perfType = PerformanceTest.PerfType.AVG_FRAME_TIME_95TH, threshold = 18, assertionType = PerformanceTest.AssertionType.LESS_OR_EQUAL ) fun testSixth() { for (i in 0 until INNER_LOOP) { val appViews = UiScrollable(UiSelector().scrollable(true)) appViews.setAsVerticalList() appViews.scrollTextIntoView("This is item 24") appViews.scrollTextIntoView("This is item 1") } }
170. @orbycius Memory Leaks
171. public class LeakActivity extends Activity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_leak); View button = findViewById(R.id.button); button.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View v) { startAsyncWork(); } }); } void startAsyncWork() { Runnable work = new Runnable() { @Override public void run() { SystemClock.sleep(20000); } }; new Thread(work).start(); } } @orbycius public class LeakActivity extends Activity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_leak); View button = findViewById(R.id.button); button.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View v) { startAsyncWork(); } }); } void startAsyncWork() { Runnable work = new Runnable() { @Override public void run() { SystemClock.sleep(20000); } }; new Thread(work).start(); } }
172. public class LeakActivity extends Activity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_leak); View button = findViewById(R.id.button); button.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View v) { startAsyncWork(); } }); } void startAsyncWork() { Runnable work = new Runnable() { @Override public void run() { SystemClock.sleep(20000); } }; new Thread(work).start(); } } public class LeakActivity extends Activity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_leak); View button = findViewById(R.id.button); button.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View v) { startAsyncWork(); } }); } void startAsyncWork() { Runnable work = new Runnable() { @Override public void run() { SystemClock.sleep(20000); } }; new Thread(work).start(); } } @orbycius
173. public class LeakActivity extends Activity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_leak); View button = findViewById(R.id.button); button.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View v) { startAsyncWork(); } }); } void startAsyncWork() { Runnable work = new Runnable() { @Override public void run() { SystemClock.sleep(20000); } }; new Thread(work).start(); } } @orbycius
174. @orbycius class SeventhTest { @get:Rule var mainActivityActivityTestRule = ActivityTestRule(LeakActivity::class.java) @Test fun testLeaks() { onView(withId(R.id.button)).perform(click()) } }
175. @orbycius dependencies { ... debugImplementation 'com.squareup.leakcanary:leakcanary-android:1.6.3' androidTestImplementation 'com.squareup.leakcanary:leakcanary-android-instrumentation:1.6.3' }
176. @orbycius dependencies { ... debugImplementation 'com.squareup.leakcanary:leakcanary-android:1.6.3' androidTestImplementation 'com.squareup.leakcanary:leakcanary-android-instrumentation:1.6.3' } defaultConfig { ... testInstrumentationRunnerArgument "listener", "com.squareup.leakcanary.FailTestOnLeakRunListener" }
177. @orbycius class SeventhTest { @get:Rule var mainActivityActivityTestRule = ActivityTestRule(LeakActivity::class.java) @Test fun testLeaks() { onView(withId(R.id.button)).perform(click()) } }
178. @orbycius class SeventhTest { @get:Rule var mainActivityActivityTestRule = ActivityTestRule(LeakActivity::class.java) @Test fun testLeaks() { onView(withId(R.id.button)).perform(click()) } } Test failed because memory leaks were detected, see leak traces below. ###################################### In com.marcosholgado.performancetest.test:null:0. * com.marcosholgado.performancetest.LeakActivity has leaked:
179. @orbycius 538ms without listener
180. @orbycius 538ms without listener 21s 52ms with listener
181. adb shell am instrument -w \ > -e listener com.squareup.leakcanary.FailTestOnLeakRunListener \ > -e class com.marcosholgado.performancetest.SeventhTest#testLeaks \ > com.marcosholgado.performancetest.test/androidx.test.runner.AndroidJUnitRunner @orbycius adb shell am instrument -w \ > -e listener com.squareup.leakcanary.FailTestOnLeakRunListener \ > -e class com.marcosholgado.performancetest.SeventhTest#testLeaks \ > com.marcosholgado.performancetest.test/androidx.test.runner.AndroidJUnitRunner
182. adb shell am instrument -w \ > -e listener com.squareup.leakcanary.FailTestOnLeakRunListener \ > -e class com.marcosholgado.performancetest.SeventhTest#testLeaks \ > com.marcosholgado.performancetest.test/androidx.test.runner.AndroidJUnitRunner @orbycius
183. @orbycius class SeventhTest { @get:Rule var mainActivityActivityTestRule = ActivityTestRule(LeakActivity::class.java) @Test fun testLeaks() { onView(withId(R.id.button)).perform(click()) } }
184. @orbycius class SeventhTest { @get:Rule var mainActivityActivityTestRule = ActivityTestRule(LeakActivity::class.java) @Test fun testLeaks() { onView(withId(R.id.button)).perform(click()) } } @LeakTest
185. @orbycius @Retention(AnnotationRetention.RUNTIME) @Target(AnnotationTarget.FUNCTION) annotation class LeakTest
186. @orbycius @Retention(AnnotationRetention.RUNTIME) @Target(AnnotationTarget.FUNCTION) annotation class LeakTest class MyOtherLeakRunListener: FailTestOnLeakRunListener() { }
187. @orbycius @Retention(AnnotationRetention.RUNTIME) @Target(AnnotationTarget.FUNCTION) annotation class LeakTest class MyOtherLeakRunListener: FailTestOnLeakRunListener() { } override fun skipLeakDetectionReason(description: Description): String? { return if(description.getAnnotation(LeakTest::class.java) != null) null else "Skip Leak test" }
188. @orbycius Startup time
189. @orbycius Startup time Janky frames
190. @orbycius Startup time Janky frames Memory leaks
191. @orbycius Janky frames adb shell dumpsys gfxinfo
192. @orbycius Janky frames adb shell dumpsys gfxinfo adb shell dumpsys batterystats
193. @orbycius Janky frames adb shell dumpsys gfxinfo adb shell dumpsys batterystats adb shell dumpsys netstats
194. How to write and automate performance tests @Orbycius Marcos Holgado
195. How to write and automate performance tests @Orbycius Marcos Holgado https://github.com/marcosholgado/performance-test https://github.com/marcosholgado/dagger-playground