How to write and automate performance tests in Android

Slide 1

Slide 1 text

How to write and automate performance tests @Orbycius Marcos Holgado

Slide 2

Slide 2 text

@orbycius Slow & clunky Some user “ ”

Slide 3

Slide 3 text

@orbycius Slow & clunky Some user “ ” “ ” Laggs massively Some other user

Slide 4

Slide 4 text

@orbycius Slow & clunky Some user “ ” “ ” Laggs massively Some other user “ ” So laggy it makes me sad Another user

Slide 5

Slide 5 text

It doesn’t matter how much value your app gives to users if they can’t use it @orbycius

Slide 6

Slide 6 text

Testing is hard @orbycius

Slide 7

Slide 7 text

We didn't write tests because… {any excuse here} @orbycius

Slide 8

Slide 8 text

We didn't write tests because… lack of time @orbycius

Slide 9

Slide 9 text

We didn't write tests because… testing is hard @orbycius

Slide 10

Slide 10 text

What’s performance testing? @orbycius

Slide 11

Slide 11 text

frames @orbycius

Slide 12

Slide 12 text

60 frames per second @orbycius

Slide 13

Slide 13 text

≈ 16ms 1000ms 60frames @orbycius

Slide 14

Slide 14 text

What else? @orbycius

Slide 15

Slide 15 text

What else? Battery usage @orbycius

Slide 16

Slide 16 text

What else? Battery usage Memory usage @orbycius

Slide 17

Slide 17 text

Pre-requisite @orbycius

Slide 18

Slide 18 text

Pre-requisite SDK 23/24 @orbycius

Slide 19

Slide 19 text

Pre-requisite SDK 23/24 Don’t use emulators @orbycius

Slide 20

Slide 20 text

@orbycius https://github.com/marcosholgado/performance-test https://github.com/marcosholgado/dagger-playground

Slide 21

Slide 21 text

@orbycius Startup time

Slide 22

Slide 22 text

@RunWith(AndroidJUnit4::class) @LargeTest class FirstTest { @get:Rule val activityRule = ActivityTestRule(MainActivity::class.java) @Before fun setUp() { // do something } @After fun after() { // do something } @Test fun testFirst() { // do something } } @orbycius @RunWith(AndroidJUnit4::class) @LargeTest class FirstTest { @get:Rule val activityRule = ActivityTestRule(MainActivity::class.java) @Before fun setUp() { // do something } @After fun after() { // do something } @Test fun testFirst() { // do something } }

Slide 23

Slide 23 text

@RunWith(AndroidJUnit4::class) @LargeTest class FirstTest { @get:Rule val activityRule = ActivityTestRule(MainActivity::class.java) @Before fun setUp() { // do something } @After fun after() { // do something } @Test fun testFirst() { // do something } } @RunWith(AndroidJUnit4::class) @LargeTest class FirstTest { @get:Rule val activityRule = ActivityTestRule(MainActivity::class.java) @Before fun setUp() { // do something } @After fun after() { // do something } @Test fun testFirst() { // do something } } @orbycius

Slide 24

Slide 24 text

@RunWith(AndroidJUnit4::class) @LargeTest class FirstTest { @get:Rule val activityRule = ActivityTestRule(MainActivity::class.java) @Before fun setUp() { // do something } @After fun after() { // do something } @Test fun testFirst() { // do something } } @RunWith(AndroidJUnit4::class) @LargeTest class FirstTest { @get:Rule val activityRule = ActivityTestRule(MainActivity::class.java) @Before fun setUp() { // do something } @After fun after() { // do something } @Test fun testFirst() { // do something } } @orbycius

Slide 25

Slide 25 text

Slide 26

Slide 26 text

@orbycius @get:Rule val activityRule = ActivityTestRule(MainActivity::class.java)

Slide 27

Slide 27 text

@orbycius class MyActivityTestRule(activityClass: Class): ActivityTestRule(activityClass) { }

Slide 28

Slide 28 text

class MyActivityTestRule(activityClass: Class): ActivityTestRule(activityClass) { var timestamp: Long = 0 @orbycius }

Slide 29

Slide 29 text

Slide 30

Slide 30 text

class MyActivityTestRule(activityClass: Class): ActivityTestRule(activityClass) { var timestamp: Long = 0 @orbycius } class MyActivityTestRule(activityClass: Class): ActivityTestRule(activityClass) { var timestamp: Long = 0 override fun beforeActivityLaunched() { timestamp = System.currentTimeMillis() super.beforeActivityLaunched() } class MyActivityTestRule(activityClass: Class): ActivityTestRule(activityClass) { var timestamp: Long = 0 override fun beforeActivityLaunched() { timestamp = System.currentTimeMillis() super.beforeActivityLaunched() } override fun afterActivityLaunched() { super.afterActivityLaunched() timestamp = System.currentTimeMillis() - timestamp }

Slide 31

Slide 31 text

@orbycius @RunWith(AndroidJUnit4::class) class FirstTest { @get:Rule val activityRule = MyActivityTestRule(MainActivity::class.java) @Test fun startUpTest() { assertWithMessage("Start up time: ${activityRule.timestamp}") .that(activityRule.timestamp) .isLessThan(300) } }

Slide 32

Slide 32 text

@orbycius FrameMetrics

Slide 33

Slide 33 text

@orbycius FrameMetrics Animation Duration Command Issue Duration Draw Duration First Draw Frame Input Handling Duration Intended Vsync Timestamp Layout Measure Duration Swap Buffers Duration Sync Duration Total Duration Unknown Delay Duration Vsync Timestamp

Slide 34

Slide 34 text

@orbycius class MainAdapter : RecyclerView.Adapter() { override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): MainViewHolder { val view = LayoutInflater .from(parent.context) .inflate(R.layout.item, parent, false) return MainViewHolder(view) } override fun onBindViewHolder(holder: MainViewHolder, pos: Int) { (holder.itemView as TextView).text = "This is item $pos" // Add sleep to recreate work on the main thread Thread.sleep(150) } override fun getItemCount(): Int = 200 }

Slide 35

Slide 35 text

class MainAdapter : RecyclerView.Adapter() { override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): MainViewHolder { val view = LayoutInflater .from(parent.context) .inflate(R.layout.item, parent, false) return MainViewHolder(view) } override fun onBindViewHolder(holder: MainViewHolder, pos: Int) { (holder.itemView as TextView).text = "This is item $pos" // Add sleep to recreate work on the main thread Thread.sleep(150) } override fun getItemCount(): Int = 200 } @orbycius

Slide 36

Slide 36 text

@orbycius

Slide 37

Slide 37 text

@orbycius var percJankyFrames = 0f

Slide 38

Slide 38 text

@orbycius @Test fun testFirst() { for (i in 0..30) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition(i)) } for (i in 30.downTo(1)) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition(i)) } assertWithMessage("Janky frames over $PERCENTAGE% value was $percJankyFrames%") .that(percJankyFrames) .isLessThan(PERCENTAGE) } var percJankyFrames = 0f

Slide 39

Slide 39 text

@Test fun testFirst() { @orbycius for (i in 0..30) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition(i)) val handler = Handler(Looper.getMainLooper()) activityRule.activity.window .addOnFrameMetricsAvailableListener(object : Window.OnFrameMetricsAvailableListener { private var totalFrames = 0 private var jankyFrames = 0 override fun onFrameMetricsAvailable( window: Window, frameMetrics: FrameMetrics, drop: Int ) { totalFrames++ val duration = (0.000001 * frameMetrics.getMetric(FrameMetrics.TOTAL_DURATION)).toFloat() if (duration > 16f) { jankyFrames++ percJankyFrames = jankyFrames.toFloat() / totalFrames * 100 } } }, handler) var percJankyFrames = 0f

Slide 40

Slide 40 text

val handler = Handler(Looper.getMainLooper()) activityRule.activity.window .addOnFrameMetricsAvailableListener(object : Window.OnFrameMetricsAvailableListener { private var totalFrames = 0 private var jankyFrames = 0 override fun onFrameMetricsAvailable( window: Window, frameMetrics: FrameMetrics, drop: Int ) { totalFrames++ val duration = (0.000001 * frameMetrics.getMetric(FrameMetrics.TOTAL_DURATION)).toFloat() if (duration > 16f) { jankyFrames++ percJankyFrames = jankyFrames.toFloat() / totalFrames * 100 } } }, handler) val handler = Handler(Looper.getMainLooper()) activityRule.activity.window .addOnFrameMetricsAvailableListener(object : Window.OnFrameMetricsAvailableListener { private var totalFrames = 0 private var jankyFrames = 0 override fun onFrameMetricsAvailable( window: Window, frameMetrics: FrameMetrics, drop: Int ) { totalFrames++ val duration = (0.000001 * frameMetrics.getMetric(FrameMetrics.TOTAL_DURATION)).toFloat() if (duration > 16f) { jankyFrames++ percJankyFrames = jankyFrames.toFloat() / totalFrames * 100 } } }, handler) @orbycius var percJankyFrames = 0f @Test fun testFirst() { for (i in 0..30) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition(i))

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

Slide 45

Slide 45 text

Slide 46

Slide 46 text

@orbycius Results companion object { private const val PERCENTAGE = 20f } // with Thread.sleep() Percentage of janky frames was 81.08% // without Thread.sleep() Percentage of janky frames was 0%

Slide 47

Slide 47 text

@orbycius Espresso vs UIAutomator

Slide 48

Slide 48 text

Slide 49

Slide 49 text

@orbycius for (i in 0..30) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition(i)) } for (i in 30.downTo(1)) { Espresso.onView(ViewMatchers.withId(R.id.recyclerView)) .perform(RecyclerViewActions.scrollToPosition(i)) } Espresso @Before fun setUp() { device = UiDevice.getInstance(InstrumentationRegistry.getInstrumentation()) } val appViews = UiScrollable(UiSelector().scrollable(true)) appViews.setAsVerticalList() appViews.scrollTextIntoView("This is item 24") appViews.scrollTextIntoView("This is item 1") UIAutomator

Slide 50

Slide 50 text

@orbycius // with Thread.sleep() Percentage of janky frames was 81.08% // without Thread.sleep() Percentage of janky frames was 0% Espresso UIAutomator // with Thread.sleep() Percentage of janky frames was 9.89% // without Thread.sleep() Percentage of janky frames was 0%

Slide 51

Slide 51 text

@orbycius 81.08% 9.89% Espresso vs UIAutomator

Slide 52

Slide 52 text

@orbycius 81.08% 9.89% 71.19% Espresso vs UIAutomator

Slide 53

Slide 53 text

@orbycius adb shell dumpsys gfxinfo

Slide 54

Slide 54 text

@orbycius

Slide 55

Slide 55 text

@orbycius

Slide 56

Slide 56 text

androidx.test.jank androidx.test.jank.annotations @orbycius androidx.test.jank androidx.test.jank.annotations

Slide 57

Slide 57 text

@UseMonitorFactory @GfxFrameStatsMonitor @GfxMonitor @JankTest androidx.test.jank androidx.test.jank.annotations @orbycius androidx.test.jank androidx.test.jank.annotations @GfxFrameStatsMonitor @GfxMonitor @JankTest @UseMonitorFactory

Slide 58

Slide 58 text

@UseMonitorFactory interface IMonitor interface IMonitorFactory @GfxFrameStatsMonitor @GfxMonitor @JankTest androidx.test.jank androidx.test.jank.annotations @orbycius androidx.test.jank androidx.test.jank.annotations @GfxFrameStatsMonitor @GfxMonitor @JankTest @UseMonitorFactory interface IMonitor interface IMonitorFactory

Slide 59

Slide 59 text

Slide 60

Slide 60 text

Slide 61

Slide 61 text

@orbycius JankTestBase

Slide 62

Slide 62 text

@orbycius Monitor JankTestBase

Slide 63

Slide 63 text

@orbycius Monitor JankTestBase monitor.startIteration()

Slide 64

Slide 64 text

@orbycius Monitor JankTestBase monitor.startIteration() test.run()

Slide 65

Slide 65 text

@orbycius Monitor JankTestBase monitor.startIteration() test.run() monitor.stopIteration()

Slide 66

Slide 66 text

@orbycius Monitor JankTestBase get results and compare monitor.startIteration() test.run() monitor.stopIteration()

Slide 67

Slide 67 text

@orbycius Monitor JankTestBase get results and compare monitor.startIteration() test.run() monitor.stopIteration() FAILS PASS

Slide 68

Slide 68 text

class SecondTest : JankTestBase() { private lateinit var device: UiDevice @Throws(Exception::class) public override fun setUp() { super.setUp() device = UiDevice.getInstance(instrumentation) } private fun launchApp(packageName: String) { val pm = instrumentation.context.packageManager val appIntent = pm.getLaunchIntentForPackage(packageName) appIntent!!.flags = Intent.FLAG_ACTIVITY_NEW_TASK instrumentation.context.startActivity(appIntent) } @Throws(OperationApplicationException::class, RemoteException::class) fun launchApp() { launchApp(PACKAGE_NAME) device.waitForIdle() } } @orbycius class SecondTest : JankTestBase() { private lateinit var device: UiDevice @Throws(Exception::class) public override fun setUp() { super.setUp() device = UiDevice.getInstance(instrumentation) } private fun launchApp(packageName: String) { val pm = instrumentation.context.packageManager val appIntent = pm.getLaunchIntentForPackage(packageName) appIntent!!.flags = Intent.FLAG_ACTIVITY_NEW_TASK instrumentation.context.startActivity(appIntent) } @Throws(OperationApplicationException::class, RemoteException::class) fun launchApp() { launchApp(PACKAGE_NAME) device.waitForIdle() } }

Slide 69

Slide 69 text

class SecondTest : JankTestBase() { private lateinit var device: UiDevice @Throws(Exception::class) public override fun setUp() { super.setUp() device = UiDevice.getInstance(instrumentation) } private fun launchApp(packageName: String) { val pm = instrumentation.context.packageManager val appIntent = pm.getLaunchIntentForPackage(packageName) appIntent!!.flags = Intent.FLAG_ACTIVITY_NEW_TASK instrumentation.context.startActivity(appIntent) } @Throws(OperationApplicationException::class, RemoteException::class) fun launchApp() { launchApp(PACKAGE_NAME) device.waitForIdle() } } class SecondTest : JankTestBase() { private lateinit var device: UiDevice @Throws(Exception::class) public override fun setUp() { super.setUp() device = UiDevice.getInstance(instrumentation) } private fun launchApp(packageName: String) { val pm = instrumentation.context.packageManager val appIntent = pm.getLaunchIntentForPackage(packageName) appIntent!!.flags = Intent.FLAG_ACTIVITY_NEW_TASK instrumentation.context.startActivity(appIntent) } @Throws(OperationApplicationException::class, RemoteException::class) fun launchApp() { launchApp(PACKAGE_NAME) device.waitForIdle() } } @orbycius

Slide 70

Slide 70 text

Slide 71

Slide 71 text

@orbycius class SecondTest : JankTestBase() { @JankTest( beforeTest = "launchApp", expectedFrames = EXPECTED_FRAMES, defaultIterationCount = 1 ) @GfxMonitor(processName = PACKAGE_NAME) fun testSecond() { for (i in 0 until INNER_LOOP) { val appViews = UiScrollable(UiSelector().scrollable(true)) appViews.setAsVerticalList() appViews.scrollTextIntoView("This is item 24") appViews.scrollTextIntoView("This is item 1") } SystemClock.sleep(200) } companion object { private const val INNER_LOOP = 2 private const val EXPECTED_FRAMES = 450 private const val PACKAGE_NAME = "com.marcosholgado.performancetest" }

Slide 72

Slide 72 text

@JankTest( beforeTest = "launchApp", expectedFrames = EXPECTED_FRAMES, defaultIterationCount = 1 ) @GfxMonitor(processName = PACKAGE_NAME) fun testSecond() { for (i in 0 until INNER_LOOP) { val appViews = UiScrollable(UiSelector().scrollable(true)) appViews.setAsVerticalList() appViews.scrollTextIntoView("This is item 24") appViews.scrollTextIntoView("This is item 1") } SystemClock.sleep(200) } companion object { private const val INNER_LOOP = 2 private const val EXPECTED_FRAMES = 450 private const val PACKAGE_NAME = "com.marcosholgado.performancetest" } @JankTest( beforeTest = "launchApp", expectedFrames = EXPECTED_FRAMES, defaultIterationCount = 1 ) @GfxMonitor(processName = PACKAGE_NAME) fun testSecond() { for (i in 0 until INNER_LOOP) { val appViews = UiScrollable(UiSelector().scrollable(true)) appViews.setAsVerticalList() appViews.scrollTextIntoView("This is item 24") appViews.scrollTextIntoView("This is item 1") } SystemClock.sleep(200) } companion object { private const val INNER_LOOP = 2 private const val EXPECTED_FRAMES = 450 private const val PACKAGE_NAME = "com.marcosholgado.performancetest" } @orbycius class SecondTest : JankTestBase() {

Slide 73

Slide 73 text

Slide 74

Slide 74 text

Slide 75

Slide 75 text

@orbycius // with Thread.sleep() Expected frames was 450. Received frames is 420

Slide 76

Slide 76 text

@orbycius // with Thread.sleep() Expected frames was 450. Received frames is 420 // without Thread.sleep() Expected frames was 450. Received frames is 476

Slide 77

Slide 77 text

@orbycius Problems with GfxMonitor

Slide 78

Slide 78 text

@orbycius How to calculate totalFrames (threshold) Problems with GfxMonitor

Slide 79

Slide 79 text

@orbycius How to calculate totalFrames (threshold) Problems with GfxMonitor Frame deadline errors

Slide 80

Slide 80 text

@orbycius How to calculate totalFrames (threshold) Problems with GfxMonitor Frame deadline errors junit.framework.AssertionFailedError: Failed to parse NUM_FRAME_DEADLINE_MISSED

Slide 81

Slide 81 text

androidx.test.jank androidx.test.jank.annotations @GfxFrameStatsMonitor @GfxMonitor @JankTest @UseMonitorFactory interface IMonitor interface IMonitorFactory class JankTestBase @UseMonitorFactory @orbycius androidx.test.jank androidx.test.jank.annotations @GfxFrameStatsMonitor @GfxMonitor @JankTest @UseMonitorFactory interface IMonitor interface IMonitorFactory class JankTestBase @UseMonitorFactory

Slide 82

Slide 82 text

Slide 83

Slide 83 text

@orbycius class ThirdTest : JankTestBase() { ... } class ThirdTest : JankTestBase() { ... }

Slide 84

Slide 84 text

@orbycius class ThirdTest : JankTestBase() { ... } @UseMonitorFactory(MyFactory::class)

Slide 85

Slide 85 text

} class MyFactory(private val instrumentation: Instrumentation): IMonitorFactory { @orbycius class MyFactory(private val instrumentation: Instrumentation): IMonitorFactory { }

Slide 86

Slide 86 text

override fun getMonitors(testMethod: Method?, instance: Any?): MutableList { } } class MyFactory(private val instrumentation: Instrumentation): IMonitorFactory { @orbycius override fun getMonitors(testMethod: Method?, instance: Any?): MutableList { }

Slide 87

Slide 87 text

val monitors = mutableListOf() override fun getMonitors(testMethod: Method?, instance: Any?): MutableList { } } class MyFactory(private val instrumentation: Instrumentation): IMonitorFactory { @orbycius val monitors = mutableListOf()

Slide 88

Slide 88 text

Slide 89

Slide 89 text

val gfxMonitorArgs = testMethod?.getAnnotation(GfxMonitor::class.java) val monitors = mutableListOf() override fun getMonitors(testMethod: Method?, instance: Any?): MutableList { } } class MyFactory(private val instrumentation: Instrumentation): IMonitorFactory { @orbycius if (gfxMonitorArgs != null) { // GfxMonitor only works on M+ if (API_LEVEL_ACTUAL <= 22) { Log.w("PerfTest", "Skipping GfxMonitor. Not supported”) } else { monitors.add(MyMonitor( instrumentation, gfxMonitorArgs.processName)) } } return monitors

Slide 90

Slide 90 text

class GfxMonitorImpl( private val instrumentation: Instrumentation, private val process: String ) : IMonitor { } @orbycius class GfxMonitorImpl( private val instrumentation: Instrumentation, private val process: String ) : IMonitor { }

Slide 91

Slide 91 text

class GfxMonitorImpl( private val instrumentation: Instrumentation, private val process: String ) : IMonitor { } @orbycius override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { }

Slide 92

Slide 92 text

@orbycius override fun startIteration() { // Clear out any previous data val stdout = executeShellCommand( String.format("dumpsys gfxinfo %s reset", process) ) val reader = BufferedReader(InputStreamReader(stdout)) reader.use { reader -> // Read the output, but don't do anything with it while (reader.readLine() != null) { } } } override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { }

Slide 93

Slide 93 text

override fun stopIteration(): Bundle { } @orbycius override fun stopIteration(): Bundle { } override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { }

Slide 94

Slide 94 text

// Dump the latest stats val stdout = executeShellCommand( String.format("dumpsys gfxinfo %s", process) ) override fun stopIteration(): Bundle { } @orbycius override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { } // Dump the latest stats val stdout = executeShellCommand( String.format("dumpsys gfxinfo %s", process) )

Slide 95

Slide 95 text

val reader = BufferedReader(InputStreamReader(stdout)) reader.use { reader -> var line: String? = reader.readLine() do { // Parse line as frame stat value } } // Dump the latest stats val stdout = executeShellCommand( String.format("dumpsys gfxinfo %s", process) ) override fun stopIteration(): Bundle { } @orbycius override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { } val reader = BufferedReader(InputStreamReader(stdout)) reader.use { reader -> var line: String? = reader.readLine() do { // Parse line as frame stat value } }

Slide 96

Slide 96 text

val reader = BufferedReader(InputStreamReader(stdout)) reader.use { reader -> var line: String? = reader.readLine() do { // Parse line as frame stat value } } // Dump the latest stats val stdout = executeShellCommand( String.format("dumpsys gfxinfo %s", process) ) override fun stopIteration(): Bundle { } @orbycius override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { } // Make sure we found all the stats ... val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret

Slide 97

Slide 97 text

@orbycius // Parse line as frame stat value // Make sure we found all the stats

Slide 98

Slide 98 text

Slide 99

Slide 99 text

@orbycius // Parse line as frame stat value enum class JankStat private constructor( private val parsePattern: Pattern, private val groupIndex: Int, internal val type: Class<*>, optional: Boolean = false ) { TOTAL_FRAMES( Pattern.compile("\\s*Total frames rendered: (\\d+)"), 1, Int::class.java ) } Total frames rendered: ### Janky frames: ### (##.##%) 50th percentile: ##ms 90th percentile: ##ms 95th percentile: ##ms 99th percentile: ##ms Number Missed Vsync: # Number High input latency: # Number Slow UI thread: # Number Slow bitmap uploads: # Number Slow draw: # Number Frame deadline missed: #

Slide 100

Slide 100 text

@orbycius // Parse line as frame stat value for (stat in JankStat.values()) { val part: String? = stat.parse(line!!) if (part != null) { when { stat.type == Int::class.java -> { val stats = accumulatedStats[stat] as MutableList stats.add(Integer.valueOf(part)) } stat.type == Double::class.java -> { val stats = accumulatedStats[stat] as MutableList stats.add(java.lang.Double.valueOf(part)) } else -> // Shouldn't get here throw IllegalStateException("Unsupported JankStat type") } break } }

Slide 101

Slide 101 text

for (stat in JankStat.values()) { val part: String? = stat.parse(line!!) if (part != null) { when { stat.type == Int::class.java -> { val stats = accumulatedStats[stat] as MutableList stats.add(Integer.valueOf(part)) } stat.type == Double::class.java -> { val stats = accumulatedStats[stat] as MutableList stats.add(java.lang.Double.valueOf(part)) } else -> // Shouldn't get here throw IllegalStateException("Unsupported JankStat type") } break } } @orbycius private val accumulatedStats = EnumMap>(JankStat::class.java) // Parse line as frame stat value

Slide 102

Slide 102 text

@orbycius // Make sure we found all the stats

Slide 103

Slide 103 text

for (stat in JankStat.values()) { if (!stat.wasParsedSuccessfully() && !stat.isOptional) { Assert.fail(String.format("Failed to parse %s", stat.name)) } stat.reset() } val jankyFreeFrames = accumulatedStats[JankStat.TOTAL_FRAMES] as List @orbycius // Make sure we found all the stats for (stat in JankStat.values()) { if (!stat.wasParsedSuccessfully() && !stat.isOptional) { Assert.fail(String.format("Failed to parse %s", stat.name)) } stat.reset() } val jankyFreeFrames = accumulatedStats[JankStat.TOTAL_FRAMES] as List

Slide 104

Slide 104 text

for (stat in JankStat.values()) { if (!stat.wasParsedSuccessfully() && !stat.isOptional) { Assert.fail(String.format("Failed to parse %s", stat.name)) } stat.reset() } val jankyFreeFrames = accumulatedStats[JankStat.TOTAL_FRAMES] as List @orbycius // Make sure we found all the stats val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret

Slide 105

Slide 105 text

override fun getMetrics(): Bundle { } @orbycius override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { } override fun getMetrics(): Bundle { }

Slide 106

Slide 106 text

override fun getMetrics(): Bundle { } @orbycius override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { } val metrics = Bundle() // Retrieve the total number of frames val totals = accumulatedStats[JankStat.TOTAL_FRAMES] as List // Store avg, min and max of total frames metrics.putInt(GfxMonitor.KEY_AVG_TOTAL_FRAMES, computeAverage(totals)) metrics.putInt(GfxMonitor.KEY_MAX_TOTAL_FRAMES, Collections.max(totals)) metrics.putInt(GfxMonitor.KEY_MIN_TOTAL_FRAMES, Collections.min(totals)) ... return metrics

Slide 107

Slide 107 text

@orbycius class GfxMonitorImpl( private val instrumentation: Instrumentation, private val process: String ) : IMonitor { override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { } }

Slide 108

Slide 108 text

Slide 109

Slide 109 text

junit.framework.AssertionFailedError: Failed to parse NUM_FRAME_DEADLINE_MISSED for (stat in JankStat.values()) { if (!stat.wasParsedSuccessfully() && !stat.isOptional) { Assert.fail(String.format("Failed to parse %s", stat.name)) } stat.reset() } @orbycius for (stat in JankStat.values()) { if (!stat.wasParsedSuccessfully() && !stat.isOptional) { Assert.fail(String.format("Failed to parse %s", stat.name)) } stat.reset() } junit.framework.AssertionFailedError: Failed to parse NUM_FRAME_DEADLINE_MISSED

Slide 110

Slide 110 text

for (stat in JankStat.values()) { if (!stat.wasParsedSuccessfully() && !stat.isOptional) { Assert.fail(String.format("Failed to parse %s", stat.name)) } stat.reset() } junit.framework.AssertionFailedError: Failed to parse NUM_FRAME_DEADLINE_MISSED for (stat in JankStat.values()) { if (!stat.wasParsedSuccessfully() && !stat.isOptional) { Assert.fail(String.format("Failed to parse %s", stat.name)) } stat.reset() } @orbycius

Slide 111

Slide 111 text

enum class JankStat private constructor( private val parsePattern: Pattern, private val groupIndex: Int, internal val type: Class<*>, optional: Boolean = false ) { TOTAL_FRAMES( Pattern.compile("\\s*Total frames rendered: (\\d+)"), 1, Int::class.java ), NUM_FRAME_DEADLINE_MISSED( Pattern.compile("\\s*Number Frame deadline missed: (\\d+)"), 1, Int::class.java junit.framework.AssertionFailedError: Failed to parse NUM_FRAME_DEADLINE_MISSED @orbycius junit.framework.AssertionFailedError: Failed to parse NUM_FRAME_DEADLINE_MISSED enum class JankStat private constructor( private val parsePattern: Pattern, private val groupIndex: Int, internal val type: Class<*>, optional: Boolean = false ) { TOTAL_FRAMES( Pattern.compile("\\s*Total frames rendered: (\\d+)"), 1, Int::class.java ), NUM_FRAME_DEADLINE_MISSED( Pattern.compile("\\s*Number Frame deadline missed: (\\d+)"), 1, Int::class.java ); );

Slide 112

Slide 112 text

Slide 113

Slide 113 text

Slide 114

Slide 114 text

@orbycius class MyMonitor( private val instrumentation: Instrumentation, private val process: String ) : } GfxMonitorImpl {

Slide 115

Slide 115 text

@orbycius class MyMonitor( private val instrumentation: Instrumentation, private val process: String ) : } GfxMonitorImpl {

Slide 116

Slide 116 text

@orbycius class MyMonitor( private val instrumentation: Instrumentation, private val process: String ) : } IMonitor {

Slide 117

Slide 117 text

@orbycius How can we get other stats?

Slide 118

Slide 118 text

val jankyFreeFrames = accumulatedStats[JankStat.TOTAL_FRAMES] as List val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret @orbycius val jankyFreeFrames = accumulatedStats[JankStat.TOTAL_FRAMES] as List val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret

Slide 119

Slide 119 text

val jankyFreeFrames = accumulatedStats[JankStat.TOTAL_FRAMES] as List val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret @orbycius

Slide 120

Slide 120 text

Slide 121

Slide 121 text

val jankyFreeFrames = accumulatedStats[JankStat.TOTAL_FRAMES] as List val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret @orbycius Total frames rendered: ### Janky frames: ### (##.##%) 50th percentile: ##ms 90th percentile: ##ms 95th percentile: ##ms 99th percentile: ##ms Number Missed Vsync: # Number High input latency: # Number Slow UI thread: # Number Slow bitmap uploads: # Number Slow draw: # Number Frame deadline missed: # JankStat.TOTAL_FRAMES JankStat.NUM_JANKY JankStat.FRAME_TIME_50TH JankStat.FRAME_TIME_90TH JankStat.FRAME_TIME_95TH JankStat.FRAME_TIME_99TH JankStat.NUM_MISSED_VSYNC JankStat.NUM_HIGH_INPUT_LATENCY JankStat.NUM_SLOW_UI_THREAD JankStat.NUM_SLOW_BITMAP_UPLOADS JankStat.NUM_SLOW_DRAW JankStat.NUM_FRAME_DEADLINE_MISSED

Slide 122

Slide 122 text

@orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret

Slide 123

Slide 123 text

public class JankTestBase extends InstrumentationTestCase { protected void runTest() throws Throwable { ... Bundle results = monitor.stopIteration(); int numFrames = results.getInt("num-frames", 0); // Fail the test if we didn't get enough frames assertTrue(String.format( "Too few frames received. Monitor: %s, Expected: %d, Received: %d.", monitor.getClass().getSimpleName(), annotation.expectedFrames(), numFrames), numFrames >= annotation.expectedFrames()); ... } } @orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret public class JankTestBase extends InstrumentationTestCase { protected void runTest() throws Throwable { ... Bundle results = monitor.stopIteration(); int numFrames = results.getInt("num-frames", 0); // Fail the test if we didn't get enough frames assertTrue(String.format( "Too few frames received. Monitor: %s, Expected: %d, Received: %d.", monitor.getClass().getSimpleName(), annotation.expectedFrames(), numFrames), numFrames >= annotation.expectedFrames()); ... } }

Slide 124

Slide 124 text

public class JankTestBase extends InstrumentationTestCase { protected void runTest() throws Throwable { ... Bundle results = monitor.stopIteration(); int numFrames = results.getInt("num-frames", 0); // Fail the test if we didn't get enough frames assertTrue(String.format( "Too few frames received. Monitor: %s, Expected: %d, Received: %d.", monitor.getClass().getSimpleName(), annotation.expectedFrames(), numFrames), numFrames >= annotation.expectedFrames()); ... } } public class JankTestBase extends InstrumentationTestCase { protected void runTest() throws Throwable { ... Bundle results = monitor.stopIteration(); int numFrames = results.getInt("num-frames", 0); // Fail the test if we didn't get enough frames assertTrue(String.format( "Too few frames received. Monitor: %s, Expected: %d, Received: %d.", monitor.getClass().getSimpleName(), annotation.expectedFrames(), numFrames), numFrames >= annotation.expectedFrames()); ... } } @orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret

Slide 125

Slide 125 text

Slide 126

Slide 126 text

@orbycius Returns just one stat

Slide 127

Slide 127 text

@orbycius Returns just one stat Always checks >=

Slide 128

Slide 128 text

@orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret monitor

Slide 129

Slide 129 text

@orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret Janky frames: ### (##.##%) numFrames >= annotation.expectedFrames() monitor JankTestBase

Slide 130

Slide 130 text

@orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret Janky frames: ### (##.##%) 10.50% numFrames >= annotation.expectedFrames() numFrames monitor JankTestBase

Slide 131

Slide 131 text

@orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret Janky frames: ### (##.##%) 10.50% numFrames >= annotation.expectedFrames() 5.00% numFrames expectedFrames monitor JankTestBase

Slide 132

Slide 132 text

@orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret Janky frames: ### (##.##%) 10.50% numFrames >= annotation.expectedFrames() >= 5.00% numFrames expectedFrames monitor JankTestBase

Slide 133

Slide 133 text

@orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List val ret = Bundle() ret.putInt("num-frames", jankyFreeFrames[jankyFreeFrames.size - 1]) return ret Janky frames: ### (##.##%) 10.50% numFrames >= annotation.expectedFrames() >= 5.00% PASS numFrames expectedFrames monitor JankTestBase

Slide 134

Slide 134 text

@orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List val ret = Bundle() ret.putInt("num-frames", return ret Janky frames: ### (##.##%) 10.50% numFrames >= annotation.expectedFrames() >= 5.00% numFrames expectedFrames monitor JankTestBase jankyFreeFrames[jankyFreeFrames.size - 1]) 100 -

Slide 135

Slide 135 text

@orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List val ret = Bundle() ret.putInt("num-frames", return ret Janky frames: ### (##.##%) numFrames >= annotation.expectedFrames() >= 5.00% numFrames expectedFrames monitor JankTestBase jankyFreeFrames[jankyFreeFrames.size - 1]) 100 - 89.50%

Slide 136

Slide 136 text

@orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List val ret = Bundle() ret.putInt("num-frames", return ret Janky frames: ### (##.##%) numFrames >= annotation.expectedFrames() >= numFrames expectedFrames monitor JankTestBase jankyFreeFrames[jankyFreeFrames.size - 1]) 100 - 89.50% 95.00%

Slide 137

Slide 137 text

@orbycius val jankyFreeFrames = accumulatedStats[JankStat.NUM_JANKY] as List val ret = Bundle() ret.putInt("num-frames", return ret Janky frames: ### (##.##%) numFrames >= annotation.expectedFrames() >= numFrames expectedFrames monitor JankTestBase jankyFreeFrames[jankyFreeFrames.size - 1]) 100 - 89.50% 95.00% FAIL

Slide 138

Slide 138 text

@orbycius Percentiles

Slide 139

Slide 139 text

@orbycius 0 40 80 120 160 0 40 80 120 160 16.299 Average

Slide 140

Slide 140 text

@orbycius avg 99th perc 95th per 90th per 50th per

Slide 141

Slide 141 text

@orbycius avg 99th perc 95th per 90th per 50th per No sleep 4ms 7ms 5ms 5ms 5ms

Slide 142

Slide 142 text

@orbycius avg 99th perc 95th per 90th per 50th per No sleep 4ms 7ms 5ms 5ms 5ms Sleep 150ms 16ms 150ms 150ms 12ms 5ms

Slide 143

Slide 143 text

@orbycius avg 99th perc 95th per 90th per 50th per No sleep 4ms 7ms 5ms 5ms 5ms Sleep 150ms 16ms 150ms 150ms 12ms 5ms Sleep 150ms %20 4.5ms 150ms 5ms 5ms 5ms

Slide 144

Slide 144 text

@orbycius JankStat.TOTAL_FRAMES JankStat.NUM_JANKY JankStat.FRAME_TIME_50TH JankStat.FRAME_TIME_90TH JankStat.FRAME_TIME_95TH JankStat.FRAME_TIME_99TH JankStat.NUM_MISSED_VSYNC JankStat.NUM_HIGH_INPUT_LATENCY JankStat.NUM_SLOW_UI_THREAD JankStat.NUM_SLOW_BITMAP_UPLOADS JankStat.NUM_SLOW_DRAW JankStat.NUM_FRAME_DEADLINE_MISSED

Slide 145

Slide 145 text

@orbycius public class JankTestBase extends InstrumentationTestCase { }

Slide 146

Slide 146 text

@orbycius public class JankTestBase extends InstrumentationTestCase { }

Slide 147

Slide 147 text

@RunWith(AndroidJUnit4::class) class FifthTest { @orbycius @RunWith(AndroidJUnit4::class) class FifthTest {

Slide 148

Slide 148 text

private var monitor: IMonitor init { if (API_LEVEL_ACTUAL <= 22) { error("Not supported by current platform.") } else { monitor = PercentileMonitor( InstrumentationRegistry.getInstrumentation(), PACKAGE_NAME ) } } @RunWith(AndroidJUnit4::class) class FifthTest { @orbycius private var monitor: IMonitor init { if (API_LEVEL_ACTUAL <= 22) { error("Not supported by current platform.") } else { monitor = PercentileMonitor( InstrumentationRegistry.getInstrumentation(), PACKAGE_NAME ) } }

Slide 149

Slide 149 text

private var monitor: IMonitor init { if (API_LEVEL_ACTUAL <= 22) { error("Not supported by current platform.") } else { monitor = PercentileMonitor( InstrumentationRegistry.getInstrumentation(), PACKAGE_NAME ) } } @RunWith(AndroidJUnit4::class) class FifthTest { @orbycius @get:Rule val activityRule = ActivityTestRule(MainActivity::class.java)

Slide 150

Slide 150 text

@get:Rule var activityRule: ActivityTestRule = object : ActivityTestRule(MainActivity::class.java) { override fun beforeActivityLaunched() { monitor.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { val results = monitor.stopIteration() val percentile = results.getInt("percentilesValue", Integer.MAX_VALUE) TestCase.assertTrue( String.format( "Monitor: %s, Expected: %d, Received: %d.", monitor::class.java.simpleName, EXPECTED_MSECS, percentile ), percentile < EXPECTED_MSECS ) super.afterActivityFinished() } } @orbycius @get:Rule var activityRule: ActivityTestRule = object : ActivityTestRule(MainActivity::class.java) { override fun beforeActivityLaunched() { monitor.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { val results = monitor.stopIteration() val percentile = results.getInt("percentilesValue", Integer.MAX_VALUE) TestCase.assertTrue( String.format( "Monitor: %s, Expected: %d, Received: %d.", monitor::class.java.simpleName, EXPECTED_MSECS, percentile ), percentile < EXPECTED_MSECS ) super.afterActivityFinished() } }

Slide 151

Slide 151 text

@get:Rule var activityRule: ActivityTestRule = object : ActivityTestRule(MainActivity::class.java) { override fun beforeActivityLaunched() { monitor.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { val results = monitor.stopIteration() val percentile = results.getInt("percentilesValue", Integer.MAX_VALUE) TestCase.assertTrue( String.format( "Monitor: %s, Expected: %d, Received: %d.", monitor::class.java.simpleName, EXPECTED_MSECS, percentile ), percentile < EXPECTED_MSECS ) super.afterActivityFinished() } } @get:Rule var activityRule: ActivityTestRule = object : ActivityTestRule(MainActivity::class.java) { override fun beforeActivityLaunched() { monitor.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { val results = monitor.stopIteration() val percentile = results.getInt("percentilesValue", Integer.MAX_VALUE) TestCase.assertTrue( String.format( "Monitor: %s, Expected: %d, Received: %d.", monitor::class.java.simpleName, EXPECTED_MSECS, percentile ), percentile < EXPECTED_MSECS ) super.afterActivityFinished() } } @orbycius

Slide 152

Slide 152 text

@get:Rule var activityRule: ActivityTestRule = object : ActivityTestRule(MainActivity::class.java) { override fun beforeActivityLaunched() { monitor.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { val results = monitor.stopIteration() val percentile = results.getInt("percentilesValue", Integer.MAX_VALUE) TestCase.assertTrue( String.format( "Monitor: %s, Expected: %d, Received: %d.", monitor::class.java.simpleName, EXPECTED_MSECS, percentile ), percentile < EXPECTED_MSECS ) super.afterActivityFinished() } } @get:Rule var activityRule: ActivityTestRule = object : ActivityTestRule(MainActivity::class.java) { override fun beforeActivityLaunched() { monitor.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { val results = monitor.stopIteration() val percentile = results.getInt("percentilesValue", Integer.MAX_VALUE) TestCase.assertTrue( String.format( "Monitor: %s, Expected: %d, Received: %d.", monitor::class.java.simpleName, EXPECTED_MSECS, percentile ), percentile < EXPECTED_MSECS ) super.afterActivityFinished() } } @orbycius

Slide 153

Slide 153 text

Slide 154

Slide 154 text

@orbycius @Test @PerformanceTest( processName = PACKAGE_NAME, perfType = PerformanceTest.PerfType.AVG_FRAME_TIME_95TH, threshold = 18, assertionType = PerformanceTest.AssertionType.LESS_OR_EQUAL ) fun testSixth() { for (i in 0 until INNER_LOOP) { val appViews = UiScrollable(UiSelector().scrollable(true)) appViews.setAsVerticalList() appViews.scrollTextIntoView("This is item 24") appViews.scrollTextIntoView("This is item 1") } }

Slide 155

Slide 155 text

@orbycius val jankyFreeFrames = accumulatedStats[JankStat.FRAME_TIME_99TH] as List val ret = Bundle() ret.putInt("percentilesValue", percentileTimeInMs[percentileTimeInMs.size - 1]) return ret monitor

Slide 156

Slide 156 text

@orbycius override fun startIteration() { } override fun stopIteration(): Bundle { } override fun getMetrics(): Bundle { } override fun getMetrics(): Bundle { } val metrics = Bundle() // Retrieve the total number of frames val totals = accumulatedStats[JankStat.TOTAL_FRAMES] as List // Store avg, min and max of total frames metrics.putInt(GfxMonitor.KEY_AVG_TOTAL_FRAMES, computeAverage(totals)) metrics.putInt(GfxMonitor.KEY_MAX_TOTAL_FRAMES, Collections.max(totals)) metrics.putInt(GfxMonitor.KEY_MIN_TOTAL_FRAMES, Collections.min(totals)) ... return metrics val jankyFreeFrames = accumulatedStats[JankStat.FRAME_TIME_99TH] as List val ret = Bundle() ret.putInt("percentilesValue", percentileTimeInMs[percentileTimeInMs.size - 1]) return ret monitor

Slide 157

Slide 157 text

Slide 158

Slide 158 text

} @Retention(AnnotationRetention.RUNTIME) @Target(AnnotationTarget.FUNCTION) annotation class PerformanceTest( val processName: String = "", val perfType: PerfType val threshold: Int = Int.MAX_VALUE, val assertionType: AssertionType ) { @orbycius @Retention(AnnotationRetention.RUNTIME) @Target(AnnotationTarget.FUNCTION) annotation class PerformanceTest( val processName: String = "", val perfType: PerfType val threshold: Int = Int.MAX_VALUE, val assertionType: AssertionType ) { } ,

Slide 159

Slide 159 text

} @Retention(AnnotationRetention.RUNTIME) @Target(AnnotationTarget.FUNCTION) annotation class PerformanceTest( val processName: String = "", val perfType: PerfType val threshold: Int = Int.MAX_VALUE, val assertionType: AssertionType ) { @orbycius , enum class PerfType(val type: String) { TOTAL_FRAMES("gfx-avg-total-frames"), MAX_TOTAL_FRAMES("gfx-max-total-frames"), MIN_TOTAL_FRAMES("gfx-min-total-frames"), … } = PerfType.TOTAL_FRAMES

Slide 160

Slide 160 text

@Retention(AnnotationRetention.RUNTIME) @Target(AnnotationTarget.FUNCTION) annotation class PerformanceTest( val processName: String = "", val perfType: PerfType = PerfType.TOTAL_FRAMES, val threshold: Int = Int.MAX_VALUE, val assertionType: AssertionType ) { } @orbycius enum class PerfType(val type: String) { TOTAL_FRAMES("gfx-avg-total-frames"), MAX_TOTAL_FRAMES("gfx-max-total-frames"), MIN_TOTAL_FRAMES("gfx-min-total-frames"), … } enum class AssertionType(val type: Int) { LESS(0), LESS_OR_EQUAL(1), GREATER(2), GREATER_OR_EQUAL(3), EQUAL(4) } = AssertionType.LESS_OR_EQUAL

Slide 161

Slide 161 text

} open class ActivityPerfTestRule(activityClass: Class): ActivityTestRule(activityClass) { @orbycius open class ActivityPerfTestRule(activityClass: Class): ActivityTestRule(activityClass) { }

Slide 162

Slide 162 text

Slide 163

Slide 163 text

} open class ActivityPerfTestRule(activityClass: Class): ActivityTestRule(activityClass) { @orbycius private var monitor: IMonitor? = null private var annotation: PerformanceTest? = null init { if (API_LEVEL_ACTUAL <= 22) { error("Not supported by current platform.") } } override fun apply(base: Statement?, description: Description?): Statement { annotation = description?.getAnnotation(PerformanceTest::class.java) annotation?.let { monitor = PerfMonitor( InstrumentationRegistry.getInstrumentation(), it.processName ) } return super.apply(base, description) }

Slide 164

Slide 164 text

@orbycius open class ActivityPerfTestRule(activityClass: Class): ActivityTestRule(activityClass) { // ... }

Slide 165

Slide 165 text

override fun beforeActivityLaunched() { monitor?.startIteration() super.beforeActivityLaunched() } @orbycius open class ActivityPerfTestRule(activityClass: Class): ActivityTestRule(activityClass) { // ... } override fun beforeActivityLaunched() { monitor?.startIteration() super.beforeActivityLaunched() }

Slide 166

Slide 166 text

override fun beforeActivityLaunched() { monitor?.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { monitor?.let { val results = it.stopIteration() val res: Double = results?.get(annotation?.perfType?.type) as Double val assertion = when(annotation?.assertionType) { PerformanceTest.AssertionType.LESS -> res < annotation!!.threshold PerformanceTest.AssertionType.GREATER -> res > annotation!!.threshold … } TestCase.assertTrue("Error msg here.", assertion) } super.afterActivityFinished() } @orbycius open class ActivityPerfTestRule(activityClass: Class): ActivityTestRule(activityClass) { // ... } override fun beforeActivityLaunched() { monitor?.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { monitor?.let { val results = it.stopIteration() val res: Double = results?.get(annotation?.perfType?.type) as Double val assertion = when(annotation?.assertionType) { PerformanceTest.AssertionType.LESS -> res < annotation!!.threshold PerformanceTest.AssertionType.GREATER -> res > annotation!!.threshold … } TestCase.assertTrue("Error msg here.", assertion) } super.afterActivityFinished() }

Slide 167

Slide 167 text

override fun beforeActivityLaunched() { monitor?.startIteration() super.beforeActivityLaunched() } override fun afterActivityFinished() { monitor?.let { val results = it.stopIteration() val res: Double = results?.get(annotation?.perfType?.type) as Double val assertion = when(annotation?.assertionType) { PerformanceTest.AssertionType.LESS -> res < annotation!!.threshold PerformanceTest.AssertionType.GREATER -> res > annotation!!.threshold … } TestCase.assertTrue("Error msg here.", assertion) } super.afterActivityFinished() } override fun afterActivityFinished() { monitor?.let { val results = it.stopIteration() val res: Double = results?.get(annotation?.perfType?.type) as Double val assertion = when(annotation?.assertionType) { PerformanceTest.AssertionType.LESS -> res < annotation!!.threshold PerformanceTest.AssertionType.GREATER -> res > annotation!!.threshold … } TestCase.assertTrue("Error msg here.", assertion) } super.afterActivityFinished() } @orbycius open class ActivityPerfTestRule(activityClass: Class): ActivityTestRule(activityClass) { // ... }

Slide 168

Slide 168 text

Slide 169

Slide 169 text

Slide 170

Slide 170 text

@orbycius Memory Leaks

Slide 171

Slide 171 text

public class LeakActivity extends Activity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_leak); View button = findViewById(R.id.button); button.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View v) { startAsyncWork(); } }); } void startAsyncWork() { Runnable work = new Runnable() { @Override public void run() { SystemClock.sleep(20000); } }; new Thread(work).start(); } } @orbycius public class LeakActivity extends Activity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_leak); View button = findViewById(R.id.button); button.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View v) { startAsyncWork(); } }); } void startAsyncWork() { Runnable work = new Runnable() { @Override public void run() { SystemClock.sleep(20000); } }; new Thread(work).start(); } }

Slide 172

Slide 172 text

public class LeakActivity extends Activity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_leak); View button = findViewById(R.id.button); button.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View v) { startAsyncWork(); } }); } void startAsyncWork() { Runnable work = new Runnable() { @Override public void run() { SystemClock.sleep(20000); } }; new Thread(work).start(); } } public class LeakActivity extends Activity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_leak); View button = findViewById(R.id.button); button.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View v) { startAsyncWork(); } }); } void startAsyncWork() { Runnable work = new Runnable() { @Override public void run() { SystemClock.sleep(20000); } }; new Thread(work).start(); } } @orbycius

Slide 173

Slide 173 text

Slide 174

Slide 174 text

@orbycius class SeventhTest { @get:Rule var mainActivityActivityTestRule = ActivityTestRule(LeakActivity::class.java) @Test fun testLeaks() { onView(withId(R.id.button)).perform(click()) } }

Slide 175

Slide 175 text

@orbycius dependencies { ... debugImplementation 'com.squareup.leakcanary:leakcanary-android:1.6.3' androidTestImplementation 'com.squareup.leakcanary:leakcanary-android-instrumentation:1.6.3' }

Slide 176

Slide 176 text

@orbycius dependencies { ... debugImplementation 'com.squareup.leakcanary:leakcanary-android:1.6.3' androidTestImplementation 'com.squareup.leakcanary:leakcanary-android-instrumentation:1.6.3' } defaultConfig { ... testInstrumentationRunnerArgument "listener", "com.squareup.leakcanary.FailTestOnLeakRunListener" }

Slide 177

Slide 177 text

@orbycius class SeventhTest { @get:Rule var mainActivityActivityTestRule = ActivityTestRule(LeakActivity::class.java) @Test fun testLeaks() { onView(withId(R.id.button)).perform(click()) } }

Slide 178

Slide 178 text

@orbycius class SeventhTest { @get:Rule var mainActivityActivityTestRule = ActivityTestRule(LeakActivity::class.java) @Test fun testLeaks() { onView(withId(R.id.button)).perform(click()) } } Test failed because memory leaks were detected, see leak traces below. ###################################### In com.marcosholgado.performancetest.test:null:0. * com.marcosholgado.performancetest.LeakActivity has leaked:

Slide 179

Slide 179 text

@orbycius 538ms without listener

Slide 180

Slide 180 text

@orbycius 538ms without listener 21s 52ms with listener

Slide 181

Slide 181 text

adb shell am instrument -w \ > -e listener com.squareup.leakcanary.FailTestOnLeakRunListener \ > -e class com.marcosholgado.performancetest.SeventhTest#testLeaks \ > com.marcosholgado.performancetest.test/androidx.test.runner.AndroidJUnitRunner @orbycius adb shell am instrument -w \ > -e listener com.squareup.leakcanary.FailTestOnLeakRunListener \ > -e class com.marcosholgado.performancetest.SeventhTest#testLeaks \ > com.marcosholgado.performancetest.test/androidx.test.runner.AndroidJUnitRunner

Slide 182

Slide 182 text

Slide 183

Slide 183 text

@orbycius class SeventhTest { @get:Rule var mainActivityActivityTestRule = ActivityTestRule(LeakActivity::class.java) @Test fun testLeaks() { onView(withId(R.id.button)).perform(click()) } }

Slide 184

Slide 184 text

Slide 185

Slide 185 text

@orbycius @Retention(AnnotationRetention.RUNTIME) @Target(AnnotationTarget.FUNCTION) annotation class LeakTest

Slide 186

Slide 186 text

@orbycius @Retention(AnnotationRetention.RUNTIME) @Target(AnnotationTarget.FUNCTION) annotation class LeakTest class MyOtherLeakRunListener: FailTestOnLeakRunListener() { }

Slide 187

Slide 187 text

@orbycius @Retention(AnnotationRetention.RUNTIME) @Target(AnnotationTarget.FUNCTION) annotation class LeakTest class MyOtherLeakRunListener: FailTestOnLeakRunListener() { } override fun skipLeakDetectionReason(description: Description): String? { return if(description.getAnnotation(LeakTest::class.java) != null) null else "Skip Leak test" }

Slide 188

Slide 188 text

@orbycius Startup time

Slide 189

Slide 189 text

@orbycius Startup time Janky frames

Slide 190

Slide 190 text

@orbycius Startup time Janky frames Memory leaks

Slide 191

Slide 191 text

@orbycius Janky frames adb shell dumpsys gfxinfo

Slide 192

Slide 192 text

@orbycius Janky frames adb shell dumpsys gfxinfo adb shell dumpsys batterystats

Slide 193

Slide 193 text

@orbycius Janky frames adb shell dumpsys gfxinfo adb shell dumpsys batterystats adb shell dumpsys netstats

Slide 194

Slide 194 text

How to write and automate performance tests @Orbycius Marcos Holgado

Slide 195

Slide 195 text

How to write and automate performance tests @Orbycius Marcos Holgado https://github.com/marcosholgado/performance-test https://github.com/marcosholgado/dagger-playground