App Performance Benchmark in Android

By :
- Andri Ginting, Mobile Engineer, Gojek
- Javent Lienata, Mobile Engineer, Gojek

Live: https://youtu.be/NYGNU8G_hxI?t=5603

Transcript

1. App performance benchmark Way of benchmarking your android app by : Javent - Andri
2. Introduction Andri Suranta Ginting Mobile Engineer @ Gojek
3. Introduction Javent Lienata Mobile Engineer @ Gojek
4. Disclaimer Everything stated here is opinionated and expressed without the influence of any instance
5. Prerequisite • You know how to operate with git • You know how to develop an Android App Using Android Studio and Kotlin • You know how to build multi module app on Android • You know how to write instrumentation tests on Android Studio • Your know how to integrate Android project into Google Firebase
6. Agenda 1. Introduction ◦ What is benchmark? ◦ Why do we need to benchmark? 2. Benchmarking Kotlin’s code ◦ measureTimeMillis/measureNanoTime 3. Jetpack Benchmark ◦ Quick Start ◦ Project Setup ◦ Example ▪ Linear/Relative/Constraint on flat hierarchy layout ◦ Additional Configuration 4. Firebase Performance Monitoring ◦ Introduction Firebase Performance Monitoring ◦ Metrics, attribute, addTrace ◦ Hidden Cost
7. What is Benchmark? Evaluate or check (something) by comparison with a standard. Google Translate
8. Introduction Why do we need to benchmark?
9. I prefer to call the comparison UX vs DX UX = User Xperience (Optimized) DX = Developer Xperience (Maintainable)
10. https://bit.ly/37RO2kS
11. https://bit.ly/37RO2kS
13. https://bit.ly/3qG70n8
14. https://bit.ly/3qG70n8
16. So why do we need Benchmark? To determine whether the result of optimisation is worth the cost of maintainability
17. Benchmarking kotlin code measureTimeMillis/measureNanoTime
19. Let’s say we want to benchmark these 2 functions to know which one is more optimized
20. We can just wrap the functions with measureTimeMillis {} and put the code we want to test inside. measureTimeMillis {} will output the time needed to execute the code inside the curly braces, in millis
21. We can create a helper/utils function so we can easily log the measureTimeMillis()
22. But how do I handle warmup and measure repeated benchmark on Android without changing the implementation code?
23. Jetpack Benchmark https://developer.android.com/jetpack/androidx/releases/benchmark Quick Start
24. Jetpack Benchmark is a Jetpack Library that measures the performance of Android app code. It helps reduce measurement errors that are easy to do in general, and is integrated into Android Studio. It is standard JUnit instrumentation tests that run on an Android device, and use a rule provided by the library to perform the measuring and reporting.
25. Quick Start Add the library to your module’s build.gradle file: project_root/module_dir/build.gradle
26. Quick Start set jvmTarget to 1.8 in your module’s build.gradle file: project_root/module_dir/build.gradle
27. Quick Start To disable debugging in the test manifest, update your <application> element to force-disable debugging temporarily as follows: project_root/module_dir/src/androidTest/AndroidManifest.xml
28. Quick Start To add your benchmark, add an instance of BenchmarkRule in a test file in the androidTest directory. For more information on writing benchmarks, see Write a benchmark. The following code snippet shows how to add a benchmark to a JUnit test:
30. Jetpack Benchmark https://developer.android.com/jetpack/androidx/releases/benchmark Project Setup
31. Setup jetpack-benchmark on your project: - Create new module for benchmark We need to force the application debuggable to false, by creating a new module we can isolate the custom configuration just to be used on benchmark module - Separate feature module from app We need our benchmark module to depend on the feature we want to benchmark. And it cannot depend on app/base module.
32. Project Setup I assume you already have this kind of structure where all the features are not on app module anymore
33. Project Setup Create a new benchmark module. It’s available on android studio 3.6
34. Project Setup Now we have 3 modules
35. Project Setup Open up your :benchmark module build.gradle and put this code to enable running benchmark on emulator
36. Project Setup we want to benchmark feature_example so we need it as a dependency on our benchmark module
37. Jetpack Benchmark https://github.com/hugosvent/jetpack-benchmark-sample Examples
38. Example 1 Let’s say we want to benchmark the time needed to inflate this 3 xml layout
39. Example 1 And we have MainActivity on feature_example
40. Example 1 We add the benchmark into androidTest in benchmark module almost like when doing Instrumentation Test
41. Example 1 We add the benchmark into androidTest in benchmark module almost like when doing Instrumentation Test
42. Example 1 Then left click and run the test on your connected device whether it’s emulator or real device
43. Example 1 The result will be shown at the test output
44. Example 1 The result will be shown at the test output
45. Example 1 The result will be shown at the test output
46. Let’s look at this repo for another example https://github.com/hugosvent/jetpack-benchmark-sample
47. Jetpack Benchmark https://developer.android.com/studio/profile/benchmark Additional Configuration
48. Enable json output in your gradle.properties enable additionalTestOutput so we can run the test from terminal to have .json output as a report for the benchmark
49. Enable json output in your benchmark module’s build.gradle add androidx.benchmark plugin
50. Enable json output when you run ./gradlew moduleName:connectedCheck for example on our module it will be ./gradlew benchmark:connectedCheck the results will be generated at $modulePath/build/outputs/connected _android_test_additional_output
52. Clock Stability Clocks on mobile devices dynamically change from high state (for performance) to low state (to save power, or when the device gets hot). These varying clocks can make your benchmark numbers vary widely, so the library provides ways to deal with this issue.
53. Configuration Errors The library detects the following conditions to ensure your project and environment are set up for release-accurate performance: - Debuggable is set to false. - A physical device, not an emulator, is being used. - Clocks are locked if the device is rooted. - Sufficient battery level on device. If any of the above checks fail, the benchmark will throw an error to discourage inaccurate measurements.
54. Jetpack Benchmark https://developer.android.com/studio/profile/run-benchmarks-in-ci Running it on CI
55. bit.ly/jetpackBenchmarkDroidconAPAC2020
56. Firebase Performance Monitoring https://firebase.google.com/docs/perf-mon
57. • Measure App startup time • Measure background/foreground process • HTTP/S network request • Rendering data per screen • Metrics by country, device, app version, and OS level
59. Measure App start time
60. HTTP/S network request
61. Filter by attribute
62. Setup Firebase Performance Monitoring on your project: dependencies { // Import the BoM for the Firebase platform implementation platform('com.google.firebase:firebase-bom:26.7.0') // When using the BoM, you don't specify versions in Firebase library dependencies implementation 'com.google.firebase:firebase-perf-ktx' }
63. Setup Firebase Performance Monitoring on your project: apply plugin: 'com.android.application' apply plugin: 'com.google.gms.google-services' // Apply the Performance Monitoring plugin apply plugin: 'com.google.firebase.firebase-perf' android { // ... }
64. Enabled log <application> <meta-data android:name="firebase_performance_logcat_enabled" android:value="true" /> </application>
65. How to create log to trace how long our codes run in app?
66. FirebasePerformance.getInstance() .newTrace("droidjam_trace") .apply { start() } Custom trace Firebase Instance Rules to follow: - Can have multiple custom traces - More than one custom trace can run at the same time - No trailing whitespace and max length is 100 chars
67. @AddTrace("droidjam_trace") fun myBoringFunction() { println("Hi droidjamers...") } Custom trace Annotation Process
68. Firebase.performance.newTrace("droidjam_trace").trace { // Update scenario. putAttribute("experiment", "A") // Reading scenario. val experimentValue = getAttribute("experiment") // Delete scenario. removeAttribute("experiment") // Read attributes. val traceAttributes = this.attributes } Attribute String value to help filter and segment data Notes: - Max length is 32 chars - Can record up to 5 custom attributes - Don’t put personal information (ex: email) Default Attribute - App version - Country - OS level - Device - Radio - Carrier - MIME type (network request)
69. How to create log to trace how many times our codes run in app?
70. val trace = FirebasePerformance.getInstance().newTrace("droidjam_trace") trace.start() // code that we want to trace (and log custom metrics) val item = cache.fetch("item") if (item != null) { trace.incrementMetric("item_cache_hit", 1) } else { trace.incrementMetric("item_cache_miss", 1) } trace.stop() Metric Int value to measured over time Notes: - Max length is 100 chars - Can record up to 32 metrics include default Duration metric
71. What about trace suspend function??
72. private suspend fun myBoringAsyncFunction() : Int { val result = 1 + 1 delay(100L) println("something to see here: $result") return result } @AddTrace("droidjam_trace") suspend fun myBoringAsyncFunction() : Int { ... }
73. inline fun <E> trace(name : String, block: (Trace) -> E): E { val trace = startTrace(name) //creates & starts a new Trace return try { block(trace) } finally { trace.stop() } }
74. inline fun <E> trace(name : String, block: (Trace) -> E): E { val trace = startTrace(name) //creates & starts a new Trace return try { block(trace) } finally { trace.stop() } }
75. inline fun <E> trace(name : String, block: (Trace) -> E): E { val trace = startTrace(name) //creates & starts a new Trace return try { block(trace) instrumented code can append information to the Trace } finally { trace.stop() } }
76. inline fun <E> trace(name : String, block: (Trace) -> E): E { val trace = startTrace(name) //creates & starts a new Trace return try { block(trace) } finally { // we stop the trace, and return value produced by block trace.stop() } }
77. suspend fun myBoringFunction() : Int { return trace("droidjam_trace") { myBoringAsyncFunction() } } private suspend fun myBoringAsyncFunction() : Int { val result = 1 + 1 delay(100L) println("something to see here: $result") return result } inline fun <E> trace(name : String, block: (Trace) -> E): E { val trace = startTrace(name) //creates & starts a new Trace return try { block(trace) } finally { trace.stop() } }
78. Hidden Cost
79. APK Size Firebase Performance Monitoring brings ~0.5mb increase to your APK download size
80. Method Count Firebase Performance Monitoring will add from 4k to ~10K
81. Solution: Using Proguard/R8/shrinking tool to reduce the downside
82. Summary - Firebase Performance Monitoring offers an easy way to monitor the performance of an app. - Some metrics has been added and monitored automatically - Analyzed data on Firebase Console
83. Jetpack Benchmark https://developer.android.com/studio/profile/run-benchmarks-in-ci Running it on CI
85. Installing Cirrus-CI https://cirrus-ci.org/guide/quick-start/
87. CI script Add .cirrus.yml file into your project
88. CI script Add .cirrus.yml file into your project v2020_06_25-08-51-15 is the last working docker image that I’ve tried, the images after it uses Jdk11 which crashed the test
89. CI script Add .cirrus.yml file into your project
90. CI script Add .cirrus.yml file into your project
91. CI script Add .cirrus.yml file into your project
92. CI script Add .cirrus.yml file into your project
93. CI script Add .cirrus.yml file into your project
94. Configuration for Json output
97. Additional Resources Jetpack Benchmark • Official Documentation https://developer.android.com/studio/profile/benchmark • Run benchmarks on CI https://developer.android.com/studio/profile/run-benchmarks-in-ci • Mindorks Blog https://blog.mindorks.com/improving-android-app-performance-with-benchmarking • droidcon Berlin 2019 Jetpack Benchmark by JB Lorenzo https://www.droidcon.com/media-detail?video=352671812 • droidcon APAC 2020 Jetpack Benchmark https://www.droidcon.com/media-detail?video=491049801 Firebase Performance Monitoring • Product Page https://firebase.google.com/products/performance • Getting Started https://firebase.google.com/docs/perf-mon/get-started-android?hl=id • Custom Code Tracing https://firebase.google.com/docs/perf-mon/custom-code-traces?platform=android • Firebase Developer Medium https://medium.com/firebase-developers/tracking-performance-in-kotlin-suspending-functions-c 81c01f87c92