Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Measure What Matters with Jetpack Benchmark

JB Lorenzo
October 24, 2019

Measure What Matters with Jetpack Benchmark

Presented in DroidconLondon

JB Lorenzo

October 24, 2019

Transcript

65 slides
  1. @bennegeek From alpha06 to beta01 to rc01 * Run GC

    each warmup to reduce leak * Added lint dependency * Use additionalTestOutputDir * Fix undetected clock frequency
  2. @bennegeek From alpha02 to alpha06 * Auto disables test coverage

    * AndroidBenchmarkRunner by default * Module structure changed to decouple JUnit4 * Change in package name * Documentation for CI integration
  3. @bennegeek Documentation for CI Integration https://developer.android.com/studio/profile/run-benchmarks-in-ci * Guidelines for benchmark

    without Gradle * Self-instrumented APK * Separate app and test APKs * Guidelines for CI * Important flags (e.g. lock clocks, json output)
  4. @bennegeek Let’s measure! (UI) @UiThreadTest @Test fun homeList() { var

    activity = activityRule.activity val fragment = activity.supportFragmentManager.fragments .first { it is HomepageAdsListFragment } as Homepage AdsListFragment val recyclerView = fragment.mainList benchmarkRule.measureRepeated { recyclerView.scrollBy(0, maxScroll) } }
  5. @bennegeek Amazon EC2 + Emulator AWS: * Does not allow

    nested VM * Need to run bare-Metal instance to run emulator Emulator: * Not advised for benchmarks Other cloud providers: * has support for nested VM
  6. @bennegeek Integrating in Gitlab CI // benchmark/build.gradle apply plugin: androidx.benchmark

    // if device/emulator is connected to CI > ./gradlew benchmark:connectedCheck
  7. @bennegeek Integrating in Gitlab CI { "context": { "build": {

    "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500,
  8. @bennegeek Integrating in Gitlab CI { "context": { "build": {

    "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500,
  9. @bennegeek Integrating in Gitlab CI { "context": { "build": {

    "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500,
  10. @bennegeek Integrating in Gitlab CI { "context": { "build": {

    "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500, 95250, ...
  11. @bennegeek Integrating in Gitlab CI { "context": { "build": {

    "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500,
  12. @bennegeek Firebase Test Lab Integration with CI Firebase Test Lab

    * Problems when running test with mocked network * $5/device/hour after Free tier * Need to configure gcloud in CI * Fetch results in google cloud storage
  13. @bennegeek Firebase Test Lab Integration with CI Firebase Test Lab

    How to get the json output from test lab?
  14. @bennegeek Gitlab CI #!/bin/sh cat benchmark/build/benchmark_reports/*/*.json | python -c '''

    import sys, json; benchmarks = json.load(sys.stdin)["benchmarks"] medians = {v["name"] : v["metrics"]["timeNs"]["median"] for v in benchmarks} … '''
  15. @bennegeek Gitlab CI #!/bin/sh cat benchmark/build/benchmark_reports/*/*.json | python -c '''

    import sys, json; benchmarks = json.load(sys.stdin)["benchmarks"] medians = {v["name"] : v["metrics"]["timeNs"]["median"] for v in benchmarks} … '''
  16. @bennegeek Gitlab CI #!/bin/sh cat benchmark/build/benchmark_reports/*/*.json | python -c '''

    import sys, json; benchmarks = json.load(sys.stdin)["benchmarks"] medians = {v["name"] : v["metrics"]["timeNs"]["median"] for v in benchmarks} … '''
  17. @bennegeek Gitlab CI #!/bin/sh cat benchmark/build/benchmark_reports/*/*.json | python -c '''

    import sys, json; benchmarks = json.load(sys.stdin)["benchmarks"] medians = {v["name"] : v["metrics"]["timeNs"]["median"] for v in benchmarks} … '''
  18. @bennegeek Gitlab CI PreviousBenchmark = 138375 Threshold = 0.10 if

    medians["benchmarkJacksonWithSample1"] > (PreviousBenchmark * Threshold): print("benchmarkJacksonWithSample1 is beyond threshold") exit(1)
  19. @bennegeek Next Steps * Separate app into modules * Mock

    more data for benchmarking * Try it out yourself
  20. @bennegeek Next Steps * Separate app into modules * Mock

    more data for benchmarking * Try it out yourself * Visit the issue tracker * File bugs
  21. @bennegeek Thank you! Questions? Get in touch with us olxgroup.com

    tech.olx.com olxtechberlin [email protected] jblorenzo JB Lorenzo | #DroidconUK

More Decks by JB Lorenzo

Other Decks in Programming