Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Jetpack Benchmark - Droidcon RO 2019

JB Lorenzo
October 04, 2019
68

Jetpack Benchmark - Droidcon RO 2019

JB Lorenzo

October 04, 2019
Tweet

Transcript

  1. @bennegeek From alpha02 to alpha06 * Auto disables test coverage

    * AndroidBenchmarkRunner by default * Module structure changed to decouple JUnit4 * Change in package name * Documentation for CI integration
  2. @bennegeek Documentation for CI Integration https://developer.android.com/studio/profile/run-benchmarks-in-ci * Guidelines for benchmark

    without Gradle * Self-instrumented APK * Separate app and test APKs * Guidelines for CI * Important flags (e.g. lock clocks, json output)
  3. @bennegeek Amazon EC2 + Emulator AWS: * Does not allow

    nested VM * Need to run bare-Metal instance to run emulator Emulator: * Not advised for benchmarks Other cloud providers: * has support for nested VM
  4. @bennegeek Integrating in Gitlab CI // benchmark/build.gradle apply plugin: androidx.benchmark

    // if device/emulator is connected to CI > ./gradlew benchmark:connectedCheck
  5. @bennegeek Integrating in Gitlab CI { "context": { "build": {

    "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500,
  6. @bennegeek Integrating in Gitlab CI { "context": { "build": {

    "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500,
  7. @bennegeek Integrating in Gitlab CI { "context": { "build": {

    "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500,
  8. @bennegeek Integrating in Gitlab CI { "context": { "build": {

    "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500, 95250, ...
  9. @bennegeek Integrating in Gitlab CI { "context": { "build": {

    "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500,
  10. @bennegeek Firebase Test Lab Integration with CI Firebase Test Lab

    * Problems when running test with mocked network * $5/device/hour after Free tier * Need to configure gcloud in CI * Fetch results in google cloud storage
  11. @bennegeek Firebase Test Lab Integration with CI Firebase Test Lab

    How to get the json output from test lab?
  12. @bennegeek Gitlab CI #!/bin/sh cat benchmark/build/benchmark_reports/*/*.json | python -c '''

    import sys, json; benchmarks = json.load(sys.stdin)["benchmarks"] medians = {v["name"] : v["metrics"]["timeNs"]["median"] for v in benchmarks} … '''
  13. @bennegeek Gitlab CI #!/bin/sh cat benchmark/build/benchmark_reports/*/*.json | python -c '''

    import sys, json; benchmarks = json.load(sys.stdin)["benchmarks"] medians = {v["name"] : v["metrics"]["timeNs"]["median"] for v in benchmarks} … '''
  14. @bennegeek Gitlab CI #!/bin/sh cat benchmark/build/benchmark_reports/*/*.json | python -c '''

    import sys, json; benchmarks = json.load(sys.stdin)["benchmarks"] medians = {v["name"] : v["metrics"]["timeNs"]["median"] for v in benchmarks} … '''
  15. @bennegeek Gitlab CI #!/bin/sh cat benchmark/build/benchmark_reports/*/*.json | python -c '''

    import sys, json; benchmarks = json.load(sys.stdin)["benchmarks"] medians = {v["name"] : v["metrics"]["timeNs"]["median"] for v in benchmarks} … '''
  16. @bennegeek Gitlab CI PreviousBenchmark = 138375 Threshold = 0.10 if

    medians["benchmarkJacksonWithSample1"] > (PreviousBenchmark * Threshold): print("benchmarkJacksonWithSample1 is beyond threshold") exit(1)
  17. @bennegeek Next Steps * Separate app into modules * Mock

    more data for benchmarking * Try it out yourself
  18. @bennegeek Next Steps * Separate app into modules * Mock

    more data for benchmarking * Try it out yourself * Visit the issue tracker * File bugs
  19. @bennegeek Multumesc! vreo întrebare? Get in touch with us olxgroup.com

    tech.olx.com olxtechberlin [email protected] jblorenzo JB Lorenzo | #DroidconRO19