Measure What Matters with Jetpack Benchmark

Presented in DroidconLondon

Transcript

1. @bennegeek Measure What Matters with Jetpack Benchmark JB Lorenzo | #droidconUK
3. @bennegeek
4. @bennegeek
5. @bennegeek
6. @bennegeek Benchmark 1.0.0-beta01 October 9, 2019 alpha01 on May 7
7. @bennegeek Benchmark 1.0.0-rc01 October 23, 2019 alpha01 on May 7
8. @bennegeek From alpha06 to beta01 to rc01 * Run GC each warmup to reduce leak * Added lint dependency * Use additionalTestOutputDir * Fix undetected clock frequency
9. @bennegeek From alpha02 to alpha06 * Auto disables test coverage * AndroidBenchmarkRunner by default * Module structure changed to decouple JUnit4 * Change in package name * Documentation for CI integration
10. @bennegeek Documentation for CI Integration https://developer.android.com/studio/profile/run-benchmarks-in-ci * Guidelines for benchmark without Gradle * Self-instrumented APK * Separate app and test APKs * Guidelines for CI * Important flags (e.g. lock clocks, json output)
11. @bennegeek Setting up Android Studio 3.5 Beta npw.benchmark.template.module=true
12. @bennegeek Setting up
13. @bennegeek Setting up your.package.name.
14. @bennegeek Setting up dependencies { androidTestImplementation “androidx.benchmark:benchmark:1.0.0-alpha06" }
15. @bennegeek Setting up <application android:debuggable="false" tools:ignore="HardcodedDebugMode" tools:replace="android:debuggable"/>
16. @bennegeek Let’s measure!
17. @bennegeek Let’s measure! @Test fun benchmarkGsonWithSample1() { benchmarkRule.measureRepeated { gsonParser.parseSample1() } }
18. @bennegeek Let’s measure! (UI) @UiThreadTest @Test fun homeList() { var activity = activityRule.activity val fragment = activity.supportFragmentManager.fragments .first { it is HomepageAdsListFragment } as Homepage AdsListFragment val recyclerView = fragment.mainList benchmarkRule.measureRepeated { recyclerView.scrollBy(0, maxScroll) } }
19. @bennegeek @mootazltaief
20. @bennegeek @mootazltaief
21. @bennegeek Firebase Test Lab Amazon EC2 + Emulator Your Own Device Farm
22. @bennegeek Firebase Test Lab Amazon EC2 + Emulator Your Own Device Farm
23. @bennegeek Firebase Test Lab Amazon EC2 + Emulator Your Own Device Farm
24. @bennegeek Amazon EC2 + Emulator AWS: * Does not allow nested VM * Need to run bare-Metal instance to run emulator Emulator: * Not advised for benchmarks Other cloud providers: * has support for nested VM
25. @bennegeek Integrating in Gitlab CI // benchmark/build.gradle apply plugin: androidx.benchmark // if device/emulator is connected to CI > ./gradlew benchmark:connectedCheck
26. @bennegeek Integrating in Gitlab CI { "context": { "build": { "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500,
27. @bennegeek Integrating in Gitlab CI { "context": { "build": { "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500,
28. @bennegeek Integrating in Gitlab CI { "context": { "build": { "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500,
29. @bennegeek Integrating in Gitlab CI { "context": { "build": { "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500, 95250, ...
30. @bennegeek Integrating in Gitlab CI { "context": { "build": { "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500,
31. @bennegeek Benchmark Warnings
32. @bennegeek Benchmark Warnings
33. @bennegeek Benchmark Warnings
34. @bennegeek Benchmark Warnings
35. @bennegeek Benchmark Warnings
36. @bennegeek Benchmark Warnings
37. @bennegeek Benchmark Warnings
38. @bennegeek Benchmark Warnings
39. @bennegeek Benchmark Warnings
40. @bennegeek Benchmark Warnings
41. @bennegeek Firebase Test Lab Amazon EC2 + Emulator Your Own Device Farm
42. @bennegeek Firebase Test Lab Amazon EC2 + Emulator Your Own Device Farm
43. @bennegeek Firebase Test Lab Integration with CI Firebase Test Lab * Problems when running test with mocked network * $5/device/hour after Free tier * Need to configure gcloud in CI * Fetch results in google cloud storage
44. @bennegeek Firebase Test Lab Integration with CI Firebase Test Lab How to get the json output from test lab?
45. @bennegeek Gitlab CI
46. @bennegeek Gitlab CI
47. @bennegeek Gitlab CI #!/bin/sh cat benchmark/build/benchmark_reports/*/*.json | python -c ''' import sys, json; benchmarks = json.load(sys.stdin)["benchmarks"] medians = {v["name"] : v["metrics"]["timeNs"]["median"] for v in benchmarks} … '''
48. @bennegeek Gitlab CI #!/bin/sh cat benchmark/build/benchmark_reports/*/*.json | python -c ''' import sys, json; benchmarks = json.load(sys.stdin)["benchmarks"] medians = {v["name"] : v["metrics"]["timeNs"]["median"] for v in benchmarks} … '''
49. @bennegeek Gitlab CI #!/bin/sh cat benchmark/build/benchmark_reports/*/*.json | python -c ''' import sys, json; benchmarks = json.load(sys.stdin)["benchmarks"] medians = {v["name"] : v["metrics"]["timeNs"]["median"] for v in benchmarks} … '''
50. @bennegeek Gitlab CI #!/bin/sh cat benchmark/build/benchmark_reports/*/*.json | python -c ''' import sys, json; benchmarks = json.load(sys.stdin)["benchmarks"] medians = {v["name"] : v["metrics"]["timeNs"]["median"] for v in benchmarks} … '''
51. @bennegeek Gitlab CI PreviousBenchmark = 138375 Threshold = 0.10 if medians["benchmarkJacksonWithSample1"] > (PreviousBenchmark * Threshold): print("benchmarkJacksonWithSample1 is beyond threshold") exit(1)
52. @bennegeek Output
53. @bennegeek Output
54. @bennegeek Output
55. @bennegeek Next Steps * Separate app into modules * Mock more data for benchmarking * Try it out yourself
56. @bennegeek Next Steps * Separate app into modules * Mock more data for benchmarking * Try it out yourself * Visit the issue tracker * File bugs
57. @bennegeek tech.olx.com Example project (No OLX code): http://github.com/mootazltaief/ android-json-benchmark
58. @bennegeek tech.olx.com https://issuetracker.google.com/ issues?q=componentid: 585351%20status:open
60. @bennegeek tech.olx.com
61. @bennegeek tech.olx.com https://medium.com/androiddevelopers/fighting-regressions-with-benchmarks-in-ci-6ea9a14b5c71
64. Berlin Buenos Aires Lisbon New Delhi Poznan Dubai
65. @bennegeek Thank you! Questions? Get in touch with us olxgroup.com tech.olx.com olxtechberlin jb.lorenzo@olx.com jblorenzo JB Lorenzo | #DroidconUK