Jetpack Benchmark - Droidcon RO 2019

Transcript

1. @bennegeek Measure What Matters with Jetpack Benchmark JB Lorenzo | #DroidconRO19
3. @bennegeek
4. @bennegeek
5. @bennegeek
6. @bennegeek Benchmark 1.0.0-alpha06 September 18, 2019 alpha01 on May 7
7. @bennegeek From alpha02 to alpha06 * Auto disables test coverage * AndroidBenchmarkRunner by default * Module structure changed to decouple JUnit4 * Change in package name * Documentation for CI integration
8. @bennegeek Documentation for CI Integration https://developer.android.com/studio/profile/run-benchmarks-in-ci * Guidelines for benchmark without Gradle * Self-instrumented APK * Separate app and test APKs * Guidelines for CI * Important flags (e.g. lock clocks, json output)
9. @bennegeek Setting up Android Studio 3.5 Beta npw.benchmark.template.module=true
10. @bennegeek Setting up
11. @bennegeek Setting up your.package.name.
12. @bennegeek Setting up dependencies { androidTestImplementation “androidx.benchmark:benchmark:1.0.0-alpha06" }
13. @bennegeek Setting up <application android:debuggable="false" tools:ignore="HardcodedDebugMode" tools:replace="android:debuggable"/>
14. @bennegeek Let’s measure!
15. @bennegeek @mootazltaief
16. @bennegeek @mootazltaief
17. @bennegeek Firebase Test Lab Amazon EC2 + Emulator Your Own Device Farm
18. @bennegeek Firebase Test Lab Amazon EC2 + Emulator Your Own Device Farm
19. @bennegeek Firebase Test Lab Amazon EC2 + Emulator Your Own Device Farm
20. @bennegeek Amazon EC2 + Emulator AWS: * Does not allow nested VM * Need to run bare-Metal instance to run emulator Emulator: * Not advised for benchmarks Other cloud providers: * has support for nested VM
21. @bennegeek Integrating in Gitlab CI // benchmark/build.gradle apply plugin: androidx.benchmark // if device/emulator is connected to CI > ./gradlew benchmark:connectedCheck
22. @bennegeek Integrating in Gitlab CI { "context": { "build": { "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500,
23. @bennegeek Integrating in Gitlab CI { "context": { "build": { "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500,
24. @bennegeek Integrating in Gitlab CI { "context": { "build": { "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500,
25. @bennegeek Integrating in Gitlab CI { "context": { "build": { "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500, 95250, ...
26. @bennegeek Integrating in Gitlab CI { "context": { "build": { "device": "generic_x86_64", "fingerprint": "google/sdk_gphone_x86_64/generic_x86_64:9/PSR1.180720.075/5124027:userdebug/dev-keys", "model": "Android SDK built for x86_64", "version": { "sdk": 28 } }, "cpuLocked": false, "sustainedPerformanceModeEnabled": false }, "benchmarks": [ { "name": "EMULATOR_UNLOCKED_benchmarkJacksonWithSample1", "className": "benchmark.JsonBenchmark", "metrics": { "timeNs": { "minimum": 94500, "maximum": 692250, "median": 138375, "runs": [ 96500,
27. @bennegeek Benchmark Warnings
28. @bennegeek Benchmark Warnings
29. @bennegeek Benchmark Warnings
30. @bennegeek Benchmark Warnings
31. @bennegeek Benchmark Warnings
32. @bennegeek Benchmark Warnings
33. @bennegeek Benchmark Warnings
34. @bennegeek Benchmark Warnings
35. @bennegeek Benchmark Warnings
36. @bennegeek Benchmark Warnings
37. @bennegeek Firebase Test Lab Amazon EC2 + Emulator Your Own Device Farm
38. @bennegeek Firebase Test Lab Amazon EC2 + Emulator Your Own Device Farm
39. @bennegeek Firebase Test Lab Integration with CI Firebase Test Lab * Problems when running test with mocked network * $5/device/hour after Free tier * Need to configure gcloud in CI * Fetch results in google cloud storage
40. @bennegeek Firebase Test Lab Integration with CI Firebase Test Lab How to get the json output from test lab?
41. @bennegeek Gitlab CI
42. @bennegeek Gitlab CI
43. @bennegeek Gitlab CI #!/bin/sh cat benchmark/build/benchmark_reports/*/*.json | python -c ''' import sys, json; benchmarks = json.load(sys.stdin)["benchmarks"] medians = {v["name"] : v["metrics"]["timeNs"]["median"] for v in benchmarks} … '''
44. @bennegeek Gitlab CI #!/bin/sh cat benchmark/build/benchmark_reports/*/*.json | python -c ''' import sys, json; benchmarks = json.load(sys.stdin)["benchmarks"] medians = {v["name"] : v["metrics"]["timeNs"]["median"] for v in benchmarks} … '''
45. @bennegeek Gitlab CI #!/bin/sh cat benchmark/build/benchmark_reports/*/*.json | python -c ''' import sys, json; benchmarks = json.load(sys.stdin)["benchmarks"] medians = {v["name"] : v["metrics"]["timeNs"]["median"] for v in benchmarks} … '''
46. @bennegeek Gitlab CI #!/bin/sh cat benchmark/build/benchmark_reports/*/*.json | python -c ''' import sys, json; benchmarks = json.load(sys.stdin)["benchmarks"] medians = {v["name"] : v["metrics"]["timeNs"]["median"] for v in benchmarks} … '''
47. @bennegeek Gitlab CI PreviousBenchmark = 138375 Threshold = 0.10 if medians["benchmarkJacksonWithSample1"] > (PreviousBenchmark * Threshold): print("benchmarkJacksonWithSample1 is beyond threshold") exit(1)
48. @bennegeek Output
49. @bennegeek Output
50. @bennegeek Output
51. @bennegeek Next Steps * Separate app into modules * Mock more data for benchmarking * Try it out yourself
52. @bennegeek Next Steps * Separate app into modules * Mock more data for benchmarking * Try it out yourself * Visit the issue tracker * File bugs
53. @bennegeek tech.olx.com Example project (No OLX code): http://github.com/mootazltaief/ android-json-benchmark
54. @bennegeek tech.olx.com https://issuetracker.google.com/ issues?q=componentid: 585351%20status:open
56. @bennegeek tech.olx.com
59. Berlin Buenos Aires Lisbon New Delhi Poznan Dubai
60. @bennegeek Multumesc! vreo întrebare? Get in touch with us olxgroup.com tech.olx.com olxtechberlin jb.lorenzo@olx.com jblorenzo JB Lorenzo | #DroidconRO19