Benchmarking your app's performance

In 1974, Donald Knuth said that “premature optimization is the root of all evil (or at least most of it) in programming”. But optimizing without benchmarks would be like following your instinct trying to get to your destination with a map but without a compass; you hope that you’re going in the right direction, but could be wasting a lot of time and energy..

Whether it is during development or at runtime, many tools exist to let you measure the performance of your applications. These measures can then be used to prioritize which parts to optimize first, as well as measure the performances gained (or lost) after a refactoring.

Based on various concrete examples, this talk will focus on measuring Android code using the Jetpack Benchmark library, as well as exploring how Datadog’s Android SDK can help measure app performance in real life.

Transcript

1. Benchmarking your app’s performance
2. Xavier F. Gouchet Senior Software Engineer at Datadog @xgouchet / @datadoghq 2
4. Introduction A few words about performance 0
5. Mobile app performance is still critical today - Low end devices - Slow networks - Short attention span 5 @priscilladupreez
6. Performance (like Android) are fragmented If you develop on a Pixel 6, make sure your app also works correctly on a Samsung Galaxy A10 (2019) 6
7. Performance must be measured You can’t just base your roadmap decision on “a hunch” 7 @youngprodigy3
8. User will tell you it feels slow Measurements will tell you what part exactly is slow. #MeasurementsMatter Changes must be measured Measurements will tell you how much your PR improves (or degrades) your feature’s performance 8
9. 9 Making sure the code works - Unit Testing - Instrumented Testing - Manual Testing - <insert testing paradigm> - … In your toolbox Making sure it works fast and efficiently - ???
10. Developer Tools Get a snapshot of your immediate performance 1
11. Immediate snapshot Get a full view of: - network - cpu - heap - rendering - … 11 @kmuza
12. 12 Tools Android Studio Profiler Systrace Perfetto
13. 13 Android Profiler
14. 14 Android Profiler Touch Activities Fragments
15. 15 Android Profiler CPU usage
16. 16 Android Profiler Memory usage
17. 17 Android Profiler Network usage
18. 18 Android Profiler Battery impact
19. 19 Android Profiler
20. Launching systrace $ systrace.py 20
21. Launching systrace $ systrace.py Starting tracing (stop with enter) 21
22. Launching systrace $ systrace.py Starting tracing (stop with enter) Tracing completed. Collecting output... Outputting Systrace results... Tracing complete, writing results 22
23. 23 Systrace (legacy)
24. 24 Perfetto showing systrace result
25. 25 Recording from Perfetto (API 28+)
26. 26 Recording from Perfetto (API 28+)
27. fun methodWithCustomEvents() { Trace.beginSection("MySectionName") // … Trace.endSection() } 27 Custom Traces
28. 28 Recording from Perfetto (API 28+)
29. 29 Recording from Perfetto (API 28+)
30. Pros 👍 ▫ (Almost) no setup required ▫ Get full system state info ▫ Controlled scenario and environment 30 Profiler / Systrace / Perfetto Cons 👎 ▫ Performance measurements are skewed ▫ Needs manual trigger ▫ Complex and dense UI/UX
31. When should you use it ? ▫ When you have a specific performance issue ▫ When you want to optimize a specific part of the code ▫ To debug a tricky algorithm (nested loops, concurrency) 31
32. Going Further ▫ Capture systrace from the device (API 28+) 32
33. 33 Capturing traces from the developer settings
34. Benchmark Standardized performance measurement 2
35. Benchmark Overview Measure timed performances of specific operations 35 @veri_ivanova
36. 36 Tools Jetpack Benchmark CI Provider
37. Jetpack Benchmark Library dependencies { // … androidTestImplementation "androidx.benchmark:benchmark-junit4:1.0.0" } 37
38. Jetpack Benchmark Library android { // … defaultConfig { // … testInstrumentationRunner "androidx.benchmark.junit4.AndroidBenchmarkRunner" } } 38
39. @RunWith(AndroidJUnit4::class) class BenchmarkTest { @get:Rule val benchmarkRule = BenchmarkRule() // … } 39 Benchmark Test Class
40. @RunWith(AndroidJUnit4::class) class BenchmarkTest { @get:Rule val benchmarkRule = BenchmarkRule() // … } 40 Benchmark Test Class
41. @RunWith(AndroidJUnit4::class) class BenchmarkTest { @get:Rule val benchmarkRule = BenchmarkRule() // … } 41 Benchmark Test Class
42. @Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { doSomeWork() } } 42 Benchmark Test Function
43. @Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { doSomeWork() } } 43 Benchmark Test Function
44. @Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { doSomeWork() } } 44 Benchmark Test Function
45. @Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { doSomeWork() } } 45 Benchmark Test Function
46. @Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { cleanState() val data = createData() doSomeWork(data) } } 46 Benchmark Test Function
47. @Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { runWithTimingDisabled { cleanState() } val data = runWithTimingDisabled { createData() } doSomeWork(data) } } 47 Benchmark Test Function
48. $ gradlew :benchmark_module:connectedCheck 48 Run the Benchmark
49. ▫ Stored in a JSON file ▫ Full timing information in nanoseconds 49 Benchmark Results
50. { "name": "benchmarkSomeWork", "className": "com.example.BenchmarkTest", "metrics": { "timeNs": { "minimum": 2561000, "maximum": 7133000, "median": 3914000, "runs": [ … ] } } 50 Benchmark Result
51. Running Benchmark in the CI ▫ Emulators are unstable, use real devices ▫ Needs some tweaks in your gradle config ▫ Use the JSON to: - Graph the evolution of performance - Fail if the measurements exceed a threshold 51
52. Pros 👍 ▫ Accurate*, reliable and consistent ▫ Measure just what you want ▫ Can be as high or low level as needed 52 Jetpack Benchmark Cons 👎 ▫ *Needs to be ran on a real device ▫ The feature to be tested must be repeatable and called from code ▫ Takes a long time in CI
53. When should you use it ? ▫ Monitor critical parts of your logic in CI ▫ When refactoring a specific feature ▫ When writing code executed on the main thread 53
54. ▫ Official Gradle plugin to lock the CPU clocks (root) ▫ Use CI script to fail on high measurements Going Further 54
55. Real User Monitoring Live performance from the trenches 3
56. What about the real users? Get performance and runtime information from the production environment 56 @robin_rednine
57. 57 Tools Real User Monitoring Error Tracking Android Vitals
58. 58 dependencies { implementation "com.datadoghq:dd-sdk-android:1.10.0" implementation "com.datadoghq:dd-sdk-android-ktx:1.10.0" // optional extensions: Compose, RxJava, Android NDK, … } Using Datadog SDK for Android
59. Single View Performance 59
60. Funnels 60
61. Network Performance 61
62. Mobile Vitals: Refresh Rate 62
63. Mobile Vitals: CPU usage 63
64. Mobile Vitals: RAM consumption 64
65. Application not responding 65
66. Application not responding 66
67. Error Tracking 67
68. Performance monitoring 68
69. Pros 👍 ▫ Real data - real use cases - real devices - real network conditions 69 App Performance Monitoring Cons 👎 ▫ Need large enough user base to have relevant data ▫ Can’t measure everything
70. When should you use it ? ▫ Monitor key aspects of your app in the wild ▫ Understand how performance and crashes impacts your users’ journey ▫ A/B testing / feature flags analytics 70
71. Going Further ▫ Library is Open Source on GitHub ▫ Also provides Logging and Tracing ▫ More features coming soon… 71
72. Wrapping up 4
73. Key takeaways “Measure twice, cut once.” 73 Choose the relevant tool to your use case. Always analyse your measurements before taking action.
74. 74 Useful links ▫ https:/ /ui.perfetto.dev/ ▫ https:/ /developer.android.com/studio/profile/benchmark ▫ https:/ /developer.android.com/studio/profile/run-benchmarks-in-ci ▫ https:/ /proandroiddev.com/jetpack-benchmark-on-firebase-test-lab-d1 4c5eae815f ▫ https:/ /github.com/DataDog/dd-sdk-android ▫ https:/ /docs.datadoghq.com/real_user_monitoring/android/
75. 75 Thank You! Any questions? ▫ @xgouchet ▫ @datadoghq @theunsteady5
76. CREDITS Special thanks to all the people who made and released these awesome resources for free: ▫ Presentation template by SlidesCarnival ▫ Photographs from Unsplash 76