Benchmarking your app's performance

In 1974, Donald Knuth said that “premature optimization is the root of all evil (or at least most of it) in programming”. But optimizing without benchmarks would be like following your instinct trying to get to your destination with a map but without a compass; you hope that you’re going in the right direction, but could be wasting a lot of time and energy..

Whether it is during development or at runtime, many tools exist to let you measure the performance of your applications. These measures can then be used to prioritize which parts to optimize first, as well as measure the performances gained (or lost) after a refactoring.

Based on various concrete examples, this talk will focus on measuring Android code using the Jetpack Benchmark library, as well as exploring how Datadog’s Android SDK can help measure app performance in real life.

Transcript

1. Benchmarking your
   app’s performance
   

   View Slide

2. Xavier F. Gouchet
   Senior Software Engineer
   at Datadog
   @xgouchet / @datadoghq
   2
   

   View Slide

3. View Slide

4. Introduction
   A few words about performance
   0
   

   View Slide

5. Mobile app
   performance is still
   critical today
   - Low end devices
   - Slow networks
   - Short attention span
   5
   @priscilladupreez
   

   View Slide

6. Performance
   (like Android)
   are fragmented
   If you develop on a Pixel 6, make sure your app also
   works correctly on a Samsung Galaxy A10 (2019)
   6
   

   View Slide

7. Performance
   must be measured
   You can’t just base your
   roadmap decision on
   “a hunch”
   7
   @youngprodigy3
   

   View Slide

8. User will tell you it feels slow
   Measurements will tell you what
   part exactly is slow.
   #MeasurementsMatter
   Changes must be measured
   Measurements will tell you how
   much your PR improves (or
   degrades) your feature’s
   performance
   8
   

   View Slide

9. 9
   Making sure the code works
   - Unit Testing
   - Instrumented Testing
   - Manual Testing
   -
   - …
   In your toolbox
   Making sure it works fast and
   efficiently
   - ???
   

   View Slide

10. Developer Tools
    Get a snapshot of your immediate performance
    1
    

    View Slide

11. Immediate snapshot
    Get a full view of:
    - network
    - cpu
    - heap
    - rendering
    - …
    11
    @kmuza
    

    View Slide

12. 12
    Tools
    Android Studio Profiler Systrace Perfetto
    

    View Slide

13. 13
    Android Profiler
    

    View Slide

14. 14
    Android Profiler
    Touch
    Activities
    Fragments
    

    View Slide

15. 15
    Android Profiler
    CPU usage
    

    View Slide

16. 16
    Android Profiler
    Memory usage
    

    View Slide

17. 17
    Android Profiler
    Network usage
    

    View Slide

18. 18
    Android Profiler
    Battery impact
    

    View Slide

19. 19
    Android Profiler
    

    View Slide

20. Launching systrace
    $ systrace.py
    20
    

    View Slide

21. Launching systrace
    $ systrace.py
    Starting tracing (stop with enter)
    21
    

    View Slide

22. Launching systrace
    $ systrace.py
    Starting tracing (stop with enter)
    Tracing completed. Collecting output...
    Outputting Systrace results...
    Tracing complete, writing results
    22
    

    View Slide

23. 23
    Systrace (legacy)
    

    View Slide

24. 24
    Perfetto showing systrace result
    

    View Slide

25. 25
    Recording from Perfetto (API 28+)
    

    View Slide

26. 26
    Recording from Perfetto (API 28+)
    

    View Slide

27. fun methodWithCustomEvents() {
    Trace.beginSection("MySectionName")
    // …
    Trace.endSection()
    }
    27
    Custom Traces
    

    View Slide

28. 28
    Recording from Perfetto (API 28+)
    

    View Slide

29. 29
    Recording from Perfetto (API 28+)
    

    View Slide

30. Pros 👍
    ▫ (Almost) no setup required
    ▫ Get full system state info
    ▫ Controlled scenario and
    environment
    30
    Profiler / Systrace / Perfetto
    Cons 👎
    ▫ Performance measurements
    are skewed
    ▫ Needs manual trigger
    ▫ Complex and dense UI/UX
    

    View Slide

31. When should you use it ?
    ▫ When you have a specific performance issue
    ▫ When you want to optimize a specific part of the code
    ▫ To debug a tricky algorithm (nested loops, concurrency)
    31
    

    View Slide

32. Going Further
    ▫ Capture systrace from the device (API 28+)
    32
    

    View Slide

33. 33
    Capturing traces from the developer settings
    

    View Slide

34. Benchmark
    Standardized performance measurement
    2
    

    View Slide

35. Benchmark Overview
    Measure timed performances
    of specific operations
    35
    @veri_ivanova
    

    View Slide

36. 36
    Tools
    Jetpack Benchmark CI Provider
    

    View Slide

37. Jetpack Benchmark Library
    dependencies {
    // …
    androidTestImplementation
    "androidx.benchmark:benchmark-junit4:1.0.0"
    }
    37
    

    View Slide

38. Jetpack Benchmark Library
    android {
    // …
    defaultConfig {
    // …
    testInstrumentationRunner
    "androidx.benchmark.junit4.AndroidBenchmarkRunner"
    }
    }
    38
    

    View Slide

39. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    39
    Benchmark Test Class
    

    View Slide

40. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    40
    Benchmark Test Class
    

    View Slide

41. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    41
    Benchmark Test Class
    

    View Slide

42. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    42
    Benchmark Test Function
    

    View Slide

43. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    43
    Benchmark Test Function
    

    View Slide

44. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    44
    Benchmark Test Function
    

    View Slide

45. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    45
    Benchmark Test Function
    

    View Slide

46. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    cleanState()
    val data = createData()
    doSomeWork(data)
    }
    }
    46
    Benchmark Test Function
    

    View Slide

47. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    runWithTimingDisabled { cleanState() }
    val data = runWithTimingDisabled { createData() }
    doSomeWork(data)
    }
    }
    47
    Benchmark Test Function
    

    View Slide

48. $ gradlew :benchmark_module:connectedCheck
    48
    Run the Benchmark
    

    View Slide

49. ▫ Stored in a JSON file
    ▫ Full timing information in nanoseconds
    49
    Benchmark Results
    

    View Slide

50. {
    "name": "benchmarkSomeWork",
    "className": "com.example.BenchmarkTest",
    "metrics": {
    "timeNs": {
    "minimum": 2561000,
    "maximum": 7133000,
    "median": 3914000,
    "runs": [ … ]
    }
    }
    50
    Benchmark Result
    

    View Slide

51. Running Benchmark in the CI
    ▫ Emulators are unstable, use real devices
    ▫ Needs some tweaks in your gradle config
    ▫ Use the JSON to:
    - Graph the evolution of performance
    - Fail if the measurements exceed a threshold
    51
    

    View Slide

52. Pros 👍
    ▫ Accurate*, reliable and
    consistent
    ▫ Measure just what you want
    ▫ Can be as high or low level as
    needed
    52
    Jetpack Benchmark
    Cons 👎
    ▫ *Needs to be ran on a real
    device
    ▫ The feature to be tested
    must be repeatable and
    called from code
    ▫ Takes a long time in CI
    

    View Slide

53. When should you use it ?
    ▫ Monitor critical parts of your logic in CI
    ▫ When refactoring a specific feature
    ▫ When writing code executed on the main thread
    53
    

    View Slide

54. ▫ Official Gradle plugin to lock the CPU clocks (root)
    ▫ Use CI script to fail on high measurements
    Going Further
    54
    

    View Slide

55. Real User Monitoring
    Live performance from the trenches
    3
    

    View Slide

56. What about the real
    users?
    Get performance and runtime
    information from the
    production environment
    56
    @robin_rednine
    

    View Slide

57. 57
    Tools
    Real User Monitoring Error Tracking Android Vitals
    

    View Slide

58. 58
    dependencies {
    implementation "com.datadoghq:dd-sdk-android:1.10.0"
    implementation "com.datadoghq:dd-sdk-android-ktx:1.10.0"
    // optional extensions: Compose, RxJava, Android NDK, …
    }
    Using Datadog SDK for Android
    

    View Slide

59. Single View Performance
    59
    

    View Slide

60. Funnels
    60
    

    View Slide

61. Network Performance
    61
    

    View Slide

62. Mobile Vitals: Refresh Rate
    62
    

    View Slide

63. Mobile Vitals: CPU usage
    63
    

    View Slide

64. Mobile Vitals: RAM consumption
    64
    

    View Slide

65. Application not responding
    65
    

    View Slide

66. Application not responding
    66
    

    View Slide

67. Error Tracking
    67
    

    View Slide

68. Performance monitoring
    68
    

    View Slide

69. Pros 👍
    ▫ Real data
    - real use cases
    - real devices
    - real network conditions
    69
    App Performance Monitoring
    Cons 👎
    ▫ Need large enough user base
    to have relevant data
    ▫ Can’t measure everything
    

    View Slide

70. When should you use it ?
    ▫ Monitor key aspects of your app in the wild
    ▫ Understand how performance and crashes impacts
    your users’ journey
    ▫ A/B testing / feature flags analytics
    70
    

    View Slide

71. Going Further
    ▫ Library is Open Source on GitHub
    ▫ Also provides Logging and Tracing
    ▫ More features coming soon…
    71
    

    View Slide

72. Wrapping up
    4
    

    View Slide

73. Key takeaways
    “Measure twice,
    cut once.”
    73
    Choose the relevant
    tool to your use case.
    Always analyse your
    measurements before
    taking action.
    

    View Slide

74. 74
    Useful links
    ▫ https:/
    /ui.perfetto.dev/
    ▫ https:/
    /developer.android.com/studio/profile/benchmark
    ▫ https:/
    /developer.android.com/studio/profile/run-benchmarks-in-ci
    ▫ https:/
    /proandroiddev.com/jetpack-benchmark-on-firebase-test-lab-d1
    4c5eae815f
    ▫ https:/
    /github.com/DataDog/dd-sdk-android
    ▫ https:/
    /docs.datadoghq.com/real_user_monitoring/android/
    

    View Slide

75. 75
    Thank You!
    Any questions?
    ▫ @xgouchet
    ▫ @datadoghq
    @theunsteady5
    

    View Slide

76. CREDITS
    Special thanks to all the people who made and released
    these awesome resources for free:
    ▫ Presentation template by SlidesCarnival
    ▫ Photographs from Unsplash
    76
    

    View Slide