Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Benchmarking your app's performance

Benchmarking your app's performance

In 1974, Donald Knuth said that “premature optimization is the root of all evil (or at least most of it) in programming”. But optimizing without benchmarks would be like following your instinct trying to get to your destination with a map but without a compass; you hope that you’re going in the right direction, but could be wasting a lot of time and energy..

Whether it is during development or at runtime, many tools exist to let you measure the performance of your applications. These measures can then be used to prioritize which parts to optimize first, as well as measure the performances gained (or lost) after a refactoring.

Based on various concrete examples, this talk will focus on measuring Android code using the Jetpack Benchmark library, as well as exploring how Datadog’s Android SDK can help measure app performance in real life.

Xavier Gouchet

April 22, 2022
Tweet

More Decks by Xavier Gouchet

Other Decks in Programming

Transcript

  1. Benchmarking your
    app’s performance

    View full-size slide

  2. Xavier F. Gouchet
    Senior Software Engineer
    at Datadog
    @xgouchet / @datadoghq
    2

    View full-size slide

  3. Introduction
    A few words about performance
    0

    View full-size slide

  4. Mobile app
    performance is still
    critical today
    - Low end devices
    - Slow networks
    - Short attention span
    5
    @priscilladupreez

    View full-size slide

  5. Performance
    (like Android)
    are fragmented
    If you develop on a Pixel 6, make sure your app also
    works correctly on a Samsung Galaxy A10 (2019)
    6

    View full-size slide

  6. Performance
    must be measured
    You can’t just base your
    roadmap decision on
    “a hunch”
    7
    @youngprodigy3

    View full-size slide

  7. User will tell you it feels slow
    Measurements will tell you what
    part exactly is slow.
    #MeasurementsMatter
    Changes must be measured
    Measurements will tell you how
    much your PR improves (or
    degrades) your feature’s
    performance
    8

    View full-size slide

  8. 9
    Making sure the code works
    - Unit Testing
    - Instrumented Testing
    - Manual Testing
    -
    - …
    In your toolbox
    Making sure it works fast and
    efficiently
    - ???

    View full-size slide

  9. Developer Tools
    Get a snapshot of your immediate performance
    1

    View full-size slide

  10. Immediate snapshot
    Get a full view of:
    - network
    - cpu
    - heap
    - rendering
    - …
    11
    @kmuza

    View full-size slide

  11. 12
    Tools
    Android Studio Profiler Systrace Perfetto

    View full-size slide

  12. 13
    Android Profiler

    View full-size slide

  13. 14
    Android Profiler
    Touch
    Activities
    Fragments

    View full-size slide

  14. 15
    Android Profiler
    CPU usage

    View full-size slide

  15. 16
    Android Profiler
    Memory usage

    View full-size slide

  16. 17
    Android Profiler
    Network usage

    View full-size slide

  17. 18
    Android Profiler
    Battery impact

    View full-size slide

  18. 19
    Android Profiler

    View full-size slide

  19. Launching systrace
    $ systrace.py
    20

    View full-size slide

  20. Launching systrace
    $ systrace.py
    Starting tracing (stop with enter)
    21

    View full-size slide

  21. Launching systrace
    $ systrace.py
    Starting tracing (stop with enter)
    Tracing completed. Collecting output...
    Outputting Systrace results...
    Tracing complete, writing results
    22

    View full-size slide

  22. 23
    Systrace (legacy)

    View full-size slide

  23. 24
    Perfetto showing systrace result

    View full-size slide

  24. 25
    Recording from Perfetto (API 28+)

    View full-size slide

  25. 26
    Recording from Perfetto (API 28+)

    View full-size slide

  26. fun methodWithCustomEvents() {
    Trace.beginSection("MySectionName")
    // …
    Trace.endSection()
    }
    27
    Custom Traces

    View full-size slide

  27. 28
    Recording from Perfetto (API 28+)

    View full-size slide

  28. 29
    Recording from Perfetto (API 28+)

    View full-size slide

  29. Pros 👍
    ▫ (Almost) no setup required
    ▫ Get full system state info
    ▫ Controlled scenario and
    environment
    30
    Profiler / Systrace / Perfetto
    Cons 👎
    ▫ Performance measurements
    are skewed
    ▫ Needs manual trigger
    ▫ Complex and dense UI/UX

    View full-size slide

  30. When should you use it ?
    ▫ When you have a specific performance issue
    ▫ When you want to optimize a specific part of the code
    ▫ To debug a tricky algorithm (nested loops, concurrency)
    31

    View full-size slide

  31. Going Further
    ▫ Capture systrace from the device (API 28+)
    32

    View full-size slide

  32. 33
    Capturing traces from the developer settings

    View full-size slide

  33. Benchmark
    Standardized performance measurement
    2

    View full-size slide

  34. Benchmark Overview
    Measure timed performances
    of specific operations
    35
    @veri_ivanova

    View full-size slide

  35. 36
    Tools
    Jetpack Benchmark CI Provider

    View full-size slide

  36. Jetpack Benchmark Library
    dependencies {
    // …
    androidTestImplementation
    "androidx.benchmark:benchmark-junit4:1.0.0"
    }
    37

    View full-size slide

  37. Jetpack Benchmark Library
    android {
    // …
    defaultConfig {
    // …
    testInstrumentationRunner
    "androidx.benchmark.junit4.AndroidBenchmarkRunner"
    }
    }
    38

    View full-size slide

  38. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    39
    Benchmark Test Class

    View full-size slide

  39. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    40
    Benchmark Test Class

    View full-size slide

  40. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    41
    Benchmark Test Class

    View full-size slide

  41. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    42
    Benchmark Test Function

    View full-size slide

  42. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    43
    Benchmark Test Function

    View full-size slide

  43. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    44
    Benchmark Test Function

    View full-size slide

  44. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    45
    Benchmark Test Function

    View full-size slide

  45. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    cleanState()
    val data = createData()
    doSomeWork(data)
    }
    }
    46
    Benchmark Test Function

    View full-size slide

  46. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    runWithTimingDisabled { cleanState() }
    val data = runWithTimingDisabled { createData() }
    doSomeWork(data)
    }
    }
    47
    Benchmark Test Function

    View full-size slide

  47. $ gradlew :benchmark_module:connectedCheck
    48
    Run the Benchmark

    View full-size slide

  48. ▫ Stored in a JSON file
    ▫ Full timing information in nanoseconds
    49
    Benchmark Results

    View full-size slide

  49. {
    "name": "benchmarkSomeWork",
    "className": "com.example.BenchmarkTest",
    "metrics": {
    "timeNs": {
    "minimum": 2561000,
    "maximum": 7133000,
    "median": 3914000,
    "runs": [ … ]
    }
    }
    50
    Benchmark Result

    View full-size slide

  50. Running Benchmark in the CI
    ▫ Emulators are unstable, use real devices
    ▫ Needs some tweaks in your gradle config
    ▫ Use the JSON to:
    - Graph the evolution of performance
    - Fail if the measurements exceed a threshold
    51

    View full-size slide

  51. Pros 👍
    ▫ Accurate*, reliable and
    consistent
    ▫ Measure just what you want
    ▫ Can be as high or low level as
    needed
    52
    Jetpack Benchmark
    Cons 👎
    ▫ *Needs to be ran on a real
    device
    ▫ The feature to be tested
    must be repeatable and
    called from code
    ▫ Takes a long time in CI

    View full-size slide

  52. When should you use it ?
    ▫ Monitor critical parts of your logic in CI
    ▫ When refactoring a specific feature
    ▫ When writing code executed on the main thread
    53

    View full-size slide

  53. ▫ Official Gradle plugin to lock the CPU clocks (root)
    ▫ Use CI script to fail on high measurements
    Going Further
    54

    View full-size slide

  54. Real User Monitoring
    Live performance from the trenches
    3

    View full-size slide

  55. What about the real
    users?
    Get performance and runtime
    information from the
    production environment
    56
    @robin_rednine

    View full-size slide

  56. 57
    Tools
    Real User Monitoring Error Tracking Android Vitals

    View full-size slide

  57. 58
    dependencies {
    implementation "com.datadoghq:dd-sdk-android:1.10.0"
    implementation "com.datadoghq:dd-sdk-android-ktx:1.10.0"
    // optional extensions: Compose, RxJava, Android NDK, …
    }
    Using Datadog SDK for Android

    View full-size slide

  58. Single View Performance
    59

    View full-size slide

  59. Network Performance
    61

    View full-size slide

  60. Mobile Vitals: Refresh Rate
    62

    View full-size slide

  61. Mobile Vitals: CPU usage
    63

    View full-size slide

  62. Mobile Vitals: RAM consumption
    64

    View full-size slide

  63. Application not responding
    65

    View full-size slide

  64. Application not responding
    66

    View full-size slide

  65. Error Tracking
    67

    View full-size slide

  66. Performance monitoring
    68

    View full-size slide

  67. Pros 👍
    ▫ Real data
    - real use cases
    - real devices
    - real network conditions
    69
    App Performance Monitoring
    Cons 👎
    ▫ Need large enough user base
    to have relevant data
    ▫ Can’t measure everything

    View full-size slide

  68. When should you use it ?
    ▫ Monitor key aspects of your app in the wild
    ▫ Understand how performance and crashes impacts
    your users’ journey
    ▫ A/B testing / feature flags analytics
    70

    View full-size slide

  69. Going Further
    ▫ Library is Open Source on GitHub
    ▫ Also provides Logging and Tracing
    ▫ More features coming soon…
    71

    View full-size slide

  70. Wrapping up
    4

    View full-size slide

  71. Key takeaways
    “Measure twice,
    cut once.”
    73
    Choose the relevant
    tool to your use case.
    Always analyse your
    measurements before
    taking action.

    View full-size slide

  72. 74
    Useful links
    ▫ https:/
    /ui.perfetto.dev/
    ▫ https:/
    /developer.android.com/studio/profile/benchmark
    ▫ https:/
    /developer.android.com/studio/profile/run-benchmarks-in-ci
    ▫ https:/
    /proandroiddev.com/jetpack-benchmark-on-firebase-test-lab-d1
    4c5eae815f
    ▫ https:/
    /github.com/DataDog/dd-sdk-android
    ▫ https:/
    /docs.datadoghq.com/real_user_monitoring/android/

    View full-size slide

  73. 75
    Thank You!
    Any questions?
    ▫ @xgouchet
    ▫ @datadoghq
    @theunsteady5

    View full-size slide

  74. CREDITS
    Special thanks to all the people who made and released
    these awesome resources for free:
    ▫ Presentation template by SlidesCarnival
    ▫ Photographs from Unsplash
    76

    View full-size slide