Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Benchmarking your app's performance

Benchmarking your app's performance

In 1974, Donald Knuth said that “premature optimization is the root of all evil (or at least most of it) in programming”. But optimizing without benchmarks would be like following your instinct trying to get to your destination with a map but without a compass; you hope that you’re going in the right direction, but could be wasting a lot of time and energy..

Whether it is during development or at runtime, many tools exist to let you measure the performance of your applications. These measures can then be used to prioritize which parts to optimize first, as well as measure the performances gained (or lost) after a refactoring.

Based on various concrete examples, this talk will focus on measuring Android code using the Jetpack Benchmark library, as well as exploring how Datadog’s Android SDK can help measure app performance in real life.

Xavier Gouchet

April 22, 2022
Tweet

More Decks by Xavier Gouchet

Other Decks in Programming

Transcript

  1. Benchmarking your
    app’s performance

    View Slide

  2. Xavier F. Gouchet
    Senior Software Engineer
    at Datadog
    @xgouchet / @datadoghq
    2

    View Slide

  3. View Slide

  4. Introduction
    A few words about performance
    0

    View Slide

  5. Mobile app
    performance is still
    critical today
    - Low end devices
    - Slow networks
    - Short attention span
    5
    @priscilladupreez

    View Slide

  6. Performance
    (like Android)
    are fragmented
    If you develop on a Pixel 6, make sure your app also
    works correctly on a Samsung Galaxy A10 (2019)
    6

    View Slide

  7. Performance
    must be measured
    You can’t just base your
    roadmap decision on
    “a hunch”
    7
    @youngprodigy3

    View Slide

  8. User will tell you it feels slow
    Measurements will tell you what
    part exactly is slow.
    #MeasurementsMatter
    Changes must be measured
    Measurements will tell you how
    much your PR improves (or
    degrades) your feature’s
    performance
    8

    View Slide

  9. 9
    Making sure the code works
    - Unit Testing
    - Instrumented Testing
    - Manual Testing
    -
    - …
    In your toolbox
    Making sure it works fast and
    efficiently
    - ???

    View Slide

  10. Developer Tools
    Get a snapshot of your immediate performance
    1

    View Slide

  11. Immediate snapshot
    Get a full view of:
    - network
    - cpu
    - heap
    - rendering
    - …
    11
    @kmuza

    View Slide

  12. 12
    Tools
    Android Studio Profiler Systrace Perfetto

    View Slide

  13. 13
    Android Profiler

    View Slide

  14. 14
    Android Profiler
    Touch
    Activities
    Fragments

    View Slide

  15. 15
    Android Profiler
    CPU usage

    View Slide

  16. 16
    Android Profiler
    Memory usage

    View Slide

  17. 17
    Android Profiler
    Network usage

    View Slide

  18. 18
    Android Profiler
    Battery impact

    View Slide

  19. 19
    Android Profiler

    View Slide

  20. Launching systrace
    $ systrace.py
    20

    View Slide

  21. Launching systrace
    $ systrace.py
    Starting tracing (stop with enter)
    21

    View Slide

  22. Launching systrace
    $ systrace.py
    Starting tracing (stop with enter)
    Tracing completed. Collecting output...
    Outputting Systrace results...
    Tracing complete, writing results
    22

    View Slide

  23. 23
    Systrace (legacy)

    View Slide

  24. 24
    Perfetto showing systrace result

    View Slide

  25. 25
    Recording from Perfetto (API 28+)

    View Slide

  26. 26
    Recording from Perfetto (API 28+)

    View Slide

  27. fun methodWithCustomEvents() {
    Trace.beginSection("MySectionName")
    // …
    Trace.endSection()
    }
    27
    Custom Traces

    View Slide

  28. 28
    Recording from Perfetto (API 28+)

    View Slide

  29. 29
    Recording from Perfetto (API 28+)

    View Slide

  30. Pros 👍
    ▫ (Almost) no setup required
    ▫ Get full system state info
    ▫ Controlled scenario and
    environment
    30
    Profiler / Systrace / Perfetto
    Cons 👎
    ▫ Performance measurements
    are skewed
    ▫ Needs manual trigger
    ▫ Complex and dense UI/UX

    View Slide

  31. When should you use it ?
    ▫ When you have a specific performance issue
    ▫ When you want to optimize a specific part of the code
    ▫ To debug a tricky algorithm (nested loops, concurrency)
    31

    View Slide

  32. Going Further
    ▫ Capture systrace from the device (API 28+)
    32

    View Slide

  33. 33
    Capturing traces from the developer settings

    View Slide

  34. Benchmark
    Standardized performance measurement
    2

    View Slide

  35. Benchmark Overview
    Measure timed performances
    of specific operations
    35
    @veri_ivanova

    View Slide

  36. 36
    Tools
    Jetpack Benchmark CI Provider

    View Slide

  37. Jetpack Benchmark Library
    dependencies {
    // …
    androidTestImplementation
    "androidx.benchmark:benchmark-junit4:1.0.0"
    }
    37

    View Slide

  38. Jetpack Benchmark Library
    android {
    // …
    defaultConfig {
    // …
    testInstrumentationRunner
    "androidx.benchmark.junit4.AndroidBenchmarkRunner"
    }
    }
    38

    View Slide

  39. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    39
    Benchmark Test Class

    View Slide

  40. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    40
    Benchmark Test Class

    View Slide

  41. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    41
    Benchmark Test Class

    View Slide

  42. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    42
    Benchmark Test Function

    View Slide

  43. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    43
    Benchmark Test Function

    View Slide

  44. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    44
    Benchmark Test Function

    View Slide

  45. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    45
    Benchmark Test Function

    View Slide

  46. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    cleanState()
    val data = createData()
    doSomeWork(data)
    }
    }
    46
    Benchmark Test Function

    View Slide

  47. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    runWithTimingDisabled { cleanState() }
    val data = runWithTimingDisabled { createData() }
    doSomeWork(data)
    }
    }
    47
    Benchmark Test Function

    View Slide

  48. $ gradlew :benchmark_module:connectedCheck
    48
    Run the Benchmark

    View Slide

  49. ▫ Stored in a JSON file
    ▫ Full timing information in nanoseconds
    49
    Benchmark Results

    View Slide

  50. {
    "name": "benchmarkSomeWork",
    "className": "com.example.BenchmarkTest",
    "metrics": {
    "timeNs": {
    "minimum": 2561000,
    "maximum": 7133000,
    "median": 3914000,
    "runs": [ … ]
    }
    }
    50
    Benchmark Result

    View Slide

  51. Running Benchmark in the CI
    ▫ Emulators are unstable, use real devices
    ▫ Needs some tweaks in your gradle config
    ▫ Use the JSON to:
    - Graph the evolution of performance
    - Fail if the measurements exceed a threshold
    51

    View Slide

  52. Pros 👍
    ▫ Accurate*, reliable and
    consistent
    ▫ Measure just what you want
    ▫ Can be as high or low level as
    needed
    52
    Jetpack Benchmark
    Cons 👎
    ▫ *Needs to be ran on a real
    device
    ▫ The feature to be tested
    must be repeatable and
    called from code
    ▫ Takes a long time in CI

    View Slide

  53. When should you use it ?
    ▫ Monitor critical parts of your logic in CI
    ▫ When refactoring a specific feature
    ▫ When writing code executed on the main thread
    53

    View Slide

  54. ▫ Official Gradle plugin to lock the CPU clocks (root)
    ▫ Use CI script to fail on high measurements
    Going Further
    54

    View Slide

  55. Real User Monitoring
    Live performance from the trenches
    3

    View Slide

  56. What about the real
    users?
    Get performance and runtime
    information from the
    production environment
    56
    @robin_rednine

    View Slide

  57. 57
    Tools
    Real User Monitoring Error Tracking Android Vitals

    View Slide

  58. 58
    dependencies {
    implementation "com.datadoghq:dd-sdk-android:1.10.0"
    implementation "com.datadoghq:dd-sdk-android-ktx:1.10.0"
    // optional extensions: Compose, RxJava, Android NDK, …
    }
    Using Datadog SDK for Android

    View Slide

  59. Single View Performance
    59

    View Slide

  60. Funnels
    60

    View Slide

  61. Network Performance
    61

    View Slide

  62. Mobile Vitals: Refresh Rate
    62

    View Slide

  63. Mobile Vitals: CPU usage
    63

    View Slide

  64. Mobile Vitals: RAM consumption
    64

    View Slide

  65. Application not responding
    65

    View Slide

  66. Application not responding
    66

    View Slide

  67. Error Tracking
    67

    View Slide

  68. Performance monitoring
    68

    View Slide

  69. Pros 👍
    ▫ Real data
    - real use cases
    - real devices
    - real network conditions
    69
    App Performance Monitoring
    Cons 👎
    ▫ Need large enough user base
    to have relevant data
    ▫ Can’t measure everything

    View Slide

  70. When should you use it ?
    ▫ Monitor key aspects of your app in the wild
    ▫ Understand how performance and crashes impacts
    your users’ journey
    ▫ A/B testing / feature flags analytics
    70

    View Slide

  71. Going Further
    ▫ Library is Open Source on GitHub
    ▫ Also provides Logging and Tracing
    ▫ More features coming soon…
    71

    View Slide

  72. Wrapping up
    4

    View Slide

  73. Key takeaways
    “Measure twice,
    cut once.”
    73
    Choose the relevant
    tool to your use case.
    Always analyse your
    measurements before
    taking action.

    View Slide

  74. 74
    Useful links
    ▫ https:/
    /ui.perfetto.dev/
    ▫ https:/
    /developer.android.com/studio/profile/benchmark
    ▫ https:/
    /developer.android.com/studio/profile/run-benchmarks-in-ci
    ▫ https:/
    /proandroiddev.com/jetpack-benchmark-on-firebase-test-lab-d1
    4c5eae815f
    ▫ https:/
    /github.com/DataDog/dd-sdk-android
    ▫ https:/
    /docs.datadoghq.com/real_user_monitoring/android/

    View Slide

  75. 75
    Thank You!
    Any questions?
    ▫ @xgouchet
    ▫ @datadoghq
    @theunsteady5

    View Slide

  76. CREDITS
    Special thanks to all the people who made and released
    these awesome resources for free:
    ▫ Presentation template by SlidesCarnival
    ▫ Photographs from Unsplash
    76

    View Slide