Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Benchmarking your app's performance

Benchmarking your app's performance

In 1974, Donald Knuth said that “premature optimization is the root of all evil (or at least most of it) in programming”. But optimizing without benchmarks would be like following your instinct trying to get to your destination with a map but without a compass; you hope that you’re going in the right direction, but could be wasting a lot of time and energy..

Whether it is during development or at runtime, many tools exist to let you measure the performance of your applications. These measures can then be used to prioritize which parts to optimize first, as well as measure the performances gained (or lost) after a refactoring.

Based on various concrete examples, this talk will focus on measuring Android code using the Jetpack Benchmark library, as well as exploring how Datadog’s Android SDK can help measure app performance in real life.

Xavier Gouchet

October 29, 2021
Tweet

More Decks by Xavier Gouchet

Other Decks in Programming

Transcript

  1. Benchmarking your
    app’s performance
    Xavier F. Gouchet, Senior Software Engineer / Team Lead
    @xgouchet (Twitter, GitHub, SpeakerDeck)

    View full-size slide

  2. Introduction
    A few words about performance
    0

    View full-size slide

  3. Mobile app
    performance is still
    critical today
    - Low end devices
    - Slow networks
    - Short attention span
    4
    @priscilladupreez

    View full-size slide

  4. Performance
    (like Android)
    are fragmented
    If you develop on a Pixel 5, make sure your app also
    works correctly on a Samsung Galaxy S8 (2017)
    5

    View full-size slide

  5. Performance
    must be measured
    You can’t just base your
    roadmap decision on
    “a hunch”
    6
    @youngprodigy3

    View full-size slide

  6. User will tell you it feels slow
    Measurements will tell you what
    part exactly is slow.
    #MeasurementsMatter
    Changes must be measured
    Measurements will tell you how
    much your PR improves (or
    degrades) your feature’s
    performance
    7

    View full-size slide

  7. 8
    Making sure the code works
    - Unit Testing
    - Instrumented Testing
    - Manual Testing
    -
    - …
    In your toolbox
    Making sure it works fast and
    efficiently
    - ???

    View full-size slide

  8. Developer Tools
    Get a snapshot of your immediate performance
    1

    View full-size slide

  9. Immediate snapshot
    Get a full view of:
    - network
    - cpu
    - heap
    - rendering
    - …
    10
    @kmuza

    View full-size slide

  10. 11
    Tools
    Android Studio Profiler Systrace Perfetto

    View full-size slide

  11. 12
    Android Profiler

    View full-size slide

  12. 13
    Android Profiler
    Touch
    Activities
    Fragments

    View full-size slide

  13. 14
    Android Profiler
    CPU usage

    View full-size slide

  14. 15
    Android Profiler
    Memory usage

    View full-size slide

  15. 16
    Android Profiler
    Network usage

    View full-size slide

  16. 17
    Android Profiler
    Battery impact

    View full-size slide

  17. 18
    Android Profiler

    View full-size slide

  18. Launching systrace
    $ systrace.py
    19

    View full-size slide

  19. Launching systrace
    $ systrace.py
    Starting tracing (stop with enter)
    20

    View full-size slide

  20. Launching systrace
    $ systrace.py
    Starting tracing (stop with enter)
    Tracing completed. Collecting output...
    Outputting Systrace results...
    Tracing complete, writing results
    21

    View full-size slide

  21. 22
    Systrace (legacy)

    View full-size slide

  22. 23
    Perfetto showing systrace result

    View full-size slide

  23. 24
    Recording from Perfetto (API 28+)

    View full-size slide

  24. 25
    Recording from Perfetto (API 28+)

    View full-size slide

  25. fun methodWithCustomEvents() {
    Trace.beginSection("MySectionName")
    // …
    Trace.endSection()
    }
    26
    Custom Traces

    View full-size slide

  26. 27
    Recording from Perfetto (API 28+)

    View full-size slide

  27. 28
    Recording from Perfetto (API 28+)

    View full-size slide

  28. Pros 👍
    ▫ (Almost) no setup required
    ▫ Get full system state info
    ▫ Controlled scenario and
    environment
    29
    Profiler / Systrace / Perfetto
    Cons 👎
    ▫ Performance measurements
    are skewed
    ▫ Needs manual trigger
    ▫ Complex and dense UI/UX

    View full-size slide

  29. When should you use it ?
    ▫ When you have a specific performance issue
    ▫ When you want to optimize a specific part of the code
    ▫ To debug a tricky algorithm (nested loops, concurrency)
    30

    View full-size slide

  30. Going Further
    ▫ Capture systrace from the device (API 28+)
    31

    View full-size slide

  31. 32
    Capturing traces from the developer settings

    View full-size slide

  32. Benchmark
    Standardized performance measurement
    2

    View full-size slide

  33. Benchmark Overview
    Measure timed performances
    of specific operations
    34
    @veri_ivanova

    View full-size slide

  34. 35
    Tools
    Jetpack Benchmark CI Provider

    View full-size slide

  35. Jetpack Benchmark Library
    dependencies {
    // …
    androidTestImplementation
    "androidx.benchmark:benchmark-junit4:1.0.0"
    }
    36

    View full-size slide

  36. Jetpack Benchmark Library
    android {
    // …
    defaultConfig {
    // …
    testInstrumentationRunner
    "androidx.benchmark.junit4.AndroidBenchmarkRunner"
    }
    }
    37

    View full-size slide

  37. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    38
    Benchmark Test Class

    View full-size slide

  38. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    39
    Benchmark Test Class

    View full-size slide

  39. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    40
    Benchmark Test Class

    View full-size slide

  40. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    41
    Benchmark Test Function

    View full-size slide

  41. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    42
    Benchmark Test Function

    View full-size slide

  42. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    43
    Benchmark Test Function

    View full-size slide

  43. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    44
    Benchmark Test Function

    View full-size slide

  44. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    cleanState()
    val data = createData()
    doSomeWork(data)
    }
    }
    45
    Benchmark Test Function

    View full-size slide

  45. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    runWithTimingDisabled { cleanState() }
    val data = runWithTimingDisabled { createData() }
    doSomeWork(data)
    }
    }
    46
    Benchmark Test Function

    View full-size slide

  46. $ gradlew :benchmark_module:connectedCheck
    47
    Run the Benchmark

    View full-size slide

  47. ▫ Stored in a JSON file
    ▫ Full timing information in nanoseconds
    48
    Benchmark Results

    View full-size slide

  48. {
    "name": "benchmarkSomeWork",
    "className": "com.example.BenchmarkTest",
    "metrics": {
    "timeNs": {
    "minimum": 2561000,
    "maximum": 7133000,
    "median": 3914000,
    "runs": [ … ]
    }
    }
    49
    Benchmark Result

    View full-size slide

  49. Running Benchmark in the CI
    ▫ Emulators are unstable, use real devices
    ▫ Needs some tweaks in your gradle config
    ▫ Use the JSON to:
    - Graph the evolution of performance
    - Fail if the measurements exceed a threshold
    50

    View full-size slide

  50. Pros 👍
    ▫ Accurate*, reliable and
    consistent
    ▫ Measure just what you want
    ▫ Can be as high or low level as
    needed
    51
    Jetpack Benchmark
    Cons 👎
    ▫ *Needs to be ran on a real
    device
    ▫ The feature to be tested
    must be repeatable and
    called from code
    ▫ Takes a long time in CI

    View full-size slide

  51. When should you use it ?
    ▫ Monitor critical parts of your logic in CI
    ▫ When refactoring a specific feature
    ▫ When writing code executed on the main thread
    52

    View full-size slide

  52. ▫ Official Gradle plugin to lock the CPU clocks (root)
    ▫ Use CI script to fail on high measurements
    Going Further
    53

    View full-size slide

  53. Real User Monitoring
    Live performance from the trenches
    3

    View full-size slide

  54. What about the real
    users?
    Get performance and runtime
    information from the
    production environment
    55
    @robin_rednine

    View full-size slide

  55. 56
    Tools
    Real User Monitoring Error Tracking Android Vitals

    View full-size slide

  56. 57
    dependencies {
    implementation "com.datadoghq:dd-sdk-android:1.10.0"
    implementation "com.datadoghq:dd-sdk-android-ktx:1.10.0"
    // optional extensions: RxJava, Android NDK, …
    }
    Using Datadog SDK for Android

    View full-size slide

  57. Single View Performance
    58

    View full-size slide

  58. Network Performance
    60

    View full-size slide

  59. Mobile Vitals: Refresh Rate
    61

    View full-size slide

  60. Mobile Vitals: CPU usage
    62

    View full-size slide

  61. Mobile Vitals: RAM consumption
    63

    View full-size slide

  62. Application not responding
    64

    View full-size slide

  63. Application not responding
    65

    View full-size slide

  64. Error Tracking
    66

    View full-size slide

  65. Performance monitoring
    67

    View full-size slide

  66. Pros 👍
    ▫ Real data
    - real use cases
    - real devices
    - real network conditions
    68
    App Performance Monitoring
    Cons 👎
    ▫ Need large enough user base
    to have relevant data
    ▫ Can’t measure everything

    View full-size slide

  67. When should you use it ?
    ▫ Monitor key aspects of your app in the wild
    ▫ Understand how performance and crashes impacts
    your users’ journey
    ▫ A/B testing / feature flags analytics
    69

    View full-size slide

  68. Going Further
    ▫ Library is Open Source on GitHub
    ▫ Also provides Logging and Tracing
    ▫ More features coming soon…
    70

    View full-size slide

  69. Wrapping up
    4

    View full-size slide

  70. Key takeaways
    “Measure twice,
    cut once.”
    72
    Choose the relevant
    tool to your use case.
    Always analyse your
    measurements before
    taking action.

    View full-size slide

  71. 73
    Useful links
    ▫ https:/
    /ui.perfetto.dev/
    ▫ https:/
    /developer.android.com/studio/profile/benchmark
    ▫ https:/
    /developer.android.com/studio/profile/run-benchmarks-in-ci
    ▫ https:/
    /proandroiddev.com/jetpack-benchmark-on-firebase-test-lab-d1
    4c5eae815f
    ▫ https:/
    /github.com/DataDog/dd-sdk-android
    ▫ https:/
    /docs.datadoghq.com/real_user_monitoring/android/

    View full-size slide

  72. 74
    Thank You!
    Any questions?
    ▫ @xgouchet
    ▫ @datadoghq
    @theunsteady5

    View full-size slide

  73. CREDITS
    Special thanks to all the people who made and released
    these awesome resources for free:
    ▫ Presentation template by SlidesCarnival
    ▫ Photographs from Unsplash
    75

    View full-size slide