Benchmarking your app's performance

In 1974, Donald Knuth said that “premature optimization is the root of all evil (or at least most of it) in programming”. But optimizing without benchmarks would be like following your instinct trying to get to your destination with a map but without a compass; you hope that you’re going in the right direction, but could be wasting a lot of time and energy..

Whether it is during development or at runtime, many tools exist to let you measure the performance of your applications. These measures can then be used to prioritize which parts to optimize first, as well as measure the performances gained (or lost) after a refactoring.

Based on various concrete examples, this talk will focus on measuring Android code using the Jetpack Benchmark library, as well as exploring how Datadog’s Android SDK can help measure app performance in real life.

Transcript

1. Benchmarking your
   app’s performance
   Xavier F. Gouchet, Senior Software Engineer / Team Lead
   @xgouchet (Twitter, GitHub, SpeakerDeck)
   

   View Slide

2. View Slide

3. Introduction
   A few words about performance
   0
   

   View Slide

4. Mobile app
   performance is still
   critical today
   - Low end devices
   - Slow networks
   - Short attention span
   4
   @priscilladupreez
   

   View Slide

5. Performance
   (like Android)
   are fragmented
   If you develop on a Pixel 5, make sure your app also
   works correctly on a Samsung Galaxy S8 (2017)
   5
   

   View Slide

6. Performance
   must be measured
   You can’t just base your
   roadmap decision on
   “a hunch”
   6
   @youngprodigy3
   

   View Slide

7. User will tell you it feels slow
   Measurements will tell you what
   part exactly is slow.
   #MeasurementsMatter
   Changes must be measured
   Measurements will tell you how
   much your PR improves (or
   degrades) your feature’s
   performance
   7
   

   View Slide

8. 8
   Making sure the code works
   - Unit Testing
   - Instrumented Testing
   - Manual Testing
   -
   - …
   In your toolbox
   Making sure it works fast and
   efficiently
   - ???
   

   View Slide

9. Developer Tools
   Get a snapshot of your immediate performance
   1
   

   View Slide

10. Immediate snapshot
    Get a full view of:
    - network
    - cpu
    - heap
    - rendering
    - …
    10
    @kmuza
    

    View Slide

11. 11
    Tools
    Android Studio Profiler Systrace Perfetto
    

    View Slide

12. 12
    Android Profiler
    

    View Slide

13. 13
    Android Profiler
    Touch
    Activities
    Fragments
    

    View Slide

14. 14
    Android Profiler
    CPU usage
    

    View Slide

15. 15
    Android Profiler
    Memory usage
    

    View Slide

16. 16
    Android Profiler
    Network usage
    

    View Slide

17. 17
    Android Profiler
    Battery impact
    

    View Slide

18. 18
    Android Profiler
    

    View Slide

19. Launching systrace
    $ systrace.py
    19
    

    View Slide

20. Launching systrace
    $ systrace.py
    Starting tracing (stop with enter)
    20
    

    View Slide

21. Launching systrace
    $ systrace.py
    Starting tracing (stop with enter)
    Tracing completed. Collecting output...
    Outputting Systrace results...
    Tracing complete, writing results
    21
    

    View Slide

22. 22
    Systrace (legacy)
    

    View Slide

23. 23
    Perfetto showing systrace result
    

    View Slide

24. 24
    Recording from Perfetto (API 28+)
    

    View Slide

25. 25
    Recording from Perfetto (API 28+)
    

    View Slide

26. fun methodWithCustomEvents() {
    Trace.beginSection("MySectionName")
    // …
    Trace.endSection()
    }
    26
    Custom Traces
    

    View Slide

27. 27
    Recording from Perfetto (API 28+)
    

    View Slide

28. 28
    Recording from Perfetto (API 28+)
    

    View Slide

29. Pros 👍
    ▫ (Almost) no setup required
    ▫ Get full system state info
    ▫ Controlled scenario and
    environment
    29
    Profiler / Systrace / Perfetto
    Cons 👎
    ▫ Performance measurements
    are skewed
    ▫ Needs manual trigger
    ▫ Complex and dense UI/UX
    

    View Slide

30. When should you use it ?
    ▫ When you have a specific performance issue
    ▫ When you want to optimize a specific part of the code
    ▫ To debug a tricky algorithm (nested loops, concurrency)
    30
    

    View Slide

31. Going Further
    ▫ Capture systrace from the device (API 28+)
    31
    

    View Slide

32. 32
    Capturing traces from the developer settings
    

    View Slide

33. Benchmark
    Standardized performance measurement
    2
    

    View Slide

34. Benchmark Overview
    Measure timed performances
    of specific operations
    34
    @veri_ivanova
    

    View Slide

35. 35
    Tools
    Jetpack Benchmark CI Provider
    

    View Slide

36. Jetpack Benchmark Library
    dependencies {
    // …
    androidTestImplementation
    "androidx.benchmark:benchmark-junit4:1.0.0"
    }
    36
    

    View Slide

37. Jetpack Benchmark Library
    android {
    // …
    defaultConfig {
    // …
    testInstrumentationRunner
    "androidx.benchmark.junit4.AndroidBenchmarkRunner"
    }
    }
    37
    

    View Slide

38. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    38
    Benchmark Test Class
    

    View Slide

39. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    39
    Benchmark Test Class
    

    View Slide

40. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    40
    Benchmark Test Class
    

    View Slide

41. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    41
    Benchmark Test Function
    

    View Slide

42. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    42
    Benchmark Test Function
    

    View Slide

43. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    43
    Benchmark Test Function
    

    View Slide

44. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    44
    Benchmark Test Function
    

    View Slide

45. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    cleanState()
    val data = createData()
    doSomeWork(data)
    }
    }
    45
    Benchmark Test Function
    

    View Slide

46. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    runWithTimingDisabled { cleanState() }
    val data = runWithTimingDisabled { createData() }
    doSomeWork(data)
    }
    }
    46
    Benchmark Test Function
    

    View Slide

47. $ gradlew :benchmark_module:connectedCheck
    47
    Run the Benchmark
    

    View Slide

48. ▫ Stored in a JSON file
    ▫ Full timing information in nanoseconds
    48
    Benchmark Results
    

    View Slide

49. {
    "name": "benchmarkSomeWork",
    "className": "com.example.BenchmarkTest",
    "metrics": {
    "timeNs": {
    "minimum": 2561000,
    "maximum": 7133000,
    "median": 3914000,
    "runs": [ … ]
    }
    }
    49
    Benchmark Result
    

    View Slide

50. Running Benchmark in the CI
    ▫ Emulators are unstable, use real devices
    ▫ Needs some tweaks in your gradle config
    ▫ Use the JSON to:
    - Graph the evolution of performance
    - Fail if the measurements exceed a threshold
    50
    

    View Slide

51. Pros 👍
    ▫ Accurate*, reliable and
    consistent
    ▫ Measure just what you want
    ▫ Can be as high or low level as
    needed
    51
    Jetpack Benchmark
    Cons 👎
    ▫ *Needs to be ran on a real
    device
    ▫ The feature to be tested
    must be repeatable and
    called from code
    ▫ Takes a long time in CI
    

    View Slide

52. When should you use it ?
    ▫ Monitor critical parts of your logic in CI
    ▫ When refactoring a specific feature
    ▫ When writing code executed on the main thread
    52
    

    View Slide

53. ▫ Official Gradle plugin to lock the CPU clocks (root)
    ▫ Use CI script to fail on high measurements
    Going Further
    53
    

    View Slide

54. Real User Monitoring
    Live performance from the trenches
    3
    

    View Slide

55. What about the real
    users?
    Get performance and runtime
    information from the
    production environment
    55
    @robin_rednine
    

    View Slide

56. 56
    Tools
    Real User Monitoring Error Tracking Android Vitals
    

    View Slide

57. 57
    dependencies {
    implementation "com.datadoghq:dd-sdk-android:1.10.0"
    implementation "com.datadoghq:dd-sdk-android-ktx:1.10.0"
    // optional extensions: RxJava, Android NDK, …
    }
    Using Datadog SDK for Android
    

    View Slide

58. Single View Performance
    58
    

    View Slide

59. Funnels
    59
    

    View Slide

60. Network Performance
    60
    

    View Slide

61. Mobile Vitals: Refresh Rate
    61
    

    View Slide

62. Mobile Vitals: CPU usage
    62
    

    View Slide

63. Mobile Vitals: RAM consumption
    63
    

    View Slide

64. Application not responding
    64
    

    View Slide

65. Application not responding
    65
    

    View Slide

66. Error Tracking
    66
    

    View Slide

67. Performance monitoring
    67
    

    View Slide

68. Pros 👍
    ▫ Real data
    - real use cases
    - real devices
    - real network conditions
    68
    App Performance Monitoring
    Cons 👎
    ▫ Need large enough user base
    to have relevant data
    ▫ Can’t measure everything
    

    View Slide

69. When should you use it ?
    ▫ Monitor key aspects of your app in the wild
    ▫ Understand how performance and crashes impacts
    your users’ journey
    ▫ A/B testing / feature flags analytics
    69
    

    View Slide

70. Going Further
    ▫ Library is Open Source on GitHub
    ▫ Also provides Logging and Tracing
    ▫ More features coming soon…
    70
    

    View Slide

71. Wrapping up
    4
    

    View Slide

72. Key takeaways
    “Measure twice,
    cut once.”
    72
    Choose the relevant
    tool to your use case.
    Always analyse your
    measurements before
    taking action.
    

    View Slide

73. 73
    Useful links
    ▫ https:/
    /ui.perfetto.dev/
    ▫ https:/
    /developer.android.com/studio/profile/benchmark
    ▫ https:/
    /developer.android.com/studio/profile/run-benchmarks-in-ci
    ▫ https:/
    /proandroiddev.com/jetpack-benchmark-on-firebase-test-lab-d1
    4c5eae815f
    ▫ https:/
    /github.com/DataDog/dd-sdk-android
    ▫ https:/
    /docs.datadoghq.com/real_user_monitoring/android/
    

    View Slide

74. 74
    Thank You!
    Any questions?
    ▫ @xgouchet
    ▫ @datadoghq
    @theunsteady5
    

    View Slide

75. CREDITS
    Special thanks to all the people who made and released
    these awesome resources for free:
    ▫ Presentation template by SlidesCarnival
    ▫ Photographs from Unsplash
    75
    

    View Slide