Benchmarking your application’s performance

Benchmarking your application’s performance

Back in 1974, Donald Knuth said that “premature optimization is the root of all evil (or at least most of it) in programming”. But optimization without benchmarks can be even worse than that. It’s like following a map without using a compass, and just following your instinct that you’re going in the right direction. Whether it is during development or at runtime, many tools exist to let you measure the performance of your applications. These measures can then be used to prioritize which parts to optimize first, as well as measure the performances gained (or lost) after a refactoring.

Cf95f93e78f6d6dd0630049396f723c6?s=128

Xavier Gouchet

April 20, 2020
Tweet

Transcript

  1. Benchmarking your app’s performance Xavier F. Gouchet, Senior Software Engineer

    @xgouchet (Twitter, GitHub, SpeakerDeck)
  2. None
  3. Introduction A few words about performance 0

  4. performance /pəˈfɔːməns/ noun 1. how well a computer, machine, application,

    etc. works. 4
  5. Mobile app performance is still critical today - Low end

    devices - Slow networks - Short attention span 5 @priscilladupreez
  6. Performance (like Android) are fragmented If you develop on a

    Pixel 4, make sure your app also works correctly on a Samsung Galaxy S7 (2016) 6
  7. Performance must be measured You can’t just base your roadmap

    decision on “a hunch” 7 @youngprodigy3
  8. User will tell you it feels slow Measurements will tell

    you what part exactly is slow. #MeasurementsMatter Changes must be measured Measurements will tell you how much your PR improves (or degrades) your feature’s performance 8
  9. “Premature optimization is the root of all evil (or at

    least most of it) in programming.” — Donald Knuth Don’t optimize early, but optimize eventually. 9
  10. 10 Making sure the code works - Unit Testing -

    Instrumented Testing - Manual Testing - Mutation Testing - … In your toolbox Making sure it works fast - ???
  11. Developer Tools Get a snapshot of your immediate performance 1

  12. Overview Get a full view of: - network - cpu

    - heap - rendering - … 12 @kmuza
  13. 13 Tools Android Studio Profiler Systrace Perfetto

  14. 14 Android Profiler

  15. 15 Android Profiler

  16. Launching systrace $ systrace.py 16

  17. Launching systrace $ systrace.py Starting tracing (stop with enter) 17

  18. Launching systrace $ systrace.py Starting tracing (stop with enter) Tracing

    completed. Collecting output... Outputting Systrace results... Tracing complete, writing results 18
  19. 19 Systrace (legacy)

  20. 20 Perfetto (API 28+)

  21. 21

  22. Additionnal info fun methodWithCustomEvents() { Trace.beginSection("MySectionName") // … Trace.endSection() }

    22
  23. Pros ▫ (Almost) no setup required ▫ Get full system

    state info ▫ Controlled scenario and environment 23 Systrace / Perfetto Cons ▫ Performance measurements are skewed ▫ Needs manual trigger ▫ Complex and dense UI/UX
  24. When should you use it ? ▫ When you have

    a specific performance issue ▫ When you want to optimize a specific part of the code ▫ To debug a tricky algorithm (nested loops, concurrency) 24
  25. Going Further ▫ Capture systrace from the device (API 28+)

    25
  26. Benchmark Standardized performance measurement 2

  27. Benchmark Overview Measure timed performances of specific operations 27 @veri_ivanova

  28. Jetpack Benchmark Library dependencies { // … androidTestImplementation "androidx.benchmark:benchmark-junit4:1.0.0" }

    28
  29. Jetpack Benchmark Library android { // … defaultConfig { //

    … testInstrumentationRunner "androidx.benchmark.junit4.AndroidBenchmarkRunner" } } 29
  30. @RunWith(AndroidJUnit4::class) class BenchmarkTest { @get:Rule val benchmarkRule = BenchmarkRule() //

    … } 30 Benchmark Test Class
  31. @RunWith(AndroidJUnit4::class) class BenchmarkTest { @get:Rule val benchmarkRule = BenchmarkRule() //

    … } 31 Benchmark Test Class
  32. @RunWith(AndroidJUnit4::class) class BenchmarkTest { @get:Rule val benchmarkRule = BenchmarkRule() //

    … } 32 Benchmark Test Class
  33. @Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { doSomeWork() } } 33

    Benchmark Test Function
  34. @Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { doSomeWork() } } 34

    Benchmark Test Function
  35. @Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { doSomeWork() } } 35

    Benchmark Test Function
  36. @Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { doSomeWork() } } 36

    Benchmark Test Function
  37. @Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { cleanState() val data =

    createData() doSomeWork(data) } } 37 Benchmark Test Function
  38. @Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { runWithTimingDisabled { cleanState() }

    val data = runWithTimingDisabled { createData() } doSomeWork(data) } } 38 Benchmark Test Function
  39. $ gradlew :benchmark_module:connectedCheck 39 Run the Benchmark

  40. ▫ Stored in a JSON file ▫ Full timing information

    in nanoseconds 40 Benchmark Results
  41. { "name": "benchmarkSomeWork", "className": "com.example.BenchmarkTest", "metrics": { "timeNs": { "minimum":

    2561000, "maximum": 7133000, "median": 3914000, "runs": [ … ] } } 41 Benchmark Result
  42. Running Benchmark in the CI ▫ Emulators are unstable, use

    real devices ▫ Needs some tweaks in your gradle config ▫ Use the JSON to: - Graph the evolution of performance - Fail if the measurements exceed a threshold 42
  43. Pros ▫ Reliable and consistent ▫ Measure just what you

    want ▫ Can be as high or low level as needed 43 Jetpack Benchmark Cons ▫ Needs to be ran on a real device ▫ The feature to be tested must be repeatable and called from code ▫ Takes a long time in CI
  44. When should you use it ? ▫ Monitor critical parts

    of your logic in CI ▫ When refactoring a specific feature ▫ When writing code executed on the main thread 44
  45. Going Further ▫ Official Gradle plugin to lock the CPU

    clocks ▫ Sample Gradle plugin to fail on high measurements ▫ Using Jetpack Benchmark on Firebase Test Lab 45
  46. App Performance Monitoring Live performance from the trenches 3

  47. What about the real users? Get performance and runtime information

    from the production environment 47 @robin_rednine
  48. 48 Datadog SDK for Android Logs Metrics Traces

  49. 49 repositories { maven { url "https://dl.bintray.com/datadog/datadog-maven" } } dependencies

    { implementation "com.datadoghq:dd-sdk-android:1.4.0" implementation "com.datadoghq:dd-sdk-android-ktx:1.4.0" } Using Datadog SDK for Android
  50. 50 Logs ▫ Send structured information about - events -

    measures - state changes
  51. 51 val logger = Logger.Builder().build() logger.i( "Activity init to resume

    took $duration nanos" ) Sending Logs
  52. 52 val logger = Logger.Builder().build() logger.i( "Activity init to resume

    took $duration nanos", attributes = mapOf( "activity.time_to_resume" to duration, "activity.classname" to javaClass.simpleName ) ) Sending Logs
  53. 53

  54. 54 Metrics ▫ Extract metrics information from logs ▫ Visualize

    critical KPIs in a dashboards ▫ Get alerts on spikes / drops
  55. 55

  56. 56 Tracing (beta in 1.4.0) ▫ Get traces across distributed

    architectures ▫ Performance information ▫ Debugging network errors
  57. 57 val okHttpClient = OkHttpClient.Builder() .addInterceptor(TracingInterceptor()) .build() OkHttp Requests

  58. 58

  59. 59 val tracer = GlobalTracer.get() val span = tracer.buildSpan("doSomething").start() doSomething()

    span.finish() Tracing custom methods
  60. 60 val tracer = GlobalTracer.get() val span = tracer.buildSpan("doSomething").start() doSomething()

    span.finish() Tracing custom methods
  61. 61 val tracer = GlobalTracer.get() val span = tracer.buildSpan("doSomething").start() doSomething()

    span.finish() Tracing custom methods
  62. 62 val total = withinSpan("doSomething") { doSomething() } Tracing custom

    methods (Kotlin Extension)
  63. Pros ▫ Real data - real use cases - real

    devices - real network conditions 63 App Performance Monitoring Cons ▫ Need large enough user base to have relevant data ▫ Can’t measure everything
  64. When should you use it ? ▫ Monitor key aspects

    of your app in the wild ▫ Understand how performance impacts your users ▫ A/B testing measures 64
  65. Going Further ▫ Library is Open Source on GitHub ▫

    More features coming soon (Real User Monitoring) 65
  66. Wrapping up 4

  67. Key takeaways “Measure twice, cut once.” 67 Choose the relevant

    tool to your use case. Always analyse your measurements before taking action.
  68. 68 Useful links ▫ https:/ /ui.perfetto.dev/ ▫ https:/ /developer.android.com/studio/profile/benchmark ▫

    https:/ /developer.android.com/studio/profile/run-benchmarks-in-ci ▫ https:/ /proandroiddev.com/jetpack-benchmark-on-firebase-test-lab-d1 4c5eae815f ▫ https:/ /github.com/DataDog/dd-sdk-android ▫ https:/ /docs.datadoghq.com/logs/log_collection/android
  69. 69 Thank You! Any questions? ▫ @xgouchet ▫ @datadoghq @theunsteady5

  70. 70 Office hours @xgouchet ▫ April 20th : 2 -

    3 pm ▫ April 21st - 10 - 11 am - 3 - 4 pm #testing #benchmark #??? @theunsteady5
  71. CREDITS Special thanks to all the people who made and

    released these awesome resources for free: ▫ Presentation template by SlidesCarnival ▫ Photographs from Unsplash 71