Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Benchmarking your app's performance

Benchmarking your app's performance

In 1974, Donald Knuth said that “premature optimization is the root of all evil (or at least most of it) in programming”. But optimizing without benchmarks would be like following your instinct trying to get to your destination with a map but without a compass; you hope that you’re going in the right direction, but could be wasting a lot of time and energy..

Whether it is during development or at runtime, many tools exist to let you measure the performance of your applications. These measures can then be used to prioritize which parts to optimize first, as well as measure the performances gained (or lost) after a refactoring.

Based on various concrete examples, this talk will focus on measuring Android code using the Jetpack Benchmark library, as well as exploring how Datadog’s Android SDK can help measure app performance in real life.

Xavier Gouchet

October 29, 2021

More Decks by Xavier Gouchet

Other Decks in Programming


  1. Benchmarking your app’s performance Xavier F. Gouchet, Senior Software Engineer

    / Team Lead @xgouchet (Twitter, GitHub, SpeakerDeck)
  2. Mobile app performance is still critical today - Low end

    devices - Slow networks - Short attention span 4 @priscilladupreez
  3. Performance (like Android) are fragmented If you develop on a

    Pixel 5, make sure your app also works correctly on a Samsung Galaxy S8 (2017) 5
  4. Performance must be measured You can’t just base your roadmap

    decision on “a hunch” 6 @youngprodigy3
  5. User will tell you it feels slow Measurements will tell

    you what part exactly is slow. #MeasurementsMatter Changes must be measured Measurements will tell you how much your PR improves (or degrades) your feature’s performance 7
  6. 8 Making sure the code works - Unit Testing -

    Instrumented Testing - Manual Testing - <insert testing paradigm> - … In your toolbox Making sure it works fast and efficiently - ???
  7. Immediate snapshot Get a full view of: - network -

    cpu - heap - rendering - … 10 @kmuza
  8. Launching systrace $ systrace.py Starting tracing (stop with enter) Tracing

    completed. Collecting output... Outputting Systrace results... Tracing complete, writing results 21
  9. Pros 👍 ▫ (Almost) no setup required ▫ Get full

    system state info ▫ Controlled scenario and environment 29 Profiler / Systrace / Perfetto Cons 👎 ▫ Performance measurements are skewed ▫ Needs manual trigger ▫ Complex and dense UI/UX
  10. When should you use it ? ▫ When you have

    a specific performance issue ▫ When you want to optimize a specific part of the code ▫ To debug a tricky algorithm (nested loops, concurrency) 30
  11. Jetpack Benchmark Library android { // … defaultConfig { //

    … testInstrumentationRunner "androidx.benchmark.junit4.AndroidBenchmarkRunner" } } 37
  12. @Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { cleanState() val data =

    createData() doSomeWork(data) } } 45 Benchmark Test Function
  13. @Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { runWithTimingDisabled { cleanState() }

    val data = runWithTimingDisabled { createData() } doSomeWork(data) } } 46 Benchmark Test Function
  14. ▫ Stored in a JSON file ▫ Full timing information

    in nanoseconds 48 Benchmark Results
  15. { "name": "benchmarkSomeWork", "className": "com.example.BenchmarkTest", "metrics": { "timeNs": { "minimum":

    2561000, "maximum": 7133000, "median": 3914000, "runs": [ … ] } } 49 Benchmark Result
  16. Running Benchmark in the CI ▫ Emulators are unstable, use

    real devices ▫ Needs some tweaks in your gradle config ▫ Use the JSON to: - Graph the evolution of performance - Fail if the measurements exceed a threshold 50
  17. Pros 👍 ▫ Accurate*, reliable and consistent ▫ Measure just

    what you want ▫ Can be as high or low level as needed 51 Jetpack Benchmark Cons 👎 ▫ *Needs to be ran on a real device ▫ The feature to be tested must be repeatable and called from code ▫ Takes a long time in CI
  18. When should you use it ? ▫ Monitor critical parts

    of your logic in CI ▫ When refactoring a specific feature ▫ When writing code executed on the main thread 52
  19. ▫ Official Gradle plugin to lock the CPU clocks (root)

    ▫ Use CI script to fail on high measurements Going Further 53
  20. What about the real users? Get performance and runtime information

    from the production environment 55 @robin_rednine
  21. Pros 👍 ▫ Real data - real use cases -

    real devices - real network conditions 68 App Performance Monitoring Cons 👎 ▫ Need large enough user base to have relevant data ▫ Can’t measure everything
  22. When should you use it ? ▫ Monitor key aspects

    of your app in the wild ▫ Understand how performance and crashes impacts your users’ journey ▫ A/B testing / feature flags analytics 69
  23. Going Further ▫ Library is Open Source on GitHub ▫

    Also provides Logging and Tracing ▫ More features coming soon… 70
  24. Key takeaways “Measure twice, cut once.” 72 Choose the relevant

    tool to your use case. Always analyse your measurements before taking action.
  25. 73 Useful links ▫ https:/ /ui.perfetto.dev/ ▫ https:/ /developer.android.com/studio/profile/benchmark ▫

    https:/ /developer.android.com/studio/profile/run-benchmarks-in-ci ▫ https:/ /proandroiddev.com/jetpack-benchmark-on-firebase-test-lab-d1 4c5eae815f ▫ https:/ /github.com/DataDog/dd-sdk-android ▫ https:/ /docs.datadoghq.com/real_user_monitoring/android/
  26. CREDITS Special thanks to all the people who made and

    released these awesome resources for free: ▫ Presentation template by SlidesCarnival ▫ Photographs from Unsplash 75