$30 off During Our Annual Pro Sale. View Details »

Benchmarking your application’s performance

Benchmarking your application’s performance

Back in 1974, Donald Knuth said that “premature optimization is the root of all evil (or at least most of it) in programming”. But optimization without benchmarks can be even worse than that. It’s like following a map without using a compass, and just following your instinct that you’re going in the right direction. Whether it is during development or at runtime, many tools exist to let you measure the performance of your applications. These measures can then be used to prioritize which parts to optimize first, as well as measure the performances gained (or lost) after a refactoring.

Xavier Gouchet

April 20, 2020
Tweet

More Decks by Xavier Gouchet

Other Decks in Programming

Transcript

  1. Benchmarking your
    app’s performance
    Xavier F. Gouchet, Senior Software Engineer
    @xgouchet (Twitter, GitHub, SpeakerDeck)

    View Slide

  2. View Slide

  3. Introduction
    A few words about performance
    0

    View Slide

  4. performance /pəˈfɔːməns/ noun
    1. how well a computer, machine,
    application, etc. works.
    4

    View Slide

  5. Mobile app
    performance is still
    critical today
    - Low end devices
    - Slow networks
    - Short attention span
    5
    @priscilladupreez

    View Slide

  6. Performance
    (like Android)
    are fragmented
    If you develop on a Pixel 4, make sure your app also
    works correctly on a Samsung Galaxy S7 (2016)
    6

    View Slide

  7. Performance
    must be measured
    You can’t just base your
    roadmap decision on
    “a hunch”
    7
    @youngprodigy3

    View Slide

  8. User will tell you it feels slow
    Measurements will tell you what
    part exactly is slow.
    #MeasurementsMatter
    Changes must be measured
    Measurements will tell you how
    much your PR improves (or
    degrades) your feature’s
    performance
    8

    View Slide

  9. “Premature optimization is the root of all evil
    (or at least most of it) in programming.”
    — Donald Knuth
    Don’t optimize early,
    but optimize eventually.
    9

    View Slide

  10. 10
    Making sure the code works
    - Unit Testing
    - Instrumented Testing
    - Manual Testing
    - Mutation Testing
    - …
    In your toolbox
    Making sure it works fast
    - ???

    View Slide

  11. Developer Tools
    Get a snapshot of your immediate performance
    1

    View Slide

  12. Overview
    Get a full view of:
    - network
    - cpu
    - heap
    - rendering
    - …
    12
    @kmuza

    View Slide

  13. 13
    Tools
    Android Studio Profiler Systrace Perfetto

    View Slide

  14. 14
    Android Profiler

    View Slide

  15. 15
    Android Profiler

    View Slide

  16. Launching systrace
    $ systrace.py
    16

    View Slide

  17. Launching systrace
    $ systrace.py
    Starting tracing (stop with enter)
    17

    View Slide

  18. Launching systrace
    $ systrace.py
    Starting tracing (stop with enter)
    Tracing completed. Collecting output...
    Outputting Systrace results...
    Tracing complete, writing results
    18

    View Slide

  19. 19
    Systrace (legacy)

    View Slide

  20. 20
    Perfetto (API 28+)

    View Slide

  21. 21

    View Slide

  22. Additionnal info
    fun methodWithCustomEvents() {
    Trace.beginSection("MySectionName")
    // …
    Trace.endSection()
    }
    22

    View Slide

  23. Pros
    ▫ (Almost) no setup required
    ▫ Get full system state info
    ▫ Controlled scenario and
    environment
    23
    Systrace / Perfetto
    Cons
    ▫ Performance measurements
    are skewed
    ▫ Needs manual trigger
    ▫ Complex and dense UI/UX

    View Slide

  24. When should you use it ?
    ▫ When you have a specific performance issue
    ▫ When you want to optimize a specific part of the code
    ▫ To debug a tricky algorithm (nested loops, concurrency)
    24

    View Slide

  25. Going Further
    ▫ Capture systrace from the device (API 28+)
    25

    View Slide

  26. Benchmark
    Standardized performance measurement
    2

    View Slide

  27. Benchmark Overview
    Measure timed performances
    of specific operations
    27
    @veri_ivanova

    View Slide

  28. Jetpack Benchmark Library
    dependencies {
    // …
    androidTestImplementation
    "androidx.benchmark:benchmark-junit4:1.0.0"
    }
    28

    View Slide

  29. Jetpack Benchmark Library
    android {
    // …
    defaultConfig {
    // …
    testInstrumentationRunner
    "androidx.benchmark.junit4.AndroidBenchmarkRunner"
    }
    }
    29

    View Slide

  30. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    30
    Benchmark Test Class

    View Slide

  31. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    31
    Benchmark Test Class

    View Slide

  32. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    32
    Benchmark Test Class

    View Slide

  33. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    33
    Benchmark Test Function

    View Slide

  34. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    34
    Benchmark Test Function

    View Slide

  35. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    35
    Benchmark Test Function

    View Slide

  36. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    36
    Benchmark Test Function

    View Slide

  37. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    cleanState()
    val data = createData()
    doSomeWork(data)
    }
    }
    37
    Benchmark Test Function

    View Slide

  38. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    runWithTimingDisabled { cleanState() }
    val data = runWithTimingDisabled { createData() }
    doSomeWork(data)
    }
    }
    38
    Benchmark Test Function

    View Slide

  39. $ gradlew :benchmark_module:connectedCheck
    39
    Run the Benchmark

    View Slide

  40. ▫ Stored in a JSON file
    ▫ Full timing information in nanoseconds
    40
    Benchmark Results

    View Slide

  41. {
    "name": "benchmarkSomeWork",
    "className": "com.example.BenchmarkTest",
    "metrics": {
    "timeNs": {
    "minimum": 2561000,
    "maximum": 7133000,
    "median": 3914000,
    "runs": [ … ]
    }
    }
    41
    Benchmark Result

    View Slide

  42. Running Benchmark in the CI
    ▫ Emulators are unstable, use real devices
    ▫ Needs some tweaks in your gradle config
    ▫ Use the JSON to:
    - Graph the evolution of performance
    - Fail if the measurements exceed a threshold
    42

    View Slide

  43. Pros
    ▫ Reliable and consistent
    ▫ Measure just what you want
    ▫ Can be as high or low level as
    needed
    43
    Jetpack Benchmark
    Cons
    ▫ Needs to be ran on a real
    device
    ▫ The feature to be tested
    must be repeatable and
    called from code
    ▫ Takes a long time in CI

    View Slide

  44. When should you use it ?
    ▫ Monitor critical parts of your logic in CI
    ▫ When refactoring a specific feature
    ▫ When writing code executed on the main thread
    44

    View Slide

  45. Going Further
    ▫ Official Gradle plugin to lock the CPU clocks
    ▫ Sample Gradle plugin to fail on high measurements
    ▫ Using Jetpack Benchmark on Firebase Test Lab
    45

    View Slide

  46. App Performance Monitoring
    Live performance from the trenches
    3

    View Slide

  47. What about the real
    users?
    Get performance and runtime
    information from the
    production environment
    47
    @robin_rednine

    View Slide

  48. 48
    Datadog SDK for Android
    Logs Metrics Traces

    View Slide

  49. 49
    repositories {
    maven { url "https://dl.bintray.com/datadog/datadog-maven" }
    }
    dependencies {
    implementation "com.datadoghq:dd-sdk-android:1.4.0"
    implementation "com.datadoghq:dd-sdk-android-ktx:1.4.0"
    }
    Using Datadog SDK for Android

    View Slide

  50. 50
    Logs
    ▫ Send structured information about
    - events
    - measures
    - state changes

    View Slide

  51. 51
    val logger = Logger.Builder().build()
    logger.i(
    "Activity init to resume took $duration nanos"
    )
    Sending Logs

    View Slide

  52. 52
    val logger = Logger.Builder().build()
    logger.i(
    "Activity init to resume took $duration nanos",
    attributes = mapOf(
    "activity.time_to_resume" to duration,
    "activity.classname" to javaClass.simpleName
    )
    )
    Sending Logs

    View Slide

  53. 53

    View Slide

  54. 54
    Metrics
    ▫ Extract metrics information from logs
    ▫ Visualize critical KPIs in a dashboards
    ▫ Get alerts on spikes / drops

    View Slide

  55. 55

    View Slide

  56. 56
    Tracing (beta in 1.4.0)
    ▫ Get traces across distributed architectures
    ▫ Performance information
    ▫ Debugging network errors

    View Slide

  57. 57
    val okHttpClient = OkHttpClient.Builder()
    .addInterceptor(TracingInterceptor())
    .build()
    OkHttp Requests

    View Slide

  58. 58

    View Slide

  59. 59
    val tracer = GlobalTracer.get()
    val span = tracer.buildSpan("doSomething").start()
    doSomething()
    span.finish()
    Tracing custom methods

    View Slide

  60. 60
    val tracer = GlobalTracer.get()
    val span = tracer.buildSpan("doSomething").start()
    doSomething()
    span.finish()
    Tracing custom methods

    View Slide

  61. 61
    val tracer = GlobalTracer.get()
    val span = tracer.buildSpan("doSomething").start()
    doSomething()
    span.finish()
    Tracing custom methods

    View Slide

  62. 62
    val total = withinSpan("doSomething") {
    doSomething()
    }
    Tracing custom methods (Kotlin Extension)

    View Slide

  63. Pros
    ▫ Real data
    - real use cases
    - real devices
    - real network conditions
    63
    App Performance Monitoring
    Cons
    ▫ Need large enough user base
    to have relevant data
    ▫ Can’t measure everything

    View Slide

  64. When should you use it ?
    ▫ Monitor key aspects of your app in the wild
    ▫ Understand how performance impacts your users
    ▫ A/B testing measures
    64

    View Slide

  65. Going Further
    ▫ Library is Open Source on GitHub
    ▫ More features coming soon (Real User Monitoring)
    65

    View Slide

  66. Wrapping up
    4

    View Slide

  67. Key takeaways
    “Measure twice,
    cut once.”
    67
    Choose the relevant
    tool to your use case.
    Always analyse your
    measurements before
    taking action.

    View Slide

  68. 68
    Useful links
    ▫ https:/
    /ui.perfetto.dev/
    ▫ https:/
    /developer.android.com/studio/profile/benchmark
    ▫ https:/
    /developer.android.com/studio/profile/run-benchmarks-in-ci
    ▫ https:/
    /proandroiddev.com/jetpack-benchmark-on-firebase-test-lab-d1
    4c5eae815f
    ▫ https:/
    /github.com/DataDog/dd-sdk-android
    ▫ https:/
    /docs.datadoghq.com/logs/log_collection/android

    View Slide

  69. 69
    Thank You!
    Any questions?
    ▫ @xgouchet
    ▫ @datadoghq
    @theunsteady5

    View Slide

  70. 70
    Office hours
    @xgouchet
    ▫ April 20th : 2 - 3 pm
    ▫ April 21st
    - 10 - 11 am
    - 3 - 4 pm
    #testing #benchmark #???
    @theunsteady5

    View Slide

  71. CREDITS
    Special thanks to all the people who made and released
    these awesome resources for free:
    ▫ Presentation template by SlidesCarnival
    ▫ Photographs from Unsplash
    71

    View Slide