Benchmarking your application’s performance

Back in 1974, Donald Knuth said that “premature optimization is the root of all evil (or at least most of it) in programming”. But optimization without benchmarks can be even worse than that. It’s like following a map without using a compass, and just following your instinct that you’re going in the right direction. Whether it is during development or at runtime, many tools exist to let you measure the performance of your applications. These measures can then be used to prioritize which parts to optimize first, as well as measure the performances gained (or lost) after a refactoring.

Transcript

1. Benchmarking your
   app’s performance
   Xavier F. Gouchet, Senior Software Engineer
   @xgouchet (Twitter, GitHub, SpeakerDeck)
   

   View Slide

2. View Slide

3. Introduction
   A few words about performance
   0
   

   View Slide

4. performance /pəˈfɔːməns/ noun
   1. how well a computer, machine,
   application, etc. works.
   4
   

   View Slide

5. Mobile app
   performance is still
   critical today
   - Low end devices
   - Slow networks
   - Short attention span
   5
   @priscilladupreez
   

   View Slide

6. Performance
   (like Android)
   are fragmented
   If you develop on a Pixel 4, make sure your app also
   works correctly on a Samsung Galaxy S7 (2016)
   6
   

   View Slide

7. Performance
   must be measured
   You can’t just base your
   roadmap decision on
   “a hunch”
   7
   @youngprodigy3
   

   View Slide

8. User will tell you it feels slow
   Measurements will tell you what
   part exactly is slow.
   #MeasurementsMatter
   Changes must be measured
   Measurements will tell you how
   much your PR improves (or
   degrades) your feature’s
   performance
   8
   

   View Slide

9. “Premature optimization is the root of all evil
   (or at least most of it) in programming.”
   — Donald Knuth
   Don’t optimize early,
   but optimize eventually.
   9
   

   View Slide

10. 10
    Making sure the code works
    - Unit Testing
    - Instrumented Testing
    - Manual Testing
    - Mutation Testing
    - …
    In your toolbox
    Making sure it works fast
    - ???
    

    View Slide

11. Developer Tools
    Get a snapshot of your immediate performance
    1
    

    View Slide

12. Overview
    Get a full view of:
    - network
    - cpu
    - heap
    - rendering
    - …
    12
    @kmuza
    

    View Slide

13. 13
    Tools
    Android Studio Profiler Systrace Perfetto
    

    View Slide

14. 14
    Android Profiler
    

    View Slide

15. 15
    Android Profiler
    

    View Slide

16. Launching systrace
    $ systrace.py
    16
    

    View Slide

17. Launching systrace
    $ systrace.py
    Starting tracing (stop with enter)
    17
    

    View Slide

18. Launching systrace
    $ systrace.py
    Starting tracing (stop with enter)
    Tracing completed. Collecting output...
    Outputting Systrace results...
    Tracing complete, writing results
    18
    

    View Slide

19. 19
    Systrace (legacy)
    

    View Slide

20. 20
    Perfetto (API 28+)
    

    View Slide

21. View Slide

22. Additionnal info
    fun methodWithCustomEvents() {
    Trace.beginSection("MySectionName")
    // …
    Trace.endSection()
    }
    22
    

    View Slide

23. Pros
    ▫ (Almost) no setup required
    ▫ Get full system state info
    ▫ Controlled scenario and
    environment
    23
    Systrace / Perfetto
    Cons
    ▫ Performance measurements
    are skewed
    ▫ Needs manual trigger
    ▫ Complex and dense UI/UX
    

    View Slide

24. When should you use it ?
    ▫ When you have a specific performance issue
    ▫ When you want to optimize a specific part of the code
    ▫ To debug a tricky algorithm (nested loops, concurrency)
    24
    

    View Slide

25. Going Further
    ▫ Capture systrace from the device (API 28+)
    25
    

    View Slide

26. Benchmark
    Standardized performance measurement
    2
    

    View Slide

27. Benchmark Overview
    Measure timed performances
    of specific operations
    27
    @veri_ivanova
    

    View Slide

28. Jetpack Benchmark Library
    dependencies {
    // …
    androidTestImplementation
    "androidx.benchmark:benchmark-junit4:1.0.0"
    }
    28
    

    View Slide

29. Jetpack Benchmark Library
    android {
    // …
    defaultConfig {
    // …
    testInstrumentationRunner
    "androidx.benchmark.junit4.AndroidBenchmarkRunner"
    }
    }
    29
    

    View Slide

30. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    30
    Benchmark Test Class
    

    View Slide

31. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    31
    Benchmark Test Class
    

    View Slide

32. @RunWith(AndroidJUnit4::class)
    class BenchmarkTest {
    @get:Rule
    val benchmarkRule = BenchmarkRule()
    // …
    }
    32
    Benchmark Test Class
    

    View Slide

33. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    33
    Benchmark Test Function
    

    View Slide

34. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    34
    Benchmark Test Function
    

    View Slide

35. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    35
    Benchmark Test Function
    

    View Slide

36. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    doSomeWork()
    }
    }
    36
    Benchmark Test Function
    

    View Slide

37. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    cleanState()
    val data = createData()
    doSomeWork(data)
    }
    }
    37
    Benchmark Test Function
    

    View Slide

38. @Test
    fun benchmarkSomeWork() {
    benchmarkRule.measureRepeated {
    runWithTimingDisabled { cleanState() }
    val data = runWithTimingDisabled { createData() }
    doSomeWork(data)
    }
    }
    38
    Benchmark Test Function
    

    View Slide

39. $ gradlew :benchmark_module:connectedCheck
    39
    Run the Benchmark
    

    View Slide

40. ▫ Stored in a JSON file
    ▫ Full timing information in nanoseconds
    40
    Benchmark Results
    

    View Slide

41. {
    "name": "benchmarkSomeWork",
    "className": "com.example.BenchmarkTest",
    "metrics": {
    "timeNs": {
    "minimum": 2561000,
    "maximum": 7133000,
    "median": 3914000,
    "runs": [ … ]
    }
    }
    41
    Benchmark Result
    

    View Slide

42. Running Benchmark in the CI
    ▫ Emulators are unstable, use real devices
    ▫ Needs some tweaks in your gradle config
    ▫ Use the JSON to:
    - Graph the evolution of performance
    - Fail if the measurements exceed a threshold
    42
    

    View Slide

43. Pros
    ▫ Reliable and consistent
    ▫ Measure just what you want
    ▫ Can be as high or low level as
    needed
    43
    Jetpack Benchmark
    Cons
    ▫ Needs to be ran on a real
    device
    ▫ The feature to be tested
    must be repeatable and
    called from code
    ▫ Takes a long time in CI
    

    View Slide

44. When should you use it ?
    ▫ Monitor critical parts of your logic in CI
    ▫ When refactoring a specific feature
    ▫ When writing code executed on the main thread
    44
    

    View Slide

45. Going Further
    ▫ Official Gradle plugin to lock the CPU clocks
    ▫ Sample Gradle plugin to fail on high measurements
    ▫ Using Jetpack Benchmark on Firebase Test Lab
    45
    

    View Slide

46. App Performance Monitoring
    Live performance from the trenches
    3
    

    View Slide

47. What about the real
    users?
    Get performance and runtime
    information from the
    production environment
    47
    @robin_rednine
    

    View Slide

48. 48
    Datadog SDK for Android
    Logs Metrics Traces
    

    View Slide

49. 49
    repositories {
    maven { url "https://dl.bintray.com/datadog/datadog-maven" }
    }
    dependencies {
    implementation "com.datadoghq:dd-sdk-android:1.4.0"
    implementation "com.datadoghq:dd-sdk-android-ktx:1.4.0"
    }
    Using Datadog SDK for Android
    

    View Slide

50. 50
    Logs
    ▫ Send structured information about
    - events
    - measures
    - state changes
    

    View Slide

51. 51
    val logger = Logger.Builder().build()
    logger.i(
    "Activity init to resume took $duration nanos"
    )
    Sending Logs
    

    View Slide

52. 52
    val logger = Logger.Builder().build()
    logger.i(
    "Activity init to resume took $duration nanos",
    attributes = mapOf(
    "activity.time_to_resume" to duration,
    "activity.classname" to javaClass.simpleName
    )
    )
    Sending Logs
    

    View Slide

53. View Slide

54. 54
    Metrics
    ▫ Extract metrics information from logs
    ▫ Visualize critical KPIs in a dashboards
    ▫ Get alerts on spikes / drops
    

    View Slide

55. View Slide

56. 56
    Tracing (beta in 1.4.0)
    ▫ Get traces across distributed architectures
    ▫ Performance information
    ▫ Debugging network errors
    

    View Slide

57. 57
    val okHttpClient = OkHttpClient.Builder()
    .addInterceptor(TracingInterceptor())
    .build()
    OkHttp Requests
    

    View Slide

58. View Slide

59. 59
    val tracer = GlobalTracer.get()
    val span = tracer.buildSpan("doSomething").start()
    doSomething()
    span.finish()
    Tracing custom methods
    

    View Slide

60. 60
    val tracer = GlobalTracer.get()
    val span = tracer.buildSpan("doSomething").start()
    doSomething()
    span.finish()
    Tracing custom methods
    

    View Slide

61. 61
    val tracer = GlobalTracer.get()
    val span = tracer.buildSpan("doSomething").start()
    doSomething()
    span.finish()
    Tracing custom methods
    

    View Slide

62. 62
    val total = withinSpan("doSomething") {
    doSomething()
    }
    Tracing custom methods (Kotlin Extension)
    

    View Slide

63. Pros
    ▫ Real data
    - real use cases
    - real devices
    - real network conditions
    63
    App Performance Monitoring
    Cons
    ▫ Need large enough user base
    to have relevant data
    ▫ Can’t measure everything
    

    View Slide

64. When should you use it ?
    ▫ Monitor key aspects of your app in the wild
    ▫ Understand how performance impacts your users
    ▫ A/B testing measures
    64
    

    View Slide

65. Going Further
    ▫ Library is Open Source on GitHub
    ▫ More features coming soon (Real User Monitoring)
    65
    

    View Slide

66. Wrapping up
    4
    

    View Slide

67. Key takeaways
    “Measure twice,
    cut once.”
    67
    Choose the relevant
    tool to your use case.
    Always analyse your
    measurements before
    taking action.
    

    View Slide

68. 68
    Useful links
    ▫ https:/
    /ui.perfetto.dev/
    ▫ https:/
    /developer.android.com/studio/profile/benchmark
    ▫ https:/
    /developer.android.com/studio/profile/run-benchmarks-in-ci
    ▫ https:/
    /proandroiddev.com/jetpack-benchmark-on-firebase-test-lab-d1
    4c5eae815f
    ▫ https:/
    /github.com/DataDog/dd-sdk-android
    ▫ https:/
    /docs.datadoghq.com/logs/log_collection/android
    

    View Slide

69. 69
    Thank You!
    Any questions?
    ▫ @xgouchet
    ▫ @datadoghq
    @theunsteady5
    

    View Slide

70. 70
    Office hours
    @xgouchet
    ▫ April 20th : 2 - 3 pm
    ▫ April 21st
    - 10 - 11 am
    - 3 - 4 pm
    #testing #benchmark #???
    @theunsteady5
    

    View Slide

71. CREDITS
    Special thanks to all the people who made and released
    these awesome resources for free:
    ▫ Presentation template by SlidesCarnival
    ▫ Photographs from Unsplash
    71
    

    View Slide