Slide 1

Slide 1 text

Benchmarking your app’s performance Xavier F. Gouchet, Senior Software Engineer @xgouchet (Twitter, GitHub, SpeakerDeck)

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Introduction A few words about performance 0

Slide 4

Slide 4 text

performance /pəˈfɔːməns/ noun 1. how well a computer, machine, application, etc. works. 4

Slide 5

Slide 5 text

Mobile app performance is still critical today - Low end devices - Slow networks - Short attention span 5 @priscilladupreez

Slide 6

Slide 6 text

Performance (like Android) are fragmented If you develop on a Pixel 4, make sure your app also works correctly on a Samsung Galaxy S7 (2016) 6

Slide 7

Slide 7 text

Performance must be measured You can’t just base your roadmap decision on “a hunch” 7 @youngprodigy3

Slide 8

Slide 8 text

User will tell you it feels slow Measurements will tell you what part exactly is slow. #MeasurementsMatter Changes must be measured Measurements will tell you how much your PR improves (or degrades) your feature’s performance 8

Slide 9

Slide 9 text

“Premature optimization is the root of all evil (or at least most of it) in programming.” — Donald Knuth Don’t optimize early, but optimize eventually. 9

Slide 10

Slide 10 text

10 Making sure the code works - Unit Testing - Instrumented Testing - Manual Testing - Mutation Testing - … In your toolbox Making sure it works fast - ???

Slide 11

Slide 11 text

Developer Tools Get a snapshot of your immediate performance 1

Slide 12

Slide 12 text

Overview Get a full view of: - network - cpu - heap - rendering - … 12 @kmuza

Slide 13

Slide 13 text

13 Tools Android Studio Profiler Systrace Perfetto

Slide 14

Slide 14 text

14 Android Profiler

Slide 15

Slide 15 text

15 Android Profiler

Slide 16

Slide 16 text

Launching systrace $ systrace.py 16

Slide 17

Slide 17 text

Launching systrace $ systrace.py Starting tracing (stop with enter) 17

Slide 18

Slide 18 text

Launching systrace $ systrace.py Starting tracing (stop with enter) Tracing completed. Collecting output... Outputting Systrace results... Tracing complete, writing results 18

Slide 19

Slide 19 text

19 Systrace (legacy)

Slide 20

Slide 20 text

20 Perfetto (API 28+)

Slide 21

Slide 21 text

21

Slide 22

Slide 22 text

Additionnal info fun methodWithCustomEvents() { Trace.beginSection("MySectionName") // … Trace.endSection() } 22

Slide 23

Slide 23 text

Pros ▫ (Almost) no setup required ▫ Get full system state info ▫ Controlled scenario and environment 23 Systrace / Perfetto Cons ▫ Performance measurements are skewed ▫ Needs manual trigger ▫ Complex and dense UI/UX

Slide 24

Slide 24 text

When should you use it ? ▫ When you have a specific performance issue ▫ When you want to optimize a specific part of the code ▫ To debug a tricky algorithm (nested loops, concurrency) 24

Slide 25

Slide 25 text

Going Further ▫ Capture systrace from the device (API 28+) 25

Slide 26

Slide 26 text

Benchmark Standardized performance measurement 2

Slide 27

Slide 27 text

Benchmark Overview Measure timed performances of specific operations 27 @veri_ivanova

Slide 28

Slide 28 text

Jetpack Benchmark Library dependencies { // … androidTestImplementation "androidx.benchmark:benchmark-junit4:1.0.0" } 28

Slide 29

Slide 29 text

Jetpack Benchmark Library android { // … defaultConfig { // … testInstrumentationRunner "androidx.benchmark.junit4.AndroidBenchmarkRunner" } } 29

Slide 30

Slide 30 text

@RunWith(AndroidJUnit4::class) class BenchmarkTest { @get:Rule val benchmarkRule = BenchmarkRule() // … } 30 Benchmark Test Class

Slide 31

Slide 31 text

@RunWith(AndroidJUnit4::class) class BenchmarkTest { @get:Rule val benchmarkRule = BenchmarkRule() // … } 31 Benchmark Test Class

Slide 32

Slide 32 text

@RunWith(AndroidJUnit4::class) class BenchmarkTest { @get:Rule val benchmarkRule = BenchmarkRule() // … } 32 Benchmark Test Class

Slide 33

Slide 33 text

@Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { doSomeWork() } } 33 Benchmark Test Function

Slide 34

Slide 34 text

@Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { doSomeWork() } } 34 Benchmark Test Function

Slide 35

Slide 35 text

@Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { doSomeWork() } } 35 Benchmark Test Function

Slide 36

Slide 36 text

@Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { doSomeWork() } } 36 Benchmark Test Function

Slide 37

Slide 37 text

@Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { cleanState() val data = createData() doSomeWork(data) } } 37 Benchmark Test Function

Slide 38

Slide 38 text

@Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { runWithTimingDisabled { cleanState() } val data = runWithTimingDisabled { createData() } doSomeWork(data) } } 38 Benchmark Test Function

Slide 39

Slide 39 text

$ gradlew :benchmark_module:connectedCheck 39 Run the Benchmark

Slide 40

Slide 40 text

▫ Stored in a JSON file ▫ Full timing information in nanoseconds 40 Benchmark Results

Slide 41

Slide 41 text

{ "name": "benchmarkSomeWork", "className": "com.example.BenchmarkTest", "metrics": { "timeNs": { "minimum": 2561000, "maximum": 7133000, "median": 3914000, "runs": [ … ] } } 41 Benchmark Result

Slide 42

Slide 42 text

Running Benchmark in the CI ▫ Emulators are unstable, use real devices ▫ Needs some tweaks in your gradle config ▫ Use the JSON to: - Graph the evolution of performance - Fail if the measurements exceed a threshold 42

Slide 43

Slide 43 text

Pros ▫ Reliable and consistent ▫ Measure just what you want ▫ Can be as high or low level as needed 43 Jetpack Benchmark Cons ▫ Needs to be ran on a real device ▫ The feature to be tested must be repeatable and called from code ▫ Takes a long time in CI

Slide 44

Slide 44 text

When should you use it ? ▫ Monitor critical parts of your logic in CI ▫ When refactoring a specific feature ▫ When writing code executed on the main thread 44

Slide 45

Slide 45 text

Going Further ▫ Official Gradle plugin to lock the CPU clocks ▫ Sample Gradle plugin to fail on high measurements ▫ Using Jetpack Benchmark on Firebase Test Lab 45

Slide 46

Slide 46 text

App Performance Monitoring Live performance from the trenches 3

Slide 47

Slide 47 text

What about the real users? Get performance and runtime information from the production environment 47 @robin_rednine

Slide 48

Slide 48 text

48 Datadog SDK for Android Logs Metrics Traces

Slide 49

Slide 49 text

49 repositories { maven { url "https://dl.bintray.com/datadog/datadog-maven" } } dependencies { implementation "com.datadoghq:dd-sdk-android:1.4.0" implementation "com.datadoghq:dd-sdk-android-ktx:1.4.0" } Using Datadog SDK for Android

Slide 50

Slide 50 text

50 Logs ▫ Send structured information about - events - measures - state changes

Slide 51

Slide 51 text

51 val logger = Logger.Builder().build() logger.i( "Activity init to resume took $duration nanos" ) Sending Logs

Slide 52

Slide 52 text

52 val logger = Logger.Builder().build() logger.i( "Activity init to resume took $duration nanos", attributes = mapOf( "activity.time_to_resume" to duration, "activity.classname" to javaClass.simpleName ) ) Sending Logs

Slide 53

Slide 53 text

53

Slide 54

Slide 54 text

54 Metrics ▫ Extract metrics information from logs ▫ Visualize critical KPIs in a dashboards ▫ Get alerts on spikes / drops

Slide 55

Slide 55 text

55

Slide 56

Slide 56 text

56 Tracing (beta in 1.4.0) ▫ Get traces across distributed architectures ▫ Performance information ▫ Debugging network errors

Slide 57

Slide 57 text

57 val okHttpClient = OkHttpClient.Builder() .addInterceptor(TracingInterceptor()) .build() OkHttp Requests

Slide 58

Slide 58 text

58

Slide 59

Slide 59 text

59 val tracer = GlobalTracer.get() val span = tracer.buildSpan("doSomething").start() doSomething() span.finish() Tracing custom methods

Slide 60

Slide 60 text

60 val tracer = GlobalTracer.get() val span = tracer.buildSpan("doSomething").start() doSomething() span.finish() Tracing custom methods

Slide 61

Slide 61 text

61 val tracer = GlobalTracer.get() val span = tracer.buildSpan("doSomething").start() doSomething() span.finish() Tracing custom methods

Slide 62

Slide 62 text

62 val total = withinSpan("doSomething") { doSomething() } Tracing custom methods (Kotlin Extension)

Slide 63

Slide 63 text

Pros ▫ Real data - real use cases - real devices - real network conditions 63 App Performance Monitoring Cons ▫ Need large enough user base to have relevant data ▫ Can’t measure everything

Slide 64

Slide 64 text

When should you use it ? ▫ Monitor key aspects of your app in the wild ▫ Understand how performance impacts your users ▫ A/B testing measures 64

Slide 65

Slide 65 text

Going Further ▫ Library is Open Source on GitHub ▫ More features coming soon (Real User Monitoring) 65

Slide 66

Slide 66 text

Wrapping up 4

Slide 67

Slide 67 text

Key takeaways “Measure twice, cut once.” 67 Choose the relevant tool to your use case. Always analyse your measurements before taking action.

Slide 68

Slide 68 text

68 Useful links ▫ https:/ /ui.perfetto.dev/ ▫ https:/ /developer.android.com/studio/profile/benchmark ▫ https:/ /developer.android.com/studio/profile/run-benchmarks-in-ci ▫ https:/ /proandroiddev.com/jetpack-benchmark-on-firebase-test-lab-d1 4c5eae815f ▫ https:/ /github.com/DataDog/dd-sdk-android ▫ https:/ /docs.datadoghq.com/logs/log_collection/android

Slide 69

Slide 69 text

69 Thank You! Any questions? ▫ @xgouchet ▫ @datadoghq @theunsteady5

Slide 70

Slide 70 text

70 Office hours @xgouchet ▫ April 20th : 2 - 3 pm ▫ April 21st - 10 - 11 am - 3 - 4 pm #testing #benchmark #??? @theunsteady5

Slide 71

Slide 71 text

CREDITS Special thanks to all the people who made and released these awesome resources for free: ▫ Presentation template by SlidesCarnival ▫ Photographs from Unsplash 71