Slide 1

Slide 1 text

Benchmarking your app’s performance Xavier F. Gouchet, Senior Software Engineer / Team Lead @xgouchet (Twitter, GitHub, SpeakerDeck)

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Introduction A few words about performance 0

Slide 4

Slide 4 text

Mobile app performance is still critical today - Low end devices - Slow networks - Short attention span 4 @priscilladupreez

Slide 5

Slide 5 text

Performance (like Android) are fragmented If you develop on a Pixel 5, make sure your app also works correctly on a Samsung Galaxy S8 (2017) 5

Slide 6

Slide 6 text

Performance must be measured You can’t just base your roadmap decision on “a hunch” 6 @youngprodigy3

Slide 7

Slide 7 text

User will tell you it feels slow Measurements will tell you what part exactly is slow. #MeasurementsMatter Changes must be measured Measurements will tell you how much your PR improves (or degrades) your feature’s performance 7

Slide 8

Slide 8 text

8 Making sure the code works - Unit Testing - Instrumented Testing - Manual Testing - - … In your toolbox Making sure it works fast and efficiently - ???

Slide 9

Slide 9 text

Developer Tools Get a snapshot of your immediate performance 1

Slide 10

Slide 10 text

Immediate snapshot Get a full view of: - network - cpu - heap - rendering - … 10 @kmuza

Slide 11

Slide 11 text

11 Tools Android Studio Profiler Systrace Perfetto

Slide 12

Slide 12 text

12 Android Profiler

Slide 13

Slide 13 text

13 Android Profiler Touch Activities Fragments

Slide 14

Slide 14 text

14 Android Profiler CPU usage

Slide 15

Slide 15 text

15 Android Profiler Memory usage

Slide 16

Slide 16 text

16 Android Profiler Network usage

Slide 17

Slide 17 text

17 Android Profiler Battery impact

Slide 18

Slide 18 text

18 Android Profiler

Slide 19

Slide 19 text

Launching systrace $ systrace.py 19

Slide 20

Slide 20 text

Launching systrace $ systrace.py Starting tracing (stop with enter) 20

Slide 21

Slide 21 text

Launching systrace $ systrace.py Starting tracing (stop with enter) Tracing completed. Collecting output... Outputting Systrace results... Tracing complete, writing results 21

Slide 22

Slide 22 text

22 Systrace (legacy)

Slide 23

Slide 23 text

23 Perfetto showing systrace result

Slide 24

Slide 24 text

24 Recording from Perfetto (API 28+)

Slide 25

Slide 25 text

25 Recording from Perfetto (API 28+)

Slide 26

Slide 26 text

fun methodWithCustomEvents() { Trace.beginSection("MySectionName") // … Trace.endSection() } 26 Custom Traces

Slide 27

Slide 27 text

27 Recording from Perfetto (API 28+)

Slide 28

Slide 28 text

28 Recording from Perfetto (API 28+)

Slide 29

Slide 29 text

Pros 👍 ▫ (Almost) no setup required ▫ Get full system state info ▫ Controlled scenario and environment 29 Profiler / Systrace / Perfetto Cons 👎 ▫ Performance measurements are skewed ▫ Needs manual trigger ▫ Complex and dense UI/UX

Slide 30

Slide 30 text

When should you use it ? ▫ When you have a specific performance issue ▫ When you want to optimize a specific part of the code ▫ To debug a tricky algorithm (nested loops, concurrency) 30

Slide 31

Slide 31 text

Going Further ▫ Capture systrace from the device (API 28+) 31

Slide 32

Slide 32 text

32 Capturing traces from the developer settings

Slide 33

Slide 33 text

Benchmark Standardized performance measurement 2

Slide 34

Slide 34 text

Benchmark Overview Measure timed performances of specific operations 34 @veri_ivanova

Slide 35

Slide 35 text

35 Tools Jetpack Benchmark CI Provider

Slide 36

Slide 36 text

Jetpack Benchmark Library dependencies { // … androidTestImplementation "androidx.benchmark:benchmark-junit4:1.0.0" } 36

Slide 37

Slide 37 text

Jetpack Benchmark Library android { // … defaultConfig { // … testInstrumentationRunner "androidx.benchmark.junit4.AndroidBenchmarkRunner" } } 37

Slide 38

Slide 38 text

@RunWith(AndroidJUnit4::class) class BenchmarkTest { @get:Rule val benchmarkRule = BenchmarkRule() // … } 38 Benchmark Test Class

Slide 39

Slide 39 text

@RunWith(AndroidJUnit4::class) class BenchmarkTest { @get:Rule val benchmarkRule = BenchmarkRule() // … } 39 Benchmark Test Class

Slide 40

Slide 40 text

@RunWith(AndroidJUnit4::class) class BenchmarkTest { @get:Rule val benchmarkRule = BenchmarkRule() // … } 40 Benchmark Test Class

Slide 41

Slide 41 text

@Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { doSomeWork() } } 41 Benchmark Test Function

Slide 42

Slide 42 text

@Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { doSomeWork() } } 42 Benchmark Test Function

Slide 43

Slide 43 text

@Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { doSomeWork() } } 43 Benchmark Test Function

Slide 44

Slide 44 text

@Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { doSomeWork() } } 44 Benchmark Test Function

Slide 45

Slide 45 text

@Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { cleanState() val data = createData() doSomeWork(data) } } 45 Benchmark Test Function

Slide 46

Slide 46 text

@Test fun benchmarkSomeWork() { benchmarkRule.measureRepeated { runWithTimingDisabled { cleanState() } val data = runWithTimingDisabled { createData() } doSomeWork(data) } } 46 Benchmark Test Function

Slide 47

Slide 47 text

$ gradlew :benchmark_module:connectedCheck 47 Run the Benchmark

Slide 48

Slide 48 text

▫ Stored in a JSON file ▫ Full timing information in nanoseconds 48 Benchmark Results

Slide 49

Slide 49 text

{ "name": "benchmarkSomeWork", "className": "com.example.BenchmarkTest", "metrics": { "timeNs": { "minimum": 2561000, "maximum": 7133000, "median": 3914000, "runs": [ … ] } } 49 Benchmark Result

Slide 50

Slide 50 text

Running Benchmark in the CI ▫ Emulators are unstable, use real devices ▫ Needs some tweaks in your gradle config ▫ Use the JSON to: - Graph the evolution of performance - Fail if the measurements exceed a threshold 50

Slide 51

Slide 51 text

Pros 👍 ▫ Accurate*, reliable and consistent ▫ Measure just what you want ▫ Can be as high or low level as needed 51 Jetpack Benchmark Cons 👎 ▫ *Needs to be ran on a real device ▫ The feature to be tested must be repeatable and called from code ▫ Takes a long time in CI

Slide 52

Slide 52 text

When should you use it ? ▫ Monitor critical parts of your logic in CI ▫ When refactoring a specific feature ▫ When writing code executed on the main thread 52

Slide 53

Slide 53 text

▫ Official Gradle plugin to lock the CPU clocks (root) ▫ Use CI script to fail on high measurements Going Further 53

Slide 54

Slide 54 text

Real User Monitoring Live performance from the trenches 3

Slide 55

Slide 55 text

What about the real users? Get performance and runtime information from the production environment 55 @robin_rednine

Slide 56

Slide 56 text

56 Tools Real User Monitoring Error Tracking Android Vitals

Slide 57

Slide 57 text

57 dependencies { implementation "com.datadoghq:dd-sdk-android:1.10.0" implementation "com.datadoghq:dd-sdk-android-ktx:1.10.0" // optional extensions: RxJava, Android NDK, … } Using Datadog SDK for Android

Slide 58

Slide 58 text

Single View Performance 58

Slide 59

Slide 59 text

Funnels 59

Slide 60

Slide 60 text

Network Performance 60

Slide 61

Slide 61 text

Mobile Vitals: Refresh Rate 61

Slide 62

Slide 62 text

Mobile Vitals: CPU usage 62

Slide 63

Slide 63 text

Mobile Vitals: RAM consumption 63

Slide 64

Slide 64 text

Application not responding 64

Slide 65

Slide 65 text

Application not responding 65

Slide 66

Slide 66 text

Error Tracking 66

Slide 67

Slide 67 text

Performance monitoring 67

Slide 68

Slide 68 text

Pros 👍 ▫ Real data - real use cases - real devices - real network conditions 68 App Performance Monitoring Cons 👎 ▫ Need large enough user base to have relevant data ▫ Can’t measure everything

Slide 69

Slide 69 text

When should you use it ? ▫ Monitor key aspects of your app in the wild ▫ Understand how performance and crashes impacts your users’ journey ▫ A/B testing / feature flags analytics 69

Slide 70

Slide 70 text

Going Further ▫ Library is Open Source on GitHub ▫ Also provides Logging and Tracing ▫ More features coming soon… 70

Slide 71

Slide 71 text

Wrapping up 4

Slide 72

Slide 72 text

Key takeaways “Measure twice, cut once.” 72 Choose the relevant tool to your use case. Always analyse your measurements before taking action.

Slide 73

Slide 73 text

73 Useful links ▫ https:/ /ui.perfetto.dev/ ▫ https:/ /developer.android.com/studio/profile/benchmark ▫ https:/ /developer.android.com/studio/profile/run-benchmarks-in-ci ▫ https:/ /proandroiddev.com/jetpack-benchmark-on-firebase-test-lab-d1 4c5eae815f ▫ https:/ /github.com/DataDog/dd-sdk-android ▫ https:/ /docs.datadoghq.com/real_user_monitoring/android/

Slide 74

Slide 74 text

74 Thank You! Any questions? ▫ @xgouchet ▫ @datadoghq @theunsteady5

Slide 75

Slide 75 text

CREDITS Special thanks to all the people who made and released these awesome resources for free: ▫ Presentation template by SlidesCarnival ▫ Photographs from Unsplash 75