Benchmarking your
app’s performance
Xavier F. Gouchet, Senior Software Engineer / Team Lead
@xgouchet (Twitter, GitHub, SpeakerDeck)
Slide 2
Slide 2 text
No content
Slide 3
Slide 3 text
Introduction
A few words about performance
0
Slide 4
Slide 4 text
Mobile app
performance is still
critical today
- Low end devices
- Slow networks
- Short attention span
4
@priscilladupreez
Slide 5
Slide 5 text
Performance
(like Android)
are fragmented
If you develop on a Pixel 5, make sure your app also
works correctly on a Samsung Galaxy S8 (2017)
5
Slide 6
Slide 6 text
Performance
must be measured
You can’t just base your
roadmap decision on
“a hunch”
6
@youngprodigy3
Slide 7
Slide 7 text
User will tell you it feels slow
Measurements will tell you what
part exactly is slow.
#MeasurementsMatter
Changes must be measured
Measurements will tell you how
much your PR improves (or
degrades) your feature’s
performance
7
Slide 8
Slide 8 text
8
Making sure the code works
- Unit Testing
- Instrumented Testing
- Manual Testing
-
- …
In your toolbox
Making sure it works fast and
efficiently
- ???
Slide 9
Slide 9 text
Developer Tools
Get a snapshot of your immediate performance
1
Slide 10
Slide 10 text
Immediate snapshot
Get a full view of:
- network
- cpu
- heap
- rendering
- …
10
@kmuza
Slide 11
Slide 11 text
11
Tools
Android Studio Profiler Systrace Perfetto
Slide 12
Slide 12 text
12
Android Profiler
Slide 13
Slide 13 text
13
Android Profiler
Touch
Activities
Fragments
Slide 14
Slide 14 text
14
Android Profiler
CPU usage
Slide 15
Slide 15 text
15
Android Profiler
Memory usage
Slide 16
Slide 16 text
16
Android Profiler
Network usage
Slide 17
Slide 17 text
17
Android Profiler
Battery impact
Slide 18
Slide 18 text
18
Android Profiler
Slide 19
Slide 19 text
Launching systrace
$ systrace.py
19
Slide 20
Slide 20 text
Launching systrace
$ systrace.py
Starting tracing (stop with enter)
20
Pros 👍
▫ (Almost) no setup required
▫ Get full system state info
▫ Controlled scenario and
environment
29
Profiler / Systrace / Perfetto
Cons 👎
▫ Performance measurements
are skewed
▫ Needs manual trigger
▫ Complex and dense UI/UX
Slide 30
Slide 30 text
When should you use it ?
▫ When you have a specific performance issue
▫ When you want to optimize a specific part of the code
▫ To debug a tricky algorithm (nested loops, concurrency)
30
Slide 31
Slide 31 text
Going Further
▫ Capture systrace from the device (API 28+)
31
Slide 32
Slide 32 text
32
Capturing traces from the developer settings
Slide 33
Slide 33 text
Benchmark
Standardized performance measurement
2
Slide 34
Slide 34 text
Benchmark Overview
Measure timed performances
of specific operations
34
@veri_ivanova
@RunWith(AndroidJUnit4::class)
class BenchmarkTest {
@get:Rule
val benchmarkRule = BenchmarkRule()
// …
}
38
Benchmark Test Class
Slide 39
Slide 39 text
@RunWith(AndroidJUnit4::class)
class BenchmarkTest {
@get:Rule
val benchmarkRule = BenchmarkRule()
// …
}
39
Benchmark Test Class
Slide 40
Slide 40 text
@RunWith(AndroidJUnit4::class)
class BenchmarkTest {
@get:Rule
val benchmarkRule = BenchmarkRule()
// …
}
40
Benchmark Test Class
Slide 41
Slide 41 text
@Test
fun benchmarkSomeWork() {
benchmarkRule.measureRepeated {
doSomeWork()
}
}
41
Benchmark Test Function
Slide 42
Slide 42 text
@Test
fun benchmarkSomeWork() {
benchmarkRule.measureRepeated {
doSomeWork()
}
}
42
Benchmark Test Function
Slide 43
Slide 43 text
@Test
fun benchmarkSomeWork() {
benchmarkRule.measureRepeated {
doSomeWork()
}
}
43
Benchmark Test Function
Slide 44
Slide 44 text
@Test
fun benchmarkSomeWork() {
benchmarkRule.measureRepeated {
doSomeWork()
}
}
44
Benchmark Test Function
Slide 45
Slide 45 text
@Test
fun benchmarkSomeWork() {
benchmarkRule.measureRepeated {
cleanState()
val data = createData()
doSomeWork(data)
}
}
45
Benchmark Test Function
Slide 46
Slide 46 text
@Test
fun benchmarkSomeWork() {
benchmarkRule.measureRepeated {
runWithTimingDisabled { cleanState() }
val data = runWithTimingDisabled { createData() }
doSomeWork(data)
}
}
46
Benchmark Test Function
Slide 47
Slide 47 text
$ gradlew :benchmark_module:connectedCheck
47
Run the Benchmark
Slide 48
Slide 48 text
▫ Stored in a JSON file
▫ Full timing information in nanoseconds
48
Benchmark Results
Running Benchmark in the CI
▫ Emulators are unstable, use real devices
▫ Needs some tweaks in your gradle config
▫ Use the JSON to:
- Graph the evolution of performance
- Fail if the measurements exceed a threshold
50
Slide 51
Slide 51 text
Pros 👍
▫ Accurate*, reliable and
consistent
▫ Measure just what you want
▫ Can be as high or low level as
needed
51
Jetpack Benchmark
Cons 👎
▫ *Needs to be ran on a real
device
▫ The feature to be tested
must be repeatable and
called from code
▫ Takes a long time in CI
Slide 52
Slide 52 text
When should you use it ?
▫ Monitor critical parts of your logic in CI
▫ When refactoring a specific feature
▫ When writing code executed on the main thread
52
Slide 53
Slide 53 text
▫ Official Gradle plugin to lock the CPU clocks (root)
▫ Use CI script to fail on high measurements
Going Further
53
Slide 54
Slide 54 text
Real User Monitoring
Live performance from the trenches
3
Slide 55
Slide 55 text
What about the real
users?
Get performance and runtime
information from the
production environment
55
@robin_rednine
Slide 56
Slide 56 text
56
Tools
Real User Monitoring Error Tracking Android Vitals
Slide 57
Slide 57 text
57
dependencies {
implementation "com.datadoghq:dd-sdk-android:1.10.0"
implementation "com.datadoghq:dd-sdk-android-ktx:1.10.0"
// optional extensions: RxJava, Android NDK, …
}
Using Datadog SDK for Android
Slide 58
Slide 58 text
Single View Performance
58
Slide 59
Slide 59 text
Funnels
59
Slide 60
Slide 60 text
Network Performance
60
Slide 61
Slide 61 text
Mobile Vitals: Refresh Rate
61
Slide 62
Slide 62 text
Mobile Vitals: CPU usage
62
Slide 63
Slide 63 text
Mobile Vitals: RAM consumption
63
Slide 64
Slide 64 text
Application not responding
64
Slide 65
Slide 65 text
Application not responding
65
Slide 66
Slide 66 text
Error Tracking
66
Slide 67
Slide 67 text
Performance monitoring
67
Slide 68
Slide 68 text
Pros 👍
▫ Real data
- real use cases
- real devices
- real network conditions
68
App Performance Monitoring
Cons 👎
▫ Need large enough user base
to have relevant data
▫ Can’t measure everything
Slide 69
Slide 69 text
When should you use it ?
▫ Monitor key aspects of your app in the wild
▫ Understand how performance and crashes impacts
your users’ journey
▫ A/B testing / feature flags analytics
69
Slide 70
Slide 70 text
Going Further
▫ Library is Open Source on GitHub
▫ Also provides Logging and Tracing
▫ More features coming soon…
70
Slide 71
Slide 71 text
Wrapping up
4
Slide 72
Slide 72 text
Key takeaways
“Measure twice,
cut once.”
72
Choose the relevant
tool to your use case.
Always analyse your
measurements before
taking action.
74
Thank You!
Any questions?
▫ @xgouchet
▫ @datadoghq
@theunsteady5
Slide 75
Slide 75 text
CREDITS
Special thanks to all the people who made and released
these awesome resources for free:
▫ Presentation template by SlidesCarnival
▫ Photographs from Unsplash
75