Benchmarking your
app’s performance
Xavier F. Gouchet, Senior Software Engineer
@xgouchet (Twitter, GitHub, SpeakerDeck)
Slide 2
Slide 2 text
No content
Slide 3
Slide 3 text
Introduction
A few words about performance
0
Slide 4
Slide 4 text
performance /pəˈfɔːməns/ noun
1. how well a computer, machine,
application, etc. works.
4
Slide 5
Slide 5 text
Mobile app
performance is still
critical today
- Low end devices
- Slow networks
- Short attention span
5
@priscilladupreez
Slide 6
Slide 6 text
Performance
(like Android)
are fragmented
If you develop on a Pixel 4, make sure your app also
works correctly on a Samsung Galaxy S7 (2016)
6
Slide 7
Slide 7 text
Performance
must be measured
You can’t just base your
roadmap decision on
“a hunch”
7
@youngprodigy3
Slide 8
Slide 8 text
User will tell you it feels slow
Measurements will tell you what
part exactly is slow.
#MeasurementsMatter
Changes must be measured
Measurements will tell you how
much your PR improves (or
degrades) your feature’s
performance
8
Slide 9
Slide 9 text
“Premature optimization is the root of all evil
(or at least most of it) in programming.”
— Donald Knuth
Don’t optimize early,
but optimize eventually.
9
Slide 10
Slide 10 text
10
Making sure the code works
- Unit Testing
- Instrumented Testing
- Manual Testing
- Mutation Testing
- …
In your toolbox
Making sure it works fast
- ???
Slide 11
Slide 11 text
Developer Tools
Get a snapshot of your immediate performance
1
Slide 12
Slide 12 text
Overview
Get a full view of:
- network
- cpu
- heap
- rendering
- …
12
@kmuza
Slide 13
Slide 13 text
13
Tools
Android Studio Profiler Systrace Perfetto
Slide 14
Slide 14 text
14
Android Profiler
Slide 15
Slide 15 text
15
Android Profiler
Slide 16
Slide 16 text
Launching systrace
$ systrace.py
16
Slide 17
Slide 17 text
Launching systrace
$ systrace.py
Starting tracing (stop with enter)
17
Additionnal info
fun methodWithCustomEvents() {
Trace.beginSection("MySectionName")
// …
Trace.endSection()
}
22
Slide 23
Slide 23 text
Pros
▫ (Almost) no setup required
▫ Get full system state info
▫ Controlled scenario and
environment
23
Systrace / Perfetto
Cons
▫ Performance measurements
are skewed
▫ Needs manual trigger
▫ Complex and dense UI/UX
Slide 24
Slide 24 text
When should you use it ?
▫ When you have a specific performance issue
▫ When you want to optimize a specific part of the code
▫ To debug a tricky algorithm (nested loops, concurrency)
24
Slide 25
Slide 25 text
Going Further
▫ Capture systrace from the device (API 28+)
25
Slide 26
Slide 26 text
Benchmark
Standardized performance measurement
2
Slide 27
Slide 27 text
Benchmark Overview
Measure timed performances
of specific operations
27
@veri_ivanova
@RunWith(AndroidJUnit4::class)
class BenchmarkTest {
@get:Rule
val benchmarkRule = BenchmarkRule()
// …
}
30
Benchmark Test Class
Slide 31
Slide 31 text
@RunWith(AndroidJUnit4::class)
class BenchmarkTest {
@get:Rule
val benchmarkRule = BenchmarkRule()
// …
}
31
Benchmark Test Class
Slide 32
Slide 32 text
@RunWith(AndroidJUnit4::class)
class BenchmarkTest {
@get:Rule
val benchmarkRule = BenchmarkRule()
// …
}
32
Benchmark Test Class
Slide 33
Slide 33 text
@Test
fun benchmarkSomeWork() {
benchmarkRule.measureRepeated {
doSomeWork()
}
}
33
Benchmark Test Function
Slide 34
Slide 34 text
@Test
fun benchmarkSomeWork() {
benchmarkRule.measureRepeated {
doSomeWork()
}
}
34
Benchmark Test Function
Slide 35
Slide 35 text
@Test
fun benchmarkSomeWork() {
benchmarkRule.measureRepeated {
doSomeWork()
}
}
35
Benchmark Test Function
Slide 36
Slide 36 text
@Test
fun benchmarkSomeWork() {
benchmarkRule.measureRepeated {
doSomeWork()
}
}
36
Benchmark Test Function
Slide 37
Slide 37 text
@Test
fun benchmarkSomeWork() {
benchmarkRule.measureRepeated {
cleanState()
val data = createData()
doSomeWork(data)
}
}
37
Benchmark Test Function
Slide 38
Slide 38 text
@Test
fun benchmarkSomeWork() {
benchmarkRule.measureRepeated {
runWithTimingDisabled { cleanState() }
val data = runWithTimingDisabled { createData() }
doSomeWork(data)
}
}
38
Benchmark Test Function
Slide 39
Slide 39 text
$ gradlew :benchmark_module:connectedCheck
39
Run the Benchmark
Slide 40
Slide 40 text
▫ Stored in a JSON file
▫ Full timing information in nanoseconds
40
Benchmark Results
Running Benchmark in the CI
▫ Emulators are unstable, use real devices
▫ Needs some tweaks in your gradle config
▫ Use the JSON to:
- Graph the evolution of performance
- Fail if the measurements exceed a threshold
42
Slide 43
Slide 43 text
Pros
▫ Reliable and consistent
▫ Measure just what you want
▫ Can be as high or low level as
needed
43
Jetpack Benchmark
Cons
▫ Needs to be ran on a real
device
▫ The feature to be tested
must be repeatable and
called from code
▫ Takes a long time in CI
Slide 44
Slide 44 text
When should you use it ?
▫ Monitor critical parts of your logic in CI
▫ When refactoring a specific feature
▫ When writing code executed on the main thread
44
Slide 45
Slide 45 text
Going Further
▫ Official Gradle plugin to lock the CPU clocks
▫ Sample Gradle plugin to fail on high measurements
▫ Using Jetpack Benchmark on Firebase Test Lab
45
Slide 46
Slide 46 text
App Performance Monitoring
Live performance from the trenches
3
Slide 47
Slide 47 text
What about the real
users?
Get performance and runtime
information from the
production environment
47
@robin_rednine
50
Logs
▫ Send structured information about
- events
- measures
- state changes
Slide 51
Slide 51 text
51
val logger = Logger.Builder().build()
logger.i(
"Activity init to resume took $duration nanos"
)
Sending Logs
Slide 52
Slide 52 text
52
val logger = Logger.Builder().build()
logger.i(
"Activity init to resume took $duration nanos",
attributes = mapOf(
"activity.time_to_resume" to duration,
"activity.classname" to javaClass.simpleName
)
)
Sending Logs
Slide 53
Slide 53 text
53
Slide 54
Slide 54 text
54
Metrics
▫ Extract metrics information from logs
▫ Visualize critical KPIs in a dashboards
▫ Get alerts on spikes / drops
Slide 55
Slide 55 text
55
Slide 56
Slide 56 text
56
Tracing (beta in 1.4.0)
▫ Get traces across distributed architectures
▫ Performance information
▫ Debugging network errors
Slide 57
Slide 57 text
57
val okHttpClient = OkHttpClient.Builder()
.addInterceptor(TracingInterceptor())
.build()
OkHttp Requests
Slide 58
Slide 58 text
58
Slide 59
Slide 59 text
59
val tracer = GlobalTracer.get()
val span = tracer.buildSpan("doSomething").start()
doSomething()
span.finish()
Tracing custom methods
Slide 60
Slide 60 text
60
val tracer = GlobalTracer.get()
val span = tracer.buildSpan("doSomething").start()
doSomething()
span.finish()
Tracing custom methods
Slide 61
Slide 61 text
61
val tracer = GlobalTracer.get()
val span = tracer.buildSpan("doSomething").start()
doSomething()
span.finish()
Tracing custom methods
Slide 62
Slide 62 text
62
val total = withinSpan("doSomething") {
doSomething()
}
Tracing custom methods (Kotlin Extension)
Slide 63
Slide 63 text
Pros
▫ Real data
- real use cases
- real devices
- real network conditions
63
App Performance Monitoring
Cons
▫ Need large enough user base
to have relevant data
▫ Can’t measure everything
Slide 64
Slide 64 text
When should you use it ?
▫ Monitor key aspects of your app in the wild
▫ Understand how performance impacts your users
▫ A/B testing measures
64
Slide 65
Slide 65 text
Going Further
▫ Library is Open Source on GitHub
▫ More features coming soon (Real User Monitoring)
65
Slide 66
Slide 66 text
Wrapping up
4
Slide 67
Slide 67 text
Key takeaways
“Measure twice,
cut once.”
67
Choose the relevant
tool to your use case.
Always analyse your
measurements before
taking action.
69
Thank You!
Any questions?
▫ @xgouchet
▫ @datadoghq
@theunsteady5
Slide 70
Slide 70 text
70
Office hours
@xgouchet
▫ April 20th : 2 - 3 pm
▫ April 21st
- 10 - 11 am
- 3 - 4 pm
#testing #benchmark #???
@theunsteady5
Slide 71
Slide 71 text
CREDITS
Special thanks to all the people who made and released
these awesome resources for free:
▫ Presentation template by SlidesCarnival
▫ Photographs from Unsplash
71