How JRebel for Android does performance testing

This talk was given at Mobilization 2016.

As developers we often employ unit- and integration testing to catch bugs early during development. But how to detect regressions in performance? Sometimes a small slowdown can go unnoticed for weeks or months until the app becomes unbearably slow and a profiler gets attached. In this talk, we'll take an in-the-trenches look at how we set up a performance testing suite for JRebel for Android. We'll talk about challenges we faced when running the tests on our CI servers. We'll also cover how we pinpoint regressions to a single commit with the help of Elasticsearch and Kibana.

Transcript

2. How JRebel for Android Does Performance Testing Madis Pink
3. JRebel for Android
4. (relatively) long-running tasks >500ms
6. Is the app fast?
8. Is the app fast enough?
9. Is this version faster than that version?
10. Is this version slower than that version?
11. Is this version slower than that version? Benchmarking!
13. Added the foo feature to thingamajig, set foo=true in the debug settings to enable it. ...
14. Added the foo feature to thingamajig, set foo=true in the debug settings to enable it. ...oh, also made the app startup two seconds slower
15. Automation We need:
16. Automation We need: → A harness to run the tests
17. Automation We need: → A harness to run the tests → Pinpoint specific commits
18. Automation We need: → A harness to run the tests → Pinpoint specific commits → Make regressions noticeable
19. Consistent Results Detecting regressions quickly
20. 8000ms -> 8800ms regression error: +/- 2000ms
21. 8000ms -> 8800ms regression error: +/- 200ms
22. Consistent Results - Nexus 5
23. Consistent Results - Nexus 5
24. Thermal Throttling
25. Consistent Results - Emulator
26. Consistent Results - Emulator
27. Consistent Results - Emulator
28. Analyzing Results
29. Kibana (ELK)
30. Kibana (ELK) → Elasticsearch (stores)
31. Kibana (ELK) → Elasticsearch (stores) → Logstash (receives)
32. Kibana (ELK) → Elasticsearch (stores) → Logstash (receives) → Kibana (displays)
33. Logstash input { http { port => 1138 } } output { elasticsearch { hosts => ["localhost"] } }
34. Logstash $ curl -XPUT 'http://127.0.0.1:1138' \ -H "content-type: application/json" \ -d '{ "version": "1.3.2", "test": "someTest", "timeMillis": 503 }'
35. Logstash class PerfTestResult { String version; String test; long timeMillis; }
36. Logstash /** * Simple Retrofit REST endpoint that * can send test results to logstash */ interface LogstashReporter { @PUT("/") @Headers("Content-Type: application/json") Call<ResponseBody> reportResult(@Body PerfTestResult result); }
37. Kibana (demo)
38. github.com/madisp/ perftesting-sample
39. Kibana A few usecases
40. A few usecases
41. Testing specific flags
42. Detecting Regressions Early
43. Fix verification
44. Improvements in Infrastructure :)
45. Long-Term Historical Data
46. Long-Term Historical Data
47. Long-Term Historical Data
48. Long-Term Historical Data
49. Long-Term Historical Data
50. Long-Term Historical Data
51. Long-Term Historical Data
52. //TODO
54. ? @madisp speakerdeck.com/madisp
55. Images Used
56. Images Used → SMBC, The superhero we need, http://smbc-comics.com/comic/ the-superhero-we-need
57. Images Used → SMBC, The superhero we need, http://smbc-comics.com/comic/ the-superhero-we-need → JRebel for Android logo, http://jrebelforandroid.com
58. Images Used → SMBC, The superhero we need, http://smbc-comics.com/comic/ the-superhero-we-need → JRebel for Android logo, http://jrebelforandroid.com → PBF comics, beach, http://pbfcomics.com/259/
59. Images Used → SMBC, The superhero we need, http://smbc-comics.com/comic/ the-superhero-we-need → JRebel for Android logo, http://jrebelforandroid.com → PBF comics, beach, http://pbfcomics.com/259/ → TF2 engineer wallpaper, http://frontierjustineer.tumblr.com/
60. Images Used → SMBC, The superhero we need, http://smbc-comics.com/comic/ the-superhero-we-need → JRebel for Android logo, http://jrebelforandroid.com → PBF comics, beach, http://pbfcomics.com/259/ → TF2 engineer wallpaper, http://frontierjustineer.tumblr.com/ → Gunshow comic, This is fine, http://gunshowcomic.com/648
61. Images Used → SMBC, The superhero we need, http://smbc-comics.com/comic/ the-superhero-we-need → JRebel for Android logo, http://jrebelforandroid.com → PBF comics, beach, http://pbfcomics.com/259/ → TF2 engineer wallpaper, http://frontierjustineer.tumblr.com/ → Gunshow comic, This is fine, http://gunshowcomic.com/648 → Random switches on eBay, http://www.ebay.com/itm/Pilot- Universal-3-Row-Chrome-Safety-Cover-Toggle-Switch-w-Red- Indicator-Light-/371614043104