Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How JRebel for Android does performance testing

How JRebel for Android does performance testing

This talk was given at Mobilization 2016.

As developers we often employ unit- and integration testing to catch bugs early during development. But how to detect regressions in performance? Sometimes a small slowdown can go unnoticed for weeks or months until the app becomes unbearably slow and a profiler gets attached. In this talk, we'll take an in-the-trenches look at how we set up a performance testing suite for JRebel for Android. We'll talk about challenges we faced when running the tests on our CI servers. We'll also cover how we pinpoint regressions to a single commit with the help of Elasticsearch and Kibana.

B5b43736709fb47edc3ee649618d84f7?s=128

Madis Pink

October 22, 2016
Tweet

Transcript

  1. None
  2. How JRebel for Android Does Performance Testing Madis Pink

  3. JRebel for Android

  4. (relatively) long-running tasks >500ms

  5. None
  6. Is the app fast?

  7. None
  8. Is the app fast enough?

  9. Is this version faster than that version?

  10. Is this version slower than that version?

  11. Is this version slower than that version? Benchmarking!

  12. None
  13. Added the foo feature to thingamajig, set foo=true in the

    debug settings to enable it. ...
  14. Added the foo feature to thingamajig, set foo=true in the

    debug settings to enable it. ...oh, also made the app startup two seconds slower
  15. Automation We need:

  16. Automation We need: → A harness to run the tests

  17. Automation We need: → A harness to run the tests

    → Pinpoint specific commits
  18. Automation We need: → A harness to run the tests

    → Pinpoint specific commits → Make regressions noticeable
  19. Consistent Results Detecting regressions quickly

  20. 8000ms -> 8800ms regression error: +/- 2000ms

  21. 8000ms -> 8800ms regression error: +/- 200ms

  22. Consistent Results - Nexus 5

  23. Consistent Results - Nexus 5

  24. Thermal Throttling

  25. Consistent Results - Emulator

  26. Consistent Results - Emulator

  27. Consistent Results - Emulator

  28. Analyzing Results

  29. Kibana (ELK)

  30. Kibana (ELK) → Elasticsearch (stores)

  31. Kibana (ELK) → Elasticsearch (stores) → Logstash (receives)

  32. Kibana (ELK) → Elasticsearch (stores) → Logstash (receives) → Kibana

    (displays)
  33. Logstash input { http { port => 1138 } }

    output { elasticsearch { hosts => ["localhost"] } }
  34. Logstash $ curl -XPUT 'http://127.0.0.1:1138' \ -H "content-type: application/json" \

    -d '{ "version": "1.3.2", "test": "someTest", "timeMillis": 503 }'
  35. Logstash class PerfTestResult { String version; String test; long timeMillis;

    }
  36. Logstash /** * Simple Retrofit REST endpoint that * can

    send test results to logstash */ interface LogstashReporter { @PUT("/") @Headers("Content-Type: application/json") Call<ResponseBody> reportResult(@Body PerfTestResult result); }
  37. Kibana (demo)

  38. github.com/madisp/ perftesting-sample

  39. Kibana A few usecases

  40. A few usecases

  41. Testing specific flags

  42. Detecting Regressions Early

  43. Fix verification

  44. Improvements in Infrastructure :)

  45. Long-Term Historical Data

  46. Long-Term Historical Data

  47. Long-Term Historical Data

  48. Long-Term Historical Data

  49. Long-Term Historical Data

  50. Long-Term Historical Data

  51. Long-Term Historical Data

  52. //TODO

  53. None
  54. ? @madisp speakerdeck.com/madisp

  55. Images Used

  56. Images Used → SMBC, The superhero we need, http://smbc-comics.com/comic/ the-superhero-we-need

  57. Images Used → SMBC, The superhero we need, http://smbc-comics.com/comic/ the-superhero-we-need

    → JRebel for Android logo, http://jrebelforandroid.com
  58. Images Used → SMBC, The superhero we need, http://smbc-comics.com/comic/ the-superhero-we-need

    → JRebel for Android logo, http://jrebelforandroid.com → PBF comics, beach, http://pbfcomics.com/259/
  59. Images Used → SMBC, The superhero we need, http://smbc-comics.com/comic/ the-superhero-we-need

    → JRebel for Android logo, http://jrebelforandroid.com → PBF comics, beach, http://pbfcomics.com/259/ → TF2 engineer wallpaper, http://frontierjustineer.tumblr.com/
  60. Images Used → SMBC, The superhero we need, http://smbc-comics.com/comic/ the-superhero-we-need

    → JRebel for Android logo, http://jrebelforandroid.com → PBF comics, beach, http://pbfcomics.com/259/ → TF2 engineer wallpaper, http://frontierjustineer.tumblr.com/ → Gunshow comic, This is fine, http://gunshowcomic.com/648
  61. Images Used → SMBC, The superhero we need, http://smbc-comics.com/comic/ the-superhero-we-need

    → JRebel for Android logo, http://jrebelforandroid.com → PBF comics, beach, http://pbfcomics.com/259/ → TF2 engineer wallpaper, http://frontierjustineer.tumblr.com/ → Gunshow comic, This is fine, http://gunshowcomic.com/648 → Random switches on eBay, http://www.ebay.com/itm/Pilot- Universal-3-Row-Chrome-Safety-Cover-Toggle-Switch-w-Red- Indicator-Light-/371614043104