Auto profiling apps on every build (Droidcon SF 2016)

Performance is always a concern when working with resource constrained environments like Android. In addition to that, developers also have to deal with another limited resource: time. It is quite common that adding new features or fixing bugs is way more relevant than ensuring application works smoothly. Just measuring the performance of certain view takes a good amount of time, so why not automate it? At Lyft we found that gathering data with every continuous integration build would help us to not only detect regressions but also to ensure a smooth ride for all our users.

Conference: http://sf.droidcon.com/
Video: https://www.youtube.com/watch?v=cCuhUrmVpSo

Transcript

1. Auto profiling apps on every build Evelio Tarazona Cáceres
8. Weekly
9. 14 full time Android Engineers
11. 14 full time Android Engineers 80+ PRs 1000s of changes Millions of users
13. 14 full time Android Engineers 80+ PRs 1000s of changes Millions of users Millions of rides
15. ❓
17. Nothing out of the ordinary. The end.
18. @eveliotc | +EvelioTarazonaCaceres http://evel.io Evelio Tarazona Cáceres Questions?
19. Agenda • Building stuff at Lyft • Testing & Automation • #xyzmatters • Mo’ money mo’ problems • Auto profiling
20. CI
21. • Automated builds • Automated deployment • Automated testing • Other stuff Continuous Integration
22. Continous Deployment
23. Automated builds & deployment Dev Per commit Dozens of us Alpha Nightly Hundreds Beta 2-3/week Thousands Prod Weekly Millions
24. Branching Model master master release-x-x-x feature
25. Automated testing
26. Testing Unit UI E2E 10% 15% 70% * or whatever % makes sense for you Maintenance Coverage Fragility Duration Cost 5%
27. Manual Testing Anti-pattern E2E UI Unit 10% 5% 0%
28. Performance Testing? E2E UI Unit Performance etc.
29. Automated testing: Do it! Unit UI E2E Performance etc. Testing Testing Testing
30. Other stuff
31. Other stuff • Common stuff like • Code review • Syncs & arch meetings • Pair programming • Refactor proposals • CI Enforced coverage, ownership, code styles, etc.
34. Other stuff • Code review, syncs, arch meetings, pair programming, etc. • CI Enforced coverage, ownership, code styles, etc. • CI Static analysis & reporting
35. This is supposed to be a performance talk
36. #perfmatters* *Disclamer
37. The problem
38. Pick One Testing Time Performance
39. Pick Two Testing Time Performance
40. RTFM
43. Android Performance 101: CPU • Do not nest multi-pass layouts • Lazily compute complex data when needed • Cache heavy computational results for re-use • Consider RenderScript for performance • Keep work off of the main thread https://corner.squareup.com/2015/08/streamlining-android-apps-tech-talks.html
44. Android Performance 101: Memory • Use object pools and caches to reduce churn • Be mindful of the overhead of enums • Do not allocate inside the draw path • Use specialized collections instead of JDK • Memory collections when appropriate (SparseArray) https://corner.squareup.com/2015/08/streamlining-android-apps-tech-talks.html
45. Android Performance 101: I/O • Batch operations with reasonable back-off policies • Use gzip or binary serialization format • Cache data offline with TTLs for reloading • Use JobScheduler API to batch across OS https://corner.squareup.com/2015/08/streamlining-android-apps-tech-talks.html
46. Android Performance 101: Other • JSON parsing only matters if you are streaming vs reading full response and then parsing • Shrink ALL the things! • Don’t do things you don’t need • Lint: use it • Watch out for 3rd party sdks e.g. analytics & ads • Dagger 2 Lazy & Producers https://corner.squareup.com/2015/08/streamlining-android-apps-tech-talks.html
47. - Some wise dude We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3% “
48. #qualitymatters • Document yourself • Understand your code • Have your own opinions • Do it with purpose • There are always tradeoffs • Prioritize, what’s best for your business?
49. Warning: Opinions • Multidex use it, but minifyEnabled true in prod • enum vs. IntDef: enums FTW! • Type safety, enforces well formed input for you • EnumSet instead of bitwise flags • EnumMap cause #perfmatters • Effective way of creating singletons • Small overhead when compared with dependencies & resources
50. Real reasons why #perfmatters • Better user experience • 100ms = instant • 1 second = noticeable delay • 2+ seconds = what’s going on????!!!!!! • = • 100ms of latency costs Amazon 1% of profit.
51. Lyft perf reasons • Delightful details • Launch time • Short sessions • Make it better • Make it faster
52. Launch time
53. Launch time
54. The solution
55. The Performance Lifecycle • Measure • Analyse • Fix • Repeat • ? • Profit
56. Measure
57. Challenges • Observer effect • Measuring code overhead • Results variance • Thread scheduling & garbage collection • Device vs. emulator • Thermal throttling • Network
58. Android Basic Profiling Toolbelt • Logs + Android Monitor on Android Studio • Rendering: Hierarchy Viewer, on device fps, overdraw, hardware layers, etc. • Memory: Memory Monitor, Heap Viewer, Allocation Tracker, MAT • Battery: Batterystats, Battery Historian • Computing: • Traceview • Systrace
60. Android Profiling Toolbelt • dumpsys • Rendering: Hierarchy Viewer, gfxinfo: fps, overdraw, hardware layers, etc. • Memory: Memory Monitor, Heap Viewer, Allocation Tracker, MAT • Battery: Batterystats, Battery Historian • Computing: • Logs • Traceview • Systrace ✋ ✋
61. Analyse
62. Findings (Not many surprises here) • Wait on async bootstrap network call to complete • Busy main thread • Huge Bitmaps resources • Complex and nested XML layouts • Third party SDKs like Google Maps • Reflection based JSON parsing
63. Fix
64. Quick Fixes • Move required bootstrap network call as early as possible • Remove/lazy load tailored Bitmap resources • Simplify XML layouts/Flattening view hierarchy • Defer/parallelize third party SDKs initialization
65. The result
66. ~800ms
67. WIP/Future Fixes • Decomposition • HTTP2 • Caching • Better JSON parsing e.g. TypeAdapters • Binary protocol: protobuf, capnproto, flatbuffers, etc. • Less stuff in the main thread • Scoop migration (Scoped down dependencies)
68. 14 full time Android Engineers 80+ PRs 1000s of changes Millions of users Millions of rides
69. Regressions
71. Why not automate it?
72. Analytics • Some manual work and code • SpaningEvents • Initiation time, end time, delta • Network calls, app launch time, etc. • Device status
74. APK Size over time
75. Pre-Analytics, thanks archive! ➜ aws s3 ls --recursive s3://lyft-client-ci/android/ | sort -k1 -n | sed 's/ / /g' > /tmp/apks-sorted.txt ➜ ./plotapksize > apksize.png
76. plotapksize script 1. #!/usr/local/bin/gnuplot 2. reset 3. set term png size 2400,1200 4. set xdata time 5. set timefmt "%Y-%m-%d %H:%M:%S" 6. set format x "%Y-%m" 7. set xlabel "date" 8. set ylabel "size in bytes" 9. set title "APK size over time" 10. set key reverse Left outside 11. set grid 12. set style data linespoints 13. plot "apks-sorted.txt" using 1:($3 == 0 ? NaN : $3) title "size"
78. Sexy & useful
79. Comercial tools • Google Cloud Test Lab • Enerlytics Essert (Battery + CI) • Nimbledroid (Droidcon) • Monkop (not at Droidcon)
80. Monkop
81. Monkop flavors
83. Too easy? Roll your own • Custom ROM/Instrumentation • Annotation processor + AspectJ • Analytics • Systrace Trace calls • Method tracing • adb commands ➜ adb logcat | grep 'ActivityManager: Displayed' … ActivityManager: Displayed me.lyft.android.dev/.ui.MainActivity: +2s217ms ➜ adb screenrecord • gfxinfo • dumpsys
84. Performance Testing Metric At Start Metric At End Your Code Pass Criteria Fail Pass Δ∆
86. Lyft’s • Appium & Espresso tests • Start/stop systrace and other collection tools at test init/end • Store them in S3 • Generate reports • and most importantly…
87. Enforce it
89. Takeaways • Define your process and streamline it • #perfmatters but #qualitymatters too • Understand your code • There is always a tradeoff, balance it out. • Testing: Do it. • Automation: Do it. ✅
90. @eveliotc | +EvelioTarazonaCaceres http://evel.io Evelio Tarazona Cáceres Q&A