ExoPlayer in RecyclerView

Presented at DroidKaigi 2019.

ExoPlayer in RecyclerView is an interesting topic that has been discussed for a long time, including this issue: https://github.com/google/ExoPlayer/issues/867

In this talk, I discuss about some challenges when implementing them, known approach and its issue. After all, I propose my approach that can not only fix those issue but also capable of doing more.

Transcript

1. eneim ExoPlayer in RecyclerView a proposal DroidKaigi 2019
2. Nam Nguyen ene.im / eneim / @ene__im 2 the simple, the best
3. In this talk - On “ExoPlayer in RecyclerView”: why and how - Discuss common approaches and their issues - Discuss my proposal concept - Demonstrate my PoC using beta test app 3
4. Before we start... 4
5. 5 DEMO
6. ExoPlayer RecyclerView 6
7. 7 RecyclerView ecosystem https://lucasr.org/2014/07/31/the-new-twowayview/ ViewHolder LayoutManager Adapter ListAdapter Paging ItemAnimator ItemDecoration SnapHelper Selection ViewPager2
8. 8 ExoPlayer ecosystem Building feature-rich media apps with ExoPlayer (Google I/O '18)
9. ExoPlayer usage val player = ExoPlayerFactory.newSimpleInstance(context) val mediaSource = ExtractorMediaSource.Factory( DefaultDataSourceFactory(context, "¯\_(ツ)_/¯") ).createMediaSource(video.toUri()) player.prepare(mediaSource) player.playWhenReady = true // playerView.player = player ⬇ use below instead PlayerView.switchTargetView(player, null, playerView) 9
10. Why ExoPlayer in RecyclerView? 10
11. How ExoPlayer in RecyclerView? 11
12. FYI: a 3.3-year-old issue 12
13. How to use ExoPlayer in a ListView or RecyclerView? 13 https://github.com/google/ExoPlayer/issues/867 Oct 15, 2015
14. What they ask 14 I want to use ExoPlayer in a RecyclerView as a part of row item. I want to make a customer view and wrap the ExoPlayer in that view. Do you have some advice? Thank you!
15. What they needs - Video player in RecyclerView, ViewPager, ScrollView, etc - ExoPlayer or whatever works - Auto play/pause on scroll? (like Facebook, Instagram, Twitter, etc) - Fullscreen back and forth, smoothly - Network usage friendly, battery friendly, UX/UI friendly, etc friendly 15
17. Take a look around 17
18. Facebook - Auto play/pause on scroll - Dialog for fullscreen player - Auto fullscreen on landscape - ಠ_ಠ: unstable under config changes, video reload on config change? 18 204.0.0.24.101
19. YouTube 14.02.54 - Good UX in single item play - The mini/overlay player - Well config change handling - Latest: optional auto play in list - ಠ_ಠ: reload when switch from list to single player? 19
20. AbemaTV 4.5.0 - ViewPager + ExoPlayer? - Fullscreen on landscape - ಠ_ಠ: [Active → Inactive → Active] shifting → black screen splash, reload on config change? 20
22. Common objectives - Players in RecyclerView or ViewPager - Auto play/pause on user interactions (scroll, swipe, etc) - Single player most of the time. - Can be multiple players. Eg: Line Live - Fullscreen playback back and forth - Playback continuity - Others (eg: auto show/hide thumbnail etc) 22
23. - RecyclerView: scrollable, infinitely - Item (ViewHolder) are supposed to be reused/recycled - Need a dynamic strategy - ViewHolder lifecycle → needs proper playback control - Created/Bound/Recycled - Attached/Detached - Many player instances = system performance down - The more player instances, the lower performance - Many PlayerView = many Surfaces = surface creation/management cost - The more Surfaces, the worse - Too strictly controlled = UX loss Challenges (1): ExoPlayer + RecyclerView 23
24. Challenges (2): to fullscreen & back - UI flow: when to fullscreen? Rotate or not rotate? - Rotate → Config changes - Config changes handling - Not only orientation change - Important criteria: playback continuity (Video, Audio), UI look and feel 24
25. Common approach - PlayerView in ViewHolder (VH) - Adapter: manage ExoPlayer and MediaSource, - MediaSource is created and bound to VH on demand - Only “will play” VH will be provided by the ExoPlayer instance - Adapter’s callback to update playbacks - onBindViewHolder → set MediaSource - onViewAttachedToWindow/onViewDetachedFromWindow → prepare/release? - onViewRecycled/onFailedToRecycleView → TODO? - RecyclerView callback to update playbacks (OnScrollListener) - Playback strategy: top-most visible → play, otherwise → pause - Reuse PlayerView: remove from “will pause” VH then add to “will play” VH 25
26. Not yet resolved - Efficient Player management = UX loss - Reuse one Player for many PlayerView → “black splash” on resume (like AbemaTV app) - Re-buffering takes time - ViewHolder lifecycle - onViewDetachedFromWindow is not always called - setAdapter(null) ← to do or not to do? - UI Flow: to fullscreen and back - Video reloads on resuming has hidden issues 26
27. To fullscreen and back 27
28. Fullscreen → Single Player 28
29. UI flow: when to single player? 29 Same orientation Landscape player Multi Windows state ಠ_ಠ UX on Android Pie
30. Single player and config changes - Opening Single Player has many chances for config changes - Handle config changes manually (= add manifest entry) - Pros: no resource reloading, playback continuity done! - Cons: no adaptive layout, or manually apply using if/else/while … (not me (>_<)) - Many config changes patterns = error prone - Handle config changes automatically (= no manifest entry) - Pros: adaptive layout for each config - Cons/Unresolved: playback continuity 30
31. 31 1 2 3 1 Click → Open single player in ‘same orientation’ 2 Click → Open single player in landscape 3 Rotate → From one form to other form of single player Config change UI flow: patterns (eg: YouTube)
32. UI flow: others - Open single player on new Activity - → How to keep the playback continuity? - → How to come back and keep the playback continuity? - Open single player on Dialog (in same Activity) - → How to keep the playback continuity? - → How to come back and keep the playback continuity? 32
33. Reload: the hidden issues 33
34. Reload: the hidden issues - Reload = playback discontinuity - Reload = playback counter is up? - Analytic metrics matters? Policy? - From User point of view: I play this once, in different UI form. Why count it as twice? - Prevent counter up = if (reloadNotCount) { /* ignore */ } else { /* counter up */ } 34
35. Sounds complicated 35
36. So How? ಠ_ಠ 36
37. The Proposal 37
38. Principle - Balance UX and performance - 2019 devices has more RAM than my Mac - And more CPU cores too - Performance loss ~ UX gain - Avoid resource reloading as much as possible. - No reload on config change - Well lifecycle control does the magic - Design lifecycle flow if need 38
39. Demo 39
40. Definitions & Rules - Playable: a piece of resource, can be played. Example: an ExoPlayer instance - Target: to which a Playable can be played on - Object to present the playback on. Eg: PlayerView (in ExoPlayer library) - Playback: when a Playable is bound to a Target, it produces a Playback. Playback represents the connection of a Playable and Target - Playable ⇆ Target binding is unique, but not required - Binding same target will produce same Playback instance. - Different Playables bind to one Target: the later wins. - Not in bound Playable will be cleaned up eventually. 40
41. Team work - Playable must be bound to Target to be “noticed” - Playback observes Target’s behavior and trigger Playable 41
42. 42 Playable PlayerView Target PlayerView Target Playable, Target and Playback model
43. 43 Playable PlayerView Target PlayerView Target Playback Playable, Target and Playback model
44. 44 Playable PlayerView Target PlayerView Target Playback Playback Playable, Target and Playback model
45. Inspiration val player = ExoPlayerFactory.newSimpleInstance(context) val mediaSource = ExtractorMediaSource.Factory( DefaultDataSourceFactory(context, "¯\_(ツ)_/¯") ).createMediaSource(video.toUri()) player.prepare(mediaSource) player.playWhenReady = true // playerView.player = player ⬇ use below instead PlayerView.switchTargetView(player, null, playerView) 45
46. Inspiration val player = ExoPlayerFactory.newSimpleInstance(context) val mediaSource = ExtractorMediaSource.Factory( DefaultDataSourceFactory(context, "¯\_(ツ)_/¯") ).createMediaSource(video.toUri()) player.prepare(mediaSource) player.playWhenReady = true // playerView.player = player ⬇ use below instead PlayerView.switchTargetView(player, null, playerView) 46
47. Definitions (2) - Manager: manages Playback instances (also, acknowledge the Playable) - Ensure uniqueness - One Playable can belong to at-most one Manager - Global singleton: manages Playables and Managers - Also manage low layer resources like ExoPlayer instances, Factories - Observe lifecycles and dispatch actions to Managers 47
48. Components’ lifecycles 48
49. Manager (Activity lifecycle) Application lifecycle 49 Playable PlayerView Target PlayerView Target Playback Playback Playable, Target and Playback model
50. Component’s lifespan - Components stay alive as long as possible - Playable: contains only resource for playback, stays in Application lifecycle - Target: View, stays in Activity/Fragment lifecycle - Playback: min(Target, Playable), stays in Activity/Fragment lifecycle - Playable survives config changes - Playable can survive lifecycle changes if need - eg: from Activity to Activity - Manager, Playback: stay in Activity/Fragment lifecycle - Do not survive config changes 50
51. Target PlayerView Manager (Activity lifecycle) 51 Playable Playback Created Added Visible by User Invisible by User Active Inactive Removed Destroyed listeners bind System events Prepare Play Pause Release play? To be continued Application lifecycle
52. Summary - Similar to Fragment (but not the same) - → Easy to imagine and learn - → Easy to be afraid of? NO! There is no IllegalStateException to worry about. - This doesn’t take into account the actual playback logic - → Not limited to Video playback, but can be anything - Easier for testing 52
53. Cross-lifecycles 53
54. Config changes 54 Activity A (normal) Application lifecycle Activity A (multi windows) Playable isChangingConfigurations = true
55. Activity A Activity B activityA.startActivity(activityB) 55 Activity to Activity B::onCreate B::onStart A::onStop A::onDestroy B takes Playable from A, rebind B::onCreate B::onStart B::takesPlayable(tag) A::onStop A::onDestroy
56. Fragment A Fragment B fragmentTransaction.replace(containerId, fragmentB).commit() 56 Fragment to Fragment (same FragmentManager) B::onCreate B::onStart A::onStop A::onDestroy B takes Playable from A, rebind B::onCreate B::onStart B::takesPlayable(tag) A::onStop A::onDestroy Nahhh (>_<)
57. Fragment A Fragment B fragmentTransaction.replace(containerId, fragmentB).setReorderingAllowed(true).commit() 57 Fragment to Fragment (same FragmentManager) B::onCreate B::onStart A::onStop A::onDestroy B takes Playable from A, rebind B::onCreate B::onStart B::takesPlayable(tag) A::onStop A::onDestroy
58. Activity B Activity A Application lifecycle 58 Combined: complex UI Flow done well! Playable Playable Playable
59. Review 59 - ☑ UL Flow - ☑ No-reload on config changes - ☑ Playback continuity - Play/Pause on User interaction - Playback management - Playable implementation
60. Play/Pause on User interaction 60 - UI System callback: Window → DecorView → ViewTreeObserver - OnScrollChangedListener: scroll → recalculate on-screen visible area - OnAttachStateChangeListener: attach → visible, detach → not visible - Event flow: Event → Manager → Playback → Playable - Roadmap: plugin system - Client can provide custom implementation (eg: RecyclerView dedicated impl). - Fallback to default implementation.
61. Playback management: the Playable 61
62. Target PlayerView Manager (Activity lifecycle) 62 Playable Playback Created Added Visible by User Invisible by User Active Inactive Removed Destroyed listeners bind System events Prepare Play Pause Release play? To be continued Application lifecycle
63. Don’t care the actual playback logic 63
64. Playable can play anything 64
65. Bridge: provides Playable with actual playback resource/logic 65
66. 66 Playable Prepare Play Pause Release Bridge Prepare Play Pause Release Bridge impl for ExoPlayer Bridge impl for MediaPlayer (fallback) Bridge impl for ijkplayer Multi platform impl? Target Bind
67. 67 My built-in implementation Dummy impl for DroidKaigi demo (build in 5 min)
68. ExoBridge: Impl for ExoPlayer 68
69. ExoBridge impl principle - “More than one ExoPlayer instance” strategy - Globally manage a Player Pool. - Create on demand, release to Pool for reuse. - Cleanup Pool when no Managers are available. - Prepare resources as late as possible: in playable.play() - Resource warming up will affect the scroll performance. - Too early = frame drop at bad timing. 69
70. ExoBridge impl in practice - Global Singleton manages Player instances in Pool - Create and Release on Bridge’s demand. - Target: PlayerView - Prepare: Do nothing to ExoPlayer resource. Just to init listeners - Play: lazily create resource. - Ensure ExoPlayer instance, create if need. - Ensure MediaSource instance, create if need. - Pause: normally pause if in playing state - Release: release MediaSource, “release” ExoPlayer instance too ExoPlayer instance Pool - ExoPlayer is actually released when no Managers are available 70
71. ExoBridge overview - Max instance number = max number of Videos on screen at a moment - Play then Pause Video + still visible = holds an instance. - Play then Pause Video + scrolled off screen → Release - Release Video = instance is stored in Pool for reuse. - On screen, Pause then Play Video: no reload required. - Bridge target is set on demand, passively - → Do not require PlayerView all the time. - Client can manage PlayerView in anyway (eg: reuse single instance). 71
72. Wrap-up 72
73. Lifecycle design for playback continuity 73
74. Implementation for Balanced performance & UX 74
75. Target PlayerView Manager 75 Playable Playback Created Added Visible by User Invisible by User Active Inactive Removed Destroyed listeners bind System events Prepare Play Pause Release play? Bridge Prepare Play Pause Release Pool Created Destroy Cleanup
76. 76 Kohii[fragment].setUp(videoUrl) .asPlayable() .bind(playerView) .observe(viewLifecycleOwner)
77. One more thing 77
78. Kohii (RecyclerView + ExoPlayer) + MotionLayout + Selection + BottomSheetBehavior 78 https://github.com/googlesamples/android-ConstraintLayoutExamples
79. Thanks for listening \m/ 79