ExoPlayer in RecyclerView

ExoPlayer in RecyclerView

Presented at DroidKaigi 2019.

ExoPlayer in RecyclerView is an interesting topic that has been discussed for a long time, including this issue: https://github.com/google/ExoPlayer/issues/867

In this talk, I discuss about some challenges when implementing them, known approach and its issue. After all, I propose my approach that can not only fix those issue but also capable of doing more.

F8ca3e94570e4dc117f34563687a3b09?s=128

Nam Nguyen Hoai

February 07, 2019
Tweet

Transcript

  1. eneim ExoPlayer in RecyclerView a proposal DroidKaigi 2019

  2. Nam Nguyen ene.im / eneim / @ene__im 2 the simple,

    the best
  3. In this talk - On “ExoPlayer in RecyclerView”: why and

    how - Discuss common approaches and their issues - Discuss my proposal concept - Demonstrate my PoC using beta test app 3
  4. Before we start... 4

  5. 5 DEMO

  6. ExoPlayer RecyclerView 6

  7. 7 RecyclerView ecosystem https://lucasr.org/2014/07/31/the-new-twowayview/ ViewHolder LayoutManager Adapter ListAdapter Paging ItemAnimator

    ItemDecoration SnapHelper Selection ViewPager2
  8. 8 ExoPlayer ecosystem Building feature-rich media apps with ExoPlayer (Google

    I/O '18)
  9. ExoPlayer usage val player = ExoPlayerFactory.newSimpleInstance(context) val mediaSource = ExtractorMediaSource.Factory(

    DefaultDataSourceFactory(context, "¯\_(ツ)_/¯") ).createMediaSource(video.toUri()) player.prepare(mediaSource) player.playWhenReady = true // playerView.player = player ⬇ use below instead PlayerView.switchTargetView(player, null, playerView) 9
  10. Why ExoPlayer in RecyclerView? 10

  11. How ExoPlayer in RecyclerView? 11

  12. FYI: a 3.3-year-old issue 12

  13. How to use ExoPlayer in a ListView or RecyclerView? 13

    https://github.com/google/ExoPlayer/issues/867 Oct 15, 2015
  14. What they ask 14 I want to use ExoPlayer in

    a RecyclerView as a part of row item. I want to make a customer view and wrap the ExoPlayer in that view. Do you have some advice? Thank you!
  15. What they needs - Video player in RecyclerView, ViewPager, ScrollView,

    etc - ExoPlayer or whatever works - Auto play/pause on scroll? (like Facebook, Instagram, Twitter, etc) - Fullscreen back and forth, smoothly - Network usage friendly, battery friendly, UX/UI friendly, etc friendly 15
  16. 16

  17. Take a look around 17

  18. Facebook - Auto play/pause on scroll - Dialog for fullscreen

    player - Auto fullscreen on landscape - ಠ_ಠ: unstable under config changes, video reload on config change? 18 204.0.0.24.101
  19. YouTube 14.02.54 - Good UX in single item play -

    The mini/overlay player - Well config change handling - Latest: optional auto play in list - ಠ_ಠ: reload when switch from list to single player? 19
  20. AbemaTV 4.5.0 - ViewPager + ExoPlayer? - Fullscreen on landscape

    - ಠ_ಠ: [Active → Inactive → Active] shifting → black screen splash, reload on config change? 20
  21. 21

  22. Common objectives - Players in RecyclerView or ViewPager - Auto

    play/pause on user interactions (scroll, swipe, etc) - Single player most of the time. - Can be multiple players. Eg: Line Live - Fullscreen playback back and forth - Playback continuity - Others (eg: auto show/hide thumbnail etc) 22
  23. - RecyclerView: scrollable, infinitely - Item (ViewHolder) are supposed to

    be reused/recycled - Need a dynamic strategy - ViewHolder lifecycle → needs proper playback control - Created/Bound/Recycled - Attached/Detached - Many player instances = system performance down - The more player instances, the lower performance - Many PlayerView = many Surfaces = surface creation/management cost - The more Surfaces, the worse - Too strictly controlled = UX loss Challenges (1): ExoPlayer + RecyclerView 23
  24. Challenges (2): to fullscreen & back - UI flow: when

    to fullscreen? Rotate or not rotate? - Rotate → Config changes - Config changes handling - Not only orientation change - Important criteria: playback continuity (Video, Audio), UI look and feel 24
  25. Common approach - PlayerView in ViewHolder (VH) - Adapter: manage

    ExoPlayer and MediaSource, - MediaSource is created and bound to VH on demand - Only “will play” VH will be provided by the ExoPlayer instance - Adapter’s callback to update playbacks - onBindViewHolder → set MediaSource - onViewAttachedToWindow/onViewDetachedFromWindow → prepare/release? - onViewRecycled/onFailedToRecycleView → TODO? - RecyclerView callback to update playbacks (OnScrollListener) - Playback strategy: top-most visible → play, otherwise → pause - Reuse PlayerView: remove from “will pause” VH then add to “will play” VH 25
  26. Not yet resolved - Efficient Player management = UX loss

    - Reuse one Player for many PlayerView → “black splash” on resume (like AbemaTV app) - Re-buffering takes time - ViewHolder lifecycle - onViewDetachedFromWindow is not always called - setAdapter(null) ← to do or not to do? - UI Flow: to fullscreen and back - Video reloads on resuming has hidden issues 26
  27. To fullscreen and back 27

  28. Fullscreen → Single Player 28

  29. UI flow: when to single player? 29 Same orientation Landscape

    player Multi Windows state ಠ_ಠ UX on Android Pie
  30. Single player and config changes - Opening Single Player has

    many chances for config changes - Handle config changes manually (= add manifest entry) - Pros: no resource reloading, playback continuity done! - Cons: no adaptive layout, or manually apply using if/else/while … (not me (>_<)) - Many config changes patterns = error prone - Handle config changes automatically (= no manifest entry) - Pros: adaptive layout for each config - Cons/Unresolved: playback continuity 30
  31. 31 1 2 3 1 Click → Open single player

    in ‘same orientation’ 2 Click → Open single player in landscape 3 Rotate → From one form to other form of single player Config change UI flow: patterns (eg: YouTube)
  32. UI flow: others - Open single player on new Activity

    - → How to keep the playback continuity? - → How to come back and keep the playback continuity? - Open single player on Dialog (in same Activity) - → How to keep the playback continuity? - → How to come back and keep the playback continuity? 32
  33. Reload: the hidden issues 33

  34. Reload: the hidden issues - Reload = playback discontinuity -

    Reload = playback counter is up? - Analytic metrics matters? Policy? - From User point of view: I play this once, in different UI form. Why count it as twice? - Prevent counter up = if (reloadNotCount) { /* ignore */ } else { /* counter up */ } 34
  35. Sounds complicated 35

  36. So How? ಠ_ಠ 36

  37. The Proposal 37

  38. Principle - Balance UX and performance - 2019 devices has

    more RAM than my Mac - And more CPU cores too - Performance loss ~ UX gain - Avoid resource reloading as much as possible. - No reload on config change - Well lifecycle control does the magic - Design lifecycle flow if need 38
  39. Demo 39

  40. Definitions & Rules - Playable: a piece of resource, can

    be played. Example: an ExoPlayer instance - Target: to which a Playable can be played on - Object to present the playback on. Eg: PlayerView (in ExoPlayer library) - Playback: when a Playable is bound to a Target, it produces a Playback. Playback represents the connection of a Playable and Target - Playable ⇆ Target binding is unique, but not required - Binding same target will produce same Playback instance. - Different Playables bind to one Target: the later wins. - Not in bound Playable will be cleaned up eventually. 40
  41. Team work - Playable must be bound to Target to

    be “noticed” - Playback observes Target’s behavior and trigger Playable 41
  42. 42 Playable PlayerView Target PlayerView Target Playable, Target and Playback

    model
  43. 43 Playable PlayerView Target PlayerView Target Playback Playable, Target and

    Playback model
  44. 44 Playable PlayerView Target PlayerView Target Playback Playback Playable, Target

    and Playback model
  45. Inspiration val player = ExoPlayerFactory.newSimpleInstance(context) val mediaSource = ExtractorMediaSource.Factory( DefaultDataSourceFactory(context,

    "¯\_(ツ)_/¯") ).createMediaSource(video.toUri()) player.prepare(mediaSource) player.playWhenReady = true // playerView.player = player ⬇ use below instead PlayerView.switchTargetView(player, null, playerView) 45
  46. Inspiration val player = ExoPlayerFactory.newSimpleInstance(context) val mediaSource = ExtractorMediaSource.Factory( DefaultDataSourceFactory(context,

    "¯\_(ツ)_/¯") ).createMediaSource(video.toUri()) player.prepare(mediaSource) player.playWhenReady = true // playerView.player = player ⬇ use below instead PlayerView.switchTargetView(player, null, playerView) 46
  47. Definitions (2) - Manager: manages Playback instances (also, acknowledge the

    Playable) - Ensure uniqueness - One Playable can belong to at-most one Manager - Global singleton: manages Playables and Managers - Also manage low layer resources like ExoPlayer instances, Factories - Observe lifecycles and dispatch actions to Managers 47
  48. Components’ lifecycles 48

  49. Manager (Activity lifecycle) Application lifecycle 49 Playable PlayerView Target PlayerView

    Target Playback Playback Playable, Target and Playback model
  50. Component’s lifespan - Components stay alive as long as possible

    - Playable: contains only resource for playback, stays in Application lifecycle - Target: View, stays in Activity/Fragment lifecycle - Playback: min(Target, Playable), stays in Activity/Fragment lifecycle - Playable survives config changes - Playable can survive lifecycle changes if need - eg: from Activity to Activity - Manager, Playback: stay in Activity/Fragment lifecycle - Do not survive config changes 50
  51. Target PlayerView Manager (Activity lifecycle) 51 Playable Playback Created Added

    Visible by User Invisible by User Active Inactive Removed Destroyed listeners bind System events Prepare Play Pause Release play? To be continued Application lifecycle
  52. Summary - Similar to Fragment (but not the same) -

    → Easy to imagine and learn - → Easy to be afraid of? NO! There is no IllegalStateException to worry about. - This doesn’t take into account the actual playback logic - → Not limited to Video playback, but can be anything - Easier for testing 52
  53. Cross-lifecycles 53

  54. Config changes 54 Activity A (normal) Application lifecycle Activity A

    (multi windows) Playable isChangingConfigurations = true
  55. Activity A Activity B activityA.startActivity(activityB) 55 Activity to Activity B::onCreate

    B::onStart A::onStop A::onDestroy B takes Playable from A, rebind B::onCreate B::onStart B::takesPlayable(tag) A::onStop A::onDestroy
  56. Fragment A Fragment B fragmentTransaction.replace(containerId, fragmentB).commit() 56 Fragment to Fragment

    (same FragmentManager) B::onCreate B::onStart A::onStop A::onDestroy B takes Playable from A, rebind B::onCreate B::onStart B::takesPlayable(tag) A::onStop A::onDestroy Nahhh (>_<)
  57. Fragment A Fragment B fragmentTransaction.replace(containerId, fragmentB).setReorderingAllowed(true).commit() 57 Fragment to Fragment

    (same FragmentManager) B::onCreate B::onStart A::onStop A::onDestroy B takes Playable from A, rebind B::onCreate B::onStart B::takesPlayable(tag) A::onStop A::onDestroy
  58. Activity B Activity A Application lifecycle 58 Combined: complex UI

    Flow done well! Playable Playable Playable
  59. Review 59 - ☑ UL Flow - ☑ No-reload on

    config changes - ☑ Playback continuity - Play/Pause on User interaction - Playback management - Playable implementation
  60. Play/Pause on User interaction 60 - UI System callback: Window

    → DecorView → ViewTreeObserver - OnScrollChangedListener: scroll → recalculate on-screen visible area - OnAttachStateChangeListener: attach → visible, detach → not visible - Event flow: Event → Manager → Playback → Playable - Roadmap: plugin system - Client can provide custom implementation (eg: RecyclerView dedicated impl). - Fallback to default implementation.
  61. Playback management: the Playable 61

  62. Target PlayerView Manager (Activity lifecycle) 62 Playable Playback Created Added

    Visible by User Invisible by User Active Inactive Removed Destroyed listeners bind System events Prepare Play Pause Release play? To be continued Application lifecycle
  63. Don’t care the actual playback logic 63

  64. Playable can play anything 64

  65. Bridge: provides Playable with actual playback resource/logic 65

  66. 66 Playable Prepare Play Pause Release Bridge Prepare Play Pause

    Release Bridge impl for ExoPlayer Bridge impl for MediaPlayer (fallback) Bridge impl for ijkplayer Multi platform impl? Target Bind
  67. 67 My built-in implementation Dummy impl for DroidKaigi demo (build

    in 5 min)
  68. ExoBridge: Impl for ExoPlayer 68

  69. ExoBridge impl principle - “More than one ExoPlayer instance” strategy

    - Globally manage a Player Pool. - Create on demand, release to Pool for reuse. - Cleanup Pool when no Managers are available. - Prepare resources as late as possible: in playable.play() - Resource warming up will affect the scroll performance. - Too early = frame drop at bad timing. 69
  70. ExoBridge impl in practice - Global Singleton manages Player instances

    in Pool - Create and Release on Bridge’s demand. - Target: PlayerView - Prepare: Do nothing to ExoPlayer resource. Just to init listeners - Play: lazily create resource. - Ensure ExoPlayer instance, create if need. - Ensure MediaSource instance, create if need. - Pause: normally pause if in playing state - Release: release MediaSource, “release” ExoPlayer instance too ExoPlayer instance Pool - ExoPlayer is actually released when no Managers are available 70
  71. ExoBridge overview - Max instance number = max number of

    Videos on screen at a moment - Play then Pause Video + still visible = holds an instance. - Play then Pause Video + scrolled off screen → Release - Release Video = instance is stored in Pool for reuse. - On screen, Pause then Play Video: no reload required. - Bridge target is set on demand, passively - → Do not require PlayerView all the time. - Client can manage PlayerView in anyway (eg: reuse single instance). 71
  72. Wrap-up 72

  73. Lifecycle design for playback continuity 73

  74. Implementation for Balanced performance & UX 74

  75. Target PlayerView Manager 75 Playable Playback Created Added Visible by

    User Invisible by User Active Inactive Removed Destroyed listeners bind System events Prepare Play Pause Release play? Bridge Prepare Play Pause Release Pool Created Destroy Cleanup
  76. 76 Kohii[fragment].setUp(videoUrl) .asPlayable() .bind(playerView) .observe(viewLifecycleOwner)

  77. One more thing 77

  78. Kohii (RecyclerView + ExoPlayer) + MotionLayout + Selection + BottomSheetBehavior

    78 https://github.com/googlesamples/android-ConstraintLayoutExamples
  79. Thanks for listening \m/ 79