Slide 1
Slide 1 text
eneim
ExoPlayer in RecyclerView
a proposal
DroidKaigi 2019
Slide 2
Slide 2 text
Nam Nguyen
ene.im / eneim / @ene__im
2
the simple, the best
Slide 3
Slide 3 text
In this talk
- On “ExoPlayer in RecyclerView”: why and how
- Discuss common approaches and their issues
- Discuss my proposal concept
- Demonstrate my PoC using beta test app
3
Slide 4
Slide 4 text
Before we start...
4
Slide 5
Slide 5 text
5
DEMO
Slide 6
Slide 6 text
ExoPlayer RecyclerView
6
Slide 7
Slide 7 text
7
RecyclerView ecosystem
https://lucasr.org/2014/07/31/the-new-twowayview/
ViewHolder
LayoutManager Adapter ListAdapter Paging
ItemAnimator ItemDecoration
SnapHelper
Selection
ViewPager2
Slide 8
Slide 8 text
8
ExoPlayer ecosystem
Building feature-rich media apps with ExoPlayer (Google I/O '18)
Slide 9
Slide 9 text
ExoPlayer
usage
val player = ExoPlayerFactory.newSimpleInstance(context)
val mediaSource = ExtractorMediaSource.Factory(
DefaultDataSourceFactory(context, "¯\_(ツ)_/¯")
).createMediaSource(video.toUri())
player.prepare(mediaSource)
player.playWhenReady = true
// playerView.player = player ⬇ use below instead
PlayerView.switchTargetView(player, null, playerView)
9
Slide 10
Slide 10 text
Why ExoPlayer in
RecyclerView?
10
Slide 11
Slide 11 text
How ExoPlayer in
RecyclerView?
11
Slide 12
Slide 12 text
FYI: a 3.3-year-old issue
12
Slide 13
Slide 13 text
How to use ExoPlayer in
a ListView or RecyclerView?
13
https://github.com/google/ExoPlayer/issues/867
Oct 15, 2015
Slide 14
Slide 14 text
What they
ask
14
I want to use ExoPlayer in a RecyclerView as
a part of row item.
I want to make a customer view and wrap
the ExoPlayer in that view.
Do you have some advice?
Thank you!
Slide 15
Slide 15 text
What they
needs
- Video player in RecyclerView,
ViewPager, ScrollView, etc
- ExoPlayer or whatever works
- Auto play/pause on scroll? (like
Facebook, Instagram, Twitter, etc)
- Fullscreen back and forth, smoothly
- Network usage friendly, battery
friendly, UX/UI friendly, etc friendly
15
Slide 16
Slide 16 text
16
Slide 17
Slide 17 text
Take a look around
17
Slide 18
Slide 18 text
Facebook
- Auto play/pause on scroll
- Dialog for fullscreen player
- Auto fullscreen on landscape
- ಠ_ಠ: unstable under config changes,
video reload on config change?
18
204.0.0.24.101
Slide 19
Slide 19 text
YouTube
14.02.54
- Good UX in single item play
- The mini/overlay player
- Well config change handling
- Latest: optional auto play in list
- ಠ_ಠ: reload when switch from list to
single player?
19
Slide 20
Slide 20 text
AbemaTV
4.5.0
- ViewPager + ExoPlayer?
- Fullscreen on landscape
- ಠ_ಠ: [Active → Inactive → Active]
shifting → black screen splash, reload
on config change?
20
Slide 21
Slide 21 text
21
Slide 22
Slide 22 text
Common objectives
- Players in RecyclerView or ViewPager
- Auto play/pause on user interactions (scroll, swipe, etc)
- Single player most of the time.
- Can be multiple players. Eg: Line Live
- Fullscreen playback back and forth
- Playback continuity
- Others (eg: auto show/hide thumbnail etc)
22
Slide 23
Slide 23 text
- RecyclerView: scrollable, infinitely
- Item (ViewHolder) are supposed to be reused/recycled
- Need a dynamic strategy
- ViewHolder lifecycle → needs proper playback control
- Created/Bound/Recycled
- Attached/Detached
- Many player instances = system performance down
- The more player instances, the lower performance
- Many PlayerView = many Surfaces = surface creation/management cost
- The more Surfaces, the worse
- Too strictly controlled = UX loss
Challenges (1): ExoPlayer + RecyclerView
23
Slide 24
Slide 24 text
Challenges (2): to fullscreen & back
- UI flow: when to fullscreen? Rotate or not rotate?
- Rotate → Config changes
- Config changes handling
- Not only orientation change
- Important criteria: playback continuity (Video, Audio), UI look and feel
24
Slide 25
Slide 25 text
Common approach
- PlayerView in ViewHolder (VH)
- Adapter: manage ExoPlayer and MediaSource,
- MediaSource is created and bound to VH on demand
- Only “will play” VH will be provided by the ExoPlayer instance
- Adapter’s callback to update playbacks
- onBindViewHolder → set MediaSource
- onViewAttachedToWindow/onViewDetachedFromWindow → prepare/release?
- onViewRecycled/onFailedToRecycleView → TODO?
- RecyclerView callback to update playbacks (OnScrollListener)
- Playback strategy: top-most visible → play, otherwise → pause
- Reuse PlayerView: remove from “will pause” VH then add to “will play” VH
25
Slide 26
Slide 26 text
Not yet resolved
- Efficient Player management = UX loss
- Reuse one Player for many PlayerView → “black splash” on resume (like AbemaTV app)
- Re-buffering takes time
- ViewHolder lifecycle
- onViewDetachedFromWindow is not always called
- setAdapter(null) ← to do or not to do?
- UI Flow: to fullscreen and back
- Video reloads on resuming has hidden issues
26
Slide 27
Slide 27 text
To fullscreen and back
27
Slide 28
Slide 28 text
Fullscreen → Single Player
28
Slide 29
Slide 29 text
UI flow: when to single player?
29
Same orientation Landscape player
Multi Windows state
ಠ_ಠ UX on
Android Pie
Slide 30
Slide 30 text
Single player and config changes
- Opening Single Player has many chances for config changes
- Handle config changes manually (= add manifest entry)
- Pros: no resource reloading, playback continuity done!
- Cons: no adaptive layout, or manually apply using if/else/while … (not me (>_<))
- Many config changes patterns = error prone
- Handle config changes automatically (= no manifest entry)
- Pros: adaptive layout for each config
- Cons/Unresolved: playback continuity
30
Slide 31
Slide 31 text
31
1
2
3
1 Click → Open single player in ‘same
orientation’
2 Click → Open single player in landscape
3 Rotate → From one form to other form of
single player
Config change
UI flow: patterns (eg: YouTube)
Slide 32
Slide 32 text
UI flow: others
- Open single player on new Activity
- → How to keep the playback continuity?
- → How to come back and keep the playback continuity?
- Open single player on Dialog (in same Activity)
- → How to keep the playback continuity?
- → How to come back and keep the playback continuity?
32
Slide 33
Slide 33 text
Reload: the hidden issues
33
Slide 34
Slide 34 text
Reload: the hidden issues
- Reload = playback discontinuity
- Reload = playback counter is up?
- Analytic metrics matters? Policy?
- From User point of view: I play this once, in different UI form. Why count it as twice?
- Prevent counter up = if (reloadNotCount) { /* ignore */ } else { /* counter up */ }
34
Slide 35
Slide 35 text
Sounds complicated
35
Slide 36
Slide 36 text
So How? ಠ_ಠ
36
Slide 37
Slide 37 text
The Proposal
37
Slide 38
Slide 38 text
Principle
- Balance UX and performance
- 2019 devices has more RAM than my Mac
- And more CPU cores too
- Performance loss ~ UX gain
- Avoid resource reloading as much as
possible.
- No reload on config change
- Well lifecycle control does the magic
- Design lifecycle flow if need
38
Slide 39
Slide 39 text
Demo
39
Slide 40
Slide 40 text
Definitions & Rules
- Playable: a piece of resource, can be played. Example: an ExoPlayer
instance
- Target: to which a Playable can be played on
- Object to present the playback on. Eg: PlayerView (in ExoPlayer library)
- Playback: when a Playable is bound to a Target, it produces a Playback.
Playback represents the connection of a Playable and Target
- Playable ⇆ Target binding is unique, but not required
- Binding same target will produce same Playback instance.
- Different Playables bind to one Target: the later wins.
- Not in bound Playable will be cleaned up eventually.
40
Slide 41
Slide 41 text
Team work
- Playable must be bound to Target to be “noticed”
- Playback observes Target’s behavior and trigger Playable
41
Slide 42
Slide 42 text
42
Playable
PlayerView
Target
PlayerView
Target
Playable, Target and Playback model
Slide 43
Slide 43 text
43
Playable
PlayerView
Target
PlayerView
Target
Playback
Playable, Target and Playback model
Slide 44
Slide 44 text
44
Playable
PlayerView
Target
PlayerView
Target
Playback
Playback
Playable, Target and Playback model
Slide 45
Slide 45 text
Inspiration
val player = ExoPlayerFactory.newSimpleInstance(context)
val mediaSource = ExtractorMediaSource.Factory(
DefaultDataSourceFactory(context, "¯\_(ツ)_/¯")
).createMediaSource(video.toUri())
player.prepare(mediaSource)
player.playWhenReady = true
// playerView.player = player ⬇ use below instead
PlayerView.switchTargetView(player, null, playerView)
45
Slide 46
Slide 46 text
Inspiration
val player = ExoPlayerFactory.newSimpleInstance(context)
val mediaSource = ExtractorMediaSource.Factory(
DefaultDataSourceFactory(context, "¯\_(ツ)_/¯")
).createMediaSource(video.toUri())
player.prepare(mediaSource)
player.playWhenReady = true
// playerView.player = player ⬇ use below instead
PlayerView.switchTargetView(player, null, playerView)
46
Slide 47
Slide 47 text
Definitions (2)
- Manager: manages Playback instances (also, acknowledge the Playable)
- Ensure uniqueness
- One Playable can belong to at-most one Manager
- Global singleton: manages Playables and Managers
- Also manage low layer resources like ExoPlayer instances, Factories
- Observe lifecycles and dispatch actions to Managers
47
Slide 48
Slide 48 text
Components’ lifecycles
48
Slide 49
Slide 49 text
Manager (Activity lifecycle)
Application lifecycle
49
Playable
PlayerView
Target
PlayerView
Target
Playback
Playback
Playable, Target and Playback model
Slide 50
Slide 50 text
Component’s lifespan
- Components stay alive as long as possible
- Playable: contains only resource for playback, stays in Application lifecycle
- Target: View, stays in Activity/Fragment lifecycle
- Playback: min(Target, Playable), stays in Activity/Fragment lifecycle
- Playable survives config changes
- Playable can survive lifecycle changes if need
- eg: from Activity to Activity
- Manager, Playback: stay in Activity/Fragment lifecycle
- Do not survive config changes
50
Slide 51
Slide 51 text
Target PlayerView
Manager (Activity lifecycle)
51
Playable Playback
Created
Added
Visible by User
Invisible by User
Active
Inactive
Removed
Destroyed
listeners
bind
System events
Prepare
Play
Pause
Release
play?
To be continued
Application lifecycle
Slide 52
Slide 52 text
Summary
- Similar to Fragment (but not the same)
- → Easy to imagine and learn
- → Easy to be afraid of? NO! There is no IllegalStateException to worry about.
- This doesn’t take into account the actual playback logic
- → Not limited to Video playback, but can be anything
- Easier for testing
52
Slide 53
Slide 53 text
Cross-lifecycles
53
Slide 54
Slide 54 text
Config changes
54
Activity A (normal)
Application lifecycle
Activity A (multi windows)
Playable
isChangingConfigurations = true
Slide 55
Slide 55 text
Activity A Activity B
activityA.startActivity(activityB)
55
Activity to Activity
B::onCreate
B::onStart
A::onStop
A::onDestroy
B takes
Playable from
A, rebind
B::onCreate B::onStart B::takesPlayable(tag) A::onStop A::onDestroy
Slide 56
Slide 56 text
Fragment A Fragment B
fragmentTransaction.replace(containerId, fragmentB).commit()
56
Fragment to Fragment (same FragmentManager)
B::onCreate
B::onStart
A::onStop
A::onDestroy
B takes
Playable from
A, rebind
B::onCreate B::onStart B::takesPlayable(tag) A::onStop A::onDestroy
Nahhh (>_<)
Slide 57
Slide 57 text
Fragment A Fragment B
fragmentTransaction.replace(containerId, fragmentB).setReorderingAllowed(true).commit()
57
Fragment to Fragment (same FragmentManager)
B::onCreate
B::onStart
A::onStop
A::onDestroy
B takes
Playable from
A, rebind
B::onCreate B::onStart B::takesPlayable(tag) A::onStop A::onDestroy
Slide 58
Slide 58 text
Activity B
Activity A
Application lifecycle
58
Combined: complex UI Flow done well!
Playable Playable Playable
Slide 59
Slide 59 text
Review
59
- ☑ UL Flow
- ☑ No-reload on config changes
- ☑ Playback continuity
- Play/Pause on User interaction
- Playback management
- Playable implementation
Slide 60
Slide 60 text
Play/Pause on User interaction
60
- UI System callback: Window → DecorView → ViewTreeObserver
- OnScrollChangedListener: scroll → recalculate on-screen visible area
- OnAttachStateChangeListener: attach → visible, detach → not visible
- Event flow: Event → Manager → Playback → Playable
- Roadmap: plugin system
- Client can provide custom implementation (eg: RecyclerView dedicated impl).
- Fallback to default implementation.
Slide 61
Slide 61 text
Playback management:
the Playable
61
Slide 62
Slide 62 text
Target PlayerView
Manager (Activity lifecycle)
62
Playable Playback
Created
Added
Visible by User
Invisible by User
Active
Inactive
Removed
Destroyed
listeners
bind
System events
Prepare
Play
Pause
Release
play?
To be continued
Application lifecycle
Slide 63
Slide 63 text
Don’t care the actual playback logic
63
Slide 64
Slide 64 text
Playable can play anything
64
Slide 65
Slide 65 text
Bridge: provides Playable with
actual playback resource/logic
65
Slide 66
Slide 66 text
66
Playable
Prepare
Play
Pause
Release
Bridge
Prepare
Play
Pause
Release
Bridge impl for
ExoPlayer
Bridge impl for
MediaPlayer (fallback)
Bridge impl for
ijkplayer
Multi platform impl?
Target
Bind
Slide 67
Slide 67 text
67
My built-in
implementation
Dummy
impl for DroidKaigi demo
(build in 5 min)
Slide 68
Slide 68 text
ExoBridge: Impl for ExoPlayer
68
Slide 69
Slide 69 text
ExoBridge impl principle
- “More than one ExoPlayer instance” strategy
- Globally manage a Player Pool.
- Create on demand, release to Pool for reuse.
- Cleanup Pool when no Managers are available.
- Prepare resources as late as possible: in playable.play()
- Resource warming up will affect the scroll performance.
- Too early = frame drop at bad timing.
69
Slide 70
Slide 70 text
ExoBridge impl in practice
- Global Singleton manages Player instances in Pool
- Create and Release on Bridge’s demand.
- Target: PlayerView
- Prepare: Do nothing to ExoPlayer resource. Just to init listeners
- Play: lazily create resource.
- Ensure ExoPlayer instance, create if need.
- Ensure MediaSource instance, create if need.
- Pause: normally pause if in playing state
- Release: release MediaSource, “release” ExoPlayer instance too ExoPlayer
instance Pool
- ExoPlayer is actually released when no Managers are available
70
Slide 71
Slide 71 text
ExoBridge overview
- Max instance number = max number of Videos on screen at a moment
- Play then Pause Video + still visible = holds an instance.
- Play then Pause Video + scrolled off screen → Release
- Release Video = instance is stored in Pool for reuse.
- On screen, Pause then Play Video: no reload required.
- Bridge target is set on demand, passively
- → Do not require PlayerView all the time.
- Client can manage PlayerView in anyway (eg: reuse single instance).
71
Slide 72
Slide 72 text
Wrap-up
72
Slide 73
Slide 73 text
Lifecycle design for
playback continuity
73
Slide 74
Slide 74 text
Implementation for
Balanced performance & UX
74
Slide 75
Slide 75 text
Target PlayerView
Manager
75
Playable Playback
Created
Added
Visible by User
Invisible by User
Active
Inactive
Removed
Destroyed
listeners
bind
System events
Prepare
Play
Pause
Release
play?
Bridge
Prepare
Play
Pause
Release
Pool
Created
Destroy Cleanup
Slide 76
Slide 76 text
76
Kohii[fragment].setUp(videoUrl)
.asPlayable()
.bind(playerView)
.observe(viewLifecycleOwner)
Slide 77
Slide 77 text
One more thing
77
Slide 78
Slide 78 text
Kohii (RecyclerView + ExoPlayer) +
MotionLayout + Selection +
BottomSheetBehavior
78
https://github.com/googlesamples/android-ConstraintLayoutExamples
Slide 79
Slide 79 text
Thanks for listening \m/
79