How we boosted ExoPlayer performance by 30%

Transcript

1. How we boosted ExoPlayer performance by 30% Alexey Bykov @nonewsss
2. 2 Alexey Bykov Senior Software Engineer 8 years Google Developer Expert @nonewsss This is me This is not
3. 3 Reddit in numbers ~100 Engineers 70% Compose 100% Kotlin ~700 Modules
4. 4 Video in numbers ~ 20+ screens
5. 5 20+ screens
6. 6 Bonus:
7. 7 Challenges
8. 8 Challenges ~ Performance: ~ Bandwidth instability: * TTFF, Rebuffering, Quality * 3g, 4g, 5g & WiFi
9. 9 Challenges ~ Performance: ~ Bandwidth instability: * TTFF, Rebuffering, Quality ~ Device capabilities: * 3g, 4g, 5g & WiFi * Decoders instances
10. 10 Challenges ~ Performance: ~ Bandwidth instability: * TTFF, Rebuffering, Quality ~ Device capabilities: * 3g, 4g, 5g & WiFi * Decoders instances ~ Observability & analytics: * Trust & platforms implementation differences
11. 11 Early 2024 In worst cases (P95), It took almost 2 seconds to render a first frame ms
12. 12 Agenda ~ ExoPlayer: under the hood ~ Cases study: ~ Delivery ~ Prefetching ~ Pools: Views, Players ~ http 2/0 & 3/0 ~ SurfaceView & TextureView ~ Performance metrics ~ Pre-warming ~ LoadControl
13. Goal Give you practical take home tips
14. 14 Verify everything that you will hear today by A/B Tests
15. 15 What’s going on when we want to play video?
16. 16 ABR (dash) Playback Backend Give me batch of videos CDN .mpd <Period duration="PT0H4M40.414S">
17. 17 .mpd or Manifest <MPD xmlns="urn:mpeg:DASH:schema:MPD:2011"> <Period duration="PT0H4M40.414S"> <AdaptationSet mimeType="video/mp4"> <Representation bandwidth="250000" id="video_lowest" /> <Representation bandwidth="500000" id="video_low" /> <Representation bandwidth="1000000" id="video_medium" /> <Representation bandwidth="2000000" id="video_high" /> </AdaptationSet> <AdaptationSet mimeType="audio/mp4"> <Representation bandwidth="64000" id="audio_low" /> <Representation bandwidth="128000" id="audio_high" /> </AdaptationSet> </Period> </MPD>
18. 18 .mpd or Manifest <MPD xmlns="urn:mpeg:DASH:schema:MPD:2011"> <Period duration="PT0H4M40.414S"> <AdaptationSet mimeType="video/mp4"> <Representation bandwidth="250000" id="video_lowest" /> <Representation bandwidth="500000" id="video_low" /> <Representation bandwidth="1000000" id="video_medium" /> <Representation bandwidth="2000000" id="video_high" /> </AdaptationSet> <AdaptationSet mimeType="audio/mp4"> <Representation bandwidth="64000" id="audio_low" /> <Representation bandwidth="128000" id="audio_high" /> </AdaptationSet> </Period> </MPD>
19. 19 .mpd or Manifest <MPD xmlns="urn:mpeg:DASH:schema:MPD:2011"> <Period duration="PT0H4M40.414S"> <AdaptationSet mimeType="video/mp4"> <Representation bandwidth="250000" id="video_lowest" /> <Representation bandwidth="500000" id="video_low" /> <Representation bandwidth="1000000" id="video_medium" /> <Representation bandwidth="2000000" id="video_high" /> </AdaptationSet> <AdaptationSet mimeType="audio/mp4"> <Representation bandwidth="64000" id="audio_low" /> <Representation bandwidth="128000" id="audio_high" /> </AdaptationSet> </Period> </MPD>
20. 20 ABR (dash) Playback Backend Give me batch of videos CDN audio (selected quality) video (selected quality) .mpd <Period duration="PT0H4M40.414S"> decoding() initDecoders()
21. 21 Media3 ExoPlayer
22. 22 ExoPlayer Architecture
24. 24 Can be: - Dash - HLS - SS - Progressive (Mp4)
28. 28 Goal: - Select audio track - Select video track - Select text track
31. 31 Goal: - Provide capabilities - Decoding
34. Goal: - Should I continue buffering? - Can I start playback?
36. player.prepare( url = "myawesomevideo.mp4" )
37. Headers
38. Headers
39. Headers
40. Headers RC RC RC
41. TS
42. TS
44. Load!
45. Load!
50. Loaded!
51. Loaded!
52. Loaded!
54. Load!
55. Load!
57. Loaded!
58. Loaded!
60. Loaded!
61. 61 Case studies You ExoPlayer and me
62. 62 Video delivery
63. 63 Dash playback CDN audio (selected quality) video (selected quality) .mpd ¯\_(ツ)_/¯ Something went wrong mp4
64. 64 HLS & Dash manifest latency ~5ms
65. 65 Playback errors: Video view: -5% +1.7% Mp4 vs Adaptive Video loading: ~30-40% faster
66. 66 >45 seconds Playback errors: Video view: -5% +1.7% Use dash/hls if: Use mp4 if: <45 seconds Mp4 vs Adaptive DO Video loading: +63% faster
67. 67 Http version
68. 68 Http version Exo uses by default: http/1 (1996) With OkHttp: http/2 (2015) With Cronet or Http engine: http/2 or http/3 (2022) https://www.linkedin.com/pulse/http-10-vs-11-20-30-swadhin-pattnaik/
69. 69 OkHttp No plans to support: http/3 https://github.com/google/cronet-transport-for-okhttp https://github.com/square/okhttp/issues/907 But possible via Cronet engine + interceptors:
70. 70 Implementation fun createEngine(): CronetEngine { return CronetEngine.Builder(context) .setStoragePath(context.cacheDir("cronetHttp2")) .enableHttpCache(HTTP_CACHE_DISK, CACHE_SIZE) .enableHttp2(true) .build() }
71. 71 DataSource CronetDataSource.Factory(engine, executor) fun createEngine(): CronetEngine { return CronetEngine.Builder(context) .setStoragePath(context.cacheDir("cronetHttp2")) .enableHttpCache(HTTP_CACHE_DISK, CACHE_SIZE) .enableHttp2(true) .build() }
72. 72 API 34+ if (Build.VERSION.SDK_INT >= 34) { // An instance of this class can be created using Builder. val httpEngine = HttpEngine .Builder(applicationContext) .setEnableHttp2(true) .build() return HttpEngineDataSource.Factory(httpEngine, executor) }
73. 73 Current results Load<500ms: +2.6% Load>1s: -7% Load>2s: -1.5% Rebuffing: -1.4 % Load<250ms: +1%
74. 74 Current results Load<500ms: +2.6% Load>1s: -7% Load>2s: -1.5% Rebuffing: -1.4 % Load<250ms: +1% DO
75. 75 and me LoadControl
76. 76 60 2.5 Blocking buffering LoadControl You and me
77. 77 LoadControl Blocking buffering Playback 2.5 60
78. 78 Blocking buffering Playback 2.5 60 50 Non-blocking buffering LoadControl
79. 79 2 flags: 2.5 60 52.5 Blocking buffering * bufferForPlaybackMs = 2.5s - First Frame - Playback stopped by user * bufferForPlaybackAfterRebuffer = 5s - Network latency problems
80. 80 2.5 60 52.5 Non-blocking buffering * minBufferMs = 50s * maxBufferMs = 50s 2 flags:
81. 81 * bufferForPlaybackMs Load control: Recommendations Default: 2.5s ~1s —> * bufferForPlaybackAfterRebuffer Default: 5s ~1s —> * minBufferMs & maxBufferMs Default: 50s ~20s —> * setPrioritizeTimeOverSizeThresholds Default: false true —> Keep them equal (drip feeding technique) !
82. 82 Results: Load control Load<500ms: +4.4% Load>1s: -11.9% Load>2s: -17.7% Video view: +1.5 %
83. 83 Results: Load control Load<500ms: +4.4% Load>1s: -11.9% DO Load>2s: -17.7%
84. 84 Code DefaultLoadControl.Builder() .setPrioritizeTimeOverSizeThresholds(true) .setBufferDurationsMs( minBufferMs = , maxBufferMs =, bufferForPlaybackMs = , bufferForPlaybackAfterRebufferMs =, ) .build()
85. 85 Pre-fetching & Pre-warming and me
86. 86 and me * Pre-fetching: Download file earlier —> cache hit * Pre-warming: Load data to memory, decode first segments —> immediate play Pre-fetching & Pre-warming
87. 87 Caching pitfalls and me
88. 88 Caching pitfalls and me * SimpleCache hits disk in constructor
89. 89 withContext(dispatcherProvider.io) { SimpleCache(…) } SimpleCache instance DO
90. 90 Caching pitfalls and me * SimpleCache hits disk in constructor * SimpleCache may clear other cache ¯\_(ツ)_/¯
91. 91 SimpleCache doc
92. 92 SimpleCache doc SimpleCache( File(androidCacheDir, "video_cache_folder") LeastRecentlyUsedCacheEvictor(maxCacheSize), StandaloneDatabaseProvider(context), )
93. 93 SimpleCache doc SimpleCache( File(androidCacheDir, "video_cache_folder") LeastRecentlyUsedCacheEvictor(maxCacheSize), StandaloneDatabaseProvider(context), )
94. 94 SimpleCache doc SimpleCache( File(androidCacheDir, "video_cache_folder") LeastRecentlyUsedCacheEvictor(maxCacheSize), StandaloneDatabaseProvider(context), )
95. 95 Caching pitfalls and me * SimpleCache hits disk in constructor * SimpleCache may clear other cache * You should have only 1 instance of SimpleCache
96. 96 Caching pitfalls and me * SimpleCache hits disk in constructor * SimpleCache may clear other cache * You should have only 1 instance of SimpleCache * Don’t forget about eviction: LeastRecentlyUsedCacheEvictor(maxCacheSize)
97. 97 Caching pitfalls and me * SimpleCache hits disk in constructor * SimpleCache may clear other cache * You should have only 1 instance of SimpleCache * Don’t forget about eviction: LeastRecentlyUsedCacheEvictor(maxCacheSize) * Don’t forget that URL == cacheKey
98. 98 Prefetching: Lazy and me
99. 99 Prefetching: Lazy and me
100. 100 Prefetching: Lazy and me
101. 101 Prefetching: Lazy and me
102. 102 Prefetching: Lazy and me
103. 103 Prefetching: Lazy and me Some Text post!…
104. 104 Prefetching: Lazy and me Some Text post!…
105. 105 Prefetching: Lazy and me Some Text post!…
106. 106 Playback errors: The same +2% Load < 500ms: Video view: +1.7% Prefetching: Lazy Load < 250ms: The same
107. 107 Prefetching: Lazy — Problem and me
108. 108 Prefetching: Lazy — Problem and me ~250ms!
109. 109 Prefetching: Early and me
110. 110 Prefetching: Early and me
111. 111 Prefetching: Aggressive and me All in batch
112. 112 Prefetching: Aggressive +2.1% Load < 500ms: Load > 2 sec: +2% Load < 250ms: Load > 1sec: -4% -4.8% Watch time: +1.2%
113. 113 Prefetching: Aggressive +2.1% Load < 500ms: Load > 2 sec: +2% Load < 250ms: Load > 1sec: -4% -4.8% TTI detail page latency: +2.5% Watch time: +1.2%
114. 114 Prefetching: Mixed approach and me
115. 115 Prefetching: Mixed approach and me
116. 116 Prefetching: Mixed approach and me N from batch
117. 117 Prefetching: Mixed approach and me N from batch
118. 118 Prefetching: Mixed (3 videos) Load: Similar data compared to agressive TTI detail page latency: ~0.5% Better (but still slightly degraded)
119. 119 Which approach to choose? Approach Load result Other requests latency Lazy Agressive Mixed Average Not affected Best Affected Good Affected, but just a bit
120. 120 DownloadManager Pause/Resume/Remove downloads Requirements: Network, battery, storage Parallelisation & Threading Suits for Dash/Hls/Mp4 Not as flexible with adaptive !
121. 121 How to deal with adaptive bitrate?
122. 122 DownloadHelper Select a proper track Paired with DownloadManager You can’t define how much data you need to download ! https://github.com/google/ExoPlayer/issues/1497 Select track manually !
123. 123 Prewarming
124. 124 Prewarming Where to call prepare
125. 125 Before TextPost VideoPost TextPost
126. 126 Before TextPost VideoPost TextPost
127. 127 Before TextPost VideoPost TextPost if (visiblity >= 0.5){ val player = createExoPlayer() player.prepare("url") }
128. 128 With LazyList, first composition happen before the content is actually visible
129. 129 Now TextPost VideoPost TextPost TextPost
130. 130 Now TextPost VideoPost TextPost TextPost val player = remember { createExoPlayer().apply { player.prepare(“url") player } }
131. Pre-warming: Result Playback errors: +19% Load < 500ms: Load > 2 sec: +16% Load < 250ms: Load > 1sec: -17% -14% +13% Watch time: +11%
132. Pre-warming: Result Playback errors: +19% Load < 500ms: Load > 2 sec: +16% Load < 250ms: Load > 1sec: -17% -14% +13% Watch time: +11% DO
133. 133 FUTURE
134. PreloadMediaSource Partial downloading Load decoded segments in runtime Easy to use Prefetching may not be that efficient ! Easier control of ExoPlayer instances
135. 135 Player Pool and me
136. 136 Problem and me It may take 100+ms, to create ExoPlayer!
137. 137 and me Navigation Reuse
138. 138 Reuse and me Navigation Scrolling
139. 139 Idea and me
140. 140 Idea and me URL
141. 141 Idea and me URL ¯\_(ツ)_/¯ Empty
142. 142 Idea and me URL Create new instance :(
143. 143 Idea and me URL
144. 144 Idea and me URL Occupied * Can’t be reused for other playbacks
145. 145 Idea and me URL IDLE * Can be reused
146. 146 Idea and me URL IDLE * Can be reused IDLE * Can be reused
147. 147 Warnings * Don’t call exoPlayer.stop() for the same playbacks!
148. 148 and me Don’t call player.stop() for the same playbacks! * Video decoder start: ~41ms * Audio decoder start: ~24ms
149. 149 Warnings * Don’t call exoPlayer.stop() for the same playbacks! * Don’t forget to evict old players!
150. 150 Implementation private val playerPool = LinkedHashMap<String, RedditVideoPlayer>()
151. 151 Implementation private val playerPool = LinkedHashMap<String, RedditVideoPlayer>() fun acquire(key: String): RedditVideoPlayer { val savedPlayer = playerPool[key] if (savedPlayer != null) { return savedPlayer } }
152. 152 Implementation fun acquire(key: String): RedditVideoPlayer { val savedPlayer = playerPool[key] if (savedPlayer != null) { return savedPlayer } val idlePlayer = playerPool.firstNotNullOfOrNull { (key, value) -> if (!value.isAttached()) { key to value } else { null } } }
153. 153 Implementation fun acquire(key: String): RedditVideoPlayer { val savedPlayer = playerPool[key] if (savedPlayer != null) { return savedPlayer } val idlePlayer = playerPool.firstNotNullOfOrNull { (key, value) -> if (!value.isAttached()) { key to value } else { null } } if (idlePlayer != null) { playerPool.remove(idlePlayer.first) playerPool[key] = idlePlayer.second idlePlayer.second.stop() return idlePlayer.second } }
154. 154 Implementation fun acquire(key: String): RedditVideoPlayer { val savedPlayer = playerPool[key] if (savedPlayer != null) { return savedPlayer } val idlePlayer = playerPool.firstNotNullOfOrNull { (key, value) -> if (!value.isAttached()) { key to value } else { null } } if (idlePlayer != null) { playerPool.remove(idlePlayer.first) playerPool[key] = idlePlayer.second idlePlayer.second.stop() return idlePlayer.second } else { val player = createPlayer() playerPool += key to player return player }
155. Player Pool: Results +1% Load < 500ms: Load > 2 sec: +0.6% Load < 250ms: Load > 1sec: -1.1% -2% Watch time: +7%
156. Player Pool: Results +1% Load < 500ms: Load > 2 sec: +0.6% Load < 250ms: Load > 1sec: -1.1% -2% Watch time: +7% Number of frames > 700ms: -2%
157. Player Pool: Results +1% Load < 500ms: Load > 2 sec: +0.6% Load < 250ms: Load > 1sec: -1.1% -2% Watch time: +7% DO Number of frames > 700ms: -2%
158. Can we do better?
159. 159 Improvements: V2 and me * androidx.startup initialisation * Create player on IO + Non-blocking
160. 160 Initialiser and me CoroutineScope(dispatcher.io + SupervisorJob()) .async { playerPool.prewarm() }
161. 161 Result and me URL
162. 162 Always have cached player and me URL IDLE * Can be reused IDLE * Can be reused
163. 163 Warnings * Don’t forget to handle multithreading (locks + double check)
164. Player Pool M2: Results +2% Load < 500ms: Load > 2 sec: +0.4% Load < 250ms: Load > 1sec: Didn’t change Didn’t change Watch time: Didn’t change
165. Player Pool M2: Results +2% Load < 500ms: Load > 2 sec: +0.4% Load < 250ms: Load > 1sec: Didn’t change Didn’t change Watch time: Didn’t change Number of frames > 16ms: -1%
166. Player Pool M2: Results +2% Load < 500ms: Load > 2 sec: +0.4% Load < 250ms: Load > 1sec: Didn’t change Didn’t change Watch time: Didn’t change Number of frames > 16ms: -1% DO
167. 167 ViewPool and me
168. 168 Problem and me We still have to use TextureView or SurfaceView
169. 169 Problem and me We still have to use TextureView or SurfaceView It may take 30+ms to inflate!
170. 170 Potential solution and me AndroidView( onReset = { // Called before view is about to be reused }, onRelease = { view -> // Called when view won’t be reused }, modifier = Modifier, factory = { context -> // create View } )
171. 171 Potential solution and me AndroidView( onReset = { // Called before view is about to be reused }, onRelease = { view -> // Called when view won’t be reused }, //… )
172. 172 and me It doesn’t work as good as we want :(
173. 173 Problems with standard pool and me * Works good only if you have the same viewTypes
174. 174 Problems with standard pool and me * Works good only if you have the same viewTypes onRelease is called too often ;(
175. 175 Problems with standard pool and me * Works good only if you have the same viewTypes * Can’t control instances count
176. 176 Problems with standard pool and me * Works good only if you have the same viewTypes * Can’t control instances count * Can’t pre-load them earlier
177. 177 Implementation and me class OldFashionedViewPool(private val maxAllocations: Int)
178. 178 Implementation and me class OldFashionedViewPool(private val maxAllocations: Int) { val viewsCache = ArrayMap<String, LinkedList<View>>() }
179. 179 Aquire and me inline fun <reified T : View> acquire(creator: () -> T): T { val clazzName = T::class.qualifiedName val typed = viewsCache[clazzName] val iterator = typed?.iterator() while (iterator?.hasNext() == true) { val view = iterator.next() iterator.remove() return view as T } return creator.invoke() }
180. 180 Aquire and me inline fun <reified T : View> acquire(creator: () -> T): T { val clazzName = T::class.qualifiedName val typed = viewsCache[clazzName] val iterator = typed?.iterator() while (iterator?.hasNext() == true) { val view = iterator.next() iterator.remove() return view as T } return creator.invoke() }
181. 181 Create if empty and me inline fun <reified T : View> acquire(creator: () -> T): T { val clazzName = T::class.qualifiedName val typed = viewsCache[clazzName] val iterator = typed?.iterator() while (iterator?.hasNext() == true) { val view = iterator.next() iterator.remove() return view as T } return creator.invoke() }
182. 182 Release and me fun release(item: View) { val clazzName = item::class.qualifiedName val typed = viewsCache.getOrPut(clazzName) { LinkedList() } if (typed.size >= maxAllocations) { typed.remove() } typed.add(item) }
183. 183 Eviction and me fun release(item: View) { val clazzName = item::class.qualifiedName val typed = viewsCache.getOrPut(clazzName) { LinkedList() } if (typed.size >= maxAllocations) { typed.remove() } typed.add(item) }
184. 184 Integration and me val viewPool = LocalOldFashionedViewPoolConfiguration.current)
185. 185 Integration and me // … AndroidView( factory = { context -> viewPool.acquire { } // …
186. 186 and me val viewPool = LocalOldFashionedViewPoolConfiguration.current) AndroidView( onRelease = viewPool::release(view) // … Integration
187. 187 and me Warnings * Use only onRelease with this, with no onReset
188. 188 and me Warnings * Use only onRelease with this, with no onReset * AndrodView::onRelease called later than onDispose
189. 189 and me Warnings * Use only onRelease with this, with no onReset * AndrodView::onRelease called later than onDispose * Always implement your own controls!
190. ViewPool: Results +4.3% Load < 500ms: Load > 2 sec: +3.1% Load < 250ms: Load > 1sec: -4% -1% Watch time: +4.3%
191. ViewPool: Results +4.3% Load < 500ms: Load > 2 sec: +3.1% Load < 250ms: Load > 1sec: -4% -1% Watch time: 4.3% DO
192. 192 and me SurfaceView? TextureView?
193. 193 Where to render? TextureView SurfaceView + Scaling/Rotation/Alpha + Performance + DRM support + Simple to use - Battery consumption - Performance - DRM Support + More accurate frame timing - Complexity - API 24+ (animations/scrolling)
194. 194 Problems: * Overscroll issue
195. 195 * Overscroll issue
196. 196 and me <SurfaceView android:id="@+id/reddit_video_surface_view" android:layout_width="match_parent" android:layout_height="match_parent" android:elevation=“1dp" android:layerType=“hardware" android:soundEffectsEnabled="false" /> Cause
197. 197 and me <SurfaceView android:id="@+id/reddit_video_surface_view" android:layout_width="match_parent" android:layout_height="match_parent" android:elevation=“1dp" android:layerType=“hardware" android:soundEffectsEnabled="false" /> Cause
198. 198 Problems: * Overscroll issue * Animations bounds
199. 199 SurfaceView TextureView
200. 200 SurfaceView in documentation SurfaceView in prod
201. 201 SurfaceView results: Load time: Neutral Frame metrics: Neutral Battery metrics: Neutral
202. 202 SurfaceView results: Load time: Neutral Frame metrics: Neutral Battery metrics: Neutral Only if you optimised everything
203. 203 6 month later…
204. 204 P95 video first frame rendering and me Before: ~1800ms After: ~1216ms
205. 205 1 month later… and me
206. 206 1 month later.. and me ~1216ms —> ~1400ms
207. 207 1 month later.. and me ~1216ms —> ~1400ms
208. 208 Performance metrics
209. 209 First rendering Rendering time
210. 210 View creation Rendering time First rendering ~30ms
211. 211 View creation Rendering time First rendering Player creation ~30ms ~100ms
212. 212 View creation Rendering time First rendering Player creation Loading ~30ms ~100ms Depends :)
213. 213 View creation Rendering time First rendering Player creation Loading Decoding ~30ms ~100ms Depends :) ~64ms
214. 214 Custom implementation https://gist.github.com/NoNews/f1d5cce1f6f939df890265ebb4713296
215. 215 Conclusions
216. 216 Conclusions ~ Removing operation
217. 217 Conclusions ~ Removing operation ~ Moving operation to IO != faster!
218. 218 Conclusions ~ Removing operation ~ Moving operation to IO ~ Pre-heating / Pre-warming
219. 219 Conclusions ~ Removing operation ~ Moving operation to IO ~ Pre-heating / Pre-warming ~ Caching & Reusability
220. 220 Conclusions ~ Removing operation ~ Moving operation to IO ~ Pre-heating / Pre-warming ~ Caching & Reusability ~ Make it faster
221. 221 Conclusions ~ Removing operation ~ Moving operation to IO ~ Pre-heating / Pre-warming ~ Caching & Reusability ~ Make it faster
222. 222 Links https://www.reddit.com/r/RedditEng/comments/1af2d8d/improving_video_playback_with_exoplayer/ https://medium.com/@filipluch/how-to-improve-buffering-by-4-times-with-drip-feeding-technique-in-exoplayer-on-android- b59eb0c4d9cc https://medium.com/pinterest-engineering/improving-the-player-on-android-8b7faf9009cf https://medium.com/pinterest-engineering/improving-abr-video-performance-at-pinterest-f0ea47a6d4fc
223. 223 Slides @nonewsss Thank you