Improving Video Playback with ExoPlayer

Video has become an integral part of our life, and we are witnessing a significant rise in the integration of video content within Android apps. Sadly, there isn't much information out there about videos on Android. I've personally had a hard time finding practical ways to make videos better.

In this talk, my primary focus will be on sharing practical approaches with ExoPlayer that go beyond what is documented: We'll discuss the common problems with playbacks, solutions and will find a performant approach to use ExoPlayer.

Transcript

1. Improving video playback with ExoPlayer Alexey Bykov @nonewsss
2. 2 Alexey Bykov Senior Software Engineer 7 years Google Developer Expert @nonewsss Let’s connect!
3. 3 Reddit in numbers 75 Engineers 56% Compose 100% Kotlin 685 Modules
4. 4 10+ Different surfaces
5. 5 1. Starter ~ What is a Video? ~ Frames ~ Codecs ~ Resolutions & Bitrate ~ Adaptive Playback ~ Android Media3
6. 6 2. Practical ~ Caching & Prefetching ~ Decoders ~ Delivery ~ BandwidthMeter ~ UI & Jetpack Compose ~ LoadControl + Buffers You ExoPlayer and me 6 ~ Improve resolution
7. Goal Give you practical 
 take home tips
8. 8 Verify everything that you will hear today by A/B Tests
9. 9 What is a video?
10. Video 10
11. 11 Video Frame Frame Frame Frame
12. 12 Motion 24 FPS * 1/24 = 41.67 ms … …
13. 13 Motion 30 FPS * 1/30 = 33.33 ms 24 FPS * 1/24 = 41.67 ms … …
14. 14 Motion … … 30 FPS * 1/30 = 33.33 ms 24 FPS * 1/24 = 41.67 ms 60 FPS * 1/60 = 16.67 ms
15. 15 Storage consumption * Resolution: 1920x1080p * Duration: 10 sec * Bitrate: 10 mbps * File size: ~12 MB ~40 KB Frame * FPS: 30 1 second = 1.2 MB
16. 16 Frames processing Frame Frame Frame Frame
17. 17 Frames processing I-Frame * Has all data * Reference point
18. 18 Frames processing I-Frame * Has all data * Reference point P-Frame * Diff from prev. frame
19. 19 Frames processing B-Frame * Diffs from prev. 
 and next frame I-Frame * Has all data * Reference point P-Frame * Diff from prev. frame
20. 20 Frames processing B-Frame * Diffs from prev. 
 and next frame I-Frame * Has all data * Reference point P-Frame * Diff from prev. frame I-Frame
21. 21 Who is that guy ???
22. 22 Mr. Codec * Goal: Compress/decompress data * Short: Coder / Decoder * Types: Software/Hardware * Variety: Just a lot of them!
23. 23 VOD Codecs in production 2022-2023 * H.264/AVC * H.265/HEVC * H.266/VVC * AV1 * VP8 * MPEG/EVC * VP9 https://bitmovin.com/wp-content/uploads/2022/12/bitmovin-6th-video-developer-report-2022-2023.pdf ~85% —> 30% ~42% —> 48% ~15% —> 29% ~14% —> 42% ~14% —> 21% ~12% —> 17% ~10% —> 30% * MPEG5/LCEVC ~10% —> 22%
24. 24 Quality, Resolution & Bitrate Frame Quality Resolution 320x240p 640x360 Low VGA 1280x720 1920x1080 3840x2160 HD Full HD 4K https://castr.com/blog/bitrate-vs-resolution-whats-the-difference/ Target Bitrate ~18 mbps ~9 mbps (c) VP9 encoding recommendations ~1.8 mbps ~0.2 mbps ~0.1 mbps
25. 25 Popular delivery methods Pure binary: mp4 - Uses a single bitrate + Easy setup + Supported everywhere - Not adaptive for network + Single roundtrip
26. 26 Popular delivery methods Pure binary: mp4 Adaptive: Dash, Hls - Uses a single bitrate + Easy setup + Supported everywhere - Not adaptive for network + Multiple bitrates + Adapts to network change - Complex setup - Additional roundtrips + Single roundtrip
27. 27 Adaptive + Works on Android & Web + Minimum 2 roundtrips: - Manifest 
 - Segment Dash: Hls: + Works everywhere - Minimum 3 roundtrips 
 - Master Manifest 
 - Variant playlist - Segment - No native support for iOS
28. 28 Adaptive Playback Mr. Backend Give me batch of videos Sr. CDN .mpd <Period duration="PT0H4M40.414S">
29. 29 .mpd or Manifest <MPD xmlns="urn:mpeg:DASH:schema:MPD:2011"> <Period duration="PT0H4M40.414S"> <AdaptationSet mimeType="video/mp4"> <Representation bandwidth="250000" id="video_lowest" /> <Representation bandwidth="500000" id="video_low" /> <Representation bandwidth="1000000" id="video_medium" /> <Representation bandwidth="2000000" id="video_high" /> </AdaptationSet> <AdaptationSet mimeType="audio/mp4"> <Representation bandwidth="64000" id="audio_low" /> <Representation bandwidth="128000" id="audio_high" /> </AdaptationSet> </Period> </MPD>
30. 30 .mpd or Manifest <MPD xmlns="urn:mpeg:DASH:schema:MPD:2011"> <Period duration="PT0H4M40.414S"> <AdaptationSet mimeType="video/mp4"> <Representation bandwidth="250000" id="video_lowest" /> <Representation bandwidth="500000" id="video_low" /> <Representation bandwidth="1000000" id="video_medium" /> <Representation bandwidth="2000000" id="video_high" /> </AdaptationSet> <AdaptationSet mimeType="audio/mp4"> <Representation bandwidth="64000" id="audio_low" /> <Representation bandwidth="128000" id="audio_high" /> </AdaptationSet> </Period> </MPD>
31. 31 .mpd or Manifest <MPD xmlns="urn:mpeg:DASH:schema:MPD:2011"> <Period duration="PT0H4M40.414S"> <AdaptationSet mimeType="video/mp4"> <Representation bandwidth="250000" id="video_lowest" /> <Representation bandwidth="500000" id="video_low" /> <Representation bandwidth="1000000" id="video_medium" /> <Representation bandwidth="2000000" id="video_high" /> </AdaptationSet> <AdaptationSet mimeType="audio/mp4"> <Representation bandwidth="64000" id="audio_low" /> <Representation bandwidth="128000" id="audio_high" /> </AdaptationSet> </Period> </MPD>
32. 32 Dash Playback Mr. Backend Give me batch of videos Sr. CDN audio (selected quality) video (selected quality) .mpd <Period duration="PT0H4M40.414S">
33. 33 upload Mr. Backend Sr. CDN Encoding service VOD uploading 1080p 720p 360p M anifest
34. 34 Media 3
35. 35 Media 3 https://github.com/androidx/media 34 libraries exoplayer exoplayer-hls exoplayer-dash exoplayer-smoothstreaming exoplayer-datasource muxer ui transformer test_utils datasource_okhttp datasource_cronet decoder_av1 decoder_vp9 decoder_ffmpeg decoder_workmanager
36. 36 Part 2: Approaches You ExoPlayer and me
37. 37 You and me ¯\_(ツ)_/¯ Something went wrong
38. 38 Proper delivery
39. 39 Dash playback Sr. CDN audio (selected quality) video (selected quality) .mpd ¯\_(ツ)_/¯ Something went wrong
40. 40 Playback errors: Video view: -5% +1.7% Mp4 vs Adaptive * This slide is my own opinion and does not reflect my employer
41. 41 >45 seconds Playback errors: Video view: -5% +1.7% Use dash/hls if: Use mp4 if: <45 seconds Mp4 vs Adaptive DO
42. 42 Prefetching and me
43. 43 You and me
44. 44 nextNext next current Prefetch
45. 45 Prefetch? DashDownloader? HlsDownloader? DashUtil? DownloadManager? DownloadHelper?
46. 46 Caching
47. 47 Caching MediaSource Hls Dash Progressive Ss
48. 48 Caching MediaSource Hls Dash Progressive Ss DataSource Http Cache
49. 49 Caching MediaSource Hls Dash Progressive Ss DataSource Http Cache SimpleCache
50. 50 Caching MediaSource Hls Dash Progressive Ss DataSource Http Cache SimpleCache File system
51. 51 Caching MediaSource Hls Dash Progressive Ss DataSource Http Cache SimpleCache File system DownloadManager
52. 52 Where to cache? context.internalDir context.externalDir Availability Isolation from other apps System can clean up this ! Availability ! Isolation from other apps ! System is not cleaning this
53. 53 ExoPlayer’s SimpleCache SimpleCache( cacheDir = file )
54. 54 ExoPlayer’s SimpleCache SimpleCache( cacheDir = file, evictor = ) LatestRecentlyUsedCacheEvictor(maxCacheSizeBytes)
55. 55 ExoPlayer’s SimpleCache SimpleCache( cacheDir = cacheDir, evictor = LeastRecentlyUsedCacheEvictor(maxCacheSizeBytes), databaseProvider = StandaloneDatabaseProvider(context), )
56. 56 SimpleCache learnings SimpleCache(…) This guy hits file system in constructor !
57. 57 withContext(dispatcherProvider.io) { SimpleCache(…) } SimpleCache learnings
58. 58 CacheDataSource.Factory() .setCacheKeyFactory { } .setCache(simpleCache) .setUpstreamDataSourceFactory(DefaultHttpDataSource.Factory()) .setFlags(FLAG_IGNORE_CACHE_ON_ERROR) SimpleCache learnings
59. 59 CacheDataSource.Factory() .setCacheKeyFactory { } .setCache(simpleCache) .setUpstreamDataSourceFactory(DefaultHttpDataSource.Factory()) .setFlags(FLAG_IGNORE_CACHE_ON_ERROR) Use it if uri is 
 not stable SimpleCache learnings
60. 60 DownloadManager Pause/Resume/Remove downloads Requirements: Network, battery, storage Parallelisation & Threading Suits for Dash/Hls/Mp4 Not as flexible with adaptive !
61. 61 DownloadHelper Select a proper track Paired with DownloadManager You can’t define how much data you need to download ! https://github.com/google/ExoPlayer/issues/1497 Select track manually !
62. 62 Playback errors: The same +2% Load < 500ms: Video view: +1.7% * This slide is my own opinion and does not reflect my employer Prefetching
63. 63 Prefetching Playback errors: The same +2% Load < 500ms: Video view: +1.7% DO
64. 64 Bandwidth-based prefetching Would it be better?
65. 65 Bandwidth-based prefetching 2 mbps 5 mbps 10 mbps 15 mbps -4% -7% Load<500ms -1% -11% * This slide is my own opinion and does not reflect my employer
66. 66 Bandwidth-based prefetching 2 mbps 5 mbps 10 mbps 15 mbps -4% -7% Load<500ms -1% -11% * This slide is my own opinion and does not reflect my employer DON’T
67. 67 Future Prefetch next N videos but 5-10 seconds each? Download them in parallel? Download segments of one in parallel?
68. 68 LoadControl and me
69. 69 LoadControl * Manages data buffering * Has a lot of customisations - shouldContinueLoading? - shouldStartPlayback?
70. 70 60 2.5 Blocking buffering LoadControl You and me
71. 71 LoadControl Blocking buffering Playback 2.5 60
72. 72 Blocking buffering Playback 2.5 60 50 Non-blocking buffering LoadControl
73. 73 Blocking buffering Playback 2.5 60 52.5 Non-blocking buffering LoadControl
74. 74 2.5 60 52.5 Non-blocking buffering * minBufferMs = 50s * maxBufferMs = 50s 2 flags:
75. 75 2 flags: 2.5 60 52.5 Blocking buffering * bufferForPlaybackMs = 2.5s - First Frame - Playback stopped by user * bufferForPlaybackAfterRebuffer = 5s - Network latency problems
76. 76 * bufferForPlaybackMs Load control: Recommendation Default: 2.5s ~1s —> * bufferForPlaybackAfterRebuffer Default: 5s ~1-2s —> * minBufferMs & maxBufferMs Default: 50s ~20-30s —> * setPrioritizeTimeOverSizeThresholds Default: false true —> Keep them equal !
77. 77 Results: Load control Load<500ms: +2.7% Load>1s: -8.8%
78. 78 Results: Load control Load<500ms: +2.7% Load>1s: -8.8% DO
79. 79 Code DefaultLoadControl.Builder() .setPrioritizeTimeOverSizeThresholds(true) .setBufferDurationsMs( minBufferMs = , maxBufferMs =, bufferForPlaybackMs = , bufferForPlaybackAfterRebufferMs =, ) .build()
80. 80 BandwidthMeter and me
81. BandwidthMeter Calculates bandwidth Uses it in different playbacks (p50)
82. 82 Dash playback Sr. CDN audio (selected quality) video (selected quality) .mpd
83. 83 Dash playback Sr. CDN audio (selected quality) video (selected quality) .mpd val bitrate = bandwidthMeter.bitrateEstimate
84. 84 Dash playback audio video .mpd val bitrate = bandwidthMeter.bitrateEstimate Sr. CDN bitrate bitrate
85. 85 Dash playback audio video .mpd val bitrate = bandwidthMeter.bitrateEstimate Sr. CDN bitrate bitrate bandwidthMeter.onTransferEnd()
86. 86 Prefetching does not contribute to bandwidth
87. 87 Fix DownloadManager( context, StandaloneDatabaseProvider(context), cache, DefaultHttpDataSource.Factory() Executors.newSingleThreadExecutor(), )
88. 88 Fix DownloadManager( context, StandaloneDatabaseProvider(context), cache, DefaultHttpDataSource.Factory() .apply { setTransferListener(bandwidthMeter.transferListener) }, Executors.newSingleThreadExecutor(), )
89. 89 Video view: +0.5% Fix result Better resolution: +1.25%
90. 90 Video view: +0.5% Fix result Better resolution: +1.25% DO
91. 91 Initial bitrate
92. 92 Initial bitrate public static final ImmutableList<Long> DEFAULT_INITIAL_BITRATE_ESTIMATES_WIFI = ImmutableList.of(4_400_000L, 3_200_000L, 2_300_000L, 1_600_000L, 810_000L); public static final ImmutableList<Long> DEFAULT_INITIAL_BITRATE_ESTIMATES_4G = ImmutableList.of(2_600_000L, 1_700_000L, 1_300_000L, 1_000_000L, 700_000L); /** Default initial 5G-NSA bitrate estimates in bits per second. */ public static final ImmutableList<Long> DEFAULT_INITIAL_BITRATE_ESTIMATES_5G_NSA = ImmutableList.of(5_700_000L, 3_700_000L, 2_300_000L, 1_700_000L, 990_000L);
93. 93 Initial bitrate public static final ImmutableList<Long> DEFAULT_INITIAL_BITRATE_ESTIMATES_WIFI = ImmutableList.of(4_400_000L, 3_200_000L, 2_300_000L, 1_600_000L, 810_000L); public static final ImmutableList<Long> DEFAULT_INITIAL_BITRATE_ESTIMATES_4G = ImmutableList.of(2_600_000L, 1_700_000L, 1_300_000L, 1_000_000L, 700_000L); public static final ImmutableList<Long> DEFAULT_INITIAL_BITRATE_ESTIMATES_5G_NSA = ImmutableList.of(5_700_000L, 3_700_000L, 2_300_000L, 1_700_000L, 990_000L);
94. 94 Initial bitrate private static int[] getInitialBitrateCountryGroupAssignment(String country) { switch (country) { case "AD": case "CW": return new int[] {2, 2, 0, 0, 2, 2}; case "AE": return new int[] {1, 4, 3, 4, 4, 2}; case "AG": return new int[] {2, 4, 3, 4, 2, 2}; case "AL": return new int[] {1, 1, 1, 3, 2, 2}; case "AM": return new int[] {2, 3, 2, 3, 2, 2}; case "AO": return new int[] {4, 4, 4, 3, 2, 2};
95. 95 GB 4G: 3G: 1.3 mbps 1.4 mbps Hardcoded Type
96. 96 WiFi 4.4 mbps Hardcoded Real Type GB 30-50 mbps (p50)
97. 97 How to change that DefaultBandwidthMeter.Builder(context) .setInitialBitrateEstimate(NETWORK_TYPE_WIFI, TODO()) .build() Keep 1 instance! !
98. 98 Ideas to experiment Decrease initial bitrate? Use last known per networkType? Use last known per networkType - threshold?
99. 99 Media 1.0.1 ExperimentalBandwidthMeter Goal: Improve bitrate calculation https://developer.android.com/reference/androidx/media3/exoplayer/upstream/experimental/ExperimentalBandwidthMeter
100. 100 Improving mp4 resolution
101. 101 Mr. Backend Sr. CDN audio (selected quality) video (selected quality) .mpd 101 Idea Give me batch of videos
102. 102 Mr. Backend Sr. CDN audio (selected quality) video (selected quality) .mpd 102 Idea 1080p 720p 480p + bitrateInfo Give me batch of videos:
103. 103 Mr. Backend Sr. CDN 103 Idea val bitrate = bandwidthMeter.bitrateEstimate 1080p 720p 480p + bitrateInfo Give me batch of videos:
104. 104 Mr. Backend Sr. CDN 104 Idea val bitrate = bandwidthMeter.bitrateEstimate val url = getUrl(bitrate) 1080p 720p 480p + bitrateInfo Give me batch of videos:
105. 105 Mr. Backend Sr. CDN 105 Idea val url = getUrl(bitrate) val bitrate = bandwidthMeter.bitrateEstimate Give me batch of videos: 1080p 720p 480p + bitrateInfo (1080p) url
106. 106 Results Better resolution: +40% Load > 2sec: +15%
107. 107 Results Better resolution: +40% Load > 2sec: +15% DON’T
108. 108 Ideas to experiment Bitrate Threshold
109. 109 What else can I do to adapt bitrate?
110. 110 AdaptiveTrackSelection params minDurationForQualityIncreaseMs = 10_000 minDurationForQualityDecreaseMs = 25_000
111. 111 Decoders
112. 112 Mr. Backend Sr. CDN audio (selected quality) video (selected quality) .mpd 112 Give me batch of videos
113. 113 Mr. Backend Sr. CDN 113 audio (selected quality) video (selected quality) .mpd Give me batch of videos decoding()
114. 114 Problem Hardware decoder may not be available Playback error: 4001
115. 115 Fix DefaultRenderersFactory(context) .setEnableDecoderFallback(true)
116. 116 Fix DefaultRenderersFactory(context) .setEnableDecoderFallback(true)
117. 117 Results 4001 error daily: -180K
118. 118 UI
119. 119 Where to render? TextureView SurfaceView + Scaling/Rotation/Alpha + Performance + DRM support + Simple to use - Battery consumption - Performance - DRM Support + More accurate frame timing - Complexity - API 24+ (animations/scrolling)
120. 120 Where to render? TextureView SurfaceView + Scaling/Rotation/Alpha + Performance + DRM support + Simple to use - Battery consumption - Performance - DRM Support + More accurate frame timing - Complexity - API 24+ (animations) Suits: Complex clipping 
 Translucency 
 Arbitrary rotations Suits: Most of the other cases
121. 121 Jetpack Compase
122. 122 Jetpack Compase ¯\_(ツ)_/¯
123. 123 Jetpack Compase AndroidView( factory = { context -> YourVideoUiContainerView(context) } )
124. 124 ViewPool AndroidView( factory = { context -> YourVideoUiContainerView(context) }, onReset = { view -> }, update = { view -> }, )
125. 125 ViewPool performance Junky frames p99 -179ms 30x time reduction
126. 126 Always implement your own controls!
127. 127 Other things Analytics first: TTFF, Rebuffing, Playback errors, Watch time Single ExoPlayer instance to reuse decoders
128. 128 Links https://medium.com/@filipluch/how-to-improve-buffering-by-4-times-with-drip-feeding-technique- in-exoplayer-on-android-b59eb0c4d9cc https://medium.com/androiddevelopers/android-hdr-migrating-from-textureview-to-surfaceview- part-1-how-to-migrate-6bfd7f4b970e https://developer.android.com/guide/topics/media/exoplayer https://exoplayer.dev/battery-consumption.html
129. 129 Slides @nonewsss Thank you