Slide 1

Slide 1 text

Improving video playback with ExoPlayer Alexey Bykov @nonewsss

Slide 2

Slide 2 text

2 Alexey Bykov Senior Software Engineer 7 years Google Developer Expert @nonewsss Let’s connect!

Slide 3

Slide 3 text

3 Reddit in numbers 75 Engineers 56% Compose 100% Kotlin 685 Modules

Slide 4

Slide 4 text

4 10+ Different surfaces

Slide 5

Slide 5 text

5 1. Starter ~ What is a Video? ~ Frames ~ Codecs ~ Resolutions & Bitrate ~ Adaptive Playback ~ Android Media3

Slide 6

Slide 6 text

6 2. Practical ~ Caching & Prefetching ~ Decoders ~ Delivery ~ BandwidthMeter ~ UI & Jetpack Compose ~ LoadControl + Buffers You ExoPlayer and me 6 ~ Improve resolution

Slide 7

Slide 7 text

Goal Give you practical 
 take home tips

Slide 8

Slide 8 text

8 Verify everything that you will hear today by A/B Tests

Slide 9

Slide 9 text

9 What is a video?

Slide 10

Slide 10 text

Video 10

Slide 11

Slide 11 text

11 Video Frame Frame Frame Frame

Slide 12

Slide 12 text

12 Motion 24 FPS * 1/24 = 41.67 ms … …

Slide 13

Slide 13 text

13 Motion 30 FPS * 1/30 = 33.33 ms 24 FPS * 1/24 = 41.67 ms … …

Slide 14

Slide 14 text

14 Motion … … 30 FPS * 1/30 = 33.33 ms 24 FPS * 1/24 = 41.67 ms 60 FPS * 1/60 = 16.67 ms

Slide 15

Slide 15 text

15 Storage consumption * Resolution: 1920x1080p * Duration: 10 sec * Bitrate: 10 mbps * File size: ~12 MB ~40 KB Frame * FPS: 30 1 second = 1.2 MB

Slide 16

Slide 16 text

16 Frames processing Frame Frame Frame Frame

Slide 17

Slide 17 text

17 Frames processing I-Frame * Has all data * Reference point

Slide 18

Slide 18 text

18 Frames processing I-Frame * Has all data * Reference point P-Frame * Diff from prev. frame

Slide 19

Slide 19 text

19 Frames processing B-Frame * Diffs from prev. 
 and next frame I-Frame * Has all data * Reference point P-Frame * Diff from prev. frame

Slide 20

Slide 20 text

20 Frames processing B-Frame * Diffs from prev. 
 and next frame I-Frame * Has all data * Reference point P-Frame * Diff from prev. frame I-Frame

Slide 21

Slide 21 text

21 Who is that guy ???

Slide 22

Slide 22 text

22 Mr. Codec * Goal: Compress/decompress data * Short: Coder / Decoder * Types: Software/Hardware * Variety: Just a lot of them!

Slide 23

Slide 23 text

23 VOD Codecs in production 2022-2023 * H.264/AVC * H.265/HEVC * H.266/VVC * AV1 * VP8 * MPEG/EVC * VP9 https://bitmovin.com/wp-content/uploads/2022/12/bitmovin-6th-video-developer-report-2022-2023.pdf ~85% —> 30% ~42% —> 48% ~15% —> 29% ~14% —> 42% ~14% —> 21% ~12% —> 17% ~10% —> 30% * MPEG5/LCEVC ~10% —> 22%

Slide 24

Slide 24 text

24 Quality, Resolution & Bitrate Frame Quality Resolution 320x240p 640x360 Low VGA 1280x720 1920x1080 3840x2160 HD Full HD 4K https://castr.com/blog/bitrate-vs-resolution-whats-the-difference/ Target Bitrate ~18 mbps ~9 mbps (c) VP9 encoding recommendations ~1.8 mbps ~0.2 mbps ~0.1 mbps

Slide 25

Slide 25 text

25 Popular delivery methods Pure binary: mp4 - Uses a single bitrate + Easy setup + Supported everywhere - Not adaptive for network + Single roundtrip

Slide 26

Slide 26 text

26 Popular delivery methods Pure binary: mp4 Adaptive: Dash, Hls - Uses a single bitrate + Easy setup + Supported everywhere - Not adaptive for network + Multiple bitrates + Adapts to network change - Complex setup - Additional roundtrips + Single roundtrip

Slide 27

Slide 27 text

27 Adaptive + Works on Android & Web + Minimum 2 roundtrips: - Manifest 
 - Segment Dash: Hls: + Works everywhere - Minimum 3 roundtrips 
 - Master Manifest 
 - Variant playlist - Segment - No native support for iOS

Slide 28

Slide 28 text

28 Adaptive Playback Mr. Backend Give me batch of videos Sr. CDN .mpd

Slide 29

Slide 29 text

29 .mpd or Manifest

Slide 30

Slide 30 text

30 .mpd or Manifest

Slide 31

Slide 31 text

31 .mpd or Manifest

Slide 32

Slide 32 text

32 Dash Playback Mr. Backend Give me batch of videos Sr. CDN audio (selected quality) video (selected quality) .mpd

Slide 33

Slide 33 text

33 upload Mr. Backend Sr. CDN Encoding service VOD uploading 1080p 720p 360p M anifest

Slide 34

Slide 34 text

34 Media 3

Slide 35

Slide 35 text

35 Media 3 https://github.com/androidx/media 34 libraries exoplayer exoplayer-hls exoplayer-dash exoplayer-smoothstreaming exoplayer-datasource muxer ui transformer test_utils datasource_okhttp datasource_cronet decoder_av1 decoder_vp9 decoder_ffmpeg decoder_workmanager

Slide 36

Slide 36 text

36 Part 2: Approaches You ExoPlayer and me

Slide 37

Slide 37 text

37 You and me ¯\_(ツ)_/¯ Something went wrong

Slide 38

Slide 38 text

38 Proper delivery

Slide 39

Slide 39 text

39 Dash playback Sr. CDN audio (selected quality) video (selected quality) .mpd ¯\_(ツ)_/¯ Something went wrong

Slide 40

Slide 40 text

40 Playback errors: Video view: -5% +1.7% Mp4 vs Adaptive * This slide is my own opinion and does not reflect my employer

Slide 41

Slide 41 text

41 >45 seconds Playback errors: Video view: -5% +1.7% Use dash/hls if: Use mp4 if: <45 seconds Mp4 vs Adaptive DO

Slide 42

Slide 42 text

42 Prefetching and me

Slide 43

Slide 43 text

43 You and me

Slide 44

Slide 44 text

44 nextNext next current Prefetch

Slide 45

Slide 45 text

45 Prefetch? DashDownloader? HlsDownloader? DashUtil? DownloadManager? DownloadHelper?

Slide 46

Slide 46 text

46 Caching

Slide 47

Slide 47 text

47 Caching MediaSource Hls Dash Progressive Ss

Slide 48

Slide 48 text

48 Caching MediaSource Hls Dash Progressive Ss DataSource Http Cache

Slide 49

Slide 49 text

49 Caching MediaSource Hls Dash Progressive Ss DataSource Http Cache SimpleCache

Slide 50

Slide 50 text

50 Caching MediaSource Hls Dash Progressive Ss DataSource Http Cache SimpleCache File system

Slide 51

Slide 51 text

51 Caching MediaSource Hls Dash Progressive Ss DataSource Http Cache SimpleCache File system DownloadManager

Slide 52

Slide 52 text

52 Where to cache? context.internalDir context.externalDir Availability Isolation from other apps System can clean up this ! Availability ! Isolation from other apps ! System is not cleaning this

Slide 53

Slide 53 text

53 ExoPlayer’s SimpleCache SimpleCache( cacheDir = file )

Slide 54

Slide 54 text

54 ExoPlayer’s SimpleCache SimpleCache( cacheDir = file, evictor = ) LatestRecentlyUsedCacheEvictor(maxCacheSizeBytes)

Slide 55

Slide 55 text

55 ExoPlayer’s SimpleCache SimpleCache( cacheDir = cacheDir, evictor = LeastRecentlyUsedCacheEvictor(maxCacheSizeBytes), databaseProvider = StandaloneDatabaseProvider(context), )

Slide 56

Slide 56 text

56 SimpleCache learnings SimpleCache(…) This guy hits file system in constructor !

Slide 57

Slide 57 text

57 withContext(dispatcherProvider.io) { SimpleCache(…) } SimpleCache learnings

Slide 58

Slide 58 text

58 CacheDataSource.Factory() .setCacheKeyFactory { } .setCache(simpleCache) .setUpstreamDataSourceFactory(DefaultHttpDataSource.Factory()) .setFlags(FLAG_IGNORE_CACHE_ON_ERROR) SimpleCache learnings

Slide 59

Slide 59 text

59 CacheDataSource.Factory() .setCacheKeyFactory { } .setCache(simpleCache) .setUpstreamDataSourceFactory(DefaultHttpDataSource.Factory()) .setFlags(FLAG_IGNORE_CACHE_ON_ERROR) Use it if uri is 
 not stable SimpleCache learnings

Slide 60

Slide 60 text

60 DownloadManager Pause/Resume/Remove downloads Requirements: Network, battery, storage Parallelisation & Threading Suits for Dash/Hls/Mp4 Not as flexible with adaptive !

Slide 61

Slide 61 text

61 DownloadHelper Select a proper track Paired with DownloadManager You can’t define how much data you need to download ! https://github.com/google/ExoPlayer/issues/1497 Select track manually !

Slide 62

Slide 62 text

62 Playback errors: The same +2% Load < 500ms: Video view: +1.7% * This slide is my own opinion and does not reflect my employer Prefetching

Slide 63

Slide 63 text

63 Prefetching Playback errors: The same +2% Load < 500ms: Video view: +1.7% DO

Slide 64

Slide 64 text

64 Bandwidth-based prefetching Would it be better?

Slide 65

Slide 65 text

65 Bandwidth-based prefetching 2 mbps 5 mbps 10 mbps 15 mbps -4% -7% Load<500ms -1% -11% * This slide is my own opinion and does not reflect my employer

Slide 66

Slide 66 text

66 Bandwidth-based prefetching 2 mbps 5 mbps 10 mbps 15 mbps -4% -7% Load<500ms -1% -11% * This slide is my own opinion and does not reflect my employer DON’T

Slide 67

Slide 67 text

67 Future Prefetch next N videos but 5-10 seconds each? Download them in parallel? Download segments of one in parallel?

Slide 68

Slide 68 text

68 LoadControl and me

Slide 69

Slide 69 text

69 LoadControl * Manages data buffering * Has a lot of customisations - shouldContinueLoading? - shouldStartPlayback?

Slide 70

Slide 70 text

70 60 2.5 Blocking buffering LoadControl You and me

Slide 71

Slide 71 text

71 LoadControl Blocking buffering Playback 2.5 60

Slide 72

Slide 72 text

72 Blocking buffering Playback 2.5 60 50 Non-blocking buffering LoadControl

Slide 73

Slide 73 text

73 Blocking buffering Playback 2.5 60 52.5 Non-blocking buffering LoadControl

Slide 74

Slide 74 text

74 2.5 60 52.5 Non-blocking buffering * minBufferMs = 50s * maxBufferMs = 50s 2 flags:

Slide 75

Slide 75 text

75 2 flags: 2.5 60 52.5 Blocking buffering * bufferForPlaybackMs = 2.5s - First Frame - Playback stopped by user * bufferForPlaybackAfterRebuffer = 5s - Network latency problems

Slide 76

Slide 76 text

76 * bufferForPlaybackMs Load control: Recommendation Default: 2.5s ~1s —> * bufferForPlaybackAfterRebuffer Default: 5s ~1-2s —> * minBufferMs & maxBufferMs Default: 50s ~20-30s —> * setPrioritizeTimeOverSizeThresholds Default: false true —> Keep them equal !

Slide 77

Slide 77 text

77 Results: Load control Load<500ms: +2.7% Load>1s: -8.8%

Slide 78

Slide 78 text

78 Results: Load control Load<500ms: +2.7% Load>1s: -8.8% DO

Slide 79

Slide 79 text

79 Code DefaultLoadControl.Builder() .setPrioritizeTimeOverSizeThresholds(true) .setBufferDurationsMs( minBufferMs = , maxBufferMs =, bufferForPlaybackMs = , bufferForPlaybackAfterRebufferMs =, ) .build()

Slide 80

Slide 80 text

80 BandwidthMeter and me

Slide 81

Slide 81 text

BandwidthMeter Calculates bandwidth Uses it in different playbacks (p50)

Slide 82

Slide 82 text

82 Dash playback Sr. CDN audio (selected quality) video (selected quality) .mpd

Slide 83

Slide 83 text

83 Dash playback Sr. CDN audio (selected quality) video (selected quality) .mpd val bitrate = bandwidthMeter.bitrateEstimate

Slide 84

Slide 84 text

84 Dash playback audio video .mpd val bitrate = bandwidthMeter.bitrateEstimate Sr. CDN bitrate bitrate

Slide 85

Slide 85 text

85 Dash playback audio video .mpd val bitrate = bandwidthMeter.bitrateEstimate Sr. CDN bitrate bitrate bandwidthMeter.onTransferEnd()

Slide 86

Slide 86 text

86 Prefetching does not contribute to bandwidth

Slide 87

Slide 87 text

87 Fix DownloadManager( context, StandaloneDatabaseProvider(context), cache, DefaultHttpDataSource.Factory() Executors.newSingleThreadExecutor(), )

Slide 88

Slide 88 text

88 Fix DownloadManager( context, StandaloneDatabaseProvider(context), cache, DefaultHttpDataSource.Factory() .apply { setTransferListener(bandwidthMeter.transferListener) }, Executors.newSingleThreadExecutor(), )

Slide 89

Slide 89 text

89 Video view: +0.5% Fix result Better resolution: +1.25%

Slide 90

Slide 90 text

90 Video view: +0.5% Fix result Better resolution: +1.25% DO

Slide 91

Slide 91 text

91 Initial bitrate

Slide 92

Slide 92 text

92 Initial bitrate public static final ImmutableList DEFAULT_INITIAL_BITRATE_ESTIMATES_WIFI = ImmutableList.of(4_400_000L, 3_200_000L, 2_300_000L, 1_600_000L, 810_000L); public static final ImmutableList DEFAULT_INITIAL_BITRATE_ESTIMATES_4G = ImmutableList.of(2_600_000L, 1_700_000L, 1_300_000L, 1_000_000L, 700_000L); /** Default initial 5G-NSA bitrate estimates in bits per second. */ public static final ImmutableList DEFAULT_INITIAL_BITRATE_ESTIMATES_5G_NSA = ImmutableList.of(5_700_000L, 3_700_000L, 2_300_000L, 1_700_000L, 990_000L);

Slide 93

Slide 93 text

93 Initial bitrate public static final ImmutableList DEFAULT_INITIAL_BITRATE_ESTIMATES_WIFI = ImmutableList.of(4_400_000L, 3_200_000L, 2_300_000L, 1_600_000L, 810_000L); public static final ImmutableList DEFAULT_INITIAL_BITRATE_ESTIMATES_4G = ImmutableList.of(2_600_000L, 1_700_000L, 1_300_000L, 1_000_000L, 700_000L); public static final ImmutableList DEFAULT_INITIAL_BITRATE_ESTIMATES_5G_NSA = ImmutableList.of(5_700_000L, 3_700_000L, 2_300_000L, 1_700_000L, 990_000L);

Slide 94

Slide 94 text

94 Initial bitrate private static int[] getInitialBitrateCountryGroupAssignment(String country) { switch (country) { case "AD": case "CW": return new int[] {2, 2, 0, 0, 2, 2}; case "AE": return new int[] {1, 4, 3, 4, 4, 2}; case "AG": return new int[] {2, 4, 3, 4, 2, 2}; case "AL": return new int[] {1, 1, 1, 3, 2, 2}; case "AM": return new int[] {2, 3, 2, 3, 2, 2}; case "AO": return new int[] {4, 4, 4, 3, 2, 2};

Slide 95

Slide 95 text

95 GB 4G: 3G: 1.3 mbps 1.4 mbps Hardcoded Type

Slide 96

Slide 96 text

96 WiFi 4.4 mbps Hardcoded Real Type GB 30-50 mbps (p50)

Slide 97

Slide 97 text

97 How to change that DefaultBandwidthMeter.Builder(context) .setInitialBitrateEstimate(NETWORK_TYPE_WIFI, TODO()) .build() Keep 1 instance! !

Slide 98

Slide 98 text

98 Ideas to experiment Decrease initial bitrate? Use last known per networkType? Use last known per networkType - threshold?

Slide 99

Slide 99 text

99 Media 1.0.1 ExperimentalBandwidthMeter Goal: Improve bitrate calculation https://developer.android.com/reference/androidx/media3/exoplayer/upstream/experimental/ExperimentalBandwidthMeter

Slide 100

Slide 100 text

100 Improving mp4 resolution

Slide 101

Slide 101 text

101 Mr. Backend Sr. CDN audio (selected quality) video (selected quality) .mpd 101 Idea Give me batch of videos

Slide 102

Slide 102 text

102 Mr. Backend Sr. CDN audio (selected quality) video (selected quality) .mpd 102 Idea 1080p 720p 480p + bitrateInfo Give me batch of videos:

Slide 103

Slide 103 text

103 Mr. Backend Sr. CDN 103 Idea val bitrate = bandwidthMeter.bitrateEstimate 1080p 720p 480p + bitrateInfo Give me batch of videos:

Slide 104

Slide 104 text

104 Mr. Backend Sr. CDN 104 Idea val bitrate = bandwidthMeter.bitrateEstimate val url = getUrl(bitrate) 1080p 720p 480p + bitrateInfo Give me batch of videos:

Slide 105

Slide 105 text

105 Mr. Backend Sr. CDN 105 Idea val url = getUrl(bitrate) val bitrate = bandwidthMeter.bitrateEstimate Give me batch of videos: 1080p 720p 480p + bitrateInfo (1080p) url

Slide 106

Slide 106 text

106 Results Better resolution: +40% Load > 2sec: +15%

Slide 107

Slide 107 text

107 Results Better resolution: +40% Load > 2sec: +15% DON’T

Slide 108

Slide 108 text

108 Ideas to experiment Bitrate Threshold

Slide 109

Slide 109 text

109 What else can I do to adapt bitrate?

Slide 110

Slide 110 text

110 AdaptiveTrackSelection params minDurationForQualityIncreaseMs = 10_000 minDurationForQualityDecreaseMs = 25_000

Slide 111

Slide 111 text

111 Decoders

Slide 112

Slide 112 text

112 Mr. Backend Sr. CDN audio (selected quality) video (selected quality) .mpd 112 Give me batch of videos

Slide 113

Slide 113 text

113 Mr. Backend Sr. CDN 113 audio (selected quality) video (selected quality) .mpd Give me batch of videos decoding()

Slide 114

Slide 114 text

114 Problem Hardware decoder may not be available Playback error: 4001

Slide 115

Slide 115 text

115 Fix DefaultRenderersFactory(context) .setEnableDecoderFallback(true)

Slide 116

Slide 116 text

116 Fix DefaultRenderersFactory(context) .setEnableDecoderFallback(true)

Slide 117

Slide 117 text

117 Results 4001 error daily: -180K

Slide 118

Slide 118 text

118 UI

Slide 119

Slide 119 text

119 Where to render? TextureView SurfaceView + Scaling/Rotation/Alpha + Performance + DRM support + Simple to use - Battery consumption - Performance - DRM Support + More accurate frame timing - Complexity - API 24+ (animations/scrolling)

Slide 120

Slide 120 text

120 Where to render? TextureView SurfaceView + Scaling/Rotation/Alpha + Performance + DRM support + Simple to use - Battery consumption - Performance - DRM Support + More accurate frame timing - Complexity - API 24+ (animations) Suits: Complex clipping 
 Translucency 
 Arbitrary rotations Suits: Most of the other cases

Slide 121

Slide 121 text

121 Jetpack Compase

Slide 122

Slide 122 text

122 Jetpack Compase ¯\_(ツ)_/¯

Slide 123

Slide 123 text

123 Jetpack Compase AndroidView( factory = { context -> YourVideoUiContainerView(context) } )

Slide 124

Slide 124 text

124 ViewPool AndroidView( factory = { context -> YourVideoUiContainerView(context) }, onReset = { view -> }, update = { view -> }, )

Slide 125

Slide 125 text

125 ViewPool performance Junky frames p99 -179ms 30x time reduction

Slide 126

Slide 126 text

126 Always implement your own controls!

Slide 127

Slide 127 text

127 Other things Analytics first: TTFF, Rebuffing, Playback errors, Watch time Single ExoPlayer instance to reuse decoders

Slide 128

Slide 128 text

128 Links https://medium.com/@filipluch/how-to-improve-buffering-by-4-times-with-drip-feeding-technique- in-exoplayer-on-android-b59eb0c4d9cc https://medium.com/androiddevelopers/android-hdr-migrating-from-textureview-to-surfaceview- part-1-how-to-migrate-6bfd7f4b970e https://developer.android.com/guide/topics/media/exoplayer https://exoplayer.dev/battery-consumption.html

Slide 129

Slide 129 text

129 Slides @nonewsss Thank you