Media streaming on Android @droidkaigi 2018

Transcript

1. Media streaming on Android TakuSemba CyberAgent.Inc
2. @takusemba https://github.com/TakuSemba
3. Player
4. API level 1 API level 14 Simple use cases Advanced use cases Black Box Customizable MediaPlayer ExoPlayer
5. ExoPlayer dependencies { implementation ‘com.google.android.exoplayer:exoplayer:x.x.x’ }
6. ExoPlayer dependencies { implementation ‘com.google.android.exoplayer:exoplayer-core:x.x.x’ implementation ‘com.google.android.exoplayer:exoplayer-ui:x.x.x’ implementation ‘com.google.android.exoplayer:exoplayer-hls:x.x.x’ }
7. ExoPlayer dependencies { implementation ‘com.google.android.exoplayer:exoplayer-core:x.x.x’ implementation ‘com.google.android.exoplayer:exoplayer-ui:x.x.x’ implementation ‘com.google.android.exoplayer:exoplayer-hls:x.x.x’ implementation ‘com.google.android.exoplayer:extension-okhttp:x.x.x’ }
8. Streaming
9. Streaming HTTP video
10. Streaming video
11. Streaming 1min 1sec × 60files video .mp4 .mp4 video
12. Streaming time video1 video2 video3 video4
13. Streaming time video1 video2 video3 video4
14. time Streaming video1 video2 video3 video4 video1
15. Streaming time video1 video2 video3 video4 video1 video2
16. Streaming time video1 video2 video3 video4 video1 video2 video3
17. time video1 video2 video3 video4 Streaming video1 video2 video3 video4
18. Streaming
19. 1 pixel = 24 bits (1byte for red, blue, green) Streaming 1 pixel
20. 1 pixel = 24 bits (1byte for red, blue, green) 1 frame = 49m bits (1080px * 1920px * 24bits) Streaming 1 frame
21. 1 pixel = 24 bits (1byte for red, blue, green) 1 frame = 49m bits (1080px * 1920px * 24bits) 1 second = 1492m bits (30fps * 49m bits) Streaming 1 second
22. 1 pixel = 24 bits (1byte for red, blue, green) 1 frame = 49m bits (1080px * 1920px * 24bits) 1 second = 1492m bits (30fps * 49m bits) 1492m bits Streaming 1 second
23. 1 pixel = 24 bits (1byte for red, blue, green) 1 frame = 49m bits (1080px * 1920px * 24bits) 1 second = 1492m bits (30fps * 49m bits) 1492m bits > 60m bps (4G LTE) Streaming 1 second
24. 1 pixel = 24 bits (1byte for red, blue, green) 1 frame = 49m bits (1080px * 1920px * 24bits) 1 second = 1492m bits (30fps * 49m bits) 1492m bits > 60m bps (4G LTE) codec (h264) Streaming 1 second
25. 1 pixel = 24 bits (1byte for red, blue, green) 1 frame = 49m bits (1080px * 1920px * 24bits) 1 second = 1492m bits (30fps * 49m bits) 1492m bits > 60m bps (4G LTE) 14.92m bits < 60m bps (4G LTE) codec (h264) Streaming 1 second
26. Resize This Window Resize This Window https://developer.android.com/guide/topics/media/media-formats.html
27. HLS
28. master playlist media playlists ts files .m3u8 .mp4 .mp4 .m3u8 .mp4 .mp4 .ts
29. master playlist media playlists ts files .m3u8 .mp4 .mp4 .m3u8 .mp4 .mp4 .ts
30. client master playlist .m3u8
31. HTTP client master playlist .m3u8
32. Master Playlist #EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000 http://mycompany.com/240p.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1400000 http://mycompany.com/480p.m3u8 … #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2400000 http://mycompany.com/720p.m3u8 …
33. Master Playlist #EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000 http://mycompany.com/240p.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1400000 http://mycompany.com/480p.m3u8 … #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2400000 http://mycompany.com/720p.m3u8 … #EXTM3U
34. Master Playlist #EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000 http://mycompany.com/240p.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1400000 http://mycompany.com/480p.m3u8 … #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2400000 http://mycompany.com/720p.m3u8 … #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000 http://mycompany.com/240p.m3u8
35. Master Playlist #EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000 http://mycompany.com/240p.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1400000 http://mycompany.com/480p.m3u8 … #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2400000 http://mycompany.com/720p.m3u8 … #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000
36. Master Playlist #EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000 http://mycompany.com/240p.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1400000 http://mycompany.com/480p.m3u8 … #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2400000 http://mycompany.com/720p.m3u8 … http://mycompany.com/240p.m3u8
37. master playlist media playlists ts files .m3u8 .mp4 .mp4 .m3u8 .mp4 .mp4 .ts
38. client 480p 240p 720p HTTP .m3u8 .m3u8 .m3u8
39. Media Playlist #EXT-X-VERSION:3 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:10.0, http://media.example.com/segment1.ts #EXTINF:9.0, http://media.example.com/segment2.ts #EXTINF:7.5, http://media.example.com/segment3.ts #EXTINF:8.5, http://media.example.com/segment4.ts …
40. Media Playlist #EXT-X-VERSION:3 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:10.0, http://media.example.com/segment1.ts #EXTINF:9.0, http://media.example.com/segment2.ts #EXTINF:7.5, http://media.example.com/segment3.ts #EXTINF:8.5, http://media.example.com/segment4.ts … #EXTINF:10.0, http://media.example.com/segment1.ts
41. Media Playlist #EXT-X-VERSION:3 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:10.0, http://media.example.com/segment1.ts #EXTINF:9.0, http://media.example.com/segment2.ts #EXTINF:7.5, http://media.example.com/segment3.ts #EXTINF:8.5, http://media.example.com/segment4.ts … #EXTINF:10.0,
42. Media Playlist #EXT-X-VERSION:3 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:10.0, http://media.example.com/segment1.ts #EXTINF:9.0, http://media.example.com/segment2.ts #EXTINF:7.5, http://media.example.com/segment3.ts #EXTINF:8.5, http://media.example.com/segment4.ts … http://media.example.com/segment1.ts
43. Media Playlist #EXT-X-VERSION:3 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:10.0, http://media.example.com/segment1.ts #EXTINF:9.0, http://media.example.com/segment2.ts #EXTINF:7.5, http://media.example.com/segment3.ts #EXTINF:8.5, http://media.example.com/segment4.ts … http://media.example.com/segment2.ts
44. Media Playlist #EXT-X-VERSION:3 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:10.0, http://media.example.com/segment1.ts #EXTINF:9.0, http://media.example.com/segment2.ts #EXTINF:7.5, http://media.example.com/segment3.ts #EXTINF:8.5, http://media.example.com/segment4.ts … http://media.example.com/segment3.ts
45. master playlist media playlists ts files .m3u8 .mp4 .mp4 .m3u8 .mp4 .mp4 .ts
46. client 480p 240p 720p HTTP .m3u8 .m3u8 .m3u8 .mp4 .mp4 .ts .mp4 .mp4 .ts .mp4 .mp4 .ts
47. client 480p 240p 720p HTTP .m3u8 .m3u8 .m3u8 .mp4 .mp4 .ts .mp4 .mp4 .ts .mp4 .mp4 .ts
48. HTTP client 480p 240p 720p .m3u8 .m3u8 .m3u8 .mp4 .mp4 .ts .mp4 .mp4 .ts .mp4 .mp4 .ts
49. HTTP client 480p 240p 720p .m3u8 .m3u8 .m3u8 .mp4 .mp4 .ts .mp4 .mp4 .ts .mp4 .mp4 .ts
50. MPEG-DASH
51. MPD fragmented mp4 file Initialization file .mpd .mp4 .mp4
52. MPD fragmented mp4 file Initialization file .mpd .mp4 .mp4
53. MPD fragmented mp4 file Initialization file .mpd .mp4 .mp4
54. MP4 ftyp: file type moov: header data mdat: audio or video data
55. ftyp: file type moof: header data sidx: index of moof + mdat Fragmented MP4 mdat: audio or video data
56. Fragmented MP4 fragmented mp4 segment files .mp4 .mp4 .mp4 .m4s
57. MPD fragmented mp4 file Initialization file .mpd .mp4 .mp4
58. MPD <MPD xmlns="urn:mpeg:dash:schema:mpd:2011" minBufferTime="PT1.500S" type="static" mediaPresentationDuration="PT0H3M10.024S" maxSegmentDuration="PT0H0M10.010S" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011"> <Period duration="PT0H3M10.024S"> <AdaptationSet segmentAlignment="true" maxWidth="1280" maxHeight="720" maxFrameRate="24000/1001" par="16:9" lang="und" subsegmentAlignment="true" subsegmentStartsWithSAP="1"> <Representation id="1" mimeType="video/mp4" codecs="avc1.64001F" width="1280" height="720" frameRate="24000/1001" sar="1:1" startWithSAP="1" bandwidth="400823"> <BaseURL>400k.mp4</BaseURL> <SegmentBase indexRangeExact="true" indexRange="919-1262"> <Initialization range="0-918"/> </SegmentBase> … </Representation> </AdaptationSet> <AdaptationSet segmentAlignment="true" lang="und" subsegmentAlignment="true" subsegmentStartsWithSAP="1"> <Representation id="3" mimeType="audio/mp4" codecs="mp4a.40.2" startWithSAP="1" bandwidth="197469"> <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/> <BaseURL>output_audio_dashinit.mp4</BaseURL> <SegmentBase indexRangeExact="true" indexRange="861-1132"> <Initialization range="0-860"/> </SegmentBase> </Representation> </AdaptationSet> </Period> </MPD>
59. MPD <MPD> <Period> <AdaptationSet> <Representation> <Segment> </Segment> </Representation> </AdaptationSet> </Period> </MPD> ɾlive or onDemand ? ɾresolution ? ɾbyte range or segment path
60. ExoPlayer
61. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
62. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() )
63. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player
64. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
65. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true player.prepare(mediaSource) player.playWhenReady = true
66. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true // clean up player.release()
67. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
68. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
69. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
70. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val okHttpClient = OkHttpClient.Builder() .connectTimeout(10L, TimeUnit.SECONDS) .writeTimeout(10L, TimeUnit.SECONDS) .readTimeout(30L, TimeUnit.SECONDS) .build() val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
71. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val okHttpClient = OkHttpClient.Builder() .connectTimeout(10L, TimeUnit.SECONDS) .writeTimeout(10L, TimeUnit.SECONDS) .readTimeout(30L, TimeUnit.SECONDS) .build() val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
72. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val okHttpClient = OkHttpClient.Builder() .connectTimeout(10L, TimeUnit.SECONDS) .writeTimeout(10L, TimeUnit.SECONDS) .readTimeout(30L, TimeUnit.SECONDS) .build() val dataSourceFactory = OkHttpDataSourceFactory(okHttpClient, “user-agent”, null) val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
73. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val okHttpClient = OkHttpClient(…) val dataSourceFactory = OkHttpDataSourceFactory(okHttpClient, “user-agent”, null) val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
74. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val okHttpClient = OkHttpClient(…) val dataSourceFactory = OkHttpDataSourceFactory(okHttpClient, “user-agent”, null) val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
75. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val okHttpClient = OkHttpClient(…) val dataSourceFactory = OkHttpDataSourceFactory(okHttpClient, “user-agent”, null) val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl()
76. Renderer val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() )
77. Renderer val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) DefaultRenderersFactory(context)
78. Renderer public class MyRenderer implements Renderer { // make your own renderer }
79. Renderer public class MyRenderer implements Renderer { // make your own renderer } DefaultRenderersFactory(..., MyRenderer())
80. TrackSelector val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() )
81. TrackSelector val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) DefaultTrackSelector(),
82. TrackSelector val selector = DefaultTrackSelector()
83. TrackSelector val factory = AdaptiveTrackSelection.Factory( DefaultBandwidthMeter(), DEFAULT_MAX_INITIAL_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS, DEFAULT_BANDWIDTH_FRACTION)
84. TrackSelector val factory = AdaptiveTrackSelection.Factory( DefaultBandwidthMeter(), DEFAULT_MAX_INITIAL_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS, DEFAULT_BANDWIDTH_FRACTION) val selector = DefaultTrackSelector(factory)
85. TrackSelector val selector = DefaultTrackSelector(factory) val factory = AdaptiveTrackSelection.Factory( DefaultBandwidthMeter(), HIEST_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS, DEFAULT_BANDWIDTH_FRACTION)
86. client 480p 240p 720p HTTP .m3u8 .m3u8 .m3u8
87. client 480p 240p 720p .m3u8 .m3u8 .m3u8 HTTP
88. TrackSelector val selector = DefaultTrackSelector(factory) val factory = AdaptiveTrackSelection.Factory( DefaultBandwidthMeter(), DEFAULT_MAX_INITIAL_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS, DEFAULT_BANDWIDTH_FRACTION)
89. TrackSelector val selector = DefaultTrackSelector(factory) val factory = AdaptiveTrackSelection.Factory( DefaultBandwidthMeter(), DEFAULT_MAX_INITIAL_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS, DEFAULT_BANDWIDTH_FRACTION) DefaultBandwidthMeter(),
90. TrackSelector public final class DefaultBandwidthMeter implements BandwidthMeter, TransferListener<Object> { @Override public long getBitrateEstimate() { return currentBitrate; } @Override public void onTransferStart(Object source, DataSpec dataSpec) { } @Override public void onBytesTransferred(Object source, int bytesTransferred) { } @Override public void onTransferEnd(Object source) { } }
91. TrackSelector public final class DefaultBandwidthMeter implements BandwidthMeter, TransferListener<Object> { @Override public long getBitrateEstimate() { return currentBitrate; } @Override public void onTransferStart(Object source, DataSpec dataSpec) { } @Override public void onBytesTransferred(Object source, int bytesTransferred) { } @Override public void onTransferEnd(Object source) { } } @Override public long getBitrateEstimate() { return currentBitrate; }
92. TrackSelector public final class MyBandwidthMeter implements BandwidthMeter, TransferListener<Object> { @Override public long getBitrateEstimate() { return Math.min(currentBitrate, limitBitrate); } @Override public void onTransferStart(Object source, DataSpec dataSpec) { } @Override public void onBytesTransferred(Object source, int bytesTransferred) { } @Override public void onTransferEnd(Object source) { } } @Override public long getBitrateEstimate() { return Math.min(currentBitrate, limitBitrate); } MyBandwidthMeter
93. TrackSelector val selector = DefaultTrackSelector(factory) val factory = AdaptiveTrackSelection.Factory( MyBandwidthMeter(), DEFAULT_MAX_INITIAL_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS, DEFAULT_BANDWIDTH_FRACTION)
94. TrackSelector val selector = DefaultTrackSelector(factory) val factory = AdaptiveTrackSelection.Factory( MyBandwidthMeter(), DEFAULT_MAX_INITIAL_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS, DEFAULT_BANDWIDTH_FRACTION) MyBandwidthMeter(),
95. client 480p 240p 720p .m3u8 .m3u8 .m3u8 HTTP
96. client 480p 240p 720p .m3u8 .m3u8 .m3u8 HTTP
97. TrackSelector public final class MyBandwidthMeter implements BandwidthMeter, TransferListener<Object> { @Override public long getBitrateEstimate() { return Math.max(currentBitrate, limitBitrate); } @Override public void onTransferStart(Object source, DataSpec dataSpec) { } @Override public void onBytesTransferred(Object source, int bytesTransferred) { } @Override public void onTransferEnd(Object source) { } }
98. TrackSelector public final class MyBandwidthMeter implements BandwidthMeter, TransferListener<Object> { @Override public long getBitrateEstimate() { return Math.max(currentBitrate, limitBitrate); } @Override public void onTransferStart(Object source, DataSpec dataSpec) { } @Override public void onBytesTransferred(Object source, int bytesTransferred) { } @Override public void onTransferEnd(Object source) { } } return Math.max(currentBitrate, limitBitrate); MyBandwidthMeter
99. client 480p 240p 720p HTTP .m3u8 .m3u8 .m3u8
100. client 480p 240p 720p HTTP .m3u8 .m3u8 .m3u8
101. LoadControl val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() )
102. LoadControl val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) DefaultLoadControl()
103. LoadControl val loadControl = DefaultLoadControl()
104. LoadControl val loadControl = DefaultLoadControl( DefaultAllocator(true, C.DEFAULT_BUFFER_SEGMENT_SIZE), DEFAULT_MIN_BUFFER_MS, DEFAULT_MAX_BUFFER_MS, DEFAULT_BUFFER_FOR_PLAYBACK_MS, DEFAULT_BUFFER_FOR_PLAYBACK_AFTER_REBUFFER_MS )
105. LoadControl val loadControl = DefaultLoadControl( DefaultAllocator(true, C.DEFAULT_BUFFER_SEGMENT_SIZE), DEFAULT_MIN_BUFFER_MS, DEFAULT_MAX_BUFFER_MS, DEFAULT_BUFFER_FOR_PLAYBACK_MS, DEFAULT_BUFFER_FOR_PLAYBACK_AFTER_REBUFFER_MS ) DEFAULT_MIN_BUFFER_MS, DEFAULT_MAX_BUFFER_MS, DEFAULT_BUFFER_FOR_PLAYBACK_MS, DEFAULT_BUFFER_FOR_PLAYBACK_AFTER_REBUFFER_MS
106. MediaSource val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true
107. MediaSource val mediaSource = DashMediaSource(Uri, …) player.prepare(mediaSource) player.playWhenReady = true
108. MediaSource val mediaSource = ExtractorMediaSource(Uri, …) player.prepare(mediaSource) player.playWhenReady = true
109. MediaSource val firstSource = HlsMediaSource(firstUri, …) val secondSource = HlsMediaSource(secondUri, …) val mediaSource = ConcatenatingMediaSource(firstSource, secondSource) // play firstSource, then secondSource player.prepare(mediaSource) player.playWhenReady = true
110. Metadata
111. Metadata video
112. Metadata video createdAt? title? thumbnail? genre?
113. Metadata ɾID3 tag
114. Metadata ɾID3 tag .ts id = 1000 place = droidkaigi
115. Metadata
116. Metadata class Logger : MetadataOutput { override fun onMetadata(metadata: Metadata) { (1..metadata.length()) .map { metadata[it] } .filterIsInstance<TextInformationFrame>() .forEach { print("metadata: ${it.value}") } } } // metadata: id=1000,place=droidkaigi
117. Metadata ɾID3 tag ɾemsg box
118. Metadata ɾID3 tag id = 1000 place = droidkaigi ɾemsg box .mp4
119. Metadata class Logger : MetadataOutput { override fun onMetadata(metadata: Metadata) { (1..metadata.length()) .map { metadata[it] } .filterIsInstance<EventMessage>() .forEach { print("metadata: ${it.value}") } } } // metadata: id=1000,place=droidkaigi
120. Publish
121. UDP TCP 1:1 1:n low latency high latency WebRTC RTMP
122. RTMP RTMP client server
123. RTMP RTMP client server users RTMP
124. RTMP RTMP client server users HLS MPEG-DASH
125. RTMP ɾdeveloped by Adobe. not “open” protocol ɾRed5 started rtmp in 2005 ɾWowza started rtmp in 2007 ɾrtmpdump released in 2008 ɾrtmp specification released 2012
126. RTMP “we will stop updating and distributing the Flash Player at the end of 2020” -Adobe
127. RTMP RTMP
128. RTMP RTMP
129. RTMP RTMP
130. RTMP RTMP
131. RTMP PRO CON ɾsmall header (basically 1, 4, 8, 12 bytes) ɾrealtime ɾdifficult to cache ɾno multiple resolutions
132. RTMP RTMP
133. RTMP RTMP RTMP
134. RTMP RTMP
135. RTMP dependencies { implementation ‘com.google.android.exoplayer:exoplayer-core:x.x.x’ implementation ‘com.google.android.exoplayer:exoplayer-ui:x.x.x’ implementation ‘com.google.android.exoplayer:extension-rtmp:x.x.x’ }
136. val mediaSource = ExtractorMediaSource( Uri.parse("rtmp://xxx.xx.xx.xxx:1935/path/stream_name") RtmpDataSourceFactory(), DefaultExtractorsFactory(), null, null) player.prepare(mediaSource) player.playWhenReady = true
137. val mediaSource = ExtractorMediaSource( Uri.parse("rtmp://xxx.xx.xx.xxx:1935/path/stream_name") RtmpDataSourceFactory(), DefaultExtractorsFactory(), null, null) player.prepare(mediaSource) player.playWhenReady = true Uri.parse("rtmp://xxx.xx.xx.xxx:1935/path/stream_name")
138. val mediaSource = ExtractorMediaSource( Uri.parse("rtmp://xxx.xx.xx.xxx:1935/path/stream_name") RtmpDataSourceFactory(), DefaultExtractorsFactory(), null, null) player.prepare(mediaSource) player.playWhenReady = true RtmpDataSourceFactory(),
139. public final class RtmpDataSource implements DataSource { private RtmpClient rtmpClient; ... @Override public int read(byte[] buffer, int offset, int readLength) throws IOException { int bytesRead = rtmpClient.read(buffer, offset, readLength); ... return bytesRead; } ... }
140. public final class RtmpDataSource implements DataSource { private RtmpClient rtmpClient; ... @Override public int read(byte[] buffer, int offset, int readLength) throws IOException { int bytesRead = rtmpClient.read(buffer, offset, readLength); ... return bytesRead; } ... } private RtmpClient rtmpClient; rtmpClient.read(buffer, offset, readLength);
141. public class RtmpClient { … public int read(byte[] data, int offset, int size) throws IOException { return nativeRead(data, offset, size, rtmpPointer); } private native int nativeRead(byte[] data, int offset, int size, long rtmpPointer) throws IOException … }
142. public class RtmpClient { … public int read(byte[] data, int offset, int size) throws IOException { return nativeRead(data, offset, size, rtmpPointer); } private native int nativeRead(byte[] data, int offset, int size, long rtmpPointer) throws IOException … } private native int nativeRead(byte[] data, int offset, int size, long rtmpPointer) public class RtmpClient
143. RTMP RTMP
144. RTMP ɾcapture data ɾencode data ɾpublish data
145. RTMP ɾcapture data ɾencode data ɾpublish data
146. RTMP ɾcapture data ɾencode data ɾpublish data val audioRecord = AudioRecord(...) Audio Data
147. RTMP ɾcapture data ɾencode data ɾpublish data val audioRecord = AudioRecord(...) // camera2 API val manager : CameraManager manager.openCamera(...) // camera API val camera = Camera.(info) Audio Data Video Data
148. RTMP ɾcapture data ɾencode data ɾpublish data
149. RTMP // MediaCodec.java val encoder = MediaCodec(…) encoder.start() val inputBuffers = encoder.getInputBuffer(index) // or val surface = encoder.createInputSurface() // get encoded data val outputBuffers = encoder.getOutputBuffer(index) ɾcapture data ɾencode data ɾpublish data
150. RTMP // MediaCodec.java val encoder = MediaCodec(…) encoder.start() val inputBuffers = encoder.getInputBuffer(index) // or val surface = encoder.createInputSurface() // get encoded data val outputBuffers = encoder.getOutputBuffer(index) ɾcapture data ɾencode data ɾpublish data // MediaCodec.java val encoder = MediaCodec(…) encoder.start()
151. RTMP ɾcapture data ɾencode data ɾpublish data // MediaCodec.java val encoder = MediaCodec(…) encoder.start() val inputBuffers = encoder.getInputBuffer(index) // or val surface = encoder.createInputSurface() // get encoded data val outputBuffers = encoder.getOutputBuffer(index) val inputBuffers = encoder.getInputBuffer(index) // or val surface = encoder.createInputSurface()
152. RTMP ɾcapture data ɾencode data ɾpublish data // MediaCodec.java val encoder = MediaCodec(…) encoder.start() val inputBuffers = encoder.getInputBuffer(index) // or val surface = encoder.createInputSurface() // get encoded data val outputBuffers = encoder.getOutputBuffer(index) // get encoded data val outputBuffers = encoder.getOutputBuffer(index)
153. RTMP ɾcapture data ɾencode data ɾpublish data
154. RTMP // rtmpdump public native int writeVideo( byte[] data, int offset, int length, int timestamp ) ɾcapture data ɾencode data ɾpublish data
155. RTMP // rtmpdump public native int writeVideo( byte[] data, int offset, int length, int timestamp ) // Socket.java val socket = Socket(“192.xx.xx.xxx", 1935) val outputStream = DataOutputStream(…) outputStream.write(byteArray) ɾcapture data ɾencode data ɾpublish data
156. https://github.com/TakuSemba/RtmpPublisher
157. Demo
158. Demo RTMP Server HLS RTMP Publisher Player Player
159. Demo rtmp://160.16.54.236:1935/live/takusemba http://160.16.54.236/hls/takusemba.m3u8 RTMP HLS
160. Media streaming on Android https://github.com/takusemba https://twitter.com/takusemba