Media streaming on Android @droidkaigi 2018

Transcript

1. Media streaming on Android TakuSemba CyberAgent.Inc

2. @takusemba https://github.com/TakuSemba

3. Player

4. API level 1 API level 14 Simple use cases Advanced

   use cases Black Box Customizable MediaPlayer ExoPlayer

5. ExoPlayer dependencies { implementation ‘com.google.android.exoplayer:exoplayer:x.x.x’ }

6. ExoPlayer dependencies { implementation ‘com.google.android.exoplayer:exoplayer-core:x.x.x’ implementation ‘com.google.android.exoplayer:exoplayer-ui:x.x.x’ implementation ‘com.google.android.exoplayer:exoplayer-hls:x.x.x’ }

7. ExoPlayer dependencies { implementation ‘com.google.android.exoplayer:exoplayer-core:x.x.x’ implementation ‘com.google.android.exoplayer:exoplayer-ui:x.x.x’ implementation ‘com.google.android.exoplayer:exoplayer-hls:x.x.x’ implementation

   ‘com.google.android.exoplayer:extension-okhttp:x.x.x’ }

8. Streaming

9. Streaming HTTP video

10. Streaming video

11. Streaming 1min 1sec × 60files video .mp4 .mp4 video

12. Streaming time video1 video2 video3 video4

13. Streaming time video1 video2 video3 video4

14. time Streaming video1 video2 video3 video4 video1

15. Streaming time video1 video2 video3 video4 video1 video2

16. Streaming time video1 video2 video3 video4 video1 video2 video3

17. time video1 video2 video3 video4 Streaming video1 video2 video3 video4

18. Streaming

19. 1 pixel = 24 bits (1byte for red, blue, green)

    Streaming 1 pixel

20. 1 pixel = 24 bits (1byte for red, blue, green)

    1 frame = 49m bits (1080px * 1920px * 24bits) Streaming 1 frame

21. 1 pixel = 24 bits (1byte for red, blue, green)

    1 frame = 49m bits (1080px * 1920px * 24bits) 1 second = 1492m bits (30fps * 49m bits) Streaming 1 second

22. 1 pixel = 24 bits (1byte for red, blue, green)

    1 frame = 49m bits (1080px * 1920px * 24bits) 1 second = 1492m bits (30fps * 49m bits) 1492m bits Streaming 1 second

23. 1 pixel = 24 bits (1byte for red, blue, green)

    1 frame = 49m bits (1080px * 1920px * 24bits) 1 second = 1492m bits (30fps * 49m bits) 1492m bits > 60m bps (4G LTE) Streaming 1 second

24. 1 pixel = 24 bits (1byte for red, blue, green)

    1 frame = 49m bits (1080px * 1920px * 24bits) 1 second = 1492m bits (30fps * 49m bits) 1492m bits > 60m bps (4G LTE) codec (h264) Streaming 1 second

25. 1 pixel = 24 bits (1byte for red, blue, green)

    1 frame = 49m bits (1080px * 1920px * 24bits) 1 second = 1492m bits (30fps * 49m bits) 1492m bits > 60m bps (4G LTE) 14.92m bits < 60m bps (4G LTE) codec (h264) Streaming 1 second

26. Resize This Window Resize This Window https://developer.android.com/guide/topics/media/media-formats.html

27. HLS

28. master playlist media playlists ts files .m3u8 .mp4 .mp4 .m3u8

    .mp4 .mp4 .ts

29. master playlist media playlists ts files .m3u8 .mp4 .mp4 .m3u8

    .mp4 .mp4 .ts

30. client master playlist .m3u8

31. HTTP client master playlist .m3u8

32. Master Playlist #EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000 http://mycompany.com/240p.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1400000 http://mycompany.com/480p.m3u8 …

    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2400000 http://mycompany.com/720p.m3u8 …

33. Master Playlist #EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000 http://mycompany.com/240p.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1400000 http://mycompany.com/480p.m3u8 …

    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2400000 http://mycompany.com/720p.m3u8 … #EXTM3U

34. Master Playlist #EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000 http://mycompany.com/240p.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1400000 http://mycompany.com/480p.m3u8 …

    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2400000 http://mycompany.com/720p.m3u8 … #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000 http://mycompany.com/240p.m3u8

35. Master Playlist #EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000 http://mycompany.com/240p.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1400000 http://mycompany.com/480p.m3u8 …

    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2400000 http://mycompany.com/720p.m3u8 … #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000

36. Master Playlist #EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000 http://mycompany.com/240p.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1400000 http://mycompany.com/480p.m3u8 …

    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2400000 http://mycompany.com/720p.m3u8 … http://mycompany.com/240p.m3u8

37. master playlist media playlists ts files .m3u8 .mp4 .mp4 .m3u8

    .mp4 .mp4 .ts

38. client 480p 240p 720p HTTP .m3u8 .m3u8 .m3u8

39. Media Playlist #EXT-X-VERSION:3 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:10.0, http://media.example.com/segment1.ts #EXTINF:9.0, http://media.example.com/segment2.ts

    #EXTINF:7.5, http://media.example.com/segment3.ts #EXTINF:8.5, http://media.example.com/segment4.ts …

40. Media Playlist #EXT-X-VERSION:3 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:10.0, http://media.example.com/segment1.ts #EXTINF:9.0, http://media.example.com/segment2.ts

    #EXTINF:7.5, http://media.example.com/segment3.ts #EXTINF:8.5, http://media.example.com/segment4.ts … #EXTINF:10.0, http://media.example.com/segment1.ts

41. Media Playlist #EXT-X-VERSION:3 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:10.0, http://media.example.com/segment1.ts #EXTINF:9.0, http://media.example.com/segment2.ts

    #EXTINF:7.5, http://media.example.com/segment3.ts #EXTINF:8.5, http://media.example.com/segment4.ts … #EXTINF:10.0,

42. Media Playlist #EXT-X-VERSION:3 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:10.0, http://media.example.com/segment1.ts #EXTINF:9.0, http://media.example.com/segment2.ts

    #EXTINF:7.5, http://media.example.com/segment3.ts #EXTINF:8.5, http://media.example.com/segment4.ts … http://media.example.com/segment1.ts

43. Media Playlist #EXT-X-VERSION:3 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:10.0, http://media.example.com/segment1.ts #EXTINF:9.0, http://media.example.com/segment2.ts

    #EXTINF:7.5, http://media.example.com/segment3.ts #EXTINF:8.5, http://media.example.com/segment4.ts … http://media.example.com/segment2.ts

44. Media Playlist #EXT-X-VERSION:3 #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:1 #EXTINF:10.0, http://media.example.com/segment1.ts #EXTINF:9.0, http://media.example.com/segment2.ts

    #EXTINF:7.5, http://media.example.com/segment3.ts #EXTINF:8.5, http://media.example.com/segment4.ts … http://media.example.com/segment3.ts

45. master playlist media playlists ts files .m3u8 .mp4 .mp4 .m3u8

    .mp4 .mp4 .ts

46. client 480p 240p 720p HTTP .m3u8 .m3u8 .m3u8 .mp4 .mp4

    .ts .mp4 .mp4 .ts .mp4 .mp4 .ts

47. client 480p 240p 720p HTTP .m3u8 .m3u8 .m3u8 .mp4 .mp4

    .ts .mp4 .mp4 .ts .mp4 .mp4 .ts

48. HTTP client 480p 240p 720p .m3u8 .m3u8 .m3u8 .mp4 .mp4

    .ts .mp4 .mp4 .ts .mp4 .mp4 .ts

49. HTTP client 480p 240p 720p .m3u8 .m3u8 .m3u8 .mp4 .mp4

    .ts .mp4 .mp4 .ts .mp4 .mp4 .ts

50. MPEG-DASH

51. MPD fragmented mp4 file Initialization file .mpd .mp4 .mp4

52. MPD fragmented mp4 file Initialization file .mpd .mp4 .mp4

53. MPD fragmented mp4 file Initialization file .mpd .mp4 .mp4

54. MP4 ftyp: file type moov: header data mdat: audio or

    video data

55. ftyp: file type moof: header data sidx: index of moof

    + mdat Fragmented MP4 mdat: audio or video data

56. Fragmented MP4 fragmented mp4 segment files .mp4 .mp4 .mp4 .m4s

57. MPD fragmented mp4 file Initialization file .mpd .mp4 .mp4

58. MPD <MPD xmlns="urn:mpeg:dash:schema:mpd:2011" minBufferTime="PT1.500S" type="static" mediaPresentationDuration="PT0H3M10.024S" maxSegmentDuration="PT0H0M10.010S" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011"> <Period duration="PT0H3M10.024S">

    <AdaptationSet segmentAlignment="true" maxWidth="1280" maxHeight="720" maxFrameRate="24000/1001" par="16:9" lang="und" subsegmentAlignment="true" subsegmentStartsWithSAP="1"> <Representation id="1" mimeType="video/mp4" codecs="avc1.64001F" width="1280" height="720" frameRate="24000/1001" sar="1:1" startWithSAP="1" bandwidth="400823"> <BaseURL>400k.mp4</BaseURL> <SegmentBase indexRangeExact="true" indexRange="919-1262"> <Initialization range="0-918"/> </SegmentBase> … </Representation> </AdaptationSet> <AdaptationSet segmentAlignment="true" lang="und" subsegmentAlignment="true" subsegmentStartsWithSAP="1"> <Representation id="3" mimeType="audio/mp4" codecs="mp4a.40.2" startWithSAP="1" bandwidth="197469"> <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/> <BaseURL>output_audio_dashinit.mp4</BaseURL> <SegmentBase indexRangeExact="true" indexRange="861-1132"> <Initialization range="0-860"/> </SegmentBase> </Representation> </AdaptationSet> </Period> </MPD>

59. MPD <MPD> <Period> <AdaptationSet> <Representation> <Segment> </Segment> </Representation> </AdaptationSet> </Period>

    </MPD> ɾlive or onDemand ? ɾresolution ? ɾbyte range or segment path

60. ExoPlayer

61. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true

62. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() )

63. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player

64. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)

65. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true player.prepare(mediaSource) player.playWhenReady = true

66. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true // clean up player.release()

67. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true

68. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player =

    player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true

69. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player =

    player val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true

70. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player =

    player val okHttpClient = OkHttpClient.Builder() .connectTimeout(10L, TimeUnit.SECONDS) .writeTimeout(10L, TimeUnit.SECONDS) .readTimeout(30L, TimeUnit.SECONDS) .build() val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent") val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true

71. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player =

    player val okHttpClient = OkHttpClient.Builder() .connectTimeout(10L, TimeUnit.SECONDS) .writeTimeout(10L, TimeUnit.SECONDS) .readTimeout(30L, TimeUnit.SECONDS) .build() val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true

72. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player =

    player val okHttpClient = OkHttpClient.Builder() .connectTimeout(10L, TimeUnit.SECONDS) .writeTimeout(10L, TimeUnit.SECONDS) .readTimeout(30L, TimeUnit.SECONDS) .build() val dataSourceFactory = OkHttpDataSourceFactory(okHttpClient, “user-agent”, null) val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true

73. val player = ExoPlayerFactory.newSimpleInstance(…) val playerView = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player =

    player val okHttpClient = OkHttpClient(…) val dataSourceFactory = OkHttpDataSourceFactory(okHttpClient, “user-agent”, null) val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true

74. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val okHttpClient = OkHttpClient(…) val dataSourceFactory = OkHttpDataSourceFactory(okHttpClient, “user-agent”, null) val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true

75. val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) val playerView

    = findViewById<SimpleExoPlayerView>(R.id.player_view) playerView.player = player val okHttpClient = OkHttpClient(…) val dataSourceFactory = OkHttpDataSourceFactory(okHttpClient, “user-agent”, null) val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady = true DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl()

76. Renderer val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() )

77. Renderer val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) DefaultRenderersFactory(context)

78. Renderer public class MyRenderer implements Renderer { // make your

    own renderer }

79. Renderer public class MyRenderer implements Renderer { // make your

    own renderer } DefaultRenderersFactory(..., MyRenderer())

80. TrackSelector val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() )

81. TrackSelector val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) DefaultTrackSelector(),

82. TrackSelector val selector = DefaultTrackSelector()

83. TrackSelector val factory = AdaptiveTrackSelection.Factory( DefaultBandwidthMeter(), DEFAULT_MAX_INITIAL_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS,

    DEFAULT_BANDWIDTH_FRACTION)

84. TrackSelector val factory = AdaptiveTrackSelection.Factory( DefaultBandwidthMeter(), DEFAULT_MAX_INITIAL_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS,

    DEFAULT_BANDWIDTH_FRACTION) val selector = DefaultTrackSelector(factory)

85. TrackSelector val selector = DefaultTrackSelector(factory) val factory = AdaptiveTrackSelection.Factory( DefaultBandwidthMeter(),

    HIEST_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS, DEFAULT_BANDWIDTH_FRACTION)

86. client 480p 240p 720p HTTP .m3u8 .m3u8 .m3u8

87. client 480p 240p 720p .m3u8 .m3u8 .m3u8 HTTP

88. TrackSelector val selector = DefaultTrackSelector(factory) val factory = AdaptiveTrackSelection.Factory( DefaultBandwidthMeter(),

    DEFAULT_MAX_INITIAL_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS, DEFAULT_BANDWIDTH_FRACTION)

89. TrackSelector val selector = DefaultTrackSelector(factory) val factory = AdaptiveTrackSelection.Factory( DefaultBandwidthMeter(),

    DEFAULT_MAX_INITIAL_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS, DEFAULT_BANDWIDTH_FRACTION) DefaultBandwidthMeter(),

90. TrackSelector public final class DefaultBandwidthMeter implements BandwidthMeter, TransferListener<Object> { @Override

    public long getBitrateEstimate() { return currentBitrate; } @Override public void onTransferStart(Object source, DataSpec dataSpec) { } @Override public void onBytesTransferred(Object source, int bytesTransferred) { } @Override public void onTransferEnd(Object source) { } }

91. TrackSelector public final class DefaultBandwidthMeter implements BandwidthMeter, TransferListener<Object> { @Override

    public long getBitrateEstimate() { return currentBitrate; } @Override public void onTransferStart(Object source, DataSpec dataSpec) { } @Override public void onBytesTransferred(Object source, int bytesTransferred) { } @Override public void onTransferEnd(Object source) { } } @Override public long getBitrateEstimate() { return currentBitrate; }

92. TrackSelector public final class MyBandwidthMeter implements BandwidthMeter, TransferListener<Object> { @Override

    public long getBitrateEstimate() { return Math.min(currentBitrate, limitBitrate); } @Override public void onTransferStart(Object source, DataSpec dataSpec) { } @Override public void onBytesTransferred(Object source, int bytesTransferred) { } @Override public void onTransferEnd(Object source) { } } @Override public long getBitrateEstimate() { return Math.min(currentBitrate, limitBitrate); } MyBandwidthMeter

93. TrackSelector val selector = DefaultTrackSelector(factory) val factory = AdaptiveTrackSelection.Factory( MyBandwidthMeter(),

    DEFAULT_MAX_INITIAL_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS, DEFAULT_BANDWIDTH_FRACTION)

94. TrackSelector val selector = DefaultTrackSelector(factory) val factory = AdaptiveTrackSelection.Factory( MyBandwidthMeter(),

    DEFAULT_MAX_INITIAL_BITRATE, DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS, DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS, DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS, DEFAULT_BANDWIDTH_FRACTION) MyBandwidthMeter(),

95. client 480p 240p 720p .m3u8 .m3u8 .m3u8 HTTP

96. client 480p 240p 720p .m3u8 .m3u8 .m3u8 HTTP

97. TrackSelector public final class MyBandwidthMeter implements BandwidthMeter, TransferListener<Object> { @Override

    public long getBitrateEstimate() { return Math.max(currentBitrate, limitBitrate); } @Override public void onTransferStart(Object source, DataSpec dataSpec) { } @Override public void onBytesTransferred(Object source, int bytesTransferred) { } @Override public void onTransferEnd(Object source) { } }

98. TrackSelector public final class MyBandwidthMeter implements BandwidthMeter, TransferListener<Object> { @Override

    public long getBitrateEstimate() { return Math.max(currentBitrate, limitBitrate); } @Override public void onTransferStart(Object source, DataSpec dataSpec) { } @Override public void onBytesTransferred(Object source, int bytesTransferred) { } @Override public void onTransferEnd(Object source) { } } return Math.max(currentBitrate, limitBitrate); MyBandwidthMeter

99. client 480p 240p 720p HTTP .m3u8 .m3u8 .m3u8

100. client 480p 240p 720p HTTP .m3u8 .m3u8 .m3u8

101. LoadControl val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() )

102. LoadControl val player = ExoPlayerFactory.newSimpleInstance( DefaultRenderersFactory(context), DefaultTrackSelector(), DefaultLoadControl() ) DefaultLoadControl()

103. LoadControl val loadControl = DefaultLoadControl()

104. LoadControl val loadControl = DefaultLoadControl( DefaultAllocator(true, C.DEFAULT_BUFFER_SEGMENT_SIZE), DEFAULT_MIN_BUFFER_MS, DEFAULT_MAX_BUFFER_MS, DEFAULT_BUFFER_FOR_PLAYBACK_MS,

     DEFAULT_BUFFER_FOR_PLAYBACK_AFTER_REBUFFER_MS )

105. LoadControl val loadControl = DefaultLoadControl( DefaultAllocator(true, C.DEFAULT_BUFFER_SEGMENT_SIZE), DEFAULT_MIN_BUFFER_MS, DEFAULT_MAX_BUFFER_MS, DEFAULT_BUFFER_FOR_PLAYBACK_MS,

     DEFAULT_BUFFER_FOR_PLAYBACK_AFTER_REBUFFER_MS ) DEFAULT_MIN_BUFFER_MS, DEFAULT_MAX_BUFFER_MS, DEFAULT_BUFFER_FOR_PLAYBACK_MS, DEFAULT_BUFFER_FOR_PLAYBACK_AFTER_REBUFFER_MS

106. MediaSource val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null) player.prepare(mediaSource) player.playWhenReady

     = true

107. MediaSource val mediaSource = DashMediaSource(Uri, …) player.prepare(mediaSource) player.playWhenReady = true

108. MediaSource val mediaSource = ExtractorMediaSource(Uri, …) player.prepare(mediaSource) player.playWhenReady = true

109. MediaSource val firstSource = HlsMediaSource(firstUri, …) val secondSource = HlsMediaSource(secondUri,

     …) val mediaSource = ConcatenatingMediaSource(firstSource, secondSource) // play firstSource, then secondSource player.prepare(mediaSource) player.playWhenReady = true

110. Metadata

111. Metadata video

112. Metadata video createdAt? title? thumbnail? genre?

113. Metadata ɾID3 tag

114. Metadata ɾID3 tag .ts id = 1000 place = droidkaigi

115. Metadata

116. Metadata class Logger : MetadataOutput { override fun onMetadata(metadata: Metadata)

     { (1..metadata.length()) .map { metadata[it] } .filterIsInstance<TextInformationFrame>() .forEach { print("metadata: ${it.value}") } } } // metadata: id=1000,place=droidkaigi

117. Metadata ɾID3 tag ɾemsg box

118. Metadata ɾID3 tag id = 1000 place = droidkaigi ɾemsg

     box .mp4

119. Metadata class Logger : MetadataOutput { override fun onMetadata(metadata: Metadata)

     { (1..metadata.length()) .map { metadata[it] } .filterIsInstance<EventMessage>() .forEach { print("metadata: ${it.value}") } } } // metadata: id=1000,place=droidkaigi

120. Publish

121. UDP TCP 1:1 1:n low latency high latency WebRTC RTMP

122. RTMP RTMP client server

123. RTMP RTMP client server users RTMP

124. RTMP RTMP client server users HLS MPEG-DASH

125. RTMP ɾdeveloped by Adobe. not “open” protocol ɾRed5 started rtmp

     in 2005 ɾWowza started rtmp in 2007 ɾrtmpdump released in 2008 ɾrtmp specification released 2012

126. RTMP “we will stop updating and distributing the Flash Player

     at the end of 2020” -Adobe

127. RTMP RTMP

128. RTMP RTMP

129. RTMP RTMP

130. RTMP RTMP

131. RTMP PRO CON ɾsmall header (basically 1, 4, 8, 12

     bytes) ɾrealtime ɾdifficult to cache ɾno multiple resolutions

132. RTMP RTMP

133. RTMP RTMP RTMP

134. RTMP RTMP

135. RTMP dependencies { implementation ‘com.google.android.exoplayer:exoplayer-core:x.x.x’ implementation ‘com.google.android.exoplayer:exoplayer-ui:x.x.x’ implementation ‘com.google.android.exoplayer:extension-rtmp:x.x.x’ }

136. val mediaSource = ExtractorMediaSource( Uri.parse("rtmp://xxx.xx.xx.xxx:1935/path/stream_name") RtmpDataSourceFactory(), DefaultExtractorsFactory(), null, null) player.prepare(mediaSource)

     player.playWhenReady = true

137. val mediaSource = ExtractorMediaSource( Uri.parse("rtmp://xxx.xx.xx.xxx:1935/path/stream_name") RtmpDataSourceFactory(), DefaultExtractorsFactory(), null, null) player.prepare(mediaSource)

     player.playWhenReady = true Uri.parse("rtmp://xxx.xx.xx.xxx:1935/path/stream_name")

138. val mediaSource = ExtractorMediaSource( Uri.parse("rtmp://xxx.xx.xx.xxx:1935/path/stream_name") RtmpDataSourceFactory(), DefaultExtractorsFactory(), null, null) player.prepare(mediaSource)

     player.playWhenReady = true RtmpDataSourceFactory(),

139. public final class RtmpDataSource implements DataSource { private RtmpClient rtmpClient;

     ... @Override public int read(byte[] buffer, int offset, int readLength) throws IOException { int bytesRead = rtmpClient.read(buffer, offset, readLength); ... return bytesRead; } ... }

140. public final class RtmpDataSource implements DataSource { private RtmpClient rtmpClient;

     ... @Override public int read(byte[] buffer, int offset, int readLength) throws IOException { int bytesRead = rtmpClient.read(buffer, offset, readLength); ... return bytesRead; } ... } private RtmpClient rtmpClient; rtmpClient.read(buffer, offset, readLength);

141. public class RtmpClient { … public int read(byte[] data, int

     offset, int size) throws IOException { return nativeRead(data, offset, size, rtmpPointer); } private native int nativeRead(byte[] data, int offset, int size, long rtmpPointer) throws IOException … }

142. public class RtmpClient { … public int read(byte[] data, int

     offset, int size) throws IOException { return nativeRead(data, offset, size, rtmpPointer); } private native int nativeRead(byte[] data, int offset, int size, long rtmpPointer) throws IOException … } private native int nativeRead(byte[] data, int offset, int size, long rtmpPointer) public class RtmpClient

143. RTMP RTMP

144. RTMP ɾcapture data ɾencode data ɾpublish data

145. RTMP ɾcapture data ɾencode data ɾpublish data

146. RTMP ɾcapture data ɾencode data ɾpublish data val audioRecord =

     AudioRecord(...) Audio Data

147. RTMP ɾcapture data ɾencode data ɾpublish data val audioRecord =

     AudioRecord(...) // camera2 API val manager : CameraManager manager.openCamera(...) // camera API val camera = Camera.(info) Audio Data Video Data

148. RTMP ɾcapture data ɾencode data ɾpublish data

149. RTMP // MediaCodec.java val encoder = MediaCodec(…) encoder.start() val inputBuffers

     = encoder.getInputBuffer(index) // or val surface = encoder.createInputSurface() // get encoded data val outputBuffers = encoder.getOutputBuffer(index) ɾcapture data ɾencode data ɾpublish data

150. RTMP // MediaCodec.java val encoder = MediaCodec(…) encoder.start() val inputBuffers

     = encoder.getInputBuffer(index) // or val surface = encoder.createInputSurface() // get encoded data val outputBuffers = encoder.getOutputBuffer(index) ɾcapture data ɾencode data ɾpublish data // MediaCodec.java val encoder = MediaCodec(…) encoder.start()

151. RTMP ɾcapture data ɾencode data ɾpublish data // MediaCodec.java val

     encoder = MediaCodec(…) encoder.start() val inputBuffers = encoder.getInputBuffer(index) // or val surface = encoder.createInputSurface() // get encoded data val outputBuffers = encoder.getOutputBuffer(index) val inputBuffers = encoder.getInputBuffer(index) // or val surface = encoder.createInputSurface()

152. RTMP ɾcapture data ɾencode data ɾpublish data // MediaCodec.java val

     encoder = MediaCodec(…) encoder.start() val inputBuffers = encoder.getInputBuffer(index) // or val surface = encoder.createInputSurface() // get encoded data val outputBuffers = encoder.getOutputBuffer(index) // get encoded data val outputBuffers = encoder.getOutputBuffer(index)

153. RTMP ɾcapture data ɾencode data ɾpublish data

154. RTMP // rtmpdump public native int writeVideo( byte[] data, int

     offset, int length, int timestamp ) ɾcapture data ɾencode data ɾpublish data

155. RTMP // rtmpdump public native int writeVideo( byte[] data, int

     offset, int length, int timestamp ) // Socket.java val socket = Socket(“192.xx.xx.xxx", 1935) val outputStream = DataOutputStream(…) outputStream.write(byteArray) ɾcapture data ɾencode data ɾpublish data

156. https://github.com/TakuSemba/RtmpPublisher

157. Demo

158. Demo RTMP Server HLS RTMP Publisher Player Player

159. Demo rtmp://160.16.54.236:1935/live/takusemba http://160.16.54.236/hls/takusemba.m3u8 RTMP HLS

160. Media streaming on Android https://github.com/takusemba https://twitter.com/takusemba