Media streaming on Android @droidkaigi 2018

Transcript

1. Media streaming
   on Android
   TakuSemba
   CyberAgent.Inc
   

   View Slide

2. @takusemba
   https://github.com/TakuSemba
   

   View Slide

3. Player
   

   View Slide

4. API level 1 API level 14
   Simple use cases Advanced use cases
   Black Box Customizable
   MediaPlayer ExoPlayer
   

   View Slide

5. ExoPlayer
   dependencies {
   implementation ‘com.google.android.exoplayer:exoplayer:x.x.x’
   }
   

   View Slide

6. ExoPlayer
   dependencies {
   implementation ‘com.google.android.exoplayer:exoplayer-core:x.x.x’
   implementation ‘com.google.android.exoplayer:exoplayer-ui:x.x.x’
   implementation ‘com.google.android.exoplayer:exoplayer-hls:x.x.x’
   }
   

   View Slide

7. ExoPlayer
   dependencies {
   implementation ‘com.google.android.exoplayer:exoplayer-core:x.x.x’
   implementation ‘com.google.android.exoplayer:exoplayer-ui:x.x.x’
   implementation ‘com.google.android.exoplayer:exoplayer-hls:x.x.x’
   implementation ‘com.google.android.exoplayer:extension-okhttp:x.x.x’
   }
   

   View Slide

8. Streaming
   

   View Slide

9. Streaming
   HTTP video
   

   View Slide

10. Streaming
    video
    

    View Slide

11. Streaming
    1min 1sec × 60files
    video .mp4
    .mp4
    video
    

    View Slide

12. Streaming
    time
    video1 video2 video3 video4
    

    View Slide

13. Streaming
    time
    video1 video2 video3 video4
    

    View Slide

14. time
    Streaming
    video1 video2 video3 video4
    video1
    

    View Slide

15. Streaming
    time
    video1 video2 video3 video4
    video1 video2
    

    View Slide

16. Streaming
    time
    video1 video2 video3 video4
    video1 video2 video3
    

    View Slide

17. time
    video1 video2 video3 video4
    Streaming
    video1 video2 video3 video4
    

    View Slide

18. Streaming
    

    View Slide

19. 1 pixel = 24 bits (1byte for red, blue, green)
    Streaming
    1 pixel
    

    View Slide

20. 1 pixel = 24 bits (1byte for red, blue, green)
    1 frame = 49m bits (1080px * 1920px * 24bits)
    Streaming
    1 frame
    

    View Slide

21. 1 pixel = 24 bits (1byte for red, blue, green)
    1 frame = 49m bits (1080px * 1920px * 24bits)
    1 second = 1492m bits (30fps * 49m bits)
    Streaming
    1 second
    

    View Slide

22. 1 pixel = 24 bits (1byte for red, blue, green)
    1 frame = 49m bits (1080px * 1920px * 24bits)
    1 second = 1492m bits (30fps * 49m bits)
    1492m bits
    Streaming
    1 second
    

    View Slide

23. 1 pixel = 24 bits (1byte for red, blue, green)
    1 frame = 49m bits (1080px * 1920px * 24bits)
    1 second = 1492m bits (30fps * 49m bits)
    1492m bits > 60m bps (4G LTE)
    Streaming
    1 second
    

    View Slide

24. 1 pixel = 24 bits (1byte for red, blue, green)
    1 frame = 49m bits (1080px * 1920px * 24bits)
    1 second = 1492m bits (30fps * 49m bits)
    1492m bits > 60m bps (4G LTE)
    codec (h264)
    Streaming
    1 second
    

    View Slide

25. 1 pixel = 24 bits (1byte for red, blue, green)
    1 frame = 49m bits (1080px * 1920px * 24bits)
    1 second = 1492m bits (30fps * 49m bits)
    1492m bits > 60m bps (4G LTE)
    14.92m bits < 60m bps (4G LTE)
    codec (h264)
    Streaming
    1 second
    

    View Slide

26. Resize This Window
    Resize This Window
    https://developer.android.com/guide/topics/media/media-formats.html
    

    View Slide

27. HLS
    

    View Slide

28. master playlist media playlists ts files
    .m3u8 .mp4
    .mp4
    .m3u8 .mp4
    .mp4
    .ts
    

    View Slide

29. master playlist media playlists ts files
    .m3u8 .mp4
    .mp4
    .m3u8 .mp4
    .mp4
    .ts
    

    View Slide

30. client master playlist
    .m3u8
    

    View Slide

31. HTTP
    client master playlist
    .m3u8
    

    View Slide

32. Master Playlist
    #EXTM3U
    
    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000
    
    http://mycompany.com/240p.m3u8
    
    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1400000
    
    http://mycompany.com/480p.m3u8
    
    …
    
    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2400000
    
    http://mycompany.com/720p.m3u8
    
    …
    

    View Slide

33. Master Playlist
    #EXTM3U
    
    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000
    
    http://mycompany.com/240p.m3u8
    
    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1400000
    
    http://mycompany.com/480p.m3u8
    
    …
    
    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2400000
    
    http://mycompany.com/720p.m3u8
    
    …
    #EXTM3U
    

    View Slide

34. Master Playlist
    #EXTM3U
    
    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000
    
    http://mycompany.com/240p.m3u8
    
    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1400000
    
    http://mycompany.com/480p.m3u8
    
    …
    
    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2400000
    
    http://mycompany.com/720p.m3u8
    
    …
    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000
    
    http://mycompany.com/240p.m3u8
    

    View Slide

35. Master Playlist
    #EXTM3U
    
    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000
    
    http://mycompany.com/240p.m3u8
    
    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1400000
    
    http://mycompany.com/480p.m3u8
    
    …
    
    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2400000
    
    http://mycompany.com/720p.m3u8
    
    …
    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000
    

    View Slide

36. Master Playlist
    #EXTM3U
    
    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=300000
    
    http://mycompany.com/240p.m3u8
    
    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=1400000
    
    http://mycompany.com/480p.m3u8
    
    …
    
    #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=2400000
    
    http://mycompany.com/720p.m3u8
    
    …
    http://mycompany.com/240p.m3u8
    

    View Slide

37. master playlist media playlists ts files
    .m3u8 .mp4
    .mp4
    .m3u8 .mp4
    .mp4
    .ts
    

    View Slide

38. client
    480p
    240p
    720p
    HTTP
    .m3u8
    .m3u8
    .m3u8
    

    View Slide

39. Media Playlist
    #EXT-X-VERSION:3
    
    #EXTM3U
    
    #EXT-X-TARGETDURATION:10
    
    #EXT-X-MEDIA-SEQUENCE:1
    
    #EXTINF:10.0,
    
    http://media.example.com/segment1.ts
    
    #EXTINF:9.0,
    
    http://media.example.com/segment2.ts
    
    #EXTINF:7.5,
    
    http://media.example.com/segment3.ts
    
    #EXTINF:8.5,
    
    http://media.example.com/segment4.ts
    
    …
    

    View Slide

40. Media Playlist
    #EXT-X-VERSION:3
    
    #EXTM3U
    
    #EXT-X-TARGETDURATION:10
    
    #EXT-X-MEDIA-SEQUENCE:1
    
    #EXTINF:10.0,
    
    http://media.example.com/segment1.ts
    
    #EXTINF:9.0,
    
    http://media.example.com/segment2.ts
    
    #EXTINF:7.5,
    
    http://media.example.com/segment3.ts
    
    #EXTINF:8.5,
    
    http://media.example.com/segment4.ts
    
    …
    #EXTINF:10.0,
    
    http://media.example.com/segment1.ts
    

    View Slide

41. Media Playlist
    #EXT-X-VERSION:3
    
    #EXTM3U
    
    #EXT-X-TARGETDURATION:10
    
    #EXT-X-MEDIA-SEQUENCE:1
    
    #EXTINF:10.0,
    
    http://media.example.com/segment1.ts
    
    #EXTINF:9.0,
    
    http://media.example.com/segment2.ts
    
    #EXTINF:7.5,
    
    http://media.example.com/segment3.ts
    
    #EXTINF:8.5,
    
    http://media.example.com/segment4.ts
    
    …
    #EXTINF:10.0,
    

    View Slide

42. Media Playlist
    #EXT-X-VERSION:3
    
    #EXTM3U
    
    #EXT-X-TARGETDURATION:10
    
    #EXT-X-MEDIA-SEQUENCE:1
    
    #EXTINF:10.0,
    
    http://media.example.com/segment1.ts
    
    #EXTINF:9.0,
    
    http://media.example.com/segment2.ts
    
    #EXTINF:7.5,
    
    http://media.example.com/segment3.ts
    
    #EXTINF:8.5,
    
    http://media.example.com/segment4.ts
    
    …
    http://media.example.com/segment1.ts
    

    View Slide

43. Media Playlist
    #EXT-X-VERSION:3
    
    #EXTM3U
    
    #EXT-X-TARGETDURATION:10
    
    #EXT-X-MEDIA-SEQUENCE:1
    
    #EXTINF:10.0,
    
    http://media.example.com/segment1.ts
    
    #EXTINF:9.0,
    
    http://media.example.com/segment2.ts
    
    #EXTINF:7.5,
    
    http://media.example.com/segment3.ts
    
    #EXTINF:8.5,
    
    http://media.example.com/segment4.ts
    
    …
    http://media.example.com/segment2.ts
    

    View Slide

44. Media Playlist
    #EXT-X-VERSION:3
    
    #EXTM3U
    
    #EXT-X-TARGETDURATION:10
    
    #EXT-X-MEDIA-SEQUENCE:1
    
    #EXTINF:10.0,
    
    http://media.example.com/segment1.ts
    
    #EXTINF:9.0,
    
    http://media.example.com/segment2.ts
    
    #EXTINF:7.5,
    
    http://media.example.com/segment3.ts
    
    #EXTINF:8.5,
    
    http://media.example.com/segment4.ts
    
    …
    http://media.example.com/segment3.ts
    

    View Slide

45. master playlist media playlists ts files
    .m3u8 .mp4
    .mp4
    .m3u8 .mp4
    .mp4
    .ts
    

    View Slide

46. client
    480p
    240p
    720p
    HTTP
    .m3u8
    .m3u8
    .m3u8
    .mp4
    .mp4
    .ts
    .mp4
    .mp4
    .ts
    .mp4
    .mp4
    .ts
    

    View Slide

47. client
    480p
    240p
    720p
    HTTP
    .m3u8
    .m3u8
    .m3u8
    .mp4
    .mp4
    .ts
    .mp4
    .mp4
    .ts
    .mp4
    .mp4
    .ts
    

    View Slide

48. HTTP
    client
    480p
    240p
    720p
    .m3u8
    .m3u8
    .m3u8
    .mp4
    .mp4
    .ts
    .mp4
    .mp4
    .ts
    .mp4
    .mp4
    .ts
    

    View Slide

49. HTTP
    client
    480p
    240p
    720p
    .m3u8
    .m3u8
    .m3u8
    .mp4
    .mp4
    .ts
    .mp4
    .mp4
    .ts
    .mp4
    .mp4
    .ts
    

    View Slide

50. MPEG-DASH
    

    View Slide

51. MPD fragmented mp4 file Initialization file
    .mpd .mp4 .mp4
    

    View Slide

52. MPD fragmented mp4 file Initialization file
    .mpd .mp4 .mp4
    

    View Slide

53. MPD fragmented mp4 file Initialization file
    .mpd .mp4 .mp4
    

    View Slide

54. MP4
    ftyp: file type
    moov: header data
    mdat: audio or video data
    

    View Slide

55. ftyp: file type
    moof: header data
    sidx: index of moof + mdat
    Fragmented MP4
    mdat: audio or video data
    

    View Slide

56. Fragmented MP4
    fragmented mp4 segment files
    .mp4 .mp4
    .mp4
    .m4s
    

    View Slide

57. MPD fragmented mp4 file Initialization file
    .mpd .mp4 .mp4
    

    View Slide

58. MPD
    maxSegmentDuration="PT0H0M10.010S" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011">
    
    subsegmentAlignment="true" subsegmentStartsWithSAP="1">
    startWithSAP="1" bandwidth="400823">
    400k.mp4
    
    
    
    …
    
    
    
    
    
    output_audio_dashinit.mp4
    
    
    
    
    
    
    
    

    View Slide

59. MPD
    
    
    
    
    
    
    
    
    
    
    ɾlive or onDemand ?
    ɾresolution ?
    ɾbyte range or segment path
    

    View Slide

60. ExoPlayer
    

    View Slide

61. val player = ExoPlayerFactory.newSimpleInstance(
    DefaultRenderersFactory(context),
    DefaultTrackSelector(),
    DefaultLoadControl()
    )
    val playerView = findViewById(R.id.player_view)
    
    playerView.player = player
    
    val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent")
    
    val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
    
    player.prepare(mediaSource)
    
    player.playWhenReady = true
    

    View Slide

62. val player = ExoPlayerFactory.newSimpleInstance(
    DefaultRenderersFactory(context),
    DefaultTrackSelector(),
    DefaultLoadControl()
    )
    val playerView = findViewById(R.id.player_view)
    
    playerView.player = player
    
    val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent")
    
    val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
    
    player.prepare(mediaSource)
    
    player.playWhenReady = true
    val player = ExoPlayerFactory.newSimpleInstance(
    DefaultRenderersFactory(context),
    DefaultTrackSelector(),
    DefaultLoadControl()
    )
    

    View Slide

63. val player = ExoPlayerFactory.newSimpleInstance(
    DefaultRenderersFactory(context),
    DefaultTrackSelector(),
    DefaultLoadControl()
    )
    val playerView = findViewById(R.id.player_view)
    
    playerView.player = player
    
    val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent")
    
    val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
    
    player.prepare(mediaSource)
    
    player.playWhenReady = true
    val playerView = findViewById(R.id.player_view)
    playerView.player = player
    

    View Slide

64. val player = ExoPlayerFactory.newSimpleInstance(
    DefaultRenderersFactory(context),
    DefaultTrackSelector(),
    DefaultLoadControl()
    )
    val playerView = findViewById(R.id.player_view)
    
    playerView.player = player
    
    val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent")
    
    val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
    
    player.prepare(mediaSource)
    
    player.playWhenReady = true
    val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent")
    
    val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
    

    View Slide

65. val player = ExoPlayerFactory.newSimpleInstance(
    DefaultRenderersFactory(context),
    DefaultTrackSelector(),
    DefaultLoadControl()
    )
    val playerView = findViewById(R.id.player_view)
    
    playerView.player = player
    
    val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent")
    
    val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
    
    player.prepare(mediaSource)
    
    player.playWhenReady = true
    player.prepare(mediaSource)
    
    player.playWhenReady = true
    

    View Slide

66. val player = ExoPlayerFactory.newSimpleInstance(
    DefaultRenderersFactory(context),
    DefaultTrackSelector(),
    DefaultLoadControl()
    )
    val playerView = findViewById(R.id.player_view)
    
    playerView.player = player
    
    val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent")
    
    val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
    
    player.prepare(mediaSource)
    
    player.playWhenReady = true
    // clean up
    
    player.release()
    

    View Slide

67. val player = ExoPlayerFactory.newSimpleInstance(
    DefaultRenderersFactory(context),
    DefaultTrackSelector(),
    DefaultLoadControl()
    )
    val playerView = findViewById(R.id.player_view)
    
    playerView.player = player
    
    val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent")
    
    val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
    
    player.prepare(mediaSource)
    
    player.playWhenReady = true
    

    View Slide

68. val player = ExoPlayerFactory.newSimpleInstance(…)
    val playerView = findViewById(R.id.player_view)
    
    playerView.player = player
    
    val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent")
    
    val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
    
    player.prepare(mediaSource)
    
    player.playWhenReady = true
    

    View Slide

69. val player = ExoPlayerFactory.newSimpleInstance(…)
    val playerView = findViewById(R.id.player_view)
    
    playerView.player = player
    
    val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent")
    
    val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
    
    player.prepare(mediaSource)
    
    player.playWhenReady = true
    

    View Slide

70. val player = ExoPlayerFactory.newSimpleInstance(…)
    val playerView = findViewById(R.id.player_view)
    
    playerView.player = player
    
    val okHttpClient = OkHttpClient.Builder()
    .connectTimeout(10L, TimeUnit.SECONDS)
    .writeTimeout(10L, TimeUnit.SECONDS)
    .readTimeout(30L, TimeUnit.SECONDS)
    .build()
    val dataSourceFactory = DefaultDataSourceFactory(context, "user-agent")
    
    val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
    
    player.prepare(mediaSource)
    
    player.playWhenReady = true
    

    View Slide

71. val player = ExoPlayerFactory.newSimpleInstance(…)
    val playerView = findViewById(R.id.player_view)
    
    playerView.player = player
    
    val okHttpClient = OkHttpClient.Builder()
    .connectTimeout(10L, TimeUnit.SECONDS)
    .writeTimeout(10L, TimeUnit.SECONDS)
    .readTimeout(30L, TimeUnit.SECONDS)
    .build()
    val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
    
    player.prepare(mediaSource)
    
    player.playWhenReady = true
    

    View Slide

72. val player = ExoPlayerFactory.newSimpleInstance(…)
    val playerView = findViewById(R.id.player_view)
    
    playerView.player = player
    
    val okHttpClient = OkHttpClient.Builder()
    .connectTimeout(10L, TimeUnit.SECONDS)
    .writeTimeout(10L, TimeUnit.SECONDS)
    .readTimeout(30L, TimeUnit.SECONDS)
    .build()
    val dataSourceFactory = OkHttpDataSourceFactory(okHttpClient, “user-agent”, null)
    
    val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
    
    player.prepare(mediaSource)
    
    player.playWhenReady = true
    

    View Slide

73. val player = ExoPlayerFactory.newSimpleInstance(…)
    val playerView = findViewById(R.id.player_view)
    
    playerView.player = player
    
    val okHttpClient = OkHttpClient(…)
    val dataSourceFactory = OkHttpDataSourceFactory(okHttpClient, “user-agent”, null)
    
    val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
    
    player.prepare(mediaSource)
    
    player.playWhenReady = true
    

    View Slide

74. val player = ExoPlayerFactory.newSimpleInstance(
    DefaultRenderersFactory(context),
    DefaultTrackSelector(),
    DefaultLoadControl()
    )
    val playerView = findViewById(R.id.player_view)
    
    playerView.player = player
    
    val okHttpClient = OkHttpClient(…)
    val dataSourceFactory = OkHttpDataSourceFactory(okHttpClient, “user-agent”, null)
    
    val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
    
    player.prepare(mediaSource)
    
    player.playWhenReady = true
    

    View Slide

75. val player = ExoPlayerFactory.newSimpleInstance(
    DefaultRenderersFactory(context),
    DefaultTrackSelector(),
    DefaultLoadControl()
    )
    val playerView = findViewById(R.id.player_view)
    
    playerView.player = player
    
    val okHttpClient = OkHttpClient(…)
    val dataSourceFactory = OkHttpDataSourceFactory(okHttpClient, “user-agent”, null)
    
    val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
    
    player.prepare(mediaSource)
    
    player.playWhenReady = true
    DefaultRenderersFactory(context),
    DefaultTrackSelector(),
    DefaultLoadControl()
    

    View Slide

76. Renderer
    val player = ExoPlayerFactory.newSimpleInstance(
    DefaultRenderersFactory(context),
    DefaultTrackSelector(),
    DefaultLoadControl()
    )
    

    View Slide

77. Renderer
    val player = ExoPlayerFactory.newSimpleInstance(
    DefaultRenderersFactory(context),
    DefaultTrackSelector(),
    DefaultLoadControl()
    )
    DefaultRenderersFactory(context)
    

    View Slide

78. Renderer
    public class MyRenderer implements Renderer {
    // make your own renderer
    }
    

    View Slide

79. Renderer
    public class MyRenderer implements Renderer {
    // make your own renderer
    }
    DefaultRenderersFactory(..., MyRenderer())
    

    View Slide

80. TrackSelector
    val player = ExoPlayerFactory.newSimpleInstance(
    DefaultRenderersFactory(context),
    DefaultTrackSelector(),
    DefaultLoadControl()
    )
    

    View Slide

81. TrackSelector
    val player = ExoPlayerFactory.newSimpleInstance(
    DefaultRenderersFactory(context),
    DefaultTrackSelector(),
    DefaultLoadControl()
    )
    DefaultTrackSelector(),
    

    View Slide

82. TrackSelector
    val selector = DefaultTrackSelector()
    

    View Slide

83. TrackSelector
    val factory = AdaptiveTrackSelection.Factory(
    DefaultBandwidthMeter(),
    DEFAULT_MAX_INITIAL_BITRATE,
    DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS,
    DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS,
    DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS,
    DEFAULT_BANDWIDTH_FRACTION)
    

    View Slide

84. TrackSelector
    val factory = AdaptiveTrackSelection.Factory(
    DefaultBandwidthMeter(),
    DEFAULT_MAX_INITIAL_BITRATE,
    DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS,
    DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS,
    DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS,
    DEFAULT_BANDWIDTH_FRACTION)
    val selector = DefaultTrackSelector(factory)
    

    View Slide

85. TrackSelector
    val selector = DefaultTrackSelector(factory)
    val factory = AdaptiveTrackSelection.Factory(
    DefaultBandwidthMeter(),
    HIEST_BITRATE,
    DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS,
    DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS,
    DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS,
    DEFAULT_BANDWIDTH_FRACTION)
    

    View Slide

86. client
    480p
    240p
    720p
    HTTP
    .m3u8
    .m3u8
    .m3u8
    

    View Slide

87. client
    480p
    240p
    720p
    .m3u8
    .m3u8
    .m3u8
    HTTP
    

    View Slide

88. TrackSelector
    val selector = DefaultTrackSelector(factory)
    val factory = AdaptiveTrackSelection.Factory(
    DefaultBandwidthMeter(),
    DEFAULT_MAX_INITIAL_BITRATE,
    DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS,
    DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS,
    DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS,
    DEFAULT_BANDWIDTH_FRACTION)
    

    View Slide

89. TrackSelector
    val selector = DefaultTrackSelector(factory)
    val factory = AdaptiveTrackSelection.Factory(
    DefaultBandwidthMeter(),
    DEFAULT_MAX_INITIAL_BITRATE,
    DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS,
    DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS,
    DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS,
    DEFAULT_BANDWIDTH_FRACTION)
    DefaultBandwidthMeter(),
    

    View Slide

90. TrackSelector
    public final class DefaultBandwidthMeter implements BandwidthMeter, TransferListener {
    @Override public long getBitrateEstimate() {
    return currentBitrate;
    }
    @Override public void onTransferStart(Object source, DataSpec dataSpec) {
    }
    @Override public void onBytesTransferred(Object source, int bytesTransferred) {
    }
    @Override public void onTransferEnd(Object source) {
    }
    }
    

    View Slide

91. TrackSelector
    public final class DefaultBandwidthMeter implements BandwidthMeter, TransferListener {
    @Override public long getBitrateEstimate() {
    return currentBitrate;
    }
    @Override public void onTransferStart(Object source, DataSpec dataSpec) {
    }
    @Override public void onBytesTransferred(Object source, int bytesTransferred) {
    }
    @Override public void onTransferEnd(Object source) {
    }
    }
    @Override public long getBitrateEstimate() {
    return currentBitrate;
    }
    

    View Slide

92. TrackSelector
    public final class MyBandwidthMeter implements BandwidthMeter,
    TransferListener {
    @Override public long getBitrateEstimate() {
    return Math.min(currentBitrate, limitBitrate);
    }
    @Override public void onTransferStart(Object source, DataSpec dataSpec) {
    }
    @Override public void onBytesTransferred(Object source, int bytesTransferred) {
    }
    @Override public void onTransferEnd(Object source) {
    }
    }
    @Override public long getBitrateEstimate() {
    return Math.min(currentBitrate, limitBitrate);
    }
    MyBandwidthMeter
    

    View Slide

93. TrackSelector
    val selector = DefaultTrackSelector(factory)
    val factory = AdaptiveTrackSelection.Factory(
    MyBandwidthMeter(),
    DEFAULT_MAX_INITIAL_BITRATE,
    DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS,
    DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS,
    DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS,
    DEFAULT_BANDWIDTH_FRACTION)
    

    View Slide

94. TrackSelector
    val selector = DefaultTrackSelector(factory)
    val factory = AdaptiveTrackSelection.Factory(
    MyBandwidthMeter(),
    DEFAULT_MAX_INITIAL_BITRATE,
    DEFAULT_MIN_DURATION_FOR_QUALITY_INCREASE_MS,
    DEFAULT_MAX_DURATION_FOR_QUALITY_DECREASE_MS,
    DEFAULT_MIN_DURATION_TO_RETAIN_AFTER_DISCARD_MS,
    DEFAULT_BANDWIDTH_FRACTION)
    MyBandwidthMeter(),
    

    View Slide

95. client
    480p
    240p
    720p
    .m3u8
    .m3u8
    .m3u8
    HTTP
    

    View Slide

96. client
    480p
    240p
    720p
    .m3u8
    .m3u8
    .m3u8
    HTTP
    

    View Slide

97. TrackSelector
    public final class MyBandwidthMeter implements BandwidthMeter,
    TransferListener {
    @Override public long getBitrateEstimate() {
    return Math.max(currentBitrate, limitBitrate);
    }
    @Override public void onTransferStart(Object source, DataSpec dataSpec) {
    }
    @Override public void onBytesTransferred(Object source, int bytesTransferred) {
    }
    @Override public void onTransferEnd(Object source) {
    }
    }
    

    View Slide

98. TrackSelector
    public final class MyBandwidthMeter implements BandwidthMeter,
    TransferListener {
    @Override public long getBitrateEstimate() {
    return Math.max(currentBitrate, limitBitrate);
    }
    @Override public void onTransferStart(Object source, DataSpec dataSpec) {
    }
    @Override public void onBytesTransferred(Object source, int bytesTransferred) {
    }
    @Override public void onTransferEnd(Object source) {
    }
    }
    return Math.max(currentBitrate, limitBitrate);
    MyBandwidthMeter
    

    View Slide

99. client
    480p
    240p
    720p
    HTTP
    .m3u8
    .m3u8
    .m3u8
    

    View Slide

100. client
     480p
     240p
     720p
     HTTP
     .m3u8
     .m3u8
     .m3u8
     

     View Slide

101. LoadControl
     val player = ExoPlayerFactory.newSimpleInstance(
     DefaultRenderersFactory(context),
     DefaultTrackSelector(),
     DefaultLoadControl()
     )
     

     View Slide

102. LoadControl
     val player = ExoPlayerFactory.newSimpleInstance(
     DefaultRenderersFactory(context),
     DefaultTrackSelector(),
     DefaultLoadControl()
     )
     DefaultLoadControl()
     

     View Slide

103. LoadControl
     val loadControl = DefaultLoadControl()
     

     View Slide

104. LoadControl
     val loadControl = DefaultLoadControl(
     DefaultAllocator(true, C.DEFAULT_BUFFER_SEGMENT_SIZE),
     DEFAULT_MIN_BUFFER_MS,
     DEFAULT_MAX_BUFFER_MS,
     DEFAULT_BUFFER_FOR_PLAYBACK_MS,
     DEFAULT_BUFFER_FOR_PLAYBACK_AFTER_REBUFFER_MS
     )
     

     View Slide

105. LoadControl
     val loadControl = DefaultLoadControl(
     DefaultAllocator(true, C.DEFAULT_BUFFER_SEGMENT_SIZE),
     DEFAULT_MIN_BUFFER_MS,
     DEFAULT_MAX_BUFFER_MS,
     DEFAULT_BUFFER_FOR_PLAYBACK_MS,
     DEFAULT_BUFFER_FOR_PLAYBACK_AFTER_REBUFFER_MS
     )
     DEFAULT_MIN_BUFFER_MS,
     DEFAULT_MAX_BUFFER_MS,
     DEFAULT_BUFFER_FOR_PLAYBACK_MS,
     DEFAULT_BUFFER_FOR_PLAYBACK_AFTER_REBUFFER_MS
     

     View Slide

106. MediaSource
     val mediaSource = HlsMediaSource(uri, dataSourceFactory, handler, null)
     player.prepare(mediaSource)
     
     player.playWhenReady = true
     

     View Slide

107. MediaSource
     val mediaSource = DashMediaSource(Uri, …)
     player.prepare(mediaSource)
     
     player.playWhenReady = true
     

     View Slide

108. MediaSource
     val mediaSource = ExtractorMediaSource(Uri, …)
     player.prepare(mediaSource)
     
     player.playWhenReady = true
     

     View Slide

109. MediaSource
     val firstSource = HlsMediaSource(firstUri, …)
     val secondSource = HlsMediaSource(secondUri, …)
     val mediaSource = ConcatenatingMediaSource(firstSource, secondSource)
     // play firstSource, then secondSource
     player.prepare(mediaSource)
     
     player.playWhenReady = true
     

     View Slide

110. Metadata
     

     View Slide

111. Metadata
     video
     

     View Slide

112. Metadata
     video
     createdAt?
     title? thumbnail?
     genre?
     

     View Slide

113. Metadata
     ɾID3 tag
     

     View Slide

114. Metadata
     ɾID3 tag
     .ts
     id = 1000
     place = droidkaigi
     

     View Slide

115. Metadata
     

     View Slide

116. Metadata
     class Logger : MetadataOutput {
     override fun onMetadata(metadata: Metadata) {
     (1..metadata.length())
     .map { metadata[it] }
     .filterIsInstance()
     .forEach {
     print("metadata: ${it.value}")
     }
     }
     } // metadata: id=1000,place=droidkaigi
     

     View Slide

117. Metadata
     ɾID3 tag
     ɾemsg box
     

     View Slide

118. Metadata
     ɾID3 tag
     id = 1000
     place = droidkaigi
     ɾemsg box .mp4
     

     View Slide

119. Metadata
     class Logger : MetadataOutput {
     override fun onMetadata(metadata: Metadata) {
     (1..metadata.length())
     .map { metadata[it] }
     .filterIsInstance()
     .forEach {
     print("metadata: ${it.value}")
     }
     }
     } // metadata: id=1000,place=droidkaigi
     

     View Slide

120. Publish
     

     View Slide

121. UDP TCP
     1:1 1:n
     low latency high latency
     WebRTC RTMP
     

     View Slide

122. RTMP
     RTMP
     client server
     

     View Slide

123. RTMP
     RTMP
     client server users
     RTMP
     

     View Slide

124. RTMP
     RTMP
     client server users
     HLS
     MPEG-DASH
     

     View Slide

125. RTMP
     ɾdeveloped by Adobe. not “open” protocol
     ɾRed5 started rtmp in 2005
     ɾWowza started rtmp in 2007
     ɾrtmpdump released in 2008
     ɾrtmp specification released 2012
     

     View Slide

126. RTMP
     “we will stop updating and distributing
     the Flash Player at the end of 2020”
     -Adobe
     

     View Slide

127. RTMP
     RTMP
     

     View Slide

128. RTMP
     RTMP
     

     View Slide

129. RTMP
     RTMP
     

     View Slide

130. RTMP
     RTMP
     

     View Slide

131. RTMP
     PRO
     CON
     ɾsmall header (basically 1, 4, 8, 12 bytes)
     ɾrealtime
     ɾdifficult to cache
     ɾno multiple resolutions
     

     View Slide

132. RTMP
     RTMP
     

     View Slide

133. RTMP
     RTMP RTMP
     

     View Slide

134. RTMP
     RTMP
     

     View Slide

135. RTMP
     dependencies {
     implementation ‘com.google.android.exoplayer:exoplayer-core:x.x.x’
     implementation ‘com.google.android.exoplayer:exoplayer-ui:x.x.x’
     implementation ‘com.google.android.exoplayer:extension-rtmp:x.x.x’
     }
     

     View Slide

136. val mediaSource = ExtractorMediaSource(
     
     Uri.parse("rtmp://xxx.xx.xx.xxx:1935/path/stream_name")
     
     RtmpDataSourceFactory(),
     
     DefaultExtractorsFactory(),
     
     null,
     
     null)
     player.prepare(mediaSource)
     
     player.playWhenReady = true
     

     View Slide

137. val mediaSource = ExtractorMediaSource(
     
     Uri.parse("rtmp://xxx.xx.xx.xxx:1935/path/stream_name")
     
     RtmpDataSourceFactory(),
     
     DefaultExtractorsFactory(),
     
     null,
     
     null)
     player.prepare(mediaSource)
     
     player.playWhenReady = true
     Uri.parse("rtmp://xxx.xx.xx.xxx:1935/path/stream_name")
     

     View Slide

138. val mediaSource = ExtractorMediaSource(
     
     Uri.parse("rtmp://xxx.xx.xx.xxx:1935/path/stream_name")
     
     RtmpDataSourceFactory(),
     
     DefaultExtractorsFactory(),
     
     null,
     
     null)
     player.prepare(mediaSource)
     
     player.playWhenReady = true
     RtmpDataSourceFactory(),
     

     View Slide

139. public final class RtmpDataSource implements DataSource {
     private RtmpClient rtmpClient;
     ...
     @Override
     public int read(byte[] buffer, int offset, int readLength) throws IOException {
     int bytesRead = rtmpClient.read(buffer, offset, readLength);
     ...
     return bytesRead;
     }
     ...
     }
     

     View Slide

140. public final class RtmpDataSource implements DataSource {
     private RtmpClient rtmpClient;
     ...
     @Override
     public int read(byte[] buffer, int offset, int readLength) throws IOException {
     int bytesRead = rtmpClient.read(buffer, offset, readLength);
     ...
     return bytesRead;
     }
     ...
     }
     private RtmpClient rtmpClient;
     rtmpClient.read(buffer, offset, readLength);
     

     View Slide

141. public class RtmpClient {
     
     …
     
     public int read(byte[] data, int offset, int size) throws IOException {
     
     return nativeRead(data, offset, size, rtmpPointer);
     
     }
     
     private native int nativeRead(byte[] data, int offset, int size, long rtmpPointer)
     throws IOException
     
     …
     
     }
     

     View Slide

142. public class RtmpClient {
     
     …
     
     public int read(byte[] data, int offset, int size) throws IOException {
     
     return nativeRead(data, offset, size, rtmpPointer);
     
     }
     
     private native int nativeRead(byte[] data, int offset, int size, long rtmpPointer)
     throws IOException
     
     …
     
     }
     private native int nativeRead(byte[] data, int offset, int size, long rtmpPointer)
     public class RtmpClient
     

     View Slide

143. RTMP
     RTMP
     

     View Slide

144. RTMP
     ɾcapture data
     ɾencode data
     ɾpublish data
     

     View Slide

145. RTMP
     ɾcapture data
     ɾencode data
     ɾpublish data
     

     View Slide

146. RTMP
     ɾcapture data
     ɾencode data
     ɾpublish data
     val audioRecord = AudioRecord(...)
     Audio Data
     

     View Slide

147. RTMP
     ɾcapture data
     ɾencode data
     ɾpublish data
     val audioRecord = AudioRecord(...)
     // camera2 API
     
     val manager : CameraManager
     
     manager.openCamera(...)
     // camera API
     
     val camera = Camera.(info)
     Audio Data
     Video Data
     

     View Slide

148. RTMP
     ɾcapture data
     ɾencode data
     ɾpublish data
     

     View Slide

149. RTMP
     // MediaCodec.java
     
     val encoder = MediaCodec(…)
     
     encoder.start()
     
     val inputBuffers = encoder.getInputBuffer(index)
     
     // or
     
     val surface = encoder.createInputSurface()
     
     // get encoded data
     
     val outputBuffers
     
     = encoder.getOutputBuffer(index)
     ɾcapture data
     ɾencode data
     ɾpublish data
     

     View Slide

150. RTMP
     // MediaCodec.java
     
     val encoder = MediaCodec(…)
     
     encoder.start()
     
     val inputBuffers = encoder.getInputBuffer(index)
     
     // or
     
     val surface = encoder.createInputSurface()
     
     // get encoded data
     
     val outputBuffers
     
     = encoder.getOutputBuffer(index)
     ɾcapture data
     ɾencode data
     ɾpublish data
     // MediaCodec.java
     
     val encoder = MediaCodec(…)
     
     encoder.start()
     

     View Slide

151. RTMP
     ɾcapture data
     ɾencode data
     ɾpublish data
     // MediaCodec.java
     
     val encoder = MediaCodec(…)
     
     encoder.start()
     
     val inputBuffers = encoder.getInputBuffer(index)
     
     // or
     
     val surface = encoder.createInputSurface()
     
     // get encoded data
     
     val outputBuffers
     
     = encoder.getOutputBuffer(index)
     val inputBuffers = encoder.getInputBuffer(index)
     
     // or
     
     val surface = encoder.createInputSurface()
     

     View Slide

152. RTMP
     ɾcapture data
     ɾencode data
     ɾpublish data
     // MediaCodec.java
     
     val encoder = MediaCodec(…)
     
     encoder.start()
     
     val inputBuffers = encoder.getInputBuffer(index)
     
     // or
     
     val surface = encoder.createInputSurface()
     
     // get encoded data
     
     val outputBuffers
     
     = encoder.getOutputBuffer(index)
     // get encoded data
     
     val outputBuffers
     
     = encoder.getOutputBuffer(index)
     

     View Slide

153. RTMP
     ɾcapture data
     ɾencode data
     ɾpublish data
     

     View Slide

154. RTMP
     // rtmpdump
     
     public native int writeVideo(
     
     byte[] data,
     
     int offset,
     
     int length,
     
     int timestamp
     
     )
     ɾcapture data
     ɾencode data
     ɾpublish data
     

     View Slide

155. RTMP
     // rtmpdump
     
     public native int writeVideo(
     
     byte[] data,
     
     int offset,
     
     int length,
     
     int timestamp
     
     )
     // Socket.java
     
     val socket = Socket(“192.xx.xx.xxx", 1935)
     
     val outputStream = DataOutputStream(…)
     
     outputStream.write(byteArray)
     ɾcapture data
     ɾencode data
     ɾpublish data
     

     View Slide

156. https://github.com/TakuSemba/RtmpPublisher
     

     View Slide

157. Demo
     

     View Slide

158. Demo
     RTMP
     Server
     HLS
     RTMP
     Publisher
     Player
     Player
     

     View Slide

159. Demo
     rtmp://160.16.54.236:1935/live/takusemba
     http://160.16.54.236/hls/takusemba.m3u8
     RTMP
     HLS
     

     View Slide

160. Media streaming
     on Android
     https://github.com/takusemba
     https://twitter.com/takusemba
     

     View Slide