Playing videos on Android? ExoPlayer2 at your service!

Playing videos on Android? ExoPlayer2 at your service!

Playing videos might be easy at first but this can get very tricky when you have to support multiple video/audio formats or offline medias. But no worries ExoPlayer2 is there for you! This session will cover the basics of the ExoPlayer such as how to play a video thanks to the library and its core features. We'll go through its architecture, how to set it up and what video format ExoPlayer supports from classic MP4 to adaptive streams like HLS or DASH with protected content. A large overview of its capabilities will be done through this talk. Then we will have a look at some interesting extensions and features such as the download manager or the ffmpeg extension for more audio formats support, building playlists or even managing you own DataSource for playing content on SMB servers.

56047a7b11797f42c2e7030d771fe803?s=128

Julien Salvi

November 26, 2019
Tweet

Transcript

  1. Julien Salvi DroidconSF Playing videos on Android? Exoplayer2 at your

    service! Place your screenshot here
  2. Android addict since Froyo Punk rock & IPA! You can

    find me at @JulienSalvi Julien Salvi Senior Android Engineer @ Innovorder
  3. × Video consumption on mobile × From MediaPlayer to ExoPlayer2

    × ExoPlayer: core features and extensions × Go beyond with ExoPlayer
  4. 1. Video consumption on mobile Gonna watch them all!

  5. “Over 50% of consumption of video is taking place on

    mobile devices!”* *According to the Q2 Global Video Index report from Brightcove
  6. Place your screenshot here Major video player on Android

  7. × Open Source project started in 2014 × Based on

    MediaCodec API introduced in Android 4.1 (API 16) × Lots of features and extensions × Regular updates and reactive developers ExoPlayer A powerful media player for Android
  8. Place your screenshot here Trusted by many Companies

  9. 2. From Mediaplayer to exoplyaer2 From zero to hero!

  10. From Mediaplayer to Exoplayer2 MediaPlayer ExoPlayer ExoPlayer2 2014 2016 Since

    API 1
  11. MediaPlayer Quick intro for playing videos! 11

  12. × Simplest way to play video on Android × Android

    Framework dependent × Not easy to customize × Limited number of media formats From Mediaplayer to Exoplayer2 What’s up MediaPlayer?
  13. Play a video with MediaPlayer val url = "http://your_url.mp4" val

    mediaPlayer: MediaPlayer? = MediaPlayer().apply { setDataSource(url) setSurface(surface) prepare() start() }
  14. Play a video with MediaPlayer val uri = Uri.parse("http://my_video.mp4") val

    mediaController = MediaController(this) videoView.apply { setVideoURI(uri) setMediaController(mediaController) setOnCompletionListener { playNextVideo() } setOnErrorListener { mp, what, extra -> displayError() } start() }
  15. Exoplayer Time to move forward! 15

  16. From Mediaplayer to Exoplayer2 ExoPlayer 1 architecture

  17. Basic setup with ExoPlayer1 // 1. Instantiate the player. player

    = ExoPlayer.Factory.newInstance(RENDERER_COUNT); // 2. Construct renderers. MediaCodecVideoTrackRenderer videoRenderer = ... MediaCodecAudioTrackRenderer audioRenderer = … // 3. Inject the renderers through prepare. player.prepare(videoRenderer, audioRenderer); // 4. Pass the surface to the video renderer. player.sendMessage(videoRenderer, MediaCodecVideoTrackRenderer.MSG_SET_SURFACE, surface); // 5. Start playback. player.setPlayWhenReady(true); ... player.release(); // Don’t forget to release when done!
  18. From Mediaplayer to Exoplayer2 MediaPlayer VS ExoPlayer

  19. Exoplayer Better structure! New features! 19

  20. × Big rewrite in 2016 to solve design limitations and

    workaround × Add a full support of DASH streams × Provide a better API for more complex use cases From Mediaplayer to Exoplayer2 ExoPlayer2 at your service!
  21. × Dynamic playlist, improved HLS support, playing ads, cast extension

    × Download manager for offline playback × 360° videos support × Improved decoder reuse (v2.9+) From Mediaplayer to Exoplayer2 Major new features in ExoPlayer2
  22. Basic setup with ExoPlayer2 player = ExoPlayerFactory.newSimpleInstance(this, new DefaultTrackSelector()); //

    Attach player to the PlayerView playerView.setPlayer(player); // Create default data source DefaultDataSourceFactory dataSourceFactory = new DefaultDataSourceFactory( this, Util.getUserAgent(this, "ExoPlayerDCSF19")); // Create media source ExtractorMediaSource mediaSource = new ExtractorMediaSource.Factory(dataSourceFactory).createMediaSource(Uri.parse(URL)); // Prepare player and start when ready player.prepare(mediaSource); player.setPlayWhenReady(true);
  23. 3. Exoplayer: core features and extensions Deep dive into ExoPlayer2

  24. ExoPlayer in a nutshell 10+ Extensions Supports DRM 3 10+

    Audio formats MP3, AAC, Flac, MIDI, Opus, WAV... Text formats 4+ Adaptive streaming 3 10+ Video formats MP4, FMP4, FLV, WebM, MKV, Ogg...
  25. Add ExoPlayer dependencies to your project repositories { google() jcenter()

    } dependencies { implementation 'com.google.android.exoplayer:exoplayer-core:2.10.8' implementation 'com.google.android.exoplayer:exoplayer-dash:2.10.8' implementation 'com.google.android.exoplayer:exoplayer-hls:2.10.8' implementation 'com.google.android.exoplayer:exoplayer-smoothstreaming:2.10.8' implementation 'com.google.android.exoplayer:exoplayer-ui:2.10.8' implementation 'com.google.android.exoplayer:extension-okhttp:2.10.8' } ExoPlayer2
  26. × Progressive download (MP4) or dynamic streams (DASH) × Dynamic

    playlist, shuffle & repeat mode, stream clipping, side-loading source... × Downloader for offline playback × UI components with custom layouts ExoPlayer2: core features The Force is strong with ExoPlayer
  27. ExoPlayer2: core features

  28. × Defined the media to be played × Injected when

    preparing the player × ExoPlayer provide default MediaSource: - ProgressiveMediaSource - DashMediaSource - HlsMediaSource, - SsMediaSource ExoPlayer2: Media source Core components
  29. Build MediaSource int type = PlayerUtils.inferContentType(mVideoUrl); switch (type) { case

    C.TYPE_DASH: return new DashMediaSource.Factory( new DefaultDashChunkSource.Factory(mediaDataSourceFactory), buildOkDataSourceFactory(true)).createMediaSource(uri); case C.TYPE_HLS: return new HlsMediaSource.Factory(buildOkDataSourceFactory(true)).createMediaSource(uri); case C.TYPE_OTHER: return new ProgressiveMediaSource.Factory(mediaDataSourceFactory).createMediaSource(uri); default: return null; }
  30. × Build advanced and custom videos playbacks × Clip, loop

    or merge different media source × Build your own composition with ConcatenatingMediaSource ExoPlayer2: Media Source MediaSource compositions
  31. Side-loaded subtitles MediaSource[] mediaSources = new MediaSource[subsSize]; for (int i

    = 0; i < subsSize; i++) { String subsUrl = mCaptions.get(i).url; mediaSources[i + 1] = new SingleSampleMediaSource.Factory(mediaDataSourceFactory) .createMediaSource(Uri.parse(subsUrl), Format.createTextSampleFormat("", PlayerUtils.getSubtitleMimeType(subsUrl), C.SELECTION_FLAG_AUTOSELECT, null, null), C.TIME_UNSET); } MergingMediaSource mergingMediaSource = new MergingMediaSource(videoSource, mediaSources); mPlayer.prepare(mergingMediaSource, mPlayerPosition == 0, mPlayerPosition == 0);
  32. Custom MediaSource composition val ms1 = ExtractorMediaSource.Factory(dataSourceFactory).createMediaSource(Uri.parse(URL)); val ms2 =

    ExtractorMediaSource.Factory(dataSourceFactory).createMediaSource(Uri.parse(URL2)); val ms3 = ExtractorMediaSource.Factory(dataSourceFactory).createMediaSource(Uri.parse(URL3)); val clippingMediaSource = ClippingMediaSource(ms1, 30000000, 35000000); val clippingMediaSource2 = ClippingMediaSource(ms2, 35000000, 40000000); val loopingMediaSource2 = LoopingMediaSource(ms2, 2); val finalMediaSource = ConcatenatingMediaSource(clippingMediaSource, loopingMediaSource2, ms3);
  33. × Components that render video, audio, text or metadata. ×

    Consumes media from the MediaSource attached to the player × Injected when the player is created ExoPlayer2: Renderers Core components
  34. × Selects tracks provided by the MediaSource to be consumed

    by each Render × You can control the max video size, the prefered audio language and more × ExoPlayer provides a default TrackSelector × Injected when the player is created ExoPlayer2: Track Selector Core components
  35. × Controls when the MediaSource buffers and how much media

    is buffered × ExoPlayer provides a default LoadControl but can be custom × Injected when the player is created ExoPlayer2: Load control Core components
  36. Build TrackSelector, Renderers and LoadControl // Init track selector DefaultTrackSelector.Parameters

    trackSelectorParameters = new DefaultTrackSelector.ParametersBuilder().build(); TrackSelection.Factory trackFactory = new AdaptiveTrackSelection.Factory(BANDWIDTH_METER); trackSelector = new DefaultTrackSelector(trackFactory); trackSelector.setParameters(trackSelectorParameters); // Build audio, video and text renderers mRenderersFactory = new VrRenderersFactory( getApplicationContext(), DefaultRenderersFactory.EXTENSION_RENDERER_MODE_PREFER); // Init player with its renderers, track selector and DRM session manager. mPlayer = new CinemurVrExoPlayer( mRenderersFactory, trackSelector, new DefaultLoadControl(), drmSessionManager);
  37. × Introduced in ExoPlayer 2.8.0 × Download dynamic and progressive

    streams for offline playback × Easy management: add, remove or query your downloads ExoPlayer2: Download manager Offline playback for the win!
  38. ExoPlayer2: Download manager Offline playback for the win!

  39. Prepare player for offline playback CacheDataSourceFactory dataSourceFactory = new CacheDataSourceFactory(

    new SimpleCache(...), upstreamDataSourceFactory); // Build the media source with the downloaded media DownloadRequest downloadRequest = getDownloadTracker().getDownloadRequest(uri); if (downloadRequest != null) { return DownloadHelper.createMediaSource(downloadRequest, dataSourceFactory); }
  40. × Basic UI for displaying and controlling your media ×

    Customize all UI components with your own layout × /!\ Use SurfaceView over TextureView when possible /!\ ExoPlayer2: UI Components Customize all the things
  41. ExoPlayer2: Event logger Track all the things! EventLogger: state [0.00,

    0.00, window=0, true, BUFFERING] EventLogger: state [0.92, 0.04, window=0, period=0, true, READY] EventLogger: state [11.53, 10.60, window=0, period=0, false, READY] EventLogger: state [14.26, 10.60, window=0, period=0, true, READY] EventLogger: state [131.89, 128.27, window=0, period=0, true, ENDED] × Add EventLogger to player × Track playback state, get player information, media tracks or decoder selection × Know what’s going on real time
  42. Exoplayer2 For more awesomeness! 42

  43. × Supporting new audio or video formats (FLAC, ffmpeg, VP9,

    Opus…) × Supporting new stream or data sources (OkHttp, RTMP, cronet…) × Adding new features (Leanback, cast, GVR…) ExoPlayer2: extensions Extensions are there for...
  44. × Most of the extensions can be added to your

    gradle dependencies × Ffmpeg, Opus or VP9 extension has to be built with the NDK before to be adding to your project × Use EXTENSION_RENDERER_MODE_ON for using renderer extensions ExoPlayer2: extensions Add extensions to your project
  45. GVR extension: spatialized audio public class VrRenderersFactory extends DefaultRenderersFactory {

    private GvrAudioProcessor gvrAudioProcessor; public VrRenderersFactory(Context context, int extensionRendererMode) { super(context, extensionRendererMode); } @Override protected AudioProcessor[] buildAudioProcessors() { gvrAudioProcessor = new GvrAudioProcessor(); return new AudioProcessor[] {gvrAudioProcessor}; } public void setHeadOrientation(float... headOrientation) { gvrAudioProcessor.updateOrientation(headOrientation[0], headOrientation[1], headOrientation[2], headOrientation[3]); } }
  46. ExoPlayer This library rocks because... × Supports a lot of

    video, audio and text formats × Highly customisable from core components to UI × DRM support from API 19 × Playlist and download management
  47. Exoplayer This library is not so great because... × Consumes

    more battery than MediaPlayer × Might increase your APK size × Might be difficult to customize (eg. Renderers)
  48. 4. Go beyond with exoplayer Amazing stuff ahead!

  49. Go beyond with exoplayer ExoPlayer for a VR/AR application Video

    plugin for Game Engines Build your own extensions
  50. Create your own extension × Build your own DataSource for

    enable the connection with the server × Inject the SmbDataSource when building the media sources × Stream your content! Playing video from SMB2/3
  51. Build the SmbDataSourceFactory val smbFile = "smb://<user>:<pwd>@<host>/<video_path>/<video>.mkv" class SmbDataSourceFactory(private val

    fileUrl: String) : DataSource.Factory { override fun createDataSource(): DataSource { val dataSpec = DataSpec(Uri.parse(fileUrl)) return SmbDataSource(dataSpec) } }
  52. Build the SmbDataSource class SmbDataSource constructor(private val dataSpec: DataSpec) :

    DataSource { override fun open(dataSpec: DataSpec): Long override fun read(buffer: ByteArray, offset: Int, readLength: Int): Int override fun getUri(): Uri override fun close() }
  53. Build the SmbDataSource class SmbDataSource constructor(private val dataSpec: DataSpec) :

    DataSource { private var hostname: String? = null private var username: String? = null private var password: String? = null private var shareName: String? = null private var path: String? = null private var smbClient: SMBClient? = null private var inputStream: InputStream? = null . . }
  54. Build the SmbDataSource override fun open(dataSpec: DataSpec): Long { smbClient

    = SMBClient() val connection = smbClient!!.connect(hostname) val ac = AuthenticationContext(username, password, "WORKGROUP") val session = connection.authenticate(ac) // Connect to Share val share = session.connectShare(shareName) as DiskShare val s = HashSet<SMB2ShareAccess>() s.add(SMB2ShareAccess.ALL.iterator().next()) // this is to get READ only val remoteSmbjFile = share.openFile(path, ...) inputStream = remoteSmbjFile.getInputStream() // Process inputStream here opened = true return bytesRemaining }
  55. Build the SmbDataSource override fun open(dataSpec: DataSpec): Long { smbClient

    = SMBClient() val connection = smbClient!!.connect(hostname) val ac = AuthenticationContext(username, password, "WORKGROUP") val session = connection.authenticate(ac) // Connect to Share val share = session.connectShare(shareName) as DiskShare val s = HashSet<SMB2ShareAccess>() s.add(SMB2ShareAccess.ALL.iterator().next()) // this is to get READ only val remoteSmbjFile = share.openFile(path, ...) inputStream = remoteSmbjFile.getInputStream() // Process inputStream here opened = true return bytesRemaining }
  56. Build the SmbDataSource override fun read(buffer: ByteArray, offset: Int, readLength:

    Int): Int { ... val bytesToRead = if (bytesRemaining == C.LENGTH_UNSET.toLong()) { readLength } else { Math.min(bytesRemaining, readLength.toLong()).toInt() } val bytesRead = inputStream.read(buffer, offset, bytesToRead) ... return bytesRead }
  57. Inject the DataSource private fun buildMediaSource(uri: Uri): MediaSource { ...

    if (PlayerUtils.isSmb(videoUrl)) { return new ExtractorMediaSource.Factory( new SmbDataSourceFactory(mVideoUrl)).createMediaSource(uri); } ... } private fun preparePlayer() { player.prepare(buildMediaSource(Uri.parse(videoUrl)), true, true); }
  58. Inject the DataSource private fun buildMediaSource(uri: Uri): MediaSource { ...

    if (PlayerUtils.isSmb(videoUrl)) { return new ExtractorMediaSource.Factory( new SmbDataSourceFactory(mVideoUrl)).createMediaSource(uri); } ... } private fun preparePlayer() { player.prepare(buildMediaSource(Uri.parse(videoUrl)), true, true); }
  59. Exoplayer for VR apps × Use the GVR for building

    your VR application × OpenGL ES for your 3D objects × Attach a SurfaceTexture to your 3D object
  60. Exoplayer Plugin for Unity For example you can use ExoPlayer

    for building a reliable player for Unity thanks to the JNI
  61. Linking native Surface with Unity Player _androidSurface = App_AndroidNative_GetSurfaceObject(); _mediaPlayer.Call("playWhenReady",

    _autoPlay); _currentActivity.Call("runOnUiThread", new AndroidJavaRunnable(() => { IntPtr methodID; methodID = AndroidJNI.GetMethodID(_mediaPlayer.GetRawClass(), "setPlayer", "(Landroid/view/Surface;)V"); jvalue[] parms = new jvalue[1]; parms[0] = new jvalue { l = _androidSurface }; AndroidJNI.CallVoidMethod(_mediaPlayer.GetRawObject(), methodID, parms); }));
  62. DEMO 62

  63. Want to know more? https://exoplayer.dev ExoPlayer documentation https://medium.com/google-exoplayer ExoPlayer developer

    blog https://medium.com/@cinemur/73ec7e83dd5c ExoPlayer for building powerful VR players on Cardboard and GearVR
  64. THANKS! Have fun with ExoPlayer and Android! Any questions? @JulienSalvi