Playing videos on Android? ExoPlayer2 at your service!

Playing videos might be easy at first but this can get very tricky when you have to support multiple video/audio formats or offline medias. But no worries ExoPlayer2 is there for you! This session will cover the basics of the ExoPlayer such as how to play a video thanks to the library and its core features. We'll go through its architecture, how to set it up and what video format ExoPlayer supports from classic MP4 to adaptive streams like HLS or DASH with protected content. A large overview of its capabilities will be done through this talk. Then we will have a look at some interesting extensions and features such as the download manager or the ffmpeg extension for more audio formats support, building playlists or even managing you own DataSource for playing content on SMB servers.

Transcript

1. Julien Salvi DroidconSF Playing videos on Android? Exoplayer2 at your service! Place your screenshot here
2. Android addict since Froyo Punk rock & IPA! You can find me at @JulienSalvi Julien Salvi Senior Android Engineer @ Innovorder
3. × Video consumption on mobile × From MediaPlayer to ExoPlayer2 × ExoPlayer: core features and extensions × Go beyond with ExoPlayer
4. 1. Video consumption on mobile Gonna watch them all!
5. “Over 50% of consumption of video is taking place on mobile devices!”* *According to the Q2 Global Video Index report from Brightcove
6. Place your screenshot here Major video player on Android
7. × Open Source project started in 2014 × Based on MediaCodec API introduced in Android 4.1 (API 16) × Lots of features and extensions × Regular updates and reactive developers ExoPlayer A powerful media player for Android
8. Place your screenshot here Trusted by many Companies
9. 2. From Mediaplayer to exoplyaer2 From zero to hero!
10. From Mediaplayer to Exoplayer2 MediaPlayer ExoPlayer ExoPlayer2 2014 2016 Since API 1
11. MediaPlayer Quick intro for playing videos! 11
12. × Simplest way to play video on Android × Android Framework dependent × Not easy to customize × Limited number of media formats From Mediaplayer to Exoplayer2 What’s up MediaPlayer?
13. Play a video with MediaPlayer val url = "http://your_url.mp4" val mediaPlayer: MediaPlayer? = MediaPlayer().apply { setDataSource(url) setSurface(surface) prepare() start() }
14. Play a video with MediaPlayer val uri = Uri.parse("http://my_video.mp4") val mediaController = MediaController(this) videoView.apply { setVideoURI(uri) setMediaController(mediaController) setOnCompletionListener { playNextVideo() } setOnErrorListener { mp, what, extra -> displayError() } start() }
15. Exoplayer Time to move forward! 15
16. From Mediaplayer to Exoplayer2 ExoPlayer 1 architecture
17. Basic setup with ExoPlayer1 // 1. Instantiate the player. player = ExoPlayer.Factory.newInstance(RENDERER_COUNT); // 2. Construct renderers. MediaCodecVideoTrackRenderer videoRenderer = ... MediaCodecAudioTrackRenderer audioRenderer = … // 3. Inject the renderers through prepare. player.prepare(videoRenderer, audioRenderer); // 4. Pass the surface to the video renderer. player.sendMessage(videoRenderer, MediaCodecVideoTrackRenderer.MSG_SET_SURFACE, surface); // 5. Start playback. player.setPlayWhenReady(true); ... player.release(); // Don’t forget to release when done!
18. From Mediaplayer to Exoplayer2 MediaPlayer VS ExoPlayer
19. Exoplayer Better structure! New features! 19
20. × Big rewrite in 2016 to solve design limitations and workaround × Add a full support of DASH streams × Provide a better API for more complex use cases From Mediaplayer to Exoplayer2 ExoPlayer2 at your service!
21. × Dynamic playlist, improved HLS support, playing ads, cast extension × Download manager for offline playback × 360° videos support × Improved decoder reuse (v2.9+) From Mediaplayer to Exoplayer2 Major new features in ExoPlayer2
22. Basic setup with ExoPlayer2 player = ExoPlayerFactory.newSimpleInstance(this, new DefaultTrackSelector()); // Attach player to the PlayerView playerView.setPlayer(player); // Create default data source DefaultDataSourceFactory dataSourceFactory = new DefaultDataSourceFactory( this, Util.getUserAgent(this, "ExoPlayerDCSF19")); // Create media source ExtractorMediaSource mediaSource = new ExtractorMediaSource.Factory(dataSourceFactory).createMediaSource(Uri.parse(URL)); // Prepare player and start when ready player.prepare(mediaSource); player.setPlayWhenReady(true);
23. 3. Exoplayer: core features and extensions Deep dive into ExoPlayer2
24. ExoPlayer in a nutshell 10+ Extensions Supports DRM 3 10+ Audio formats MP3, AAC, Flac, MIDI, Opus, WAV... Text formats 4+ Adaptive streaming 3 10+ Video formats MP4, FMP4, FLV, WebM, MKV, Ogg...
25. Add ExoPlayer dependencies to your project repositories { google() jcenter() } dependencies { implementation 'com.google.android.exoplayer:exoplayer-core:2.10.8' implementation 'com.google.android.exoplayer:exoplayer-dash:2.10.8' implementation 'com.google.android.exoplayer:exoplayer-hls:2.10.8' implementation 'com.google.android.exoplayer:exoplayer-smoothstreaming:2.10.8' implementation 'com.google.android.exoplayer:exoplayer-ui:2.10.8' implementation 'com.google.android.exoplayer:extension-okhttp:2.10.8' } ExoPlayer2
26. × Progressive download (MP4) or dynamic streams (DASH) × Dynamic playlist, shuffle & repeat mode, stream clipping, side-loading source... × Downloader for offline playback × UI components with custom layouts ExoPlayer2: core features The Force is strong with ExoPlayer
27. ExoPlayer2: core features
28. × Defined the media to be played × Injected when preparing the player × ExoPlayer provide default MediaSource: - ProgressiveMediaSource - DashMediaSource - HlsMediaSource, - SsMediaSource ExoPlayer2: Media source Core components
29. Build MediaSource int type = PlayerUtils.inferContentType(mVideoUrl); switch (type) { case C.TYPE_DASH: return new DashMediaSource.Factory( new DefaultDashChunkSource.Factory(mediaDataSourceFactory), buildOkDataSourceFactory(true)).createMediaSource(uri); case C.TYPE_HLS: return new HlsMediaSource.Factory(buildOkDataSourceFactory(true)).createMediaSource(uri); case C.TYPE_OTHER: return new ProgressiveMediaSource.Factory(mediaDataSourceFactory).createMediaSource(uri); default: return null; }
30. × Build advanced and custom videos playbacks × Clip, loop or merge different media source × Build your own composition with ConcatenatingMediaSource ExoPlayer2: Media Source MediaSource compositions
31. Side-loaded subtitles MediaSource[] mediaSources = new MediaSource[subsSize]; for (int i = 0; i < subsSize; i++) { String subsUrl = mCaptions.get(i).url; mediaSources[i + 1] = new SingleSampleMediaSource.Factory(mediaDataSourceFactory) .createMediaSource(Uri.parse(subsUrl), Format.createTextSampleFormat("", PlayerUtils.getSubtitleMimeType(subsUrl), C.SELECTION_FLAG_AUTOSELECT, null, null), C.TIME_UNSET); } MergingMediaSource mergingMediaSource = new MergingMediaSource(videoSource, mediaSources); mPlayer.prepare(mergingMediaSource, mPlayerPosition == 0, mPlayerPosition == 0);
32. Custom MediaSource composition val ms1 = ExtractorMediaSource.Factory(dataSourceFactory).createMediaSource(Uri.parse(URL)); val ms2 = ExtractorMediaSource.Factory(dataSourceFactory).createMediaSource(Uri.parse(URL2)); val ms3 = ExtractorMediaSource.Factory(dataSourceFactory).createMediaSource(Uri.parse(URL3)); val clippingMediaSource = ClippingMediaSource(ms1, 30000000, 35000000); val clippingMediaSource2 = ClippingMediaSource(ms2, 35000000, 40000000); val loopingMediaSource2 = LoopingMediaSource(ms2, 2); val finalMediaSource = ConcatenatingMediaSource(clippingMediaSource, loopingMediaSource2, ms3);
33. × Components that render video, audio, text or metadata. × Consumes media from the MediaSource attached to the player × Injected when the player is created ExoPlayer2: Renderers Core components
34. × Selects tracks provided by the MediaSource to be consumed by each Render × You can control the max video size, the prefered audio language and more × ExoPlayer provides a default TrackSelector × Injected when the player is created ExoPlayer2: Track Selector Core components
35. × Controls when the MediaSource buffers and how much media is buffered × ExoPlayer provides a default LoadControl but can be custom × Injected when the player is created ExoPlayer2: Load control Core components
36. Build TrackSelector, Renderers and LoadControl // Init track selector DefaultTrackSelector.Parameters trackSelectorParameters = new DefaultTrackSelector.ParametersBuilder().build(); TrackSelection.Factory trackFactory = new AdaptiveTrackSelection.Factory(BANDWIDTH_METER); trackSelector = new DefaultTrackSelector(trackFactory); trackSelector.setParameters(trackSelectorParameters); // Build audio, video and text renderers mRenderersFactory = new VrRenderersFactory( getApplicationContext(), DefaultRenderersFactory.EXTENSION_RENDERER_MODE_PREFER); // Init player with its renderers, track selector and DRM session manager. mPlayer = new CinemurVrExoPlayer( mRenderersFactory, trackSelector, new DefaultLoadControl(), drmSessionManager);
37. × Introduced in ExoPlayer 2.8.0 × Download dynamic and progressive streams for offline playback × Easy management: add, remove or query your downloads ExoPlayer2: Download manager Offline playback for the win!
38. ExoPlayer2: Download manager Offline playback for the win!
39. Prepare player for offline playback CacheDataSourceFactory dataSourceFactory = new CacheDataSourceFactory( new SimpleCache(...), upstreamDataSourceFactory); // Build the media source with the downloaded media DownloadRequest downloadRequest = getDownloadTracker().getDownloadRequest(uri); if (downloadRequest != null) { return DownloadHelper.createMediaSource(downloadRequest, dataSourceFactory); }
40. × Basic UI for displaying and controlling your media × Customize all UI components with your own layout × /!\ Use SurfaceView over TextureView when possible /!\ ExoPlayer2: UI Components Customize all the things
41. ExoPlayer2: Event logger Track all the things! EventLogger: state [0.00, 0.00, window=0, true, BUFFERING] EventLogger: state [0.92, 0.04, window=0, period=0, true, READY] EventLogger: state [11.53, 10.60, window=0, period=0, false, READY] EventLogger: state [14.26, 10.60, window=0, period=0, true, READY] EventLogger: state [131.89, 128.27, window=0, period=0, true, ENDED] × Add EventLogger to player × Track playback state, get player information, media tracks or decoder selection × Know what’s going on real time
42. Exoplayer2 For more awesomeness! 42
43. × Supporting new audio or video formats (FLAC, ffmpeg, VP9, Opus…) × Supporting new stream or data sources (OkHttp, RTMP, cronet…) × Adding new features (Leanback, cast, GVR…) ExoPlayer2: extensions Extensions are there for...
44. × Most of the extensions can be added to your gradle dependencies × Ffmpeg, Opus or VP9 extension has to be built with the NDK before to be adding to your project × Use EXTENSION_RENDERER_MODE_ON for using renderer extensions ExoPlayer2: extensions Add extensions to your project
45. GVR extension: spatialized audio public class VrRenderersFactory extends DefaultRenderersFactory { private GvrAudioProcessor gvrAudioProcessor; public VrRenderersFactory(Context context, int extensionRendererMode) { super(context, extensionRendererMode); } @Override protected AudioProcessor[] buildAudioProcessors() { gvrAudioProcessor = new GvrAudioProcessor(); return new AudioProcessor[] {gvrAudioProcessor}; } public void setHeadOrientation(float... headOrientation) { gvrAudioProcessor.updateOrientation(headOrientation[0], headOrientation[1], headOrientation[2], headOrientation[3]); } }
46. ExoPlayer This library rocks because... × Supports a lot of video, audio and text formats × Highly customisable from core components to UI × DRM support from API 19 × Playlist and download management
47. Exoplayer This library is not so great because... × Consumes more battery than MediaPlayer × Might increase your APK size × Might be difficult to customize (eg. Renderers)
48. 4. Go beyond with exoplayer Amazing stuff ahead!
49. Go beyond with exoplayer ExoPlayer for a VR/AR application Video plugin for Game Engines Build your own extensions
50. Create your own extension × Build your own DataSource for enable the connection with the server × Inject the SmbDataSource when building the media sources × Stream your content! Playing video from SMB2/3
51. Build the SmbDataSourceFactory val smbFile = "smb://<user>:<pwd>@<host>/<video_path>/<video>.mkv" class SmbDataSourceFactory(private val fileUrl: String) : DataSource.Factory { override fun createDataSource(): DataSource { val dataSpec = DataSpec(Uri.parse(fileUrl)) return SmbDataSource(dataSpec) } }
52. Build the SmbDataSource class SmbDataSource constructor(private val dataSpec: DataSpec) : DataSource { override fun open(dataSpec: DataSpec): Long override fun read(buffer: ByteArray, offset: Int, readLength: Int): Int override fun getUri(): Uri override fun close() }
53. Build the SmbDataSource class SmbDataSource constructor(private val dataSpec: DataSpec) : DataSource { private var hostname: String? = null private var username: String? = null private var password: String? = null private var shareName: String? = null private var path: String? = null private var smbClient: SMBClient? = null private var inputStream: InputStream? = null . . }
54. Build the SmbDataSource override fun open(dataSpec: DataSpec): Long { smbClient = SMBClient() val connection = smbClient!!.connect(hostname) val ac = AuthenticationContext(username, password, "WORKGROUP") val session = connection.authenticate(ac) // Connect to Share val share = session.connectShare(shareName) as DiskShare val s = HashSet<SMB2ShareAccess>() s.add(SMB2ShareAccess.ALL.iterator().next()) // this is to get READ only val remoteSmbjFile = share.openFile(path, ...) inputStream = remoteSmbjFile.getInputStream() // Process inputStream here opened = true return bytesRemaining }
55. Build the SmbDataSource override fun open(dataSpec: DataSpec): Long { smbClient = SMBClient() val connection = smbClient!!.connect(hostname) val ac = AuthenticationContext(username, password, "WORKGROUP") val session = connection.authenticate(ac) // Connect to Share val share = session.connectShare(shareName) as DiskShare val s = HashSet<SMB2ShareAccess>() s.add(SMB2ShareAccess.ALL.iterator().next()) // this is to get READ only val remoteSmbjFile = share.openFile(path, ...) inputStream = remoteSmbjFile.getInputStream() // Process inputStream here opened = true return bytesRemaining }
56. Build the SmbDataSource override fun read(buffer: ByteArray, offset: Int, readLength: Int): Int { ... val bytesToRead = if (bytesRemaining == C.LENGTH_UNSET.toLong()) { readLength } else { Math.min(bytesRemaining, readLength.toLong()).toInt() } val bytesRead = inputStream.read(buffer, offset, bytesToRead) ... return bytesRead }
57. Inject the DataSource private fun buildMediaSource(uri: Uri): MediaSource { ... if (PlayerUtils.isSmb(videoUrl)) { return new ExtractorMediaSource.Factory( new SmbDataSourceFactory(mVideoUrl)).createMediaSource(uri); } ... } private fun preparePlayer() { player.prepare(buildMediaSource(Uri.parse(videoUrl)), true, true); }
58. Inject the DataSource private fun buildMediaSource(uri: Uri): MediaSource { ... if (PlayerUtils.isSmb(videoUrl)) { return new ExtractorMediaSource.Factory( new SmbDataSourceFactory(mVideoUrl)).createMediaSource(uri); } ... } private fun preparePlayer() { player.prepare(buildMediaSource(Uri.parse(videoUrl)), true, true); }
59. Exoplayer for VR apps × Use the GVR for building your VR application × OpenGL ES for your 3D objects × Attach a SurfaceTexture to your 3D object
60. Exoplayer Plugin for Unity For example you can use ExoPlayer for building a reliable player for Unity thanks to the JNI
61. Linking native Surface with Unity Player _androidSurface = App_AndroidNative_GetSurfaceObject(); _mediaPlayer.Call("playWhenReady", _autoPlay); _currentActivity.Call("runOnUiThread", new AndroidJavaRunnable(() => { IntPtr methodID; methodID = AndroidJNI.GetMethodID(_mediaPlayer.GetRawClass(), "setPlayer", "(Landroid/view/Surface;)V"); jvalue[] parms = new jvalue[1]; parms[0] = new jvalue { l = _androidSurface }; AndroidJNI.CallVoidMethod(_mediaPlayer.GetRawObject(), methodID, parms); }));
62. DEMO 62
63. Want to know more? https://exoplayer.dev ExoPlayer documentation https://medium.com/google-exoplayer ExoPlayer developer blog https://medium.com/@cinemur/73ec7e83dd5c ExoPlayer for building powerful VR players on Cardboard and GearVR
64. THANKS! Have fun with ExoPlayer and Android! Any questions? @JulienSalvi