Video Streaming

Transcript

1. Video Streaming iOS / tvOS How to get the most out of Apple’s SDK
2. How Video Streaming Works - Terms to be learned - Video on Demand (VOD) - Online streaming - Adaptive streaming - Live streaming - Program stream vs Transport stream - Streaming Protocol / Manifest / HLS - DRM
3. HLS Manifest - Media Segments (chunks) - One media split into multiple files (segments) - /video/s0.mp4, /video/s1.mp4, ... - Separate for Audio and Video - /video/s0.mp4, /audio/s0.mp4 - Various durations (2-10 seconds) - Various qualities - /audio/128kbps/segment0.mp4 … - Various languages - audio tracks - /audio/french/128kbps/segment0.mp4 … - Various subtitles - /audio/french/segment0.vtt …
4. Video Streaming Pros - Allows progressive download for specific seek position - Basic static http hosting only needed - Change quality or language on fly - Adaptive streaming - Quick start - No need to download whole media
5. Video Streaming Cons - A lot of space on server - /video/720p/segment0.mp4 - /video/1080p/segment0.mp4 - /audio/english/128kbps/segment0.mp4 - /audio/english/256kbps/segment0.mp4 - /audio/french/256kbps/segment0.mp4 - Even worse when audio and video in one file. - Network bandwidth and stability - Complicated download for offline playback
6. Live Streaming - Simplified by the fact that media are segmented - Freshly created segments are quickly consumed by video players - Delayed compared to TV stream (segment duration + encoding time + request / response + latencies + playhead) - Streaming Protocol / Manifest to help identify existing segments
7. - Structured file - DASH (XML), Smooth Streaming (XML), HLS (M3U) - Describes media - Available video and audio qualities - Segment duration - Languages - Available segments (for live) - Some are codec/encoding agnostic Streaming Protocol / Manifest
8. HLS - HTTP Live Streaming (not just for live) - Streaming protocol developed by Apple - Launched in summer 2009 (with iPhone 3) - Replaced Quicktime - Manifests/playlists in the M3U8 format (M3U encoded in UTF-8) - Main index (playlist) usually points to other playlists - quality/language specific - Supports closed captions - Codec/Encoding H264/H265 + AAC (more options for audio) - Container MPEG-2 Transport Stream (.ts files)
9. DRM - Digital Rights Management - Access control technology to limit the use of digital content - Inhibits the non desired use of content - Prevents copying, converting, redistributing - Hardware accelerated - HLS -> AES encryption - A layer of security (not Hollywood grade) - AES-128 (all segments same key) - SAMPLE-AES (each segment by its own AES) - FairPlay - Developed by Apple in and used in iTunes initially - Regular MP4 container with encrypted AAC audio layer (by AES)
10. iOS / tvOS Player • AVPlayerViewController ??? !!! ??? • AVPlayer ◦ AVPlayerLayer • AVPlayerItem ◦ AVPlayerItem(url: URL) - Maybe ◦ AVPlayerItem(asset: AVAsset) - Good choice • AVURLAsset ◦ AVURLAsset(url: URL, options: [String : Any]?) • AVPictureInPictureController ◦ AVPictureInPictureControllerDelegate
11. iOS / tvOS Player - handling playback events playerItem.observe(\AVPlayerItem.status) { [weak self] _, _ in player.addPeriodicTimeObserver(forInterval: timeInterval, queue: DispatchQueue.main) { [weak self] (elapsedTime: CMTime) -> Void in NotificationCenter.default.addObserver(self, selector: #selector(playerDidFinishPlaying(notification:)), name: NSNotification.Name.AVPlayerItemDidPlayToEndTime, object: player.currentItem) NotificationCenter.default.addObserver(self, selector: #selector(playerStalled(notification:)), name: NSNotification.Name.AVPlayerItemPlaybackStalled, object: player.currentItem)
12. iOS / tvOS Player - tips & tricks • Override layer and define it as AVPlayerLayer • Use your own layer animation for resizing if you need special effects • Check if player.currentItem?.status == AVPlayerItem.Status.readyToPlay ◦ Now you can use player.seek() to resume playback • Unhappy with black screen and spinner? ◦ Use your own stalled playback detection ▪ player.rate > 0 ▪ videoTrack.currentVideoFrameRate > 5 ▪ playerLayer.isReadyForDisplay - check during the playback progress ◦ Use playback recovery techniques ▪ player.pause() and player.resume() ▪ player.replaceCurrentItem(with: nil) and player.replaceCurrentItem(with: playerItem)
13. iOS Downloads • URLSessionConfiguration • URLSession or AVAssetDownloadURLSession ◦ downloadQueue = OperationQueue() ◦ session = URLSession(configuration: configuration, delegate: self, delegateQueue: downloadQueue) • URLSessionDownloadTask - iOS 9, 10, 11, 12 (only file download - MP4, no HLS) ◦ session.downloadTask(with: URLRequest) • AVAssetDownloadTask - iOS 10, 11, 12 (HLS without automatic download of tracks) ◦ downloadTask = assetSessionManager.makeAssetDownloadTask(asset: urlAsset, ...) • AVAggregateAssetDownloadTask - iOS 11, 12 (HLS with automatic download of tracks) ◦ downloadTask = assetSessionManager.aggregateAssetDownloadTask(with: urlAsset, mediaSelections: urlAsset.allMediaSelections, ...)
14. iOS Downloads - handling download events - page 1 extension AxisDownloadProvider: URLSessionDownloadDelegate { func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didWriteData bytesWritten: Int64, totalBytesWritten: Int64, totalBytesExpectedToWrite: Int64) {} func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didFinishDownloadingTo location: URL) {} func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) {} }
15. iOS Downloads - handling download events - page 2 extension HlsDownloadProvider: AVAssetDownloadDelegate { func urlSession(_ session: URLSession, assetDownloadTask: AVAssetDownloadTask, didLoad timeRange: CMTimeRange, totalTimeRangesLoaded loadedTimeRanges: [NSValue], timeRangeExpectedToLoad: CMTimeRange) {} func urlSession(_ session: URLSession, assetDownloadTask: AVAssetDownloadTask, didFinishDownloadingTo location: URL) {} func urlSession(_ session: URLSession, assetDownloadTask: AVAssetDownloadTask, didResolve resolvedMediaSelection: AVMediaSelection) {} }
16. iOS Downloads - handling download events - page 3 extension HlsDownloadProvider: AVAssetDownloadDelegate { func urlSession(_ session: URLSession, aggregateAssetDownloadTask: AVAggregateAssetDownloadTask, didLoad timeRange: CMTimeRange, totalTimeRangesLoaded loadedTimeRanges: [NSValue], timeRangeExpectedToLoad: CMTimeRange, for mediaSelection: AVMediaSelection) {} func urlSession(_ session: URLSession, aggregateAssetDownloadTask: AVAggregateAssetDownloadTask, didCompleteFor mediaSelection: AVMediaSelection) {} func urlSession(_ session: URLSession, aggregateAssetDownloadTask: AVAggregateAssetDownloadTask, willDownloadTo location: URL) {} }
17. iOS Downloads - tips & tricks - page 1 let configuration = URLSessionConfiguration.background(withIdentifier: urlBackgroundIdentifier) configuration.isDiscretionary = false (consider carefully if you set to true - recommended by Apple configuration.sessionSendsLaunchEvents = true configuration.shouldUseExtendedBackgroundIdleMode = true downloadQueue.qualityOfService = .userInteractive (not on battery, use .utility - recommended by Apple) sessionManager = URLSession(configuration: configuration, delegate: self, delegateQueue: downloadQueue)
18. iOS Downloads - tips & tricks - page 2 AVAssetDownloadTask - download assets (media selection options) programmatically func urlSession(_ session: URLSession, assetDownloadTask: AVAssetDownloadTask, didResolve resolvedMediaSelection: AVMediaSelection) { // save resolvedMediaSelection somewhere } • AVURLAsset ◦ holds asset cache AVAssetCache • AVAssetCache ◦ check which assets have been loaded and load the remaining • AVMediaCharacteristic.audible, AVMediaCharacteristic.legible • AVMediaSelectionGroup ◦ list of media selection options [AVMediaSelectionOption] • AVMediaSelectionOption ◦ check if this selection option is in cache, if not downlaod it
19. Links • How video streaming works on the web ◦ https://medium.com/canal-tech/how-video-streaming-works-on-the-web-an-introduction-7919739f7e1 • HTTP Live Streaming ◦ https://en.wikipedia.org/wiki/HTTP_Live_Streaming#Using_fragmented_MP4 • HLS Streaming ◦ https://developer.apple.com/streaming/ • HLS Authoring ◦ https://developer.apple.com/documentation/http_live_streaming/hls_authoring_specification_for_apple_devices • HLS for Offline Playback ◦ https://developer.apple.com/documentation/avfoundation/media_assets_playback_and_editing/using_avfoundation_to_ play_and_persist_http_live_streams • FairPlay ◦ https://en.wikipedia.org/wiki/FairPlay