HEVC Video with Alpha Channel @TachibanaKaoru 2019/7/24 #potatotips

About Me @TachibanaKaoru Freelance iOS Engineer झຯɿϘϧμϦϯάɺBotWͰཾͷྡྷङ ࠷ۙͷςʔϚɿཱྀΛ͠ͳ͕Β࢓ࣄΛ͢Δ όϦౡɺαϯτϦʔχౡɺόϯίΫɺόϧηϩφɺΞΠεϥϯυ

Alpha Channel alpha channel each pixel include transparency information popular in images

Alpha Channel alpha channel each pixel include transparency information popular in images

video codec support alpha channel Apple ProRes 4444 WebM VP8

media with alpha Image Video High bitrate PNG, TIFF Apple ProRes 4444 Flexible bitrate HEIF, HEIFS HEVC with alpha

HEVC support os iOS 13~ tvOS13 ~ macOS Catalina ~

Playback AVPlayerͰͷ࠶ੜ

Demo Sample Code https://github.com/TachibanaKaoru/ HEVCAlphaSample

Encoding HEVC video with API Color Chroma key from Apple ProRes 4444 to HEVC with alpha

Color Chroma key

let asset = AVAsset(url: sourceURL) guard let exportSession = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetHEVCHighestQualityWithAlpha) else { print("Failed to create export session to HEVC with alpha.") handleExportCompletion(.failed) return } // Make CIFilter let filter = makeChromaKeyFilter(usingHueFrom: 0.3, to: 0.4, brightnessFrom: 0.05, to: 1.0 ) let chromaKeyComposition = AVVideoComposition(asset: asset, applyingCIFiltersWithHandler: { request in let source = request.sourceImage.clampedToExtent() filter.setValue(source, forKey: kCIInputImageKey) let output = filter.outputImage! // Provide the filter output to the composition request.finish(with: output, context: nil) }) // Export exportSession.outputURL = destinationURL exportSession.outputFileType = .mov exportSession.videoComposition = chromaKeyComposition exportSession.exportAsynchronously { handleExportCompletion(exportSession.status) }

let asset = AVAsset(url: sourceURL) guard let exportSession = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetHEVCHighestQualityWithAlpha) else { print("Failed to create export session to HEVC with alpha.") handleExportCompletion(.failed) return } // Make CIFilter let filter = makeChromaKeyFilter(usingHueFrom: 0.3, to: 0.4, brightnessFrom: 0.05, to: 1.0 ) let chromaKeyComposition = AVVideoComposition(asset: asset, applyingCIFiltersWithHandler: { request in let source = request.sourceImage.clampedToExtent() filter.setValue(source, forKey: kCIInputImageKey) let output = filter.outputImage! // Provide the filter output to the composition request.finish(with: output, context: nil) }) // Export exportSession.outputURL = destinationURL exportSession.outputFileType = .mov exportSession.videoComposition = chromaKeyComposition exportSession.exportAsynchronously { handleExportCompletion(exportSession.status) }

let asset = AVAsset(url: sourceURL) guard let exportSession = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetHEVCHighestQualityWithAlpha) else { print("Failed to create export session to HEVC with alpha.") handleExportCompletion(.failed) return } // Make CIFilter let filter = makeChromaKeyFilter(usingHueFrom: 0.3, to: 0.4, brightnessFrom: 0.05, to: 1.0 ) let chromaKeyComposition = AVVideoComposition(asset: asset, applyingCIFiltersWithHandler: { request in let source = request.sourceImage.clampedToExtent() filter.setValue(source, forKey: kCIInputImageKey) let output = filter.outputImage! // Provide the filter output to the composition request.finish(with: output, context: nil) }) // Export exportSession.outputURL = destinationURL exportSession.outputFileType = .mov exportSession.videoComposition = chromaKeyComposition exportSession.exportAsynchronously { handleExportCompletion(exportSession.status) }

from Apple ProRes 4444 to HEVC with alpha

let asset = AVAsset(url: sourceURL) AVAssetExportSession.determineCompatibility(ofExportPreset: AVAssetExportPresetHEVCHighestQualityWithAlpha, with: asset, outputFileType: .mov) { compatible in if compatible { guard let exportSession = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetHEVCHighestQualityWithAlpha) else { handleExportCompletion(.failed) return } // Export exportSession.outputURL = destinationURL exportSession.outputFileType = .mov exportSession.exportAsynchronously { handleExportCompletion(exportSession.status) } } else { handleExportCompletion(.failed) } }

let asset = AVAsset(url: sourceURL) AVAssetExportSession.determineCompatibility(ofExportPreset: AVAssetExportPresetHEVCHighestQualityWithAlpha, with: asset, outputFileType: .mov) { compatible in if compatible { guard let exportSession = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetHEVCHighestQualityWithAlpha) else { handleExportCompletion(.failed) return } // Export exportSession.outputURL = destinationURL exportSession.outputFileType = .mov exportSession.exportAsynchronously { handleExportCompletion(exportSession.status) } } else { handleExportCompletion(.failed) } }

let asset = AVAsset(url: sourceURL) AVAssetExportSession.determineCompatibility(ofExportPreset: AVAssetExportPresetHEVCHighestQualityWithAlpha, with: asset, outputFileType: .mov) { compatible in if compatible { guard let exportSession = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetHEVCHighestQualityWithAlpha) else { handleExportCompletion(.failed) return } // Export exportSession.outputURL = destinationURL exportSession.outputFileType = .mov exportSession.exportAsynchronously { handleExportCompletion(exportSession.status) } } else { handleExportCompletion(.failed) } }

·ͱΊ HEVC͸iOS13͔Β࢖͑ΔAlpha͖ͭಈըϑΥʔϚοτͰ͢ɻ ࠶ੜ͸AVPlayer΍WKWebViewͰ؆୯ʹͰ͖·͢ɻ SpriteKitɺSceneKitɺMetalͰ΋࠶ੜՄೳɻARͰ΋࢖͑·͢ɻ

Links HEVC Video with Alpha https://developer.apple.com/videos/play/wwdc2019/506/ Using HEVC Video with Alpha https://developer.apple.com/documentation/avfoundation/ media_assets_playback_and_editing/using_hevc_video_with_alpha HEVC Video with Alpha https://developer.apple.com/av-foundation/HEVC-Video-with-Alpha-Interoperability- Profile.pdf