Let's make an Immersive Video with APMP

This document summarizes a brief explanation of APMP, which was announced at WWDC25.

Transcript

1. ͓खݩͷಈըΛAPMPʹ ม׵͓ͯ͠खܰʹΠϚʔ γϰͳମݧΛ࡞Ζ͏ʂ
3. visionOSͰݟΔ͜ͱ͕Ͱ͖Δಈըͷछྨ
4. Apple Projected Media Profile
5. Apple Projected Media Profile 360°ಈըͷΑ͏ͳ౤ӨܕϝσΟΞΛѻ͏ͨΊʹISOBMFFͷ֦ு ͱͯ͠࡞ΒΕͨϓϩϑΝΠϧ
6. ISOBMFF (ISO Base Media File Format) ߏ଄ ISOBMFF File ftyp (File Type Box) ϑΝΠϧλΠϓɺϒϥϯυɺޓ׵ੑ৘ใ moov (Movie Box) - ϝλσʔλίϯςφ mvhd (Movie Header) શମͷ࣌ؒ৘ใ trak (Track Box) τϥοΫ৘ใ mvex (Movie Extends) ϑϥάϝϯτ༻֦ு tkhd ϔομʔ mdia ϝσΟΞ mdhd hdlr minf mdat (Media Data Box) ࣮ࡍͷϝσΟΞσʔλʢԻ੠ɾө૾ɾࣈນͳͲʣ ௨ৗϑΝΠϧͷେ෦෼Λ઎ΊΔ free/skip (Free Space) ະ࢖༻ྖҬ udta (User Data) Ϣʔβʔσʔλ ͦͷଞͷBox ֦ுɾΧελϜBox ஫ɿBox ͸֊૚ߏ଄Λ࣋ͪɺ֤Box ͸ size (4bytes) + type (4bytes) + data Ͱߏ੒͞ΕΔ ISOBMFFɺQTFFͱ͸ • ISOBMFF͸ϏσΦ΍ΦʔσΟΦͳͲ ͷϚϧνϝσΟΞσʔλΛؚΉϑΝΠ ϧͷҰൠతͳߏ଄͕ఆٛ͞Εͨࠃࡍ ج४ͷϑΥʔϚοτ • QTFF͕ϕʔε • MP4͸ISOBMFFʹج͍࣮ͮͨࡍͷ ϑΝΠϧܗࣜ
7. ISOBMFFͷ֦ுʹͭ ͍ͯ • Apple͕ۭؒϏσΦͷͨΊʹ௥Ճ͠ ֦ͨு: vexu(Video Extended Usage)
8. APMPʹରԠ͢ΔͱͲ͏ͳΔͷ͔ʁ • visonOSͷඪ४ͷࣸਅΞϓϦ΍ࣗ෼ͷΞϓϦ͔ΒΠϚʔγϒ ͳಈըͷ࠶ੜ͕Ͱ͖ΔΑ͏ʹͳΔ
9. APMPܗࣜͷಈըͷ࡞Γํ 1. CLIπʔϧʢavconvertʣ·ͨ͸macOSʢbetaʣͷػೳΛ࢖ͬ ͯม׵͢Δ 2. طଘͷҰ෦ͷΧϝϥͰ࿥ͬͨಈը͸ࣗಈͰม׵Ͱ͖Δ 3. APIΛ࢖ͬͯม׵͢Δ(Appleͷαϯϓϧίʔυ͋Γ)
10. 1. avconvert
11. 2. ࣗಈม׵ 1. ·ͣInsta360͔Go ProΛߪೖ͠·͠ΐ͏ 2. Insta360 StudioͰMP4΁ग़ྗ͠·͢ 3. AirDropͰAVPʢvisionOS 26 betaʣ΁ૹΔ 4. visionOSͰ౰֘ϑΝΠϧΛ։͘ࡍʹม׵͢Δ͔Ͳ͏͔μΠΞ ϩά͕ग़ͯ͘ΔͷͰબͿ
12. 3. APIΛ࢖ͬͯม׵͢Δ AppleͷαϯϓϧίʔυProjectedMediaConversionΛ࢖༻͠· ͢ https://developer.apple.com/documentation/AVFoundation/ converting-projected-video-to-apple-projected-media-profile
13. ॲཧͷྲྀΕ 1. ݩಈըಡΈࠐΈ 2. ύοΩϯάઃఆ 3. ѹॖϓϩύςΟઃఆ 4. Τϯίʔμʔͷઃఆ 5. ॻ͖ग़͠
14. ύοΩϯά ཱମࢹʢstereoscopicʣͷ৔߹͸ύοΩ ϯά͕ҎԼͷ͍ͣΕ͔Ͱ͋Δ͔Λࢦఆ͠ ·͢ • SideBySide • OverUnder
15. ύοΩϯά if let viewPackingKind = projectedMediaMetadata.viewPackingKind { isFramePacked = true if viewPackingKind.caseInsensitiveCompare("SideBySide") == .orderedSame { horizontalScale = 2.0 isSideBySide = true } else if viewPackingKind.caseInsensitiveCompare("OverUnder") == .orderedSame { verticalScale = 2.0 } } let eyeFrameSize = CGSize(width: sourceVideoFrameSize.width / horizontalScale, height: sourceVideoFrameSize.height / verticalScale) ... let cropRectDict = [ kCVImageBufferCleanApertureHorizontalOffsetKey: apertureHorizontalOffset, kCVImageBufferCleanApertureVerticalOffsetKey: apertureVerticalOffset, kCVImageBufferCleanApertureWidthKey: eyeFrameSize.width, kCVImageBufferCleanApertureHeightKey: eyeFrameSize.height ]
16. ѹॖϓϩύςΟͷઃఆ let MVHEVCVideoLayerIDs = [0, 1] let MVHEVCViewIDs = [0, 1] let MVHEVCLeftAndRightViewIDs = [0, 1] ... let stereoCompressionProperties: [CFString: Any] = [ kVTCompressionPropertyKey_MVHEVCVideoLayerIDs: MVHEVCVideoLayerIDs, kVTCompressionPropertyKey_MVHEVCViewIDs: MVHEVCViewIDs, kVTCompressionPropertyKey_MVHEVCLeftAndRightViewIDs: MVHEVCLeftAndRightViewIDs, kVTCompressionPropertyKey_HasLeftStereoEyeView: true, kVTCompressionPropertyKey_HasRightStereoEyeView: true ]
17. ѹॖϓϩύςΟͷઃఆʢ2ʣ let projectionKind = projectedMediaMetadata.projectionKind ... compressionProperties[kVTCompressionPropertyKey_ProjectionKind] = kCMFormatDescriptionProjectionKind_HalfEquirectangular ... let baselineInMicrometers = UInt32(1000.0 * baselineInMillimeters) compressionProperties[kVTCompressionPropertyKey_StereoCameraBaseline] = baselineInMicrometers ... let encodedHorizontalFOV = UInt32(1000.0 * horizontalFOV) compressionProperties[kVTCompressionPropertyKey_HorizontalFieldOfView] = encodedHorizontalFOV
18. Τϯίʔμʔͷઃఆ let outputSettings: [String: Any] = [ AVVideoCodecKey: AVVideoCodecType.hevc, AVVideoWidthKey: eyeFrameSize.width, AVVideoHeightKey: eyeFrameSize.height, AVVideoCompressionPropertiesKey: compressionProperties ]
19. ॻ͖ग़͠ for (layerID, eye) in zip(MVHEVCVideoLayerIDs, eyes) { let pixelBuffer = try pixelBufferPool.makeMutablePixelBuffer() ... ੾ΓऔΓൣғࢉग़ॲཧʢলུʣ ... CVBufferSetAttachment(imageBuffer, kCVImageBufferCleanApertureKey, cropRectDict as CFDictionary, CVAttachmentMode.shouldPropagate) VTSessionSetProperty(session, key: kVTPixelTransferPropertyKey_ScalingMode, value: kVTScalingMode_CropSourceToCleanAperture) pixelBuffer.withUnsafeBuffer { cvPixelBuffer in guard VTPixelTransferSessionTransferImage(session, from: imageBuffer, to: cvPixelBuffer) == noErr else { fatalError("Error during pixel transfer session for layer \(layerID)") } } // Create and append a tagged buffer for this eye. let tags: [CMTag] = [.videoLayerID(Int64(layerID)), .stereoView(eye)] taggedBuffers.append(.init(tags: tags, content: .pixelBuffer(.init(pixelBuffer)))) }
20. ॻ͖ग़͠ // Create and append a tagged buffer for this eye. let tags: [CMTag] = [.videoLayerID(Int64(layerID)), .stereoView(eye)] taggedBuffers.append(.init(tags: tags, content: .pixelBuffer(.init(pixelBuffer))))
21. ग़དྷ্͕Γʂ
22. !
23. ࣗݾ঺հ ۄ৓৴ޛ iOS DeveloperʢϑϦʔϥϯεʣ ԭೄࡏॅ ίϛϡχςΟ׆ಈ - try! Swift TokyoӡӦ - ԭೄϞόΠϧΞϓϦ։ൃษڧձӡӦ
24. Okinawa.swiftͲ͏Ͱ͠ΐ͏ʁ
25. Ҏ্