2022.3.15 - iOS Tech Talk Naruki Chigira @_naru_jpn / Horizon Forbidden West ʹϋϚͬͯ·͢ PiP Ͱ࣮ݱ͢ΔϛϥςΟϒͷ഑৴ίϝϯτόʔ

1. PiP ͱ഑৴ίϝϯτόʔ 2. iOS15 ͱ഑৴ίϝϯτόʔ 3. ഑৴ίϝϯτόʔͷඳը

1. PiP ͱ഑৴ίϝϯτόʔ

PiPʢϐΫνϟɾΠϯɾϐΫνϟʣ ϗʔϜը໘΍ΞϓϦը໘্ͰಈըΛ࠶ੜͰ͖Δػೳ

഑৴ίϝϯτόʔ ϗʔϜը໘΍ΞϓϦը໘্ͰϛϥςΟϒ഑৴ͷίϝϯτͳͲΛ͓ಧ͚͢Δػೳ

࣮ػ্Ͱͷݟ͑ํ

഑৴ίϝϯτόʔ ϗʔϜը໘΍ΞϓϦը໘্ͰϛϥςΟϒ഑৴ͷίϝϯτͳͲΛ͓ಧ͚͢Δػೳ ࢹௌऀ͔Βͷίϝϯτ ഑৴࣌ؒ ࢹௌऀ਺ ϚΠΫON/OFF ετοΫ͞ΕͨΪϑτͷ਺ ഑৴ऀͷΤϞϞ

2. iOS15 ͱ഑৴ίϝϯτόʔ

iOS 15 Ҏલ ɾAVPictureInPictureController ͸ iOS 9.0 ͔Β͋Δ ɾAVPictureInPictureController ͸ PiP ͷ࣮૷ʹ࢖͏Ϋϥε ɾAVPlayerLayer ্Ͱ࠶ੜ͢Δಈըʹ͔͠ରԠ͍ͯ͠ͳ͔ͬͨ

iOS 15 ͰͰ͖ΔΑ͏ʹͳͬͨ͜ͱ ɾiOS 15.0 Ͱ AVPictureInPictureController ͷػೳ͕֦ு͞Εͯ ɹAVSampleBufferDisplayLayer ʹ΋ରԠͨ͠ʢ2021೥7݄ʣ ɾϐΫνϟɾΠϯɾϐΫνϟͷࣗ༝౓͕૿ͨ͠ ɾࣾ಺Ͱ R&D ͷػӡ͕ߴ·͖ͬͯͨʢ2021೥10݄ॳ಄ʣ ɾ഑৴ίϝϯτόʔͷࢼ࡞඼͕Ͱ͖͸͡Ίͨʢ2021೥10݄຤ʣ WWDC2021 What's new in AVKit

iOS 15.1 Ͱ੍ݶ͞Εͨ͜ͱ ɾը໘ڞ༗தͷϓογϡ௨஌Λ੍ݶ͢Δઃఆ͕૿͑ͨʢσϑΥONʣ ɾ഑৴த͸ը໘ڞ༗தͱͯ͠ѻΘΕΔͷͰϓογϡ௨஌੍͕ݶ͞ΕΔ ɾϛϥςΟϒ͸ίϝϯτΛϓογϡ௨஌Ͱಧ͚͍ͯΔͷͰࠔΔʢ2021೥10݄຤ʣ ɾ഑৴ίϝϯτόʔ͕࣮ݱͰ͖Ε͹ϓογϡ௨஌ͷ੍໿Λड͚ͣʹࡁΉ

഑৴ίϝϯτόʔͷ։ൃ 7/1 iOS15 public β 10݄ॳ಄ 10݄຤ 11/25 R&D ࢼ࡞ ௨஌੍ݶ ػೳϦϦʔε ຊ֨։ൃ!

3. ഑৴ίϝϯτόʔͷ࣮૷

ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ CMSampleBuffer ɾϝσΟΞσʔλʢө૾΍Ի੠ʣΛอ͍࣋ͯ͠ΔΦϒδΣΫτ

ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ AVSampleBufferDisplayLayer ɾCMSampleBuffer Λ౉͢ͱදࣔͯ͘͠ΕΔ sampleBufferDisplayLayer.enqueue(sampleBuffer)

ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ AVPictureInPictureController ɾϐΫνϟɾΠϯɾϐΫνϟͷ؅ཧΛ͢Δ // ද͍ࣔͨ͠ AVSampleBufferDisplayLayer Λࢦఆͯ͠ੜ੒ let controller = AVPictureInPictureController( contentSource: .init( sampleBufferDisplayLayer: sampleBufferDisplayLayer, playbackDelegate: self ) )

ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ AVPictureInPictureController ɾϐΫνϟɾΠϯɾϐΫνϟͷ؅ཧΛ͢Δ // ϐΫνϟɾΠϯɾϐΫνϟͷ։࢝ controller.startPictureInPicture() // ϐΫνϟɾΠϯɾϐΫνϟͷఀࢭ controller.stopPictureInPicture()

ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ AVPictureInPictureController ͕؅ཧ AVSampleBufferDisplayLayer ͷඳը಺༰ enqueue(sampleBuffer)

ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ AVPictureInPictureController ͕؅ཧ AVSampleBufferDisplayLayer ͷඳը಺༰ enqueue(sampleBuffer)

CMSampleBuffer ͷѻ͍ ɾCMSampleBuffer ΛͲͷ͘Β͍ͷස౓Ͱߋ৽͢Δ͔ʢFPSʣ ɾCMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔ ্ه2఺͕ΞϓϦ΁ͷෛՙʹେ͖͘ӨڹΛ༩͑Δ ϛϥςΟϒͷ৔߹ɺΞϓϦ্Ͱ Unity ͕ಈ࡞͍ͯ͠Δঢ়گͰ΋ϐΫνϟɾΠϯɾϐ ΫνϟΛಉ࣌ʹಈ࡞ͤ͞Δඞཁ͕͋ΓɺύϑΥʔϚϯεΛؾʹ͢Δඞཁ͕͋Δ

CMSampleBuffer ΛͲͷ͘Β͍ͷස౓Ͱߋ৽͢Δ͔ ɾϛϥςΟϒͰ͸1ඵ͋ͨΓ1ճඳը಺༰Λߋ৽͍ͯ͠Δʢ1FPSʣ ɾ഑৴࣌ؒ͸ඵ୯ҐͰද͍ࣔͯ͠Δ ɾίϝϯτ΋ߴස౓Ͱߋ৽͞ΕΔඞཁ͸ͳ͍

CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔ ɾCMSampleBuffer ͸৭ʑͳํ๏Ͱੜ੒Ͱ͖Δ ɾϏοτϚοϓ͔Βੜ੒͢ΔɺUIImage, CGImage, CIImage ͔Βੜ੒͢Δɺ౳… CVPixelBuffer ܦ༝ͰϏοτϚοϓΛฤूͨ͠ࢼ࡞඼

CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔ ɾϛϥςΟϒͰ͸ UIView ͷඳը಺༰Λ CMSampleBuffer ʹॻ͖ࠐΜͰ͍Δ ɾ࡞Γ͍ͨ UI ͕ΞϓϦతͰɺAutoLayout ΋׆༻͔ͨͬͨ͠ ɾίʔυϨΠΞ΢τͰ UI Λ࡞੒͠ɺCMSampleBuffer ʹඳը͢Δ

CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔ ɾϛϥςΟϒͰ͸ UIKit ͷඳը݁ՌΛ CMSampleBuffer ʹॻ͖ࠐΜͰ͍Δ let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)! let pixelBuffer = imageBuffer as CVPixelBuffer let renderer = UIGraphicsImageRenderer(size: renderingSize) let image = renderer.image { context in view.layer.draw(in: context) } if let cgImage = image.cgImage { cicontext.render(CIImage(cgImage: cgImage), to: pixelBuffer) } ɾUIView → UIImage → CVPixelBuffer → CMSampleBuffer

CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔ ɾඳըपΓͷύϑΥʔϚϯεͷվળΊͪΌؤுͬͨ ɾ౰ॳ 50ms ͔͔͍ͬͯͨඳըॲཧ͕ 5ms ʹʂ ߋ৽ස౓ͷগͳ͍෦෼(ࠨ)͸͋Β͔͡Ίඳըͯ͠1ຕͷը૾ʹͯ͠Ωϟογϡ͍ͯ͠Δ ഑৴ίϝϯτόʔ ʙ PiP ඳըύϑΥʔϚϯεͱͷ޲͖߹͍ํ

ଞʹ΋͍Ζ͍Ζ͋Δ͚Ͳ… ɾAVAudioSession ͱ PiP ͷؔ܎ ɾ഑৴ϓϩηεͱ഑৴ίϝϯτόʔͷಉظ ɾAVPictureInPictureController ͷ᠘ ɾR&D ͷ։ൃͰֶͼʹͳͬͨ͜ͱ ɾiOS15 Ҏલͷ PiP ׂѪ͠·͢ʂ

͋Γ͕ͱ͏͍͟͝·ͨ͠