PiP で実現するミラティブの配信コメントバー / pip-streaming-comment-bar

Transcript

1. 2022.3.15 - iOS Tech Talk Naruki Chigira @_naru_jpn / Horizon Forbidden West ʹϋϚͬͯ·͢ PiP Ͱ࣮ݱ͢ΔϛϥςΟϒͷ഑৴ίϝϯτόʔ
2. 1. PiP ͱ഑৴ίϝϯτόʔ 2. iOS15 ͱ഑৴ίϝϯτόʔ 3. ഑৴ίϝϯτόʔͷඳը
3. 1. PiP ͱ഑৴ίϝϯτόʔ
4. PiPʢϐΫνϟɾΠϯɾϐΫνϟʣ ϗʔϜը໘΍ΞϓϦը໘্ͰಈըΛ࠶ੜͰ͖Δػೳ
5. ഑৴ίϝϯτόʔ ϗʔϜը໘΍ΞϓϦը໘্ͰϛϥςΟϒ഑৴ͷίϝϯτͳͲΛ͓ಧ͚͢Δػೳ
6. ࣮ػ্Ͱͷݟ͑ํ
7. ഑৴ίϝϯτόʔ ϗʔϜը໘΍ΞϓϦը໘্ͰϛϥςΟϒ഑৴ͷίϝϯτͳͲΛ͓ಧ͚͢Δػೳ ࢹௌऀ͔Βͷίϝϯτ ഑৴࣌ؒ ࢹௌऀ਺ ϚΠΫON/OFF ετοΫ͞ΕͨΪϑτͷ਺ ഑৴ऀͷΤϞϞ
8. 2. iOS15 ͱ഑৴ίϝϯτόʔ
9. iOS 15 Ҏલ ɾAVPictureInPictureController ͸ iOS 9.0 ͔Β͋Δ ɾAVPictureInPictureController ͸ PiP ͷ࣮૷ʹ࢖͏Ϋϥε ɾAVPlayerLayer ্Ͱ࠶ੜ͢Δಈըʹ͔͠ରԠ͍ͯ͠ͳ͔ͬͨ
10. iOS 15 ͰͰ͖ΔΑ͏ʹͳͬͨ͜ͱ ɾiOS 15.0 Ͱ AVPictureInPictureController ͷػೳ͕֦ு͞Εͯ ɹAVSampleBufferDisplayLayer ʹ΋ରԠͨ͠ʢ2021೥7݄ʣ ɾϐΫνϟɾΠϯɾϐΫνϟͷࣗ༝౓͕૿ͨ͠ ɾࣾ಺Ͱ R&D ͷػӡ͕ߴ·͖ͬͯͨʢ2021೥10݄ॳ಄ʣ ɾ഑৴ίϝϯτόʔͷࢼ࡞඼͕Ͱ͖͸͡Ίͨʢ2021೥10݄຤ʣ WWDC2021 What's new in AVKit
11. iOS 15.1 Ͱ੍ݶ͞Εͨ͜ͱ ɾը໘ڞ༗தͷϓογϡ௨஌Λ੍ݶ͢Δઃఆ͕૿͑ͨʢσϑΥONʣ ɾ഑৴த͸ը໘ڞ༗தͱͯ͠ѻΘΕΔͷͰϓογϡ௨஌੍͕ݶ͞ΕΔ ɾϛϥςΟϒ͸ίϝϯτΛϓογϡ௨஌Ͱಧ͚͍ͯΔͷͰࠔΔʢ2021೥10݄຤ʣ ɾ഑৴ίϝϯτόʔ͕࣮ݱͰ͖Ε͹ϓογϡ௨஌ͷ੍໿Λड͚ͣʹࡁΉ
12. ഑৴ίϝϯτόʔͷ։ൃ 7/1 iOS15 public β 10݄ॳ಄ 10݄຤ 11/25 R&D ࢼ࡞ ௨஌੍ݶ ػೳϦϦʔε ຊ֨։ൃ!
13. 3. ഑৴ίϝϯτόʔͷ࣮૷
14. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ CMSampleBuffer ɾϝσΟΞσʔλʢө૾΍Ի੠ʣΛอ͍࣋ͯ͠ΔΦϒδΣΫτ
15. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ AVSampleBufferDisplayLayer ɾCMSampleBuffer Λ౉͢ͱදࣔͯ͘͠ΕΔ sampleBufferDisplayLayer.enqueue(sampleBuffer)
16. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ AVPictureInPictureController ɾϐΫνϟɾΠϯɾϐΫνϟͷ؅ཧΛ͢Δ // ද͍ࣔͨ͠ AVSampleBufferDisplayLayer Λࢦఆͯ͠ੜ੒ let controller = AVPictureInPictureController( contentSource: .init( sampleBufferDisplayLayer: sampleBufferDisplayLayer, playbackDelegate: self ) )
17. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ AVPictureInPictureController ɾϐΫνϟɾΠϯɾϐΫνϟͷ؅ཧΛ͢Δ // ϐΫνϟɾΠϯɾϐΫνϟͷ։࢝ controller.startPictureInPicture() // ϐΫνϟɾΠϯɾϐΫνϟͷఀࢭ controller.stopPictureInPicture()
18. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ AVPictureInPictureController ͕؅ཧ AVSampleBufferDisplayLayer ͷඳը಺༰ enqueue(sampleBuffer)
19. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ AVPictureInPictureController ͕؅ཧ AVSampleBufferDisplayLayer ͷඳը಺༰ enqueue(sampleBuffer)
20. CMSampleBuffer ͷѻ͍ ɾCMSampleBuffer ΛͲͷ͘Β͍ͷස౓Ͱߋ৽͢Δ͔ʢFPSʣ ɾCMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔ ্ه2఺͕ΞϓϦ΁ͷෛՙʹେ͖͘ӨڹΛ༩͑Δ ϛϥςΟϒͷ৔߹ɺΞϓϦ্Ͱ Unity ͕ಈ࡞͍ͯ͠Δঢ়گͰ΋ϐΫνϟɾΠϯɾϐ ΫνϟΛಉ࣌ʹಈ࡞ͤ͞Δඞཁ͕͋ΓɺύϑΥʔϚϯεΛؾʹ͢Δඞཁ͕͋Δ
21. CMSampleBuffer ΛͲͷ͘Β͍ͷස౓Ͱߋ৽͢Δ͔ ɾϛϥςΟϒͰ͸1ඵ͋ͨΓ1ճඳը಺༰Λߋ৽͍ͯ͠Δʢ1FPSʣ ɾ഑৴࣌ؒ͸ඵ୯ҐͰද͍ࣔͯ͠Δ ɾίϝϯτ΋ߴස౓Ͱߋ৽͞ΕΔඞཁ͸ͳ͍
22. CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔ ɾCMSampleBuffer ͸৭ʑͳํ๏Ͱੜ੒Ͱ͖Δ ɾϏοτϚοϓ͔Βੜ੒͢ΔɺUIImage, CGImage, CIImage ͔Βੜ੒͢Δɺ౳… CVPixelBuffer ܦ༝ͰϏοτϚοϓΛฤूͨ͠ࢼ࡞඼
23. CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔ ɾϛϥςΟϒͰ͸ UIView ͷඳը಺༰Λ CMSampleBuffer ʹॻ͖ࠐΜͰ͍Δ ɾ࡞Γ͍ͨ UI ͕ΞϓϦతͰɺAutoLayout ΋׆༻͔ͨͬͨ͠ ɾίʔυϨΠΞ΢τͰ UI Λ࡞੒͠ɺCMSampleBuffer ʹඳը͢Δ
24. CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔ ɾϛϥςΟϒͰ͸ UIKit ͷඳը݁ՌΛ CMSampleBuffer ʹॻ͖ࠐΜͰ͍Δ let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)! let pixelBuffer = imageBuffer as CVPixelBuffer let renderer = UIGraphicsImageRenderer(size: renderingSize) let image = renderer.image { context in view.layer.draw(in: context) } if let cgImage = image.cgImage { cicontext.render(CIImage(cgImage: cgImage), to: pixelBuffer) } ɾUIView → UIImage → CVPixelBuffer → CMSampleBuffer
25. CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔ ɾඳըपΓͷύϑΥʔϚϯεͷվળΊͪΌؤுͬͨ ɾ౰ॳ 50ms ͔͔͍ͬͯͨඳըॲཧ͕ 5ms ʹʂ ߋ৽ස౓ͷগͳ͍෦෼(ࠨ)͸͋Β͔͡Ίඳըͯ͠1ຕͷը૾ʹͯ͠Ωϟογϡ͍ͯ͠Δ ഑৴ίϝϯτόʔ ʙ PiP ඳըύϑΥʔϚϯεͱͷ޲͖߹͍ํ
26. ଞʹ΋͍Ζ͍Ζ͋Δ͚Ͳ… ɾAVAudioSession ͱ PiP ͷؔ܎ ɾ഑৴ϓϩηεͱ഑৴ίϝϯτόʔͷಉظ ɾAVPictureInPictureController ͷ᠘ ɾR&D ͷ։ൃͰֶͼʹͳͬͨ͜ͱ ɾiOS15 Ҏલͷ PiP ׂѪ͠·͢ʂ
27. ͋Γ͕ͱ͏͍͟͝·ͨ͠