PiP で実現するミラティブの配信コメントバー / pip-streaming-comment-bar

Transcript

1. 2022.3.15 - iOS Tech Talk
   Naruki Chigira
   @_naru_jpn / Horizon Forbidden West ʹϋϚͬͯ·͢
   PiP Ͱ࣮ݱ͢ΔϛϥςΟϒͷ഑৴ίϝϯτόʔ
   

   View Slide

2. 1. PiP ͱ഑৴ίϝϯτόʔ
   2. iOS15 ͱ഑৴ίϝϯτόʔ
   3. ഑৴ίϝϯτόʔͷඳը
   

   View Slide

3. 1. PiP ͱ഑৴ίϝϯτόʔ
   

   View Slide

4. PiPʢϐΫνϟɾΠϯɾϐΫνϟʣ
   ϗʔϜը໘΍ΞϓϦը໘্ͰಈըΛ࠶ੜͰ͖Δػೳ
   

   View Slide

5. ഑৴ίϝϯτόʔ
   ϗʔϜը໘΍ΞϓϦը໘্ͰϛϥςΟϒ഑৴ͷίϝϯτͳͲΛ͓ಧ͚͢Δػೳ
   

   View Slide

6. ࣮ػ্Ͱͷݟ͑ํ
   

   View Slide

7. ഑৴ίϝϯτόʔ
   ϗʔϜը໘΍ΞϓϦը໘্ͰϛϥςΟϒ഑৴ͷίϝϯτͳͲΛ͓ಧ͚͢Δػೳ
   ࢹௌऀ͔Βͷίϝϯτ
   ഑৴࣌ؒ
   ࢹௌऀ਺
   ϚΠΫON/OFF
   ετοΫ͞ΕͨΪϑτͷ਺
   ഑৴ऀͷΤϞϞ
   

   View Slide

8. 2. iOS15 ͱ഑৴ίϝϯτόʔ
   

   View Slide

9. iOS 15 Ҏલ
   ɾAVPictureInPictureController ͸ iOS 9.0 ͔Β͋Δ
   ɾAVPictureInPictureController ͸ PiP ͷ࣮૷ʹ࢖͏Ϋϥε
   ɾAVPlayerLayer ্Ͱ࠶ੜ͢Δಈըʹ͔͠ରԠ͍ͯ͠ͳ͔ͬͨ
   

   View Slide

10. iOS 15 ͰͰ͖ΔΑ͏ʹͳͬͨ͜ͱ
    ɾiOS 15.0 Ͱ AVPictureInPictureController ͷػೳ͕֦ு͞Εͯ
    ɹAVSampleBufferDisplayLayer ʹ΋ରԠͨ͠ʢ2021೥7݄ʣ
    ɾϐΫνϟɾΠϯɾϐΫνϟͷࣗ༝౓͕૿ͨ͠
    ɾࣾ಺Ͱ R&D ͷػӡ͕ߴ·͖ͬͯͨʢ2021೥10݄ॳ಄ʣ
    ɾ഑৴ίϝϯτόʔͷࢼ࡞඼͕Ͱ͖͸͡Ίͨʢ2021೥10݄຤ʣ
    WWDC2021 What's new in AVKit
    

    View Slide

11. iOS 15.1 Ͱ੍ݶ͞Εͨ͜ͱ
    ɾը໘ڞ༗தͷϓογϡ௨஌Λ੍ݶ͢Δઃఆ͕૿͑ͨʢσϑΥONʣ
    ɾ഑৴த͸ը໘ڞ༗தͱͯ͠ѻΘΕΔͷͰϓογϡ௨஌੍͕ݶ͞ΕΔ
    ɾϛϥςΟϒ͸ίϝϯτΛϓογϡ௨஌Ͱಧ͚͍ͯΔͷͰࠔΔʢ2021೥10݄຤ʣ
    ɾ഑৴ίϝϯτόʔ͕࣮ݱͰ͖Ε͹ϓογϡ௨஌ͷ੍໿Λड͚ͣʹࡁΉ
    

    View Slide

12. ഑৴ίϝϯτόʔͷ։ൃ
    7/1
    iOS15 public β
    10݄ॳ಄ 10݄຤ 11/25
    R&D
    ࢼ࡞
    ௨஌੍ݶ ػೳϦϦʔε
    ຊ֨։ൃ!
    

    View Slide

13. 3. ഑৴ίϝϯτόʔͷ࣮૷
    

    View Slide

14. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ
    CMSampleBuffer
    ɾϝσΟΞσʔλʢө૾΍Ի੠ʣΛอ͍࣋ͯ͠ΔΦϒδΣΫτ
    

    View Slide

15. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ
    AVSampleBufferDisplayLayer
    ɾCMSampleBuffer Λ౉͢ͱදࣔͯ͘͠ΕΔ
    sampleBufferDisplayLayer.enqueue(sampleBuffer)
    

    View Slide

16. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ
    AVPictureInPictureController
    ɾϐΫνϟɾΠϯɾϐΫνϟͷ؅ཧΛ͢Δ
    // ද͍ࣔͨ͠ AVSampleBufferDisplayLayer Λࢦఆͯ͠ੜ੒
    
    
    let controller = AVPictureInPictureController(
    
    
    contentSource: .init(
    
    
    sampleBufferDisplayLayer: sampleBufferDisplayLayer,
    
    
    playbackDelegate: self
    
    
    )
    
    
    )
    

    View Slide

17. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ
    AVPictureInPictureController
    ɾϐΫνϟɾΠϯɾϐΫνϟͷ؅ཧΛ͢Δ
    // ϐΫνϟɾΠϯɾϐΫνϟͷ։࢝
    
    
    controller.startPictureInPicture()
    
    
    // ϐΫνϟɾΠϯɾϐΫνϟͷఀࢭ
    
    
    controller.stopPictureInPicture()
    

    View Slide

18. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ
    AVPictureInPictureController ͕؅ཧ
    AVSampleBufferDisplayLayer ͷඳը಺༰
    enqueue(sampleBuffer)
    

    View Slide

19. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ
    AVPictureInPictureController ͕؅ཧ
    AVSampleBufferDisplayLayer ͷඳը಺༰
    enqueue(sampleBuffer)
    

    View Slide

20. CMSampleBuffer ͷѻ͍
    ɾCMSampleBuffer ΛͲͷ͘Β͍ͷස౓Ͱߋ৽͢Δ͔ʢFPSʣ
    ɾCMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔
    ্ه2఺͕ΞϓϦ΁ͷෛՙʹେ͖͘ӨڹΛ༩͑Δ
    ϛϥςΟϒͷ৔߹ɺΞϓϦ্Ͱ Unity ͕ಈ࡞͍ͯ͠Δঢ়گͰ΋ϐΫνϟɾΠϯɾϐ
    ΫνϟΛಉ࣌ʹಈ࡞ͤ͞Δඞཁ͕͋ΓɺύϑΥʔϚϯεΛؾʹ͢Δඞཁ͕͋Δ
    

    View Slide

21. CMSampleBuffer ΛͲͷ͘Β͍ͷස౓Ͱߋ৽͢Δ͔
    ɾϛϥςΟϒͰ͸1ඵ͋ͨΓ1ճඳը಺༰Λߋ৽͍ͯ͠Δʢ1FPSʣ
    ɾ഑৴࣌ؒ͸ඵ୯ҐͰද͍ࣔͯ͠Δ
    ɾίϝϯτ΋ߴස౓Ͱߋ৽͞ΕΔඞཁ͸ͳ͍
    

    View Slide

22. CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔
    ɾCMSampleBuffer ͸৭ʑͳํ๏Ͱੜ੒Ͱ͖Δ
    ɾϏοτϚοϓ͔Βੜ੒͢ΔɺUIImage, CGImage, CIImage ͔Βੜ੒͢Δɺ౳…
    CVPixelBuffer ܦ༝ͰϏοτϚοϓΛฤूͨ͠ࢼ࡞඼
    

    View Slide

23. CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔
    ɾϛϥςΟϒͰ͸ UIView ͷඳը಺༰Λ CMSampleBuffer ʹॻ͖ࠐΜͰ͍Δ
    ɾ࡞Γ͍ͨ UI ͕ΞϓϦతͰɺAutoLayout ΋׆༻͔ͨͬͨ͠
    ɾίʔυϨΠΞ΢τͰ UI Λ࡞੒͠ɺCMSampleBuffer ʹඳը͢Δ
    

    View Slide

24. CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔
    ɾϛϥςΟϒͰ͸ UIKit ͷඳը݁ՌΛ CMSampleBuffer ʹॻ͖ࠐΜͰ͍Δ
    let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
    
    
    let pixelBuffer = imageBuffer as CVPixelBuffer
    
    
    let renderer = UIGraphicsImageRenderer(size: renderingSize)
    
    
    let image = renderer.image { context in
    
    
    view.layer.draw(in: context)
    
    
    }
    
    
    if let cgImage = image.cgImage {
    
    
    cicontext.render(CIImage(cgImage: cgImage), to: pixelBuffer)
    
    
    }
    ɾUIView → UIImage → CVPixelBuffer → CMSampleBuffer
    

    View Slide

25. CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔
    ɾඳըपΓͷύϑΥʔϚϯεͷվળΊͪΌؤுͬͨ
    ɾ౰ॳ 50ms ͔͔͍ͬͯͨඳըॲཧ͕ 5ms ʹʂ
    ߋ৽ස౓ͷগͳ͍෦෼(ࠨ)͸͋Β͔͡Ίඳըͯ͠1ຕͷը૾ʹͯ͠Ωϟογϡ͍ͯ͠Δ
    ഑৴ίϝϯτόʔ ʙ PiP ඳըύϑΥʔϚϯεͱͷ޲͖߹͍ํ
    

    View Slide

26. ଞʹ΋͍Ζ͍Ζ͋Δ͚Ͳ…
    ɾAVAudioSession ͱ PiP ͷؔ܎
    ɾ഑৴ϓϩηεͱ഑৴ίϝϯτόʔͷಉظ
    ɾAVPictureInPictureController ͷ᠘
    ɾR&D ͷ։ൃͰֶͼʹͳͬͨ͜ͱ
    ɾiOS15 Ҏલͷ PiP
    ׂѪ͠·͢ʂ
    

    View Slide

27. ͋Γ͕ͱ͏͍͟͝·ͨ͠
    

    View Slide