Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PiP で実現するミラティブの配信コメントバー / pip-streaming-comment-bar

PiP で実現するミラティブの配信コメントバー / pip-streaming-comment-bar

naru-jpn

March 15, 2022
Tweet

More Decks by naru-jpn

Other Decks in Programming

Transcript

  1. 2022.3.15 - iOS Tech Talk
    Naruki Chigira
    @_naru_jpn / Horizon Forbidden West ʹϋϚͬͯ·͢
    PiP Ͱ࣮ݱ͢ΔϛϥςΟϒͷ഑৴ίϝϯτόʔ

    View full-size slide

  2. 1. PiP ͱ഑৴ίϝϯτόʔ
    2. iOS15 ͱ഑৴ίϝϯτόʔ
    3. ഑৴ίϝϯτόʔͷඳը

    View full-size slide

  3. 1. PiP ͱ഑৴ίϝϯτόʔ

    View full-size slide

  4. PiPʢϐΫνϟɾΠϯɾϐΫνϟʣ
    ϗʔϜը໘΍ΞϓϦը໘্ͰಈըΛ࠶ੜͰ͖Δػೳ

    View full-size slide

  5. ഑৴ίϝϯτόʔ
    ϗʔϜը໘΍ΞϓϦը໘্ͰϛϥςΟϒ഑৴ͷίϝϯτͳͲΛ͓ಧ͚͢Δػೳ

    View full-size slide

  6. ࣮ػ্Ͱͷݟ͑ํ

    View full-size slide

  7. ഑৴ίϝϯτόʔ
    ϗʔϜը໘΍ΞϓϦը໘্ͰϛϥςΟϒ഑৴ͷίϝϯτͳͲΛ͓ಧ͚͢Δػೳ
    ࢹௌऀ͔Βͷίϝϯτ
    ഑৴࣌ؒ
    ࢹௌऀ਺
    ϚΠΫON/OFF
    ετοΫ͞ΕͨΪϑτͷ਺
    ഑৴ऀͷΤϞϞ

    View full-size slide

  8. 2. iOS15 ͱ഑৴ίϝϯτόʔ

    View full-size slide

  9. iOS 15 Ҏલ
    ɾAVPictureInPictureController ͸ iOS 9.0 ͔Β͋Δ
    ɾAVPictureInPictureController ͸ PiP ͷ࣮૷ʹ࢖͏Ϋϥε
    ɾAVPlayerLayer ্Ͱ࠶ੜ͢Δಈըʹ͔͠ରԠ͍ͯ͠ͳ͔ͬͨ

    View full-size slide

  10. iOS 15 ͰͰ͖ΔΑ͏ʹͳͬͨ͜ͱ
    ɾiOS 15.0 Ͱ AVPictureInPictureController ͷػೳ͕֦ு͞Εͯ
    ɹAVSampleBufferDisplayLayer ʹ΋ରԠͨ͠ʢ2021೥7݄ʣ
    ɾϐΫνϟɾΠϯɾϐΫνϟͷࣗ༝౓͕૿ͨ͠
    ɾࣾ಺Ͱ R&D ͷػӡ͕ߴ·͖ͬͯͨʢ2021೥10݄ॳ಄ʣ
    ɾ഑৴ίϝϯτόʔͷࢼ࡞඼͕Ͱ͖͸͡Ίͨʢ2021೥10݄຤ʣ
    WWDC2021 What's new in AVKit

    View full-size slide

  11. iOS 15.1 Ͱ੍ݶ͞Εͨ͜ͱ
    ɾը໘ڞ༗தͷϓογϡ௨஌Λ੍ݶ͢Δઃఆ͕૿͑ͨʢσϑΥONʣ
    ɾ഑৴த͸ը໘ڞ༗தͱͯ͠ѻΘΕΔͷͰϓογϡ௨஌੍͕ݶ͞ΕΔ
    ɾϛϥςΟϒ͸ίϝϯτΛϓογϡ௨஌Ͱಧ͚͍ͯΔͷͰࠔΔʢ2021೥10݄຤ʣ
    ɾ഑৴ίϝϯτόʔ͕࣮ݱͰ͖Ε͹ϓογϡ௨஌ͷ੍໿Λड͚ͣʹࡁΉ

    View full-size slide

  12. ഑৴ίϝϯτόʔͷ։ൃ
    7/1
    iOS15 public β
    10݄ॳ಄ 10݄຤ 11/25
    R&D
    ࢼ࡞
    ௨஌੍ݶ ػೳϦϦʔε
    ຊ֨։ൃ!

    View full-size slide

  13. 3. ഑৴ίϝϯτόʔͷ࣮૷

    View full-size slide

  14. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ
    CMSampleBuffer
    ɾϝσΟΞσʔλʢө૾΍Ի੠ʣΛอ͍࣋ͯ͠ΔΦϒδΣΫτ

    View full-size slide

  15. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ
    AVSampleBufferDisplayLayer
    ɾCMSampleBuffer Λ౉͢ͱදࣔͯ͘͠ΕΔ
    sampleBufferDisplayLayer.enqueue(sampleBuffer)

    View full-size slide

  16. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ
    AVPictureInPictureController
    ɾϐΫνϟɾΠϯɾϐΫνϟͷ؅ཧΛ͢Δ
    // ද͍ࣔͨ͠ AVSampleBufferDisplayLayer Λࢦఆͯ͠ੜ੒


    let controller = AVPictureInPictureController(


    contentSource: .init(


    sampleBufferDisplayLayer: sampleBufferDisplayLayer,


    playbackDelegate: self


    )


    )

    View full-size slide

  17. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ
    AVPictureInPictureController
    ɾϐΫνϟɾΠϯɾϐΫνϟͷ؅ཧΛ͢Δ
    // ϐΫνϟɾΠϯɾϐΫνϟͷ։࢝


    controller.startPictureInPicture()


    // ϐΫνϟɾΠϯɾϐΫνϟͷఀࢭ


    controller.stopPictureInPicture()

    View full-size slide

  18. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ
    AVPictureInPictureController ͕؅ཧ
    AVSampleBufferDisplayLayer ͷඳը಺༰
    enqueue(sampleBuffer)

    View full-size slide

  19. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ
    AVPictureInPictureController ͕؅ཧ
    AVSampleBufferDisplayLayer ͷඳը಺༰
    enqueue(sampleBuffer)

    View full-size slide

  20. CMSampleBuffer ͷѻ͍
    ɾCMSampleBuffer ΛͲͷ͘Β͍ͷස౓Ͱߋ৽͢Δ͔ʢFPSʣ
    ɾCMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔
    ্ه2఺͕ΞϓϦ΁ͷෛՙʹେ͖͘ӨڹΛ༩͑Δ
    ϛϥςΟϒͷ৔߹ɺΞϓϦ্Ͱ Unity ͕ಈ࡞͍ͯ͠Δঢ়گͰ΋ϐΫνϟɾΠϯɾϐ
    ΫνϟΛಉ࣌ʹಈ࡞ͤ͞Δඞཁ͕͋ΓɺύϑΥʔϚϯεΛؾʹ͢Δඞཁ͕͋Δ

    View full-size slide

  21. CMSampleBuffer ΛͲͷ͘Β͍ͷස౓Ͱߋ৽͢Δ͔
    ɾϛϥςΟϒͰ͸1ඵ͋ͨΓ1ճඳը಺༰Λߋ৽͍ͯ͠Δʢ1FPSʣ
    ɾ഑৴࣌ؒ͸ඵ୯ҐͰද͍ࣔͯ͠Δ
    ɾίϝϯτ΋ߴස౓Ͱߋ৽͞ΕΔඞཁ͸ͳ͍

    View full-size slide

  22. CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔
    ɾCMSampleBuffer ͸৭ʑͳํ๏Ͱੜ੒Ͱ͖Δ
    ɾϏοτϚοϓ͔Βੜ੒͢ΔɺUIImage, CGImage, CIImage ͔Βੜ੒͢Δɺ౳…
    CVPixelBuffer ܦ༝ͰϏοτϚοϓΛฤूͨ͠ࢼ࡞඼

    View full-size slide

  23. CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔
    ɾϛϥςΟϒͰ͸ UIView ͷඳը಺༰Λ CMSampleBuffer ʹॻ͖ࠐΜͰ͍Δ
    ɾ࡞Γ͍ͨ UI ͕ΞϓϦతͰɺAutoLayout ΋׆༻͔ͨͬͨ͠
    ɾίʔυϨΠΞ΢τͰ UI Λ࡞੒͠ɺCMSampleBuffer ʹඳը͢Δ

    View full-size slide

  24. CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔
    ɾϛϥςΟϒͰ͸ UIKit ͷඳը݁ՌΛ CMSampleBuffer ʹॻ͖ࠐΜͰ͍Δ
    let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!


    let pixelBuffer = imageBuffer as CVPixelBuffer


    let renderer = UIGraphicsImageRenderer(size: renderingSize)


    let image = renderer.image { context in


    view.layer.draw(in: context)


    }


    if let cgImage = image.cgImage {


    cicontext.render(CIImage(cgImage: cgImage), to: pixelBuffer)


    }
    ɾUIView → UIImage → CVPixelBuffer → CMSampleBuffer

    View full-size slide

  25. CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔
    ɾඳըपΓͷύϑΥʔϚϯεͷվળΊͪΌؤுͬͨ
    ɾ౰ॳ 50ms ͔͔͍ͬͯͨඳըॲཧ͕ 5ms ʹʂ
    ߋ৽ස౓ͷগͳ͍෦෼(ࠨ)͸͋Β͔͡Ίඳըͯ͠1ຕͷը૾ʹͯ͠Ωϟογϡ͍ͯ͠Δ
    ഑৴ίϝϯτόʔ ʙ PiP ඳըύϑΥʔϚϯεͱͷ޲͖߹͍ํ

    View full-size slide

  26. ଞʹ΋͍Ζ͍Ζ͋Δ͚Ͳ…
    ɾAVAudioSession ͱ PiP ͷؔ܎
    ɾ഑৴ϓϩηεͱ഑৴ίϝϯτόʔͷಉظ
    ɾAVPictureInPictureController ͷ᠘
    ɾR&D ͷ։ൃͰֶͼʹͳͬͨ͜ͱ
    ɾiOS15 Ҏલͷ PiP
    ׂѪ͠·͢ʂ

    View full-size slide

  27. ͋Γ͕ͱ͏͍͟͝·ͨ͠

    View full-size slide