Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PiP で実現するミラティブの配信コメントバー / pip-streaming-comment-bar

PiP で実現するミラティブの配信コメントバー / pip-streaming-comment-bar

2bfb16e53b154afc9f141f831b4aaaf2?s=128

naru-jpn

March 15, 2022
Tweet

More Decks by naru-jpn

Other Decks in Programming

Transcript

  1. 2022.3.15 - iOS Tech Talk Naruki Chigira @_naru_jpn / Horizon

    Forbidden West ʹϋϚͬͯ·͢ PiP Ͱ࣮ݱ͢ΔϛϥςΟϒͷ഑৴ίϝϯτόʔ
  2. 1. PiP ͱ഑৴ίϝϯτόʔ 2. iOS15 ͱ഑৴ίϝϯτόʔ 3. ഑৴ίϝϯτόʔͷඳը

  3. 1. PiP ͱ഑৴ίϝϯτόʔ

  4. PiPʢϐΫνϟɾΠϯɾϐΫνϟʣ ϗʔϜը໘΍ΞϓϦը໘্ͰಈըΛ࠶ੜͰ͖Δػೳ

  5. ഑৴ίϝϯτόʔ ϗʔϜը໘΍ΞϓϦը໘্ͰϛϥςΟϒ഑৴ͷίϝϯτͳͲΛ͓ಧ͚͢Δػೳ

  6. ࣮ػ্Ͱͷݟ͑ํ

  7. ഑৴ίϝϯτόʔ ϗʔϜը໘΍ΞϓϦը໘্ͰϛϥςΟϒ഑৴ͷίϝϯτͳͲΛ͓ಧ͚͢Δػೳ ࢹௌऀ͔Βͷίϝϯτ ഑৴࣌ؒ ࢹௌऀ਺ ϚΠΫON/OFF ετοΫ͞ΕͨΪϑτͷ਺ ഑৴ऀͷΤϞϞ

  8. 2. iOS15 ͱ഑৴ίϝϯτόʔ

  9. iOS 15 Ҏલ ɾAVPictureInPictureController ͸ iOS 9.0 ͔Β͋Δ ɾAVPictureInPictureController ͸

    PiP ͷ࣮૷ʹ࢖͏Ϋϥε ɾAVPlayerLayer ্Ͱ࠶ੜ͢Δಈըʹ͔͠ରԠ͍ͯ͠ͳ͔ͬͨ
  10. iOS 15 ͰͰ͖ΔΑ͏ʹͳͬͨ͜ͱ ɾiOS 15.0 Ͱ AVPictureInPictureController ͷػೳ͕֦ு͞Εͯ ɹAVSampleBufferDisplayLayer ʹ΋ରԠͨ͠ʢ2021೥7݄ʣ

    ɾϐΫνϟɾΠϯɾϐΫνϟͷࣗ༝౓͕૿ͨ͠ ɾࣾ಺Ͱ R&D ͷػӡ͕ߴ·͖ͬͯͨʢ2021೥10݄ॳ಄ʣ ɾ഑৴ίϝϯτόʔͷࢼ࡞඼͕Ͱ͖͸͡Ίͨʢ2021೥10݄຤ʣ WWDC2021 What's new in AVKit
  11. iOS 15.1 Ͱ੍ݶ͞Εͨ͜ͱ ɾը໘ڞ༗தͷϓογϡ௨஌Λ੍ݶ͢Δઃఆ͕૿͑ͨʢσϑΥONʣ ɾ഑৴த͸ը໘ڞ༗தͱͯ͠ѻΘΕΔͷͰϓογϡ௨஌੍͕ݶ͞ΕΔ ɾϛϥςΟϒ͸ίϝϯτΛϓογϡ௨஌Ͱಧ͚͍ͯΔͷͰࠔΔʢ2021೥10݄຤ʣ ɾ഑৴ίϝϯτόʔ͕࣮ݱͰ͖Ε͹ϓογϡ௨஌ͷ੍໿Λड͚ͣʹࡁΉ

  12. ഑৴ίϝϯτόʔͷ։ൃ 7/1 iOS15 public β 10݄ॳ಄ 10݄຤ 11/25 R&D ࢼ࡞

    ௨஌੍ݶ ػೳϦϦʔε ຊ֨։ൃ!
  13. 3. ഑৴ίϝϯτόʔͷ࣮૷

  14. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ CMSampleBuffer ɾϝσΟΞσʔλʢө૾΍Ի੠ʣΛอ͍࣋ͯ͠ΔΦϒδΣΫτ

  15. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ AVSampleBufferDisplayLayer ɾCMSampleBuffer Λ౉͢ͱදࣔͯ͘͠ΕΔ sampleBufferDisplayLayer.enqueue(sampleBuffer)

  16. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ AVPictureInPictureController ɾϐΫνϟɾΠϯɾϐΫνϟͷ؅ཧΛ͢Δ // ද͍ࣔͨ͠ AVSampleBufferDisplayLayer Λࢦఆͯ͠ੜ੒ let controller =

    AVPictureInPictureController( contentSource: .init( sampleBufferDisplayLayer: sampleBufferDisplayLayer, playbackDelegate: self ) )
  17. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ AVPictureInPictureController ɾϐΫνϟɾΠϯɾϐΫνϟͷ؅ཧΛ͢Δ // ϐΫνϟɾΠϯɾϐΫνϟͷ։࢝ controller.startPictureInPicture() // ϐΫνϟɾΠϯɾϐΫνϟͷఀࢭ controller.stopPictureInPicture()

  18. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ AVPictureInPictureController ͕؅ཧ AVSampleBufferDisplayLayer ͷඳը಺༰ enqueue(sampleBuffer)

  19. ϐΫνϟɾΠϯɾϐΫνϟͷಈ࡞ͷ֓ཁ AVPictureInPictureController ͕؅ཧ AVSampleBufferDisplayLayer ͷඳը಺༰ enqueue(sampleBuffer)

  20. CMSampleBuffer ͷѻ͍ ɾCMSampleBuffer ΛͲͷ͘Β͍ͷස౓Ͱߋ৽͢Δ͔ʢFPSʣ ɾCMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔ ্ه2఺͕ΞϓϦ΁ͷෛՙʹେ͖͘ӨڹΛ༩͑Δ ϛϥςΟϒͷ৔߹ɺΞϓϦ্Ͱ Unity ͕ಈ࡞͍ͯ͠Δঢ়گͰ΋ϐΫνϟɾΠϯɾϐ

    ΫνϟΛಉ࣌ʹಈ࡞ͤ͞Δඞཁ͕͋ΓɺύϑΥʔϚϯεΛؾʹ͢Δඞཁ͕͋Δ
  21. CMSampleBuffer ΛͲͷ͘Β͍ͷස౓Ͱߋ৽͢Δ͔ ɾϛϥςΟϒͰ͸1ඵ͋ͨΓ1ճඳը಺༰Λߋ৽͍ͯ͠Δʢ1FPSʣ ɾ഑৴࣌ؒ͸ඵ୯ҐͰද͍ࣔͯ͠Δ ɾίϝϯτ΋ߴස౓Ͱߋ৽͞ΕΔඞཁ͸ͳ͍

  22. CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔ ɾCMSampleBuffer ͸৭ʑͳํ๏Ͱੜ੒Ͱ͖Δ ɾϏοτϚοϓ͔Βੜ੒͢ΔɺUIImage, CGImage, CIImage ͔Βੜ੒͢Δɺ౳… CVPixelBuffer ܦ༝ͰϏοτϚοϓΛฤूͨ͠ࢼ࡞඼

  23. CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔ ɾϛϥςΟϒͰ͸ UIView ͷඳը಺༰Λ CMSampleBuffer ʹॻ͖ࠐΜͰ͍Δ ɾ࡞Γ͍ͨ UI ͕ΞϓϦతͰɺAutoLayout

    ΋׆༻͔ͨͬͨ͠ ɾίʔυϨΠΞ΢τͰ UI Λ࡞੒͠ɺCMSampleBuffer ʹඳը͢Δ
  24. CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔ ɾϛϥςΟϒͰ͸ UIKit ͷඳը݁ՌΛ CMSampleBuffer ʹॻ͖ࠐΜͰ͍Δ let imageBuffer =

    CMSampleBufferGetImageBuffer(sampleBuffer)! let pixelBuffer = imageBuffer as CVPixelBuffer let renderer = UIGraphicsImageRenderer(size: renderingSize) let image = renderer.image { context in view.layer.draw(in: context) } if let cgImage = image.cgImage { cicontext.render(CIImage(cgImage: cgImage), to: pixelBuffer) } ɾUIView → UIImage → CVPixelBuffer → CMSampleBuffer
  25. CMSampleBuffer ΛͲ͏΍ͬͯߋ৽͢Δ͔ ɾඳըपΓͷύϑΥʔϚϯεͷվળΊͪΌؤுͬͨ ɾ౰ॳ 50ms ͔͔͍ͬͯͨඳըॲཧ͕ 5ms ʹʂ ߋ৽ස౓ͷগͳ͍෦෼(ࠨ)͸͋Β͔͡Ίඳըͯ͠1ຕͷը૾ʹͯ͠Ωϟογϡ͍ͯ͠Δ ഑৴ίϝϯτόʔ

    ʙ PiP ඳըύϑΥʔϚϯεͱͷ޲͖߹͍ํ
  26. ଞʹ΋͍Ζ͍Ζ͋Δ͚Ͳ… ɾAVAudioSession ͱ PiP ͷؔ܎ ɾ഑৴ϓϩηεͱ഑৴ίϝϯτόʔͷಉظ ɾAVPictureInPictureController ͷ᠘ ɾR&D ͷ։ൃͰֶͼʹͳͬͨ͜ͱ

    ɾiOS15 Ҏલͷ PiP ׂѪ͠·͢ʂ
  27. ͋Γ͕ͱ͏͍͟͝·ͨ͠