Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PiPを応用した配信コメントバー機能の開発秘話と技術の詳解 / pip_streaming_comment_bar

naru-jpn
September 10, 2022

PiPを応用した配信コメントバー機能の開発秘話と技術の詳解 / pip_streaming_comment_bar

ゲーム配信アプリミラティブに実装され多くのユーザーに利用されている、視聴者からのコメントや各種配信情報をアプリ外で表示する「配信コメントバー」機能の開発の裏側と技術の詳細についてご紹介します。

naru-jpn

September 10, 2022
Tweet

More Decks by naru-jpn

Other Decks in Programming

Transcript

  1. 1J1ΛԠ༻ͨ͠഑৴ίϝϯτόʔػೳͷ
    ։ൃൿ࿩ͱٕज़ͷৄղ
    גࣜձࣾϛϥςΟϒɹઍ٢ྑ੒ل
    @OBSV@KQO OBSVKQO
    aϐΫνϟɾΠϯɾϐΫνϟ

    View full-size slide

  2. ͜ͷτʔΫͰ͓࿩͢͠Δ͜ͱ
    ɾ1J1جຊͷৼΔ෣͍ͱ࣮૷
    ɾϛϥςΟϒͷ഑৴ͱʮ഑৴ίϝϯτόʔʯػೳ
    ɾ3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ
    ɾʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ
    ɾඳըίετͱͷઓ͍

    View full-size slide

  3. 1J1 ϐΫνϟɾΠϯɾϐΫνϟ
    ͷجຊ
    ৼΔ෣͍

    View full-size slide

  4. ΞϓϦ֎Ͱͷಈըίϯςϯπͷ࠶ੜҠಈ
    1J1 ϐΫνϟɾΠϯɾϐΫνϟ
    ͷجຊৼΔ෣͍

    View full-size slide

  5. ը໘ͷ୺ʹӅ͢
    1J1 ϐΫνϟɾΠϯɾϐΫνϟ
    ͷجຊৼΔ෣͍

    View full-size slide

  6. ϐϯνɾΠϯϐϯνɾΞ΢τͰαΠζͷมߋ
    1J1 ϐΫνϟɾΠϯɾϐΫνϟ
    ͷجຊৼΔ෣͍

    View full-size slide

  7. Ϣʔβʔૢ࡞ʹΑΔ։࢝ఀࢭ
    1J1 ϐΫνϟɾΠϯɾϐΫνϟ
    ͷجຊৼΔ෣͍

    View full-size slide

  8. όοΫάϥ΢ϯυҠߦͰࣗಈදࣔ
    1J1 ϐΫνϟɾΠϯɾϐΫνϟ
    ͷجຊৼΔ෣͍

    View full-size slide

  9. ઃఆˠϐΫνϟΠϯϐΫνϟͰʮࣗಈతʹ։࢝ʯͷมߋ
    1J1 ϐΫνϟɾΠϯɾϐΫνϟ
    ͷجຊৼΔ෣͍

    View full-size slide

  10. 1J1 ϐΫνϟɾΠϯɾϐΫνϟ
    ͷجຊ
    "71JDUVSF*O1JDUVSF$POUSPMMFS

    View full-size slide

  11. 1J1 ϐΫνϟɾΠϯɾϐΫνϟ
    ͷجຊ"71JDUVSF*O1JDUVSF$POUSPMMFS
    "71JDUVSF*O1JDUVSF$POUSPMMFS
    ɾ1J1ͷ࣮૷ͷͨΊʹ࢖͏
    ɾ"71JDUVSF*O1JDUVSF$POUSPMMFS͸J04͔Βଘࡏ͍ͯ͠Δ
    ɹɾJ04࣌఺Ͱ͸Ұ෦ͷJ1BEͷΈରԠ
    ɹɾJ04͔ΒJ1IPOFͰ΋1J1͕࢖͑ΔΑ͏ʹ
    ɾ$POUFOU4PVSDF͔Βੜ੒͢Δ˞

    ɹɾinit(contentSource: AVPictureInPictureController.ContentSource)
    ˞ଞͷΠχγϟϥΠβ΋͋Δ͕DPOWFOJFODFͳͷͰ͜͜Ͱ͸ແࢹ͢Δ

    View full-size slide

  12. 1J1 ϐΫνϟɾΠϯɾϐΫνϟ
    ͷجຊ"71JDUVSF*O1JDUVSF$POUSPMMFS
    "71JDUVSF*O1JDUVSF$POUSPMMFSͷ֓ཁ
    ɾ$POUFOU4PVSDFͷछྨ
    ɹɾछྨͷϨΠϠʔ͔Βੜ੒Ͱ͖Δ˞
    ɹɾ"71MBZFS-BZFS͔"74BNQMF#VGGFS%JTQMBZ-BZFS
    ˞"71JDUVSF*O1JDUVSF7JEFP$BMM7JFX$POUSPMMFS͔Β΋ੜ੒Ͱ͖Δ͕ಛघͳػೳͰ"QQMFͷڐՄ͕ͳ͍ͱ։ൃͰ͖ͳ͍ͷͰແࢹ
    init(


    sampleBufferDisplayLayer: AVSampleBufferDisplayLayer,


    playbackDelegate: AVPictureInPictureSampleBufferPlaybackDelegate


    )
    init(playerLayer: AVPlayerLayer) J04
    J04

    View full-size slide

  13. "71MBZFS-BZFSΛ࢖࣮ͬͨ૷

    View full-size slide

  14. 1J1 ϐΫνϟɾΠϯɾϐΫνϟ
    ͷجຊ"71MBZFS-BZFS
    ४උ
    $BQBCJMJUJFT#BDLHSPVOE.PEFT1JDUVSFJO1JDUVSFΛ༗ޮʹ͢Δ
    "7"VEJP4FTTJPOͷΧςΰϦΛQMBZCBDL͔QMBZ"OE3FDPSEʹ͢Δ
    USZ"7"VEJP4FTTJPOTIBSFE*OTUBODF
    TFU$BUFHPSZ QMBZCBDL

    "7'PVOEBUJPO1J11MBZFS1JDUVSFJO1JDUVSF1MBZCBDLXJUI"7,JU

    View full-size slide

  15. 1J1 ϐΫνϟɾΠϯɾϐΫνϟ
    ͷجຊ"71MBZFS-BZFS
    ߏ੒
    "71MBZFS-BZFS
    6*7JFX$POUSPMMFS
    6*#VUUPO
    "71JDUVSF*O1JDUVSF$POUSPMMFS

    View full-size slide

  16. "71JDUVSF*O1JDUVSF$POUSPMMFSͷॳظԽ
    // AVPictureInPictureController の生成


    let controller = AVPictureInPictureController(playerLayer: playerLayer)


    // バックグラウンド移行時に自動でピクチャ・イン・ピクチャを起動するか


    controller.canStartPictureInPictureAutomaticallyFromInline = true
    QMBZFS-BZFS
    1J1 ϐΫνϟɾΠϯɾϐΫνϟ
    ͷجຊ"71MBZFS-BZFS

    View full-size slide

  17. Ϣʔβʔૢ࡞ʹΑΔ։࢝ఀࢭ
    /// ピクチャ・イン・ピクチャの開始


    @IBAction private func didTapStartButton(_ sender: UIButton) {


    pictureInPictureController.startPictureInPicture()


    }


    /// ピクチャ・イン・ピクチャの停止


    @IBAction private func didTapStopButton(_ sender: UIButton) {


    pictureInPictureController.stopPictureInPicture()


    }
    1J1 ϐΫνϟɾΠϯɾϐΫνϟ
    ͷجຊ"71MBZFS-BZFS

    View full-size slide

  18. OBSVKQOQJQFYFSDJTF
    FYBNQMFTQJQCBTJD
    "71MBZFS-BZFSΛ࢖ͬͨ1J1ͷαϯϓϧ࣮૷

    View full-size slide

  19. "74BNQMF#VGGFS%JTQMBZ-BZFSΛ࢖ͬͨ1J1ͷαϯϓϧ࣮૷
    FYBNQMFTQJQDVTUPN FYBNQMFTQJQDBNFSB

    View full-size slide

  20. ϛϥςΟϒͷ഑৴ͱʮ഑৴ίϝϯτόʔʯػೳ

    View full-size slide

  21. ϛϥςΟϒͷʮ഑৴ίϝϯτόʔʯػೳ
    ϛϥςΟϒͰͷ഑৴தͷίϝϯτͱϓογϡ௨஌
    ɾϛϥςΟϒ͸഑৴ΞϓϦͳͷͰࢹௌऀ͕͍Δ
    ɹɾࢹௌऀ͸ςΩετͰίϝϯτΛ͢Δ
    ɹɾ഑৴ऀ͸جຊతʹϛϥςΟϒΞϓϦΛ։͍͍ͯͳ͍
    ɹɹɾήʔϜͳͲଞͷΞϓϦΛ։͍ͯ഑৴͍ͯ͠Δ
    ɹɾ഑৴ऀʹࢹௌऀ͔ΒͷίϝϯτΛಧ͚Δඞཁ͕͋Δ
    ɾJ04Ͱ͸ैདྷ͔Βϓογϡ௨஌ͰϢʔβʔʹίϝϯτΛಧ͚͍ͯΔ

    View full-size slide

  22. ϓογϡ௨஌ʹΑΔ഑৴ͷίϝϯτදࣔʢ഑৴ίϝϯτόʔະ࢖༻ʣ

    View full-size slide

  23. ϛϥςΟϒͷʮ഑৴ίϝϯτόʔʯػೳ
    ഑৴ίϝϯτόʔ
    ɾ഑৴ίϝϯτόʔ͸ϓογϡ௨஌ʹมΘͬͯίϝϯτΛಧ͚Δ
    ɹɾ1J1Λ׆༻ͯ͠ಠࣗͷ6*ͰίϝϯτΛಧ͚Δ
    ɹɾৼΔ෣͍͸ී௨ͷ1J1ͱಉ͡
    ɹɾ1J1͕දࣔ͞Ε͍ͯΔؒɺϓογϡ௨஌͸දࣔ͞Εͳ͍

    View full-size slide

  24. ഑৴ίϝϯτόʔʹΑΔ഑৴ͷίϝϯτදࣔ

    View full-size slide

  25. ϛϥςΟϒͷʮ഑৴ίϝϯτόʔʯػೳ
    ίϝϯτ΍Ϊϑτͷ಺༰ ࠷େߦ
    ഑৴ͷܦա࣌ؒ
    ࢹௌऀ਺
    ଃΒΕͨΪϑτͷ਺
    ഑৴Ի੠ͷPOPGG
    ഑৴ऀͷΤϞϞ Ξόλʔ

    ഑৴ऀͷΤϞϞͷഎܠը૾
    ഑৴ίϝϯτόʔͷཁૉ

    View full-size slide

  26. 3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ

    View full-size slide

  27. 3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ
    ໌ࣔతʹ3%ͱͯ͠։ൃΛਐΊͨ
    ɾੈͷதʹ·ͩͳ͍࣮૷͕ͩͬͨԿਓ͔͸ՄೳੑΛײ͍ͯͨ͡
    ɾԿ͕Ͳ͜·Ͱ࣮૷Ͱ͖Δ͔෼͔Βͳ͔ͬͨ
    ɾΤϯδχΞ͕ٕज़ͷݕূΛ͠ͳ͕Βओମతʹ։ൃΛਐΊ͍ͯ
    ͘ඞཁ͕͋ͬͨ
    ɾࣦഊͯ͠΋ؾʹ͠ͳ͍

    View full-size slide

  28. 3%։ൃ͔Βੜ·Εͨʮ഑৴ίϝϯτόʔʯ
    ഑৴ίϝϯτόʔػೳͷϦϦʔε·Ͱ
    ɹJ04ͷσϕϩούʔϕʔλ൛ϦϦʔε
    ɹࣾ಺ͰϓϩτλΠϓ͕࡞ΒΕͨ
    ɹJ04ϦϦʔε
    ɹ3%։࢝
    ɹJ04ϦϦʔε
    ɹ഑৴ίϝϯτόʔػೳϦϦʔε




    View full-size slide

  29. ࣾ಺ͰϓϩτλΠϓ͕࡞ΒΕͨ
    ɾ"71JDUVSF*O1JDUVSF$POUSPMMFSͱ"74BNQMF#VGGFS%JTQMBZ-BZFS
    Λ૊Έ߹ΘͤͨϓϩτλΠϓ͕ͻͬͦΓͱ࡞ΒΕͨ
    ɾςΩετΛදࣔ͢Δ͚ͩͷγϯϓϧͳ಺༰͕ͩɺՄೳੑΛײ͡Δ
    ʹ͸े෼ͳ࣮૷ͩͬͨ
    3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ




    View full-size slide

  30. 3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ

    View full-size slide

  31. 3%։࢝
    ɾ6*,JUతͳݟͨ໨Λ࠶ݱͰ͖Δͷ͔
    ɹɾϝϯςφϯεੑͷ͋Δ࣮૷͕Ͱ͖Δ͔
    ɾ3FQMBZ,JU΍8FC35$Λ࢖༻͍ͯ͠ΔϛϥςΟϒʹ૊ΈࠐΊΔ͔
    ɾٕज़తʹԿ͕Ͱ͖ΔͰ͖ͳ͍Λ1.΍σβΠφʔʹ఻͑Δ
    ɾࢼ࡞඼Λ࡞ٕͬͯज़తͳݶքΛ୳͍ͬͯͨ
    3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ




    View full-size slide

  32. ϏοτϚοϓͷՃ޻'14
    3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ
    6*,JUͷ࠶ݱ
    ࢼ࡞඼Λ࡞Γͳ͕Βٕज़తͳ໰୊Λ୳͍ͬͯͨ

    View full-size slide

  33. J04ϦϦʔε
    ɾ഑৴தʹϓογϡ௨஌͕දࣔ͞Εͳ͘ͳΔͱ͍͏ࣄ͕݅ى͖ͨ
    ɾ഑৴தͷϓογϡ௨஌Λ੍ݶ͢Δػೳ͕௥Ճ͞Ε͍ͯͨ
    ɹɾઃఆΛมߋ͠ͳ͍ͱࢹௌऀͷίϝϯτ͕ಧ͔ͳ͘ͳͬͯ͠·͏




    3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ
    σϑΥϧτ͸Φϑ

    View full-size slide

  34. J04ϦϦʔε
    ɾ1J1ͷίϝϯτදࣔػೳͳΒઃఆؔ܎ͳ͘࢖͑Δ
    ɾ͜Ε͕௥͍෩ʹͳΓਖ਼ࣜʹεέδϡʔϧ͕੾ΒΕͨ




    3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ
    σϑΥϧτ͸Φϑ

    View full-size slide

  35. 3%Ͱ੔͑ΒΕͨϦϦʔε·ͰͷಓͷΓ
    ɾॳظͷϓϩτλΠϓ͕ଘࡏͨ͠
    ɾՄೳੑΛ୳ΔͨΊͷ3%Λձ͕ࣾਪਐͨ͠
    ɹɾ1.΍σβΠφʔ΋͔ͳΓڠྗతͩͬͨ
    ɾJ04ͷࣄނ͕ޙԡ͠ʹͳͬͨ




    3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ

    View full-size slide

  36. ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ

    View full-size slide

  37. ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ
    1J1ͱಈըͷؔ܎
    1J1͸ΞϓϦʹ഑ஔ͞ΕͨϨΠϠʔ্Ͱ࠶ੜ͞Ε͍ͯΔಈը
    ΛΞϓϦͷྖҬ֎Ͱ࠶ੜ͢Δ࢓૊Έ
    ຊདྷಈը͕࠶ੜ͞
    Ε͍ͯΔϨΠϠʔ

    View full-size slide

  38. ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ
    ಈը͕࠶ੜ͞Εͯ
    ͍ΔྖҬ
    ͭ·Γ1J1͕࠶ੜ͞Ε͍ͯΔظؒ͸ɺΞϓϦͷͲ͔͜Ͱಈը
    ͕࠶ੜ͞Ε͍ͯΔ
    1J1ͱಈըͷؔ܎

    View full-size slide

  39. ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ
    ഑৴ίϝϯτόʔͰ͸Ͳ͏ͳ͍ͬͯΔ͔
    Ͳ͔͜Βͱ΋ͳ͘഑৴ίϝϯτόʔ͕χϣΩοͱग़͖͍ͯͯ
    ΔΑ͏ʹݟ͑Δ

    View full-size slide

  40. ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ
    ഑৴ίϝϯτόʔͰ͸Ͳ͏ͳ͍ͬͯΔ͔
    εϩʔө૾

    View full-size slide

  41. ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ
    ഑৴ίϝϯτόʔͰ͸Ͳ͏ͳ͍ͬͯΔ͔
    ͜ͷਧ͖ग़͠ͷը૾ͷཪଆʹΊͪΌͪ͘Όখ͍͞
    "74BNQMF#VGGFS%JTQMBZ-BZFSΛஔ͍ͯಈըΛ࠶ੜ͍ͯ͠Δ

    View full-size slide

  42. ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ
    "74BNQMF#VGGFS%JTQMBZ-BZFS্ͰಈըΛ࠶ੜ͢Δ
    ɾ"74BNQMF#VGGFS%JTQMBZ-BZFS͸$.4BNQMF#VGGFSΛड͚औͬ
    ͯ಺༰Λඳը͢Δ
    ɹ


    ɾ഑৴ίϝϯτόʔͰ͸ɺ$.4BNQMF#VGGFS͸ಈըͷ֤ϑϨʔϜ
    ʹରԠ͢Δը૾σʔλΛอ͍࣋ͯ͠Δ˞
    ˞$.4BNQMF#VGGFS͸ಈըͷϑϨʔϜ͚ͩͰͳ͘ɺྫ͑͹ෳ਺νϟωϧͷԻ੠σʔλͳͲ΋දݱͰ͖Δ
    func enqueue(_ sampleBuffer: CMSampleBuffer)

    View full-size slide

  43. ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ
    1J1ͷඳըʹ༻͍Δ$.4BNQMF#VGGFSͷߏ଄
    $.4BNQMF#VGGFS
    ɾ$.4BNQMF#VGGFS͸಺෦ʹ$71JYFM#VGGFSΛอ࣋͢Δ
    ɹɾ$71JYFM#VGGFS͸Ұຕͷը૾ʹ૬౰͢ΔσʔλΛ࣋ͭ
    ɾ͜ΕΛ"74BNQMF#VGGFS%JTQMBZ-BZFSʹ౉ͯ͠දࣔ͢Δ
    $71JYFM#VGGFS
    Y
    ˞࣮ࡍʹ͸6*4DSFFONBJOTDBMFഒͷେ͖͞
    ˞
    "3(#

    View full-size slide

  44. ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ
    ഑৴ίϝϯτόʔͰ͸6*,JUͷੈքΛ࠶ݱ͍ͨ͠
    ɾຊ౰͸ಈը͕ͩ6*,JUͰݟͨ໨Λ࡞͍ͬͯΔΑ͏ʹݟ͍ͤͨ
    ɾ৬ਓٕ͗ͯ͢୭΋ϝϯςφϯεͰ͖ͳ͍࢓૊ΈͩͱࠔΔ
    ɾ"VUP-BZPVU΋׆༻͍ͨ͠
    ˠ"VUP-BZPVUͰ഑ஔ͞Εͨ6*7JFXͷඳը಺༰Λ$71JYFM#VGGFS
    ʹॻ͖ࠐΈ͍ͨ

    View full-size slide

  45. ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ
    6*7JFXͷඳը಺༰Λ$71JYFM#VGGFSʹॻ͖ࠐΉ
    let renderer = UIGraphicsImageRenderer(size: size)


    // UIView → UIImage の変換


    let uiImage = renderer.image { context in


    view.layer.render(in: context.cgContext)


    }


    // UIImage → CGImage → CIImage → CVPixelBuffer の変換


    if let cgImage = uiImage.cgImage {


    let ciImage = CIImage(cgImage: cgImage)


    CIContext(options: nil).render(ciImage, to: pixelBuffer)


    }

    View full-size slide

  46. ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ
    ඳը಺༰ͷߋ৽ස౓
    ɾ഑৴࣌ؒ͸))NNTTܗࣜͳͷͰඵʹճߋ৽͢Ε͹े෼
    ɾίϝϯτ͸͋·Γߴස౓Ͱߋ৽ͯ͠΋ಡΊͳ͍
    ˠඳը಺༰ͷߋ৽ස౓ '14
    ͸ͱͨ͠

    View full-size slide

  47. ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ
    ඞཁͳॲཧΛ·ͱΊΔͱ
    ҰఆִؒͰඳը༻ϧʔϓΛ૸ΒͤΔ
    ಈըͷϑϨʔϜʹରԠ͢Δ$.4BNQMF#VGGFSΛੜ੒͢Δ
    $.4BNQMF#VGGFSΛϨϯμϥʔʹ౉͢
    ඳըॲཧΛߦ͏
    Ճ޻ͨ͠$.4BNQMF#VGGFSΛ"74BNQMF#VGGFS%JTQMBZ-BZFSʹ౉͢

    View full-size slide

  48. ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ
    "74BNQMF#VGGFS%JTQMBZ-BZFS
    $POUSPMMFS
    -PPQFS 4BNQMF#VGGFS'BDUPSZ
    3FOEFSFS
    ϧʔϓॲཧ '14
    ͷ੍ޚ $.4BNQMF#VGGFSͷੜ੒
    $71JYFM#VGGFSͷՃ޻
    1J1ͷ੍ޚ ։࢝ऴྃ




    FYBNQMFTQJQDVTUPN͸͜ͷߏ੒Ͱ࣮૷͞Ε͍ͯ·͢

    View full-size slide

  49. ඳըίετͱͷઓ͍

    View full-size slide

  50. ඳըίετͱͷઓ͍
    ඳըίετ͕େ͖͗͢Δ໰୊
    ɾࢼ࡞ஈ֊͔Βඳըίετʹ໰୊͕͋Δ͜ͱ͸෼͔͍ͬͯͨ
    ɾϛϥςΟϒ͸6OJUZͰΤϞϞΛඳը͍ͯ͠Δ
    ɾϝΠϯεϨου΁ͷେ͖͍ෛՙ͸6OJUZͷඳըʹӨڹΛ༩͑Δ
    ɾը໘ͷେ͖͍୺຤͸6OJUZͷඳըίετ΋େ͖͘ɺϝΠϯεϨο
    υ΁ͷෛՙͷӨڹΛड͚΍͍͢

    View full-size slide

  51. ඳըίετͱͷઓ͍
    1J1ͷෛՙͰΤϞϞͷඳըʹࢧো͕ग़͍ͯΔ༷ࢠ

    View full-size slide

  52. $*$POUFYUͷੜ੒͸Ұ౓͚ͩ
    $71JYFM#VGGFS1PPMͷར༻
    ޮ཰ͷΑ͍ඳըํ๏
    ඳըͷճ਺ΛݮΒ͢
    $71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ
    ඳըίετͱͷઓ͍
    ඳըίετͷվળʹ༗ޮͩͬͨରࡦ

    View full-size slide

  53. FYBNQMFTQJQQFSGPSNBODF
    1J1ʹؔ࿈͢Δॲཧͷฏۉॲཧ͕࣌ؒܭଌͰ͖·͢

    View full-size slide

  54. ඳըίετͱͷઓ͍$*$POUFYU
    $*$POUFYUͷੜ੒͸Ұ౓͚ͩ
    let renderer = UIGraphicsImageRenderer(size: size)


    let uiImage = renderer.image { context in


    view.layer.render(in: context.cgContext)


    }


    if let cgImage = uiImage.cgImage {


    let ciImage = CIImage(cgImage: cgImage)


    CIContext(options: nil).render(ciImage, to: pixelBuffer)


    }

    View full-size slide

  55. ඳըίετͱͷઓ͍$*$POUFYU
    $*$POUFYUͷੜ੒͸Ұ౓͚ͩ
    ɾ$*$POUFYUͷੜ੒͸஗͍
    ɾJ1IPOFNJOJͰ͓Α͔͔ͦΔ
    ɾ$PSF*NBHF1SPHSBNNJOH(VJEFʹ΋ॻ͍ͯ͋Δ
    ɾ࡞੒ͨ͠ΠϯελϯεΛอ͓࣋ͯ͜͠͏
    Core Image Programming Guide - Getting the Best Performance

    View full-size slide

  56. ඳըίετͱͷઓ͍$*$POUFYU
    $*$POUFYUͷੜ੒͸Ұ౓͚ͩ
    lazy var context = CIContext(options: nil)


    // …


    let renderer = UIGraphicsImageRenderer(size: size)


    let uiImage = renderer.image { context in


    view.layer.render(in: context.cgContext)


    }


    if let cgImage = uiImage.cgImage {


    let ciImage = CIImage(cgImage: cgImage)


    context.render(ciImage, to: pixelBuffer)


    }

    View full-size slide

  57. ඳըίετͱͷઓ͍$71JYFM#VGGFS1PPM
    $71JYFM#VGGFS1PPMͷར༻
    ɾಈըͷϑϨʔϜʹରԠ͢Δ$.4BNQMF#VGGFSΛੜ੒͢Δ
    ࡍʹࡐྉͱͳΔ$71JYFM#VGGFS
    $.4BNQMF#VGGFS
    $71JYFM#VGGFS
    Y
    "3(#

    View full-size slide

  58. ඳըίετͱͷઓ͍$71JYFM#VGGFS1PPM
    $71JYFM#VGGFS1PPMͷར༻
    ɾ$71JYFM#VGGFSͷੜ੒΋·͋·͋஗͍
    ɾd͘Β͍͔͔Δ
    CVPixelBufferCreate(


    kCFAllocatorDefault,


    Int(size.width),


    Int(size.height),


    kCVPixelFormatType_32ARBG,


    attributes,


    &pixelBufferOut


    )

    View full-size slide

  59. ඳըίετͱͷઓ͍$71JYFM#VGGFS1PPM
    $71JYFM#VGGFS1PPMͷར༻
    ɾ$71JYFM#VGGFSͷ࠶ར༻ͷͨΊͷ$71JYFM#VGGFS1PPM
    ɹɾ6*$PMMFDUJPO7JFXͱ$FMMͷؔ܎ੑ
    ɾ$71JYFM#VGGFSͷऔಘ͕ఔ౓ͰࡁΉ
    CVPixelBufferPoolCreate(kCFAllocatorDefault, nil, attributes, &pixelBufferPool)


    CVPixelBufferPoolCreatePixelBuffer(


    kCFAllocatorDefault,


    pixelBufferPool,


    &pixelBufferOut


    )

    View full-size slide

  60. FYBNQMFTQJQDVTUPN
    4BNQMF#VGGFS'BDUPSZTXJGUʹ$71JYFM#VGGFS1PPMΛར༻࣮ͨ͠૷͕͋Γ·͢

    View full-size slide

  61. ඳըίετͱͷઓ͍ޮ཰ͷΑ͍ඳըํ๏
    ޮ཰ͷΑ͍ඳըํ๏
    ɾ6*7JFX͔Β6**NBHFͷੜ੒Λ͢Δํ๏͸͍͔ͭ͋͘Δ
    ɾESBX)JFSBSDIZํࣜͱMBZFSSFOEFSํࣜͰൺֱͯ͠ΈΔ

    View full-size slide

  62. ESBX)JFSBSDIZํࣜ
    ɾESBX)JFSBSDIZΛ࢖ͬͯ6**NBHFΛੜ੒
    ɾ$71JYFM#VGGFSʹॻ͖ࠐΉ·Ͱd͔͔Δ
    UIGraphicsBeginImageContextWithOptions(renderingSize, false, 0.0)


    // 描画


    view.drawHierarchy(in: frame, afterScreenUpdates: true)


    let image = UIGraphicsGetImageFromCurrentImageContext()


    UIGraphicsEndImageContext()


    if let cgImage = image?.cgImage {


    renderContext.render(CIImage(cgImage: cgImage), to: pixelBuffer)


    }
    ඳըίετͱͷઓ͍ޮ཰ͷΑ͍ඳըํ๏

    View full-size slide

  63. MBZFSSFOEFSํࣜ
    ɾMBZFSSFOEFSΛ࢖ͬͯ6**NBHFΛੜ੒
    ɾ$71JYFM#VGGFSʹॻ͖ࠐΉ·Ͱd͔͔Δ
    let imageRenderer = UIGraphicsImageRenderer(size: renderingSize)


    let image = imageRenderer.image { context in


    // 描画


    view.layer.render(in: context.cgContext)


    }


    if let cgImage = image.cgImage {


    renderContext.render(CIImage(cgImage: cgImage), to: pixelBuffer)


    }
    ඳըίετͱͷઓ͍ޮ཰ͷΑ͍ඳըํ๏

    View full-size slide

  64. ඳըίετͱͷઓ͍ޮ཰ͷΑ͍ඳըํ๏
    ޮ཰ͷΑ͍ඳըํ๏
    ɾESBX)JFSBSDIZํࣜΑΓ΋MBZFSSFOEFSํࣜͷํ͕
    ΄Ͳૣ͍
    ɾESBX)JFSBSDIZ͸6*7JFXͷΞχϝʔγϣϯʹ௥ैͯ͠ඳըͰ
    ͖ΔͳͲͷಛੑ͕͋Δ͕ɺ஗͍
    ɹɾΞχϝʔγϣϯ͸ͤ͞ͳ͍ͷͰMBZFSSFOEFSΛ࠾༻

    View full-size slide

  65. ඳըίετͱͷઓ͍ඳըͷճ਺
    ·ͩ·ͩඳըͷίετ͕ߴ͍
    ɾϏϡʔͷඳըॲཧ͸·ͩக໋తʹίετ͕ߴ͍
    ɾۙ͘ॲཧ͕͔͔࣌ؒͬͯ͠·͏
    // 以下のコードで配信コメントバーの全てのビューを


    // 再起的に描画しようとすると 30[ms] 近くかかってしまう


    let uiImage = renderer.image { context in


    view.layer.render(in: context.cgContext)


    }

    View full-size slide

  66. ඳըίετͱͷઓ͍ඳըͷճ਺
    Ͳ͜ʹίετ͕͔͔͍ͬͯΔ͔
    ɾը૾ 6**NBHF7JFX
    Λඳըͨ͠ͱ͖ͷίετ͕ߴ͍
    ɹɾը૾ʹෆಁ໌ϚεΫ΋͔͚͍ͯͨΓ΋͢Δ
    ɾը૾Λ͋Β͔͡Ίॖখͯ͠࢖༻ͯ͠΋े෼ͳޮՌ͸ͳ͍
    let uiImage = renderer.image { context in


    // view の階層が深くなればなるほど時間がかかる


    view.layer.render(in: context.cgContext)


    }

    View full-size slide

  67. ඳըίετͱͷઓ͍ඳըͷճ਺
    SFOEFS JODPOUFYU
    ΛݺͿճ਺ΛݮΒͯ͠Έͨ
    ɾ͍ΘΏΔυϩʔίʔϧΛݮΒ͢
    ɾߋ৽ස౓͕௿͍ཁૉɺߴ͍ཁૉΛ6*7JFXͷUBHͰ෼ྨ͢Δ
    ɾߋ৽ස౓͕௿͍ཁૉ͸·ͱΊͯҰͭͷը૾ʹ͓ͯ͘͠
    ߋ৽ස౓͕௿͍ཁૉ සൟʹߋ৽͞ΕΔཁૉ

    View full-size slide

  68. ߋ৽ස౓͕௿͍ཁૉͷΈ͔Βੜ੒ͨ͠ը૾͸
    ඞཁͳ͚࣌ͩߋ৽ͯ͠ɺΩϟογϡ͓ͯ͘͠
    ɾΞόλʔ͕ߋ৽͞Εͨͱ͖
    ɾഎܠը૾͕ߋ৽͞Εͨͱ͖
    ɾԻ੠ͷ0/0''ͷ੾Γସ͕͑͞Εͨͱ͖
    ߋ৽ස౓͕ߴ͍ϏϡʔͷBMQIBΛʹͯ͠6*7JFX͔Β6**NBHFΛੜ੒͢Δ
    ඳըίετͱͷઓ͍ඳըͷճ਺

    View full-size slide

  69. ߋ৽ස౓͕ߴ͍ཁૉ͸Ωϟογϡͨ͠ը૾ͷ্ʹ
    ελϯϓܗࣜʢΠϝʔδʣͰඳը͍ͯ͘͠
    ඳըίετͱͷઓ͍ඳըͷճ਺
    func renderRedrawnContents(on view: UIView, in context: CGContext) {


    if view.tag == DrawingPolicy.redrawn.tag {


    view.layer.draw(in: context)


    }


    for subview in view.subviews {


    let origin = subview.frame.origin


    context.translateBy(x: origin.x, y: origin.y)


    renderRedrawnContents(on: subview, in: context)


    context.translateBy(x: -origin.x, y: -origin.y)


    }


    }
    ݸผʹඳը

    View full-size slide

  70. Ұຕֆͷߋ৽ͱίϝϯτόʔͷඳըͷྲྀΕͷ֓೦ਤ
    // エモモや背景などが更新された時だけ一枚絵を再生成する


    if needsToUpdateStableContents {


    updateStableContentsImage()


    }


    let uiImage = renderer.image { context in


    // 更新頻度の低い要素をまとめた画像の描画


    renderStableContents(in: context.cgContext)


    // 更新頻度の高い要素は個別に描画する


    renderRedrawnContents(on: view, in: context.cgContext)


    }
    ඳըίετͱͷઓ͍ඳըͷճ਺

    View full-size slide

  71. ඳըͷճ਺ΛݮΒ͢ޮՌ
    ɾඳըʹ͔͔Δ͕࣌ؒd͔Βdʹվળ͞Εͨ
    ɾ͜ͷվળʹΑͬͯը໘ͷνϥ͖ͭ͸શ୺຤Ͱͳ͘ͳͬͨ
    ɾ࣮͸͜ͷ࣮૷͸2"ͷ࠷ऴ೔Ͱ׬੒ͨ͠ʢʂʣ
    .JSSBUJW5FDI#MPH
    ഑৴ίϝϯτόʔʙ1J1ඳըύϑΥʔϚϯεͱͷ޲͖߹͍ํ
    ඳըίετͱͷઓ͍ඳըͷճ਺

    View full-size slide

  72. ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ
    ͞ΒʹඳըॲཧͷશମΛݟ௚ͯ͠ΈΔ
    ɾ7JFXͷ಺༰͕$71JYFM#VGGFSʹඳը͞ΕΔ·Ͱͷܦ࿏
    ɾͲ͏͍ͬͨதؒ෺͕ੜ੒͞Ε͍ͯΔ͔
    let imageRenderer = UIGraphicsImageRenderer(size: renderingSize)


    let image = imageRenderer.image { context in


    // 描画


    view.layer.render(in: context.cgContext)


    }


    if let cgImage = image.cgImage {


    renderContext.render(CIImage(cgImage: cgImage), to: pixelBuffer)


    }

    View full-size slide

  73. let imageRenderer = UIGraphicsImageRenderer(size: renderingSize)


    let image = imageRenderer.image { context in


    // 描画


    view.layer.render(in: context.cgContext)


    }


    if let cgImage = image.cgImage {


    renderContext.render(CIImage(cgImage: cgImage), to: pixelBuffer)


    }
    ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ
    6*7JFXˠ$($POUFYUˠ6**NBHFˠ$**NBHFˠ$71JYFM#VGGFS

    View full-size slide

  74. ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ
    $.4BNQMF#VGGFS
    $71JYFM#VGGFS
    Y
    "3(#
    6*7JFX $($POUFYU 6**NBHF $**NBHF
    ඳը ඳը

    View full-size slide

  75. ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ
    $71JYFM#VGGFSͱ$($POUFYUͷؔ܎
    ɾ$71JYFM#VGGFS͸"3(#ܗࣜͷσʔλΛอ͍࣋ͯ͠Δ
    ɾ$($POUFYU͸"3(#ܗࣜͷσʔλΛࢀরͯ͠ੜ੒͢Δ͜ͱ͕
    Ͱ͖Δ

    View full-size slide

  76. ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ
    CVPixelBufferLockBaseAddress(pixelBuffer, .readOnly)


    // CVPixelBuffer が保持しているデータのアドレスを取得する


    let data = CVPixelBufferGetBaseAddress(pixelBuffer)


    CVPixelBufferUnlockBaseAddress(pixelBuffer, .readOnly)


    // CVPixelBuffer が保持しているデータを参照した CGContext を生成する


    guard let context = CGContext(


    data: data,


    width: width,


    height: height,


    bitsPerComponent: 8,


    bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer),


    space: CGColorSpaceCreateDeviceRGB(),


    bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue // ARGB


    ) else {


    return


    }

    View full-size slide

  77. ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ
    $.4BNQMF#VGGFS
    $71JYFM#VGGFS
    Y
    "3(#
    $($POUFYU

    View full-size slide

  78. ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ
    $.4BNQMF#VGGFS
    $71JYFM#VGGFS
    Y
    "3(#
    6*7JFX $($POUFYU
    ඳը

    View full-size slide

  79. ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ
    $.4BNQMF#VGGFS
    $71JYFM#VGGFS
    Y
    "3(#
    6*7JFX $($POUFYU
    ඳը
    6*7JFX $($POUFYU 6**NBHF $**NBHF
    ඳը ඳը
    ϝϞϦͷ֬อ΍్தͷॲཧΛݮΒ͢͜ͱͰɺdͷ
    ඳըίετ͕࡟ݮͰ͖Δ

    View full-size slide

  80. ඳըίετͱͷઓ͍
    ඳըίετͱͷઓ͍ͷ݁Ռ
    ɾ࠷ѱͷঢ়ଶͰ͸ϑϨʔϜ͋ͨΓఔ౓ͷඳըίετ͕
    ͔͔͕ͬͨɺdఔ౓ʹվળͨ͠
    ɾը໘ͷେ͖͍୺຤Ͱ΋6OJUZͷඳըͳͲʹࢧো͸ͳ͘ɺগͳ͍
    ෛՙͰ഑৴ίϝϯτόʔͷػೳ͕࣮ݱͰ͖ͨ

    View full-size slide

  81. ·ͱΊ
    ɾ഑৴ίϝϯτόʔ͸3%ʹΑͬͯੜ·Εֵͨ৽తͳػೳ
    ɹɾ1J1্ʹ6*,JUతͳݟͨ໨Λ࠶ݱͰ͖ͨ
    ɹɹɾ"VUP-BZPVUͰ഑ஔ͞ΕɺϝϯςφϯεੑΛ୲อͨ͠
    ɹɾෛՙ΋ඞཁ࠷௿ݶʹ཈͑ΒΕͨ
    ɾ3%Λͨ͠Ձ஋͕͋ͬͨ
    ɹɾϓϨογϟʔ΋େ͖͔͕ͬͨɺಘΒΕΔܦݧ΋ଟ͔ͬͨ
    ɹɾ৽͍͠ػೳ͕ੜ·Εɺٕज़ൃ৴ʹ΋ܨ͕ͬͨ

    View full-size slide

  82. ͋Γ͕ͱ͏͍͟͝·ͨ͠

    View full-size slide