PiPを応用した配信コメントバー機能の開発秘話と技術の詳解 / pip_streaming_comment_bar

by naru-jpn

Slide 1

Slide 1 text

1J1ΛԠ༻ͨ͠഑৴ίϝϯτόʔػೳͷ ։ൃൿ࿩ͱٕज़ͷৄղ גࣜձࣾϛϥςΟϒɹઍ٢ྑ੒ل @OBSV@KQO OBSVKQO aϐΫνϟɾΠϯɾϐΫνϟ

Slide 2

Slide 2 text

͜ͷτʔΫͰ͓࿩͢͠Δ͜ͱ ɾ1J1جຊͷৼΔ෣͍ͱ࣮૷ ɾϛϥςΟϒͷ഑৴ͱʮ഑৴ίϝϯτόʔʯػೳ ɾ3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ ɾʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ ɾඳըίετͱͷઓ͍

Slide 3

Slide 3 text

1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊ ৼΔ෣͍

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

ΞϓϦ֎Ͱͷಈըίϯςϯπͷ࠶ੜҠಈ 1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊৼΔ෣͍

Slide 6

Slide 6 text

ը໘ͷ୺ʹӅ͢ 1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊৼΔ෣͍

Slide 7

Slide 7 text

ϐϯνɾΠϯϐϯνɾΞ΢τͰαΠζͷมߋ 1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊৼΔ෣͍

Slide 8

Slide 8 text

Ϣʔβʔૢ࡞ʹΑΔ։࢝ఀࢭ 1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊৼΔ෣͍

Slide 9

Slide 9 text

όοΫάϥ΢ϯυҠߦͰࣗಈදࣔ 1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊৼΔ෣͍

Slide 10

Slide 10 text

ઃఆˠϐΫνϟΠϯϐΫνϟͰʮࣗಈతʹ։࢝ʯͷมߋ 1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊৼΔ෣͍

Slide 11

Slide 11 text

1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊ "71JDUVSF*O1JDUVSF$POUSPMMFS

Slide 12

Slide 12 text

1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊ"71JDUVSF*O1JDUVSF$POUSPMMFS "71JDUVSF*O1JDUVSF$POUSPMMFS ɾ1J1ͷ࣮૷ͷͨΊʹ࢖͏ ɾ"71JDUVSF*O1JDUVSF$POUSPMMFS͸J04͔Βଘࡏ͍ͯ͠Δ ɹɾJ04࣌఺Ͱ͸Ұ෦ͷJ1BEͷΈରԠ ɹɾJ04͔ΒJ1IPOFͰ΋1J1͕࢖͑ΔΑ͏ʹ ɾ$POUFOU4PVSDF͔Βੜ੒͢Δ˞ ɹɾinit(contentSource: AVPictureInPictureController.ContentSource) ˞ଞͷΠχγϟϥΠβ΋͋Δ͕DPOWFOJFODFͳͷͰ͜͜Ͱ͸ແࢹ͢Δ

Slide 13

Slide 13 text

1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊ"71JDUVSF*O1JDUVSF$POUSPMMFS "71JDUVSF*O1JDUVSF$POUSPMMFSͷ֓ཁ ɾ$POUFOU4PVSDFͷछྨ ɹɾछྨͷϨΠϠʔ͔Βੜ੒Ͱ͖Δ˞ ɹɾ"71MBZFS-BZFS͔"74BNQMF#VGGFS%JTQMBZ-BZFS ˞"71JDUVSF*O1JDUVSF7JEFP$BMM7JFX$POUSPMMFS͔Β΋ੜ੒Ͱ͖Δ͕ಛघͳػೳͰ"QQMFͷڐՄ͕ͳ͍ͱ։ൃͰ͖ͳ͍ͷͰແࢹ init( sampleBufferDisplayLayer: AVSampleBufferDisplayLayer, playbackDelegate: AVPictureInPictureSampleBufferPlaybackDelegate ) init(playerLayer: AVPlayerLayer) J04 J04

Slide 14

Slide 14 text

"71MBZFS-BZFSΛ࢖࣮ͬͨ૷

Slide 15

Slide 15 text

1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊ"71MBZFS-BZFS ४උ $BQBCJMJUJFT#BDLHSPVOE.PEFT1JDUVSFJO1JDUVSFΛ༗ޮʹ͢Δ "7"VEJP4FTTJPOͷΧςΰϦΛQMBZCBDL͔QMBZ"OE3FDPSEʹ͢Δ USZ"7"VEJP4FTTJPOTIBSFE*OTUBODF TFU$BUFHPSZ QMBZCBDL "7'PVOEBUJPO1J11MBZFS1JDUVSFJO1JDUVSF1MBZCBDLXJUI"7,JU

Slide 16

Slide 16 text

1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊ"71MBZFS-BZFS ߏ੒ "71MBZFS-BZFS 6*7JFX$POUSPMMFS 6*#VUUPO "71JDUVSF*O1JDUVSF$POUSPMMFS

Slide 17

Slide 17 text

"71JDUVSF*O1JDUVSF$POUSPMMFSͷॳظԽ // AVPictureInPictureController の生成 let controller = AVPictureInPictureController(playerLayer: playerLayer) // バックグラウンド移行時に自動でピクチャ・イン・ピクチャを起動するか controller.canStartPictureInPictureAutomaticallyFromInline = true QMBZFS-BZFS 1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊ"71MBZFS-BZFS

Slide 18

Slide 18 text

Ϣʔβʔૢ࡞ʹΑΔ։࢝ఀࢭ /// ピクチャ・イン・ピクチャの開始 @IBAction private func didTapStartButton(_ sender: UIButton) { pictureInPictureController.startPictureInPicture() } /// ピクチャ・イン・ピクチャの停止 @IBAction private func didTapStopButton(_ sender: UIButton) { pictureInPictureController.stopPictureInPicture() } 1J1 ϐΫνϟɾΠϯɾϐΫνϟ ͷجຊ"71MBZFS-BZFS

Slide 19

Slide 19 text

OBSVKQOQJQFYFSDJTF FYBNQMFTQJQCBTJD "71MBZFS-BZFSΛ࢖ͬͨ1J1ͷαϯϓϧ࣮૷

Slide 20

Slide 20 text

"74BNQMF#VGGFS%JTQMBZ-BZFSΛ࢖ͬͨ1J1ͷαϯϓϧ࣮૷ FYBNQMFTQJQDVTUPN FYBNQMFTQJQDBNFSB

Slide 21

Slide 21 text

ϛϥςΟϒͷ഑৴ͱʮ഑৴ίϝϯτόʔʯػೳ

Slide 22

Slide 22 text

ϛϥςΟϒͷʮ഑৴ίϝϯτόʔʯػೳ ϛϥςΟϒͰͷ഑৴தͷίϝϯτͱϓογϡ௨஌ ɾϛϥςΟϒ͸഑৴ΞϓϦͳͷͰࢹௌऀ͕͍Δ ɹɾࢹௌऀ͸ςΩετͰίϝϯτΛ͢Δ ɹɾ഑৴ऀ͸جຊతʹϛϥςΟϒΞϓϦΛ։͍͍ͯͳ͍ ɹɹɾήʔϜͳͲଞͷΞϓϦΛ։͍ͯ഑৴͍ͯ͠Δ ɹɾ഑৴ऀʹࢹௌऀ͔ΒͷίϝϯτΛಧ͚Δඞཁ͕͋Δ ɾJ04Ͱ͸ैདྷ͔Βϓογϡ௨஌ͰϢʔβʔʹίϝϯτΛಧ͚͍ͯΔ

Slide 23

Slide 23 text

ϓογϡ௨஌ʹΑΔ഑৴ͷίϝϯτදࣔʢ഑৴ίϝϯτόʔະ࢖༻ʣ

Slide 24

Slide 24 text

ϛϥςΟϒͷʮ഑৴ίϝϯτόʔʯػೳ ഑৴ίϝϯτόʔ ɾ഑৴ίϝϯτόʔ͸ϓογϡ௨஌ʹมΘͬͯίϝϯτΛಧ͚Δ ɹɾ1J1Λ׆༻ͯ͠ಠࣗͷ6*ͰίϝϯτΛಧ͚Δ ɹɾৼΔ෣͍͸ී௨ͷ1J1ͱಉ͡ ɹɾ1J1͕දࣔ͞Ε͍ͯΔؒɺϓογϡ௨஌͸දࣔ͞Εͳ͍

Slide 25

Slide 25 text

഑৴ίϝϯτόʔʹΑΔ഑৴ͷίϝϯτදࣔ

Slide 26

Slide 26 text

ϛϥςΟϒͷʮ഑৴ίϝϯτόʔʯػೳ ίϝϯτ΍Ϊϑτͷ಺༰ ࠷େߦ ഑৴ͷܦա࣌ؒ ࢹௌऀ਺ ଃΒΕͨΪϑτͷ਺ ഑৴Ի੠ͷPOPGG ഑৴ऀͷΤϞϞ Ξόλʔ ഑৴ऀͷΤϞϞͷഎܠը૾ ഑৴ίϝϯτόʔͷཁૉ

Slide 27

Slide 27 text

3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ

Slide 28

Slide 28 text

3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ ໌ࣔతʹ3%ͱͯ͠։ൃΛਐΊͨ ɾੈͷதʹ·ͩͳ͍࣮૷͕ͩͬͨԿਓ͔͸ՄೳੑΛײ͍ͯͨ͡ ɾԿ͕Ͳ͜·Ͱ࣮૷Ͱ͖Δ͔෼͔Βͳ͔ͬͨ ɾΤϯδχΞ͕ٕज़ͷݕূΛ͠ͳ͕Βओମతʹ։ൃΛਐΊ͍ͯ ͘ඞཁ͕͋ͬͨ ɾࣦഊͯ͠΋ؾʹ͠ͳ͍

Slide 29

Slide 29 text

3%։ൃ͔Βੜ·Εͨʮ഑৴ίϝϯτόʔʯ ഑৴ίϝϯτόʔػೳͷϦϦʔε·Ͱ ɹJ04ͷσϕϩούʔϕʔλ൛ϦϦʔε ɹࣾ಺ͰϓϩτλΠϓ͕࡞ΒΕͨ ɹJ04ϦϦʔε ɹ3%։࢝ ɹJ04ϦϦʔε ɹ഑৴ίϝϯτόʔػೳϦϦʔε

Slide 30

Slide 30 text

ࣾ಺ͰϓϩτλΠϓ͕࡞ΒΕͨ ɾ"71JDUVSF*O1JDUVSF$POUSPMMFSͱ"74BNQMF#VGGFS%JTQMBZ-BZFS Λ૊Έ߹ΘͤͨϓϩτλΠϓ͕ͻͬͦΓͱ࡞ΒΕͨ ɾςΩετΛදࣔ͢Δ͚ͩͷγϯϓϧͳ಺༰͕ͩɺՄೳੑΛײ͡Δ ʹ͸े෼ͳ࣮૷ͩͬͨ 3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ

Slide 31

Slide 31 text

3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ

Slide 32

Slide 32 text

3%։࢝ ɾ6*,JUతͳݟͨ໨Λ࠶ݱͰ͖Δͷ͔ ɹɾϝϯςφϯεੑͷ͋Δ࣮૷͕Ͱ͖Δ͔ ɾ3FQMBZ,JU΍8FC35$Λ࢖༻͍ͯ͠ΔϛϥςΟϒʹ૊ΈࠐΊΔ͔ ɾٕज़తʹԿ͕Ͱ͖ΔͰ͖ͳ͍Λ1.΍σβΠφʔʹ఻͑Δ ɾࢼ࡞඼Λ࡞ٕͬͯज़తͳݶքΛ୳͍ͬͯͨ 3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ

Slide 33

Slide 33 text

ϏοτϚοϓͷՃ޻'14 3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ 6*,JUͷ࠶ݱ ࢼ࡞඼Λ࡞Γͳ͕Βٕज़తͳ໰୊Λ୳͍ͬͯͨ

Slide 34

Slide 34 text

J04ϦϦʔε ɾ഑৴தʹϓογϡ௨஌͕දࣔ͞Εͳ͘ͳΔͱ͍͏ࣄ͕݅ى͖ͨ ɾ഑৴தͷϓογϡ௨஌Λ੍ݶ͢Δػೳ͕௥Ճ͞Ε͍ͯͨ ɹɾઃఆΛมߋ͠ͳ͍ͱࢹௌऀͷίϝϯτ͕ಧ͔ͳ͘ͳͬͯ͠·͏ 3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ σϑΥϧτ͸Φϑ

Slide 35

Slide 35 text

J04ϦϦʔε ɾ1J1ͷίϝϯτදࣔػೳͳΒઃఆؔ܎ͳ͘࢖͑Δ ɾ͜Ε͕௥͍෩ʹͳΓਖ਼ࣜʹεέδϡʔϧ͕੾ΒΕͨ 3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ σϑΥϧτ͸Φϑ

Slide 36

Slide 36 text

3%Ͱ੔͑ΒΕͨϦϦʔε·ͰͷಓͷΓ ɾॳظͷϓϩτλΠϓ͕ଘࡏͨ͠ ɾՄೳੑΛ୳ΔͨΊͷ3%Λձ͕ࣾਪਐͨ͠ ɹɾ1.΍σβΠφʔ΋͔ͳΓڠྗతͩͬͨ ɾJ04ͷࣄނ͕ޙԡ͠ʹͳͬͨ 3%Ͱੜ·Εͨʮ഑৴ίϝϯτόʔʯ

Slide 37

Slide 37 text

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ

Slide 38

Slide 38 text

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ 1J1ͱಈըͷؔ܎ 1J1͸ΞϓϦʹ഑ஔ͞ΕͨϨΠϠʔ্Ͱ࠶ੜ͞Ε͍ͯΔಈը ΛΞϓϦͷྖҬ֎Ͱ࠶ੜ͢Δ࢓૊Έ ຊདྷಈը͕࠶ੜ͞ Ε͍ͯΔϨΠϠʔ

Slide 39

Slide 39 text

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ ಈը͕࠶ੜ͞Εͯ ͍ΔྖҬ ͭ·Γ1J1͕࠶ੜ͞Ε͍ͯΔظؒ͸ɺΞϓϦͷͲ͔͜Ͱಈը ͕࠶ੜ͞Ε͍ͯΔ 1J1ͱಈըͷؔ܎

Slide 40

Slide 40 text

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ ഑৴ίϝϯτόʔͰ͸Ͳ͏ͳ͍ͬͯΔ͔ Ͳ͔͜Βͱ΋ͳ͘഑৴ίϝϯτόʔ͕χϣΩοͱग़͖͍ͯͯ ΔΑ͏ʹݟ͑Δ

Slide 41

Slide 41 text

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ ഑৴ίϝϯτόʔͰ͸Ͳ͏ͳ͍ͬͯΔ͔ εϩʔө૾

Slide 42

Slide 42 text

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ ഑৴ίϝϯτόʔͰ͸Ͳ͏ͳ͍ͬͯΔ͔ ͜ͷਧ͖ग़͠ͷը૾ͷཪଆʹΊͪΌͪ͘Όখ͍͞ "74BNQMF#VGGFS%JTQMBZ-BZFSΛஔ͍ͯಈըΛ࠶ੜ͍ͯ͠Δ

Slide 43

Slide 43 text

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ "74BNQMF#VGGFS%JTQMBZ-BZFS্ͰಈըΛ࠶ੜ͢Δ ɾ"74BNQMF#VGGFS%JTQMBZ-BZFS͸$.4BNQMF#VGGFSΛड͚औͬ ͯ಺༰Λඳը͢Δ ɹ ɾ഑৴ίϝϯτόʔͰ͸ɺ$.4BNQMF#VGGFS͸ಈըͷ֤ϑϨʔϜ ʹରԠ͢Δը૾σʔλΛอ͍࣋ͯ͠Δ˞ ˞$.4BNQMF#VGGFS͸ಈըͷϑϨʔϜ͚ͩͰͳ͘ɺྫ͑͹ෳ਺νϟωϧͷԻ੠σʔλͳͲ΋දݱͰ͖Δ func enqueue(_ sampleBuffer: CMSampleBuffer)

Slide 44

Slide 44 text

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ 1J1ͷඳըʹ༻͍Δ$.4BNQMF#VGGFSͷߏ଄ $.4BNQMF#VGGFS ɾ$.4BNQMF#VGGFS͸಺෦ʹ$71JYFM#VGGFSΛอ࣋͢Δ ɹɾ$71JYFM#VGGFS͸Ұຕͷը૾ʹ૬౰͢ΔσʔλΛ࣋ͭ ɾ͜ΕΛ"74BNQMF#VGGFS%JTQMBZ-BZFSʹ౉ͯ͠දࣔ͢Δ $71JYFM#VGGFS Y ˞࣮ࡍʹ͸6*4DSFFONBJOTDBMFഒͷେ͖͞ ˞ "3(#

Slide 45

Slide 45 text

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ ഑৴ίϝϯτόʔͰ͸6*,JUͷੈքΛ࠶ݱ͍ͨ͠ ɾຊ౰͸ಈը͕ͩ6*,JUͰݟͨ໨Λ࡞͍ͬͯΔΑ͏ʹݟ͍ͤͨ ɾ৬ਓٕ͗ͯ͢୭΋ϝϯςφϯεͰ͖ͳ͍࢓૊ΈͩͱࠔΔ ɾ"VUP-BZPVU΋׆༻͍ͨ͠ ˠ"VUP-BZPVUͰ഑ஔ͞Εͨ6*7JFXͷඳը಺༰Λ$71JYFM#VGGFS ʹॻ͖ࠐΈ͍ͨ

Slide 46

Slide 46 text

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ 6*7JFXͷඳը಺༰Λ$71JYFM#VGGFSʹॻ͖ࠐΉ let renderer = UIGraphicsImageRenderer(size: size) // UIView → UIImage の変換 let uiImage = renderer.image { context in view.layer.render(in: context.cgContext) } // UIImage → CGImage → CIImage → CVPixelBuffer の変換 if let cgImage = uiImage.cgImage { let ciImage = CIImage(cgImage: cgImage) CIContext(options: nil).render(ciImage, to: pixelBuffer) } ྫ

Slide 47

Slide 47 text

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ ඳը಺༰ͷߋ৽ස౓ ɾ഑৴࣌ؒ͸))NNTTܗࣜͳͷͰඵʹճߋ৽͢Ε͹े෼ ɾίϝϯτ͸͋·Γߴස౓Ͱߋ৽ͯ͠΋ಡΊͳ͍ ˠඳը಺༰ͷߋ৽ස౓ '14 ͸ͱͨ͠

Slide 48

Slide 48 text

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ ඞཁͳॲཧΛ·ͱΊΔͱ ҰఆִؒͰඳը༻ϧʔϓΛ૸ΒͤΔ ಈըͷϑϨʔϜʹରԠ͢Δ$.4BNQMF#VGGFSΛੜ੒͢Δ $.4BNQMF#VGGFSΛϨϯμϥʔʹ౉͢ ඳըॲཧΛߦ͏ Ճ޻ͨ͠$.4BNQMF#VGGFSΛ"74BNQMF#VGGFS%JTQMBZ-BZFSʹ౉͢

Slide 49

Slide 49 text

ʮ഑৴ίϝϯτόʔʯ࣮૷ͷৄղ "74BNQMF#VGGFS%JTQMBZ-BZFS $POUSPMMFS -PPQFS 4BNQMF#VGGFS'BDUPSZ 3FOEFSFS ϧʔϓॲཧ '14 ͷ੍ޚ $.4BNQMF#VGGFSͷੜ੒ $71JYFM#VGGFSͷՃ޻ 1J1ͷ੍ޚ ։࢝ऴྃ FYBNQMFTQJQDVTUPN͸͜ͷߏ੒Ͱ࣮૷͞Ε͍ͯ·͢

Slide 50

Slide 50 text

ඳըίετͱͷઓ͍

Slide 51

Slide 51 text

ඳըίετͱͷઓ͍ ඳըίετ͕େ͖͗͢Δ໰୊ ɾࢼ࡞ஈ֊͔Βඳըίετʹ໰୊͕͋Δ͜ͱ͸෼͔͍ͬͯͨ ɾϛϥςΟϒ͸6OJUZͰΤϞϞΛඳը͍ͯ͠Δ ɾϝΠϯεϨου΁ͷେ͖͍ෛՙ͸6OJUZͷඳըʹӨڹΛ༩͑Δ ɾը໘ͷେ͖͍୺຤͸6OJUZͷඳըίετ΋େ͖͘ɺϝΠϯεϨο υ΁ͷෛՙͷӨڹΛड͚΍͍͢

Slide 52

Slide 52 text

ඳըίετͱͷઓ͍ 1J1ͷෛՙͰΤϞϞͷඳըʹࢧো͕ग़͍ͯΔ༷ࢠ

Slide 53

Slide 53 text

$*$POUFYUͷੜ੒͸Ұ౓͚ͩ $71JYFM#VGGFS1PPMͷར༻ ޮ཰ͷΑ͍ඳըํ๏ ඳըͷճ਺ΛݮΒ͢ $71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ ඳըίετͱͷઓ͍ ඳըίετͷվળʹ༗ޮͩͬͨରࡦ

Slide 54

Slide 54 text

FYBNQMFTQJQQFSGPSNBODF 1J1ʹؔ࿈͢Δॲཧͷฏۉॲཧ͕࣌ؒܭଌͰ͖·͢

Slide 55

Slide 55 text

ඳըίετͱͷઓ͍$*$POUFYU $*$POUFYUͷੜ੒͸Ұ౓͚ͩ let renderer = UIGraphicsImageRenderer(size: size) let uiImage = renderer.image { context in view.layer.render(in: context.cgContext) } if let cgImage = uiImage.cgImage { let ciImage = CIImage(cgImage: cgImage) CIContext(options: nil).render(ciImage, to: pixelBuffer) }

Slide 56

Slide 56 text

ඳըίετͱͷઓ͍$*$POUFYU $*$POUFYUͷੜ੒͸Ұ౓͚ͩ ɾ$*$POUFYUͷੜ੒͸஗͍ ɾJ1IPOFNJOJͰ͓Α͔͔ͦΔ ɾ$PSF*NBHF1SPHSBNNJOH(VJEFʹ΋ॻ͍ͯ͋Δ ɾ࡞੒ͨ͠ΠϯελϯεΛอ͓࣋ͯ͜͠͏ Core Image Programming Guide - Getting the Best Performance

Slide 57

Slide 57 text

ඳըίετͱͷઓ͍$*$POUFYU $*$POUFYUͷੜ੒͸Ұ౓͚ͩ lazy var context = CIContext(options: nil) // … let renderer = UIGraphicsImageRenderer(size: size) let uiImage = renderer.image { context in view.layer.render(in: context.cgContext) } if let cgImage = uiImage.cgImage { let ciImage = CIImage(cgImage: cgImage) context.render(ciImage, to: pixelBuffer) }

Slide 58

Slide 58 text

ඳըίετͱͷઓ͍$71JYFM#VGGFS1PPM $71JYFM#VGGFS1PPMͷར༻ ɾಈըͷϑϨʔϜʹରԠ͢Δ$.4BNQMF#VGGFSΛੜ੒͢Δ ࡍʹࡐྉͱͳΔ$71JYFM#VGGFS $.4BNQMF#VGGFS $71JYFM#VGGFS Y "3(#

Slide 59

Slide 59 text

ඳըίετͱͷઓ͍$71JYFM#VGGFS1PPM $71JYFM#VGGFS1PPMͷར༻ ɾ$71JYFM#VGGFSͷੜ੒΋·͋·͋஗͍ ɾd͘Β͍͔͔Δ CVPixelBufferCreate( kCFAllocatorDefault, Int(size.width), Int(size.height), kCVPixelFormatType_32ARBG, attributes, &pixelBufferOut )

Slide 60

Slide 60 text

ඳըίετͱͷઓ͍$71JYFM#VGGFS1PPM $71JYFM#VGGFS1PPMͷར༻ ɾ$71JYFM#VGGFSͷ࠶ར༻ͷͨΊͷ$71JYFM#VGGFS1PPM ɹɾ6*$PMMFDUJPO7JFXͱ$FMMͷؔ܎ੑ ɾ$71JYFM#VGGFSͷऔಘ͕ఔ౓ͰࡁΉ CVPixelBufferPoolCreate(kCFAllocatorDefault, nil, attributes, &pixelBufferPool) CVPixelBufferPoolCreatePixelBuffer( kCFAllocatorDefault, pixelBufferPool, &pixelBufferOut )

Slide 61

Slide 61 text

FYBNQMFTQJQDVTUPN 4BNQMF#VGGFS'BDUPSZTXJGUʹ$71JYFM#VGGFS1PPMΛར༻࣮ͨ͠૷͕͋Γ·͢

Slide 62

Slide 62 text

ඳըίετͱͷઓ͍ޮ཰ͷΑ͍ඳըํ๏ ޮ཰ͷΑ͍ඳըํ๏ ɾ6*7JFX͔Β6**NBHFͷੜ੒Λ͢Δํ๏͸͍͔ͭ͋͘Δ ɾESBX)JFSBSDIZํࣜͱMBZFSSFOEFSํࣜͰൺֱͯ͠ΈΔ

Slide 63

Slide 63 text

ESBX)JFSBSDIZํࣜ ɾESBX)JFSBSDIZΛ࢖ͬͯ6**NBHFΛੜ੒ ɾ$71JYFM#VGGFSʹॻ͖ࠐΉ·Ͱd͔͔Δ UIGraphicsBeginImageContextWithOptions(renderingSize, false, 0.0) // 描画 view.drawHierarchy(in: frame, afterScreenUpdates: true) let image = UIGraphicsGetImageFromCurrentImageContext() UIGraphicsEndImageContext() if let cgImage = image?.cgImage { renderContext.render(CIImage(cgImage: cgImage), to: pixelBuffer) } ඳըίετͱͷઓ͍ޮ཰ͷΑ͍ඳըํ๏

Slide 64

Slide 64 text

MBZFSSFOEFSํࣜ ɾMBZFSSFOEFSΛ࢖ͬͯ6**NBHFΛੜ੒ ɾ$71JYFM#VGGFSʹॻ͖ࠐΉ·Ͱd͔͔Δ let imageRenderer = UIGraphicsImageRenderer(size: renderingSize) let image = imageRenderer.image { context in // 描画 view.layer.render(in: context.cgContext) } if let cgImage = image.cgImage { renderContext.render(CIImage(cgImage: cgImage), to: pixelBuffer) } ඳըίετͱͷઓ͍ޮ཰ͷΑ͍ඳըํ๏

Slide 65

Slide 65 text

ඳըίετͱͷઓ͍ޮ཰ͷΑ͍ඳըํ๏ ޮ཰ͷΑ͍ඳըํ๏ ɾESBX)JFSBSDIZํࣜΑΓ΋MBZFSSFOEFSํࣜͷํ͕ ΄Ͳૣ͍ ɾESBX)JFSBSDIZ͸6*7JFXͷΞχϝʔγϣϯʹ௥ैͯ͠ඳըͰ ͖ΔͳͲͷಛੑ͕͋Δ͕ɺ஗͍ ɹɾΞχϝʔγϣϯ͸ͤ͞ͳ͍ͷͰMBZFSSFOEFSΛ࠾༻

Slide 66

Slide 66 text

ඳըίετͱͷઓ͍ඳըͷճ਺ ·ͩ·ͩඳըͷίετ͕ߴ͍ ɾϏϡʔͷඳըॲཧ͸·ͩக໋తʹίετ͕ߴ͍ ɾۙ͘ॲཧ͕͔͔࣌ؒͬͯ͠·͏ // 以下のコードで配信コメントバーの全てのビューを // 再起的に描画しようとすると 30[ms] 近くかかってしまう let uiImage = renderer.image { context in view.layer.render(in: context.cgContext) }

Slide 67

Slide 67 text

ඳըίετͱͷઓ͍ඳըͷճ਺ Ͳ͜ʹίετ͕͔͔͍ͬͯΔ͔ ɾը૾ 6**NBHF7JFX Λඳըͨ͠ͱ͖ͷίετ͕ߴ͍ ɹɾը૾ʹෆಁ໌ϚεΫ΋͔͚͍ͯͨΓ΋͢Δ ɾը૾Λ͋Β͔͡Ίॖখͯ͠࢖༻ͯ͠΋े෼ͳޮՌ͸ͳ͍ let uiImage = renderer.image { context in // view の階層が深くなればなるほど時間がかかる view.layer.render(in: context.cgContext) }

Slide 68

Slide 68 text

ඳըίετͱͷઓ͍ඳըͷճ਺ SFOEFS JODPOUFYU ΛݺͿճ਺ΛݮΒͯ͠Έͨ ɾ͍ΘΏΔυϩʔίʔϧΛݮΒ͢ ɾߋ৽ස౓͕௿͍ཁૉɺߴ͍ཁૉΛ6*7JFXͷUBHͰ෼ྨ͢Δ ɾߋ৽ස౓͕௿͍ཁૉ͸·ͱΊͯҰͭͷը૾ʹ͓ͯ͘͠ ߋ৽ස౓͕௿͍ཁૉ සൟʹߋ৽͞ΕΔཁૉ

Slide 69

Slide 69 text

ߋ৽ස౓͕௿͍ཁૉͷΈ͔Βੜ੒ͨ͠ը૾͸ ඞཁͳ͚࣌ͩߋ৽ͯ͠ɺΩϟογϡ͓ͯ͘͠ ɾΞόλʔ͕ߋ৽͞Εͨͱ͖ ɾഎܠը૾͕ߋ৽͞Εͨͱ͖ ɾԻ੠ͷ0/0''ͷ੾Γସ͕͑͞Εͨͱ͖ ߋ৽ස౓͕ߴ͍ϏϡʔͷBMQIBΛʹͯ͠6*7JFX͔Β6**NBHFΛੜ੒͢Δ ඳըίετͱͷઓ͍ඳըͷճ਺

Slide 70

Slide 70 text

ߋ৽ස౓͕ߴ͍ཁૉ͸Ωϟογϡͨ͠ը૾ͷ্ʹ ελϯϓܗࣜʢΠϝʔδʣͰඳը͍ͯ͘͠ ඳըίετͱͷઓ͍ඳըͷճ਺ func renderRedrawnContents(on view: UIView, in context: CGContext) { if view.tag == DrawingPolicy.redrawn.tag { view.layer.draw(in: context) } for subview in view.subviews { let origin = subview.frame.origin context.translateBy(x: origin.x, y: origin.y) renderRedrawnContents(on: subview, in: context) context.translateBy(x: -origin.x, y: -origin.y) } } ݸผʹඳը

Slide 71

Slide 71 text

Ұຕֆͷߋ৽ͱίϝϯτόʔͷඳըͷྲྀΕͷ֓೦ਤ // エモモや背景などが更新された時だけ一枚絵を再生成する if needsToUpdateStableContents { updateStableContentsImage() } let uiImage = renderer.image { context in // 更新頻度の低い要素をまとめた画像の描画 renderStableContents(in: context.cgContext) // 更新頻度の高い要素は個別に描画する renderRedrawnContents(on: view, in: context.cgContext) } ඳըίετͱͷઓ͍ඳըͷճ਺

Slide 72

Slide 72 text

ඳըͷճ਺ΛݮΒ͢ޮՌ ɾඳըʹ͔͔Δ͕࣌ؒd͔Βdʹվળ͞Εͨ ɾ͜ͷվળʹΑͬͯը໘ͷνϥ͖ͭ͸શ୺຤Ͱͳ͘ͳͬͨ ɾ࣮͸͜ͷ࣮૷͸2"ͷ࠷ऴ೔Ͱ׬੒ͨ͠ʢʂʣ .JSSBUJW5FDI#MPH ഑৴ίϝϯτόʔʙ1J1ඳըύϑΥʔϚϯεͱͷ޲͖߹͍ํ ඳըίετͱͷઓ͍ඳըͷճ਺

Slide 73

Slide 73 text

ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ ͞ΒʹඳըॲཧͷશମΛݟ௚ͯ͠ΈΔ ɾ7JFXͷ಺༰͕$71JYFM#VGGFSʹඳը͞ΕΔ·Ͱͷܦ࿏ ɾͲ͏͍ͬͨதؒ෺͕ੜ੒͞Ε͍ͯΔ͔ let imageRenderer = UIGraphicsImageRenderer(size: renderingSize) let image = imageRenderer.image { context in // 描画 view.layer.render(in: context.cgContext) } if let cgImage = image.cgImage { renderContext.render(CIImage(cgImage: cgImage), to: pixelBuffer) }

Slide 74

Slide 74 text

let imageRenderer = UIGraphicsImageRenderer(size: renderingSize) let image = imageRenderer.image { context in // 描画 view.layer.render(in: context.cgContext) } if let cgImage = image.cgImage { renderContext.render(CIImage(cgImage: cgImage), to: pixelBuffer) } ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ 6*7JFXˠ$($POUFYUˠ6**NBHFˠ$**NBHFˠ$71JYFM#VGGFS

Slide 75

Slide 75 text

ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ $.4BNQMF#VGGFS $71JYFM#VGGFS Y "3(# 6*7JFX $($POUFYU 6**NBHF $**NBHF ඳը ඳը

Slide 76

Slide 76 text

ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ $71JYFM#VGGFSͱ$($POUFYUͷؔ܎ ɾ$71JYFM#VGGFS͸"3(#ܗࣜͷσʔλΛอ͍࣋ͯ͠Δ ɾ$($POUFYU͸"3(#ܗࣜͷσʔλΛࢀরͯ͠ੜ੒͢Δ͜ͱ͕ Ͱ͖Δ

Slide 77

Slide 77 text

ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ CVPixelBufferLockBaseAddress(pixelBuffer, .readOnly) // CVPixelBuffer が保持しているデータのアドレスを取得する let data = CVPixelBufferGetBaseAddress(pixelBuffer) CVPixelBufferUnlockBaseAddress(pixelBuffer, .readOnly) // CVPixelBuffer が保持しているデータを参照した CGContext を生成する guard let context = CGContext( data: data, width: width, height: height, bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer), space: CGColorSpaceCreateDeviceRGB(), bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue // ARGB ) else { return }

Slide 78

Slide 78 text

ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ $.4BNQMF#VGGFS $71JYFM#VGGFS Y "3(# $($POUFYU

Slide 79

Slide 79 text

ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ $.4BNQMF#VGGFS $71JYFM#VGGFS Y "3(# 6*7JFX $($POUFYU ඳը

Slide 80

Slide 80 text

ඳըίετͱͷઓ͍$71JYFM#VGGFSͷՃ޻Λ޻෉͢Δ $.4BNQMF#VGGFS $71JYFM#VGGFS Y "3(# 6*7JFX $($POUFYU ඳը 6*7JFX $($POUFYU 6**NBHF $**NBHF ඳը ඳը ϝϞϦͷ֬อ΍్தͷॲཧΛݮΒ͢͜ͱͰɺdͷ ඳըίετ͕࡟ݮͰ͖Δ

Slide 81

Slide 81 text

ඳըίετͱͷઓ͍ ඳըίετͱͷઓ͍ͷ݁Ռ ɾ࠷ѱͷঢ়ଶͰ͸ϑϨʔϜ͋ͨΓఔ౓ͷඳըίετ͕ ͔͔͕ͬͨɺdఔ౓ʹվળͨ͠ ɾը໘ͷେ͖͍୺຤Ͱ΋6OJUZͷඳըͳͲʹࢧো͸ͳ͘ɺগͳ͍ ෛՙͰ഑৴ίϝϯτόʔͷػೳ͕࣮ݱͰ͖ͨ

Slide 82

Slide 82 text

·ͱΊ ɾ഑৴ίϝϯτόʔ͸3%ʹΑͬͯੜ·Εֵͨ৽తͳػೳ ɹɾ1J1্ʹ6*,JUతͳݟͨ໨Λ࠶ݱͰ͖ͨ ɹɹɾ"VUP-BZPVUͰ഑ஔ͞ΕɺϝϯςφϯεੑΛ୲อͨ͠ ɹɾෛՙ΋ඞཁ࠷௿ݶʹ཈͑ΒΕͨ ɾ3%Λͨ͠Ձ஋͕͋ͬͨ ɹɾϓϨογϟʔ΋େ͖͔͕ͬͨɺಘΒΕΔܦݧ΋ଟ͔ͬͨ ɹɾ৽͍͠ػೳ͕ੜ·Εɺٕज़ൃ৴ʹ΋ܨ͕ͬͨ

Slide 83

Slide 83 text

͋Γ͕ͱ͏͍͟͝·ͨ͠