Slide 1

Slide 1 text

ೋบ͘Β͍͋Δ ը໘ऩ࿥͔Βͷੜ์ૹ NBUTVPLBI

Slide 2

Slide 2 text

01&/3&$UW$ZCFS; *OD $ZCFS"HFOU *OD 5XJUUFS!NBUTVPLBI@ (JUIVCNBUVPLBI झຯ ήʔϜ࣮گΛݟΔ͜ͱɺ࣮گੜ์ૹ͢Δ͜ͱ ಛʹ16#( 4QMBUPPO দԬलथ

Slide 3

Slide 3 text

• https://openrec.tv • ήʔϜಛԽͷੜ์ૹαʔϏε • MAU 200ສ • ߴը࣭ɺ௿஗Ԇ഑৴ɺ௥͍͔͚࠶ੜɺ̐ϲࠃޠ(CJK+E)ରԠ • Web/iOS/tvOS/Android/AndroidTV

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

ൃදʹҠΔલʹ

Slide 6

Slide 6 text

E CyberAgent group iOS BeerBash։࠵! ݄೔ Ր ɹ࣌෼։࢝ʢ࣌ΑΓड෇։࢝ʣ αΠόʔΤʔδΣϯτʢौ୩ϓϥΠϜϓϥβ'ɹΫϦΤΠςΟϒϥ΢ϯδʣ dɹ֓ཁઆ໌ɺסഋ dɹ׻ஊ dɹ৽ઃͷ৽نαʔϏε։ൃ૊৫ɹ$"54ʢΫϥΠΞϯτٕज़ྖҬΛݗҾ͢Δ૊৫ʣʹ͍ͭͯ dɹϚονϯάΞϓϦʮλοϓϧ஀ੜʯʹ͓͚Δ։ൃͷมԽ dɹ׻ஊ ͝Ԡื͸23ίʔυ͔Β ͪ͜Β͔ΒͷΤϯτϦʔ͸શһ͝ࢀՃ͍͚ͨͩ·͢ connpass͔ΒΤϯτϦʔͷ৔߹ɺࢀՃ͸நબͱͳΓ·͢ ฐάϧʔϓͷiOSDCొஃऀ͕ࢀՃ͢ΔɹBeerBashͰ͢ʂ দԬ΋ࢀՃ͠·͢ʂ ೔࣌ ৔ॴ ಺༰ ɹɹɹɹɹɹɹגࣜձࣾαΠόʔΤʔδΣϯτ$MJFOU"EWBODFE5FDIOPMPHZ4UVEJP $"54 ɹҏ౻ɹګฏ ɹɹɹɹɹɹɹגࣜձࣾϚονϯάΤʔδΣϯτɹ∁ڮɹ༏հ

Slide 7

Slide 7 text

͜ͷൃදͰ͸Sli.doΛ࢖͍·͢ ൃදͷ్தʹSli.do͔ΒΞϯέʔτΛ͠ͳ͕Β࠷ޙʹൃද ్தͰ࣭໰ͨ͘͠ͳͬͨΒSli.doʹ! ࠷ޙͷ࣭໰λΠϜͰर͍·͢ Event Code: iosdc2018_openrec URL: https://app2.sli.do/event/0kwbnzqe

Slide 8

Slide 8 text

ͦΕͰ͸ൃදʹҠΓ·͢

Slide 9

Slide 9 text

ΞδΣϯμ •HLSͷ֓ཁ •ReplayKitͷ֓ཁ •ը໘ऩ࿥ͷϥΠϑαΠΫϧ •SampleBufferͷฤू •ੜ์ૹதͷૢ࡞ •഑৴ͷҟৗऴྃͱ࠶։ •iOS AppͱBroadcaster Upload Extensionͷσʔλڞ༗ •ExtensionΛσόοά͢Δ

Slide 10

Slide 10 text

HLSͷ֓ཁ

Slide 11

Slide 11 text

HLSͷ֓ཁ HTTP Live Streaming(RFC8216)ͷུ ϓϨʔϠʔ͕ϓϨΠϦετͱࡉ੾Εʹͳ͍ͬͯΔಈըΛ HTTPϦΫΤετͰμ΢ϯϩʔυ͠ɺ ϓϨΠϦετʹఆٛ͞Ε͍ͯΔॱʹ࿈ଓతʹ࠶ੜ͢Δ https://iosdc.com/playlist.m3u8 https://iosdc2018.com/1.ts https://iosdc2018.com/2.ts https://iosdc2018.com/3.ts ϓϨΠϦετ 2ඵຖͷಈը ≒mp4

Slide 12

Slide 12 text

playlit.m3u8ͷத਎ #EXTM3U #EXT-X-VERSION:3 #EXT-X-TARGETDURATION:2 #EXT-X-MEDIA-SEQUENCE:1 #EXT-X-PROGRAM-DATE-TIME: 2018-08-31T12:15:12.713+09:00 #EXTINF:2.0, 1.ts #EXTINF:2.0, 2.ts #EXTINF:2.0, 3.ts #EXT-X-ENDLIST

Slide 13

Slide 13 text

࢓༷ʹଇͬͨϑΝΠϧΛΞοϓϩʔυͯ͠ ϓϨΠϠʔʹμ΢ϯϩʔυͯ͠΋Β͏ CDNΛ༗ޮ׆༻Ͱ͖ΔͨΊ εέʔϥϏϦςΟ/ίετͷ໘Ͱ ੜ์ૹͷٕज़తϋʔυϧΛԼ͛ΒΕΔ HLSͷ࠷େͷϝϦοτ

Slide 14

Slide 14 text

HLSʹΑΔ ੜ์ૹαʔϏεͷશମ૾

Slide 15

Slide 15 text

഑৴ΞϓϦ OBS ഑৴ج൫ ΞϓϦ frame .ts .m3u8 ഑৴ऀ HLSʹΑΔಈը഑৴ͷશମ૾ Storage/CDN ಈը৘ใ URI

Slide 16

Slide 16 text

഑৴ج൫ͷ໾ׂ 1. ड͚औͬͨϝσΟΞͷϑϨʔϜΛ߹੒ͯ͠ ɹ਺ඵ୯ҐͷಈըΛੜ੒͠ɺtsϑΝΠϧͱͯ͠σϓϩΠ 2. σϓϩΠ͞Εͨಈը৘ใΛϓϨΠϦετʹ௥ه͢Δ ഑৴ΞϓϦ OBS ഑৴ج൫ frame .ts .m3u8 Storage/CDN

Slide 17

Slide 17 text

഑৴ΞϓϦ OBS ഑৴ج൫ ΞϓϦ ಈը৘ใ ࢹௌऀ frame .ts .m3u8 ಈը৘ใ .ts .m3u8 ഑৴ऀ Storage/CDN HLSʹΑΔಈը഑৴ͷશମ૾ URI

Slide 18

Slide 18 text

഑৴ΞϓϦ OBS Storage/CDN ഑৴ج൫ ΞϓϦ ಈը৘ใ ࢹௌऀ .ts .m3u8 frame .ts .m3u8 ಈը৘ใ ഑৴ऀ ࠓճѻ͏෦෼ ɾReplayKit ɾBroadcast Upload Extension URI

Slide 19

Slide 19 text

ReplayKitͷ֓ཁ

Slide 20

Slide 20 text

ReplayKitͷ֓ཁ(1) •iOSͷεΫϦʔϯΩϟϓνϟΛѻ͏SDK •ϚΠΫ΍ΞϓϦͷԻ΋࿥ԻՄೳ •iOS9͔Βར༻Մೳ •ήʔϜ΍ϖΠϯτܥΞϓϦɺVTuberΞϓϦ ʹ૊Έࠐ·Ε͍ͯΔ

Slide 21

Slide 21 text

iOS9 iOS10 iOS11 ΞϓϦʹ૊ΈࠐΜͰ࿥ը͕Ͱ͖Δ ૊ΈࠐΜͩΞϓϦ͕ ExtensionʹରԠ͍ͯ͠Δ഑৴ΞϓϦʹॲཧΛҕৡͰ͖Δ ඪ४Ͱը໘ऩ࿥͕࣮૷ ReplayKitͷ֓ཁ(2)

Slide 22

Slide 22 text

•഑৴ͷ࣮૷ΛϓϥοτϑΥʔϜଆʹ೚ͤΒΕΔ •୺຤ͷϥΠϑαΠΫϧͷϋϯυϦϯά͕ߟྀࡁΈ •ࣗલ࿥ըΑΓύϑΥʔϚϯε͕ྑ͍ •UI΍௨஌ͳͲ͕Ωϟϓνϟ͞ΕΔ ϝϦοτ σϝϦοτ ReplayKitΛ͔ͭ͏ϝϦσϝ

Slide 23

Slide 23 text

https://developer.apple.com/videos/play/wwdc2018/601/ WWDC2018Ͱ΋ൃද

Slide 24

Slide 24 text

ը໘ऩ࿥Λड͚෇͚Δʹ͸ Broadcast Upload Extension Λ࣮૷͢Ε͹OK

Slide 25

Slide 25 text

Broadcast Upload Extension λʔήοτΛ࡞੒

Slide 26

Slide 26 text

Broadcast Upload Extension λʔήοτΛ࡞੒ ͜ͷΑ͏ʹɺը໘ऩ࿥ͷબ୒ࢶʹ ΞϓϦ͕ग़ΔΑ͏ʹͳΓ·͢ɻ

Slide 27

Slide 27 text

ը໘ऩ࿥ͷϥΠϑαΠΫϧ

Slide 28

Slide 28 text

ը໘ऩ࿥ͷϥΠϑαΠΫϧ Start Recording Finish ϕʔεͱͳΔϥΠϑαΠΫϧ αʔϏεຖͷ࣮૷͕֤ॴʹ૊Έࠐ·ΕΔ RPBroadcastSampleHandlerͷ ϝιουͱඥ෇͘

Slide 29

Slide 29 text

class SampleHandler: RPBroadcastSampleHandler { override func broadcastStarted(…) { // ഑৴։࢝ } override func processSampleBuffer(…) { // ը໘ऩ࿥ϓϩηε͔ΒΩϟϓνϟ͞ΕͨϑϨʔϜͷड͚औΓ } override func finishBroadcastWithError() { // ഑৴ऴྃ } override func finishBroadcastWithError(…) { // ҟৗऴྃ } } ΤϯτϦʔϙΠϯτ SampleHandler.swift https://developer.apple.com/documentation/replaykit/rpbroadcastsamplehandler

Slide 30

Slide 30 text

֤ϥΠϑαΠΫϧຖͷΞΫγϣϯ Start Recording Finish εϖοΫνΣοΫ/ೝূ ഑৴ઌͷղܾ ഑৴ج൫ʹ઀ଓ Τϯίʔυͱ഑৴ ઀ଓΛ੾அ ഑৴த ഑৴։࢝ ഑৴ऴྃ ঢ়ଶ ঢ়ଶຖͷΞΫγϣϯ ୺຤ͷঢ়ଶΛ؂ࢹ

Slide 31

Slide 31 text

Start εϖοΫνΣοΫ/ೝূ ഑৴ઌͷղܾ ഑৴ج൫ʹ઀ଓ ɾ഑৴Ͱ͖ΔεϖοΫͷ୺຤͔νΣοΫ ɾೝূ৘ใɺ഑৴ݖݶνΣοΫ ɾதஅ͞Ε͍ͯΔ࿮͕͋Δ͔νΣοΫ ɾ഑৴࿮ͷ࡞੒ ɾ഑৴ج൫ͷΞΫηεઌΛαʔόʔ͔Βऔಘ ɾRTMPͷ઀ଓ(ө૾/Ի੠) ɾWebSocketͷ઀ଓ(ίϝϯτͷண৴)

Slide 32

Slide 32 text

Recording Τϯίʔυͱ഑৴ ɾϝσΟΞຖʹΤϯίʔυ(h264/AAC)͠ɺૹ৴ ɾόοϑΝͷฤू/ࠩ͠ସ͑ ୺຤ͷঢ়ଶΛ؂ࢹ ɾը໘ͷճస৘ใ ɾ഑৴ઃఆͷมߋ ɾը໘ڞ༗ΦϑϞʔυ ɾ௨஌ͷON/OFF ɾ௨࿩ͷண৴ ɾ୺຤ͷϩοΫɺి஑੾Ε ɾωοτϫʔΫঢ়گͷ؂ࢹ ɾΞϓϦ಺͔Βͷऴྃ ಈըͷݟ͑ํ தஅ/ऴྃ/ϦτϥΠ

Slide 33

Slide 33 text

Finish ઀ଓΛ੾அ ɾRTMPΛ੾அ ɾWebSocketΛ੾அ ɾ഑৴ऴྃAPI ഑৴ͷઃఆ৘ใΛ࡟আ ɾAppGroup಺ͷ഑৴ઃఆ৘ใΛ࡟আ

Slide 34

Slide 34 text

Recordingதͷ ϝσΟΞαϯϓϦϯά

Slide 35

Slide 35 text

όοϑΝͷૹ৴ HaishinKitʹͯରԠͨ͠ͷͰࠓճ͸Τϯίʔυɾ RTMPʹΑΔόοϑΝͷૹ৴ʹ͸৮Ε·ͤΜɻ HaishinKit͸ϝσΟΞͷΤϯίʔυ͓ΑͼɺετϦʔϛϯάαʔόʔ ͱͷRTMP௨৴Λड͚࣋ͬͯ͘ΕΔϥΠϒϥϦͰ͢ɻ https://github.com/shogo4405/HaishinKit.swift

Slide 36

Slide 36 text

ը໘ऩ࿥ϓϩηε͔Βड͚औ ΔϝσΟΞσʔλ func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) { // process each media data } ϝσΟΞͷੜσʔλɻόΠτ৘ใ͔Β ֤ϝσΟΞʹ෮ݩͯ͠ฤूͨ͠ΓΤϯίʔυͨ͠Γ͢Δ video, audioApp, audioMicͷ̏छྨ sampleBufferType sampleBuffer

Slide 37

Slide 37 text

videoϑϨʔϜ

Slide 38

Slide 38 text

SampleBuffer(video) ϑϨʔϜͷ৘ใؚ͕·Ε͍ͯΔ ɾը૾ͷੜσʔλ ɾը૾ͷճస৘ใ ɾը૾ͷαΠζ ɾ࠶ੜ։࢝࣌ؒ(Presentation Time Stamp)

Slide 39

Slide 39 text

NT NT NT NT NT NT SampleBuffer(video) t 154 #VGGFS αϯϓϦϯάλΠϛϯά͸ը໘ʹߋ৽͕͋ͬͨͱ͖ Duration͸લճͷαϯϓϦϯά͔ΒࠓճͷαϯϓϦϯά

Slide 40

Slide 40 text

ϗʔϜը໘ͷ഑৴Λ͠ͳ͕Β ์ஔ͍ͯͨ͠Β… ҆ఆ࠶ੜͰ͖ͳ͍ࣄ৅ʹؕͬͨ ΩʔϑϨʔϜΛੜ੒ִؒΛ2ඵʹ͍ͯ͠Δʹ΋ ؔΘΒͣ6ඵʹͳͬͯ͠·ͬͨΓ 1ts͋ͨΓಈըͷ୹͞΍ ҆ఆ͠ͳ͍tsͷduration͸ϥάʹ௚݁͢Δ

Slide 41

Slide 41 text

NT NT ը໘ͷߋ৽͕ͳ͍৔߹ 154 t #VGGFS ͜ͷؒ͸ߋ৽͕ͳ͘ϑϨʔϜ͕ૹΒΕͣ ΩʔϑϨʔϜ΋ૹΕͳ͍ͨΊ ಈըͷ۠੾Γ͕͏·͘Ͱ͖ͳ͍

Slide 42

Slide 42 text

ಈըΛ࠶ݱ͢ΔͨΊʹbitmap৘ใΛ͢΂ؚͯΜͩϑϨʔϜ ࣮ࡍͷ ϑϨʔϜ t ಈը ΩʔϑϨʔϜ(I Frame)ͱ͸

Slide 43

Slide 43 text

࣮ࡍͷ ϑϨʔϜ I Frame P Frame ࠩ෼ͷΈΛ࣋ͭ͜ͱͰσʔλྔΛѹॖ HLSʹ͓͍ͯI Frame͸1ͭͷtsϑΝΠϧຖʹ ࠷ॳͷϑϨʔϜͱؚͯ͠ΜͰ͍ͳ͚Ε͹ͳΒͳ͍ ΩʔϑϨʔϜ(I Frame)ͱ͸ ಈըΛ࠶ݱ͢ΔͨΊʹbitmap৘ใΛ͢΂ؚͯΜͩϑϨʔϜ

Slide 44

Slide 44 text

NT NT 154 t #VGGFS videoϑϨʔϜͷϦΧόϦͰରԠ ؒΛຒΊΔϑϨʔϜΛ͍ΕΕ͹OK

Slide 45

Slide 45 text

t NT NT NT NT NT NT videoϑϨʔϜͷϦΧόϦͰରԠ 154 ϑϨʔϜΛૹ৴ͯ͠Ұఆ͕࣌ؒܦͬͯ΋࣍ͷϑϨʔϜ͕ͳ͍࣌ ࠷ޙʹૹ৴ͨ͠ϑϨʔϜΛɺPTSΛ෇͚ସ͑ͯૹ৴͢Δ #VGGFS

Slide 46

Slide 46 text

PTS͕ॏཁ ɾPTSͷζϨ͸ಈըͷԻζϨͷݪҼ ɾόοϑΝͷૢ࡞Ͱ΋͜ͷPTSΛѻ͏ ɾPTSؚΊɺಈը΍Ի੠ͷ࣌ؒ৘ใ͸CMTimeʹΑͬ ͯϑϨʔϜͰѻ͑Δܗࣜʹͳ͍ͬͯΔ

Slide 47

Slide 47 text

CMTime ࣌ؒ৘ใΛ཭ࢄత(ϑϨʔϜ)ʹ੔਺Ͱѻ͏ public struct CMTime { public var value: CMTimeValue public var timescale: CMTimeScale public var flags: CMTimeFlags public var epoch: CMTimeEpoch public init() public init(value: CMTimeValue, timescale: CMTimeScale, flags: CMTimeFlags, epoch: CMTimeEpoch) } extension CMTime { public init(seconds: Double, preferredTimescale: CMTimeScale) public init(value: CMTimeValue, timescale: CMTimeScale) } value: ϑϨʔϜ਺ timescale: 1ඵ͋ͨΓΛԿϑϨʔϜʹ෼ׂ͢Δ͔ʁ

Slide 48

Slide 48 text

CMTimeͷσʔλͷྫ 7JEFP CMTime(value: 1, timeScale: 30) "VEJP CMTime(value: 1024, timeScale: 44100) ≒0.33333…

Slide 49

Slide 49 text

ΩʔϑϨʔϜૹ৴ִؒ ֤αʔϏεͷਪ঑KeyFrame Interval͸2ඵ ɾYouTube ɾTwitch ɾOPENREC.tv ɾFRESH LIVE 1ͭͷts͋ͨΓΛ2ඵͷಈըʹ͍ͨ͠

Slide 50

Slide 50 text

audioϑϨʔϜ

Slide 51

Slide 51 text

CMSampleBuffer(audioMic) ϑϨʔϜͷ৘ใؚ͕·Ε͍ͯΔ ɾԻ੠ͷੜσʔλ ɾLinear PCM ɾ࠶ੜ։࢝࣌ؒ(Presentation Time Stamp) ɾ௕͞(Duration)

Slide 52

Slide 52 text

NT NT NT NT NT CMSampleBuffer(audioMic) 154 㽈 㽈 㽈 㽈 t 㽈 αϯϓϦϯάλΠϛϯά͸ఆظతɻ ִؒ͸1024/44100ඵ(≒23ms) ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS

Slide 53

Slide 53 text

NT NT NT NT NT CMSampleBuffer(audioMic) 154 㽈 㽈 㽈 㽈 t 㽈 ϏσΦόοϑΝ͸ࠓͷදࣔͷόοϑΝʹର͠ ΦʔσΟΦόοϑΝ͸ࠓ·Ͱ࿥Ի͞ΕͨόοϑΝΛड͚औΔ 㽈 㽈 㽈 㽈 㽈 औಘ λΠϛϯά ᶃ ᶄ ᶅ ᶆ ᶇ ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS

Slide 54

Slide 54 text

NT NT NT NT NT CMSampleBuffer(audioApp) 154 㽈 㽈 㽈 㽈 t 㽈 ϚΠΫಉ༷αϯϓϦϯάλΠϛϯά͸΄΅ఆظతɻ ִؒ͸22050/44100ඵલޙ(≒500ms) ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS

Slide 55

Slide 55 text

ө૾ͷฤू

Slide 56

Slide 56 text

ө૾ͷฤू ө૾ͷฤूͱ͍ͬͯ΋CIImageʹΑΔฤू ·ͣCMSampleBuffer͔ΒCIImageܗࣜʹม׵͢Δ extension CMSampleBuffer { var ciImage: CIImage? { guard let pixelBuffer = CMSampleBufferGetImageBuffer(self) else { return nil } CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.readOnly); defer { CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.readOnly) } return CIImage(cvPixelBuffer: pixelBuffer) } } CMSampleBuffer > CVPixelBuffer > CIImage

Slide 57

Slide 57 text

CIImageͰͰ͖Δ͜ͱ ɾ֦େ/ॖখ(CGAffineTransform) ɾҠಈ(CGAffineTransform) ɾ੾ΓऔΓ(crop) ɾճస(oriented) ɾ΅͔͠(applyingGaussianBlur) ɾϑΟϧλʔ

Slide 58

Slide 58 text

CMSampleBufferʹ໭͢ // BlurΛద༻͢ΔͳͲͷฤू let newImage = sampleBuffer.ciImage.applyingGaussianBlur(sigma: 10) // ۣܗ৘ใ let dimensions = CMVideoFormatDescriptionGetDimensions(CMSampleBufferGetFormatDescription(sampleBuffer)!) // ϐΫηϧ৘ใͷॻ͖ࠐΈઌͷόοϑΝΛ࡞੒ var outputPixelBuffer: CVPixelBuffer? = nil CVPixelBufferCreate(kCFAllocatorSystemDefault, Int(dimensions.width), Int(dimensions.height), kCVPixelFormatType_32BGRA, nil, &outputPixelBuffer) CVPixelBufferLockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)); defer { CVPixelBufferUnlockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)) } // όοϑΝʹCIImageΛॻ͖ࠐΈ let ciContext = ContextHolder.shared.context ciContext.render(newImage, to: outputPixelBuffer!) // ৽͍͠SampleBufferΛ࡞੒ var videoFormatDescription: CMVideoFormatDescription? = nil CMVideoFormatDescriptionCreateForImageBuffer(nil, outputPixelBuffer!, &videoFormatDescription) var sampleTimingInfo = self.sampleTimingInfo var newSampleBuffer: CMSampleBuffer? = nil CMSampleBufferCreateForImageBuffer(nil, outputPixelBuffer!, true, nil, nil, videoFormatDescription!, &sampleTimingInfo, &newSampleBuffer)

Slide 59

Slide 59 text

// BlurΛద༻͢ΔͳͲͷฤू let newImage = sampleBuffer.ciImage.applyingGaussianBlur(sigma: 10) CMSampleBuffer͔ΒCIImageΛऔಘ͠ɺ ϑΟϧλʔͳͲΛద༻͢Δ CMSampleBufferʹ໭͢

Slide 60

Slide 60 text

var outputPixelBuffer: CVPixelBuffer? = nil CVPixelBufferCreate(kCFAllocatorSystemDefault, Int(dimensions.width), Int(dimensions.height), kCVPixelFormatType_32BGRA, nil, &outputPixelBuffer) CVPixelBufferLockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)); defer { CVPixelBufferUnlockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)) } CIImageͷॻ͖ࠐΈઌͷόοϑΝΛ࡞੒͢Δ CMSampleBufferʹ໭͢ᶃ

Slide 61

Slide 61 text

// όοϑΝʹCIImageΛॻ͖ࠐΈ let ciContext = ContextHolder.shared.context ciContext.render(newImage, to: outputPixelBuffer!) ฤूͨ͠CIImageΛόοϑΝʹॻ͖ࠐΉ CMSampleBufferʹ໭͢ᶄ

Slide 62

Slide 62 text

var videoFormatDescription: CMVideoFormatDescription? = nil CMVideoFormatDescriptionCreateForImageBuffer(nil, outputPixelBuffer!, &videoFormatDescription) var sampleTimingInfo = self.sampleTimingInfo var newSampleBuffer: CMSampleBuffer? = nil CMSampleBufferCreateForImageBuffer(nil, outputPixelBuffer!, true, nil, nil, videoFormatDescription!, &sampleTimingInfo, &newSampleBuffer) ॻ͖ࠐΜͩBufferͱݩͷCMSampleBufferͷ ࣌ؒ৘ใΛ࢖ͬͯ৽ͨͳCMSampleBufferΛੜ੒͢Δ CMSampleBufferʹ໭͢ᶅ

Slide 63

Slide 63 text

ө૾ͷฤू ֖ֆʹࠩ͠ସ͑

Slide 64

Slide 64 text

͍ͨͩ·४උதʂ ͍ΘΏΔɺྲྀͤΔө૾͕ͳ͍ ࣌ʹ”֖”Λ͢ΔͨΊͷը૾ͷ ͜ͱ ֖ֆͱ͸

Slide 65

Slide 65 text

ɾύεϫʔυೖྗ ɾݸਓ৘ใͱͳΔΞϓϦΛ։͘ ɾʮҰ౓഑৴Λऴྃ͢Ε͹͍͍ʯ͸ඍົ ΅͔͠Λݕ౼͕ͨ͠ɺ ύεϫʔυೖྗͰ͸ ΩʔϘʔυͷҐஔ͔Β ೖྗ͕ਪଌͰ͖ΔͨΊ ׬શͳ֖ֆʹ ֖ֆͷར༻γʔϯ

Slide 66

Slide 66 text

ɾ֖ֆͷը૾৘ใΛ࣋ͬͨSampleBufferΛىಈ࣌ʹੜ੒͓ͯ͘͠ ɾը໘ऩ࿥ϓϩηε͔Βड͚औͬͨSampleBufferͷPTSʹࠩ͠ସ ֖͑ͯֆͷόοϑΝΛૹ৴͢Δ ը໘ऩ࿥͔Β ड͚औͬͨόοϑΝ ֖ֆ όοϑΝ PTSΛ ΋Β͏ ഑৴ج൫ όοϑΝͷࠩ͠ସ͑

Slide 67

Slide 67 text

NT NT NT NT NT NT t 154 'SBNF ڞ༗OFF όοϑΝͷࠩ͠ସ͑

Slide 68

Slide 68 text

NT NT NT NT NT NT t 154 'SBNF ڞ༗OFF όοϑΝͷࠩ͠ସ͑

Slide 69

Slide 69 text

όοϑΝͷࠩ͠ସ͑ ֖ֆͷCMSampleBufferΛϗϧμʔʹอ͓͖࣋ͯ͠ ετϦʔϜͷϋϯυϦϯάଆͰόοϑΝΛࠩ͠ସ͑ class MaskHolder { static let shared = MaskHolder() private let maskBuffer: CMSampleBuffer func mask(with sampleTimingInfo: CMSampleTimingInfo) -> CMSampleBuffer { return maskBuffer.copy(with: sampleTimingInfo) } } class BroadcastService { var isEnabledMask: Bool let rtmpStream: RTMPStream func processSampleBuffer(sampleBuffer: CMSampleBuffer) { let buffer = isEnabledMask ? MaskHolder.shared.mask(with: sampleBuffer.sampleTimingInfo) : sampleBuffer rtmpStream.appendSampleBuffer(sampleBuffer: buffer) } }

Slide 70

Slide 70 text

extension CMSampleBuffer { func copy(with timing: CMSampleTimingInfo) -> CMSampleBuffer? { var sampleTiming = timing var copied: CMSampleBuffer? = nil CMSampleBufferCreateCopyWithNewTiming(kCFAllocatorSystemDefault, self, 1, &sampleTiming, &copied) return copied } var sampleTimingInfo: CMSampleTimingInfo { return CMSampleTimingInfo(duration: self.duration, presentationTimeStamp: self.presentationTimeStamp, decodeTimeStamp: self.decodeTimeStamp) } var duration: CMTime { return CMSampleBufferGetDuration(self) } var presentationTimeStamp: CMTime { return CMSampleBufferGetPresentationTimeStamp(self) } var decodeTimeStamp: CMTime { return CMSampleBufferGetDecodeTimeStamp(self) } } CMSampleBuffer͸ ؔ਺ݺͼग़͠ʹΑΔϓϩύςΟऔಘ͕ଟ͍ͷͰ extensionʹcomputed propertyΛੜ΍͓ͯ͘͠ͱศར όοϑΝͷࠩ͠ସ͑

Slide 71

Slide 71 text

ө૾ͷճస

Slide 72

Slide 72 text

iOSͷը໘͸഑৴தʹํ޲͕มΘΔ ͜ΕΛͲ͏ѻ͏͔ͱ͍͏τϐοΫ ήʔϜ͸ԣը໘΋ଟ͍ɻԻήʔ΍ΧʔυήʔϜͳͲɻ ө૾ͷճసͱ͸ʁ

Slide 73

Slide 73 text

ɾCMSampleBufferͷத਎͸ॎ௕ͷը૾ ɾCMSampleBufferͷϝλ৘ใʹճస৘ใؚ͕·Ε͍ͯΔ ճస৘ใͷऔಘ

Slide 74

Slide 74 text

extension CMSampleBuffer { var orientation: CGImagePropertyOrientation { guard let attachment = CMGetAttachment(self, RPVideoSampleOrientationKey, nil) as? NSNumber else { return .up } if let orientation = CGImagePropertyOrientation(rawValue: orientationAttachment.uint32Value) { return orientation } return .up } } VQ MFGU SJHIU EPXO https://developer.apple.com/documentation/imageio/cgimagepropertyorientation CGImagePropertyOrientation

Slide 75

Slide 75 text

ɾυΩϡϝϯτͰ͸iOS11.0+͕ͩ… ɾiOS11.2.1ҎલͰ͸ىಈ࣌ʹϦϯΫΤϥʔͰڧ੍ऴྃ ɾiOS11.2.5Ҏ߱͸໰୊ͳ͠ CMGetAttachment(self, "RPSampleBufferVideoOrientation" as CFString, nil) RPVideoSampleOrientationKey ɾ11.2.5Ҏ߱ͰऔಘͰ͖Δఆ਺Λϕλଧͪ ɾ11.2.1ҎલͰ͸nil͕औಘ͞ΕΔ ɾ11.2.1ҎલͰ΋ىಈͰ͖Δ͜ͱ͕ॏཁ https://developer.apple.com/documentation/replaykit/rpvideosampleorientationkey

Slide 76

Slide 76 text

഑৴։࢝࣌ʹํ޲ͱղ૾౓Λݻఆɺ ը૾Λճస(CoreImageͷoriented)ͤͯ͞Scale Fit͢Δ ѻ͍ํ1 ॎݻఆ ԣݻఆ

Slide 77

Slide 77 text

ϝϦοτ σϝϦοτ ɾࢹௌऀ͕ݟ΍͍͢ํ޲ʹճస͍ͯ͠Δ ɾϑϨʔϜͷม׵͕ੜ͡Δ ɾ༨ന͕ೖͬͯ͠·͏ ɾෳ਺ͷΞϓϦͷߦ͖དྷΛ͢Δͱ͖ʹɺ ͜ͷඳըʹͳΔ͜ͱΛ഑৴ऀ͕ؾʹ͠ͳ͚Ε͹ͳΒͳ͍ ѻ͍ํ1

Slide 78

Slide 78 text

ಈը͸ॎݻఆʹͯ͠ɺө૾͸ճసͤ͞ͳ͍ɻ αʔόʔʹճస৘ใΛૹΓɺϓϨʔϠʔଆͰճసΛαϙʔτ up right left down ѻ͍ํ2

Slide 79

Slide 79 text

ϝϦοτ σϝϦοτ ɾ༨ന͕ͳ͍ ɾճసॲཧͷඞཁ͕ͳ͍ ɾࢹௌऀ͕ݟΔॠؒʹΑͬͯ͸ํ޲͕Ϛον͠ͳ͍ ɾϓϨΠϠʔ(ϓϥοτϑΥʔϜ)ຖʹରԠ͕ඞཁ ίετ͕ߴ͍ը૾ॲཧΛ1ͭল͚Δˍڐ༰Ͱ͖ΔσϝϦοτ ѻ͍ํ2

Slide 80

Slide 80 text

഑৴ͷऴྃ

Slide 81

Slide 81 text

഑৴ͷऴྃγʔέϯε •WebSocketΛ੾Δ •RTMPΛ੾அ͢Δ •αʔόʔʹ഑৴ऴྃAPIΛ౤͛Δ •AppͱExtensionͷڞ௨ྖҬʹอଘ͍ͯ͠Δ഑৴ʹؔ ࿈͢ΔσʔλΛ࡟আ͢Δ •ҟৗऴྃͰ͋Ε͹ͦͷঢ়ଶΛอଘ͢Δ ഑৴ͷऴྃAPIͱσʔλ࡟আ͸τϥϯβΫγϣϯ

Slide 82

Slide 82 text

DispatchGroup override open func broadcastFinished() { let dispathGroup = DispatchGroup() dispathGroup.enter() broadcastService?.finish { dispathGroup.leave() } dispathGroup.wait() } DispatchGroupͰεϨουΛఀࢭͤ͞Δ

Slide 83

Slide 83 text

BroadcastService func finish(completion: (()->Void)) { api.finishStream(movieID: 123) { completion?() } } αʔϏε಺ͰAPIΛड͚औͬͨΒcompletionΛ࣮ߦ

Slide 84

Slide 84 text

ը໘ऩ࿥Λऴྃͤ͞Δํ๏ •εςʔλεόʔ͔Βऴྃ •ίϯτϩʔϧηϯλʔ͔Βऴྃ •ΞϓϦ͔Βऴྃ

Slide 85

Slide 85 text

ΞϓϦέʔγϣϯ͔Βऴྃ override open func finishBroadcastWithError(_ error: Error) { super.finishBroadcastWithError(error) } RPBroadcastSampleHandlerͰ Τϥʔऴྃͤ͞Δඞཁ͕͋Δ https://developer.apple.com/documentation/replaykit/rpbroadcastsamplehandler/2721526-finishbroadcastwitherror

Slide 86

Slide 86 text

഑৴ͷҟৗऴྃͱ࠶։

Slide 87

Slide 87 text

഑৴Λҟৗऴྃͤ͞Δ override open func finishBroadcastWithError(_ error: Error) { super.finishBroadcastWithError(error) } ద੾ʹΤϥʔΛૹΔ͜ͱͰɺ μΠΞϩάʹऴྃཧ༝͕දࣔͰ͖Δ

Slide 88

Slide 88 text

഑৴Λҟৗऴྃͤ͞Δ let message = “ωοτϫʔΫ͕੾அ͞Ε·ͨ͠" let error = NSError( domain: Bundle.main.bundleIdentifier ?? “" , code: 1 , userInfo: [NSLocalizedFailureReasonErrorKey: message]) finishBroadcastWithError(error) userInfoʹϝοηʔδΛࢦఆ͠ͳ͍ͱ ऴྃཧ༝͕(null)ʹͳΓ·͢ɻ

Slide 89

Slide 89 text

ҙਤ͠ͳ͍ऴྃͱͯ͠ ૝ఆ͞ΕΔέʔε •ը໘͕ϩοΫ͞ΕΔ(࣋ͪํͰిݯϘλϯΛԡͨ͠) •ి஑͕੾ΕΔ •ి࿩͕དྷΔ •BadAccessͰམͪΔ

Slide 90

Slide 90 text

ҟৗ഑৴͕ఀࢭͨ͠৔߹ɺ࠶ ։͍ͤͯ͋͛ͨ͞ •഑৴ͷ਺ࣈͷ෼ࢄ(ࢹௌ਺΍ίϝϯτɺΤʔϧ) •ࢹௌऀͷମݧ •৽͍͠഑৴࿮΁ͷҠಈʹΑΔτϥϑΟοΫͷε ύΠΫ

Slide 91

Slide 91 text

ి஑੾ΕɺిݯOFFɺϩοΫ • RPBroadcastSampleHandlerͷfinish()͕ݺ͹ΕΔ • ਖ਼ৗऴྃѻ͍ʹ͢Δ͔͠ͳ͍ • OSͷ௨஌͸͋Δ͕ඇެ։APIͷͨΊϦδΣΫτ͞ΕΔ

Slide 92

Slide 92 text

௨࿩ͷண৴ iOS͸௨࿩Λண৴͢Δͱɺ RPBroadcastSampleHandlerͷfinish͕ݺ͹ΕΔ ͦͷͨΊɺਖ਼ৗܥͷ഑৴ऴྃγʔέϯεʹೖͬͯ ͠·͏ ண৴Λݕ஌ͨ࣌͠͸ɺ finish͕ݺ͹Εͯ΋தஅѻ͍ʹ͢Δ

Slide 93

Slide 93 text

௨࿩ͷண৴ import CallKit class SampleHandler: RPBroadcastSampleHandler { let callObserver = CXCallObserver() override init() { super.init() callObserver.setDelegate(self, queue: nil) } } extension SampleHandler: CXCallObserverDelegate { func callObserver(…) { broadcastService.suspendStreaming() // ഑৴Λதஅ͢Δ } }

Slide 94

Slide 94 text

ωοτϫʔΫͷ੾அ • Reachability.swiftΛ࠾༻ • RxͰ؂ࢹ͓ͯ͘͠ • ωοτϫʔΫ͕੾Εͨ৔߹͸ҟৗऴྃͱ͠ɺ ୺຤ʹதஅঢ়ଶΛอଘ͢Δ

Slide 95

Slide 95 text

AppͱExtensionͷσʔλڞ༗

Slide 96

Slide 96 text

ڞ༗͢Δσʔλ • ೝূ৘ใ • ഑৴ઃఆ(ը࣭) • ഑৴࿮(഑৴ج൫ͷ઀ଓઌͳͲͷ৘ใ) • ୺຤ઃఆ(௨஌ɺ֖ֆ) • தஅ৘ใ

Slide 97

Slide 97 text

AppGroupΛ࢖ͬͯσʔλΛ ڞ༗͢Δ • ෳ਺ͷΞϓϦؒͰͷσʔλڞ༗ઃఆ • ಉ͡σϕϩούʔͷΞϓϦؒͰ༗ޮ • AppID͝ͱʹ༗ޮԽ͠ɺEntitlementͳͲͷ ઃఆ͕ඞཁ let userDefaults: UserDefaults = UserDefaults(suiteName: “groups.jp.iosdc2018”) IDΛࢦఆ͢Δ͜ͱͰάϧʔϓؒͷڞ༗ྖҬʹΞΫηεͰ͖Δ

Slide 98

Slide 98 text

ExtensionΛσόοά͢Δ

Slide 99

Slide 99 text

ϒϨΠΫϙΠϯτͷషΓํ

Slide 100

Slide 100 text

ϒϨΠΫϙΠϯτͷషΓํ

Slide 101

Slide 101 text

Waitingঢ়ଶʹͳ͔ͬͯΒɺ ഑৴Λ։࢝͢Δ

Slide 102

Slide 102 text

Appͱಉ͡Α͏ʹΠϯεϖΫ τͰ͖·͢

Slide 103

Slide 103 text

Appͱಉ͡Α͏ʹΠϯεϖΫ τͰ͖·͢

Slide 104

Slide 104 text

ΩϟϓνϟͷεϨου BreakpointͰࢭΊͯΔͱ ഑৴ϓϩηε͕ࢮʹ·͢

Slide 105

Slide 105 text

ͳͷͰLoggerΛ࢖͍·͠ΐ͏ • TCP Socket Logger • Logboard/LogboardConsole • JustLog • File Logger • XCGLogger • SwiftyBeaver

Slide 106

Slide 106 text

·ͱΊ

Slide 107

Slide 107 text

·ͱΊ • ReplayKitʹΑͬͯΧδϡΞϧʹੜ์ૹαʔϏε ͕࣮૷Ͱ͖ΔΑ͏ʹͳͬͨʂ • CMSampleBufferΛฤू͢Δ͜ͱͰө૾Λࠩ͠ସ ͑ͨΓΤϑΣΫτΛ͔͚ͨΓ͢Δ͜ͱ͕Ͱ͖Δ! • ഑৴தʹ୺຤͕ى͜ΓಘΔΠϕϯτͷϋϯυϦϯ ά͸ΞϓϦʹൺ΂ͯൈ͚ͳ͘΍Βͳͯ͘͸ͳΒͳ ͍

Slide 108

Slide 108 text

Thanks • HaishinKit(https://github.com/ shogo4405/HaishinKit.swift)