Live Streaming with Screen Recording

Transcript

1. ೋบ͘Β͍͋Δ
   ը໘ऩ࿥͔Βͷੜ์ૹ NBUTVPLBI
2. 01&/3&$UW
   $ZCFS;
   *OD $ZCFS"HFOU
   *OD
   5XJUUFS
   !NBUTVPLBI@
   (JUIVC
   NBUVPLBI
   झຯ
   ήʔϜ࣮گΛݟΔ͜ͱɺ࣮گੜ์ૹ͢Δ͜ͱ
   ಛʹ16#(
   4QMBUPPO দԬलथ
3. • https://openrec.tv • ήʔϜಛԽͷੜ์ૹαʔϏε • MAU 200ສ • ߴը࣭ɺ௿஗Ԇ഑৴ɺ௥͍͔͚࠶ੜɺ̐ϲࠃޠ(CJK+E)ରԠ • Web/iOS/tvOS/Android/AndroidTV
5. ൃදʹҠΔલʹ
6. E CyberAgent group iOS BeerBash։࠵! ݄೔ Ր ɹ࣌෼։࢝ʢ࣌ΑΓड෇։࢝ʣ
   αΠόʔΤʔδΣϯτʢौ୩ϓϥΠϜϓϥβ'ɹΫϦΤΠςΟϒϥ΢ϯδʣ
   dɹ֓ཁઆ໌ɺסഋ
   dɹ׻ஊ
   dɹ৽ઃͷ৽نαʔϏε։ൃ૊৫ɹ$"54ʢΫϥΠΞϯτٕज़ྖҬΛݗҾ͢Δ૊৫ʣʹ͍ͭͯ
   dɹϚονϯάΞϓϦʮλοϓϧ஀ੜʯʹ͓͚Δ։ൃͷมԽ
   dɹ׻ஊ ͝Ԡื͸23ίʔυ͔Β ͪ͜Β͔ΒͷΤϯτϦʔ͸શһ͝ࢀՃ͍͚ͨͩ·͢ connpass͔ΒΤϯτϦʔͷ৔߹ɺࢀՃ͸நબͱͳΓ·͢ ฐάϧʔϓͷiOSDCొஃऀ͕ࢀՃ͢ΔɹBeerBashͰ͢ʂ দԬ΋ࢀՃ͠·͢ʂ ೔࣌ ৔ॴ ಺༰ ɹɹɹɹɹɹɹ
   
   
   
   גࣜձࣾαΠόʔΤʔδΣϯτ
   $MJFOU
   "EWBODFE
   5FDIOPMPHZ
   4UVEJP $"54 ɹҏ౻ɹګฏ
   
   ɹɹɹɹɹɹɹ
   
   
   
   גࣜձࣾϚονϯάΤʔδΣϯτɹ∁ڮɹ༏հ
   
7. ͜ͷൃදͰ͸Sli.doΛ࢖͍·͢ ൃදͷ్தʹSli.do͔ΒΞϯέʔτΛ͠ͳ͕Β࠷ޙʹൃද ్தͰ࣭໰ͨ͘͠ͳͬͨΒSli.doʹ! ࠷ޙͷ࣭໰λΠϜͰर͍·͢ Event Code: iosdc2018_openrec URL: https://app2.sli.do/event/0kwbnzqe
8. ͦΕͰ͸ൃදʹҠΓ·͢
9. ΞδΣϯμ •HLSͷ֓ཁ •ReplayKitͷ֓ཁ •ը໘ऩ࿥ͷϥΠϑαΠΫϧ •SampleBufferͷฤू •ੜ์ૹதͷૢ࡞ •഑৴ͷҟৗऴྃͱ࠶։ •iOS AppͱBroadcaster Upload Extensionͷσʔλڞ༗ •ExtensionΛσόοά͢Δ
10. HLSͷ֓ཁ
11. HLSͷ֓ཁ HTTP Live Streaming(RFC8216)ͷུ ϓϨʔϠʔ͕ϓϨΠϦετͱࡉ੾Εʹͳ͍ͬͯΔಈըΛ HTTPϦΫΤετͰμ΢ϯϩʔυ͠ɺ ϓϨΠϦετʹఆٛ͞Ε͍ͯΔॱʹ࿈ଓతʹ࠶ੜ͢Δ https://iosdc.com/playlist.m3u8 https://iosdc2018.com/1.ts https://iosdc2018.com/2.ts https://iosdc2018.com/3.ts ϓϨΠϦετ 2ඵຖͷಈը ≒mp4
12. playlit.m3u8ͷத਎ #EXTM3U #EXT-X-VERSION:3 #EXT-X-TARGETDURATION:2 #EXT-X-MEDIA-SEQUENCE:1 #EXT-X-PROGRAM-DATE-TIME: 2018-08-31T12:15:12.713+09:00 #EXTINF:2.0, 1.ts #EXTINF:2.0, 2.ts #EXTINF:2.0, 3.ts #EXT-X-ENDLIST
13. ࢓༷ʹଇͬͨϑΝΠϧΛΞοϓϩʔυͯ͠ ϓϨΠϠʔʹμ΢ϯϩʔυͯ͠΋Β͏ CDNΛ༗ޮ׆༻Ͱ͖ΔͨΊ εέʔϥϏϦςΟ/ίετͷ໘Ͱ ੜ์ૹͷٕज़తϋʔυϧΛԼ͛ΒΕΔ HLSͷ࠷େͷϝϦοτ
14. HLSʹΑΔ ੜ์ૹαʔϏεͷશମ૾
15. ഑৴ΞϓϦ OBS ഑৴ج൫ ΞϓϦ frame .ts .m3u8 ഑৴ऀ HLSʹΑΔಈը഑৴ͷશମ૾ Storage/CDN ಈը৘ใ URI
16. ഑৴ج൫ͷ໾ׂ 1. ड͚औͬͨϝσΟΞͷϑϨʔϜΛ߹੒ͯ͠ ɹ਺ඵ୯ҐͷಈըΛੜ੒͠ɺtsϑΝΠϧͱͯ͠σϓϩΠ 2. σϓϩΠ͞Εͨಈը৘ใΛϓϨΠϦετʹ௥ه͢Δ ഑৴ΞϓϦ OBS ഑৴ج൫ frame .ts .m3u8 Storage/CDN
17. ഑৴ΞϓϦ OBS ഑৴ج൫ ΞϓϦ ಈը৘ใ ࢹௌऀ frame .ts .m3u8 ಈը৘ใ .ts .m3u8 ഑৴ऀ Storage/CDN HLSʹΑΔಈը഑৴ͷશମ૾ URI
18. ഑৴ΞϓϦ OBS Storage/CDN ഑৴ج൫ ΞϓϦ ಈը৘ใ ࢹௌऀ .ts .m3u8 frame .ts .m3u8 ಈը৘ใ ഑৴ऀ ࠓճѻ͏෦෼ ɾReplayKit ɾBroadcast Upload Extension URI
19. ReplayKitͷ֓ཁ
20. ReplayKitͷ֓ཁ(1) •iOSͷεΫϦʔϯΩϟϓνϟΛѻ͏SDK •ϚΠΫ΍ΞϓϦͷԻ΋࿥ԻՄೳ •iOS9͔Βར༻Մೳ •ήʔϜ΍ϖΠϯτܥΞϓϦɺVTuberΞϓϦ ʹ૊Έࠐ·Ε͍ͯΔ
21. iOS9 iOS10 iOS11 ΞϓϦʹ૊ΈࠐΜͰ࿥ը͕Ͱ͖Δ ૊ΈࠐΜͩΞϓϦ͕ ExtensionʹରԠ͍ͯ͠Δ഑৴ΞϓϦʹॲཧΛҕৡͰ͖Δ ඪ४Ͱը໘ऩ࿥͕࣮૷ ReplayKitͷ֓ཁ(2)
22. •഑৴ͷ࣮૷ΛϓϥοτϑΥʔϜଆʹ೚ͤΒΕΔ •୺຤ͷϥΠϑαΠΫϧͷϋϯυϦϯά͕ߟྀࡁΈ •ࣗલ࿥ըΑΓύϑΥʔϚϯε͕ྑ͍ •UI΍௨஌ͳͲ͕Ωϟϓνϟ͞ΕΔ ϝϦοτ σϝϦοτ ReplayKitΛ͔ͭ͏ϝϦσϝ
23. https://developer.apple.com/videos/play/wwdc2018/601/ WWDC2018Ͱ΋ൃද
24. ը໘ऩ࿥Λड͚෇͚Δʹ͸ Broadcast Upload Extension Λ࣮૷͢Ε͹OK
25. Broadcast Upload Extension λʔήοτΛ࡞੒
26. Broadcast Upload Extension λʔήοτΛ࡞੒ ͜ͷΑ͏ʹɺը໘ऩ࿥ͷબ୒ࢶʹ ΞϓϦ͕ग़ΔΑ͏ʹͳΓ·͢ɻ
27. ը໘ऩ࿥ͷϥΠϑαΠΫϧ
28. ը໘ऩ࿥ͷϥΠϑαΠΫϧ Start Recording Finish ϕʔεͱͳΔϥΠϑαΠΫϧ αʔϏεຖͷ࣮૷͕֤ॴʹ૊Έࠐ·ΕΔ RPBroadcastSampleHandlerͷ ϝιουͱඥ෇͘
29. class SampleHandler: RPBroadcastSampleHandler { override func broadcastStarted(…) { // ഑৴։࢝ } override func processSampleBuffer(…) { // ը໘ऩ࿥ϓϩηε͔ΒΩϟϓνϟ͞ΕͨϑϨʔϜͷड͚औΓ } override func finishBroadcastWithError() { // ഑৴ऴྃ } override func finishBroadcastWithError(…) { // ҟৗऴྃ } } ΤϯτϦʔϙΠϯτ SampleHandler.swift https://developer.apple.com/documentation/replaykit/rpbroadcastsamplehandler
30. ֤ϥΠϑαΠΫϧຖͷΞΫγϣϯ Start Recording Finish εϖοΫνΣοΫ/ೝূ ഑৴ઌͷղܾ ഑৴ج൫ʹ઀ଓ Τϯίʔυͱ഑৴ ઀ଓΛ੾அ ഑৴த ഑৴։࢝ ഑৴ऴྃ ঢ়ଶ ঢ়ଶຖͷΞΫγϣϯ ୺຤ͷঢ়ଶΛ؂ࢹ
31. Start εϖοΫνΣοΫ/ೝূ ഑৴ઌͷղܾ ഑৴ج൫ʹ઀ଓ ɾ഑৴Ͱ͖ΔεϖοΫͷ୺຤͔νΣοΫ ɾೝূ৘ใɺ഑৴ݖݶνΣοΫ ɾதஅ͞Ε͍ͯΔ࿮͕͋Δ͔νΣοΫ ɾ഑৴࿮ͷ࡞੒ ɾ഑৴ج൫ͷΞΫηεઌΛαʔόʔ͔Βऔಘ ɾRTMPͷ઀ଓ(ө૾/Ի੠) ɾWebSocketͷ઀ଓ(ίϝϯτͷண৴)
32. Recording Τϯίʔυͱ഑৴ ɾϝσΟΞຖʹΤϯίʔυ(h264/AAC)͠ɺૹ৴ ɾόοϑΝͷฤू/ࠩ͠ସ͑ ୺຤ͷঢ়ଶΛ؂ࢹ ɾը໘ͷճస৘ใ ɾ഑৴ઃఆͷมߋ ɾը໘ڞ༗ΦϑϞʔυ ɾ௨஌ͷON/OFF ɾ௨࿩ͷண৴ ɾ୺຤ͷϩοΫɺి஑੾Ε ɾωοτϫʔΫঢ়گͷ؂ࢹ ɾΞϓϦ಺͔Βͷऴྃ ಈըͷݟ͑ํ தஅ/ऴྃ/ϦτϥΠ
33. Finish ઀ଓΛ੾அ ɾRTMPΛ੾அ ɾWebSocketΛ੾அ ɾ഑৴ऴྃAPI ഑৴ͷઃఆ৘ใΛ࡟আ ɾAppGroup಺ͷ഑৴ઃఆ৘ใΛ࡟আ
34. Recordingதͷ ϝσΟΞαϯϓϦϯά
35. όοϑΝͷૹ৴ HaishinKitʹͯରԠͨ͠ͷͰࠓճ͸Τϯίʔυɾ RTMPʹΑΔόοϑΝͷૹ৴ʹ͸৮Ε·ͤΜɻ HaishinKit͸ϝσΟΞͷΤϯίʔυ͓ΑͼɺετϦʔϛϯάαʔόʔ ͱͷRTMP௨৴Λड͚࣋ͬͯ͘ΕΔϥΠϒϥϦͰ͢ɻ https://github.com/shogo4405/HaishinKit.swift
36. ը໘ऩ࿥ϓϩηε͔Βड͚औ ΔϝσΟΞσʔλ func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) { // process each media data } ϝσΟΞͷੜσʔλɻόΠτ৘ใ͔Β ֤ϝσΟΞʹ෮ݩͯ͠ฤूͨ͠ΓΤϯίʔυͨ͠Γ͢Δ video, audioApp, audioMicͷ̏छྨ sampleBufferType sampleBuffer
37. videoϑϨʔϜ
38. SampleBuffer(video) ϑϨʔϜͷ৘ใؚ͕·Ε͍ͯΔ ɾը૾ͷੜσʔλ ɾը૾ͷճస৘ใ ɾը૾ͷαΠζ ɾ࠶ੜ։࢝࣌ؒ(Presentation Time Stamp)
39. NT NT NT NT NT NT SampleBuffer(video) t 154 #VGGFS αϯϓϦϯάλΠϛϯά͸ը໘ʹߋ৽͕͋ͬͨͱ͖ Duration͸લճͷαϯϓϦϯά͔ΒࠓճͷαϯϓϦϯά
40. ϗʔϜը໘ͷ഑৴Λ͠ͳ͕Β ์ஔ͍ͯͨ͠Β… ҆ఆ࠶ੜͰ͖ͳ͍ࣄ৅ʹؕͬͨ ΩʔϑϨʔϜΛੜ੒ִؒΛ2ඵʹ͍ͯ͠Δʹ΋ ؔΘΒͣ6ඵʹͳͬͯ͠·ͬͨΓ 1ts͋ͨΓಈըͷ୹͞΍ ҆ఆ͠ͳ͍tsͷduration͸ϥάʹ௚݁͢Δ
41. NT NT ը໘ͷߋ৽͕ͳ͍৔߹ 154 t #VGGFS ͜ͷؒ͸ߋ৽͕ͳ͘ϑϨʔϜ͕ૹΒΕͣ ΩʔϑϨʔϜ΋ૹΕͳ͍ͨΊ ಈըͷ۠੾Γ͕͏·͘Ͱ͖ͳ͍
42. ಈըΛ࠶ݱ͢ΔͨΊʹbitmap৘ใΛ͢΂ؚͯΜͩϑϨʔϜ ࣮ࡍͷ ϑϨʔϜ t ಈը ΩʔϑϨʔϜ(I Frame)ͱ͸
43. ࣮ࡍͷ ϑϨʔϜ I Frame P Frame ࠩ෼ͷΈΛ࣋ͭ͜ͱͰσʔλྔΛѹॖ HLSʹ͓͍ͯI Frame͸1ͭͷtsϑΝΠϧຖʹ ࠷ॳͷϑϨʔϜͱؚͯ͠ΜͰ͍ͳ͚Ε͹ͳΒͳ͍ ΩʔϑϨʔϜ(I Frame)ͱ͸ ಈըΛ࠶ݱ͢ΔͨΊʹbitmap৘ใΛ͢΂ؚͯΜͩϑϨʔϜ
44. NT NT 154 t #VGGFS videoϑϨʔϜͷϦΧόϦͰରԠ ؒΛຒΊΔϑϨʔϜΛ͍ΕΕ͹OK
45. t NT NT NT NT NT NT videoϑϨʔϜͷϦΧόϦͰରԠ 154 ϑϨʔϜΛૹ৴ͯ͠Ұఆ͕࣌ؒܦͬͯ΋࣍ͷϑϨʔϜ͕ͳ͍࣌ ࠷ޙʹૹ৴ͨ͠ϑϨʔϜΛɺPTSΛ෇͚ସ͑ͯૹ৴͢Δ #VGGFS
46. PTS͕ॏཁ ɾPTSͷζϨ͸ಈըͷԻζϨͷݪҼ ɾόοϑΝͷૢ࡞Ͱ΋͜ͷPTSΛѻ͏ ɾPTSؚΊɺಈը΍Ի੠ͷ࣌ؒ৘ใ͸CMTimeʹΑͬ ͯϑϨʔϜͰѻ͑Δܗࣜʹͳ͍ͬͯΔ
47. CMTime ࣌ؒ৘ใΛ཭ࢄత(ϑϨʔϜ)ʹ੔਺Ͱѻ͏ public struct CMTime { public var value: CMTimeValue public var timescale: CMTimeScale public var flags: CMTimeFlags public var epoch: CMTimeEpoch public init() public init(value: CMTimeValue, timescale: CMTimeScale, flags: CMTimeFlags, epoch: CMTimeEpoch) } extension CMTime { public init(seconds: Double, preferredTimescale: CMTimeScale) public init(value: CMTimeValue, timescale: CMTimeScale) } value: ϑϨʔϜ਺ timescale: 1ඵ͋ͨΓΛԿϑϨʔϜʹ෼ׂ͢Δ͔ʁ
48. CMTimeͷσʔλͷྫ 7JEFP CMTime(value: 1, timeScale: 30) "VEJP CMTime(value: 1024, timeScale: 44100) ≒0.33333…
49. ΩʔϑϨʔϜૹ৴ִؒ ֤αʔϏεͷਪ঑KeyFrame Interval͸2ඵ ɾYouTube ɾTwitch ɾOPENREC.tv ɾFRESH LIVE 1ͭͷts͋ͨΓΛ2ඵͷಈըʹ͍ͨ͠
50. audioϑϨʔϜ
51. CMSampleBuffer(audioMic) ϑϨʔϜͷ৘ใؚ͕·Ε͍ͯΔ ɾԻ੠ͷੜσʔλ ɾLinear PCM ɾ࠶ੜ։࢝࣌ؒ(Presentation Time Stamp) ɾ௕͞(Duration)
52. NT NT NT NT NT CMSampleBuffer(audioMic) 154 㽈 㽈 㽈 㽈 t 㽈 αϯϓϦϯάλΠϛϯά͸ఆظతɻ ִؒ͸1024/44100ඵ(≒23ms) ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS
53. NT NT NT NT NT CMSampleBuffer(audioMic) 154 㽈 㽈 㽈 㽈 t 㽈 ϏσΦόοϑΝ͸ࠓͷදࣔͷόοϑΝʹର͠ ΦʔσΟΦόοϑΝ͸ࠓ·Ͱ࿥Ի͞ΕͨόοϑΝΛड͚औΔ 㽈 㽈 㽈 㽈 㽈 औಘ
    λΠϛϯά ᶃ ᶄ ᶅ ᶆ ᶇ ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS
54. NT NT NT NT NT CMSampleBuffer(audioApp) 154 㽈 㽈 㽈 㽈 t 㽈 ϚΠΫಉ༷αϯϓϦϯάλΠϛϯά͸΄΅ఆظతɻ ִؒ͸22050/44100ඵલޙ(≒500ms) ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS
55. ө૾ͷฤू
56. ө૾ͷฤू ө૾ͷฤूͱ͍ͬͯ΋CIImageʹΑΔฤू ·ͣCMSampleBuffer͔ΒCIImageܗࣜʹม׵͢Δ extension CMSampleBuffer { var ciImage: CIImage? { guard let pixelBuffer = CMSampleBufferGetImageBuffer(self) else { return nil } CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.readOnly); defer { CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.readOnly) } return CIImage(cvPixelBuffer: pixelBuffer) } } CMSampleBuffer > CVPixelBuffer > CIImage
57. CIImageͰͰ͖Δ͜ͱ ɾ֦େ/ॖখ(CGAffineTransform) ɾҠಈ(CGAffineTransform) ɾ੾ΓऔΓ(crop) ɾճస(oriented) ɾ΅͔͠(applyingGaussianBlur) ɾϑΟϧλʔ
58. CMSampleBufferʹ໭͢ // BlurΛద༻͢ΔͳͲͷฤू let newImage = sampleBuffer.ciImage.applyingGaussianBlur(sigma: 10) // ۣܗ৘ใ let dimensions = CMVideoFormatDescriptionGetDimensions(CMSampleBufferGetFormatDescription(sampleBuffer)!) // ϐΫηϧ৘ใͷॻ͖ࠐΈઌͷόοϑΝΛ࡞੒ var outputPixelBuffer: CVPixelBuffer? = nil CVPixelBufferCreate(kCFAllocatorSystemDefault, Int(dimensions.width), Int(dimensions.height), kCVPixelFormatType_32BGRA, nil, &outputPixelBuffer) CVPixelBufferLockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)); defer { CVPixelBufferUnlockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)) } // όοϑΝʹCIImageΛॻ͖ࠐΈ let ciContext = ContextHolder.shared.context ciContext.render(newImage, to: outputPixelBuffer!) // ৽͍͠SampleBufferΛ࡞੒ var videoFormatDescription: CMVideoFormatDescription? = nil CMVideoFormatDescriptionCreateForImageBuffer(nil, outputPixelBuffer!, &videoFormatDescription) var sampleTimingInfo = self.sampleTimingInfo var newSampleBuffer: CMSampleBuffer? = nil CMSampleBufferCreateForImageBuffer(nil, outputPixelBuffer!, true, nil, nil, videoFormatDescription!, &sampleTimingInfo, &newSampleBuffer)
59. // BlurΛద༻͢ΔͳͲͷฤू let newImage = sampleBuffer.ciImage.applyingGaussianBlur(sigma: 10) CMSampleBuffer͔ΒCIImageΛऔಘ͠ɺ ϑΟϧλʔͳͲΛద༻͢Δ CMSampleBufferʹ໭͢
60. var outputPixelBuffer: CVPixelBuffer? = nil CVPixelBufferCreate(kCFAllocatorSystemDefault, Int(dimensions.width), Int(dimensions.height), kCVPixelFormatType_32BGRA, nil, &outputPixelBuffer) CVPixelBufferLockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)); defer { CVPixelBufferUnlockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)) } CIImageͷॻ͖ࠐΈઌͷόοϑΝΛ࡞੒͢Δ CMSampleBufferʹ໭͢ᶃ
61. // όοϑΝʹCIImageΛॻ͖ࠐΈ let ciContext = ContextHolder.shared.context ciContext.render(newImage, to: outputPixelBuffer!) ฤूͨ͠CIImageΛόοϑΝʹॻ͖ࠐΉ CMSampleBufferʹ໭͢ᶄ
62. var videoFormatDescription: CMVideoFormatDescription? = nil CMVideoFormatDescriptionCreateForImageBuffer(nil, outputPixelBuffer!, &videoFormatDescription) var sampleTimingInfo = self.sampleTimingInfo var newSampleBuffer: CMSampleBuffer? = nil CMSampleBufferCreateForImageBuffer(nil, outputPixelBuffer!, true, nil, nil, videoFormatDescription!, &sampleTimingInfo, &newSampleBuffer) ॻ͖ࠐΜͩBufferͱݩͷCMSampleBufferͷ ࣌ؒ৘ใΛ࢖ͬͯ৽ͨͳCMSampleBufferΛੜ੒͢Δ CMSampleBufferʹ໭͢ᶅ
63. ө૾ͷฤू ֖ֆʹࠩ͠ସ͑
64. ͍ͨͩ·४උதʂ ͍ΘΏΔɺྲྀͤΔө૾͕ͳ͍ ࣌ʹ”֖”Λ͢ΔͨΊͷը૾ͷ ͜ͱ ֖ֆͱ͸
65. ɾύεϫʔυೖྗ ɾݸਓ৘ใͱͳΔΞϓϦΛ։͘ ɾʮҰ౓഑৴Λऴྃ͢Ε͹͍͍ʯ͸ඍົ ΅͔͠Λݕ౼͕ͨ͠ɺ ύεϫʔυೖྗͰ͸ ΩʔϘʔυͷҐஔ͔Β ೖྗ͕ਪଌͰ͖ΔͨΊ ׬શͳ֖ֆʹ ֖ֆͷར༻γʔϯ
66. ɾ֖ֆͷը૾৘ใΛ࣋ͬͨSampleBufferΛىಈ࣌ʹੜ੒͓ͯ͘͠ ɾը໘ऩ࿥ϓϩηε͔Βड͚औͬͨSampleBufferͷPTSʹࠩ͠ସ ֖͑ͯֆͷόοϑΝΛૹ৴͢Δ ը໘ऩ࿥͔Β ड͚औͬͨόοϑΝ ֖ֆ όοϑΝ PTSΛ ΋Β͏ ഑৴ج൫ όοϑΝͷࠩ͠ସ͑
67. NT NT NT NT NT NT t 154 'SBNF ڞ༗OFF όοϑΝͷࠩ͠ସ͑
68. NT NT NT NT NT NT t 154 'SBNF ڞ༗OFF όοϑΝͷࠩ͠ସ͑
69. όοϑΝͷࠩ͠ସ͑ ֖ֆͷCMSampleBufferΛϗϧμʔʹอ͓͖࣋ͯ͠ ετϦʔϜͷϋϯυϦϯάଆͰόοϑΝΛࠩ͠ସ͑ class MaskHolder { static let shared = MaskHolder() private let maskBuffer: CMSampleBuffer func mask(with sampleTimingInfo: CMSampleTimingInfo) -> CMSampleBuffer { return maskBuffer.copy(with: sampleTimingInfo) } } class BroadcastService { var isEnabledMask: Bool let rtmpStream: RTMPStream func processSampleBuffer(sampleBuffer: CMSampleBuffer) { let buffer = isEnabledMask ? MaskHolder.shared.mask(with: sampleBuffer.sampleTimingInfo) : sampleBuffer rtmpStream.appendSampleBuffer(sampleBuffer: buffer) } }
70. extension CMSampleBuffer { func copy(with timing: CMSampleTimingInfo) -> CMSampleBuffer? { var sampleTiming = timing var copied: CMSampleBuffer? = nil CMSampleBufferCreateCopyWithNewTiming(kCFAllocatorSystemDefault, self, 1, &sampleTiming, &copied) return copied } var sampleTimingInfo: CMSampleTimingInfo { return CMSampleTimingInfo(duration: self.duration, presentationTimeStamp: self.presentationTimeStamp, decodeTimeStamp: self.decodeTimeStamp) } var duration: CMTime { return CMSampleBufferGetDuration(self) } var presentationTimeStamp: CMTime { return CMSampleBufferGetPresentationTimeStamp(self) } var decodeTimeStamp: CMTime { return CMSampleBufferGetDecodeTimeStamp(self) } } CMSampleBuffer͸ ؔ਺ݺͼग़͠ʹΑΔϓϩύςΟऔಘ͕ଟ͍ͷͰ extensionʹcomputed propertyΛੜ΍͓ͯ͘͠ͱศར όοϑΝͷࠩ͠ସ͑
71. ө૾ͷճస
72. iOSͷը໘͸഑৴தʹํ޲͕มΘΔ ͜ΕΛͲ͏ѻ͏͔ͱ͍͏τϐοΫ ήʔϜ͸ԣը໘΋ଟ͍ɻԻήʔ΍ΧʔυήʔϜͳͲɻ ө૾ͷճసͱ͸ʁ
73. ɾCMSampleBufferͷத਎͸ॎ௕ͷը૾ ɾCMSampleBufferͷϝλ৘ใʹճస৘ใؚ͕·Ε͍ͯΔ ճస৘ใͷऔಘ
74. extension CMSampleBuffer { var orientation: CGImagePropertyOrientation { guard let attachment = CMGetAttachment(self, RPVideoSampleOrientationKey, nil) as? NSNumber else { return .up } if let orientation = CGImagePropertyOrientation(rawValue: orientationAttachment.uint32Value) { return orientation } return .up } } VQ MFGU SJHIU EPXO https://developer.apple.com/documentation/imageio/cgimagepropertyorientation CGImagePropertyOrientation
75. ɾυΩϡϝϯτͰ͸iOS11.0+͕ͩ… ɾiOS11.2.1ҎલͰ͸ىಈ࣌ʹϦϯΫΤϥʔͰڧ੍ऴྃ ɾiOS11.2.5Ҏ߱͸໰୊ͳ͠ CMGetAttachment(self, "RPSampleBufferVideoOrientation" as CFString, nil) RPVideoSampleOrientationKey ɾ11.2.5Ҏ߱ͰऔಘͰ͖Δఆ਺Λϕλଧͪ ɾ11.2.1ҎલͰ͸nil͕औಘ͞ΕΔ ɾ11.2.1ҎલͰ΋ىಈͰ͖Δ͜ͱ͕ॏཁ https://developer.apple.com/documentation/replaykit/rpvideosampleorientationkey
76. ഑৴։࢝࣌ʹํ޲ͱղ૾౓Λݻఆɺ ը૾Λճస(CoreImageͷoriented)ͤͯ͞Scale Fit͢Δ ѻ͍ํ1 ॎݻఆ ԣݻఆ
77. ϝϦοτ σϝϦοτ ɾࢹௌऀ͕ݟ΍͍͢ํ޲ʹճస͍ͯ͠Δ ɾϑϨʔϜͷม׵͕ੜ͡Δ ɾ༨ന͕ೖͬͯ͠·͏ ɾෳ਺ͷΞϓϦͷߦ͖དྷΛ͢Δͱ͖ʹɺ ͜ͷඳըʹͳΔ͜ͱΛ഑৴ऀ͕ؾʹ͠ͳ͚Ε͹ͳΒͳ͍ ѻ͍ํ1
78. ಈը͸ॎݻఆʹͯ͠ɺө૾͸ճసͤ͞ͳ͍ɻ αʔόʔʹճస৘ใΛૹΓɺϓϨʔϠʔଆͰճసΛαϙʔτ up right left down ѻ͍ํ2
79. ϝϦοτ σϝϦοτ ɾ༨ന͕ͳ͍ ɾճసॲཧͷඞཁ͕ͳ͍ ɾࢹௌऀ͕ݟΔॠؒʹΑͬͯ͸ํ޲͕Ϛον͠ͳ͍ ɾϓϨΠϠʔ(ϓϥοτϑΥʔϜ)ຖʹରԠ͕ඞཁ ίετ͕ߴ͍ը૾ॲཧΛ1ͭল͚Δˍڐ༰Ͱ͖ΔσϝϦοτ ѻ͍ํ2
80. ഑৴ͷऴྃ
81. ഑৴ͷऴྃγʔέϯε •WebSocketΛ੾Δ •RTMPΛ੾அ͢Δ •αʔόʔʹ഑৴ऴྃAPIΛ౤͛Δ •AppͱExtensionͷڞ௨ྖҬʹอଘ͍ͯ͠Δ഑৴ʹؔ ࿈͢ΔσʔλΛ࡟আ͢Δ •ҟৗऴྃͰ͋Ε͹ͦͷঢ়ଶΛอଘ͢Δ ഑৴ͷऴྃAPIͱσʔλ࡟আ͸τϥϯβΫγϣϯ
82. DispatchGroup override open func broadcastFinished() { let dispathGroup = DispatchGroup() dispathGroup.enter() broadcastService?.finish { dispathGroup.leave() } dispathGroup.wait() } DispatchGroupͰεϨουΛఀࢭͤ͞Δ
83. BroadcastService func finish(completion: (()->Void)) { api.finishStream(movieID: 123) { completion?() } } αʔϏε಺ͰAPIΛड͚औͬͨΒcompletionΛ࣮ߦ
84. ը໘ऩ࿥Λऴྃͤ͞Δํ๏ •εςʔλεόʔ͔Βऴྃ •ίϯτϩʔϧηϯλʔ͔Βऴྃ •ΞϓϦ͔Βऴྃ
85. ΞϓϦέʔγϣϯ͔Βऴྃ override open func finishBroadcastWithError(_ error: Error) { super.finishBroadcastWithError(error) } RPBroadcastSampleHandlerͰ Τϥʔऴྃͤ͞Δඞཁ͕͋Δ https://developer.apple.com/documentation/replaykit/rpbroadcastsamplehandler/2721526-finishbroadcastwitherror
86. ഑৴ͷҟৗऴྃͱ࠶։
87. ഑৴Λҟৗऴྃͤ͞Δ override open func finishBroadcastWithError(_ error: Error) { super.finishBroadcastWithError(error) } ద੾ʹΤϥʔΛૹΔ͜ͱͰɺ μΠΞϩάʹऴྃཧ༝͕දࣔͰ͖Δ
88. ഑৴Λҟৗऴྃͤ͞Δ let message = “ωοτϫʔΫ͕੾அ͞Ε·ͨ͠" let error = NSError( domain: Bundle.main.bundleIdentifier ?? “" , code: 1 , userInfo: [NSLocalizedFailureReasonErrorKey: message]) finishBroadcastWithError(error) userInfoʹϝοηʔδΛࢦఆ͠ͳ͍ͱ ऴྃཧ༝͕(null)ʹͳΓ·͢ɻ
89. ҙਤ͠ͳ͍ऴྃͱͯ͠ ૝ఆ͞ΕΔέʔε •ը໘͕ϩοΫ͞ΕΔ(࣋ͪํͰిݯϘλϯΛԡͨ͠) •ి஑͕੾ΕΔ •ి࿩͕དྷΔ •BadAccessͰམͪΔ
90. ҟৗ഑৴͕ఀࢭͨ͠৔߹ɺ࠶ ։͍ͤͯ͋͛ͨ͞ •഑৴ͷ਺ࣈͷ෼ࢄ(ࢹௌ਺΍ίϝϯτɺΤʔϧ) •ࢹௌऀͷମݧ •৽͍͠഑৴࿮΁ͷҠಈʹΑΔτϥϑΟοΫͷε ύΠΫ
91. ి஑੾ΕɺిݯOFFɺϩοΫ • RPBroadcastSampleHandlerͷfinish()͕ݺ͹ΕΔ • ਖ਼ৗऴྃѻ͍ʹ͢Δ͔͠ͳ͍ • OSͷ௨஌͸͋Δ͕ඇެ։APIͷͨΊϦδΣΫτ͞ΕΔ
92. ௨࿩ͷண৴ iOS͸௨࿩Λண৴͢Δͱɺ RPBroadcastSampleHandlerͷfinish͕ݺ͹ΕΔ ͦͷͨΊɺਖ਼ৗܥͷ഑৴ऴྃγʔέϯεʹೖͬͯ ͠·͏ ண৴Λݕ஌ͨ࣌͠͸ɺ finish͕ݺ͹Εͯ΋தஅѻ͍ʹ͢Δ
93. ௨࿩ͷண৴ import CallKit class SampleHandler: RPBroadcastSampleHandler { let callObserver = CXCallObserver() override init() { super.init() callObserver.setDelegate(self, queue: nil) } } extension SampleHandler: CXCallObserverDelegate { func callObserver(…) { broadcastService.suspendStreaming() // ഑৴Λதஅ͢Δ } }
94. ωοτϫʔΫͷ੾அ • Reachability.swiftΛ࠾༻ • RxͰ؂ࢹ͓ͯ͘͠ • ωοτϫʔΫ͕੾Εͨ৔߹͸ҟৗऴྃͱ͠ɺ ୺຤ʹதஅঢ়ଶΛอଘ͢Δ
95. AppͱExtensionͷσʔλڞ༗
96. ڞ༗͢Δσʔλ • ೝূ৘ใ • ഑৴ઃఆ(ը࣭) • ഑৴࿮(഑৴ج൫ͷ઀ଓઌͳͲͷ৘ใ) • ୺຤ઃఆ(௨஌ɺ֖ֆ) • தஅ৘ใ
97. AppGroupΛ࢖ͬͯσʔλΛ ڞ༗͢Δ • ෳ਺ͷΞϓϦؒͰͷσʔλڞ༗ઃఆ • ಉ͡σϕϩούʔͷΞϓϦؒͰ༗ޮ • AppID͝ͱʹ༗ޮԽ͠ɺEntitlementͳͲͷ ઃఆ͕ඞཁ let userDefaults: UserDefaults = UserDefaults(suiteName: “groups.jp.iosdc2018”) IDΛࢦఆ͢Δ͜ͱͰάϧʔϓؒͷڞ༗ྖҬʹΞΫηεͰ͖Δ
98. ExtensionΛσόοά͢Δ
99. ϒϨΠΫϙΠϯτͷషΓํ
100. ϒϨΠΫϙΠϯτͷషΓํ
101. Waitingঢ়ଶʹͳ͔ͬͯΒɺ ഑৴Λ։࢝͢Δ
102. Appͱಉ͡Α͏ʹΠϯεϖΫ τͰ͖·͢
103. Appͱಉ͡Α͏ʹΠϯεϖΫ τͰ͖·͢
104. ΩϟϓνϟͷεϨου BreakpointͰࢭΊͯΔͱ ഑৴ϓϩηε͕ࢮʹ·͢
105. ͳͷͰLoggerΛ࢖͍·͠ΐ͏ • TCP Socket Logger • Logboard/LogboardConsole • JustLog • File Logger • XCGLogger • SwiftyBeaver
106. ·ͱΊ
107. ·ͱΊ • ReplayKitʹΑͬͯΧδϡΞϧʹੜ์ૹαʔϏε ͕࣮૷Ͱ͖ΔΑ͏ʹͳͬͨʂ • CMSampleBufferΛฤू͢Δ͜ͱͰө૾Λࠩ͠ସ ͑ͨΓΤϑΣΫτΛ͔͚ͨΓ͢Δ͜ͱ͕Ͱ͖Δ! • ഑৴தʹ୺຤͕ى͜ΓಘΔΠϕϯτͷϋϯυϦϯ ά͸ΞϓϦʹൺ΂ͯൈ͚ͳ͘΍Βͳͯ͘͸ͳΒͳ ͍
108. Thanks • HaishinKit(https://github.com/ shogo4405/HaishinKit.swift)