Live Streaming with Screen Recording

Transcript

1. ೋบ͘Β͍͋Δ
   ը໘ऩ࿥͔Βͷੜ์ૹ NBUTVPLBI

2. 01&/3&$UW
   $ZCFS;
   *OD $ZCFS"HFOU
   *OD
   5XJUUFS
   !NBUTVPLBI@
   (JUIVC
   NBUVPLBI
   झຯ
   ήʔϜ࣮گΛݟΔ͜ͱɺ࣮گੜ์ૹ͢Δ͜ͱ
   ಛʹ16#(

   
   4QMBUPPO দԬलथ

3. • https://openrec.tv • ήʔϜಛԽͷੜ์ૹαʔϏε • MAU 200ສ • ߴը࣭ɺ௿஗Ԇ഑৴ɺ௥͍͔͚࠶ੜɺ̐ϲࠃޠ(CJK+E)ରԠ •

   Web/iOS/tvOS/Android/AndroidTV
4. None

5. ൃදʹҠΔલʹ

6. E CyberAgent group iOS BeerBash։࠵! ݄೔ Ր ɹ࣌෼։࢝ʢ࣌ΑΓड෇։࢝ʣ
   αΠόʔΤʔδΣϯτʢौ୩ϓϥΠϜϓϥβ'ɹΫϦΤΠςΟϒϥ΢ϯδʣ
   dɹ֓ཁઆ໌ɺסഋ
   

   dɹ׻ஊ
   dɹ৽ઃͷ৽نαʔϏε։ൃ૊৫ɹ$"54ʢΫϥΠΞϯτٕज़ྖҬΛݗҾ͢Δ૊৫ʣʹ͍ͭͯ
   dɹϚονϯάΞϓϦʮλοϓϧ஀ੜʯʹ͓͚Δ։ൃͷมԽ
   dɹ׻ஊ ͝Ԡื͸23ίʔυ͔Β ͪ͜Β͔ΒͷΤϯτϦʔ͸શһ͝ࢀՃ͍͚ͨͩ·͢ connpass͔ΒΤϯτϦʔͷ৔߹ɺࢀՃ͸நબͱͳΓ·͢ ฐάϧʔϓͷiOSDCొஃऀ͕ࢀՃ͢ΔɹBeerBashͰ͢ʂ দԬ΋ࢀՃ͠·͢ʂ ೔࣌ ৔ॴ ಺༰ ɹɹɹɹɹɹɹ
   
   
   
   גࣜձࣾαΠόʔΤʔδΣϯτ
   $MJFOU
   "EWBODFE
   5FDIOPMPHZ
   4UVEJP $"54 ɹҏ౻ɹګฏ
   
   ɹɹɹɹɹɹɹ
   
   
   
   גࣜձࣾϚονϯάΤʔδΣϯτɹ∁ڮɹ༏հ
   

7. ͜ͷൃදͰ͸Sli.doΛ࢖͍·͢ ൃදͷ్தʹSli.do͔ΒΞϯέʔτΛ͠ͳ͕Β࠷ޙʹൃද ్தͰ࣭໰ͨ͘͠ͳͬͨΒSli.doʹ! ࠷ޙͷ࣭໰λΠϜͰर͍·͢ Event Code: iosdc2018_openrec URL: https://app2.sli.do/event/0kwbnzqe

8. ͦΕͰ͸ൃදʹҠΓ·͢

9. ΞδΣϯμ •HLSͷ֓ཁ •ReplayKitͷ֓ཁ •ը໘ऩ࿥ͷϥΠϑαΠΫϧ •SampleBufferͷฤू •ੜ์ૹதͷૢ࡞ •഑৴ͷҟৗऴྃͱ࠶։ •iOS AppͱBroadcaster Upload

   Extensionͷσʔλڞ༗ •ExtensionΛσόοά͢Δ

10. HLSͷ֓ཁ

11. HLSͷ֓ཁ HTTP Live Streaming(RFC8216)ͷུ ϓϨʔϠʔ͕ϓϨΠϦετͱࡉ੾Εʹͳ͍ͬͯΔಈըΛ HTTPϦΫΤετͰμ΢ϯϩʔυ͠ɺ ϓϨΠϦετʹఆٛ͞Ε͍ͯΔॱʹ࿈ଓతʹ࠶ੜ͢Δ https://iosdc.com/playlist.m3u8 https://iosdc2018.com/1.ts https://iosdc2018.com/2.ts

    https://iosdc2018.com/3.ts ϓϨΠϦετ 2ඵຖͷಈը ≒mp4

12. playlit.m3u8ͷத਎ #EXTM3U #EXT-X-VERSION:3 #EXT-X-TARGETDURATION:2 #EXT-X-MEDIA-SEQUENCE:1 #EXT-X-PROGRAM-DATE-TIME: 2018-08-31T12:15:12.713+09:00 #EXTINF:2.0, 1.ts #EXTINF:2.0,

    2.ts #EXTINF:2.0, 3.ts #EXT-X-ENDLIST

13. ࢓༷ʹଇͬͨϑΝΠϧΛΞοϓϩʔυͯ͠ ϓϨΠϠʔʹμ΢ϯϩʔυͯ͠΋Β͏ CDNΛ༗ޮ׆༻Ͱ͖ΔͨΊ εέʔϥϏϦςΟ/ίετͷ໘Ͱ ੜ์ૹͷٕज़తϋʔυϧΛԼ͛ΒΕΔ HLSͷ࠷େͷϝϦοτ

14. HLSʹΑΔ ੜ์ૹαʔϏεͷશମ૾

15. ഑৴ΞϓϦ OBS ഑৴ج൫ ΞϓϦ frame .ts .m3u8 ഑৴ऀ HLSʹΑΔಈը഑৴ͷશମ૾ Storage/CDN

    ಈը৘ใ URI

16. ഑৴ج൫ͷ໾ׂ 1. ड͚औͬͨϝσΟΞͷϑϨʔϜΛ߹੒ͯ͠ ɹ਺ඵ୯ҐͷಈըΛੜ੒͠ɺtsϑΝΠϧͱͯ͠σϓϩΠ 2. σϓϩΠ͞Εͨಈը৘ใΛϓϨΠϦετʹ௥ه͢Δ ഑৴ΞϓϦ OBS ഑৴ج൫ frame

    .ts .m3u8 Storage/CDN

17. ഑৴ΞϓϦ OBS ഑৴ج൫ ΞϓϦ ಈը৘ใ ࢹௌऀ frame .ts .m3u8 ಈը৘ใ

    .ts .m3u8 ഑৴ऀ Storage/CDN HLSʹΑΔಈը഑৴ͷશମ૾ URI

18. ഑৴ΞϓϦ OBS Storage/CDN ഑৴ج൫ ΞϓϦ ಈը৘ใ ࢹௌऀ .ts .m3u8 frame

    .ts .m3u8 ಈը৘ใ ഑৴ऀ ࠓճѻ͏෦෼ ɾReplayKit ɾBroadcast Upload Extension URI

19. ReplayKitͷ֓ཁ

20. ReplayKitͷ֓ཁ(1) •iOSͷεΫϦʔϯΩϟϓνϟΛѻ͏SDK •ϚΠΫ΍ΞϓϦͷԻ΋࿥ԻՄೳ •iOS9͔Βར༻Մೳ •ήʔϜ΍ϖΠϯτܥΞϓϦɺVTuberΞϓϦ ʹ૊Έࠐ·Ε͍ͯΔ

21. iOS9 iOS10 iOS11 ΞϓϦʹ૊ΈࠐΜͰ࿥ը͕Ͱ͖Δ ૊ΈࠐΜͩΞϓϦ͕ ExtensionʹରԠ͍ͯ͠Δ഑৴ΞϓϦʹॲཧΛҕৡͰ͖Δ ඪ४Ͱը໘ऩ࿥͕࣮૷ ReplayKitͷ֓ཁ(2)

22. •഑৴ͷ࣮૷ΛϓϥοτϑΥʔϜଆʹ೚ͤΒΕΔ •୺຤ͷϥΠϑαΠΫϧͷϋϯυϦϯά͕ߟྀࡁΈ •ࣗલ࿥ըΑΓύϑΥʔϚϯε͕ྑ͍ •UI΍௨஌ͳͲ͕Ωϟϓνϟ͞ΕΔ ϝϦοτ σϝϦοτ ReplayKitΛ͔ͭ͏ϝϦσϝ

23. https://developer.apple.com/videos/play/wwdc2018/601/ WWDC2018Ͱ΋ൃද

24. ը໘ऩ࿥Λड͚෇͚Δʹ͸ Broadcast Upload Extension Λ࣮૷͢Ε͹OK

25. Broadcast Upload Extension λʔήοτΛ࡞੒

26. Broadcast Upload Extension λʔήοτΛ࡞੒ ͜ͷΑ͏ʹɺը໘ऩ࿥ͷબ୒ࢶʹ ΞϓϦ͕ग़ΔΑ͏ʹͳΓ·͢ɻ

27. ը໘ऩ࿥ͷϥΠϑαΠΫϧ

28. ը໘ऩ࿥ͷϥΠϑαΠΫϧ Start Recording Finish ϕʔεͱͳΔϥΠϑαΠΫϧ αʔϏεຖͷ࣮૷͕֤ॴʹ૊Έࠐ·ΕΔ RPBroadcastSampleHandlerͷ ϝιουͱඥ෇͘

29. class SampleHandler: RPBroadcastSampleHandler { override func broadcastStarted(…) { // ഑৴։࢝

    } override func processSampleBuffer(…) { // ը໘ऩ࿥ϓϩηε͔ΒΩϟϓνϟ͞ΕͨϑϨʔϜͷड͚औΓ } override func finishBroadcastWithError() { // ഑৴ऴྃ } override func finishBroadcastWithError(…) { // ҟৗऴྃ } } ΤϯτϦʔϙΠϯτ SampleHandler.swift https://developer.apple.com/documentation/replaykit/rpbroadcastsamplehandler

30. ֤ϥΠϑαΠΫϧຖͷΞΫγϣϯ Start Recording Finish εϖοΫνΣοΫ/ೝূ ഑৴ઌͷղܾ ഑৴ج൫ʹ઀ଓ Τϯίʔυͱ഑৴ ઀ଓΛ੾அ ഑৴த

    ഑৴։࢝ ഑৴ऴྃ ঢ়ଶ ঢ়ଶຖͷΞΫγϣϯ ୺຤ͷঢ়ଶΛ؂ࢹ

31. Start εϖοΫνΣοΫ/ೝূ ഑৴ઌͷղܾ ഑৴ج൫ʹ઀ଓ ɾ഑৴Ͱ͖ΔεϖοΫͷ୺຤͔νΣοΫ ɾೝূ৘ใɺ഑৴ݖݶνΣοΫ ɾதஅ͞Ε͍ͯΔ࿮͕͋Δ͔νΣοΫ ɾ഑৴࿮ͷ࡞੒ ɾ഑৴ج൫ͷΞΫηεઌΛαʔόʔ͔Βऔಘ ɾRTMPͷ઀ଓ(ө૾/Ի੠)

    ɾWebSocketͷ઀ଓ(ίϝϯτͷண৴)

32. Recording Τϯίʔυͱ഑৴ ɾϝσΟΞຖʹΤϯίʔυ(h264/AAC)͠ɺૹ৴ ɾόοϑΝͷฤू/ࠩ͠ସ͑ ୺຤ͷঢ়ଶΛ؂ࢹ ɾը໘ͷճస৘ใ ɾ഑৴ઃఆͷมߋ ɾը໘ڞ༗ΦϑϞʔυ ɾ௨஌ͷON/OFF ɾ௨࿩ͷண৴

    ɾ୺຤ͷϩοΫɺి஑੾Ε ɾωοτϫʔΫঢ়گͷ؂ࢹ ɾΞϓϦ಺͔Βͷऴྃ ಈըͷݟ͑ํ தஅ/ऴྃ/ϦτϥΠ

33. Finish ઀ଓΛ੾அ ɾRTMPΛ੾அ ɾWebSocketΛ੾அ ɾ഑৴ऴྃAPI ഑৴ͷઃఆ৘ใΛ࡟আ ɾAppGroup಺ͷ഑৴ઃఆ৘ใΛ࡟আ

34. Recordingதͷ ϝσΟΞαϯϓϦϯά

35. όοϑΝͷૹ৴ HaishinKitʹͯରԠͨ͠ͷͰࠓճ͸Τϯίʔυɾ RTMPʹΑΔόοϑΝͷૹ৴ʹ͸৮Ε·ͤΜɻ HaishinKit͸ϝσΟΞͷΤϯίʔυ͓ΑͼɺετϦʔϛϯάαʔόʔ ͱͷRTMP௨৴Λड͚࣋ͬͯ͘ΕΔϥΠϒϥϦͰ͢ɻ https://github.com/shogo4405/HaishinKit.swift

36. ը໘ऩ࿥ϓϩηε͔Βड͚औ ΔϝσΟΞσʔλ func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {

    // process each media data } ϝσΟΞͷੜσʔλɻόΠτ৘ใ͔Β ֤ϝσΟΞʹ෮ݩͯ͠ฤूͨ͠ΓΤϯίʔυͨ͠Γ͢Δ video, audioApp, audioMicͷ̏छྨ sampleBufferType sampleBuffer

37. videoϑϨʔϜ

38. SampleBuffer(video) ϑϨʔϜͷ৘ใؚ͕·Ε͍ͯΔ ɾը૾ͷੜσʔλ ɾը૾ͷճస৘ใ ɾը૾ͷαΠζ ɾ࠶ੜ։࢝࣌ؒ(Presentation Time Stamp)

39. NT NT NT NT NT NT SampleBuffer(video) t 154 #VGGFS

    αϯϓϦϯάλΠϛϯά͸ը໘ʹߋ৽͕͋ͬͨͱ͖ Duration͸લճͷαϯϓϦϯά͔ΒࠓճͷαϯϓϦϯά

40. ϗʔϜը໘ͷ഑৴Λ͠ͳ͕Β ์ஔ͍ͯͨ͠Β… ҆ఆ࠶ੜͰ͖ͳ͍ࣄ৅ʹؕͬͨ ΩʔϑϨʔϜΛੜ੒ִؒΛ2ඵʹ͍ͯ͠Δʹ΋ ؔΘΒͣ6ඵʹͳͬͯ͠·ͬͨΓ 1ts͋ͨΓಈըͷ୹͞΍ ҆ఆ͠ͳ͍tsͷduration͸ϥάʹ௚݁͢Δ

41. NT NT ը໘ͷߋ৽͕ͳ͍৔߹ 154 t #VGGFS ͜ͷؒ͸ߋ৽͕ͳ͘ϑϨʔϜ͕ૹΒΕͣ ΩʔϑϨʔϜ΋ૹΕͳ͍ͨΊ ಈըͷ۠੾Γ͕͏·͘Ͱ͖ͳ͍

42. ಈըΛ࠶ݱ͢ΔͨΊʹbitmap৘ใΛ͢΂ؚͯΜͩϑϨʔϜ ࣮ࡍͷ ϑϨʔϜ t ಈը ΩʔϑϨʔϜ(I Frame)ͱ͸

43. ࣮ࡍͷ ϑϨʔϜ I Frame P Frame ࠩ෼ͷΈΛ࣋ͭ͜ͱͰσʔλྔΛѹॖ HLSʹ͓͍ͯI Frame͸1ͭͷtsϑΝΠϧຖʹ ࠷ॳͷϑϨʔϜͱؚͯ͠ΜͰ͍ͳ͚Ε͹ͳΒͳ͍

    ΩʔϑϨʔϜ(I Frame)ͱ͸ ಈըΛ࠶ݱ͢ΔͨΊʹbitmap৘ใΛ͢΂ؚͯΜͩϑϨʔϜ

44. NT NT 154 t #VGGFS videoϑϨʔϜͷϦΧόϦͰରԠ ؒΛຒΊΔϑϨʔϜΛ͍ΕΕ͹OK

45. t NT NT NT NT NT NT videoϑϨʔϜͷϦΧόϦͰରԠ 154 ϑϨʔϜΛૹ৴ͯ͠Ұఆ͕࣌ؒܦͬͯ΋࣍ͷϑϨʔϜ͕ͳ͍࣌

    ࠷ޙʹૹ৴ͨ͠ϑϨʔϜΛɺPTSΛ෇͚ସ͑ͯૹ৴͢Δ #VGGFS

46. PTS͕ॏཁ ɾPTSͷζϨ͸ಈըͷԻζϨͷݪҼ ɾόοϑΝͷૢ࡞Ͱ΋͜ͷPTSΛѻ͏ ɾPTSؚΊɺಈը΍Ի੠ͷ࣌ؒ৘ใ͸CMTimeʹΑͬ ͯϑϨʔϜͰѻ͑Δܗࣜʹͳ͍ͬͯΔ

47. CMTime ࣌ؒ৘ใΛ཭ࢄత(ϑϨʔϜ)ʹ੔਺Ͱѻ͏ public struct CMTime { public var value: CMTimeValue

    public var timescale: CMTimeScale public var flags: CMTimeFlags public var epoch: CMTimeEpoch public init() public init(value: CMTimeValue, timescale: CMTimeScale, flags: CMTimeFlags, epoch: CMTimeEpoch) } extension CMTime { public init(seconds: Double, preferredTimescale: CMTimeScale) public init(value: CMTimeValue, timescale: CMTimeScale) } value: ϑϨʔϜ਺ timescale: 1ඵ͋ͨΓΛԿϑϨʔϜʹ෼ׂ͢Δ͔ʁ

48. CMTimeͷσʔλͷྫ 7JEFP CMTime(value: 1, timeScale: 30) "VEJP CMTime(value: 1024, timeScale:

    44100) ≒0.33333…

49. ΩʔϑϨʔϜૹ৴ִؒ ֤αʔϏεͷਪ঑KeyFrame Interval͸2ඵ ɾYouTube ɾTwitch ɾOPENREC.tv ɾFRESH LIVE 1ͭͷts͋ͨΓΛ2ඵͷಈըʹ͍ͨ͠

50. audioϑϨʔϜ

51. CMSampleBuffer(audioMic) ϑϨʔϜͷ৘ใؚ͕·Ε͍ͯΔ ɾԻ੠ͷੜσʔλ ɾLinear PCM ɾ࠶ੜ։࢝࣌ؒ(Presentation Time Stamp) ɾ௕͞(Duration)

52. NT NT NT NT NT CMSampleBuffer(audioMic) 154 㽈 㽈 㽈

    㽈 t 㽈 αϯϓϦϯάλΠϛϯά͸ఆظతɻ ִؒ͸1024/44100ඵ(≒23ms) ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS

53. NT NT NT NT NT CMSampleBuffer(audioMic) 154 㽈 㽈 㽈

    㽈 t 㽈 ϏσΦόοϑΝ͸ࠓͷදࣔͷόοϑΝʹର͠ ΦʔσΟΦόοϑΝ͸ࠓ·Ͱ࿥Ի͞ΕͨόοϑΝΛड͚औΔ 㽈 㽈 㽈 㽈 㽈 औಘ
    λΠϛϯά ᶃ ᶄ ᶅ ᶆ ᶇ ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS

54. NT NT NT NT NT CMSampleBuffer(audioApp) 154 㽈 㽈 㽈

    㽈 t 㽈 ϚΠΫಉ༷αϯϓϦϯάλΠϛϯά͸΄΅ఆظతɻ ִؒ͸22050/44100ඵલޙ(≒500ms) ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS

55. ө૾ͷฤू

56. ө૾ͷฤू ө૾ͷฤूͱ͍ͬͯ΋CIImageʹΑΔฤू ·ͣCMSampleBuffer͔ΒCIImageܗࣜʹม׵͢Δ extension CMSampleBuffer { var ciImage: CIImage? {

    guard let pixelBuffer = CMSampleBufferGetImageBuffer(self) else { return nil } CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.readOnly); defer { CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.readOnly) } return CIImage(cvPixelBuffer: pixelBuffer) } } CMSampleBuffer > CVPixelBuffer > CIImage

57. CIImageͰͰ͖Δ͜ͱ ɾ֦େ/ॖখ(CGAffineTransform) ɾҠಈ(CGAffineTransform) ɾ੾ΓऔΓ(crop) ɾճస(oriented) ɾ΅͔͠(applyingGaussianBlur) ɾϑΟϧλʔ

58. CMSampleBufferʹ໭͢ // BlurΛద༻͢ΔͳͲͷฤू let newImage = sampleBuffer.ciImage.applyingGaussianBlur(sigma: 10) // ۣܗ৘ใ

    let dimensions = CMVideoFormatDescriptionGetDimensions(CMSampleBufferGetFormatDescription(sampleBuffer)!) // ϐΫηϧ৘ใͷॻ͖ࠐΈઌͷόοϑΝΛ࡞੒ var outputPixelBuffer: CVPixelBuffer? = nil CVPixelBufferCreate(kCFAllocatorSystemDefault, Int(dimensions.width), Int(dimensions.height), kCVPixelFormatType_32BGRA, nil, &outputPixelBuffer) CVPixelBufferLockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)); defer { CVPixelBufferUnlockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)) } // όοϑΝʹCIImageΛॻ͖ࠐΈ let ciContext = ContextHolder.shared.context ciContext.render(newImage, to: outputPixelBuffer!) // ৽͍͠SampleBufferΛ࡞੒ var videoFormatDescription: CMVideoFormatDescription? = nil CMVideoFormatDescriptionCreateForImageBuffer(nil, outputPixelBuffer!, &videoFormatDescription) var sampleTimingInfo = self.sampleTimingInfo var newSampleBuffer: CMSampleBuffer? = nil CMSampleBufferCreateForImageBuffer(nil, outputPixelBuffer!, true, nil, nil, videoFormatDescription!, &sampleTimingInfo, &newSampleBuffer)

59. // BlurΛద༻͢ΔͳͲͷฤू let newImage = sampleBuffer.ciImage.applyingGaussianBlur(sigma: 10) CMSampleBuffer͔ΒCIImageΛऔಘ͠ɺ ϑΟϧλʔͳͲΛద༻͢Δ CMSampleBufferʹ໭͢

60. var outputPixelBuffer: CVPixelBuffer? = nil CVPixelBufferCreate(kCFAllocatorSystemDefault, Int(dimensions.width), Int(dimensions.height), kCVPixelFormatType_32BGRA, nil,

    &outputPixelBuffer) CVPixelBufferLockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)); defer { CVPixelBufferUnlockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)) } CIImageͷॻ͖ࠐΈઌͷόοϑΝΛ࡞੒͢Δ CMSampleBufferʹ໭͢ᶃ

61. // όοϑΝʹCIImageΛॻ͖ࠐΈ let ciContext = ContextHolder.shared.context ciContext.render(newImage, to: outputPixelBuffer!) ฤूͨ͠CIImageΛόοϑΝʹॻ͖ࠐΉ

    CMSampleBufferʹ໭͢ᶄ

62. var videoFormatDescription: CMVideoFormatDescription? = nil CMVideoFormatDescriptionCreateForImageBuffer(nil, outputPixelBuffer!, &videoFormatDescription) var sampleTimingInfo

    = self.sampleTimingInfo var newSampleBuffer: CMSampleBuffer? = nil CMSampleBufferCreateForImageBuffer(nil, outputPixelBuffer!, true, nil, nil, videoFormatDescription!, &sampleTimingInfo, &newSampleBuffer) ॻ͖ࠐΜͩBufferͱݩͷCMSampleBufferͷ ࣌ؒ৘ใΛ࢖ͬͯ৽ͨͳCMSampleBufferΛੜ੒͢Δ CMSampleBufferʹ໭͢ᶅ

63. ө૾ͷฤू ֖ֆʹࠩ͠ସ͑

64. ͍ͨͩ·४උதʂ ͍ΘΏΔɺྲྀͤΔө૾͕ͳ͍ ࣌ʹ”֖”Λ͢ΔͨΊͷը૾ͷ ͜ͱ ֖ֆͱ͸

65. ɾύεϫʔυೖྗ ɾݸਓ৘ใͱͳΔΞϓϦΛ։͘ ɾʮҰ౓഑৴Λऴྃ͢Ε͹͍͍ʯ͸ඍົ ΅͔͠Λݕ౼͕ͨ͠ɺ ύεϫʔυೖྗͰ͸ ΩʔϘʔυͷҐஔ͔Β ೖྗ͕ਪଌͰ͖ΔͨΊ ׬શͳ֖ֆʹ ֖ֆͷར༻γʔϯ

66. ɾ֖ֆͷը૾৘ใΛ࣋ͬͨSampleBufferΛىಈ࣌ʹੜ੒͓ͯ͘͠ ɾը໘ऩ࿥ϓϩηε͔Βड͚औͬͨSampleBufferͷPTSʹࠩ͠ସ ֖͑ͯֆͷόοϑΝΛૹ৴͢Δ ը໘ऩ࿥͔Β ड͚औͬͨόοϑΝ ֖ֆ όοϑΝ PTSΛ ΋Β͏ ഑৴ج൫

    όοϑΝͷࠩ͠ସ͑

67. NT NT NT NT NT NT t 154 'SBNF ڞ༗OFF

    όοϑΝͷࠩ͠ସ͑

68. NT NT NT NT NT NT t 154 'SBNF ڞ༗OFF

    όοϑΝͷࠩ͠ସ͑

69. όοϑΝͷࠩ͠ସ͑ ֖ֆͷCMSampleBufferΛϗϧμʔʹอ͓͖࣋ͯ͠ ετϦʔϜͷϋϯυϦϯάଆͰόοϑΝΛࠩ͠ସ͑ class MaskHolder { static let shared =

    MaskHolder() private let maskBuffer: CMSampleBuffer func mask(with sampleTimingInfo: CMSampleTimingInfo) -> CMSampleBuffer { return maskBuffer.copy(with: sampleTimingInfo) } } class BroadcastService { var isEnabledMask: Bool let rtmpStream: RTMPStream func processSampleBuffer(sampleBuffer: CMSampleBuffer) { let buffer = isEnabledMask ? MaskHolder.shared.mask(with: sampleBuffer.sampleTimingInfo) : sampleBuffer rtmpStream.appendSampleBuffer(sampleBuffer: buffer) } }

70. extension CMSampleBuffer { func copy(with timing: CMSampleTimingInfo) -> CMSampleBuffer? {

    var sampleTiming = timing var copied: CMSampleBuffer? = nil CMSampleBufferCreateCopyWithNewTiming(kCFAllocatorSystemDefault, self, 1, &sampleTiming, &copied) return copied } var sampleTimingInfo: CMSampleTimingInfo { return CMSampleTimingInfo(duration: self.duration, presentationTimeStamp: self.presentationTimeStamp, decodeTimeStamp: self.decodeTimeStamp) } var duration: CMTime { return CMSampleBufferGetDuration(self) } var presentationTimeStamp: CMTime { return CMSampleBufferGetPresentationTimeStamp(self) } var decodeTimeStamp: CMTime { return CMSampleBufferGetDecodeTimeStamp(self) } } CMSampleBuffer͸ ؔ਺ݺͼग़͠ʹΑΔϓϩύςΟऔಘ͕ଟ͍ͷͰ extensionʹcomputed propertyΛੜ΍͓ͯ͘͠ͱศར όοϑΝͷࠩ͠ସ͑

71. ө૾ͷճస

72. iOSͷը໘͸഑৴தʹํ޲͕มΘΔ ͜ΕΛͲ͏ѻ͏͔ͱ͍͏τϐοΫ ήʔϜ͸ԣը໘΋ଟ͍ɻԻήʔ΍ΧʔυήʔϜͳͲɻ ө૾ͷճసͱ͸ʁ

73. ɾCMSampleBufferͷத਎͸ॎ௕ͷը૾ ɾCMSampleBufferͷϝλ৘ใʹճస৘ใؚ͕·Ε͍ͯΔ ճస৘ใͷऔಘ

74. extension CMSampleBuffer { var orientation: CGImagePropertyOrientation { guard let attachment

    = CMGetAttachment(self, RPVideoSampleOrientationKey, nil) as? NSNumber else { return .up } if let orientation = CGImagePropertyOrientation(rawValue: orientationAttachment.uint32Value) { return orientation } return .up } } VQ MFGU SJHIU EPXO https://developer.apple.com/documentation/imageio/cgimagepropertyorientation CGImagePropertyOrientation

75. ɾυΩϡϝϯτͰ͸iOS11.0+͕ͩ… ɾiOS11.2.1ҎલͰ͸ىಈ࣌ʹϦϯΫΤϥʔͰڧ੍ऴྃ ɾiOS11.2.5Ҏ߱͸໰୊ͳ͠ CMGetAttachment(self, "RPSampleBufferVideoOrientation" as CFString, nil) RPVideoSampleOrientationKey ɾ11.2.5Ҏ߱ͰऔಘͰ͖Δఆ਺Λϕλଧͪ

    ɾ11.2.1ҎલͰ͸nil͕औಘ͞ΕΔ ɾ11.2.1ҎલͰ΋ىಈͰ͖Δ͜ͱ͕ॏཁ https://developer.apple.com/documentation/replaykit/rpvideosampleorientationkey

76. ഑৴։࢝࣌ʹํ޲ͱղ૾౓Λݻఆɺ ը૾Λճస(CoreImageͷoriented)ͤͯ͞Scale Fit͢Δ ѻ͍ํ1 ॎݻఆ ԣݻఆ

77. ϝϦοτ σϝϦοτ ɾࢹௌऀ͕ݟ΍͍͢ํ޲ʹճస͍ͯ͠Δ ɾϑϨʔϜͷม׵͕ੜ͡Δ ɾ༨ന͕ೖͬͯ͠·͏ ɾෳ਺ͷΞϓϦͷߦ͖དྷΛ͢Δͱ͖ʹɺ ͜ͷඳըʹͳΔ͜ͱΛ഑৴ऀ͕ؾʹ͠ͳ͚Ε͹ͳΒͳ͍ ѻ͍ํ1

78. ಈը͸ॎݻఆʹͯ͠ɺө૾͸ճసͤ͞ͳ͍ɻ αʔόʔʹճస৘ใΛૹΓɺϓϨʔϠʔଆͰճసΛαϙʔτ up right left down ѻ͍ํ2

79. ϝϦοτ σϝϦοτ ɾ༨ന͕ͳ͍ ɾճసॲཧͷඞཁ͕ͳ͍ ɾࢹௌऀ͕ݟΔॠؒʹΑͬͯ͸ํ޲͕Ϛον͠ͳ͍ ɾϓϨΠϠʔ(ϓϥοτϑΥʔϜ)ຖʹରԠ͕ඞཁ ίετ͕ߴ͍ը૾ॲཧΛ1ͭল͚Δˍڐ༰Ͱ͖ΔσϝϦοτ ѻ͍ํ2

80. ഑৴ͷऴྃ

81. ഑৴ͷऴྃγʔέϯε •WebSocketΛ੾Δ •RTMPΛ੾அ͢Δ •αʔόʔʹ഑৴ऴྃAPIΛ౤͛Δ •AppͱExtensionͷڞ௨ྖҬʹอଘ͍ͯ͠Δ഑৴ʹؔ ࿈͢ΔσʔλΛ࡟আ͢Δ •ҟৗऴྃͰ͋Ε͹ͦͷঢ়ଶΛอଘ͢Δ ഑৴ͷऴྃAPIͱσʔλ࡟আ͸τϥϯβΫγϣϯ

82. DispatchGroup override open func broadcastFinished() { let dispathGroup = DispatchGroup()

    dispathGroup.enter() broadcastService?.finish { dispathGroup.leave() } dispathGroup.wait() } DispatchGroupͰεϨουΛఀࢭͤ͞Δ

83. BroadcastService func finish(completion: (()->Void)) { api.finishStream(movieID: 123) { completion?() }

    } αʔϏε಺ͰAPIΛड͚औͬͨΒcompletionΛ࣮ߦ

84. ը໘ऩ࿥Λऴྃͤ͞Δํ๏ •εςʔλεόʔ͔Βऴྃ •ίϯτϩʔϧηϯλʔ͔Βऴྃ •ΞϓϦ͔Βऴྃ

85. ΞϓϦέʔγϣϯ͔Βऴྃ override open func finishBroadcastWithError(_ error: Error) { super.finishBroadcastWithError(error) }

    RPBroadcastSampleHandlerͰ Τϥʔऴྃͤ͞Δඞཁ͕͋Δ https://developer.apple.com/documentation/replaykit/rpbroadcastsamplehandler/2721526-finishbroadcastwitherror

86. ഑৴ͷҟৗऴྃͱ࠶։

87. ഑৴Λҟৗऴྃͤ͞Δ override open func finishBroadcastWithError(_ error: Error) { super.finishBroadcastWithError(error) }

    ద੾ʹΤϥʔΛૹΔ͜ͱͰɺ μΠΞϩάʹऴྃཧ༝͕දࣔͰ͖Δ

88. ഑৴Λҟৗऴྃͤ͞Δ let message = “ωοτϫʔΫ͕੾அ͞Ε·ͨ͠" let error = NSError( domain:

    Bundle.main.bundleIdentifier ?? “" , code: 1 , userInfo: [NSLocalizedFailureReasonErrorKey: message]) finishBroadcastWithError(error) userInfoʹϝοηʔδΛࢦఆ͠ͳ͍ͱ ऴྃཧ༝͕(null)ʹͳΓ·͢ɻ

89. ҙਤ͠ͳ͍ऴྃͱͯ͠ ૝ఆ͞ΕΔέʔε •ը໘͕ϩοΫ͞ΕΔ(࣋ͪํͰిݯϘλϯΛԡͨ͠) •ి஑͕੾ΕΔ •ి࿩͕དྷΔ •BadAccessͰམͪΔ

90. ҟৗ഑৴͕ఀࢭͨ͠৔߹ɺ࠶ ։͍ͤͯ͋͛ͨ͞ •഑৴ͷ਺ࣈͷ෼ࢄ(ࢹௌ਺΍ίϝϯτɺΤʔϧ) •ࢹௌऀͷମݧ •৽͍͠഑৴࿮΁ͷҠಈʹΑΔτϥϑΟοΫͷε ύΠΫ

91. ి஑੾ΕɺిݯOFFɺϩοΫ • RPBroadcastSampleHandlerͷfinish()͕ݺ͹ΕΔ • ਖ਼ৗऴྃѻ͍ʹ͢Δ͔͠ͳ͍ • OSͷ௨஌͸͋Δ͕ඇެ։APIͷͨΊϦδΣΫτ͞ΕΔ

92. ௨࿩ͷண৴ iOS͸௨࿩Λண৴͢Δͱɺ RPBroadcastSampleHandlerͷfinish͕ݺ͹ΕΔ ͦͷͨΊɺਖ਼ৗܥͷ഑৴ऴྃγʔέϯεʹೖͬͯ ͠·͏ ண৴Λݕ஌ͨ࣌͠͸ɺ finish͕ݺ͹Εͯ΋தஅѻ͍ʹ͢Δ

93. ௨࿩ͷண৴ import CallKit class SampleHandler: RPBroadcastSampleHandler { let callObserver =

    CXCallObserver() override init() { super.init() callObserver.setDelegate(self, queue: nil) } } extension SampleHandler: CXCallObserverDelegate { func callObserver(…) { broadcastService.suspendStreaming() // ഑৴Λதஅ͢Δ } }

94. ωοτϫʔΫͷ੾அ • Reachability.swiftΛ࠾༻ • RxͰ؂ࢹ͓ͯ͘͠ • ωοτϫʔΫ͕੾Εͨ৔߹͸ҟৗऴྃͱ͠ɺ ୺຤ʹதஅঢ়ଶΛอଘ͢Δ

95. AppͱExtensionͷσʔλڞ༗

96. ڞ༗͢Δσʔλ • ೝূ৘ใ • ഑৴ઃఆ(ը࣭) • ഑৴࿮(഑৴ج൫ͷ઀ଓઌͳͲͷ৘ใ) • ୺຤ઃఆ(௨஌ɺ֖ֆ) •

    தஅ৘ใ

97. AppGroupΛ࢖ͬͯσʔλΛ ڞ༗͢Δ • ෳ਺ͷΞϓϦؒͰͷσʔλڞ༗ઃఆ • ಉ͡σϕϩούʔͷΞϓϦؒͰ༗ޮ • AppID͝ͱʹ༗ޮԽ͠ɺEntitlementͳͲͷ ઃఆ͕ඞཁ let

    userDefaults: UserDefaults = UserDefaults(suiteName: “groups.jp.iosdc2018”) IDΛࢦఆ͢Δ͜ͱͰάϧʔϓؒͷڞ༗ྖҬʹΞΫηεͰ͖Δ

98. ExtensionΛσόοά͢Δ

99. ϒϨΠΫϙΠϯτͷషΓํ

100. ϒϨΠΫϙΠϯτͷషΓํ

101. Waitingঢ়ଶʹͳ͔ͬͯΒɺ ഑৴Λ։࢝͢Δ

102. Appͱಉ͡Α͏ʹΠϯεϖΫ τͰ͖·͢

103. Appͱಉ͡Α͏ʹΠϯεϖΫ τͰ͖·͢

104. ΩϟϓνϟͷεϨου BreakpointͰࢭΊͯΔͱ ഑৴ϓϩηε͕ࢮʹ·͢

105. ͳͷͰLoggerΛ࢖͍·͠ΐ͏ • TCP Socket Logger • Logboard/LogboardConsole • JustLog •

     File Logger • XCGLogger • SwiftyBeaver

106. ·ͱΊ

107. ·ͱΊ • ReplayKitʹΑͬͯΧδϡΞϧʹੜ์ૹαʔϏε ͕࣮૷Ͱ͖ΔΑ͏ʹͳͬͨʂ • CMSampleBufferΛฤू͢Δ͜ͱͰө૾Λࠩ͠ସ ͑ͨΓΤϑΣΫτΛ͔͚ͨΓ͢Δ͜ͱ͕Ͱ͖Δ! • ഑৴தʹ୺຤͕ى͜ΓಘΔΠϕϯτͷϋϯυϦϯ ά͸ΞϓϦʹൺ΂ͯൈ͚ͳ͘΍Βͳͯ͘͸ͳΒͳ

     ͍

108. Thanks • HaishinKit(https://github.com/ shogo4405/HaishinKit.swift)