Live Streaming with Screen Recording

Live Streaming with Screen Recording

5405b3d871f38164ef419e95e9467197?s=128

Hideki Matsuoka

September 01, 2018
Tweet

Transcript

  1. ೋบ͘Β͍͋Δ ը໘ऩ࿥͔Βͷੜ์ૹ NBUTVPLBI

  2. 01&/3&$UW$ZCFS; *OD $ZCFS"HFOU *OD  5XJUUFS!NBUTVPLBI@ (JUIVCNBUVPLBI झຯ ήʔϜ࣮گΛݟΔ͜ͱɺ࣮گੜ์ૹ͢Δ͜ͱ ಛʹ16#(

    4QMBUPPO দԬलथ
  3. • https://openrec.tv • ήʔϜಛԽͷੜ์ૹαʔϏε • MAU 200ສ • ߴը࣭ɺ௿஗Ԇ഑৴ɺ௥͍͔͚࠶ੜɺ̐ϲࠃޠ(CJK+E)ରԠ •

    Web/iOS/tvOS/Android/AndroidTV
  4. None
  5. ൃදʹҠΔલʹ

  6. E CyberAgent group iOS BeerBash։࠵! ݄೔ Ր ɹ࣌෼։࢝ʢ࣌ΑΓड෇։࢝ʣ αΠόʔΤʔδΣϯτʢौ୩ϓϥΠϜϓϥβ'ɹΫϦΤΠςΟϒϥ΢ϯδʣ dɹ֓ཁઆ໌ɺסഋ

    dɹ׻ஊ dɹ৽ઃͷ৽نαʔϏε։ൃ૊৫ɹ$"54ʢΫϥΠΞϯτٕज़ྖҬΛݗҾ͢Δ૊৫ʣʹ͍ͭͯ dɹϚονϯάΞϓϦʮλοϓϧ஀ੜʯʹ͓͚Δ։ൃͷมԽ dɹ׻ஊ ͝Ԡื͸23ίʔυ͔Β ͪ͜Β͔ΒͷΤϯτϦʔ͸શһ͝ࢀՃ͍͚ͨͩ·͢ connpass͔ΒΤϯτϦʔͷ৔߹ɺࢀՃ͸நબͱͳΓ·͢ ฐάϧʔϓͷiOSDCొஃऀ͕ࢀՃ͢ΔɹBeerBashͰ͢ʂ দԬ΋ࢀՃ͠·͢ʂ ೔࣌ ৔ॴ ಺༰ ɹɹɹɹɹɹɹגࣜձࣾαΠόʔΤʔδΣϯτ$MJFOU"EWBODFE5FDIOPMPHZ4UVEJP $"54 ɹҏ౻ɹګฏ ɹɹɹɹɹɹɹגࣜձࣾϚονϯάΤʔδΣϯτɹ∁ڮɹ༏հ
  7. ͜ͷൃදͰ͸Sli.doΛ࢖͍·͢ ൃදͷ్தʹSli.do͔ΒΞϯέʔτΛ͠ͳ͕Β࠷ޙʹൃද ్தͰ࣭໰ͨ͘͠ͳͬͨΒSli.doʹ! ࠷ޙͷ࣭໰λΠϜͰर͍·͢ Event Code: iosdc2018_openrec URL: https://app2.sli.do/event/0kwbnzqe

  8. ͦΕͰ͸ൃදʹҠΓ·͢

  9. ΞδΣϯμ •HLSͷ֓ཁ •ReplayKitͷ֓ཁ •ը໘ऩ࿥ͷϥΠϑαΠΫϧ •SampleBufferͷฤू •ੜ์ૹதͷૢ࡞ •഑৴ͷҟৗऴྃͱ࠶։ •iOS AppͱBroadcaster Upload

    Extensionͷσʔλڞ༗ •ExtensionΛσόοά͢Δ
  10. HLSͷ֓ཁ

  11. HLSͷ֓ཁ HTTP Live Streaming(RFC8216)ͷུ ϓϨʔϠʔ͕ϓϨΠϦετͱࡉ੾Εʹͳ͍ͬͯΔಈըΛ HTTPϦΫΤετͰμ΢ϯϩʔυ͠ɺ ϓϨΠϦετʹఆٛ͞Ε͍ͯΔॱʹ࿈ଓతʹ࠶ੜ͢Δ https://iosdc.com/playlist.m3u8 https://iosdc2018.com/1.ts https://iosdc2018.com/2.ts

    https://iosdc2018.com/3.ts ϓϨΠϦετ 2ඵຖͷಈը ≒mp4
  12. playlit.m3u8ͷத਎ #EXTM3U #EXT-X-VERSION:3 #EXT-X-TARGETDURATION:2 #EXT-X-MEDIA-SEQUENCE:1 #EXT-X-PROGRAM-DATE-TIME: 2018-08-31T12:15:12.713+09:00 #EXTINF:2.0, 1.ts #EXTINF:2.0,

    2.ts #EXTINF:2.0, 3.ts #EXT-X-ENDLIST
  13. ࢓༷ʹଇͬͨϑΝΠϧΛΞοϓϩʔυͯ͠ ϓϨΠϠʔʹμ΢ϯϩʔυͯ͠΋Β͏ CDNΛ༗ޮ׆༻Ͱ͖ΔͨΊ εέʔϥϏϦςΟ/ίετͷ໘Ͱ ੜ์ૹͷٕज़తϋʔυϧΛԼ͛ΒΕΔ HLSͷ࠷େͷϝϦοτ

  14. HLSʹΑΔ ੜ์ૹαʔϏεͷશମ૾

  15. ഑৴ΞϓϦ OBS ഑৴ج൫ ΞϓϦ frame .ts .m3u8 ഑৴ऀ HLSʹΑΔಈը഑৴ͷશମ૾ Storage/CDN

    ಈը৘ใ URI
  16. ഑৴ج൫ͷ໾ׂ 1. ड͚औͬͨϝσΟΞͷϑϨʔϜΛ߹੒ͯ͠ ɹ਺ඵ୯ҐͷಈըΛੜ੒͠ɺtsϑΝΠϧͱͯ͠σϓϩΠ 2. σϓϩΠ͞Εͨಈը৘ใΛϓϨΠϦετʹ௥ه͢Δ ഑৴ΞϓϦ OBS ഑৴ج൫ frame

    .ts .m3u8 Storage/CDN
  17. ഑৴ΞϓϦ OBS ഑৴ج൫ ΞϓϦ ಈը৘ใ ࢹௌऀ frame .ts .m3u8 ಈը৘ใ

    .ts .m3u8 ഑৴ऀ Storage/CDN HLSʹΑΔಈը഑৴ͷશମ૾ URI
  18. ഑৴ΞϓϦ OBS Storage/CDN ഑৴ج൫ ΞϓϦ ಈը৘ใ ࢹௌऀ .ts .m3u8 frame

    .ts .m3u8 ಈը৘ใ ഑৴ऀ ࠓճѻ͏෦෼ ɾReplayKit ɾBroadcast Upload Extension URI
  19. ReplayKitͷ֓ཁ

  20. ReplayKitͷ֓ཁ(1) •iOSͷεΫϦʔϯΩϟϓνϟΛѻ͏SDK •ϚΠΫ΍ΞϓϦͷԻ΋࿥ԻՄೳ •iOS9͔Βར༻Մೳ •ήʔϜ΍ϖΠϯτܥΞϓϦɺVTuberΞϓϦ ʹ૊Έࠐ·Ε͍ͯΔ

  21. iOS9 iOS10 iOS11 ΞϓϦʹ૊ΈࠐΜͰ࿥ը͕Ͱ͖Δ ૊ΈࠐΜͩΞϓϦ͕ ExtensionʹରԠ͍ͯ͠Δ഑৴ΞϓϦʹॲཧΛҕৡͰ͖Δ ඪ४Ͱը໘ऩ࿥͕࣮૷ ReplayKitͷ֓ཁ(2)

  22. •഑৴ͷ࣮૷ΛϓϥοτϑΥʔϜଆʹ೚ͤΒΕΔ •୺຤ͷϥΠϑαΠΫϧͷϋϯυϦϯά͕ߟྀࡁΈ •ࣗલ࿥ըΑΓύϑΥʔϚϯε͕ྑ͍ •UI΍௨஌ͳͲ͕Ωϟϓνϟ͞ΕΔ ϝϦοτ σϝϦοτ ReplayKitΛ͔ͭ͏ϝϦσϝ

  23. https://developer.apple.com/videos/play/wwdc2018/601/ WWDC2018Ͱ΋ൃද

  24. ը໘ऩ࿥Λड͚෇͚Δʹ͸ Broadcast Upload Extension Λ࣮૷͢Ε͹OK

  25. Broadcast Upload Extension λʔήοτΛ࡞੒

  26. Broadcast Upload Extension λʔήοτΛ࡞੒ ͜ͷΑ͏ʹɺը໘ऩ࿥ͷબ୒ࢶʹ ΞϓϦ͕ग़ΔΑ͏ʹͳΓ·͢ɻ

  27. ը໘ऩ࿥ͷϥΠϑαΠΫϧ

  28. ը໘ऩ࿥ͷϥΠϑαΠΫϧ Start Recording Finish ϕʔεͱͳΔϥΠϑαΠΫϧ αʔϏεຖͷ࣮૷͕֤ॴʹ૊Έࠐ·ΕΔ RPBroadcastSampleHandlerͷ ϝιουͱඥ෇͘

  29. class SampleHandler: RPBroadcastSampleHandler { override func broadcastStarted(…) { // ഑৴։࢝

    } override func processSampleBuffer(…) { // ը໘ऩ࿥ϓϩηε͔ΒΩϟϓνϟ͞ΕͨϑϨʔϜͷड͚औΓ } override func finishBroadcastWithError() { // ഑৴ऴྃ } override func finishBroadcastWithError(…) { // ҟৗऴྃ } } ΤϯτϦʔϙΠϯτ SampleHandler.swift https://developer.apple.com/documentation/replaykit/rpbroadcastsamplehandler
  30. ֤ϥΠϑαΠΫϧຖͷΞΫγϣϯ Start Recording Finish εϖοΫνΣοΫ/ೝূ ഑৴ઌͷղܾ ഑৴ج൫ʹ઀ଓ Τϯίʔυͱ഑৴ ઀ଓΛ੾அ ഑৴த

    ഑৴։࢝ ഑৴ऴྃ ঢ়ଶ ঢ়ଶຖͷΞΫγϣϯ ୺຤ͷঢ়ଶΛ؂ࢹ
  31. Start εϖοΫνΣοΫ/ೝূ ഑৴ઌͷղܾ ഑৴ج൫ʹ઀ଓ ɾ഑৴Ͱ͖ΔεϖοΫͷ୺຤͔νΣοΫ ɾೝূ৘ใɺ഑৴ݖݶνΣοΫ ɾதஅ͞Ε͍ͯΔ࿮͕͋Δ͔νΣοΫ ɾ഑৴࿮ͷ࡞੒ ɾ഑৴ج൫ͷΞΫηεઌΛαʔόʔ͔Βऔಘ ɾRTMPͷ઀ଓ(ө૾/Ի੠)

    ɾWebSocketͷ઀ଓ(ίϝϯτͷண৴)
  32. Recording Τϯίʔυͱ഑৴ ɾϝσΟΞຖʹΤϯίʔυ(h264/AAC)͠ɺૹ৴ ɾόοϑΝͷฤू/ࠩ͠ସ͑ ୺຤ͷঢ়ଶΛ؂ࢹ ɾը໘ͷճస৘ใ ɾ഑৴ઃఆͷมߋ ɾը໘ڞ༗ΦϑϞʔυ ɾ௨஌ͷON/OFF ɾ௨࿩ͷண৴

    ɾ୺຤ͷϩοΫɺి஑੾Ε ɾωοτϫʔΫঢ়گͷ؂ࢹ ɾΞϓϦ಺͔Βͷऴྃ ಈըͷݟ͑ํ தஅ/ऴྃ/ϦτϥΠ
  33. Finish ઀ଓΛ੾அ ɾRTMPΛ੾அ ɾWebSocketΛ੾அ ɾ഑৴ऴྃAPI ഑৴ͷઃఆ৘ใΛ࡟আ ɾAppGroup಺ͷ഑৴ઃఆ৘ใΛ࡟আ

  34. Recordingதͷ ϝσΟΞαϯϓϦϯά

  35. όοϑΝͷૹ৴ HaishinKitʹͯରԠͨ͠ͷͰࠓճ͸Τϯίʔυɾ RTMPʹΑΔόοϑΝͷૹ৴ʹ͸৮Ε·ͤΜɻ HaishinKit͸ϝσΟΞͷΤϯίʔυ͓ΑͼɺετϦʔϛϯάαʔόʔ ͱͷRTMP௨৴Λड͚࣋ͬͯ͘ΕΔϥΠϒϥϦͰ͢ɻ https://github.com/shogo4405/HaishinKit.swift

  36. ը໘ऩ࿥ϓϩηε͔Βड͚औ ΔϝσΟΞσʔλ func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {

    // process each media data } ϝσΟΞͷੜσʔλɻόΠτ৘ใ͔Β ֤ϝσΟΞʹ෮ݩͯ͠ฤूͨ͠ΓΤϯίʔυͨ͠Γ͢Δ video, audioApp, audioMicͷ̏छྨ sampleBufferType sampleBuffer
  37. videoϑϨʔϜ

  38. SampleBuffer(video) ϑϨʔϜͷ৘ใؚ͕·Ε͍ͯΔ ɾը૾ͷੜσʔλ ɾը૾ͷճస৘ใ ɾը૾ͷαΠζ ɾ࠶ੜ։࢝࣌ؒ(Presentation Time Stamp)

  39. NT NT NT NT NT NT SampleBuffer(video) t 154 #VGGFS

    αϯϓϦϯάλΠϛϯά͸ը໘ʹߋ৽͕͋ͬͨͱ͖ Duration͸લճͷαϯϓϦϯά͔ΒࠓճͷαϯϓϦϯά
  40. ϗʔϜը໘ͷ഑৴Λ͠ͳ͕Β ์ஔ͍ͯͨ͠Β… ҆ఆ࠶ੜͰ͖ͳ͍ࣄ৅ʹؕͬͨ ΩʔϑϨʔϜΛੜ੒ִؒΛ2ඵʹ͍ͯ͠Δʹ΋ ؔΘΒͣ6ඵʹͳͬͯ͠·ͬͨΓ 1ts͋ͨΓಈըͷ୹͞΍ ҆ఆ͠ͳ͍tsͷduration͸ϥάʹ௚݁͢Δ

  41. NT NT ը໘ͷߋ৽͕ͳ͍৔߹ 154 t #VGGFS ͜ͷؒ͸ߋ৽͕ͳ͘ϑϨʔϜ͕ૹΒΕͣ ΩʔϑϨʔϜ΋ૹΕͳ͍ͨΊ ಈըͷ۠੾Γ͕͏·͘Ͱ͖ͳ͍

  42. ಈըΛ࠶ݱ͢ΔͨΊʹbitmap৘ใΛ͢΂ؚͯΜͩϑϨʔϜ ࣮ࡍͷ ϑϨʔϜ t ಈը ΩʔϑϨʔϜ(I Frame)ͱ͸

  43. ࣮ࡍͷ ϑϨʔϜ I Frame P Frame ࠩ෼ͷΈΛ࣋ͭ͜ͱͰσʔλྔΛѹॖ HLSʹ͓͍ͯI Frame͸1ͭͷtsϑΝΠϧຖʹ ࠷ॳͷϑϨʔϜͱؚͯ͠ΜͰ͍ͳ͚Ε͹ͳΒͳ͍

    ΩʔϑϨʔϜ(I Frame)ͱ͸ ಈըΛ࠶ݱ͢ΔͨΊʹbitmap৘ใΛ͢΂ؚͯΜͩϑϨʔϜ
  44. NT NT 154 t #VGGFS videoϑϨʔϜͷϦΧόϦͰରԠ ؒΛຒΊΔϑϨʔϜΛ͍ΕΕ͹OK

  45. t NT NT NT NT NT NT videoϑϨʔϜͷϦΧόϦͰରԠ 154 ϑϨʔϜΛૹ৴ͯ͠Ұఆ͕࣌ؒܦͬͯ΋࣍ͷϑϨʔϜ͕ͳ͍࣌

    ࠷ޙʹૹ৴ͨ͠ϑϨʔϜΛɺPTSΛ෇͚ସ͑ͯૹ৴͢Δ #VGGFS
  46. PTS͕ॏཁ ɾPTSͷζϨ͸ಈըͷԻζϨͷݪҼ ɾόοϑΝͷૢ࡞Ͱ΋͜ͷPTSΛѻ͏ ɾPTSؚΊɺಈը΍Ի੠ͷ࣌ؒ৘ใ͸CMTimeʹΑͬ ͯϑϨʔϜͰѻ͑Δܗࣜʹͳ͍ͬͯΔ

  47. CMTime ࣌ؒ৘ใΛ཭ࢄత(ϑϨʔϜ)ʹ੔਺Ͱѻ͏ public struct CMTime { public var value: CMTimeValue

    public var timescale: CMTimeScale public var flags: CMTimeFlags public var epoch: CMTimeEpoch public init() public init(value: CMTimeValue, timescale: CMTimeScale, flags: CMTimeFlags, epoch: CMTimeEpoch) } extension CMTime { public init(seconds: Double, preferredTimescale: CMTimeScale) public init(value: CMTimeValue, timescale: CMTimeScale) } value: ϑϨʔϜ਺ timescale: 1ඵ͋ͨΓΛԿϑϨʔϜʹ෼ׂ͢Δ͔ʁ
  48. CMTimeͷσʔλͷྫ 7JEFP CMTime(value: 1, timeScale: 30) "VEJP CMTime(value: 1024, timeScale:

    44100) ≒0.33333…
  49. ΩʔϑϨʔϜૹ৴ִؒ ֤αʔϏεͷਪ঑KeyFrame Interval͸2ඵ ɾYouTube ɾTwitch ɾOPENREC.tv ɾFRESH LIVE 1ͭͷts͋ͨΓΛ2ඵͷಈըʹ͍ͨ͠

  50. audioϑϨʔϜ

  51. CMSampleBuffer(audioMic) ϑϨʔϜͷ৘ใؚ͕·Ε͍ͯΔ ɾԻ੠ͷੜσʔλ ɾLinear PCM ɾ࠶ੜ։࢝࣌ؒ(Presentation Time Stamp) ɾ௕͞(Duration)

  52. NT NT NT NT NT CMSampleBuffer(audioMic) 154 㽈 㽈 㽈

    㽈 t 㽈 αϯϓϦϯάλΠϛϯά͸ఆظతɻ ִؒ͸1024/44100ඵ(≒23ms) ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS
  53. NT NT NT NT NT CMSampleBuffer(audioMic) 154 㽈 㽈 㽈

    㽈 t 㽈 ϏσΦόοϑΝ͸ࠓͷදࣔͷόοϑΝʹର͠ ΦʔσΟΦόοϑΝ͸ࠓ·Ͱ࿥Ի͞ΕͨόοϑΝΛड͚औΔ 㽈 㽈 㽈 㽈 㽈 औಘ λΠϛϯά ᶃ ᶄ ᶅ ᶆ ᶇ ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS
  54. NT NT NT NT NT CMSampleBuffer(audioApp) 154 㽈 㽈 㽈

    㽈 t 㽈 ϚΠΫಉ༷αϯϓϦϯάλΠϛϯά͸΄΅ఆظతɻ ִؒ͸22050/44100ඵલޙ(≒500ms) ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS
  55. ө૾ͷฤू

  56. ө૾ͷฤू ө૾ͷฤूͱ͍ͬͯ΋CIImageʹΑΔฤू ·ͣCMSampleBuffer͔ΒCIImageܗࣜʹม׵͢Δ extension CMSampleBuffer { var ciImage: CIImage? {

    guard let pixelBuffer = CMSampleBufferGetImageBuffer(self) else { return nil } CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.readOnly); defer { CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.readOnly) } return CIImage(cvPixelBuffer: pixelBuffer) } } CMSampleBuffer > CVPixelBuffer > CIImage
  57. CIImageͰͰ͖Δ͜ͱ ɾ֦େ/ॖখ(CGAffineTransform) ɾҠಈ(CGAffineTransform) ɾ੾ΓऔΓ(crop) ɾճస(oriented) ɾ΅͔͠(applyingGaussianBlur) ɾϑΟϧλʔ

  58. CMSampleBufferʹ໭͢ // BlurΛద༻͢ΔͳͲͷฤू let newImage = sampleBuffer.ciImage.applyingGaussianBlur(sigma: 10) // ۣܗ৘ใ

    let dimensions = CMVideoFormatDescriptionGetDimensions(CMSampleBufferGetFormatDescription(sampleBuffer)!) // ϐΫηϧ৘ใͷॻ͖ࠐΈઌͷόοϑΝΛ࡞੒ var outputPixelBuffer: CVPixelBuffer? = nil CVPixelBufferCreate(kCFAllocatorSystemDefault, Int(dimensions.width), Int(dimensions.height), kCVPixelFormatType_32BGRA, nil, &outputPixelBuffer) CVPixelBufferLockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)); defer { CVPixelBufferUnlockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)) } // όοϑΝʹCIImageΛॻ͖ࠐΈ let ciContext = ContextHolder.shared.context ciContext.render(newImage, to: outputPixelBuffer!) // ৽͍͠SampleBufferΛ࡞੒ var videoFormatDescription: CMVideoFormatDescription? = nil CMVideoFormatDescriptionCreateForImageBuffer(nil, outputPixelBuffer!, &videoFormatDescription) var sampleTimingInfo = self.sampleTimingInfo var newSampleBuffer: CMSampleBuffer? = nil CMSampleBufferCreateForImageBuffer(nil, outputPixelBuffer!, true, nil, nil, videoFormatDescription!, &sampleTimingInfo, &newSampleBuffer)
  59. // BlurΛద༻͢ΔͳͲͷฤू let newImage = sampleBuffer.ciImage.applyingGaussianBlur(sigma: 10) CMSampleBuffer͔ΒCIImageΛऔಘ͠ɺ ϑΟϧλʔͳͲΛద༻͢Δ CMSampleBufferʹ໭͢

  60. var outputPixelBuffer: CVPixelBuffer? = nil CVPixelBufferCreate(kCFAllocatorSystemDefault, Int(dimensions.width), Int(dimensions.height), kCVPixelFormatType_32BGRA, nil,

    &outputPixelBuffer) CVPixelBufferLockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)); defer { CVPixelBufferUnlockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)) } CIImageͷॻ͖ࠐΈઌͷόοϑΝΛ࡞੒͢Δ CMSampleBufferʹ໭͢ᶃ
  61. // όοϑΝʹCIImageΛॻ͖ࠐΈ let ciContext = ContextHolder.shared.context ciContext.render(newImage, to: outputPixelBuffer!) ฤूͨ͠CIImageΛόοϑΝʹॻ͖ࠐΉ

    CMSampleBufferʹ໭͢ᶄ
  62. var videoFormatDescription: CMVideoFormatDescription? = nil CMVideoFormatDescriptionCreateForImageBuffer(nil, outputPixelBuffer!, &videoFormatDescription) var sampleTimingInfo

    = self.sampleTimingInfo var newSampleBuffer: CMSampleBuffer? = nil CMSampleBufferCreateForImageBuffer(nil, outputPixelBuffer!, true, nil, nil, videoFormatDescription!, &sampleTimingInfo, &newSampleBuffer) ॻ͖ࠐΜͩBufferͱݩͷCMSampleBufferͷ ࣌ؒ৘ใΛ࢖ͬͯ৽ͨͳCMSampleBufferΛੜ੒͢Δ CMSampleBufferʹ໭͢ᶅ
  63. ө૾ͷฤू ֖ֆʹࠩ͠ସ͑

  64. ͍ͨͩ·४උதʂ ͍ΘΏΔɺྲྀͤΔө૾͕ͳ͍ ࣌ʹ”֖”Λ͢ΔͨΊͷը૾ͷ ͜ͱ ֖ֆͱ͸

  65. ɾύεϫʔυೖྗ ɾݸਓ৘ใͱͳΔΞϓϦΛ։͘ ɾʮҰ౓഑৴Λऴྃ͢Ε͹͍͍ʯ͸ඍົ ΅͔͠Λݕ౼͕ͨ͠ɺ ύεϫʔυೖྗͰ͸ ΩʔϘʔυͷҐஔ͔Β ೖྗ͕ਪଌͰ͖ΔͨΊ ׬શͳ֖ֆʹ ֖ֆͷར༻γʔϯ

  66. ɾ֖ֆͷը૾৘ใΛ࣋ͬͨSampleBufferΛىಈ࣌ʹੜ੒͓ͯ͘͠ ɾը໘ऩ࿥ϓϩηε͔Βड͚औͬͨSampleBufferͷPTSʹࠩ͠ସ ֖͑ͯֆͷόοϑΝΛૹ৴͢Δ ը໘ऩ࿥͔Β ड͚औͬͨόοϑΝ ֖ֆ όοϑΝ PTSΛ ΋Β͏ ഑৴ج൫

    όοϑΝͷࠩ͠ସ͑
  67. NT NT NT NT NT NT t 154 'SBNF ڞ༗OFF

    όοϑΝͷࠩ͠ସ͑
  68. NT NT NT NT NT NT t 154 'SBNF ڞ༗OFF

    όοϑΝͷࠩ͠ସ͑
  69. όοϑΝͷࠩ͠ସ͑ ֖ֆͷCMSampleBufferΛϗϧμʔʹอ͓͖࣋ͯ͠ ετϦʔϜͷϋϯυϦϯάଆͰόοϑΝΛࠩ͠ସ͑ class MaskHolder { static let shared =

    MaskHolder() private let maskBuffer: CMSampleBuffer func mask(with sampleTimingInfo: CMSampleTimingInfo) -> CMSampleBuffer { return maskBuffer.copy(with: sampleTimingInfo) } } class BroadcastService { var isEnabledMask: Bool let rtmpStream: RTMPStream func processSampleBuffer(sampleBuffer: CMSampleBuffer) { let buffer = isEnabledMask ? MaskHolder.shared.mask(with: sampleBuffer.sampleTimingInfo) : sampleBuffer rtmpStream.appendSampleBuffer(sampleBuffer: buffer) } }
  70. extension CMSampleBuffer { func copy(with timing: CMSampleTimingInfo) -> CMSampleBuffer? {

    var sampleTiming = timing var copied: CMSampleBuffer? = nil CMSampleBufferCreateCopyWithNewTiming(kCFAllocatorSystemDefault, self, 1, &sampleTiming, &copied) return copied } var sampleTimingInfo: CMSampleTimingInfo { return CMSampleTimingInfo(duration: self.duration, presentationTimeStamp: self.presentationTimeStamp, decodeTimeStamp: self.decodeTimeStamp) } var duration: CMTime { return CMSampleBufferGetDuration(self) } var presentationTimeStamp: CMTime { return CMSampleBufferGetPresentationTimeStamp(self) } var decodeTimeStamp: CMTime { return CMSampleBufferGetDecodeTimeStamp(self) } } CMSampleBuffer͸ ؔ਺ݺͼग़͠ʹΑΔϓϩύςΟऔಘ͕ଟ͍ͷͰ extensionʹcomputed propertyΛੜ΍͓ͯ͘͠ͱศར όοϑΝͷࠩ͠ସ͑
  71. ө૾ͷճస

  72. iOSͷը໘͸഑৴தʹํ޲͕มΘΔ ͜ΕΛͲ͏ѻ͏͔ͱ͍͏τϐοΫ ήʔϜ͸ԣը໘΋ଟ͍ɻԻήʔ΍ΧʔυήʔϜͳͲɻ ө૾ͷճసͱ͸ʁ

  73. ɾCMSampleBufferͷத਎͸ॎ௕ͷը૾ ɾCMSampleBufferͷϝλ৘ใʹճస৘ใؚ͕·Ε͍ͯΔ ճస৘ใͷऔಘ

  74. extension CMSampleBuffer { var orientation: CGImagePropertyOrientation { guard let attachment

    = CMGetAttachment(self, RPVideoSampleOrientationKey, nil) as? NSNumber else { return .up } if let orientation = CGImagePropertyOrientation(rawValue: orientationAttachment.uint32Value) { return orientation } return .up } } VQ MFGU SJHIU EPXO https://developer.apple.com/documentation/imageio/cgimagepropertyorientation CGImagePropertyOrientation
  75. ɾυΩϡϝϯτͰ͸iOS11.0+͕ͩ… ɾiOS11.2.1ҎલͰ͸ىಈ࣌ʹϦϯΫΤϥʔͰڧ੍ऴྃ ɾiOS11.2.5Ҏ߱͸໰୊ͳ͠ CMGetAttachment(self, "RPSampleBufferVideoOrientation" as CFString, nil) RPVideoSampleOrientationKey ɾ11.2.5Ҏ߱ͰऔಘͰ͖Δఆ਺Λϕλଧͪ

    ɾ11.2.1ҎલͰ͸nil͕औಘ͞ΕΔ ɾ11.2.1ҎલͰ΋ىಈͰ͖Δ͜ͱ͕ॏཁ https://developer.apple.com/documentation/replaykit/rpvideosampleorientationkey
  76. ഑৴։࢝࣌ʹํ޲ͱղ૾౓Λݻఆɺ ը૾Λճస(CoreImageͷoriented)ͤͯ͞Scale Fit͢Δ ѻ͍ํ1 ॎݻఆ ԣݻఆ

  77. ϝϦοτ σϝϦοτ ɾࢹௌऀ͕ݟ΍͍͢ํ޲ʹճస͍ͯ͠Δ ɾϑϨʔϜͷม׵͕ੜ͡Δ ɾ༨ന͕ೖͬͯ͠·͏ ɾෳ਺ͷΞϓϦͷߦ͖དྷΛ͢Δͱ͖ʹɺ ͜ͷඳըʹͳΔ͜ͱΛ഑৴ऀ͕ؾʹ͠ͳ͚Ε͹ͳΒͳ͍ ѻ͍ํ1

  78. ಈը͸ॎݻఆʹͯ͠ɺө૾͸ճసͤ͞ͳ͍ɻ αʔόʔʹճస৘ใΛૹΓɺϓϨʔϠʔଆͰճసΛαϙʔτ up right left down ѻ͍ํ2

  79. ϝϦοτ σϝϦοτ ɾ༨ന͕ͳ͍ ɾճసॲཧͷඞཁ͕ͳ͍ ɾࢹௌऀ͕ݟΔॠؒʹΑͬͯ͸ํ޲͕Ϛον͠ͳ͍ ɾϓϨΠϠʔ(ϓϥοτϑΥʔϜ)ຖʹରԠ͕ඞཁ ίετ͕ߴ͍ը૾ॲཧΛ1ͭল͚Δˍڐ༰Ͱ͖ΔσϝϦοτ ѻ͍ํ2

  80. ഑৴ͷऴྃ

  81. ഑৴ͷऴྃγʔέϯε •WebSocketΛ੾Δ •RTMPΛ੾அ͢Δ •αʔόʔʹ഑৴ऴྃAPIΛ౤͛Δ •AppͱExtensionͷڞ௨ྖҬʹอଘ͍ͯ͠Δ഑৴ʹؔ ࿈͢ΔσʔλΛ࡟আ͢Δ •ҟৗऴྃͰ͋Ε͹ͦͷঢ়ଶΛอଘ͢Δ ഑৴ͷऴྃAPIͱσʔλ࡟আ͸τϥϯβΫγϣϯ

  82. DispatchGroup override open func broadcastFinished() { let dispathGroup = DispatchGroup()

    dispathGroup.enter() broadcastService?.finish { dispathGroup.leave() } dispathGroup.wait() } DispatchGroupͰεϨουΛఀࢭͤ͞Δ
  83. BroadcastService func finish(completion: (()->Void)) { api.finishStream(movieID: 123) { completion?() }

    } αʔϏε಺ͰAPIΛड͚औͬͨΒcompletionΛ࣮ߦ
  84. ը໘ऩ࿥Λऴྃͤ͞Δํ๏ •εςʔλεόʔ͔Βऴྃ •ίϯτϩʔϧηϯλʔ͔Βऴྃ •ΞϓϦ͔Βऴྃ

  85. ΞϓϦέʔγϣϯ͔Βऴྃ override open func finishBroadcastWithError(_ error: Error) { super.finishBroadcastWithError(error) }

    RPBroadcastSampleHandlerͰ Τϥʔऴྃͤ͞Δඞཁ͕͋Δ https://developer.apple.com/documentation/replaykit/rpbroadcastsamplehandler/2721526-finishbroadcastwitherror
  86. ഑৴ͷҟৗऴྃͱ࠶։

  87. ഑৴Λҟৗऴྃͤ͞Δ override open func finishBroadcastWithError(_ error: Error) { super.finishBroadcastWithError(error) }

    ద੾ʹΤϥʔΛૹΔ͜ͱͰɺ μΠΞϩάʹऴྃཧ༝͕දࣔͰ͖Δ
  88. ഑৴Λҟৗऴྃͤ͞Δ let message = “ωοτϫʔΫ͕੾அ͞Ε·ͨ͠" let error = NSError( domain:

    Bundle.main.bundleIdentifier ?? “" , code: 1 , userInfo: [NSLocalizedFailureReasonErrorKey: message]) finishBroadcastWithError(error) userInfoʹϝοηʔδΛࢦఆ͠ͳ͍ͱ ऴྃཧ༝͕(null)ʹͳΓ·͢ɻ
  89. ҙਤ͠ͳ͍ऴྃͱͯ͠ ૝ఆ͞ΕΔέʔε •ը໘͕ϩοΫ͞ΕΔ(࣋ͪํͰిݯϘλϯΛԡͨ͠) •ి஑͕੾ΕΔ •ి࿩͕དྷΔ •BadAccessͰམͪΔ

  90. ҟৗ഑৴͕ఀࢭͨ͠৔߹ɺ࠶ ։͍ͤͯ͋͛ͨ͞ •഑৴ͷ਺ࣈͷ෼ࢄ(ࢹௌ਺΍ίϝϯτɺΤʔϧ) •ࢹௌऀͷମݧ •৽͍͠഑৴࿮΁ͷҠಈʹΑΔτϥϑΟοΫͷε ύΠΫ

  91. ి஑੾ΕɺిݯOFFɺϩοΫ • RPBroadcastSampleHandlerͷfinish()͕ݺ͹ΕΔ • ਖ਼ৗऴྃѻ͍ʹ͢Δ͔͠ͳ͍ • OSͷ௨஌͸͋Δ͕ඇެ։APIͷͨΊϦδΣΫτ͞ΕΔ

  92. ௨࿩ͷண৴ iOS͸௨࿩Λண৴͢Δͱɺ RPBroadcastSampleHandlerͷfinish͕ݺ͹ΕΔ ͦͷͨΊɺਖ਼ৗܥͷ഑৴ऴྃγʔέϯεʹೖͬͯ ͠·͏ ண৴Λݕ஌ͨ࣌͠͸ɺ finish͕ݺ͹Εͯ΋தஅѻ͍ʹ͢Δ

  93. ௨࿩ͷண৴ import CallKit class SampleHandler: RPBroadcastSampleHandler { let callObserver =

    CXCallObserver() override init() { super.init() callObserver.setDelegate(self, queue: nil) } } extension SampleHandler: CXCallObserverDelegate { func callObserver(…) { broadcastService.suspendStreaming() // ഑৴Λதஅ͢Δ } }
  94. ωοτϫʔΫͷ੾அ • Reachability.swiftΛ࠾༻ • RxͰ؂ࢹ͓ͯ͘͠ • ωοτϫʔΫ͕੾Εͨ৔߹͸ҟৗऴྃͱ͠ɺ ୺຤ʹதஅঢ়ଶΛอଘ͢Δ

  95. AppͱExtensionͷσʔλڞ༗

  96. ڞ༗͢Δσʔλ • ೝূ৘ใ • ഑৴ઃఆ(ը࣭) • ഑৴࿮(഑৴ج൫ͷ઀ଓઌͳͲͷ৘ใ) • ୺຤ઃఆ(௨஌ɺ֖ֆ) •

    தஅ৘ใ
  97. AppGroupΛ࢖ͬͯσʔλΛ ڞ༗͢Δ • ෳ਺ͷΞϓϦؒͰͷσʔλڞ༗ઃఆ • ಉ͡σϕϩούʔͷΞϓϦؒͰ༗ޮ • AppID͝ͱʹ༗ޮԽ͠ɺEntitlementͳͲͷ ઃఆ͕ඞཁ let

    userDefaults: UserDefaults = UserDefaults(suiteName: “groups.jp.iosdc2018”) IDΛࢦఆ͢Δ͜ͱͰάϧʔϓؒͷڞ༗ྖҬʹΞΫηεͰ͖Δ
  98. ExtensionΛσόοά͢Δ

  99. ϒϨΠΫϙΠϯτͷషΓํ

  100. ϒϨΠΫϙΠϯτͷషΓํ

  101. Waitingঢ়ଶʹͳ͔ͬͯΒɺ ഑৴Λ։࢝͢Δ

  102. Appͱಉ͡Α͏ʹΠϯεϖΫ τͰ͖·͢

  103. Appͱಉ͡Α͏ʹΠϯεϖΫ τͰ͖·͢

  104. ΩϟϓνϟͷεϨου BreakpointͰࢭΊͯΔͱ ഑৴ϓϩηε͕ࢮʹ·͢

  105. ͳͷͰLoggerΛ࢖͍·͠ΐ͏ • TCP Socket Logger • Logboard/LogboardConsole • JustLog •

    File Logger • XCGLogger • SwiftyBeaver
  106. ·ͱΊ

  107. ·ͱΊ • ReplayKitʹΑͬͯΧδϡΞϧʹੜ์ૹαʔϏε ͕࣮૷Ͱ͖ΔΑ͏ʹͳͬͨʂ • CMSampleBufferΛฤू͢Δ͜ͱͰө૾Λࠩ͠ସ ͑ͨΓΤϑΣΫτΛ͔͚ͨΓ͢Δ͜ͱ͕Ͱ͖Δ! • ഑৴தʹ୺຤͕ى͜ΓಘΔΠϕϯτͷϋϯυϦϯ ά͸ΞϓϦʹൺ΂ͯൈ͚ͳ͘΍Βͳͯ͘͸ͳΒͳ

    ͍
  108. Thanks • HaishinKit(https://github.com/ shogo4405/HaishinKit.swift)