Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Live Streaming with Screen Recording

Hideki Matsuoka
September 01, 2018

Live Streaming with Screen Recording

Hideki Matsuoka

September 01, 2018
Tweet

More Decks by Hideki Matsuoka

Other Decks in Technology

Transcript

  1. ೋบ͘Β͍͋Δ ը໘ऩ࿥͔Βͷੜ์ૹ NBUTVPLBI

  2. 01&/3&$UW$ZCFS; *OD $ZCFS"HFOU *OD  5XJUUFS[email protected] (JUIVCNBUVPLBI झຯ ήʔϜ࣮گΛݟΔ͜ͱɺ࣮گੜ์ૹ͢Δ͜ͱ ಛʹ16#(

    4QMBUPPO দԬलथ
  3. • https://openrec.tv • ήʔϜಛԽͷੜ์ૹαʔϏε • MAU 200ສ • ߴը࣭ɺ௿஗Ԇ഑৴ɺ௥͍͔͚࠶ੜɺ̐ϲࠃޠ(CJK+E)ରԠ •

    Web/iOS/tvOS/Android/AndroidTV
  4. None
  5. ൃදʹҠΔલʹ

  6. E CyberAgent group iOS BeerBash։࠵! ݄೔ Ր ɹ࣌෼։࢝ʢ࣌ΑΓड෇։࢝ʣ αΠόʔΤʔδΣϯτʢौ୩ϓϥΠϜϓϥβ'ɹΫϦΤΠςΟϒϥ΢ϯδʣ dɹ֓ཁઆ໌ɺסഋ

    dɹ׻ஊ dɹ৽ઃͷ৽نαʔϏε։ൃ૊৫ɹ$"54ʢΫϥΠΞϯτٕज़ྖҬΛݗҾ͢Δ૊৫ʣʹ͍ͭͯ dɹϚονϯάΞϓϦʮλοϓϧ஀ੜʯʹ͓͚Δ։ൃͷมԽ dɹ׻ஊ ͝Ԡื͸23ίʔυ͔Β ͪ͜Β͔ΒͷΤϯτϦʔ͸શһ͝ࢀՃ͍͚ͨͩ·͢ connpass͔ΒΤϯτϦʔͷ৔߹ɺࢀՃ͸நબͱͳΓ·͢ ฐάϧʔϓͷiOSDCొஃऀ͕ࢀՃ͢ΔɹBeerBashͰ͢ʂ দԬ΋ࢀՃ͠·͢ʂ ೔࣌ ৔ॴ ಺༰ ɹɹɹɹɹɹɹגࣜձࣾαΠόʔΤʔδΣϯτ$MJFOU"EWBODFE5FDIOPMPHZ4UVEJP $"54 ɹҏ౻ɹګฏ ɹɹɹɹɹɹɹגࣜձࣾϚονϯάΤʔδΣϯτɹ∁ڮɹ༏հ
  7. ͜ͷൃදͰ͸Sli.doΛ࢖͍·͢ ൃදͷ్தʹSli.do͔ΒΞϯέʔτΛ͠ͳ͕Β࠷ޙʹൃද ్தͰ࣭໰ͨ͘͠ͳͬͨΒSli.doʹ! ࠷ޙͷ࣭໰λΠϜͰर͍·͢ Event Code: iosdc2018_openrec URL: https://app2.sli.do/event/0kwbnzqe

  8. ͦΕͰ͸ൃදʹҠΓ·͢

  9. ΞδΣϯμ •HLSͷ֓ཁ •ReplayKitͷ֓ཁ •ը໘ऩ࿥ͷϥΠϑαΠΫϧ •SampleBufferͷฤू •ੜ์ૹதͷૢ࡞ •഑৴ͷҟৗऴྃͱ࠶։ •iOS AppͱBroadcaster Upload

    Extensionͷσʔλڞ༗ •ExtensionΛσόοά͢Δ
  10. HLSͷ֓ཁ

  11. HLSͷ֓ཁ HTTP Live Streaming(RFC8216)ͷུ ϓϨʔϠʔ͕ϓϨΠϦετͱࡉ੾Εʹͳ͍ͬͯΔಈըΛ HTTPϦΫΤετͰμ΢ϯϩʔυ͠ɺ ϓϨΠϦετʹఆٛ͞Ε͍ͯΔॱʹ࿈ଓతʹ࠶ੜ͢Δ https://iosdc.com/playlist.m3u8 https://iosdc2018.com/1.ts https://iosdc2018.com/2.ts

    https://iosdc2018.com/3.ts ϓϨΠϦετ 2ඵຖͷಈը ≒mp4
  12. playlit.m3u8ͷத਎ #EXTM3U #EXT-X-VERSION:3 #EXT-X-TARGETDURATION:2 #EXT-X-MEDIA-SEQUENCE:1 #EXT-X-PROGRAM-DATE-TIME: 2018-08-31T12:15:12.713+09:00 #EXTINF:2.0, 1.ts #EXTINF:2.0,

    2.ts #EXTINF:2.0, 3.ts #EXT-X-ENDLIST
  13. ࢓༷ʹଇͬͨϑΝΠϧΛΞοϓϩʔυͯ͠ ϓϨΠϠʔʹμ΢ϯϩʔυͯ͠΋Β͏ CDNΛ༗ޮ׆༻Ͱ͖ΔͨΊ εέʔϥϏϦςΟ/ίετͷ໘Ͱ ੜ์ૹͷٕज़తϋʔυϧΛԼ͛ΒΕΔ HLSͷ࠷େͷϝϦοτ

  14. HLSʹΑΔ ੜ์ૹαʔϏεͷશମ૾

  15. ഑৴ΞϓϦ OBS ഑৴ج൫ ΞϓϦ frame .ts .m3u8 ഑৴ऀ HLSʹΑΔಈը഑৴ͷશମ૾ Storage/CDN

    ಈը৘ใ URI
  16. ഑৴ج൫ͷ໾ׂ 1. ड͚औͬͨϝσΟΞͷϑϨʔϜΛ߹੒ͯ͠ ɹ਺ඵ୯ҐͷಈըΛੜ੒͠ɺtsϑΝΠϧͱͯ͠σϓϩΠ 2. σϓϩΠ͞Εͨಈը৘ใΛϓϨΠϦετʹ௥ه͢Δ ഑৴ΞϓϦ OBS ഑৴ج൫ frame

    .ts .m3u8 Storage/CDN
  17. ഑৴ΞϓϦ OBS ഑৴ج൫ ΞϓϦ ಈը৘ใ ࢹௌऀ frame .ts .m3u8 ಈը৘ใ

    .ts .m3u8 ഑৴ऀ Storage/CDN HLSʹΑΔಈը഑৴ͷશମ૾ URI
  18. ഑৴ΞϓϦ OBS Storage/CDN ഑৴ج൫ ΞϓϦ ಈը৘ใ ࢹௌऀ .ts .m3u8 frame

    .ts .m3u8 ಈը৘ใ ഑৴ऀ ࠓճѻ͏෦෼ ɾReplayKit ɾBroadcast Upload Extension URI
  19. ReplayKitͷ֓ཁ

  20. ReplayKitͷ֓ཁ(1) •iOSͷεΫϦʔϯΩϟϓνϟΛѻ͏SDK •ϚΠΫ΍ΞϓϦͷԻ΋࿥ԻՄೳ •iOS9͔Βར༻Մೳ •ήʔϜ΍ϖΠϯτܥΞϓϦɺVTuberΞϓϦ ʹ૊Έࠐ·Ε͍ͯΔ

  21. iOS9 iOS10 iOS11 ΞϓϦʹ૊ΈࠐΜͰ࿥ը͕Ͱ͖Δ ૊ΈࠐΜͩΞϓϦ͕ ExtensionʹରԠ͍ͯ͠Δ഑৴ΞϓϦʹॲཧΛҕৡͰ͖Δ ඪ४Ͱը໘ऩ࿥͕࣮૷ ReplayKitͷ֓ཁ(2)

  22. •഑৴ͷ࣮૷ΛϓϥοτϑΥʔϜଆʹ೚ͤΒΕΔ •୺຤ͷϥΠϑαΠΫϧͷϋϯυϦϯά͕ߟྀࡁΈ •ࣗલ࿥ըΑΓύϑΥʔϚϯε͕ྑ͍ •UI΍௨஌ͳͲ͕Ωϟϓνϟ͞ΕΔ ϝϦοτ σϝϦοτ ReplayKitΛ͔ͭ͏ϝϦσϝ

  23. https://developer.apple.com/videos/play/wwdc2018/601/ WWDC2018Ͱ΋ൃද

  24. ը໘ऩ࿥Λड͚෇͚Δʹ͸ Broadcast Upload Extension Λ࣮૷͢Ε͹OK

  25. Broadcast Upload Extension λʔήοτΛ࡞੒

  26. Broadcast Upload Extension λʔήοτΛ࡞੒ ͜ͷΑ͏ʹɺը໘ऩ࿥ͷબ୒ࢶʹ ΞϓϦ͕ग़ΔΑ͏ʹͳΓ·͢ɻ

  27. ը໘ऩ࿥ͷϥΠϑαΠΫϧ

  28. ը໘ऩ࿥ͷϥΠϑαΠΫϧ Start Recording Finish ϕʔεͱͳΔϥΠϑαΠΫϧ αʔϏεຖͷ࣮૷͕֤ॴʹ૊Έࠐ·ΕΔ RPBroadcastSampleHandlerͷ ϝιουͱඥ෇͘

  29. class SampleHandler: RPBroadcastSampleHandler { override func broadcastStarted(…) { // ഑৴։࢝

    } override func processSampleBuffer(…) { // ը໘ऩ࿥ϓϩηε͔ΒΩϟϓνϟ͞ΕͨϑϨʔϜͷड͚औΓ } override func finishBroadcastWithError() { // ഑৴ऴྃ } override func finishBroadcastWithError(…) { // ҟৗऴྃ } } ΤϯτϦʔϙΠϯτ SampleHandler.swift https://developer.apple.com/documentation/replaykit/rpbroadcastsamplehandler
  30. ֤ϥΠϑαΠΫϧຖͷΞΫγϣϯ Start Recording Finish εϖοΫνΣοΫ/ೝূ ഑৴ઌͷղܾ ഑৴ج൫ʹ઀ଓ Τϯίʔυͱ഑৴ ઀ଓΛ੾அ ഑৴த

    ഑৴։࢝ ഑৴ऴྃ ঢ়ଶ ঢ়ଶຖͷΞΫγϣϯ ୺຤ͷঢ়ଶΛ؂ࢹ
  31. Start εϖοΫνΣοΫ/ೝূ ഑৴ઌͷղܾ ഑৴ج൫ʹ઀ଓ ɾ഑৴Ͱ͖ΔεϖοΫͷ୺຤͔νΣοΫ ɾೝূ৘ใɺ഑৴ݖݶνΣοΫ ɾதஅ͞Ε͍ͯΔ࿮͕͋Δ͔νΣοΫ ɾ഑৴࿮ͷ࡞੒ ɾ഑৴ج൫ͷΞΫηεઌΛαʔόʔ͔Βऔಘ ɾRTMPͷ઀ଓ(ө૾/Ի੠)

    ɾWebSocketͷ઀ଓ(ίϝϯτͷண৴)
  32. Recording Τϯίʔυͱ഑৴ ɾϝσΟΞຖʹΤϯίʔυ(h264/AAC)͠ɺૹ৴ ɾόοϑΝͷฤू/ࠩ͠ସ͑ ୺຤ͷঢ়ଶΛ؂ࢹ ɾը໘ͷճస৘ใ ɾ഑৴ઃఆͷมߋ ɾը໘ڞ༗ΦϑϞʔυ ɾ௨஌ͷON/OFF ɾ௨࿩ͷண৴

    ɾ୺຤ͷϩοΫɺి஑੾Ε ɾωοτϫʔΫঢ়گͷ؂ࢹ ɾΞϓϦ಺͔Βͷऴྃ ಈըͷݟ͑ํ தஅ/ऴྃ/ϦτϥΠ
  33. Finish ઀ଓΛ੾அ ɾRTMPΛ੾அ ɾWebSocketΛ੾அ ɾ഑৴ऴྃAPI ഑৴ͷઃఆ৘ใΛ࡟আ ɾAppGroup಺ͷ഑৴ઃఆ৘ใΛ࡟আ

  34. Recordingதͷ ϝσΟΞαϯϓϦϯά

  35. όοϑΝͷૹ৴ HaishinKitʹͯରԠͨ͠ͷͰࠓճ͸Τϯίʔυɾ RTMPʹΑΔόοϑΝͷૹ৴ʹ͸৮Ε·ͤΜɻ HaishinKit͸ϝσΟΞͷΤϯίʔυ͓ΑͼɺετϦʔϛϯάαʔόʔ ͱͷRTMP௨৴Λड͚࣋ͬͯ͘ΕΔϥΠϒϥϦͰ͢ɻ https://github.com/shogo4405/HaishinKit.swift

  36. ը໘ऩ࿥ϓϩηε͔Βड͚औ ΔϝσΟΞσʔλ func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {

    // process each media data } ϝσΟΞͷੜσʔλɻόΠτ৘ใ͔Β ֤ϝσΟΞʹ෮ݩͯ͠ฤूͨ͠ΓΤϯίʔυͨ͠Γ͢Δ video, audioApp, audioMicͷ̏छྨ sampleBufferType sampleBuffer
  37. videoϑϨʔϜ

  38. SampleBuffer(video) ϑϨʔϜͷ৘ใؚ͕·Ε͍ͯΔ ɾը૾ͷੜσʔλ ɾը૾ͷճస৘ใ ɾը૾ͷαΠζ ɾ࠶ੜ։࢝࣌ؒ(Presentation Time Stamp)

  39. NT NT NT NT NT NT SampleBuffer(video) t 154 #VGGFS

    αϯϓϦϯάλΠϛϯά͸ը໘ʹߋ৽͕͋ͬͨͱ͖ Duration͸લճͷαϯϓϦϯά͔ΒࠓճͷαϯϓϦϯά
  40. ϗʔϜը໘ͷ഑৴Λ͠ͳ͕Β ์ஔ͍ͯͨ͠Β… ҆ఆ࠶ੜͰ͖ͳ͍ࣄ৅ʹؕͬͨ ΩʔϑϨʔϜΛੜ੒ִؒΛ2ඵʹ͍ͯ͠Δʹ΋ ؔΘΒͣ6ඵʹͳͬͯ͠·ͬͨΓ 1ts͋ͨΓಈըͷ୹͞΍ ҆ఆ͠ͳ͍tsͷduration͸ϥάʹ௚݁͢Δ

  41. NT NT ը໘ͷߋ৽͕ͳ͍৔߹ 154 t #VGGFS ͜ͷؒ͸ߋ৽͕ͳ͘ϑϨʔϜ͕ૹΒΕͣ ΩʔϑϨʔϜ΋ૹΕͳ͍ͨΊ ಈըͷ۠੾Γ͕͏·͘Ͱ͖ͳ͍

  42. ಈըΛ࠶ݱ͢ΔͨΊʹbitmap৘ใΛ͢΂ؚͯΜͩϑϨʔϜ ࣮ࡍͷ ϑϨʔϜ t ಈը ΩʔϑϨʔϜ(I Frame)ͱ͸

  43. ࣮ࡍͷ ϑϨʔϜ I Frame P Frame ࠩ෼ͷΈΛ࣋ͭ͜ͱͰσʔλྔΛѹॖ HLSʹ͓͍ͯI Frame͸1ͭͷtsϑΝΠϧຖʹ ࠷ॳͷϑϨʔϜͱؚͯ͠ΜͰ͍ͳ͚Ε͹ͳΒͳ͍

    ΩʔϑϨʔϜ(I Frame)ͱ͸ ಈըΛ࠶ݱ͢ΔͨΊʹbitmap৘ใΛ͢΂ؚͯΜͩϑϨʔϜ
  44. NT NT 154 t #VGGFS videoϑϨʔϜͷϦΧόϦͰରԠ ؒΛຒΊΔϑϨʔϜΛ͍ΕΕ͹OK

  45. t NT NT NT NT NT NT videoϑϨʔϜͷϦΧόϦͰରԠ 154 ϑϨʔϜΛૹ৴ͯ͠Ұఆ͕࣌ؒܦͬͯ΋࣍ͷϑϨʔϜ͕ͳ͍࣌

    ࠷ޙʹૹ৴ͨ͠ϑϨʔϜΛɺPTSΛ෇͚ସ͑ͯૹ৴͢Δ #VGGFS
  46. PTS͕ॏཁ ɾPTSͷζϨ͸ಈըͷԻζϨͷݪҼ ɾόοϑΝͷૢ࡞Ͱ΋͜ͷPTSΛѻ͏ ɾPTSؚΊɺಈը΍Ի੠ͷ࣌ؒ৘ใ͸CMTimeʹΑͬ ͯϑϨʔϜͰѻ͑Δܗࣜʹͳ͍ͬͯΔ

  47. CMTime ࣌ؒ৘ใΛ཭ࢄత(ϑϨʔϜ)ʹ੔਺Ͱѻ͏ public struct CMTime { public var value: CMTimeValue

    public var timescale: CMTimeScale public var flags: CMTimeFlags public var epoch: CMTimeEpoch public init() public init(value: CMTimeValue, timescale: CMTimeScale, flags: CMTimeFlags, epoch: CMTimeEpoch) } extension CMTime { public init(seconds: Double, preferredTimescale: CMTimeScale) public init(value: CMTimeValue, timescale: CMTimeScale) } value: ϑϨʔϜ਺ timescale: 1ඵ͋ͨΓΛԿϑϨʔϜʹ෼ׂ͢Δ͔ʁ
  48. CMTimeͷσʔλͷྫ 7JEFP CMTime(value: 1, timeScale: 30) "VEJP CMTime(value: 1024, timeScale:

    44100) ≒0.33333…
  49. ΩʔϑϨʔϜૹ৴ִؒ ֤αʔϏεͷਪ঑KeyFrame Interval͸2ඵ ɾYouTube ɾTwitch ɾOPENREC.tv ɾFRESH LIVE 1ͭͷts͋ͨΓΛ2ඵͷಈըʹ͍ͨ͠

  50. audioϑϨʔϜ

  51. CMSampleBuffer(audioMic) ϑϨʔϜͷ৘ใؚ͕·Ε͍ͯΔ ɾԻ੠ͷੜσʔλ ɾLinear PCM ɾ࠶ੜ։࢝࣌ؒ(Presentation Time Stamp) ɾ௕͞(Duration)

  52. NT NT NT NT NT CMSampleBuffer(audioMic) 154 㽈 㽈 㽈

    㽈 t 㽈 αϯϓϦϯάλΠϛϯά͸ఆظతɻ ִؒ͸1024/44100ඵ(≒23ms) ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS
  53. NT NT NT NT NT CMSampleBuffer(audioMic) 154 㽈 㽈 㽈

    㽈 t 㽈 ϏσΦόοϑΝ͸ࠓͷදࣔͷόοϑΝʹର͠ ΦʔσΟΦόοϑΝ͸ࠓ·Ͱ࿥Ի͞ΕͨόοϑΝΛड͚औΔ 㽈 㽈 㽈 㽈 㽈 औಘ λΠϛϯά ᶃ ᶄ ᶅ ᶆ ᶇ ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS
  54. NT NT NT NT NT CMSampleBuffer(audioApp) 154 㽈 㽈 㽈

    㽈 t 㽈 ϚΠΫಉ༷αϯϓϦϯάλΠϛϯά͸΄΅ఆظతɻ ִؒ͸22050/44100ඵલޙ(≒500ms) ᶃ ᶄ ᶅ ᶆ ᶇ #VGGFS
  55. ө૾ͷฤू

  56. ө૾ͷฤू ө૾ͷฤूͱ͍ͬͯ΋CIImageʹΑΔฤू ·ͣCMSampleBuffer͔ΒCIImageܗࣜʹม׵͢Δ extension CMSampleBuffer { var ciImage: CIImage? {

    guard let pixelBuffer = CMSampleBufferGetImageBuffer(self) else { return nil } CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.readOnly); defer { CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.readOnly) } return CIImage(cvPixelBuffer: pixelBuffer) } } CMSampleBuffer > CVPixelBuffer > CIImage
  57. CIImageͰͰ͖Δ͜ͱ ɾ֦େ/ॖখ(CGAffineTransform) ɾҠಈ(CGAffineTransform) ɾ੾ΓऔΓ(crop) ɾճస(oriented) ɾ΅͔͠(applyingGaussianBlur) ɾϑΟϧλʔ

  58. CMSampleBufferʹ໭͢ // BlurΛద༻͢ΔͳͲͷฤू let newImage = sampleBuffer.ciImage.applyingGaussianBlur(sigma: 10) // ۣܗ৘ใ

    let dimensions = CMVideoFormatDescriptionGetDimensions(CMSampleBufferGetFormatDescription(sampleBuffer)!) // ϐΫηϧ৘ใͷॻ͖ࠐΈઌͷόοϑΝΛ࡞੒ var outputPixelBuffer: CVPixelBuffer? = nil CVPixelBufferCreate(kCFAllocatorSystemDefault, Int(dimensions.width), Int(dimensions.height), kCVPixelFormatType_32BGRA, nil, &outputPixelBuffer) CVPixelBufferLockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)); defer { CVPixelBufferUnlockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)) } // όοϑΝʹCIImageΛॻ͖ࠐΈ let ciContext = ContextHolder.shared.context ciContext.render(newImage, to: outputPixelBuffer!) // ৽͍͠SampleBufferΛ࡞੒ var videoFormatDescription: CMVideoFormatDescription? = nil CMVideoFormatDescriptionCreateForImageBuffer(nil, outputPixelBuffer!, &videoFormatDescription) var sampleTimingInfo = self.sampleTimingInfo var newSampleBuffer: CMSampleBuffer? = nil CMSampleBufferCreateForImageBuffer(nil, outputPixelBuffer!, true, nil, nil, videoFormatDescription!, &sampleTimingInfo, &newSampleBuffer)
  59. // BlurΛద༻͢ΔͳͲͷฤू let newImage = sampleBuffer.ciImage.applyingGaussianBlur(sigma: 10) CMSampleBuffer͔ΒCIImageΛऔಘ͠ɺ ϑΟϧλʔͳͲΛద༻͢Δ CMSampleBufferʹ໭͢

  60. var outputPixelBuffer: CVPixelBuffer? = nil CVPixelBufferCreate(kCFAllocatorSystemDefault, Int(dimensions.width), Int(dimensions.height), kCVPixelFormatType_32BGRA, nil,

    &outputPixelBuffer) CVPixelBufferLockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)); defer { CVPixelBufferUnlockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0)) } CIImageͷॻ͖ࠐΈઌͷόοϑΝΛ࡞੒͢Δ CMSampleBufferʹ໭͢ᶃ
  61. // όοϑΝʹCIImageΛॻ͖ࠐΈ let ciContext = ContextHolder.shared.context ciContext.render(newImage, to: outputPixelBuffer!) ฤूͨ͠CIImageΛόοϑΝʹॻ͖ࠐΉ

    CMSampleBufferʹ໭͢ᶄ
  62. var videoFormatDescription: CMVideoFormatDescription? = nil CMVideoFormatDescriptionCreateForImageBuffer(nil, outputPixelBuffer!, &videoFormatDescription) var sampleTimingInfo

    = self.sampleTimingInfo var newSampleBuffer: CMSampleBuffer? = nil CMSampleBufferCreateForImageBuffer(nil, outputPixelBuffer!, true, nil, nil, videoFormatDescription!, &sampleTimingInfo, &newSampleBuffer) ॻ͖ࠐΜͩBufferͱݩͷCMSampleBufferͷ ࣌ؒ৘ใΛ࢖ͬͯ৽ͨͳCMSampleBufferΛੜ੒͢Δ CMSampleBufferʹ໭͢ᶅ
  63. ө૾ͷฤू ֖ֆʹࠩ͠ସ͑

  64. ͍ͨͩ·४උதʂ ͍ΘΏΔɺྲྀͤΔө૾͕ͳ͍ ࣌ʹ”֖”Λ͢ΔͨΊͷը૾ͷ ͜ͱ ֖ֆͱ͸

  65. ɾύεϫʔυೖྗ ɾݸਓ৘ใͱͳΔΞϓϦΛ։͘ ɾʮҰ౓഑৴Λऴྃ͢Ε͹͍͍ʯ͸ඍົ ΅͔͠Λݕ౼͕ͨ͠ɺ ύεϫʔυೖྗͰ͸ ΩʔϘʔυͷҐஔ͔Β ೖྗ͕ਪଌͰ͖ΔͨΊ ׬શͳ֖ֆʹ ֖ֆͷར༻γʔϯ

  66. ɾ֖ֆͷը૾৘ใΛ࣋ͬͨSampleBufferΛىಈ࣌ʹੜ੒͓ͯ͘͠ ɾը໘ऩ࿥ϓϩηε͔Βड͚औͬͨSampleBufferͷPTSʹࠩ͠ସ ֖͑ͯֆͷόοϑΝΛૹ৴͢Δ ը໘ऩ࿥͔Β ड͚औͬͨόοϑΝ ֖ֆ όοϑΝ PTSΛ ΋Β͏ ഑৴ج൫

    όοϑΝͷࠩ͠ସ͑
  67. NT NT NT NT NT NT t 154 'SBNF ڞ༗OFF

    όοϑΝͷࠩ͠ସ͑
  68. NT NT NT NT NT NT t 154 'SBNF ڞ༗OFF

    όοϑΝͷࠩ͠ସ͑
  69. όοϑΝͷࠩ͠ସ͑ ֖ֆͷCMSampleBufferΛϗϧμʔʹอ͓͖࣋ͯ͠ ετϦʔϜͷϋϯυϦϯάଆͰόοϑΝΛࠩ͠ସ͑ class MaskHolder { static let shared =

    MaskHolder() private let maskBuffer: CMSampleBuffer func mask(with sampleTimingInfo: CMSampleTimingInfo) -> CMSampleBuffer { return maskBuffer.copy(with: sampleTimingInfo) } } class BroadcastService { var isEnabledMask: Bool let rtmpStream: RTMPStream func processSampleBuffer(sampleBuffer: CMSampleBuffer) { let buffer = isEnabledMask ? MaskHolder.shared.mask(with: sampleBuffer.sampleTimingInfo) : sampleBuffer rtmpStream.appendSampleBuffer(sampleBuffer: buffer) } }
  70. extension CMSampleBuffer { func copy(with timing: CMSampleTimingInfo) -> CMSampleBuffer? {

    var sampleTiming = timing var copied: CMSampleBuffer? = nil CMSampleBufferCreateCopyWithNewTiming(kCFAllocatorSystemDefault, self, 1, &sampleTiming, &copied) return copied } var sampleTimingInfo: CMSampleTimingInfo { return CMSampleTimingInfo(duration: self.duration, presentationTimeStamp: self.presentationTimeStamp, decodeTimeStamp: self.decodeTimeStamp) } var duration: CMTime { return CMSampleBufferGetDuration(self) } var presentationTimeStamp: CMTime { return CMSampleBufferGetPresentationTimeStamp(self) } var decodeTimeStamp: CMTime { return CMSampleBufferGetDecodeTimeStamp(self) } } CMSampleBuffer͸ ؔ਺ݺͼग़͠ʹΑΔϓϩύςΟऔಘ͕ଟ͍ͷͰ extensionʹcomputed propertyΛੜ΍͓ͯ͘͠ͱศར όοϑΝͷࠩ͠ସ͑
  71. ө૾ͷճస

  72. iOSͷը໘͸഑৴தʹํ޲͕มΘΔ ͜ΕΛͲ͏ѻ͏͔ͱ͍͏τϐοΫ ήʔϜ͸ԣը໘΋ଟ͍ɻԻήʔ΍ΧʔυήʔϜͳͲɻ ө૾ͷճసͱ͸ʁ

  73. ɾCMSampleBufferͷத਎͸ॎ௕ͷը૾ ɾCMSampleBufferͷϝλ৘ใʹճస৘ใؚ͕·Ε͍ͯΔ ճస৘ใͷऔಘ

  74. extension CMSampleBuffer { var orientation: CGImagePropertyOrientation { guard let attachment

    = CMGetAttachment(self, RPVideoSampleOrientationKey, nil) as? NSNumber else { return .up } if let orientation = CGImagePropertyOrientation(rawValue: orientationAttachment.uint32Value) { return orientation } return .up } } VQ MFGU SJHIU EPXO https://developer.apple.com/documentation/imageio/cgimagepropertyorientation CGImagePropertyOrientation
  75. ɾυΩϡϝϯτͰ͸iOS11.0+͕ͩ… ɾiOS11.2.1ҎલͰ͸ىಈ࣌ʹϦϯΫΤϥʔͰڧ੍ऴྃ ɾiOS11.2.5Ҏ߱͸໰୊ͳ͠ CMGetAttachment(self, "RPSampleBufferVideoOrientation" as CFString, nil) RPVideoSampleOrientationKey ɾ11.2.5Ҏ߱ͰऔಘͰ͖Δఆ਺Λϕλଧͪ

    ɾ11.2.1ҎલͰ͸nil͕औಘ͞ΕΔ ɾ11.2.1ҎલͰ΋ىಈͰ͖Δ͜ͱ͕ॏཁ https://developer.apple.com/documentation/replaykit/rpvideosampleorientationkey
  76. ഑৴։࢝࣌ʹํ޲ͱղ૾౓Λݻఆɺ ը૾Λճస(CoreImageͷoriented)ͤͯ͞Scale Fit͢Δ ѻ͍ํ1 ॎݻఆ ԣݻఆ

  77. ϝϦοτ σϝϦοτ ɾࢹௌऀ͕ݟ΍͍͢ํ޲ʹճస͍ͯ͠Δ ɾϑϨʔϜͷม׵͕ੜ͡Δ ɾ༨ന͕ೖͬͯ͠·͏ ɾෳ਺ͷΞϓϦͷߦ͖དྷΛ͢Δͱ͖ʹɺ ͜ͷඳըʹͳΔ͜ͱΛ഑৴ऀ͕ؾʹ͠ͳ͚Ε͹ͳΒͳ͍ ѻ͍ํ1

  78. ಈը͸ॎݻఆʹͯ͠ɺө૾͸ճసͤ͞ͳ͍ɻ αʔόʔʹճస৘ใΛૹΓɺϓϨʔϠʔଆͰճసΛαϙʔτ up right left down ѻ͍ํ2

  79. ϝϦοτ σϝϦοτ ɾ༨ന͕ͳ͍ ɾճసॲཧͷඞཁ͕ͳ͍ ɾࢹௌऀ͕ݟΔॠؒʹΑͬͯ͸ํ޲͕Ϛον͠ͳ͍ ɾϓϨΠϠʔ(ϓϥοτϑΥʔϜ)ຖʹରԠ͕ඞཁ ίετ͕ߴ͍ը૾ॲཧΛ1ͭল͚Δˍڐ༰Ͱ͖ΔσϝϦοτ ѻ͍ํ2

  80. ഑৴ͷऴྃ

  81. ഑৴ͷऴྃγʔέϯε •WebSocketΛ੾Δ •RTMPΛ੾அ͢Δ •αʔόʔʹ഑৴ऴྃAPIΛ౤͛Δ •AppͱExtensionͷڞ௨ྖҬʹอଘ͍ͯ͠Δ഑৴ʹؔ ࿈͢ΔσʔλΛ࡟আ͢Δ •ҟৗऴྃͰ͋Ε͹ͦͷঢ়ଶΛอଘ͢Δ ഑৴ͷऴྃAPIͱσʔλ࡟আ͸τϥϯβΫγϣϯ

  82. DispatchGroup override open func broadcastFinished() { let dispathGroup = DispatchGroup()

    dispathGroup.enter() broadcastService?.finish { dispathGroup.leave() } dispathGroup.wait() } DispatchGroupͰεϨουΛఀࢭͤ͞Δ
  83. BroadcastService func finish(completion: (()->Void)) { api.finishStream(movieID: 123) { completion?() }

    } αʔϏε಺ͰAPIΛड͚औͬͨΒcompletionΛ࣮ߦ
  84. ը໘ऩ࿥Λऴྃͤ͞Δํ๏ •εςʔλεόʔ͔Βऴྃ •ίϯτϩʔϧηϯλʔ͔Βऴྃ •ΞϓϦ͔Βऴྃ

  85. ΞϓϦέʔγϣϯ͔Βऴྃ override open func finishBroadcastWithError(_ error: Error) { super.finishBroadcastWithError(error) }

    RPBroadcastSampleHandlerͰ Τϥʔऴྃͤ͞Δඞཁ͕͋Δ https://developer.apple.com/documentation/replaykit/rpbroadcastsamplehandler/2721526-finishbroadcastwitherror
  86. ഑৴ͷҟৗऴྃͱ࠶։

  87. ഑৴Λҟৗऴྃͤ͞Δ override open func finishBroadcastWithError(_ error: Error) { super.finishBroadcastWithError(error) }

    ద੾ʹΤϥʔΛૹΔ͜ͱͰɺ μΠΞϩάʹऴྃཧ༝͕දࣔͰ͖Δ
  88. ഑৴Λҟৗऴྃͤ͞Δ let message = “ωοτϫʔΫ͕੾அ͞Ε·ͨ͠" let error = NSError( domain:

    Bundle.main.bundleIdentifier ?? “" , code: 1 , userInfo: [NSLocalizedFailureReasonErrorKey: message]) finishBroadcastWithError(error) userInfoʹϝοηʔδΛࢦఆ͠ͳ͍ͱ ऴྃཧ༝͕(null)ʹͳΓ·͢ɻ
  89. ҙਤ͠ͳ͍ऴྃͱͯ͠ ૝ఆ͞ΕΔέʔε •ը໘͕ϩοΫ͞ΕΔ(࣋ͪํͰిݯϘλϯΛԡͨ͠) •ి஑͕੾ΕΔ •ి࿩͕དྷΔ •BadAccessͰམͪΔ

  90. ҟৗ഑৴͕ఀࢭͨ͠৔߹ɺ࠶ ։͍ͤͯ͋͛ͨ͞ •഑৴ͷ਺ࣈͷ෼ࢄ(ࢹௌ਺΍ίϝϯτɺΤʔϧ) •ࢹௌऀͷମݧ •৽͍͠഑৴࿮΁ͷҠಈʹΑΔτϥϑΟοΫͷε ύΠΫ

  91. ి஑੾ΕɺిݯOFFɺϩοΫ • RPBroadcastSampleHandlerͷfinish()͕ݺ͹ΕΔ • ਖ਼ৗऴྃѻ͍ʹ͢Δ͔͠ͳ͍ • OSͷ௨஌͸͋Δ͕ඇެ։APIͷͨΊϦδΣΫτ͞ΕΔ

  92. ௨࿩ͷண৴ iOS͸௨࿩Λண৴͢Δͱɺ RPBroadcastSampleHandlerͷfinish͕ݺ͹ΕΔ ͦͷͨΊɺਖ਼ৗܥͷ഑৴ऴྃγʔέϯεʹೖͬͯ ͠·͏ ண৴Λݕ஌ͨ࣌͠͸ɺ finish͕ݺ͹Εͯ΋தஅѻ͍ʹ͢Δ

  93. ௨࿩ͷண৴ import CallKit class SampleHandler: RPBroadcastSampleHandler { let callObserver =

    CXCallObserver() override init() { super.init() callObserver.setDelegate(self, queue: nil) } } extension SampleHandler: CXCallObserverDelegate { func callObserver(…) { broadcastService.suspendStreaming() // ഑৴Λதஅ͢Δ } }
  94. ωοτϫʔΫͷ੾அ • Reachability.swiftΛ࠾༻ • RxͰ؂ࢹ͓ͯ͘͠ • ωοτϫʔΫ͕੾Εͨ৔߹͸ҟৗऴྃͱ͠ɺ ୺຤ʹதஅঢ়ଶΛอଘ͢Δ

  95. AppͱExtensionͷσʔλڞ༗

  96. ڞ༗͢Δσʔλ • ೝূ৘ใ • ഑৴ઃఆ(ը࣭) • ഑৴࿮(഑৴ج൫ͷ઀ଓઌͳͲͷ৘ใ) • ୺຤ઃఆ(௨஌ɺ֖ֆ) •

    தஅ৘ใ
  97. AppGroupΛ࢖ͬͯσʔλΛ ڞ༗͢Δ • ෳ਺ͷΞϓϦؒͰͷσʔλڞ༗ઃఆ • ಉ͡σϕϩούʔͷΞϓϦؒͰ༗ޮ • AppID͝ͱʹ༗ޮԽ͠ɺEntitlementͳͲͷ ઃఆ͕ඞཁ let

    userDefaults: UserDefaults = UserDefaults(suiteName: “groups.jp.iosdc2018”) IDΛࢦఆ͢Δ͜ͱͰάϧʔϓؒͷڞ༗ྖҬʹΞΫηεͰ͖Δ
  98. ExtensionΛσόοά͢Δ

  99. ϒϨΠΫϙΠϯτͷషΓํ

  100. ϒϨΠΫϙΠϯτͷషΓํ

  101. Waitingঢ়ଶʹͳ͔ͬͯΒɺ ഑৴Λ։࢝͢Δ

  102. Appͱಉ͡Α͏ʹΠϯεϖΫ τͰ͖·͢

  103. Appͱಉ͡Α͏ʹΠϯεϖΫ τͰ͖·͢

  104. ΩϟϓνϟͷεϨου BreakpointͰࢭΊͯΔͱ ഑৴ϓϩηε͕ࢮʹ·͢

  105. ͳͷͰLoggerΛ࢖͍·͠ΐ͏ • TCP Socket Logger • Logboard/LogboardConsole • JustLog •

    File Logger • XCGLogger • SwiftyBeaver
  106. ·ͱΊ

  107. ·ͱΊ • ReplayKitʹΑͬͯΧδϡΞϧʹੜ์ૹαʔϏε ͕࣮૷Ͱ͖ΔΑ͏ʹͳͬͨʂ • CMSampleBufferΛฤू͢Δ͜ͱͰө૾Λࠩ͠ସ ͑ͨΓΤϑΣΫτΛ͔͚ͨΓ͢Δ͜ͱ͕Ͱ͖Δ! • ഑৴தʹ୺຤͕ى͜ΓಘΔΠϕϯτͷϋϯυϦϯ ά͸ΞϓϦʹൺ΂ͯൈ͚ͳ͘΍Βͳͯ͘͸ͳΒͳ

    ͍
  108. Thanks • HaishinKit(https://github.com/ shogo4405/HaishinKit.swift)