$30 off During Our Annual Pro Sale. View Details »

Live Streaming with Screen Recording

Live Streaming with Screen Recording

Hideki Matsuoka

September 01, 2018
Tweet

More Decks by Hideki Matsuoka

Other Decks in Technology

Transcript

  1. ೋบ͘Β͍͋Δ
    ը໘ऩ࿥͔Βͷੜ์ૹ
    NBUTVPLBI

    View Slide

  2. 01&/3&$UW$ZCFS; *OD $ZCFS"HFOU *OD

    5XJUUFS!NBUTVPLBI@
    (JUIVCNBUVPLBI
    झຯ
    ήʔϜ࣮گΛݟΔ͜ͱɺ࣮گੜ์ૹ͢Δ͜ͱ
    ಛʹ16#( 4QMBUPPO
    দԬलथ

    View Slide

  3. • https://openrec.tv
    • ήʔϜಛԽͷੜ์ૹαʔϏε
    • MAU 200ສ
    • ߴը࣭ɺ௿஗Ԇ഑৴ɺ௥͍͔͚࠶ੜɺ̐ϲࠃޠ(CJK+E)ରԠ
    • Web/iOS/tvOS/Android/AndroidTV

    View Slide

  4. View Slide

  5. ൃදʹҠΔલʹ

    View Slide

  6. E
    CyberAgent group iOS BeerBash։࠵!
    ݄೔ Ր
    ɹ࣌෼։࢝ʢ࣌ΑΓड෇։࢝ʣ
    αΠόʔΤʔδΣϯτʢौ୩ϓϥΠϜϓϥβ'ɹΫϦΤΠςΟϒϥ΢ϯδʣ
    dɹ֓ཁઆ໌ɺסഋ
    dɹ׻ஊ
    dɹ৽ઃͷ৽نαʔϏε։ൃ૊৫ɹ$"54ʢΫϥΠΞϯτٕज़ྖҬΛݗҾ͢Δ૊৫ʣʹ͍ͭͯ
    dɹϚονϯάΞϓϦʮλοϓϧ஀ੜʯʹ͓͚Δ։ൃͷมԽ
    dɹ׻ஊ
    ͝Ԡื͸23ίʔυ͔Β
    ͪ͜Β͔ΒͷΤϯτϦʔ͸શһ͝ࢀՃ͍͚ͨͩ·͢
    connpass͔ΒΤϯτϦʔͷ৔߹ɺࢀՃ͸நબͱͳΓ·͢
    ฐάϧʔϓͷiOSDCొஃऀ͕ࢀՃ͢ΔɹBeerBashͰ͢ʂ
    দԬ΋ࢀՃ͠·͢ʂ
    ೔࣌
    ৔ॴ
    ಺༰
    ɹɹɹɹɹɹɹגࣜձࣾαΠόʔΤʔδΣϯτ$MJFOU"EWBODFE5FDIOPMPHZ4UVEJP $"54
    ɹҏ౻ɹګฏ
    ɹɹɹɹɹɹɹגࣜձࣾϚονϯάΤʔδΣϯτɹ∁ڮɹ༏հ

    View Slide

  7. ͜ͷൃදͰ͸Sli.doΛ࢖͍·͢
    ൃදͷ్தʹSli.do͔ΒΞϯέʔτΛ͠ͳ͕Β࠷ޙʹൃද
    ్தͰ࣭໰ͨ͘͠ͳͬͨΒSli.doʹ!
    ࠷ޙͷ࣭໰λΠϜͰर͍·͢
    Event Code: iosdc2018_openrec
    URL: https://app2.sli.do/event/0kwbnzqe

    View Slide

  8. ͦΕͰ͸ൃදʹҠΓ·͢

    View Slide

  9. ΞδΣϯμ
    •HLSͷ֓ཁ
    •ReplayKitͷ֓ཁ
    •ը໘ऩ࿥ͷϥΠϑαΠΫϧ
    •SampleBufferͷฤू
    •ੜ์ૹதͷૢ࡞
    •഑৴ͷҟৗऴྃͱ࠶։
    •iOS AppͱBroadcaster Upload Extensionͷσʔλڞ༗
    •ExtensionΛσόοά͢Δ

    View Slide

  10. HLSͷ֓ཁ

    View Slide

  11. HLSͷ֓ཁ
    HTTP Live Streaming(RFC8216)ͷུ
    ϓϨʔϠʔ͕ϓϨΠϦετͱࡉ੾Εʹͳ͍ͬͯΔಈըΛ
    HTTPϦΫΤετͰμ΢ϯϩʔυ͠ɺ
    ϓϨΠϦετʹఆٛ͞Ε͍ͯΔॱʹ࿈ଓతʹ࠶ੜ͢Δ
    https://iosdc.com/playlist.m3u8
    https://iosdc2018.com/1.ts
    https://iosdc2018.com/2.ts
    https://iosdc2018.com/3.ts
    ϓϨΠϦετ
    2ඵຖͷಈը
    ≒mp4

    View Slide

  12. playlit.m3u8ͷத਎
    #EXTM3U
    #EXT-X-VERSION:3
    #EXT-X-TARGETDURATION:2
    #EXT-X-MEDIA-SEQUENCE:1
    #EXT-X-PROGRAM-DATE-TIME:
    2018-08-31T12:15:12.713+09:00
    #EXTINF:2.0,
    1.ts
    #EXTINF:2.0,
    2.ts
    #EXTINF:2.0,
    3.ts
    #EXT-X-ENDLIST

    View Slide

  13. ࢓༷ʹଇͬͨϑΝΠϧΛΞοϓϩʔυͯ͠
    ϓϨΠϠʔʹμ΢ϯϩʔυͯ͠΋Β͏
    CDNΛ༗ޮ׆༻Ͱ͖ΔͨΊ
    εέʔϥϏϦςΟ/ίετͷ໘Ͱ
    ੜ์ૹͷٕज़తϋʔυϧΛԼ͛ΒΕΔ
    HLSͷ࠷େͷϝϦοτ

    View Slide

  14. HLSʹΑΔ
    ੜ์ૹαʔϏεͷશମ૾

    View Slide

  15. ഑৴ΞϓϦ
    OBS
    ഑৴ج൫
    ΞϓϦ
    frame
    .ts
    .m3u8
    ഑৴ऀ
    HLSʹΑΔಈը഑৴ͷશମ૾
    Storage/CDN
    ಈը৘ใ
    URI

    View Slide

  16. ഑৴ج൫ͷ໾ׂ
    1. ड͚औͬͨϝσΟΞͷϑϨʔϜΛ߹੒ͯ͠
    ɹ਺ඵ୯ҐͷಈըΛੜ੒͠ɺtsϑΝΠϧͱͯ͠σϓϩΠ
    2. σϓϩΠ͞Εͨಈը৘ใΛϓϨΠϦετʹ௥ه͢Δ
    ഑৴ΞϓϦ
    OBS
    ഑৴ج൫
    frame
    .ts
    .m3u8
    Storage/CDN

    View Slide

  17. ഑৴ΞϓϦ
    OBS
    ഑৴ج൫
    ΞϓϦ
    ಈը৘ใ
    ࢹௌऀ
    frame
    .ts
    .m3u8
    ಈը৘ใ
    .ts
    .m3u8
    ഑৴ऀ
    Storage/CDN
    HLSʹΑΔಈը഑৴ͷશମ૾
    URI

    View Slide

  18. ഑৴ΞϓϦ
    OBS
    Storage/CDN
    ഑৴ج൫
    ΞϓϦ
    ಈը৘ใ
    ࢹௌऀ
    .ts
    .m3u8
    frame
    .ts
    .m3u8
    ಈը৘ใ
    ഑৴ऀ
    ࠓճѻ͏෦෼
    ɾReplayKit
    ɾBroadcast Upload Extension
    URI

    View Slide

  19. ReplayKitͷ֓ཁ

    View Slide

  20. ReplayKitͷ֓ཁ(1)
    •iOSͷεΫϦʔϯΩϟϓνϟΛѻ͏SDK
    •ϚΠΫ΍ΞϓϦͷԻ΋࿥ԻՄೳ
    •iOS9͔Βར༻Մೳ
    •ήʔϜ΍ϖΠϯτܥΞϓϦɺVTuberΞϓϦ
    ʹ૊Έࠐ·Ε͍ͯΔ

    View Slide

  21. iOS9
    iOS10
    iOS11
    ΞϓϦʹ૊ΈࠐΜͰ࿥ը͕Ͱ͖Δ
    ૊ΈࠐΜͩΞϓϦ͕
    ExtensionʹରԠ͍ͯ͠Δ഑৴ΞϓϦʹॲཧΛҕৡͰ͖Δ
    ඪ४Ͱը໘ऩ࿥͕࣮૷
    ReplayKitͷ֓ཁ(2)

    View Slide

  22. •഑৴ͷ࣮૷ΛϓϥοτϑΥʔϜଆʹ೚ͤΒΕΔ
    •୺຤ͷϥΠϑαΠΫϧͷϋϯυϦϯά͕ߟྀࡁΈ
    •ࣗલ࿥ըΑΓύϑΥʔϚϯε͕ྑ͍
    •UI΍௨஌ͳͲ͕Ωϟϓνϟ͞ΕΔ
    ϝϦοτ
    σϝϦοτ
    ReplayKitΛ͔ͭ͏ϝϦσϝ

    View Slide

  23. https://developer.apple.com/videos/play/wwdc2018/601/
    WWDC2018Ͱ΋ൃද

    View Slide

  24. ը໘ऩ࿥Λड͚෇͚Δʹ͸
    Broadcast Upload Extension
    Λ࣮૷͢Ε͹OK

    View Slide

  25. Broadcast Upload
    Extension λʔήοτΛ࡞੒

    View Slide

  26. Broadcast Upload
    Extension λʔήοτΛ࡞੒
    ͜ͷΑ͏ʹɺը໘ऩ࿥ͷબ୒ࢶʹ
    ΞϓϦ͕ग़ΔΑ͏ʹͳΓ·͢ɻ

    View Slide

  27. ը໘ऩ࿥ͷϥΠϑαΠΫϧ

    View Slide

  28. ը໘ऩ࿥ͷϥΠϑαΠΫϧ
    Start
    Recording
    Finish
    ϕʔεͱͳΔϥΠϑαΠΫϧ
    αʔϏεຖͷ࣮૷͕֤ॴʹ૊Έࠐ·ΕΔ
    RPBroadcastSampleHandlerͷ
    ϝιουͱඥ෇͘

    View Slide

  29. class SampleHandler: RPBroadcastSampleHandler {
    override func broadcastStarted(…) {
    // ഑৴։࢝
    }
    override func processSampleBuffer(…) {
    // ը໘ऩ࿥ϓϩηε͔ΒΩϟϓνϟ͞ΕͨϑϨʔϜͷड͚औΓ
    }
    override func finishBroadcastWithError() {
    // ഑৴ऴྃ
    }
    override func finishBroadcastWithError(…) {
    // ҟৗऴྃ
    }
    }
    ΤϯτϦʔϙΠϯτ
    SampleHandler.swift
    https://developer.apple.com/documentation/replaykit/rpbroadcastsamplehandler

    View Slide

  30. ֤ϥΠϑαΠΫϧຖͷΞΫγϣϯ
    Start
    Recording
    Finish
    εϖοΫνΣοΫ/ೝূ
    ഑৴ઌͷղܾ
    ഑৴ج൫ʹ઀ଓ
    Τϯίʔυͱ഑৴
    ઀ଓΛ੾அ
    ഑৴த
    ഑৴։࢝
    ഑৴ऴྃ
    ঢ়ଶ ঢ়ଶຖͷΞΫγϣϯ
    ୺຤ͷঢ়ଶΛ؂ࢹ

    View Slide

  31. Start
    εϖοΫνΣοΫ/ೝূ
    ഑৴ઌͷղܾ
    ഑৴ج൫ʹ઀ଓ
    ɾ഑৴Ͱ͖ΔεϖοΫͷ୺຤͔νΣοΫ
    ɾೝূ৘ใɺ഑৴ݖݶνΣοΫ
    ɾதஅ͞Ε͍ͯΔ࿮͕͋Δ͔νΣοΫ
    ɾ഑৴࿮ͷ࡞੒
    ɾ഑৴ج൫ͷΞΫηεઌΛαʔόʔ͔Βऔಘ
    ɾRTMPͷ઀ଓ(ө૾/Ի੠)
    ɾWebSocketͷ઀ଓ(ίϝϯτͷண৴)

    View Slide

  32. Recording
    Τϯίʔυͱ഑৴
    ɾϝσΟΞຖʹΤϯίʔυ(h264/AAC)͠ɺૹ৴
    ɾόοϑΝͷฤू/ࠩ͠ସ͑
    ୺຤ͷঢ়ଶΛ؂ࢹ
    ɾը໘ͷճస৘ใ
    ɾ഑৴ઃఆͷมߋ
    ɾը໘ڞ༗ΦϑϞʔυ
    ɾ௨஌ͷON/OFF
    ɾ௨࿩ͷண৴
    ɾ୺຤ͷϩοΫɺి஑੾Ε
    ɾωοτϫʔΫঢ়گͷ؂ࢹ
    ɾΞϓϦ಺͔Βͷऴྃ
    ಈըͷݟ͑ํ தஅ/ऴྃ/ϦτϥΠ

    View Slide

  33. Finish
    ઀ଓΛ੾அ
    ɾRTMPΛ੾அ
    ɾWebSocketΛ੾அ
    ɾ഑৴ऴྃAPI
    ഑৴ͷઃఆ৘ใΛ࡟আ
    ɾAppGroup಺ͷ഑৴ઃఆ৘ใΛ࡟আ

    View Slide

  34. Recordingதͷ
    ϝσΟΞαϯϓϦϯά

    View Slide

  35. όοϑΝͷૹ৴
    HaishinKitʹͯରԠͨ͠ͷͰࠓճ͸Τϯίʔυɾ
    RTMPʹΑΔόοϑΝͷૹ৴ʹ͸৮Ε·ͤΜɻ
    HaishinKit͸ϝσΟΞͷΤϯίʔυ͓ΑͼɺετϦʔϛϯάαʔόʔ
    ͱͷRTMP௨৴Λड͚࣋ͬͯ͘ΕΔϥΠϒϥϦͰ͢ɻ
    https://github.com/shogo4405/HaishinKit.swift

    View Slide

  36. ը໘ऩ࿥ϓϩηε͔Βड͚औ
    ΔϝσΟΞσʔλ
    func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with
    sampleBufferType: RPSampleBufferType) {
    // process each media data
    }
    ϝσΟΞͷੜσʔλɻόΠτ৘ใ͔Β
    ֤ϝσΟΞʹ෮ݩͯ͠ฤूͨ͠ΓΤϯίʔυͨ͠Γ͢Δ
    video, audioApp, audioMicͷ̏छྨ
    sampleBufferType
    sampleBuffer

    View Slide

  37. videoϑϨʔϜ

    View Slide

  38. SampleBuffer(video)
    ϑϨʔϜͷ৘ใؚ͕·Ε͍ͯΔ
    ɾը૾ͷੜσʔλ
    ɾը૾ͷճస৘ใ
    ɾը૾ͷαΠζ
    ɾ࠶ੜ։࢝࣌ؒ(Presentation Time Stamp)

    View Slide

  39. NT NT NT NT NT NT
    SampleBuffer(video)
    t
    154
    #VGGFS
    αϯϓϦϯάλΠϛϯά͸ը໘ʹߋ৽͕͋ͬͨͱ͖
    Duration͸લճͷαϯϓϦϯά͔ΒࠓճͷαϯϓϦϯά

    View Slide

  40. ϗʔϜը໘ͷ഑৴Λ͠ͳ͕Β
    ์ஔ͍ͯͨ͠Β…
    ҆ఆ࠶ੜͰ͖ͳ͍ࣄ৅ʹؕͬͨ
    ΩʔϑϨʔϜΛੜ੒ִؒΛ2ඵʹ͍ͯ͠Δʹ΋
    ؔΘΒͣ6ඵʹͳͬͯ͠·ͬͨΓ
    1ts͋ͨΓಈըͷ୹͞΍
    ҆ఆ͠ͳ͍tsͷduration͸ϥάʹ௚݁͢Δ

    View Slide

  41. NT NT
    ը໘ͷߋ৽͕ͳ͍৔߹
    154
    t
    #VGGFS
    ͜ͷؒ͸ߋ৽͕ͳ͘ϑϨʔϜ͕ૹΒΕͣ
    ΩʔϑϨʔϜ΋ૹΕͳ͍ͨΊ
    ಈըͷ۠੾Γ͕͏·͘Ͱ͖ͳ͍

    View Slide

  42. ಈըΛ࠶ݱ͢ΔͨΊʹbitmap৘ใΛ͢΂ؚͯΜͩϑϨʔϜ
    ࣮ࡍͷ
    ϑϨʔϜ
    t
    ಈը
    ΩʔϑϨʔϜ(I Frame)ͱ͸

    View Slide

  43. ࣮ࡍͷ
    ϑϨʔϜ
    I Frame P Frame
    ࠩ෼ͷΈΛ࣋ͭ͜ͱͰσʔλྔΛѹॖ
    HLSʹ͓͍ͯI Frame͸1ͭͷtsϑΝΠϧຖʹ
    ࠷ॳͷϑϨʔϜͱؚͯ͠ΜͰ͍ͳ͚Ε͹ͳΒͳ͍
    ΩʔϑϨʔϜ(I Frame)ͱ͸
    ಈըΛ࠶ݱ͢ΔͨΊʹbitmap৘ใΛ͢΂ؚͯΜͩϑϨʔϜ

    View Slide

  44. NT NT
    154
    t
    #VGGFS
    videoϑϨʔϜͷϦΧόϦͰରԠ
    ؒΛຒΊΔϑϨʔϜΛ͍ΕΕ͹OK

    View Slide

  45. t
    NT NT NT NT NT NT
    videoϑϨʔϜͷϦΧόϦͰରԠ
    154
    ϑϨʔϜΛૹ৴ͯ͠Ұఆ͕࣌ؒܦͬͯ΋࣍ͷϑϨʔϜ͕ͳ͍࣌
    ࠷ޙʹૹ৴ͨ͠ϑϨʔϜΛɺPTSΛ෇͚ସ͑ͯૹ৴͢Δ
    #VGGFS

    View Slide

  46. PTS͕ॏཁ
    ɾPTSͷζϨ͸ಈըͷԻζϨͷݪҼ
    ɾόοϑΝͷૢ࡞Ͱ΋͜ͷPTSΛѻ͏
    ɾPTSؚΊɺಈը΍Ի੠ͷ࣌ؒ৘ใ͸CMTimeʹΑͬ
    ͯϑϨʔϜͰѻ͑Δܗࣜʹͳ͍ͬͯΔ

    View Slide

  47. CMTime
    ࣌ؒ৘ใΛ཭ࢄత(ϑϨʔϜ)ʹ੔਺Ͱѻ͏
    public struct CMTime {
    public var value: CMTimeValue
    public var timescale: CMTimeScale
    public var flags: CMTimeFlags
    public var epoch: CMTimeEpoch
    public init()
    public init(value: CMTimeValue, timescale: CMTimeScale, flags: CMTimeFlags, epoch: CMTimeEpoch)
    }
    extension CMTime {
    public init(seconds: Double, preferredTimescale: CMTimeScale)
    public init(value: CMTimeValue, timescale: CMTimeScale)
    }
    value: ϑϨʔϜ਺
    timescale: 1ඵ͋ͨΓΛԿϑϨʔϜʹ෼ׂ͢Δ͔ʁ

    View Slide

  48. CMTimeͷσʔλͷྫ
    7JEFP CMTime(value: 1, timeScale: 30)
    "VEJP CMTime(value: 1024, timeScale: 44100)
    ≒0.33333…

    View Slide

  49. ΩʔϑϨʔϜૹ৴ִؒ
    ֤αʔϏεͷਪ঑KeyFrame Interval͸2ඵ
    ɾYouTube
    ɾTwitch
    ɾOPENREC.tv
    ɾFRESH LIVE
    1ͭͷts͋ͨΓΛ2ඵͷಈըʹ͍ͨ͠

    View Slide

  50. audioϑϨʔϜ

    View Slide

  51. CMSampleBuffer(audioMic)
    ϑϨʔϜͷ৘ใؚ͕·Ε͍ͯΔ
    ɾԻ੠ͷੜσʔλ
    ɾLinear PCM
    ɾ࠶ੜ։࢝࣌ؒ(Presentation Time Stamp)
    ɾ௕͞(Duration)

    View Slide

  52. NT NT NT NT NT
    CMSampleBuffer(audioMic)
    154
    㽈 㽈 㽈 㽈
    t

    αϯϓϦϯάλΠϛϯά͸ఆظతɻ
    ִؒ͸1024/44100ඵ(≒23ms)
    ᶃ ᶄ ᶅ ᶆ ᶇ
    #VGGFS

    View Slide

  53. NT NT NT NT NT
    CMSampleBuffer(audioMic)
    154
    㽈 㽈 㽈 㽈
    t

    ϏσΦόοϑΝ͸ࠓͷදࣔͷόοϑΝʹର͠
    ΦʔσΟΦόοϑΝ͸ࠓ·Ͱ࿥Ի͞ΕͨόοϑΝΛड͚औΔ
    㽈 㽈 㽈 㽈 㽈
    औಘ
    λΠϛϯά
    ᶃ ᶄ ᶅ ᶆ ᶇ
    ᶃ ᶄ ᶅ ᶆ ᶇ
    #VGGFS

    View Slide

  54. NT NT NT NT NT
    CMSampleBuffer(audioApp)
    154
    㽈 㽈 㽈 㽈
    t

    ϚΠΫಉ༷αϯϓϦϯάλΠϛϯά͸΄΅ఆظతɻ
    ִؒ͸22050/44100ඵલޙ(≒500ms)
    ᶃ ᶄ ᶅ ᶆ ᶇ
    #VGGFS

    View Slide

  55. ө૾ͷฤू

    View Slide

  56. ө૾ͷฤू
    ө૾ͷฤूͱ͍ͬͯ΋CIImageʹΑΔฤू
    ·ͣCMSampleBuffer͔ΒCIImageܗࣜʹม׵͢Δ
    extension CMSampleBuffer {
    var ciImage: CIImage? {
    guard let pixelBuffer = CMSampleBufferGetImageBuffer(self) else {
    return nil
    }
    CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.readOnly);
    defer {
    CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags.readOnly)
    }
    return CIImage(cvPixelBuffer: pixelBuffer)
    }
    }
    CMSampleBuffer > CVPixelBuffer > CIImage

    View Slide

  57. CIImageͰͰ͖Δ͜ͱ
    ɾ֦େ/ॖখ(CGAffineTransform)
    ɾҠಈ(CGAffineTransform)
    ɾ੾ΓऔΓ(crop)
    ɾճస(oriented)
    ɾ΅͔͠(applyingGaussianBlur)
    ɾϑΟϧλʔ

    View Slide

  58. CMSampleBufferʹ໭͢
    // BlurΛద༻͢ΔͳͲͷฤू
    let newImage = sampleBuffer.ciImage.applyingGaussianBlur(sigma: 10)
    // ۣܗ৘ใ
    let dimensions = CMVideoFormatDescriptionGetDimensions(CMSampleBufferGetFormatDescription(sampleBuffer)!)
    // ϐΫηϧ৘ใͷॻ͖ࠐΈઌͷόοϑΝΛ࡞੒
    var outputPixelBuffer: CVPixelBuffer? = nil
    CVPixelBufferCreate(kCFAllocatorSystemDefault, Int(dimensions.width), Int(dimensions.height), kCVPixelFormatType_32BGRA, nil,
    &outputPixelBuffer)
    CVPixelBufferLockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0));
    defer {
    CVPixelBufferUnlockBaseAddress(outputPixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
    }
    // όοϑΝʹCIImageΛॻ͖ࠐΈ
    let ciContext = ContextHolder.shared.context
    ciContext.render(newImage, to: outputPixelBuffer!)
    // ৽͍͠SampleBufferΛ࡞੒
    var videoFormatDescription: CMVideoFormatDescription? = nil
    CMVideoFormatDescriptionCreateForImageBuffer(nil, outputPixelBuffer!, &videoFormatDescription)
    var sampleTimingInfo = self.sampleTimingInfo
    var newSampleBuffer: CMSampleBuffer? = nil
    CMSampleBufferCreateForImageBuffer(nil, outputPixelBuffer!, true, nil, nil, videoFormatDescription!, &sampleTimingInfo, &newSampleBuffer)

    View Slide

  59. // BlurΛద༻͢ΔͳͲͷฤू
    let newImage = sampleBuffer.ciImage.applyingGaussianBlur(sigma:
    10)
    CMSampleBuffer͔ΒCIImageΛऔಘ͠ɺ
    ϑΟϧλʔͳͲΛద༻͢Δ
    CMSampleBufferʹ໭͢

    View Slide

  60. var outputPixelBuffer: CVPixelBuffer? = nil
    CVPixelBufferCreate(kCFAllocatorSystemDefault,
    Int(dimensions.width), Int(dimensions.height),
    kCVPixelFormatType_32BGRA, nil, &outputPixelBuffer)
    CVPixelBufferLockBaseAddress(outputPixelBuffer!,
    CVPixelBufferLockFlags(rawValue: 0));
    defer {
    CVPixelBufferUnlockBaseAddress(outputPixelBuffer!,
    CVPixelBufferLockFlags(rawValue: 0))
    }
    CIImageͷॻ͖ࠐΈઌͷόοϑΝΛ࡞੒͢Δ
    CMSampleBufferʹ໭͢ᶃ

    View Slide

  61. // όοϑΝʹCIImageΛॻ͖ࠐΈ
    let ciContext = ContextHolder.shared.context
    ciContext.render(newImage, to: outputPixelBuffer!)
    ฤूͨ͠CIImageΛόοϑΝʹॻ͖ࠐΉ
    CMSampleBufferʹ໭͢ᶄ

    View Slide

  62. var videoFormatDescription: CMVideoFormatDescription? = nil
    CMVideoFormatDescriptionCreateForImageBuffer(nil,
    outputPixelBuffer!, &videoFormatDescription)
    var sampleTimingInfo = self.sampleTimingInfo
    var newSampleBuffer: CMSampleBuffer? = nil
    CMSampleBufferCreateForImageBuffer(nil, outputPixelBuffer!,
    true, nil, nil, videoFormatDescription!, &sampleTimingInfo,
    &newSampleBuffer)
    ॻ͖ࠐΜͩBufferͱݩͷCMSampleBufferͷ
    ࣌ؒ৘ใΛ࢖ͬͯ৽ͨͳCMSampleBufferΛੜ੒͢Δ
    CMSampleBufferʹ໭͢ᶅ

    View Slide

  63. ө૾ͷฤू
    ֖ֆʹࠩ͠ସ͑

    View Slide

  64. ͍ͨͩ·४උதʂ
    ͍ΘΏΔɺྲྀͤΔө૾͕ͳ͍
    ࣌ʹ”֖”Λ͢ΔͨΊͷը૾ͷ
    ͜ͱ
    ֖ֆͱ͸

    View Slide

  65. ɾύεϫʔυೖྗ
    ɾݸਓ৘ใͱͳΔΞϓϦΛ։͘
    ɾʮҰ౓഑৴Λऴྃ͢Ε͹͍͍ʯ͸ඍົ
    ΅͔͠Λݕ౼͕ͨ͠ɺ
    ύεϫʔυೖྗͰ͸
    ΩʔϘʔυͷҐஔ͔Β
    ೖྗ͕ਪଌͰ͖ΔͨΊ
    ׬શͳ֖ֆʹ
    ֖ֆͷར༻γʔϯ

    View Slide

  66. ɾ֖ֆͷը૾৘ใΛ࣋ͬͨSampleBufferΛىಈ࣌ʹੜ੒͓ͯ͘͠
    ɾը໘ऩ࿥ϓϩηε͔Βड͚औͬͨSampleBufferͷPTSʹࠩ͠ସ
    ֖͑ͯֆͷόοϑΝΛૹ৴͢Δ
    ը໘ऩ࿥͔Β
    ड͚औͬͨόοϑΝ
    ֖ֆ
    όοϑΝ
    PTSΛ
    ΋Β͏ ഑৴ج൫
    όοϑΝͷࠩ͠ସ͑

    View Slide

  67. NT NT NT NT NT NT
    t
    154
    'SBNF
    ڞ༗OFF
    όοϑΝͷࠩ͠ସ͑

    View Slide

  68. NT NT NT NT NT NT
    t
    154
    'SBNF
    ڞ༗OFF
    όοϑΝͷࠩ͠ସ͑

    View Slide

  69. όοϑΝͷࠩ͠ସ͑
    ֖ֆͷCMSampleBufferΛϗϧμʔʹอ͓͖࣋ͯ͠
    ετϦʔϜͷϋϯυϦϯάଆͰόοϑΝΛࠩ͠ସ͑
    class MaskHolder {
    static let shared = MaskHolder()
    private let maskBuffer: CMSampleBuffer
    func mask(with sampleTimingInfo: CMSampleTimingInfo) -> CMSampleBuffer {
    return maskBuffer.copy(with: sampleTimingInfo)
    }
    }
    class BroadcastService {
    var isEnabledMask: Bool
    let rtmpStream: RTMPStream
    func processSampleBuffer(sampleBuffer: CMSampleBuffer) {
    let buffer = isEnabledMask ? MaskHolder.shared.mask(with: sampleBuffer.sampleTimingInfo) :
    sampleBuffer
    rtmpStream.appendSampleBuffer(sampleBuffer: buffer)
    }
    }

    View Slide

  70. extension CMSampleBuffer {
    func copy(with timing: CMSampleTimingInfo) -> CMSampleBuffer? {
    var sampleTiming = timing
    var copied: CMSampleBuffer? = nil
    CMSampleBufferCreateCopyWithNewTiming(kCFAllocatorSystemDefault, self, 1, &sampleTiming, &copied)
    return copied
    }
    var sampleTimingInfo: CMSampleTimingInfo {
    return CMSampleTimingInfo(duration: self.duration, presentationTimeStamp: self.presentationTimeStamp,
    decodeTimeStamp: self.decodeTimeStamp)
    }
    var duration: CMTime {
    return CMSampleBufferGetDuration(self)
    }
    var presentationTimeStamp: CMTime {
    return CMSampleBufferGetPresentationTimeStamp(self)
    }
    var decodeTimeStamp: CMTime {
    return CMSampleBufferGetDecodeTimeStamp(self)
    }
    }
    CMSampleBuffer͸
    ؔ਺ݺͼग़͠ʹΑΔϓϩύςΟऔಘ͕ଟ͍ͷͰ
    extensionʹcomputed propertyΛੜ΍͓ͯ͘͠ͱศར
    όοϑΝͷࠩ͠ସ͑

    View Slide

  71. ө૾ͷճస

    View Slide

  72. iOSͷը໘͸഑৴தʹํ޲͕มΘΔ
    ͜ΕΛͲ͏ѻ͏͔ͱ͍͏τϐοΫ
    ήʔϜ͸ԣը໘΋ଟ͍ɻԻήʔ΍ΧʔυήʔϜͳͲɻ
    ө૾ͷճసͱ͸ʁ

    View Slide

  73. ɾCMSampleBufferͷத਎͸ॎ௕ͷը૾
    ɾCMSampleBufferͷϝλ৘ใʹճస৘ใؚ͕·Ε͍ͯΔ
    ճస৘ใͷऔಘ

    View Slide

  74. extension CMSampleBuffer {
    var orientation: CGImagePropertyOrientation {
    guard let attachment = CMGetAttachment(self, RPVideoSampleOrientationKey, nil) as? NSNumber else {
    return .up
    }
    if let orientation = CGImagePropertyOrientation(rawValue: orientationAttachment.uint32Value) {
    return orientation
    }
    return .up
    }
    }
    VQ MFGU SJHIU EPXO
    https://developer.apple.com/documentation/imageio/cgimagepropertyorientation
    CGImagePropertyOrientation

    View Slide

  75. ɾυΩϡϝϯτͰ͸iOS11.0+͕ͩ…
    ɾiOS11.2.1ҎલͰ͸ىಈ࣌ʹϦϯΫΤϥʔͰڧ੍ऴྃ
    ɾiOS11.2.5Ҏ߱͸໰୊ͳ͠
    CMGetAttachment(self, "RPSampleBufferVideoOrientation" as CFString, nil)
    RPVideoSampleOrientationKey
    ɾ11.2.5Ҏ߱ͰऔಘͰ͖Δఆ਺Λϕλଧͪ
    ɾ11.2.1ҎલͰ͸nil͕औಘ͞ΕΔ
    ɾ11.2.1ҎલͰ΋ىಈͰ͖Δ͜ͱ͕ॏཁ
    https://developer.apple.com/documentation/replaykit/rpvideosampleorientationkey

    View Slide

  76. ഑৴։࢝࣌ʹํ޲ͱղ૾౓Λݻఆɺ
    ը૾Λճస(CoreImageͷoriented)ͤͯ͞Scale Fit͢Δ
    ѻ͍ํ1
    ॎݻఆ ԣݻఆ

    View Slide

  77. ϝϦοτ
    σϝϦοτ
    ɾࢹௌऀ͕ݟ΍͍͢ํ޲ʹճస͍ͯ͠Δ
    ɾϑϨʔϜͷม׵͕ੜ͡Δ
    ɾ༨ന͕ೖͬͯ͠·͏
    ɾෳ਺ͷΞϓϦͷߦ͖དྷΛ͢Δͱ͖ʹɺ
    ͜ͷඳըʹͳΔ͜ͱΛ഑৴ऀ͕ؾʹ͠ͳ͚Ε͹ͳΒͳ͍
    ѻ͍ํ1

    View Slide

  78. ಈը͸ॎݻఆʹͯ͠ɺө૾͸ճసͤ͞ͳ͍ɻ
    αʔόʔʹճస৘ใΛૹΓɺϓϨʔϠʔଆͰճసΛαϙʔτ
    up right left down
    ѻ͍ํ2

    View Slide

  79. ϝϦοτ
    σϝϦοτ
    ɾ༨ന͕ͳ͍
    ɾճసॲཧͷඞཁ͕ͳ͍
    ɾࢹௌऀ͕ݟΔॠؒʹΑͬͯ͸ํ޲͕Ϛον͠ͳ͍
    ɾϓϨΠϠʔ(ϓϥοτϑΥʔϜ)ຖʹରԠ͕ඞཁ
    ίετ͕ߴ͍ը૾ॲཧΛ1ͭল͚Δˍڐ༰Ͱ͖ΔσϝϦοτ
    ѻ͍ํ2

    View Slide

  80. ഑৴ͷऴྃ

    View Slide

  81. ഑৴ͷऴྃγʔέϯε
    •WebSocketΛ੾Δ
    •RTMPΛ੾அ͢Δ
    •αʔόʔʹ഑৴ऴྃAPIΛ౤͛Δ
    •AppͱExtensionͷڞ௨ྖҬʹอଘ͍ͯ͠Δ഑৴ʹؔ
    ࿈͢ΔσʔλΛ࡟আ͢Δ
    •ҟৗऴྃͰ͋Ε͹ͦͷঢ়ଶΛอଘ͢Δ
    ഑৴ͷऴྃAPIͱσʔλ࡟আ͸τϥϯβΫγϣϯ

    View Slide

  82. DispatchGroup
    override open func broadcastFinished() {
    let dispathGroup = DispatchGroup()
    dispathGroup.enter()
    broadcastService?.finish {
    dispathGroup.leave()
    }
    dispathGroup.wait()
    }
    DispatchGroupͰεϨουΛఀࢭͤ͞Δ

    View Slide

  83. BroadcastService
    func finish(completion: (()->Void)) {
    api.finishStream(movieID: 123) {
    completion?()
    }
    }
    αʔϏε಺ͰAPIΛड͚औͬͨΒcompletionΛ࣮ߦ

    View Slide

  84. ը໘ऩ࿥Λऴྃͤ͞Δํ๏
    •εςʔλεόʔ͔Βऴྃ
    •ίϯτϩʔϧηϯλʔ͔Βऴྃ
    •ΞϓϦ͔Βऴྃ

    View Slide

  85. ΞϓϦέʔγϣϯ͔Βऴྃ
    override open func finishBroadcastWithError(_ error: Error) {
    super.finishBroadcastWithError(error)
    }
    RPBroadcastSampleHandlerͰ
    Τϥʔऴྃͤ͞Δඞཁ͕͋Δ
    https://developer.apple.com/documentation/replaykit/rpbroadcastsamplehandler/2721526-finishbroadcastwitherror

    View Slide

  86. ഑৴ͷҟৗऴྃͱ࠶։

    View Slide

  87. ഑৴Λҟৗऴྃͤ͞Δ
    override open func finishBroadcastWithError(_ error: Error) {
    super.finishBroadcastWithError(error)
    }
    ద੾ʹΤϥʔΛૹΔ͜ͱͰɺ
    μΠΞϩάʹऴྃཧ༝͕දࣔͰ͖Δ

    View Slide

  88. ഑৴Λҟৗऴྃͤ͞Δ
    let message = “ωοτϫʔΫ͕੾அ͞Ε·ͨ͠"
    let error = NSError(
    domain: Bundle.main.bundleIdentifier ?? “"
    , code: 1
    , userInfo: [NSLocalizedFailureReasonErrorKey: message])
    finishBroadcastWithError(error)
    userInfoʹϝοηʔδΛࢦఆ͠ͳ͍ͱ
    ऴྃཧ༝͕(null)ʹͳΓ·͢ɻ

    View Slide

  89. ҙਤ͠ͳ͍ऴྃͱͯ͠
    ૝ఆ͞ΕΔέʔε
    •ը໘͕ϩοΫ͞ΕΔ(࣋ͪํͰిݯϘλϯΛԡͨ͠)
    •ి஑͕੾ΕΔ
    •ి࿩͕དྷΔ
    •BadAccessͰམͪΔ

    View Slide

  90. ҟৗ഑৴͕ఀࢭͨ͠৔߹ɺ࠶
    ։͍ͤͯ͋͛ͨ͞
    •഑৴ͷ਺ࣈͷ෼ࢄ(ࢹௌ਺΍ίϝϯτɺΤʔϧ)
    •ࢹௌऀͷମݧ
    •৽͍͠഑৴࿮΁ͷҠಈʹΑΔτϥϑΟοΫͷε
    ύΠΫ

    View Slide

  91. ి஑੾ΕɺిݯOFFɺϩοΫ
    • RPBroadcastSampleHandlerͷfinish()͕ݺ͹ΕΔ
    • ਖ਼ৗऴྃѻ͍ʹ͢Δ͔͠ͳ͍
    • OSͷ௨஌͸͋Δ͕ඇެ։APIͷͨΊϦδΣΫτ͞ΕΔ

    View Slide

  92. ௨࿩ͷண৴
    iOS͸௨࿩Λண৴͢Δͱɺ
    RPBroadcastSampleHandlerͷfinish͕ݺ͹ΕΔ
    ͦͷͨΊɺਖ਼ৗܥͷ഑৴ऴྃγʔέϯεʹೖͬͯ
    ͠·͏
    ண৴Λݕ஌ͨ࣌͠͸ɺ
    finish͕ݺ͹Εͯ΋தஅѻ͍ʹ͢Δ

    View Slide

  93. ௨࿩ͷண৴
    import CallKit
    class SampleHandler: RPBroadcastSampleHandler {
    let callObserver = CXCallObserver()
    override init() {
    super.init()
    callObserver.setDelegate(self, queue: nil)
    }
    }
    extension SampleHandler: CXCallObserverDelegate {
    func callObserver(…) {
    broadcastService.suspendStreaming() // ഑৴Λதஅ͢Δ
    }
    }

    View Slide

  94. ωοτϫʔΫͷ੾அ
    • Reachability.swiftΛ࠾༻
    • RxͰ؂ࢹ͓ͯ͘͠
    • ωοτϫʔΫ͕੾Εͨ৔߹͸ҟৗऴྃͱ͠ɺ
    ୺຤ʹதஅঢ়ଶΛอଘ͢Δ

    View Slide

  95. AppͱExtensionͷσʔλڞ༗

    View Slide

  96. ڞ༗͢Δσʔλ
    • ೝূ৘ใ
    • ഑৴ઃఆ(ը࣭)
    • ഑৴࿮(഑৴ج൫ͷ઀ଓઌͳͲͷ৘ใ)
    • ୺຤ઃఆ(௨஌ɺ֖ֆ)
    • தஅ৘ใ

    View Slide

  97. AppGroupΛ࢖ͬͯσʔλΛ
    ڞ༗͢Δ
    • ෳ਺ͷΞϓϦؒͰͷσʔλڞ༗ઃఆ
    • ಉ͡σϕϩούʔͷΞϓϦؒͰ༗ޮ
    • AppID͝ͱʹ༗ޮԽ͠ɺEntitlementͳͲͷ
    ઃఆ͕ඞཁ
    let userDefaults: UserDefaults
    = UserDefaults(suiteName: “groups.jp.iosdc2018”)
    IDΛࢦఆ͢Δ͜ͱͰάϧʔϓؒͷڞ༗ྖҬʹΞΫηεͰ͖Δ

    View Slide

  98. ExtensionΛσόοά͢Δ

    View Slide

  99. ϒϨΠΫϙΠϯτͷషΓํ

    View Slide

  100. ϒϨΠΫϙΠϯτͷషΓํ

    View Slide

  101. Waitingঢ়ଶʹͳ͔ͬͯΒɺ
    ഑৴Λ։࢝͢Δ

    View Slide

  102. Appͱಉ͡Α͏ʹΠϯεϖΫ
    τͰ͖·͢

    View Slide

  103. Appͱಉ͡Α͏ʹΠϯεϖΫ
    τͰ͖·͢

    View Slide

  104. ΩϟϓνϟͷεϨου
    BreakpointͰࢭΊͯΔͱ
    ഑৴ϓϩηε͕ࢮʹ·͢

    View Slide

  105. ͳͷͰLoggerΛ࢖͍·͠ΐ͏
    • TCP Socket Logger
    • Logboard/LogboardConsole
    • JustLog
    • File Logger
    • XCGLogger
    • SwiftyBeaver

    View Slide

  106. ·ͱΊ

    View Slide

  107. ·ͱΊ
    • ReplayKitʹΑͬͯΧδϡΞϧʹੜ์ૹαʔϏε
    ͕࣮૷Ͱ͖ΔΑ͏ʹͳͬͨʂ
    • CMSampleBufferΛฤू͢Δ͜ͱͰө૾Λࠩ͠ସ
    ͑ͨΓΤϑΣΫτΛ͔͚ͨΓ͢Δ͜ͱ͕Ͱ͖Δ!
    • ഑৴தʹ୺຤͕ى͜ΓಘΔΠϕϯτͷϋϯυϦϯ
    ά͸ΞϓϦʹൺ΂ͯൈ͚ͳ͘΍Βͳͯ͘͸ͳΒͳ
    ͍

    View Slide

  108. Thanks
    • HaishinKit(https://github.com/
    shogo4405/HaishinKit.swift)

    View Slide