iOSDC 2019 Day 0 17:50~ Track A
εΫϦʔϯ৴ػೳͷ࣮͕େมͩͬͨͷͰݟΛ͓͑͠·͢iOSDC Japan 20192019/09/05 17:50~ Track Apixiv.incFromAtom
View Slide
• Swift 1.2 ͔ΒiOSΞϓϦ։ൃͷੈք• ΞϓϦͷςοΫΛϦʔυ͍ͯ͠·͢• pixiv Sketch / pixiv Sketch LIVE iOSΞϓϦ։ൃ@FromAtom
͜ͷൃදͷΰʔϧ
ϥΠϒ৴ྲྀߦͬͯΔ͠ɺ͏ͪϥΠϒ৴Δͧʂʂʂ
ͱɺͳͬͨͱ͖ʹࠔΒͳ͍Α͏ʹ
εΫϦʔϯ৴ػೳ
εΫϦʔϯ৴ػೳͱεϚʔτϑΥϯͷը໘ΛϥΠϒ৴Ͱ͖Δػೳ
• εϚϗ୯ମͰϥΠϒ৴͕Մೳ• ήʔϜ࣮گ͓ֆ͔͖ͷ৴ʹར༻͞Ε͍ͯΔ• ࠷ۙVTuberք۾Ͱར༻͞Ε͍ͯΔεΫϦʔϯ৴ػೳͱ
ReplayKitͱεΫϦʔϯ৴
• ReplayKitΛར༻͢ΔͱεΫϦʔϯͷऩLIVE৴͕؆୯ʹͰ͖Δ• ϚΠΫԻΞϓϦ͕ग़ྗ͍ͯ͠ΔԻऔಘͰ͖Δ• ͨͩ͠ɺiOSͷόʔδϣϯʹΑͬͯͰ͖Δ͜ͱ͕ҧ͏ReplayKitͱεΫϦʔϯ৴
• iOS 9+‣ ΈࠐΜͩΞϓϦͷը໘ΛըͰ͖Δ‣ ৴Ͱ͖ͳ͍• iOS 10+‣ Broadcast Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ• iOS 11+‣ ༷ʑͳը໘͕৴ɾըͰ͖Δ‣ ίϯτϩʔϧηϯλʔ͔Βը໘ըͰ͖Δͷͱಉ͡ΈReplayKitͱεΫϦʔϯ৴
ࠓ͔Β࡞ΔͳΒ࠷ͰiOS 11+
͡Ό͋ReplayKitʹ͍ͭͯௐͯΈΑ͏
5͔݅͠هࣄ͕Ͷ͐ʂ
ͳͷͰɺͦΕͧΕͷ࡞ΓํΛղઆ
• ࣮ػͰ͔͠σόοάͰ͖ͳ͍‣ ΧϝϥPush௨ͱಉ͡• iOSඪ४ͷը໘ऩQuickTimeܦ༝Ͱͷը໘ըͱڝ߹͢Δ‣ σϞͷࡱӨΛ͢Δલʹؾ͍ͮͯযΒͳ͍Α͏ʹ͠·͠ΐ͏‣ Pixel3ͰiPhoneXΛࡱӨ͢ΔʑΛա͝͠·ͨͪ͠ΐͬͱͨ͠ҙ
ΞϓϦͷը໘ΛըͰ͖ΔiOS 9+
ը։࢝
ըऴྃ
iOS 10+ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ
• Broadcast Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ• ৴Λ͍ͨ͠ΞϓϦͱ৴͢ΔΞϓϦͷํͰ࣮͕ඞཁiOS10Ͱͷը໘৴
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮ -৴ऴྃ-
৴͢ΔΞϓϦͰͷ࣮• Broadcast Upload ExtensionΛ࣮͢Δ• ৄ͘͜͠ͷޙ iOS11+ ͷͱҰॹʹ͠·͢
iOS 11+Ͳͷը໘৴ɾըͰ͖Δ
• iOS11+ ͔ΒͰ͖ΔΑ͏ʹͳͬͨࣄ‣ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ৴iOS11+Ͱͷը໘৴
• ࣮ͨ͠ΞϓϦͰը໘ͷΩϟϓνϟ͕Ͱ͖Δ• ը໘ͷCMSampleBufferΛऔಘͯ͠ըɾ৴͕Մೳʹ• ϚΠΫԻΞϓϦԻऔಘՄೳRPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ
࣮
• ίϯτϩʔϧηϯλʔͷ˕͔ΒબͰ͖Δ• iOS9+ͱҧ͍ը໘શମ͕ऩͰ͖Δ‣ ௨ऩ͞ΕΔͷͰҙίϯτϩʔϧηϯλʔܦ༝ͷ৴
iOS 11+͍͜͠ͷͰҰ୴ཧ
• iOS 9+‣ ΈࠐΜͩΞϓϦͷը໘ΛըͰ͖Δ͕৴Ͱ͖ͳ͍• iOS 10+‣ Broadcast Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ• iOS 11+‣ Broadcast Upload ExtensionΛܦ༝͠ͳ͍ը໘ͷऩɾը‣ ίϯτϩʔϧηϯλʔܦ༝ͷ৴ReplayKitͱεΫϦʔϯ৴
iOS 11+͜ΕͰReplayKitͰͰ͖Δ͜ͱͷઆ໌ྃ
iOS 11+Broadcast Upload Extension
• ࠓճ͜ͷExtensionͷ• Լهͷ2ͭʹ֘‣ iOS10+- ଞͷΞϓϦܦ༝Ͱ৴Ͱ͖Δ‣ iOS11+- ίϯτϩʔϧηϯλʔܦ༝Ͱ৴Ͱ͖ΔBroadcast Upload Extension
࣮खॱ
iOS 11+SetupUI Extensionͬͯͳʹʁʁ
SetupUI Extensionͱ
• RPBroadcastActivityViewControllerܦ༝Ͱදࣔ͞ΕΔ• ͜ͷಋઢ͕ෆཁͳ߹Target͔Βআ͢ΕΑ͍SetupUI Extensionͱ
࣮खॱ• ίʔυ͚ͩͰUIΛ࡞Δ߹ "BroadcastSetupViewController" Λ͍͡ΕOK‣ ͋ͱόγόγίʔυΛॻ͍ͯͶ• StoryboardϕʔεͰUIΛ࡞Γ͍ͨ߹·ͣStoryboardΛ࡞Γ·͠ΐ͏
࣮खॱ• TARGETS͔ΒSetupUI ExtensionΛબ• GeneralͷDeployment Info͔ΒMain InterfaceΛมߋ
࣮खॱ• InfoλϒΛ։͍ͯ "NSExtension" ͷ "NSExtensionPrincipalClass" Λফ͢• "NSExtension" Ͱ "NSExtensionMainStoryboard" ͕͋Δ͜ͱΛ֬ೝ
࣮खॱ• ޙ "BroadcastSetupViewController.swift" ͷαϯϓϧίʔυΛݟͳ͕Β࣮• ৴༻ͷॲཧBroadcast Upload ExtensionଆͰ࣮• SetupUI ExtensionͰLIVE৴ʹඞཁͳใΛೖྗͯ͠Β͏‣ e.g. ৴ͷλΠτϧɺίϝϯτɺެ։ൣғ
iOS 11+Ұ୴ཧ
iOS 10+ͷ߹৴͍ͨ͠ΞϓϦ৴ΞϓϦBroadcast SetupUI ExtensionBroadcast Upload Extension
iOS 10+ͷ߹৴͍ͨ͠ΞϓϦ৴ΞϓϦBroadcast SetupUI ExtensionBroadcast Upload Extension৴͍ͨ͠
iOS 10+ͷ߹৴͍ͨ͠ΞϓϦ৴ΞϓϦBroadcast SetupUI ExtensionBroadcast Upload ExtensionSetupUI৴͍ͨ͠
iOS 10+ͷ߹৴͍ͨ͠ΞϓϦ৴ΞϓϦBroadcast SetupUI ExtensionBroadcast Upload ExtensionSetupUI৴͍ͨ͠CMSampleBuffer
iOS 10+ͷ߹৴͍ͨ͠ΞϓϦ৴ΞϓϦBroadcast SetupUI ExtensionBroadcast Upload ExtensionSetupUI৴͍ͨ͠CMSampleBufferੈք
iOS 11+ͷ߹৴͍ͨ͠ΞϓϦ৴ΞϓϦBroadcast SetupUI ExtensionBroadcast Upload Extensionίϯτϩʔϧηϯλʔ
iOS 11+ͷ߹৴͍ͨ͠ΞϓϦ৴ΞϓϦBroadcast SetupUI ExtensionBroadcast Upload Extensionίϯτϩʔϧηϯλʔ৴͍ͨ͠
iOS 11+ͷ߹৴͍ͨ͠ΞϓϦ৴ΞϓϦBroadcast SetupUI ExtensionBroadcast Upload ExtensionίϯτϩʔϧηϯλʔCMSampleBuffer৴͍ͨ͠
iOS 11+ͷ߹৴͍ͨ͠ΞϓϦ৴ΞϓϦBroadcast SetupUI ExtensionBroadcast Upload ExtensionίϯτϩʔϧηϯλʔCMSampleBuffer৴͍ͨ͠ ੈք
• ॳظঢ়ଶͰϑΝΠϧ2ͭ• "SampleHandler.swift" ʹ࣮Λॻ͍͍࣮ͯ͘खॱ
iOS 11+SampleHandler.swift ʹ࣮Λॻ͍͍ͯ͘
• ৴։࢝࣌ʹݺΕΔ• ϩάΠϯॲཧ• ৴༻ͷԼॲཧ
• ৴͕ϙʔζɾ࠶։͞ΕͨΒݺΕΔ
• broadcastFinishedɿ৴ऴྃ࣌ʹݺΕΔ• finishBroadcastWithErrorɿΤϥʔ͕ൃੜͯ͠ऴྃ͢Δͱ͖ʹݺΕΔ
• ͜͜ͰCMSampleBufferΛड͚औΔ• ͋ͱαʔόαΠυʹૹ৴͢ΕΑ͍
• αʔϏε͕࠾༻͍ͯ͠ΔϓϩτίϧʹΑΔ• RTMP(S)͔HLSͳΒHaishinKitΛར༻͢Δͷ͕͓͢͢Ί‣ https://github.com/shogo4405/HaishinKit.swift• ࠓճWebRTCͷ‣ ࣌Ӎಊ͞ΜͷSoraΛར༻‣ Sora iOS SDK͕͋ΔͷͰiOSͪ͜ΒΛར༻- https://github.com/shiguredo/shiguredo-webrtc-iosCMSampleBufferΛૹΔ
• Broadcast Upload ExtensionͰར༻Ͱ͖ΔϝϞϦ50MB·Ͱ• ͑Δͱແ༻ͰKill͞ΕΔ• Sora iOS SDK͕CMSampleBufferΛVP8Τϯίʔυ͢Δࡍʹ͑Δ‣ VP8CPUΤϯίʔυͳͷͰϝϞϦΛফඅ͢Δ
ղܾํ๏• CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢CMSampleBuffer
ղܾํ๏• CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢CMSampleBufferCVPixelBuffer
ղܾํ๏• CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢CMSampleBufferCVPixelBufferCIImage
ղܾํ๏• CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢CMSampleBufferCVPixelBufferCIImage CIImagetransformed(by:)
ղܾํ๏• CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢CMSampleBufferCVPixelBufferCIImageCMSampleBufferCVPixelBufferCIImagetransformed(by:)
ղܾํ๏• CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢CMSampleBufferCVPixelBufferCIImageCMSampleBufferCVPixelBufferCIImagetransformed(by:)SDK
ϦαΠζͨ͠ΒૹΕͨ
• FPSɾը࣭ɾϝϞϦফඅྔͷτϨʔυΦϑ͔Β࠷దղΛࢦ͢‣ αʔϏεͷੑ࣭ʹΑͬͯऔࣺબ͕มԽ͢Δ• ΤϯίʔυॲཧҎ֎ͰϝϞϦ͏ͷͰҙ੍ݶΛ͑ͳ͍αΠζΛ୳Δ
iOS 11+Ի
• ϚΠΫԻͱΞϓϦԻ• ͳʹ͠ͳ͍ͱϚΠΫԻऔಘͰ͖ͳ͍• ͜ͷίʔυΛ"broadcastStarted"ʹॻ͚औಘͰ͖ΔԻ
iOS 11+Sora iOS SDKlibwebrtcͷ্༷Extension্ͰԻ͕ͱΕͳ͍
• iOS11+ͷRPScreenRecorderΛར༻࣮ͨ͠ͳΒऔಘͰ͖Δ• Broadcast Upload ExtensionͩͱऔಘͰ͖ͳ͍• libwebrtcΛվ͢Δ͔……ʁ• ࣌Ӎಊ͞ΜղܾࡦΛ୳Γத• RTMP(S) or HLSͰHaishinKitΛར༻͢ΕऔಘՄೳSora iOS SDK with libwebrtc
To Be Continued
iOS 11+͔͜͜Βࡉ͔ͳϊϋूͰ͢
• ը໘ʹมԽ͕͋Δͱ͖͚ͩCMSampleBuffer͕ৼͬͯ͘Δ• ղܾࡦ2छྨ‣ ԻͷมԽʹ߹ΘͤͯલͷϑϨʔϜΛૹΔ- ແԻ৴ͩͱ٧Ή‣ αʔόαΠυʹHeartbeatΛఆظతʹૹΔ- ΫϥΠΞϯταΠυͰϑϨʔϜΛੜ͢Δํ๏- ΫϥΠΞϯταΠυͰAPIΛୟ͚ͩ͘ͰɺαʔόαΠυͰҎલͷϑϨʔϜΛͱʹੜ͢Δը໘ʹมԽ͕ແ͍ͱ͖
• ը໘ऩ͞Εͨ͘ͳ͍ίϯςϯπΛӅ͢‣ e.g. Netflix, Amazon Prime Video• ࠓɺը໘͕Ωϟϓνϟ͞ΕͯΔ͔ผ͢Δ• ը໘ऩͷ։࢝ɾऴྃΛݕ͢Δը໘ऩΛݕ͍ͨ͠ͱ͖
• print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍• BreakpointͰࢭ·Βͳ͍ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘
• print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍• BreakpointͰࢭ·Βͳ͍ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘͔͜͜ΒSchemeΛมߋ͠·͠ΐ͏
• ※·ͩਂ͘ௐ͍ࠪͯ͠ͳ͍ใ• ը໘ऩதʹGPUΛ͏ΤϯίʔυʢH.264ʣΛ͢Δͱୣ͍߹͏• ͏·͘ΤϯίʔυͰ͖ͣɺSora iOS SDKͰࠔ͍ͬͯΔϙΠϯτ• ৄ͍͠ํ͍·ͨ͠Β͝ڭ͍ࣔͩ͘͞ʂReplayKitͱH.264Τϯίʔυͷ૬ੑ͕ѱ͍
• εΫϦʔϯ৴ػೳΛ࡞Γ͍ͨͱ͖ReplayKitΛ͍·͠ΐ͏• iOS11+ͷαϙʔτʹ͢Δͷ͕͓͢͢ΊͰ͢• Broadcast Upload ExtensionΛ࣮͠·͠ΐ͏• ϝϞϦ͕50MB͔͑͠ͳ͍ͷͰҙ͠·͠ΐ͏·ͱΊ
fromatom
megabitsenmzq
hirosatogamo
oracle4engineer
mamohacy
bonjiri_niwa
koic
nenonaninu
bitkey
drumath2237
nikkei_engineer_recruiting
lhazy
oekazuma
lara
chriscoyier
bkeepers
geeforr
danielanewman
brad_frost
keathley
hatefulcrawdad
geoffreycrofte
bcantrill
aleyda