Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019
FromAtom
September 05, 2019
Technology
5
8.1k
スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019
iOSDC 2019
Day 0 17:50~
Track A
FromAtom
September 05, 2019
Tweet
Share
More Decks by FromAtom
See All by FromAtom
fromatom
9
2.3k
fromatom
1
3.1k
fromatom
8
4.8k
fromatom
0
3k
fromatom
4
4.8k
fromatom
0
200
fromatom
1
810
Other Decks in Technology
See All in Technology
finengine
0
130
lmi
1
970
ykanazawa
0
310
hanasuke
0
220
jozono
6
1.1k
fabiangosebrink
0
210
daisukehirama41
1
290
nnstt1
2
110
kanaugust
PRO
0
190
line_developers
PRO
2
370
surumegohan
1
130
gracia
0
820
Featured
See All Featured
danielanewman
2
550
malarkey
119
16k
mza
80
4.2k
sferik
612
55k
brettharned
93
3.1k
cherdarchuk
71
260k
maltzj
502
36k
marktimemedia
7
440
lara
16
2.8k
jakevdp
776
200k
garrettdimon
289
110k
pedronauck
652
110k
Transcript
εΫϦʔϯ৴ػೳͷ ࣮͕େมͩͬͨͷͰ ݟΛ͓͑͠·͢ iOSDC Japan 2019 2019/09/05 17:50~ Track A
pixiv.inc FromAtom
• Swift 1.2 ͔ΒiOSΞϓϦ։ൃͷੈք • ΞϓϦͷςοΫΛϦʔυ͍ͯ͠·͢ • pixiv Sketch /
pixiv Sketch LIVE iOSΞϓϦ։ൃ @FromAtom
͜ͷൃදͷΰʔϧ
ϥΠϒ৴ྲྀߦͬͯΔ͠ɺ ͏ͪϥΠϒ৴Δͧʂʂʂ
ͱɺͳͬͨͱ͖ʹࠔΒͳ͍Α͏ʹ
εΫϦʔϯ৴ػೳ
εΫϦʔϯ৴ػೳͱ εϚʔτϑΥϯͷը໘ΛϥΠϒ৴Ͱ͖Δػೳ
• εϚϗ୯ମͰϥΠϒ৴͕Մೳ • ήʔϜ࣮گ͓ֆ͔͖ͷ৴ʹར༻͞Ε͍ͯΔ • ࠷ۙVTuberք۾Ͱར༻͞Ε͍ͯΔ εΫϦʔϯ৴ػೳͱ
ReplayKitͱεΫϦʔϯ৴
• ReplayKitΛར༻͢ΔͱεΫϦʔϯͷऩLIVE৴͕؆୯ʹͰ͖Δ • ϚΠΫԻΞϓϦ͕ग़ྗ͍ͯ͠ΔԻऔಘͰ͖Δ • ͨͩ͠ɺiOSͷόʔδϣϯʹΑͬͯͰ͖Δ͜ͱ͕ҧ͏ ReplayKitͱεΫϦʔϯ৴
• iOS 9+ ‣ ΈࠐΜͩΞϓϦͷը໘ΛըͰ͖Δ ‣ ৴Ͱ͖ͳ͍ • iOS 10+
‣ Broadcast Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ • iOS 11+ ‣ ༷ʑͳը໘͕৴ɾըͰ͖Δ ‣ ίϯτϩʔϧηϯλʔ͔Βը໘ըͰ͖Δͷͱಉ͡Έ ReplayKitͱεΫϦʔϯ৴
ࠓ͔Β࡞ΔͳΒ ࠷ͰiOS 11+
͡Ό͋ReplayKitʹ͍ͭͯௐͯΈΑ͏
None
None
5͔݅͠هࣄ͕Ͷ͐ʂ
ͳͷͰɺͦΕͧΕͷ࡞ΓํΛղઆ
• ࣮ػͰ͔͠σόοάͰ͖ͳ͍ ‣ ΧϝϥPush௨ͱಉ͡ • iOSඪ४ͷը໘ऩQuickTimeܦ༝Ͱͷը໘ըͱڝ߹͢Δ ‣ σϞͷࡱӨΛ͢Δલʹؾ͍ͮͯযΒͳ͍Α͏ʹ͠·͠ΐ͏ ‣ Pixel3ͰiPhoneXΛࡱӨ͢ΔʑΛա͝͠·ͨ͠
ͪΐͬͱͨ͠ҙ
ΞϓϦͷը໘ΛըͰ͖Δ iOS 9+
ը։࢝
ըऴྃ
iOS 10+ ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ
• Broadcast Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ • ৴Λ͍ͨ͠ΞϓϦͱ৴͢ΔΞϓϦͷํͰ࣮͕ඞཁ iOS10Ͱͷը໘৴
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮ -৴ऴྃ-
৴͢ΔΞϓϦͰͷ࣮ • Broadcast Upload ExtensionΛ࣮͢Δ • ৄ͘͜͠ͷޙ iOS11+
ͷͱҰॹʹ͠·͢
iOS 11+ Ͳͷը໘৴ɾըͰ͖Δ
• iOS11+ ͔ΒͰ͖ΔΑ͏ʹͳͬͨࣄ ‣ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ৴ iOS11+Ͱͷը໘৴
• ࣮ͨ͠ΞϓϦͰը໘ͷΩϟϓνϟ͕Ͱ͖Δ • ը໘ͷCMSampleBufferΛऔಘͯ͠ըɾ৴͕Մೳʹ • ϚΠΫԻΞϓϦԻऔಘՄೳ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ
࣮
• ίϯτϩʔϧηϯλʔͷ˕͔ΒબͰ͖Δ • iOS9+ͱҧ͍ը໘શମ͕ऩͰ͖Δ ‣ ௨ऩ͞ΕΔͷͰҙ ίϯτϩʔϧηϯλʔܦ༝ͷ৴
iOS 11+ ͍͜͠ͷͰҰ୴ཧ
• iOS 9+ ‣ ΈࠐΜͩΞϓϦͷը໘ΛըͰ͖Δ͕৴Ͱ͖ͳ͍ • iOS 10+ ‣ Broadcast
Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ • iOS 11+ ‣ Broadcast Upload ExtensionΛܦ༝͠ͳ͍ը໘ͷऩɾը ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ৴ ReplayKitͱεΫϦʔϯ৴
iOS 11+ ͜ΕͰReplayKitͰ Ͱ͖Δ͜ͱͷઆ໌ྃ
iOS 11+ Broadcast Upload Extension
• ࠓճ͜ͷExtensionͷ • Լهͷ2ͭʹ֘ ‣ iOS10+ - ଞͷΞϓϦܦ༝Ͱ৴Ͱ͖Δ ‣ iOS11+
- ίϯτϩʔϧηϯλʔܦ༝Ͱ৴Ͱ͖Δ Broadcast Upload Extension
࣮खॱ
࣮खॱ
࣮खॱ
࣮खॱ
iOS 11+ SetupUI Extensionͬͯͳʹʁʁ
SetupUI Extensionͱ
SetupUI Extensionͱ
• RPBroadcastActivityViewControllerܦ༝Ͱදࣔ͞ΕΔ • ͜ͷಋઢ͕ෆཁͳ߹Target͔Βআ͢ΕΑ͍ SetupUI Extensionͱ
࣮खॱ • ίʔυ͚ͩͰUIΛ࡞Δ߹ "BroadcastSetupViewController" Λ͍͡ΕOK ‣ ͋ͱόγόγίʔυΛॻ͍ͯͶ •
StoryboardϕʔεͰUIΛ࡞Γ͍ͨ߹·ͣStoryboardΛ࡞Γ·͠ΐ͏
࣮खॱ • ίʔυ͚ͩͰUIΛ࡞Δ߹ "BroadcastSetupViewController" Λ͍͡ΕOK ‣ ͋ͱόγόγίʔυΛॻ͍ͯͶ •
StoryboardϕʔεͰUIΛ࡞Γ͍ͨ߹·ͣStoryboardΛ࡞Γ·͠ΐ͏
࣮खॱ • TARGETS͔ΒSetupUI ExtensionΛબ • GeneralͷDeployment Info͔ΒMain InterfaceΛมߋ
࣮खॱ • InfoλϒΛ։͍ͯ "NSExtension" ͷ "NSExtensionPrincipalClass" Λফ͢ •
"NSExtension" Ͱ "NSExtensionMainStoryboard" ͕͋Δ͜ͱΛ֬ೝ
࣮खॱ • ޙ "BroadcastSetupViewController.swift" ͷαϯϓϧίʔυΛݟͳ͕Β࣮ • ৴༻ͷॲཧBroadcast Upload
ExtensionଆͰ࣮ • SetupUI ExtensionͰLIVE৴ʹඞཁͳใΛೖྗͯ͠Β͏ ‣ e.g. ৴ͷλΠτϧɺίϝϯτɺެ։ൣғ
iOS 11+ Ұ୴ཧ
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ৴͍ͨ͠
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension SetupUI ৴͍ͨ͠
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension SetupUI ৴͍ͨ͠ CMSampleBuffer
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension SetupUI ৴͍ͨ͠ CMSampleBuffer ੈք
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ ৴͍ͨ͠
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ৴͍ͨ͠
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ৴͍ͨ͠ ੈք
iOS 11+ Broadcast Upload Extension
• ॳظঢ়ଶͰϑΝΠϧ2ͭ • "SampleHandler.swift" ʹ࣮Λॻ͍͍ͯ͘ ࣮खॱ
iOS 11+ SampleHandler.swift ʹ ࣮Λॻ͍͍ͯ͘
None
• ৴։࢝࣌ʹݺΕΔ • ϩάΠϯॲཧ • ৴༻ͷԼॲཧ
• ৴͕ϙʔζɾ࠶։͞ΕͨΒݺΕΔ
• broadcastFinishedɿ৴ऴྃ࣌ʹݺΕΔ • finishBroadcastWithErrorɿΤϥʔ͕ൃੜͯ͠ऴྃ͢Δͱ͖ʹݺΕΔ
• ͜͜ͰCMSampleBufferΛड͚औΔ • ͋ͱαʔόαΠυʹૹ৴͢ΕΑ͍
None
• αʔϏε͕࠾༻͍ͯ͠ΔϓϩτίϧʹΑΔ • RTMP(S)͔HLSͳΒHaishinKitΛར༻͢Δͷ͕͓͢͢Ί ‣ https://github.com/shogo4405/HaishinKit.swift • ࠓճWebRTCͷ ‣ ࣌Ӎಊ͞ΜͷSoraΛར༻
‣ Sora iOS SDK͕͋ΔͷͰiOSͪ͜ΒΛར༻ - https://github.com/shiguredo/shiguredo-webrtc-ios CMSampleBufferΛૹΔ
࣮खॱ
• Broadcast Upload ExtensionͰར༻Ͱ͖ΔϝϞϦ50MB·Ͱ • ͑Δͱແ༻ͰKill͞ΕΔ •
Sora iOS SDK͕CMSampleBufferΛVP8Τϯίʔυ͢Δࡍʹ͑Δ ‣ VP8CPUΤϯίʔυͳͷͰϝϞϦΛফඅ͢Δ
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CIImage transformed(by:)
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer
CIImage transformed(by:)
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer
CIImage transformed(by:) SDK
ϦαΠζͨ͠ΒૹΕͨ
• FPSɾը࣭ɾϝϞϦফඅྔͷτϨʔυΦϑ͔Β࠷దղΛࢦ͢ ‣ αʔϏεͷੑ࣭ʹΑͬͯऔࣺબ͕มԽ͢Δ • ΤϯίʔυॲཧҎ֎ͰϝϞϦ͏ͷͰҙ ੍ݶΛ͑ͳ͍αΠζΛ୳Δ
iOS 11+ Ի
• ϚΠΫԻͱΞϓϦԻ • ͳʹ͠ͳ͍ͱϚΠΫԻऔಘͰ͖ͳ͍ • ͜ͷίʔυΛ"broadcastStarted"ʹॻ͚औಘͰ͖Δ Ի
iOS 11+ Sora iOS SDKlibwebrtcͷ্༷ Extension্ͰԻ͕ͱΕͳ͍
• iOS11+ͷRPScreenRecorderΛར༻࣮ͨ͠ͳΒऔಘͰ͖Δ • Broadcast Upload ExtensionͩͱऔಘͰ͖ͳ͍ • libwebrtcΛվ͢Δ͔……ʁ • ࣌Ӎಊ͞ΜղܾࡦΛ୳Γத
• RTMP(S) or HLSͰHaishinKitΛར༻͢ΕऔಘՄೳ Sora iOS SDK with libwebrtc
To Be Continued
iOS 11+ ͔͜͜Βࡉ͔ͳϊϋूͰ͢
• ը໘ʹมԽ͕͋Δͱ͖͚ͩCMSampleBuffer͕ৼͬͯ͘Δ • ղܾࡦ2छྨ ‣ ԻͷมԽʹ߹ΘͤͯલͷϑϨʔϜΛૹΔ - ແԻ৴ͩͱ٧Ή ‣ αʔόαΠυʹHeartbeatΛఆظతʹૹΔ
- ΫϥΠΞϯταΠυͰϑϨʔϜΛੜ͢Δํ๏ - ΫϥΠΞϯταΠυͰAPIΛୟ͚ͩ͘ͰɺαʔόαΠυͰҎલͷϑϨʔϜΛͱʹੜ͢Δ ը໘ʹมԽ͕ແ͍ͱ͖
• ը໘ऩ͞Εͨ͘ͳ͍ίϯςϯπΛӅ͢ ‣ e.g. Netflix, Amazon Prime Video • ࠓɺը໘͕Ωϟϓνϟ͞ΕͯΔ͔ผ͢Δ
• ը໘ऩͷ։࢝ɾऴྃΛݕ͢Δ ը໘ऩΛݕ͍ͨ͠ͱ͖
• print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍ ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘
• print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍ ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘ ͔͜͜ΒSchemeΛมߋ͠·͠ΐ͏
• ※·ͩਂ͘ௐ͍ࠪͯ͠ͳ͍ใ • ը໘ऩதʹGPUΛ͏ΤϯίʔυʢH.264ʣΛ͢Δͱୣ͍߹͏ • ͏·͘ΤϯίʔυͰ͖ͣɺSora iOS SDKͰࠔ͍ͬͯΔϙΠϯτ • ৄ͍͠ํ͍·ͨ͠Β͝ڭ͍ࣔͩ͘͞ʂ
ReplayKitͱH.264Τϯίʔυͷ૬ੑ͕ѱ͍
• εΫϦʔϯ৴ػೳΛ࡞Γ͍ͨͱ͖ReplayKitΛ͍·͠ΐ͏ • iOS11+ͷαϙʔτʹ͢Δͷ͕͓͢͢ΊͰ͢ • Broadcast Upload ExtensionΛ࣮͠·͠ΐ͏ • ϝϞϦ͕50MB͔͑͠ͳ͍ͷͰҙ͠·͠ΐ͏
·ͱΊ