スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019

iOSDC 2019
Day 0 17:50~
Track A

Transcript

1. εΫϦʔϯ഑৴ػೳͷ ࣮૷͕େมͩͬͨͷͰ ஌ݟΛ͓఻͑͠·͢ iOSDC Japan 2019 2019/09/05 17:50~ Track A pixiv.inc FromAtom
2. • Swift 1.2 ͔ΒiOSΞϓϦ։ൃͷੈք΁ • ΞϓϦͷςοΫΛϦʔυ͍ͯ͠·͢ • pixiv Sketch / pixiv Sketch LIVE iOSΞϓϦ։ൃ
    @FromAtom
3. 
   
4. 
    ͜ͷൃදͷΰʔϧ
5. 
    ϥΠϒ഑৴ྲྀߦͬͯΔ͠ɺ ͏ͪ΋ϥΠϒ഑৴΍Δͧʂʂʂ
6. 
    ͱɺͳͬͨͱ͖ʹࠔΒͳ͍Α͏ʹ
7. 
    εΫϦʔϯ഑৴ػೳ
8. 
    εΫϦʔϯ഑৴ػೳͱ͸ εϚʔτϑΥϯͷը໘ΛϥΠϒ഑৴Ͱ͖Δػೳ
9. • εϚϗ୯ମͰϥΠϒ഑৴͕Մೳ • ήʔϜ࣮گ΍͓ֆ͔͖ͷ഑৴ʹར༻͞Ε͍ͯΔ • ࠷ۙ͸VTuberք۾Ͱ΋ར༻͞Ε͍ͯΔ
    εΫϦʔϯ഑৴ػೳͱ͸
10. 
     ReplayKitͱεΫϦʔϯ഑৴
11. • ReplayKitΛར༻͢ΔͱεΫϦʔϯͷऩ࿥΍LIVE഑৴͕؆୯ʹͰ͖Δ • ϚΠΫԻ੠΍ΞϓϦ͕ग़ྗ͍ͯ͠ΔԻ΋औಘͰ͖Δ • ͨͩ͠ɺiOSͷόʔδϣϯʹΑͬͯͰ͖Δ͜ͱ͕ҧ͏
     ReplayKitͱεΫϦʔϯ഑৴
12. • iOS 9+ ‣ ૊ΈࠐΜͩΞϓϦͷը໘Λ࿥ըͰ͖Δ ‣ ഑৴͸Ͱ͖ͳ͍ • iOS 10+ ‣ Broadcast Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ • iOS 11+ ‣ ༷ʑͳը໘͕഑৴ɾ࿥ըͰ͖Δ ‣ ίϯτϩʔϧηϯλʔ͔Βը໘࿥ըͰ͖Δͷͱಉ͡࢓૊Έ
     ReplayKitͱεΫϦʔϯ഑৴
13. 
     ࠓ͔Β࡞ΔͳΒ ࠷௿Ͱ΋iOS 11+
14. 
     ͡Ό͋ReplayKitʹ͍ͭͯௐ΂ͯΈΑ͏
17. 5͔݅͠هࣄ͕Ͷ͐ʂ
18. 
     ͳͷͰɺͦΕͧΕͷ࡞ΓํΛղઆ
19. • ࣮ػͰ͔͠σόοάͰ͖ͳ͍ ‣ Χϝϥ΍Push௨஌ͱಉ͡ • iOSඪ४ͷը໘ऩ࿥΍QuickTimeܦ༝Ͱͷը໘࿥ըͱڝ߹͢Δ ‣ σϞͷࡱӨΛ͢Δ௚લʹؾ͍ͮͯযΒͳ͍Α͏ʹ͠·͠ΐ͏ ‣ Pixel3ͰiPhoneXΛࡱӨ͢Δ೔ʑΛա͝͠·ͨ͠
     ͪΐͬͱͨ͠஫ҙ఺
20. 
     ΞϓϦͷը໘Λ࿥ըͰ͖Δ iOS 9+
21. 
    
22. ࿥ը։࢝
23. 
     ࿥ըऴྃ
24. 
     iOS 10+ ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ
25. • Broadcast Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ • ഑৴Λ͍ͨ͠ΞϓϦͱ഑৴͢ΔΞϓϦͷ૒ํͰ࣮૷͕ඞཁ
     iOS10Ͱͷը໘഑৴
26. 
    
27. 
     ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷
28. 
     ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷
29. 
     ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷
30. 
     ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷
31. 
    
32. 
     ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷ -഑৴ऴྃ-
33. 
     ഑৴͢ΔΞϓϦͰͷ࣮૷ • Broadcast Upload ExtensionΛ࣮૷͢Δ • ৄ͘͠͸͜ͷޙ iOS11+ ͷ࿩ͱҰॹʹ͠·͢
34. 
     iOS 11+ Ͳͷը໘΋഑৴ɾ࿥ըͰ͖Δ
35. • iOS11+ ͔ΒͰ͖ΔΑ͏ʹͳͬͨࣄ ‣ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴
     iOS11+Ͱͷը໘഑৴
36. • ࣮૷ͨ͠ΞϓϦͰը໘ͷΩϟϓνϟ͕Ͱ͖Δ • ը໘ͷCMSampleBufferΛऔಘͯ͠࿥ըɾ഑৴͕Մೳʹ • ϚΠΫԻ੠΍ΞϓϦԻ੠΋औಘՄೳ
     RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ
37. 
     ࣮૷
38. • ίϯτϩʔϧηϯλʔͷ˕͔Βબ୒Ͱ͖Δ • iOS9+ͱ͸ҧ͍ը໘શମ͕ऩ࿥Ͱ͖Δ ‣ ௨஌΋ऩ࿥͞ΕΔͷͰ஫ҙ
     ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴
39. 
    
40. 
     iOS 11+ ΍΍͍͜͠ͷͰҰ୴੔ཧ
41. • iOS 9+ ‣ ૊ΈࠐΜͩΞϓϦͷը໘Λ࿥ըͰ͖Δ͕഑৴͸Ͱ͖ͳ͍ • iOS 10+ ‣ Broadcast Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ • iOS 11+ ‣ Broadcast Upload ExtensionΛܦ༝͠ͳ͍ը໘ͷऩ࿥ɾ࿥ը ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴
     ReplayKitͱεΫϦʔϯ഑৴
42. 
     iOS 11+ ͜ΕͰReplayKitͰ Ͱ͖Δ͜ͱͷઆ໌׬ྃ
43. 
     iOS 11+ Broadcast Upload Extension
44. • ࠓճ͸͜ͷExtensionͷ࿩ • Լهͷ2ͭʹ֘౰ ‣ iOS10+ - ଞͷΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ ‣ iOS11+ - ίϯτϩʔϧηϯλʔܦ༝Ͱ഑৴Ͱ͖Δ
     Broadcast Upload Extension
45. 
     ࣮૷खॱ
46. 
     ࣮૷खॱ
47. 
     ࣮૷खॱ
48. 
     ࣮૷खॱ
49. 
     iOS 11+ SetupUI Extensionͬͯͳʹʁʁ
50. 
     SetupUI Extensionͱ͸
51. 
     SetupUI Extensionͱ͸
52. • RPBroadcastActivityViewControllerܦ༝Ͱදࣔ͞ΕΔ • ͜ͷಋઢ͕ෆཁͳ৔߹͸Target͔Β࡟আ͢Ε͹Α͍
     SetupUI Extensionͱ͸
53. 
     ࣮૷खॱ • ίʔυ͚ͩͰUIΛ࡞Δ৔߹͸ "BroadcastSetupViewController" Λ͍͡Ε͹OK ‣ ͋ͱ͸όγόγίʔυΛॻ͍ͯͶ • StoryboardϕʔεͰUIΛ࡞Γ͍ͨ৔߹͸·ͣStoryboardΛ࡞Γ·͠ΐ͏
54. 
     ࣮૷खॱ • ίʔυ͚ͩͰUIΛ࡞Δ৔߹͸ "BroadcastSetupViewController" Λ͍͡Ε͹OK ‣ ͋ͱ͸όγόγίʔυΛॻ͍ͯͶ • StoryboardϕʔεͰUIΛ࡞Γ͍ͨ৔߹͸·ͣStoryboardΛ࡞Γ·͠ΐ͏
55. 
     ࣮૷खॱ • TARGETS͔ΒSetupUI ExtensionΛબ୒ • General಺ͷDeployment Info͔ΒMain InterfaceΛมߋ
56. 
     ࣮૷खॱ • InfoλϒΛ։͍ͯ "NSExtension" ಺ͷ "NSExtensionPrincipalClass" Λফ͢ • "NSExtension" ಺Ͱ "NSExtensionMainStoryboard" ͕͋Δ͜ͱΛ֬ೝ
57. 
     ࣮૷खॱ • ޙ͸ "BroadcastSetupViewController.swift" ͷαϯϓϧίʔυΛݟͳ͕Β࣮૷ • ഑৴༻ͷॲཧ͸Broadcast Upload ExtensionଆͰ࣮૷ • SetupUI ExtensionͰ͸LIVE഑৴ʹඞཁͳ৘ใΛೖྗͯ͠΋Β͏ ‣ e.g. ഑৴ͷλΠτϧɺίϝϯτɺެ։ൣғ
58. 
     iOS 11+ Ұ୴੔ཧ
59. 
     iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension
60. 
     iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension ഑৴͍ͨ͠
61. 
     iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension SetupUI ഑৴͍ͨ͠
62. 
     iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension SetupUI ഑৴͍ͨ͠ CMSampleBuffer
63. 
     iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension SetupUI ഑৴͍ͨ͠ CMSampleBuffer ੈք
64. 
     iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension ίϯτϩʔϧηϯλʔ
65. 
     iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension ίϯτϩʔϧηϯλʔ ഑৴͍ͨ͠
66. 
     iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ഑৴͍ͨ͠
67. 
     iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ഑৴͍ͨ͠ ੈք
68. 
     iOS 11+ Broadcast Upload Extension
69. • ॳظঢ়ଶͰ͸ϑΝΠϧ͸2ͭ • "SampleHandler.swift" ʹ࣮૷Λॻ͍͍ͯ͘
     ࣮૷खॱ
70. 
     iOS 11+ SampleHandler.swift ʹ ࣮૷Λॻ͍͍ͯ͘
72. • ഑৴։࢝࣌ʹݺ͹ΕΔ • ϩάΠϯॲཧ • ഑৴༻ͷԼॲཧ
73. • ഑৴͕ϙʔζɾ࠶։͞ΕͨΒݺ͹ΕΔ
74. • broadcastFinishedɿ഑৴ऴྃ࣌ʹݺ͹ΕΔ • finishBroadcastWithErrorɿΤϥʔ͕ൃੜͯ͠ऴྃ͢Δͱ͖ʹݺ͹ΕΔ
75. • ͜͜ͰCMSampleBufferΛड͚औΔ • ͋ͱ͸αʔόαΠυʹૹ৴͢Ε͹Α͍
77. • αʔϏε͕࠾༻͍ͯ͠ΔϓϩτίϧʹΑΔ • RTMP(S)͔HLSͳΒHaishinKitΛར༻͢Δͷ͕͓͢͢Ί ‣ https://github.com/shogo4405/HaishinKit.swift • ࠓճ͸WebRTCͷ࿩ ‣ ࣌Ӎಊ͞ΜͷSoraΛར༻ ‣ Sora iOS SDK͕͋ΔͷͰiOS͸ͪ͜ΒΛར༻ - https://github.com/shiguredo/shiguredo-webrtc-ios
     CMSampleBufferΛૹΔ
78. 
     ࣮૷खॱ
79. 
     ໰୊ • Broadcast Upload ExtensionͰར༻Ͱ͖ΔϝϞϦ͸50MB·Ͱ • ௒͑Δͱ໰౴ແ༻ͰKill͞ΕΔ • Sora iOS SDK͕CMSampleBufferΛVP8Τϯίʔυ͢Δࡍʹ௒͑Δ ‣ VP8͸CPUΤϯίʔυͳͷͰϝϞϦΛফඅ͢Δ
80. 
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer
81. 
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer
82. 
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage
83. 
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CIImage transformed(by:)
84. 
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer CIImage transformed(by:)
85. 
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer CIImage transformed(by:) SDK΁
86. 
     ϦαΠζͨ͠ΒૹΕͨ
87. • FPSɾը࣭ɾϝϞϦফඅྔͷτϨʔυΦϑ͔Β࠷దղΛ໨ࢦ͢ ‣ αʔϏεͷੑ࣭ʹΑͬͯऔࣺબ୒͕มԽ͢Δ • ΤϯίʔυॲཧҎ֎Ͱ΋ϝϞϦ͸࢖͏ͷͰ஫ҙ
     ੍ݶΛ௒͑ͳ͍αΠζΛ୳Δ
88. 
     iOS 11+ Ի੠
89. • ϚΠΫԻ੠ͱΞϓϦԻ੠ • ͳʹ΋͠ͳ͍ͱϚΠΫԻ੠͸औಘͰ͖ͳ͍ • ͜ͷίʔυΛ"broadcastStarted"ʹॻ͚͹औಘͰ͖Δ
     Ի੠
90. 
     iOS 11+ Sora iOS SDK͸libwebrtcͷ࢓্༷ Extension্ͰԻ੠͕ͱΕͳ͍
91. • iOS11+ͷRPScreenRecorderΛར༻࣮ͨ͠૷ͳΒऔಘͰ͖Δ • Broadcast Upload ExtensionͩͱऔಘͰ͖ͳ͍ • libwebrtcΛվ଄͢Δ͔……ʁ • ࣌Ӎಊ͞ΜղܾࡦΛ୳Γத • RTMP(S) or HLSͰHaishinKitΛར༻͢Ε͹औಘՄೳ
     Sora iOS SDK with libwebrtc
92. To Be Continued
93. 
     iOS 11+ ͔͜͜Β͸ࡉ͔ͳϊ΢ϋ΢ूͰ͢
94. • ը໘ʹมԽ͕͋Δͱ͖͚ͩCMSampleBuffer͕ৼͬͯ͘Δ • ղܾࡦ͸2छྨ ‣ Ի੠ͷมԽʹ߹ΘͤͯલͷϑϨʔϜΛૹΔ - ແԻ഑৴ͩͱ٧Ή ‣ αʔόαΠυʹHeartbeatΛఆظతʹૹΔ - ΫϥΠΞϯταΠυͰϑϨʔϜΛੜ੒͢Δํ๏ - ΫϥΠΞϯταΠυͰ͸APIΛୟ͚ͩ͘ͰɺαʔόαΠυͰҎલͷϑϨʔϜΛ΋ͱʹੜ੒͢Δ
     ը໘ʹมԽ͕ແ͍ͱ͖
95. • ը໘ऩ࿥͞Εͨ͘ͳ͍ίϯςϯπΛӅ͢ ‣ e.g. Netflix, Amazon Prime Video • ࠓɺը໘͕Ωϟϓνϟ͞ΕͯΔ͔൑ผ͢Δ • ը໘ऩ࿥ͷ։࢝ɾऴྃΛݕ஌͢Δ
     ը໘ऩ࿥Λݕ஌͍ͨ͠ͱ͖
96. • print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍
     ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘
97. • print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍
     ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘ ͔͜͜ΒSchemeΛมߋ͠·͠ΐ͏
98. • ※·ͩਂ͘ௐ͍ࠪͯ͠ͳ͍৘ใ • ը໘ऩ࿥தʹGPUΛ࢖͏ΤϯίʔυʢH.264ʣΛ͢Δͱୣ͍߹͏ • ͏·͘ΤϯίʔυͰ͖ͣɺSora iOS SDKͰ΋ࠔ͍ͬͯΔϙΠϯτ • ৄ͍͠ํ͍·ͨ͠Β͝ڭ͍ࣔͩ͘͞ʂ
     ReplayKitͱH.264Τϯίʔυͷ૬ੑ͕ѱ͍
99. • εΫϦʔϯ഑৴ػೳΛ࡞Γ͍ͨͱ͖͸ReplayKitΛ࢖͍·͠ΐ͏ • iOS11+ͷαϙʔτʹ͢Δͷ͕͓͢͢ΊͰ͢ • Broadcast Upload ExtensionΛ࣮૷͠·͠ΐ͏ • ϝϞϦ͕50MB͔͠࢖͑ͳ͍ͷͰ஫ҙ͠·͠ΐ͏
     ·ͱΊ