スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019

Transcript

1. 1.

   εΫϦʔϯ഑৴ػೳͷ ࣮૷͕େมͩͬͨͷͰ ஌ݟΛ͓఻͑͠·͢ iOSDC Japan 2019 2019/09/05 17:50~ Track A

   pixiv.inc FromAtom
2. 2.

   • Swift 1.2 ͔ΒiOSΞϓϦ։ൃͷੈք΁ • ΞϓϦͷςοΫΛϦʔυ͍ͯ͠·͢ • pixiv Sketch /

   pixiv Sketch LIVE iOSΞϓϦ։ൃ
    @FromAtom
3. 3.

   
   

4. 4.

   
    ͜ͷൃදͷΰʔϧ

5. 5.

   
    ϥΠϒ഑৴ྲྀߦͬͯΔ͠ɺ ͏ͪ΋ϥΠϒ഑৴΍Δͧʂʂʂ

6. 6.

   
    ͱɺͳͬͨͱ͖ʹࠔΒͳ͍Α͏ʹ

7. 7.

   
    εΫϦʔϯ഑৴ػೳ

8. 8.

   
    εΫϦʔϯ഑৴ػೳͱ͸ εϚʔτϑΥϯͷը໘ΛϥΠϒ഑৴Ͱ͖Δػೳ

9. 9.

   • εϚϗ୯ମͰϥΠϒ഑৴͕Մೳ • ήʔϜ࣮گ΍͓ֆ͔͖ͷ഑৴ʹར༻͞Ε͍ͯΔ • ࠷ۙ͸VTuberք۾Ͱ΋ར༻͞Ε͍ͯΔ
    εΫϦʔϯ഑৴ػೳͱ͸

10. 10.

    
     ReplayKitͱεΫϦʔϯ഑৴

11. 11.

    • ReplayKitΛར༻͢ΔͱεΫϦʔϯͷऩ࿥΍LIVE഑৴͕؆୯ʹͰ͖Δ • ϚΠΫԻ੠΍ΞϓϦ͕ग़ྗ͍ͯ͠ΔԻ΋औಘͰ͖Δ • ͨͩ͠ɺiOSͷόʔδϣϯʹΑͬͯͰ͖Δ͜ͱ͕ҧ͏
     ReplayKitͱεΫϦʔϯ഑৴

12. 12.

    • iOS 9+ ‣ ૊ΈࠐΜͩΞϓϦͷը໘Λ࿥ըͰ͖Δ ‣ ഑৴͸Ͱ͖ͳ͍ • iOS 10+

    ‣ Broadcast Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ • iOS 11+ ‣ ༷ʑͳը໘͕഑৴ɾ࿥ըͰ͖Δ ‣ ίϯτϩʔϧηϯλʔ͔Βը໘࿥ըͰ͖Δͷͱಉ͡࢓૊Έ
     ReplayKitͱεΫϦʔϯ഑৴
13. 13.

    
     ࠓ͔Β࡞ΔͳΒ ࠷௿Ͱ΋iOS 11+

14. 14.

    
     ͡Ό͋ReplayKitʹ͍ͭͯௐ΂ͯΈΑ͏

15. 15.

    None
16. 16.

    None
17. 17.

    5͔݅͠هࣄ͕Ͷ͐ʂ

18. 18.

    
     ͳͷͰɺͦΕͧΕͷ࡞ΓํΛղઆ

19. 19.

    • ࣮ػͰ͔͠σόοάͰ͖ͳ͍ ‣ Χϝϥ΍Push௨஌ͱಉ͡ • iOSඪ४ͷը໘ऩ࿥΍QuickTimeܦ༝Ͱͷը໘࿥ըͱڝ߹͢Δ ‣ σϞͷࡱӨΛ͢Δ௚લʹؾ͍ͮͯযΒͳ͍Α͏ʹ͠·͠ΐ͏ ‣ Pixel3ͰiPhoneXΛࡱӨ͢Δ೔ʑΛա͝͠·ͨ͠

    
     ͪΐͬͱͨ͠஫ҙ఺
20. 20.

    
     ΞϓϦͷը໘Λ࿥ըͰ͖Δ iOS 9+

21. 21.

    
    

22. 22.

    ࿥ը։࢝

23. 23.

    
     ࿥ըऴྃ

24. 24.

    
     iOS 10+ ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ

25. 25.

    • Broadcast Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ • ഑৴Λ͍ͨ͠ΞϓϦͱ഑৴͢ΔΞϓϦͷ૒ํͰ࣮૷͕ඞཁ
     iOS10Ͱͷը໘഑৴

26. 26.

    
    

27. 27.

    
     ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

28. 28.

    
     ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

29. 29.

    
     ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

30. 30.

    
     ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

31. 31.

    
    

32. 32.

    
     ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷ -഑৴ऴྃ-

33. 33.

    
     ഑৴͢ΔΞϓϦͰͷ࣮૷ • Broadcast Upload ExtensionΛ࣮૷͢Δ • ৄ͘͠͸͜ͷޙ iOS11+

    ͷ࿩ͱҰॹʹ͠·͢
34. 34.

    
     iOS 11+ Ͳͷը໘΋഑৴ɾ࿥ըͰ͖Δ

35. 35.

    • iOS11+ ͔ΒͰ͖ΔΑ͏ʹͳͬͨࣄ ‣ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴
     iOS11+Ͱͷը໘഑৴

36. 36.

    • ࣮૷ͨ͠ΞϓϦͰը໘ͷΩϟϓνϟ͕Ͱ͖Δ • ը໘ͷCMSampleBufferΛऔಘͯ͠࿥ըɾ഑৴͕Մೳʹ • ϚΠΫԻ੠΍ΞϓϦԻ੠΋औಘՄೳ
     RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ

37. 37.

    
     ࣮૷

38. 38.

    • ίϯτϩʔϧηϯλʔͷ˕͔Βબ୒Ͱ͖Δ • iOS9+ͱ͸ҧ͍ը໘શମ͕ऩ࿥Ͱ͖Δ ‣ ௨஌΋ऩ࿥͞ΕΔͷͰ஫ҙ
     ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴

39. 39.

    
    

40. 40.

    
     iOS 11+ ΍΍͍͜͠ͷͰҰ୴੔ཧ

41. 41.

    • iOS 9+ ‣ ૊ΈࠐΜͩΞϓϦͷը໘Λ࿥ըͰ͖Δ͕഑৴͸Ͱ͖ͳ͍ • iOS 10+ ‣ Broadcast

    Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ • iOS 11+ ‣ Broadcast Upload ExtensionΛܦ༝͠ͳ͍ը໘ͷऩ࿥ɾ࿥ը ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴
     ReplayKitͱεΫϦʔϯ഑৴
42. 42.

    
     iOS 11+ ͜ΕͰReplayKitͰ Ͱ͖Δ͜ͱͷઆ໌׬ྃ

43. 43.

    
     iOS 11+ Broadcast Upload Extension

44. 44.

    • ࠓճ͸͜ͷExtensionͷ࿩ • Լهͷ2ͭʹ֘౰ ‣ iOS10+ - ଞͷΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ ‣ iOS11+

    - ίϯτϩʔϧηϯλʔܦ༝Ͱ഑৴Ͱ͖Δ
     Broadcast Upload Extension
45. 45.

    
     ࣮૷खॱ

46. 46.

    
     ࣮૷खॱ

47. 47.

    
     ࣮૷खॱ

48. 48.

    
     ࣮૷खॱ

49. 49.

    
     iOS 11+ SetupUI Extensionͬͯͳʹʁʁ

50. 50.

    
     SetupUI Extensionͱ͸

51. 51.

    
     SetupUI Extensionͱ͸

52. 52.

    • RPBroadcastActivityViewControllerܦ༝Ͱදࣔ͞ΕΔ • ͜ͷಋઢ͕ෆཁͳ৔߹͸Target͔Β࡟আ͢Ε͹Α͍
     SetupUI Extensionͱ͸

53. 53.

    
     ࣮૷खॱ • ίʔυ͚ͩͰUIΛ࡞Δ৔߹͸ "BroadcastSetupViewController" Λ͍͡Ε͹OK ‣ ͋ͱ͸όγόγίʔυΛॻ͍ͯͶ •

    StoryboardϕʔεͰUIΛ࡞Γ͍ͨ৔߹͸·ͣStoryboardΛ࡞Γ·͠ΐ͏
54. 54.

    
     ࣮૷खॱ • ίʔυ͚ͩͰUIΛ࡞Δ৔߹͸ "BroadcastSetupViewController" Λ͍͡Ε͹OK ‣ ͋ͱ͸όγόγίʔυΛॻ͍ͯͶ •

    StoryboardϕʔεͰUIΛ࡞Γ͍ͨ৔߹͸·ͣStoryboardΛ࡞Γ·͠ΐ͏
55. 55.

    
     ࣮૷खॱ • TARGETS͔ΒSetupUI ExtensionΛબ୒ • General಺ͷDeployment Info͔ΒMain InterfaceΛมߋ

56. 56.

    
     ࣮૷खॱ • InfoλϒΛ։͍ͯ "NSExtension" ಺ͷ "NSExtensionPrincipalClass" Λফ͢ •

    "NSExtension" ಺Ͱ "NSExtensionMainStoryboard" ͕͋Δ͜ͱΛ֬ೝ
57. 57.

    
     ࣮૷खॱ • ޙ͸ "BroadcastSetupViewController.swift" ͷαϯϓϧίʔυΛݟͳ͕Β࣮૷ • ഑৴༻ͷॲཧ͸Broadcast Upload

    ExtensionଆͰ࣮૷ • SetupUI ExtensionͰ͸LIVE഑৴ʹඞཁͳ৘ใΛೖྗͯ͠΋Β͏ ‣ e.g. ഑৴ͷλΠτϧɺίϝϯτɺެ։ൣғ
58. 58.

    
     iOS 11+ Ұ୴੔ཧ

59. 59.

    
     iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension
60. 60.

    
     iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension ഑৴͍ͨ͠
61. 61.

    
     iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension SetupUI ഑৴͍ͨ͠
62. 62.

    
     iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension SetupUI ഑৴͍ͨ͠ CMSampleBuffer
63. 63.

    
     iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension SetupUI ഑৴͍ͨ͠ CMSampleBuffer ੈք
64. 64.

    
     iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension ίϯτϩʔϧηϯλʔ
65. 65.

    
     iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension ίϯτϩʔϧηϯλʔ ഑৴͍ͨ͠
66. 66.

    
     iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ഑৴͍ͨ͠
67. 67.

    
     iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ഑৴͍ͨ͠ ੈք
68. 68.

    
     iOS 11+ Broadcast Upload Extension

69. 69.

    • ॳظঢ়ଶͰ͸ϑΝΠϧ͸2ͭ • "SampleHandler.swift" ʹ࣮૷Λॻ͍͍ͯ͘
     ࣮૷खॱ

70. 70.

    
     iOS 11+ SampleHandler.swift ʹ ࣮૷Λॻ͍͍ͯ͘

71. 71.

    None
72. 72.

    • ഑৴։࢝࣌ʹݺ͹ΕΔ • ϩάΠϯॲཧ • ഑৴༻ͷԼॲཧ

73. 73.

    • ഑৴͕ϙʔζɾ࠶։͞ΕͨΒݺ͹ΕΔ

74. 74.

    • broadcastFinishedɿ഑৴ऴྃ࣌ʹݺ͹ΕΔ • finishBroadcastWithErrorɿΤϥʔ͕ൃੜͯ͠ऴྃ͢Δͱ͖ʹݺ͹ΕΔ

75. 75.

    • ͜͜ͰCMSampleBufferΛड͚औΔ • ͋ͱ͸αʔόαΠυʹૹ৴͢Ε͹Α͍

76. 76.

    None
77. 77.

    • αʔϏε͕࠾༻͍ͯ͠ΔϓϩτίϧʹΑΔ • RTMP(S)͔HLSͳΒHaishinKitΛར༻͢Δͷ͕͓͢͢Ί ‣ https://github.com/shogo4405/HaishinKit.swift • ࠓճ͸WebRTCͷ࿩ ‣ ࣌Ӎಊ͞ΜͷSoraΛར༻

    ‣ Sora iOS SDK͕͋ΔͷͰiOS͸ͪ͜ΒΛར༻ - https://github.com/shiguredo/shiguredo-webrtc-ios
     CMSampleBufferΛૹΔ
78. 78.

    
     ࣮૷खॱ

79. 79.

    
     ໰୊ • Broadcast Upload ExtensionͰར༻Ͱ͖ΔϝϞϦ͸50MB·Ͱ • ௒͑Δͱ໰౴ແ༻ͰKill͞ΕΔ •

    Sora iOS SDK͕CMSampleBufferΛVP8Τϯίʔυ͢Δࡍʹ௒͑Δ ‣ VP8͸CPUΤϯίʔυͳͷͰϝϞϦΛফඅ͢Δ
80. 80.

    
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer

81. 81.

    
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer

82. 82.

    
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage

83. 83.

    
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CIImage transformed(by:)

84. 84.

    
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer

    CIImage transformed(by:)
85. 85.

    
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer

    CIImage transformed(by:) SDK΁
86. 86.

    
     ϦαΠζͨ͠ΒૹΕͨ

87. 87.

    • FPSɾը࣭ɾϝϞϦফඅྔͷτϨʔυΦϑ͔Β࠷దղΛ໨ࢦ͢ ‣ αʔϏεͷੑ࣭ʹΑͬͯऔࣺબ୒͕มԽ͢Δ • ΤϯίʔυॲཧҎ֎Ͱ΋ϝϞϦ͸࢖͏ͷͰ஫ҙ
     ੍ݶΛ௒͑ͳ͍αΠζΛ୳Δ

88. 88.

    
     iOS 11+ Ի੠

89. 89.

    • ϚΠΫԻ੠ͱΞϓϦԻ੠ • ͳʹ΋͠ͳ͍ͱϚΠΫԻ੠͸औಘͰ͖ͳ͍ • ͜ͷίʔυΛ"broadcastStarted"ʹॻ͚͹औಘͰ͖Δ
     Ի੠

90. 90.

    
     iOS 11+ Sora iOS SDK͸libwebrtcͷ࢓্༷ Extension্ͰԻ੠͕ͱΕͳ͍

91. 91.

    • iOS11+ͷRPScreenRecorderΛར༻࣮ͨ͠૷ͳΒऔಘͰ͖Δ • Broadcast Upload ExtensionͩͱऔಘͰ͖ͳ͍ • libwebrtcΛվ଄͢Δ͔……ʁ • ࣌Ӎಊ͞ΜղܾࡦΛ୳Γத

    • RTMP(S) or HLSͰHaishinKitΛར༻͢Ε͹औಘՄೳ
     Sora iOS SDK with libwebrtc
92. 92.

    To Be Continued

93. 93.

    
     iOS 11+ ͔͜͜Β͸ࡉ͔ͳϊ΢ϋ΢ूͰ͢

94. 94.

    • ը໘ʹมԽ͕͋Δͱ͖͚ͩCMSampleBuffer͕ৼͬͯ͘Δ • ղܾࡦ͸2छྨ ‣ Ի੠ͷมԽʹ߹ΘͤͯલͷϑϨʔϜΛૹΔ - ແԻ഑৴ͩͱ٧Ή ‣ αʔόαΠυʹHeartbeatΛఆظతʹૹΔ

    - ΫϥΠΞϯταΠυͰϑϨʔϜΛੜ੒͢Δํ๏ - ΫϥΠΞϯταΠυͰ͸APIΛୟ͚ͩ͘ͰɺαʔόαΠυͰҎલͷϑϨʔϜΛ΋ͱʹੜ੒͢Δ
     ը໘ʹมԽ͕ແ͍ͱ͖
95. 95.

    • ը໘ऩ࿥͞Εͨ͘ͳ͍ίϯςϯπΛӅ͢ ‣ e.g. Netflix, Amazon Prime Video • ࠓɺը໘͕Ωϟϓνϟ͞ΕͯΔ͔൑ผ͢Δ

    • ը໘ऩ࿥ͷ։࢝ɾऴྃΛݕ஌͢Δ
     ը໘ऩ࿥Λݕ஌͍ͨ͠ͱ͖
96. 96.

    • print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍
     ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘

97. 97.

    • print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍
     ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘ ͔͜͜ΒSchemeΛมߋ͠·͠ΐ͏

98. 98.

    • ※·ͩਂ͘ௐ͍ࠪͯ͠ͳ͍৘ใ • ը໘ऩ࿥தʹGPUΛ࢖͏ΤϯίʔυʢH.264ʣΛ͢Δͱୣ͍߹͏ • ͏·͘ΤϯίʔυͰ͖ͣɺSora iOS SDKͰ΋ࠔ͍ͬͯΔϙΠϯτ • ৄ͍͠ํ͍·ͨ͠Β͝ڭ͍ࣔͩ͘͞ʂ

    
     ReplayKitͱH.264Τϯίʔυͷ૬ੑ͕ѱ͍
99. 99.

    • εΫϦʔϯ഑৴ػೳΛ࡞Γ͍ͨͱ͖͸ReplayKitΛ࢖͍·͠ΐ͏ • iOS11+ͷαϙʔτʹ͢Δͷ͕͓͢͢ΊͰ͢ • Broadcast Upload ExtensionΛ࣮૷͠·͠ΐ͏ • ϝϞϦ͕50MB͔͠࢖͑ͳ͍ͷͰ஫ҙ͠·͠ΐ͏

    
     ·ͱΊ