スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019

Transcript

1. εΫϦʔϯ഑৴ػೳͷ ࣮૷͕େมͩͬͨͷͰ ஌ݟΛ͓఻͑͠·͢ iOSDC Japan 2019 2019/09/05 17:50~ Track A

   pixiv.inc FromAtom

2. • Swift 1.2 ͔ΒiOSΞϓϦ։ൃͷੈք΁ • ΞϓϦͷςοΫΛϦʔυ͍ͯ͠·͢ • pixiv Sketch /

   pixiv Sketch LIVE iOSΞϓϦ։ൃ
    @FromAtom

3. 
   

4. 
    ͜ͷൃදͷΰʔϧ

5. 
    ϥΠϒ഑৴ྲྀߦͬͯΔ͠ɺ ͏ͪ΋ϥΠϒ഑৴΍Δͧʂʂʂ

6. 
    ͱɺͳͬͨͱ͖ʹࠔΒͳ͍Α͏ʹ

7. 
    εΫϦʔϯ഑৴ػೳ

8. 
    εΫϦʔϯ഑৴ػೳͱ͸ εϚʔτϑΥϯͷը໘ΛϥΠϒ഑৴Ͱ͖Δػೳ

9. • εϚϗ୯ମͰϥΠϒ഑৴͕Մೳ • ήʔϜ࣮گ΍͓ֆ͔͖ͷ഑৴ʹར༻͞Ε͍ͯΔ • ࠷ۙ͸VTuberք۾Ͱ΋ར༻͞Ε͍ͯΔ
    εΫϦʔϯ഑৴ػೳͱ͸

10. 
     ReplayKitͱεΫϦʔϯ഑৴

11. • ReplayKitΛར༻͢ΔͱεΫϦʔϯͷऩ࿥΍LIVE഑৴͕؆୯ʹͰ͖Δ • ϚΠΫԻ੠΍ΞϓϦ͕ग़ྗ͍ͯ͠ΔԻ΋औಘͰ͖Δ • ͨͩ͠ɺiOSͷόʔδϣϯʹΑͬͯͰ͖Δ͜ͱ͕ҧ͏
     ReplayKitͱεΫϦʔϯ഑৴

12. • iOS 9+ ‣ ૊ΈࠐΜͩΞϓϦͷը໘Λ࿥ըͰ͖Δ ‣ ഑৴͸Ͱ͖ͳ͍ • iOS 10+

    ‣ Broadcast Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ • iOS 11+ ‣ ༷ʑͳը໘͕഑৴ɾ࿥ըͰ͖Δ ‣ ίϯτϩʔϧηϯλʔ͔Βը໘࿥ըͰ͖Δͷͱಉ͡࢓૊Έ
     ReplayKitͱεΫϦʔϯ഑৴

13. 
     ࠓ͔Β࡞ΔͳΒ ࠷௿Ͱ΋iOS 11+

14. 
     ͡Ό͋ReplayKitʹ͍ͭͯௐ΂ͯΈΑ͏

15. None
16. None

17. 5͔݅͠هࣄ͕Ͷ͐ʂ

18. 
     ͳͷͰɺͦΕͧΕͷ࡞ΓํΛղઆ

19. • ࣮ػͰ͔͠σόοάͰ͖ͳ͍ ‣ Χϝϥ΍Push௨஌ͱಉ͡ • iOSඪ४ͷը໘ऩ࿥΍QuickTimeܦ༝Ͱͷը໘࿥ըͱڝ߹͢Δ ‣ σϞͷࡱӨΛ͢Δ௚લʹؾ͍ͮͯযΒͳ͍Α͏ʹ͠·͠ΐ͏ ‣ Pixel3ͰiPhoneXΛࡱӨ͢Δ೔ʑΛա͝͠·ͨ͠

    
     ͪΐͬͱͨ͠஫ҙ఺

20. 
     ΞϓϦͷը໘Λ࿥ըͰ͖Δ iOS 9+

21. 
    

22. ࿥ը։࢝

23. 
     ࿥ըऴྃ

24. 
     iOS 10+ ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ

25. • Broadcast Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ • ഑৴Λ͍ͨ͠ΞϓϦͱ഑৴͢ΔΞϓϦͷ૒ํͰ࣮૷͕ඞཁ
     iOS10Ͱͷը໘഑৴

26. 
    

27. 
     ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

28. 
     ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

29. 
     ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

30. 
     ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

31. 
    

32. 
     ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷ -഑৴ऴྃ-

33. 
     ഑৴͢ΔΞϓϦͰͷ࣮૷ • Broadcast Upload ExtensionΛ࣮૷͢Δ • ৄ͘͠͸͜ͷޙ iOS11+

    ͷ࿩ͱҰॹʹ͠·͢

34. 
     iOS 11+ Ͳͷը໘΋഑৴ɾ࿥ըͰ͖Δ

35. • iOS11+ ͔ΒͰ͖ΔΑ͏ʹͳͬͨࣄ ‣ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴
     iOS11+Ͱͷը໘഑৴

36. • ࣮૷ͨ͠ΞϓϦͰը໘ͷΩϟϓνϟ͕Ͱ͖Δ • ը໘ͷCMSampleBufferΛऔಘͯ͠࿥ըɾ഑৴͕Մೳʹ • ϚΠΫԻ੠΍ΞϓϦԻ੠΋औಘՄೳ
     RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ

37. 
     ࣮૷

38. • ίϯτϩʔϧηϯλʔͷ˕͔Βબ୒Ͱ͖Δ • iOS9+ͱ͸ҧ͍ը໘શମ͕ऩ࿥Ͱ͖Δ ‣ ௨஌΋ऩ࿥͞ΕΔͷͰ஫ҙ
     ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴

39. 
    

40. 
     iOS 11+ ΍΍͍͜͠ͷͰҰ୴੔ཧ

41. • iOS 9+ ‣ ૊ΈࠐΜͩΞϓϦͷը໘Λ࿥ըͰ͖Δ͕഑৴͸Ͱ͖ͳ͍ • iOS 10+ ‣ Broadcast

    Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ • iOS 11+ ‣ Broadcast Upload ExtensionΛܦ༝͠ͳ͍ը໘ͷऩ࿥ɾ࿥ը ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴
     ReplayKitͱεΫϦʔϯ഑৴

42. 
     iOS 11+ ͜ΕͰReplayKitͰ Ͱ͖Δ͜ͱͷઆ໌׬ྃ

43. 
     iOS 11+ Broadcast Upload Extension

44. • ࠓճ͸͜ͷExtensionͷ࿩ • Լهͷ2ͭʹ֘౰ ‣ iOS10+ - ଞͷΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ ‣ iOS11+

    - ίϯτϩʔϧηϯλʔܦ༝Ͱ഑৴Ͱ͖Δ
     Broadcast Upload Extension

45. 
     ࣮૷खॱ

46. 
     ࣮૷खॱ

47. 
     ࣮૷खॱ

48. 
     ࣮૷खॱ

49. 
     iOS 11+ SetupUI Extensionͬͯͳʹʁʁ

50. 
     SetupUI Extensionͱ͸

51. 
     SetupUI Extensionͱ͸

52. • RPBroadcastActivityViewControllerܦ༝Ͱදࣔ͞ΕΔ • ͜ͷಋઢ͕ෆཁͳ৔߹͸Target͔Β࡟আ͢Ε͹Α͍
     SetupUI Extensionͱ͸

53. 
     ࣮૷खॱ • ίʔυ͚ͩͰUIΛ࡞Δ৔߹͸ "BroadcastSetupViewController" Λ͍͡Ε͹OK ‣ ͋ͱ͸όγόγίʔυΛॻ͍ͯͶ •

    StoryboardϕʔεͰUIΛ࡞Γ͍ͨ৔߹͸·ͣStoryboardΛ࡞Γ·͠ΐ͏

54. 
     ࣮૷खॱ • ίʔυ͚ͩͰUIΛ࡞Δ৔߹͸ "BroadcastSetupViewController" Λ͍͡Ε͹OK ‣ ͋ͱ͸όγόγίʔυΛॻ͍ͯͶ •

    StoryboardϕʔεͰUIΛ࡞Γ͍ͨ৔߹͸·ͣStoryboardΛ࡞Γ·͠ΐ͏

55. 
     ࣮૷खॱ • TARGETS͔ΒSetupUI ExtensionΛબ୒ • General಺ͷDeployment Info͔ΒMain InterfaceΛมߋ

56. 
     ࣮૷खॱ • InfoλϒΛ։͍ͯ "NSExtension" ಺ͷ "NSExtensionPrincipalClass" Λফ͢ •

    "NSExtension" ಺Ͱ "NSExtensionMainStoryboard" ͕͋Δ͜ͱΛ֬ೝ

57. 
     ࣮૷खॱ • ޙ͸ "BroadcastSetupViewController.swift" ͷαϯϓϧίʔυΛݟͳ͕Β࣮૷ • ഑৴༻ͷॲཧ͸Broadcast Upload

    ExtensionଆͰ࣮૷ • SetupUI ExtensionͰ͸LIVE഑৴ʹඞཁͳ৘ใΛೖྗͯ͠΋Β͏ ‣ e.g. ഑৴ͷλΠτϧɺίϝϯτɺެ։ൣғ

58. 
     iOS 11+ Ұ୴੔ཧ

59. 
     iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension

60. 
     iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension ഑৴͍ͨ͠

61. 
     iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension SetupUI ഑৴͍ͨ͠

62. 
     iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension SetupUI ഑৴͍ͨ͠ CMSampleBuffer

63. 
     iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension SetupUI ഑৴͍ͨ͠ CMSampleBuffer ੈք

64. 
     iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension ίϯτϩʔϧηϯλʔ

65. 
     iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension ίϯτϩʔϧηϯλʔ ഑৴͍ͨ͠

66. 
     iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ഑৴͍ͨ͠

67. 
     iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ഑৴͍ͨ͠ ੈք

68. 
     iOS 11+ Broadcast Upload Extension

69. • ॳظঢ়ଶͰ͸ϑΝΠϧ͸2ͭ • "SampleHandler.swift" ʹ࣮૷Λॻ͍͍ͯ͘
     ࣮૷खॱ

70. 
     iOS 11+ SampleHandler.swift ʹ ࣮૷Λॻ͍͍ͯ͘

71. None

72. • ഑৴։࢝࣌ʹݺ͹ΕΔ • ϩάΠϯॲཧ • ഑৴༻ͷԼॲཧ

73. • ഑৴͕ϙʔζɾ࠶։͞ΕͨΒݺ͹ΕΔ

74. • broadcastFinishedɿ഑৴ऴྃ࣌ʹݺ͹ΕΔ • finishBroadcastWithErrorɿΤϥʔ͕ൃੜͯ͠ऴྃ͢Δͱ͖ʹݺ͹ΕΔ

75. • ͜͜ͰCMSampleBufferΛड͚औΔ • ͋ͱ͸αʔόαΠυʹૹ৴͢Ε͹Α͍

76. None

77. • αʔϏε͕࠾༻͍ͯ͠ΔϓϩτίϧʹΑΔ • RTMP(S)͔HLSͳΒHaishinKitΛར༻͢Δͷ͕͓͢͢Ί ‣ https://github.com/shogo4405/HaishinKit.swift • ࠓճ͸WebRTCͷ࿩ ‣ ࣌Ӎಊ͞ΜͷSoraΛར༻

    ‣ Sora iOS SDK͕͋ΔͷͰiOS͸ͪ͜ΒΛར༻ - https://github.com/shiguredo/shiguredo-webrtc-ios
     CMSampleBufferΛૹΔ

78. 
     ࣮૷खॱ

79. 
     ໰୊ • Broadcast Upload ExtensionͰར༻Ͱ͖ΔϝϞϦ͸50MB·Ͱ • ௒͑Δͱ໰౴ແ༻ͰKill͞ΕΔ •

    Sora iOS SDK͕CMSampleBufferΛVP8Τϯίʔυ͢Δࡍʹ௒͑Δ ‣ VP8͸CPUΤϯίʔυͳͷͰϝϞϦΛফඅ͢Δ

80. 
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer

81. 
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer

82. 
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage

83. 
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CIImage transformed(by:)

84. 
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer

    CIImage transformed(by:)

85. 
     ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer

    CIImage transformed(by:) SDK΁

86. 
     ϦαΠζͨ͠ΒૹΕͨ

87. • FPSɾը࣭ɾϝϞϦফඅྔͷτϨʔυΦϑ͔Β࠷దղΛ໨ࢦ͢ ‣ αʔϏεͷੑ࣭ʹΑͬͯऔࣺબ୒͕มԽ͢Δ • ΤϯίʔυॲཧҎ֎Ͱ΋ϝϞϦ͸࢖͏ͷͰ஫ҙ
     ੍ݶΛ௒͑ͳ͍αΠζΛ୳Δ

88. 
     iOS 11+ Ի੠

89. • ϚΠΫԻ੠ͱΞϓϦԻ੠ • ͳʹ΋͠ͳ͍ͱϚΠΫԻ੠͸औಘͰ͖ͳ͍ • ͜ͷίʔυΛ"broadcastStarted"ʹॻ͚͹औಘͰ͖Δ
     Ի੠

90. 
     iOS 11+ Sora iOS SDK͸libwebrtcͷ࢓্༷ Extension্ͰԻ੠͕ͱΕͳ͍

91. • iOS11+ͷRPScreenRecorderΛར༻࣮ͨ͠૷ͳΒऔಘͰ͖Δ • Broadcast Upload ExtensionͩͱऔಘͰ͖ͳ͍ • libwebrtcΛվ଄͢Δ͔……ʁ • ࣌Ӎಊ͞ΜղܾࡦΛ୳Γத

    • RTMP(S) or HLSͰHaishinKitΛར༻͢Ε͹औಘՄೳ
     Sora iOS SDK with libwebrtc

92. To Be Continued

93. 
     iOS 11+ ͔͜͜Β͸ࡉ͔ͳϊ΢ϋ΢ूͰ͢

94. • ը໘ʹมԽ͕͋Δͱ͖͚ͩCMSampleBuffer͕ৼͬͯ͘Δ • ղܾࡦ͸2छྨ ‣ Ի੠ͷมԽʹ߹ΘͤͯલͷϑϨʔϜΛૹΔ - ແԻ഑৴ͩͱ٧Ή ‣ αʔόαΠυʹHeartbeatΛఆظతʹૹΔ

    - ΫϥΠΞϯταΠυͰϑϨʔϜΛੜ੒͢Δํ๏ - ΫϥΠΞϯταΠυͰ͸APIΛୟ͚ͩ͘ͰɺαʔόαΠυͰҎલͷϑϨʔϜΛ΋ͱʹੜ੒͢Δ
     ը໘ʹมԽ͕ແ͍ͱ͖

95. • ը໘ऩ࿥͞Εͨ͘ͳ͍ίϯςϯπΛӅ͢ ‣ e.g. Netflix, Amazon Prime Video • ࠓɺը໘͕Ωϟϓνϟ͞ΕͯΔ͔൑ผ͢Δ

    • ը໘ऩ࿥ͷ։࢝ɾऴྃΛݕ஌͢Δ
     ը໘ऩ࿥Λݕ஌͍ͨ͠ͱ͖

96. • print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍
     ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘

97. • print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍
     ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘ ͔͜͜ΒSchemeΛมߋ͠·͠ΐ͏

98. • ※·ͩਂ͘ௐ͍ࠪͯ͠ͳ͍৘ใ • ը໘ऩ࿥தʹGPUΛ࢖͏ΤϯίʔυʢH.264ʣΛ͢Δͱୣ͍߹͏ • ͏·͘ΤϯίʔυͰ͖ͣɺSora iOS SDKͰ΋ࠔ͍ͬͯΔϙΠϯτ • ৄ͍͠ํ͍·ͨ͠Β͝ڭ͍ࣔͩ͘͞ʂ

    
     ReplayKitͱH.264Τϯίʔυͷ૬ੑ͕ѱ͍

99. • εΫϦʔϯ഑৴ػೳΛ࡞Γ͍ͨͱ͖͸ReplayKitΛ࢖͍·͠ΐ͏ • iOS11+ͷαϙʔτʹ͢Δͷ͕͓͢͢ΊͰ͢ • Broadcast Upload ExtensionΛ࣮૷͠·͠ΐ͏ • ϝϞϦ͕50MB͔͠࢖͑ͳ͍ͷͰ஫ҙ͠·͠ΐ͏

    
     ·ͱΊ