$30 off During Our Annual Pro Sale. View Details »

スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019

FromAtom
September 05, 2019

スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019

iOSDC 2019
Day 0 17:50~
Track A

FromAtom

September 05, 2019
Tweet

More Decks by FromAtom

Other Decks in Technology

Transcript

  1. εΫϦʔϯ഑৴ػೳͷ
    ࣮૷͕େมͩͬͨͷͰ
    ஌ݟΛ͓఻͑͠·͢
    iOSDC Japan 2019
    2019/09/05 17:50~ Track A
    pixiv.inc
    FromAtom

    View Slide

  2. • Swift 1.2 ͔ΒiOSΞϓϦ։ൃͷੈք΁
    • ΞϓϦͷςοΫΛϦʔυ͍ͯ͠·͢
    • pixiv Sketch / pixiv Sketch LIVE iOSΞϓϦ։ൃ


    @FromAtom

    View Slide



  3. View Slide



  4. ͜ͷൃදͷΰʔϧ

    View Slide



  5. ϥΠϒ഑৴ྲྀߦͬͯΔ͠ɺ
    ͏ͪ΋ϥΠϒ഑৴΍Δͧʂʂʂ

    View Slide



  6. ͱɺͳͬͨͱ͖ʹࠔΒͳ͍Α͏ʹ

    View Slide



  7. εΫϦʔϯ഑৴ػೳ

    View Slide



  8. εΫϦʔϯ഑৴ػೳͱ͸
    εϚʔτϑΥϯͷը໘ΛϥΠϒ഑৴Ͱ͖Δػೳ

    View Slide

  9. • εϚϗ୯ମͰϥΠϒ഑৴͕Մೳ
    • ήʔϜ࣮گ΍͓ֆ͔͖ͷ഑৴ʹར༻͞Ε͍ͯΔ
    • ࠷ۙ͸VTuberք۾Ͱ΋ར༻͞Ε͍ͯΔ


    εΫϦʔϯ഑৴ػೳͱ͸

    View Slide



  10. ReplayKitͱεΫϦʔϯ഑৴

    View Slide

  11. • ReplayKitΛར༻͢ΔͱεΫϦʔϯͷऩ࿥΍LIVE഑৴͕؆୯ʹͰ͖Δ
    • ϚΠΫԻ੠΍ΞϓϦ͕ग़ྗ͍ͯ͠ΔԻ΋औಘͰ͖Δ
    • ͨͩ͠ɺiOSͷόʔδϣϯʹΑͬͯͰ͖Δ͜ͱ͕ҧ͏


    ReplayKitͱεΫϦʔϯ഑৴

    View Slide

  12. • iOS 9+
    ‣ ૊ΈࠐΜͩΞϓϦͷը໘Λ࿥ըͰ͖Δ
    ‣ ഑৴͸Ͱ͖ͳ͍
    • iOS 10+
    ‣ Broadcast Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ
    • iOS 11+
    ‣ ༷ʑͳը໘͕഑৴ɾ࿥ըͰ͖Δ
    ‣ ίϯτϩʔϧηϯλʔ͔Βը໘࿥ըͰ͖Δͷͱಉ͡࢓૊Έ


    ReplayKitͱεΫϦʔϯ഑৴

    View Slide



  13. ࠓ͔Β࡞ΔͳΒ
    ࠷௿Ͱ΋iOS 11+

    View Slide



  14. ͡Ό͋ReplayKitʹ͍ͭͯௐ΂ͯΈΑ͏

    View Slide

  15. View Slide

  16. View Slide

  17. 5͔݅͠هࣄ͕Ͷ͐ʂ

    View Slide



  18. ͳͷͰɺͦΕͧΕͷ࡞ΓํΛղઆ

    View Slide

  19. • ࣮ػͰ͔͠σόοάͰ͖ͳ͍
    ‣ Χϝϥ΍Push௨஌ͱಉ͡
    • iOSඪ४ͷը໘ऩ࿥΍QuickTimeܦ༝Ͱͷը໘࿥ըͱڝ߹͢Δ
    ‣ σϞͷࡱӨΛ͢Δ௚લʹؾ͍ͮͯযΒͳ͍Α͏ʹ͠·͠ΐ͏
    ‣ Pixel3ͰiPhoneXΛࡱӨ͢Δ೔ʑΛա͝͠·ͨ͠


    ͪΐͬͱͨ͠஫ҙ఺

    View Slide



  20. ΞϓϦͷը໘Λ࿥ըͰ͖Δ
    iOS 9+

    View Slide



  21. View Slide

  22. ࿥ը։࢝

    View Slide



  23. ࿥ըऴྃ

    View Slide



  24. iOS 10+
    ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ

    View Slide

  25. • Broadcast Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ
    • ഑৴Λ͍ͨ͠ΞϓϦͱ഑৴͢ΔΞϓϦͷ૒ํͰ࣮૷͕ඞཁ


    iOS10Ͱͷը໘഑৴

    View Slide



  26. View Slide



  27. ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

    View Slide



  28. ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

    View Slide



  29. ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

    View Slide



  30. ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

    View Slide



  31. View Slide



  32. ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷ -഑৴ऴྃ-

    View Slide



  33. ഑৴͢ΔΞϓϦͰͷ࣮૷
    • Broadcast Upload ExtensionΛ࣮૷͢Δ
    • ৄ͘͠͸͜ͷޙ iOS11+ ͷ࿩ͱҰॹʹ͠·͢

    View Slide



  34. iOS 11+
    Ͳͷը໘΋഑৴ɾ࿥ըͰ͖Δ

    View Slide

  35. • iOS11+ ͔ΒͰ͖ΔΑ͏ʹͳͬͨࣄ
    ‣ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ
    ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴


    iOS11+Ͱͷը໘഑৴

    View Slide

  36. • ࣮૷ͨ͠ΞϓϦͰը໘ͷΩϟϓνϟ͕Ͱ͖Δ
    • ը໘ͷCMSampleBufferΛऔಘͯ͠࿥ըɾ഑৴͕Մೳʹ
    • ϚΠΫԻ੠΍ΞϓϦԻ੠΋औಘՄೳ


    RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ

    View Slide



  37. ࣮૷

    View Slide

  38. • ίϯτϩʔϧηϯλʔͷ˕͔Βબ୒Ͱ͖Δ
    • iOS9+ͱ͸ҧ͍ը໘શମ͕ऩ࿥Ͱ͖Δ
    ‣ ௨஌΋ऩ࿥͞ΕΔͷͰ஫ҙ


    ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴

    View Slide



  39. View Slide



  40. iOS 11+
    ΍΍͍͜͠ͷͰҰ୴੔ཧ

    View Slide

  41. • iOS 9+
    ‣ ૊ΈࠐΜͩΞϓϦͷը໘Λ࿥ըͰ͖Δ͕഑৴͸Ͱ͖ͳ͍
    • iOS 10+
    ‣ Broadcast Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ
    • iOS 11+
    ‣ Broadcast Upload ExtensionΛܦ༝͠ͳ͍ը໘ͷऩ࿥ɾ࿥ը
    ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴


    ReplayKitͱεΫϦʔϯ഑৴

    View Slide



  42. iOS 11+
    ͜ΕͰReplayKitͰ
    Ͱ͖Δ͜ͱͷઆ໌׬ྃ

    View Slide



  43. iOS 11+
    Broadcast Upload Extension

    View Slide

  44. • ࠓճ͸͜ͷExtensionͷ࿩
    • Լهͷ2ͭʹ֘౰
    ‣ iOS10+
    - ଞͷΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ
    ‣ iOS11+
    - ίϯτϩʔϧηϯλʔܦ༝Ͱ഑৴Ͱ͖Δ


    Broadcast Upload Extension

    View Slide



  45. ࣮૷खॱ

    View Slide



  46. ࣮૷खॱ

    View Slide



  47. ࣮૷खॱ

    View Slide



  48. ࣮૷खॱ

    View Slide



  49. iOS 11+
    SetupUI Extensionͬͯͳʹʁʁ

    View Slide



  50. SetupUI Extensionͱ͸

    View Slide



  51. SetupUI Extensionͱ͸

    View Slide

  52. • RPBroadcastActivityViewControllerܦ༝Ͱදࣔ͞ΕΔ
    • ͜ͷಋઢ͕ෆཁͳ৔߹͸Target͔Β࡟আ͢Ε͹Α͍


    SetupUI Extensionͱ͸

    View Slide



  53. ࣮૷खॱ
    • ίʔυ͚ͩͰUIΛ࡞Δ৔߹͸ "BroadcastSetupViewController" Λ͍͡Ε͹OK
    ‣ ͋ͱ͸όγόγίʔυΛॻ͍ͯͶ
    • StoryboardϕʔεͰUIΛ࡞Γ͍ͨ৔߹͸·ͣStoryboardΛ࡞Γ·͠ΐ͏

    View Slide



  54. ࣮૷खॱ
    • ίʔυ͚ͩͰUIΛ࡞Δ৔߹͸ "BroadcastSetupViewController" Λ͍͡Ε͹OK
    ‣ ͋ͱ͸όγόγίʔυΛॻ͍ͯͶ
    • StoryboardϕʔεͰUIΛ࡞Γ͍ͨ৔߹͸·ͣStoryboardΛ࡞Γ·͠ΐ͏

    View Slide



  55. ࣮૷खॱ
    • TARGETS͔ΒSetupUI ExtensionΛબ୒
    • General಺ͷDeployment Info͔ΒMain InterfaceΛมߋ

    View Slide



  56. ࣮૷खॱ
    • InfoλϒΛ։͍ͯ "NSExtension" ಺ͷ "NSExtensionPrincipalClass" Λফ͢
    • "NSExtension" ಺Ͱ "NSExtensionMainStoryboard" ͕͋Δ͜ͱΛ֬ೝ

    View Slide



  57. ࣮૷खॱ
    • ޙ͸ "BroadcastSetupViewController.swift" ͷαϯϓϧίʔυΛݟͳ͕Β࣮૷
    • ഑৴༻ͷॲཧ͸Broadcast Upload ExtensionଆͰ࣮૷
    • SetupUI ExtensionͰ͸LIVE഑৴ʹඞཁͳ৘ใΛೖྗͯ͠΋Β͏
    ‣ e.g. ഑৴ͷλΠτϧɺίϝϯτɺެ։ൣғ

    View Slide



  58. iOS 11+
    Ұ୴੔ཧ

    View Slide



  59. iOS 10+ͷ৔߹
    ഑৴͍ͨ͠ΞϓϦ
    ഑৴ΞϓϦ
    Broadcast SetupUI Extension
    Broadcast Upload Extension

    View Slide



  60. iOS 10+ͷ৔߹
    ഑৴͍ͨ͠ΞϓϦ
    ഑৴ΞϓϦ
    Broadcast SetupUI Extension
    Broadcast Upload Extension
    ഑৴͍ͨ͠

    View Slide



  61. iOS 10+ͷ৔߹
    ഑৴͍ͨ͠ΞϓϦ
    ഑৴ΞϓϦ
    Broadcast SetupUI Extension
    Broadcast Upload Extension
    SetupUI
    ഑৴͍ͨ͠

    View Slide



  62. iOS 10+ͷ৔߹
    ഑৴͍ͨ͠ΞϓϦ
    ഑৴ΞϓϦ
    Broadcast SetupUI Extension
    Broadcast Upload Extension
    SetupUI
    ഑৴͍ͨ͠
    CMSampleBuffer

    View Slide



  63. iOS 10+ͷ৔߹
    ഑৴͍ͨ͠ΞϓϦ
    ഑৴ΞϓϦ
    Broadcast SetupUI Extension
    Broadcast Upload Extension
    SetupUI
    ഑৴͍ͨ͠
    CMSampleBuffer
    ੈք

    View Slide



  64. iOS 11+ͷ৔߹
    ഑৴͍ͨ͠ΞϓϦ
    ഑৴ΞϓϦ
    Broadcast SetupUI Extension
    Broadcast Upload Extension
    ίϯτϩʔϧηϯλʔ

    View Slide



  65. iOS 11+ͷ৔߹
    ഑৴͍ͨ͠ΞϓϦ
    ഑৴ΞϓϦ
    Broadcast SetupUI Extension
    Broadcast Upload Extension
    ίϯτϩʔϧηϯλʔ
    ഑৴͍ͨ͠

    View Slide



  66. iOS 11+ͷ৔߹
    ഑৴͍ͨ͠ΞϓϦ
    ഑৴ΞϓϦ
    Broadcast SetupUI Extension
    Broadcast Upload Extension
    ίϯτϩʔϧηϯλʔ
    CMSampleBuffer
    ഑৴͍ͨ͠

    View Slide



  67. iOS 11+ͷ৔߹
    ഑৴͍ͨ͠ΞϓϦ
    ഑৴ΞϓϦ
    Broadcast SetupUI Extension
    Broadcast Upload Extension
    ίϯτϩʔϧηϯλʔ
    CMSampleBuffer
    ഑৴͍ͨ͠ ੈք

    View Slide



  68. iOS 11+
    Broadcast Upload Extension

    View Slide

  69. • ॳظঢ়ଶͰ͸ϑΝΠϧ͸2ͭ
    • "SampleHandler.swift" ʹ࣮૷Λॻ͍͍ͯ͘


    ࣮૷खॱ

    View Slide



  70. iOS 11+
    SampleHandler.swift ʹ
    ࣮૷Λॻ͍͍ͯ͘

    View Slide

  71. View Slide

  72. • ഑৴։࢝࣌ʹݺ͹ΕΔ
    • ϩάΠϯॲཧ
    • ഑৴༻ͷԼॲཧ

    View Slide

  73. • ഑৴͕ϙʔζɾ࠶։͞ΕͨΒݺ͹ΕΔ

    View Slide

  74. • broadcastFinishedɿ഑৴ऴྃ࣌ʹݺ͹ΕΔ
    • finishBroadcastWithErrorɿΤϥʔ͕ൃੜͯ͠ऴྃ͢Δͱ͖ʹݺ͹ΕΔ

    View Slide

  75. • ͜͜ͰCMSampleBufferΛड͚औΔ
    • ͋ͱ͸αʔόαΠυʹૹ৴͢Ε͹Α͍

    View Slide

  76. View Slide

  77. • αʔϏε͕࠾༻͍ͯ͠ΔϓϩτίϧʹΑΔ
    • RTMP(S)͔HLSͳΒHaishinKitΛར༻͢Δͷ͕͓͢͢Ί
    ‣ https://github.com/shogo4405/HaishinKit.swift
    • ࠓճ͸WebRTCͷ࿩
    ‣ ࣌Ӎಊ͞ΜͷSoraΛར༻
    ‣ Sora iOS SDK͕͋ΔͷͰiOS͸ͪ͜ΒΛར༻
    - https://github.com/shiguredo/shiguredo-webrtc-ios


    CMSampleBufferΛૹΔ

    View Slide



  78. ࣮૷खॱ

    View Slide



  79. ໰୊
    • Broadcast Upload ExtensionͰར༻Ͱ͖ΔϝϞϦ͸50MB·Ͱ
    • ௒͑Δͱ໰౴ແ༻ͰKill͞ΕΔ
    • Sora iOS SDK͕CMSampleBufferΛVP8Τϯίʔυ͢Δࡍʹ௒͑Δ
    ‣ VP8͸CPUΤϯίʔυͳͷͰϝϞϦΛফඅ͢Δ

    View Slide



  80. ղܾํ๏
    • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢
    CMSampleBuffer

    View Slide



  81. ղܾํ๏
    • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢
    CMSampleBuffer
    CVPixelBuffer

    View Slide



  82. ղܾํ๏
    • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢
    CMSampleBuffer
    CVPixelBuffer
    CIImage

    View Slide



  83. ղܾํ๏
    • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢
    CMSampleBuffer
    CVPixelBuffer
    CIImage CIImage
    transformed(by:)

    View Slide



  84. ղܾํ๏
    • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢
    CMSampleBuffer
    CVPixelBuffer
    CIImage
    CMSampleBuffer
    CVPixelBuffer
    CIImage
    transformed(by:)

    View Slide



  85. ղܾํ๏
    • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢
    CMSampleBuffer
    CVPixelBuffer
    CIImage
    CMSampleBuffer
    CVPixelBuffer
    CIImage
    transformed(by:)
    SDK΁

    View Slide




  86. ϦαΠζͨ͠ΒૹΕͨ

    View Slide

  87. • FPSɾը࣭ɾϝϞϦফඅྔͷτϨʔυΦϑ͔Β࠷దղΛ໨ࢦ͢
    ‣ αʔϏεͷੑ࣭ʹΑͬͯऔࣺબ୒͕มԽ͢Δ
    • ΤϯίʔυॲཧҎ֎Ͱ΋ϝϞϦ͸࢖͏ͷͰ஫ҙ


    ੍ݶΛ௒͑ͳ͍αΠζΛ୳Δ

    View Slide



  88. iOS 11+
    Ի੠

    View Slide

  89. • ϚΠΫԻ੠ͱΞϓϦԻ੠
    • ͳʹ΋͠ͳ͍ͱϚΠΫԻ੠͸औಘͰ͖ͳ͍
    • ͜ͷίʔυΛ"broadcastStarted"ʹॻ͚͹औಘͰ͖Δ


    Ի੠

    View Slide



  90. iOS 11+
    Sora iOS SDK͸libwebrtcͷ࢓্༷
    Extension্ͰԻ੠͕ͱΕͳ͍

    View Slide

  91. • iOS11+ͷRPScreenRecorderΛར༻࣮ͨ͠૷ͳΒऔಘͰ͖Δ
    • Broadcast Upload ExtensionͩͱऔಘͰ͖ͳ͍
    • libwebrtcΛվ଄͢Δ͔……ʁ
    • ࣌Ӎಊ͞ΜղܾࡦΛ୳Γத
    • RTMP(S) or HLSͰHaishinKitΛར༻͢Ε͹औಘՄೳ


    Sora iOS SDK with libwebrtc

    View Slide

  92. To Be Continued

    View Slide



  93. iOS 11+
    ͔͜͜Β͸ࡉ͔ͳϊ΢ϋ΢ूͰ͢

    View Slide

  94. • ը໘ʹมԽ͕͋Δͱ͖͚ͩCMSampleBuffer͕ৼͬͯ͘Δ
    • ղܾࡦ͸2छྨ
    ‣ Ի੠ͷมԽʹ߹ΘͤͯલͷϑϨʔϜΛૹΔ
    - ແԻ഑৴ͩͱ٧Ή
    ‣ αʔόαΠυʹHeartbeatΛఆظతʹૹΔ
    - ΫϥΠΞϯταΠυͰϑϨʔϜΛੜ੒͢Δํ๏
    - ΫϥΠΞϯταΠυͰ͸APIΛୟ͚ͩ͘ͰɺαʔόαΠυͰҎલͷϑϨʔϜΛ΋ͱʹੜ੒͢Δ


    ը໘ʹมԽ͕ແ͍ͱ͖

    View Slide

  95. • ը໘ऩ࿥͞Εͨ͘ͳ͍ίϯςϯπΛӅ͢
    ‣ e.g. Netflix, Amazon Prime Video
    • ࠓɺը໘͕Ωϟϓνϟ͞ΕͯΔ͔൑ผ͢Δ
    • ը໘ऩ࿥ͷ։࢝ɾऴྃΛݕ஌͢Δ


    ը໘ऩ࿥Λݕ஌͍ͨ͠ͱ͖

    View Slide

  96. • print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍
    • BreakpointͰࢭ·Βͳ͍


    ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘

    View Slide

  97. • print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍
    • BreakpointͰࢭ·Βͳ͍


    ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘
    ͔͜͜ΒSchemeΛมߋ͠·͠ΐ͏

    View Slide

  98. • ※·ͩਂ͘ௐ͍ࠪͯ͠ͳ͍৘ใ
    • ը໘ऩ࿥தʹGPUΛ࢖͏ΤϯίʔυʢH.264ʣΛ͢Δͱୣ͍߹͏
    • ͏·͘ΤϯίʔυͰ͖ͣɺSora iOS SDKͰ΋ࠔ͍ͬͯΔϙΠϯτ
    • ৄ͍͠ํ͍·ͨ͠Β͝ڭ͍ࣔͩ͘͞ʂ


    ReplayKitͱH.264Τϯίʔυͷ૬ੑ͕ѱ͍

    View Slide

  99. • εΫϦʔϯ഑৴ػೳΛ࡞Γ͍ͨͱ͖͸ReplayKitΛ࢖͍·͠ΐ͏
    • iOS11+ͷαϙʔτʹ͢Δͷ͕͓͢͢ΊͰ͢
    • Broadcast Upload ExtensionΛ࣮૷͠·͠ΐ͏
    • ϝϞϦ͕50MB͔͠࢖͑ͳ͍ͷͰ஫ҙ͠·͠ΐ͏


    ·ͱΊ

    View Slide