スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019

3bf6da1f63cd8b7a2f802f95c269ed80?s=47 FromAtom
September 05, 2019

スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019

iOSDC 2019
Day 0 17:50~
Track A

3bf6da1f63cd8b7a2f802f95c269ed80?s=128

FromAtom

September 05, 2019
Tweet

Transcript

  1. εΫϦʔϯ഑৴ػೳͷ ࣮૷͕େมͩͬͨͷͰ ஌ݟΛ͓఻͑͠·͢ iOSDC Japan 2019 2019/09/05 17:50~ Track A

    pixiv.inc FromAtom
  2. • Swift 1.2 ͔ΒiOSΞϓϦ։ൃͷੈք΁ • ΞϓϦͷςοΫΛϦʔυ͍ͯ͠·͢ • pixiv Sketch /

    pixiv Sketch LIVE iOSΞϓϦ։ൃ   @FromAtom
  3.  

  4.   ͜ͷൃදͷΰʔϧ

  5.   ϥΠϒ഑৴ྲྀߦͬͯΔ͠ɺ ͏ͪ΋ϥΠϒ഑৴΍Δͧʂʂʂ

  6.   ͱɺͳͬͨͱ͖ʹࠔΒͳ͍Α͏ʹ

  7.   εΫϦʔϯ഑৴ػೳ

  8.   εΫϦʔϯ഑৴ػೳͱ͸ εϚʔτϑΥϯͷը໘ΛϥΠϒ഑৴Ͱ͖Δػೳ

  9. • εϚϗ୯ମͰϥΠϒ഑৴͕Մೳ • ήʔϜ࣮گ΍͓ֆ͔͖ͷ഑৴ʹར༻͞Ε͍ͯΔ • ࠷ۙ͸VTuberք۾Ͱ΋ར༻͞Ε͍ͯΔ   εΫϦʔϯ഑৴ػೳͱ͸

  10.   ReplayKitͱεΫϦʔϯ഑৴

  11. • ReplayKitΛར༻͢ΔͱεΫϦʔϯͷऩ࿥΍LIVE഑৴͕؆୯ʹͰ͖Δ • ϚΠΫԻ੠΍ΞϓϦ͕ग़ྗ͍ͯ͠ΔԻ΋औಘͰ͖Δ • ͨͩ͠ɺiOSͷόʔδϣϯʹΑͬͯͰ͖Δ͜ͱ͕ҧ͏   ReplayKitͱεΫϦʔϯ഑৴

  12. • iOS 9+ ‣ ૊ΈࠐΜͩΞϓϦͷը໘Λ࿥ըͰ͖Δ ‣ ഑৴͸Ͱ͖ͳ͍ • iOS 10+

    ‣ Broadcast Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ • iOS 11+ ‣ ༷ʑͳը໘͕഑৴ɾ࿥ըͰ͖Δ ‣ ίϯτϩʔϧηϯλʔ͔Βը໘࿥ըͰ͖Δͷͱಉ͡࢓૊Έ   ReplayKitͱεΫϦʔϯ഑৴
  13.   ࠓ͔Β࡞ΔͳΒ ࠷௿Ͱ΋iOS 11+

  14.   ͡Ό͋ReplayKitʹ͍ͭͯௐ΂ͯΈΑ͏

  15. None
  16. None
  17. 5͔݅͠هࣄ͕Ͷ͐ʂ

  18.   ͳͷͰɺͦΕͧΕͷ࡞ΓํΛղઆ

  19. • ࣮ػͰ͔͠σόοάͰ͖ͳ͍ ‣ Χϝϥ΍Push௨஌ͱಉ͡ • iOSඪ४ͷը໘ऩ࿥΍QuickTimeܦ༝Ͱͷը໘࿥ըͱڝ߹͢Δ ‣ σϞͷࡱӨΛ͢Δ௚લʹؾ͍ͮͯযΒͳ͍Α͏ʹ͠·͠ΐ͏ ‣ Pixel3ͰiPhoneXΛࡱӨ͢Δ೔ʑΛա͝͠·ͨ͠

      ͪΐͬͱͨ͠஫ҙ఺
  20.   ΞϓϦͷը໘Λ࿥ըͰ͖Δ iOS 9+

  21.  

  22. ࿥ը։࢝

  23.   ࿥ըऴྃ

  24.   iOS 10+ ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ

  25. • Broadcast Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ • ഑৴Λ͍ͨ͠ΞϓϦͱ഑৴͢ΔΞϓϦͷ૒ํͰ࣮૷͕ඞཁ   iOS10Ͱͷը໘഑৴

  26.  

  27.   ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

  28.   ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

  29.   ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

  30.   ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷

  31.  

  32.   ഑৴͍ͨ͠ΞϓϦଆͰͷ࣮૷ -഑৴ऴྃ-

  33.   ഑৴͢ΔΞϓϦͰͷ࣮૷ • Broadcast Upload ExtensionΛ࣮૷͢Δ • ৄ͘͠͸͜ͷޙ iOS11+

    ͷ࿩ͱҰॹʹ͠·͢
  34.   iOS 11+ Ͳͷը໘΋഑৴ɾ࿥ըͰ͖Δ

  35. • iOS11+ ͔ΒͰ͖ΔΑ͏ʹͳͬͨࣄ ‣ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴   iOS11+Ͱͷը໘഑৴

  36. • ࣮૷ͨ͠ΞϓϦͰը໘ͷΩϟϓνϟ͕Ͱ͖Δ • ը໘ͷCMSampleBufferΛऔಘͯ͠࿥ըɾ഑৴͕Մೳʹ • ϚΠΫԻ੠΍ΞϓϦԻ੠΋औಘՄೳ   RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ

  37.   ࣮૷

  38. • ίϯτϩʔϧηϯλʔͷ˕͔Βબ୒Ͱ͖Δ • iOS9+ͱ͸ҧ͍ը໘શମ͕ऩ࿥Ͱ͖Δ ‣ ௨஌΋ऩ࿥͞ΕΔͷͰ஫ҙ   ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴

  39.  

  40.   iOS 11+ ΍΍͍͜͠ͷͰҰ୴੔ཧ

  41. • iOS 9+ ‣ ૊ΈࠐΜͩΞϓϦͷը໘Λ࿥ըͰ͖Δ͕഑৴͸Ͱ͖ͳ͍ • iOS 10+ ‣ Broadcast

    Upload ExtensionΛ࣮૷͍ͯ͠ΔΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ • iOS 11+ ‣ Broadcast Upload ExtensionΛܦ༝͠ͳ͍ը໘ͷऩ࿥ɾ࿥ը ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ഑৴   ReplayKitͱεΫϦʔϯ഑৴
  42.   iOS 11+ ͜ΕͰReplayKitͰ Ͱ͖Δ͜ͱͷઆ໌׬ྃ

  43.   iOS 11+ Broadcast Upload Extension

  44. • ࠓճ͸͜ͷExtensionͷ࿩ • Լهͷ2ͭʹ֘౰ ‣ iOS10+ - ଞͷΞϓϦܦ༝Ͱ഑৴Ͱ͖Δ ‣ iOS11+

    - ίϯτϩʔϧηϯλʔܦ༝Ͱ഑৴Ͱ͖Δ   Broadcast Upload Extension
  45.   ࣮૷खॱ

  46.   ࣮૷खॱ

  47.   ࣮૷खॱ

  48.   ࣮૷खॱ

  49.   iOS 11+ SetupUI Extensionͬͯͳʹʁʁ

  50.   SetupUI Extensionͱ͸

  51.   SetupUI Extensionͱ͸

  52. • RPBroadcastActivityViewControllerܦ༝Ͱදࣔ͞ΕΔ • ͜ͷಋઢ͕ෆཁͳ৔߹͸Target͔Β࡟আ͢Ε͹Α͍   SetupUI Extensionͱ͸

  53.   ࣮૷खॱ • ίʔυ͚ͩͰUIΛ࡞Δ৔߹͸ "BroadcastSetupViewController" Λ͍͡Ε͹OK ‣ ͋ͱ͸όγόγίʔυΛॻ͍ͯͶ •

    StoryboardϕʔεͰUIΛ࡞Γ͍ͨ৔߹͸·ͣStoryboardΛ࡞Γ·͠ΐ͏
  54.   ࣮૷खॱ • ίʔυ͚ͩͰUIΛ࡞Δ৔߹͸ "BroadcastSetupViewController" Λ͍͡Ε͹OK ‣ ͋ͱ͸όγόγίʔυΛॻ͍ͯͶ •

    StoryboardϕʔεͰUIΛ࡞Γ͍ͨ৔߹͸·ͣStoryboardΛ࡞Γ·͠ΐ͏
  55.   ࣮૷खॱ • TARGETS͔ΒSetupUI ExtensionΛબ୒ • General಺ͷDeployment Info͔ΒMain InterfaceΛมߋ

  56.   ࣮૷खॱ • InfoλϒΛ։͍ͯ "NSExtension" ಺ͷ "NSExtensionPrincipalClass" Λফ͢ •

    "NSExtension" ಺Ͱ "NSExtensionMainStoryboard" ͕͋Δ͜ͱΛ֬ೝ
  57.   ࣮૷खॱ • ޙ͸ "BroadcastSetupViewController.swift" ͷαϯϓϧίʔυΛݟͳ͕Β࣮૷ • ഑৴༻ͷॲཧ͸Broadcast Upload

    ExtensionଆͰ࣮૷ • SetupUI ExtensionͰ͸LIVE഑৴ʹඞཁͳ৘ใΛೖྗͯ͠΋Β͏ ‣ e.g. ഑৴ͷλΠτϧɺίϝϯτɺެ։ൣғ
  58.   iOS 11+ Ұ୴੔ཧ

  59.   iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension
  60.   iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension ഑৴͍ͨ͠
  61.   iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension SetupUI ഑৴͍ͨ͠
  62.   iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension SetupUI ഑৴͍ͨ͠ CMSampleBuffer
  63.   iOS 10+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension SetupUI ഑৴͍ͨ͠ CMSampleBuffer ੈք
  64.   iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension ίϯτϩʔϧηϯλʔ
  65.   iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension ίϯτϩʔϧηϯλʔ ഑৴͍ͨ͠
  66.   iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ഑৴͍ͨ͠
  67.   iOS 11+ͷ৔߹ ഑৴͍ͨ͠ΞϓϦ ഑৴ΞϓϦ Broadcast SetupUI Extension Broadcast

    Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ഑৴͍ͨ͠ ੈք
  68.   iOS 11+ Broadcast Upload Extension

  69. • ॳظঢ়ଶͰ͸ϑΝΠϧ͸2ͭ • "SampleHandler.swift" ʹ࣮૷Λॻ͍͍ͯ͘   ࣮૷खॱ

  70.   iOS 11+ SampleHandler.swift ʹ ࣮૷Λॻ͍͍ͯ͘

  71. None
  72. • ഑৴։࢝࣌ʹݺ͹ΕΔ • ϩάΠϯॲཧ • ഑৴༻ͷԼॲཧ

  73. • ഑৴͕ϙʔζɾ࠶։͞ΕͨΒݺ͹ΕΔ

  74. • broadcastFinishedɿ഑৴ऴྃ࣌ʹݺ͹ΕΔ • finishBroadcastWithErrorɿΤϥʔ͕ൃੜͯ͠ऴྃ͢Δͱ͖ʹݺ͹ΕΔ

  75. • ͜͜ͰCMSampleBufferΛड͚औΔ • ͋ͱ͸αʔόαΠυʹૹ৴͢Ε͹Α͍

  76. None
  77. • αʔϏε͕࠾༻͍ͯ͠ΔϓϩτίϧʹΑΔ • RTMP(S)͔HLSͳΒHaishinKitΛར༻͢Δͷ͕͓͢͢Ί ‣ https://github.com/shogo4405/HaishinKit.swift • ࠓճ͸WebRTCͷ࿩ ‣ ࣌Ӎಊ͞ΜͷSoraΛར༻

    ‣ Sora iOS SDK͕͋ΔͷͰiOS͸ͪ͜ΒΛར༻ - https://github.com/shiguredo/shiguredo-webrtc-ios   CMSampleBufferΛૹΔ
  78.   ࣮૷खॱ

  79.   ໰୊ • Broadcast Upload ExtensionͰར༻Ͱ͖ΔϝϞϦ͸50MB·Ͱ • ௒͑Δͱ໰౴ແ༻ͰKill͞ΕΔ •

    Sora iOS SDK͕CMSampleBufferΛVP8Τϯίʔυ͢Δࡍʹ௒͑Δ ‣ VP8͸CPUΤϯίʔυͳͷͰϝϞϦΛফඅ͢Δ
  80.   ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer

  81.   ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer

  82.   ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage

  83.   ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CIImage transformed(by:)

  84.   ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer

    CIImage transformed(by:)
  85.   ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer

    CIImage transformed(by:) SDK΁
  86.   ϦαΠζͨ͠ΒૹΕͨ

  87. • FPSɾը࣭ɾϝϞϦফඅྔͷτϨʔυΦϑ͔Β࠷దղΛ໨ࢦ͢ ‣ αʔϏεͷੑ࣭ʹΑͬͯऔࣺબ୒͕มԽ͢Δ • ΤϯίʔυॲཧҎ֎Ͱ΋ϝϞϦ͸࢖͏ͷͰ஫ҙ   ੍ݶΛ௒͑ͳ͍αΠζΛ୳Δ

  88.   iOS 11+ Ի੠

  89. • ϚΠΫԻ੠ͱΞϓϦԻ੠ • ͳʹ΋͠ͳ͍ͱϚΠΫԻ੠͸औಘͰ͖ͳ͍ • ͜ͷίʔυΛ"broadcastStarted"ʹॻ͚͹औಘͰ͖Δ   Ի੠

  90.   iOS 11+ Sora iOS SDK͸libwebrtcͷ࢓্༷ Extension্ͰԻ੠͕ͱΕͳ͍

  91. • iOS11+ͷRPScreenRecorderΛར༻࣮ͨ͠૷ͳΒऔಘͰ͖Δ • Broadcast Upload ExtensionͩͱऔಘͰ͖ͳ͍ • libwebrtcΛվ଄͢Δ͔……ʁ • ࣌Ӎಊ͞ΜղܾࡦΛ୳Γத

    • RTMP(S) or HLSͰHaishinKitΛར༻͢Ε͹औಘՄೳ   Sora iOS SDK with libwebrtc
  92. To Be Continued

  93.   iOS 11+ ͔͜͜Β͸ࡉ͔ͳϊ΢ϋ΢ूͰ͢

  94. • ը໘ʹมԽ͕͋Δͱ͖͚ͩCMSampleBuffer͕ৼͬͯ͘Δ • ղܾࡦ͸2छྨ ‣ Ի੠ͷมԽʹ߹ΘͤͯલͷϑϨʔϜΛૹΔ - ແԻ഑৴ͩͱ٧Ή ‣ αʔόαΠυʹHeartbeatΛఆظతʹૹΔ

    - ΫϥΠΞϯταΠυͰϑϨʔϜΛੜ੒͢Δํ๏ - ΫϥΠΞϯταΠυͰ͸APIΛୟ͚ͩ͘ͰɺαʔόαΠυͰҎલͷϑϨʔϜΛ΋ͱʹੜ੒͢Δ   ը໘ʹมԽ͕ແ͍ͱ͖
  95. • ը໘ऩ࿥͞Εͨ͘ͳ͍ίϯςϯπΛӅ͢ ‣ e.g. Netflix, Amazon Prime Video • ࠓɺը໘͕Ωϟϓνϟ͞ΕͯΔ͔൑ผ͢Δ

    • ը໘ऩ࿥ͷ։࢝ɾऴྃΛݕ஌͢Δ   ը໘ऩ࿥Λݕ஌͍ͨ͠ͱ͖
  96. • print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍   ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘

  97. • print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍   ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘ ͔͜͜ΒSchemeΛมߋ͠·͠ΐ͏

  98. • ※·ͩਂ͘ௐ͍ࠪͯ͠ͳ͍৘ใ • ը໘ऩ࿥தʹGPUΛ࢖͏ΤϯίʔυʢH.264ʣΛ͢Δͱୣ͍߹͏ • ͏·͘ΤϯίʔυͰ͖ͣɺSora iOS SDKͰ΋ࠔ͍ͬͯΔϙΠϯτ • ৄ͍͠ํ͍·ͨ͠Β͝ڭ͍ࣔͩ͘͞ʂ

      ReplayKitͱH.264Τϯίʔυͷ૬ੑ͕ѱ͍
  99. • εΫϦʔϯ഑৴ػೳΛ࡞Γ͍ͨͱ͖͸ReplayKitΛ࢖͍·͠ΐ͏ • iOS11+ͷαϙʔτʹ͢Δͷ͕͓͢͢ΊͰ͢ • Broadcast Upload ExtensionΛ࣮૷͠·͠ΐ͏ • ϝϞϦ͕50MB͔͠࢖͑ͳ͍ͷͰ஫ҙ͠·͠ΐ͏

      ·ͱΊ