Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019
Search
FromAtom
September 05, 2019
Technology
7
13k
スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019
iOSDC 2019
Day 0 17:50~
Track A
FromAtom
September 05, 2019
Tweet
Share
More Decks by FromAtom
See All by FromAtom
「UIは英語なのにアプリ内リンクは日本語だ!」を防ぐコツ / pixiv App Night 2024-10-24
fromatom
0
81
『SwiftUIならiOS, macOSの両方で動くエディタアプリが簡単に作れる』 と思ったら大間違いだよ! / pixiv App Night 2024-01-25
fromatom
1
530
君だけのGFMエディタを作ろう! / iOSDC Japan 2023
fromatom
0
2k
サポートiOSバージョンを定期的にあげる仕組みづくり / iOSDC Japan 2022
fromatom
1
3.8k
僕たちが 『Appのプライバシーに関する質問への回答』 そして『ATT』に対応するまでの物語 / iOSDC Japan 2021
fromatom
1
4k
やってみよう! iOSDCデザインスポンサー! / iOSDC Japan 2021 LT
fromatom
2
1.4k
デバッグメニューのメンテナンスが大変だったので、専用アプリを作りました。 / iOSDC Japan 2020
fromatom
9
6.5k
🎉 esa 生誕5周年記念パーティー(\( ⁰⊖⁰)/) 🎉 / esa the 5th anniversary
fromatom
1
4.4k
複数のライブ映像を同期再生するのが大変だったので知見をお伝えします / iOSDC Japan 2018
fromatom
8
7.9k
Other Decks in Technology
See All in Technology
データ基盤の管理者からGoogle Cloud全体の管理者になっていた話
zozotech
PRO
0
180
マルチモーダル基盤モデルに基づく動画と音の解析技術
lycorptech_jp
PRO
4
410
経理出身PdMがAIプロダクト開発を_ハンズオンで学んだ話.pdf
shunsukenarita
1
290
2025新卒研修・HTML/CSS #弁護士ドットコム
bengo4com
3
4.7k
robocopy の怖い話/scary-story-about-robocopy
emiki
0
440
2時間で300+テーブルをデータ基盤に連携するためのAI活用 / FukuokaDataEngineer
sansan_randd
0
110
【2025 Japan AWS Jr. Champions Ignition】点から線、線から面へ〜僕たちが起こすコラボレーション・ムーブメント〜
amixedcolor
1
110
東京海上日動におけるセキュアな開発プロセスの取り組み
miyabit
0
220
LIFF CLIとngrokを使ったLIFF/LINEミニアプリのお手軽実機確認
diggymo
0
150
製造業の課題解決に向けた機械学習の活用と、製造業特化LLM開発への挑戦
knt44kw
0
140
2025-07-25 NOT A HOTEL TECH TALK ━ スマートホーム開発の最前線 ━ SOFTWARE
wakinchan
0
200
Mambaで物体検出 完全に理解した
shirarei24
2
180
Featured
See All Featured
[RailsConf 2023] Rails as a piece of cake
palkan
56
5.7k
The Cost Of JavaScript in 2023
addyosmani
51
8.7k
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.5k
Measuring & Analyzing Core Web Vitals
bluesmoon
7
530
Intergalactic Javascript Robots from Outer Space
tanoku
272
27k
Java REST API Framework Comparison - PWX 2021
mraible
31
8.7k
Gamification - CAS2011
davidbonilla
81
5.4k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
Navigating Team Friction
lara
188
15k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
21
1.4k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
3k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Transcript
εΫϦʔϯ৴ػೳͷ ࣮͕େมͩͬͨͷͰ ݟΛ͓͑͠·͢ iOSDC Japan 2019 2019/09/05 17:50~ Track A
pixiv.inc FromAtom
• Swift 1.2 ͔ΒiOSΞϓϦ։ൃͷੈք • ΞϓϦͷςοΫΛϦʔυ͍ͯ͠·͢ • pixiv Sketch /
pixiv Sketch LIVE iOSΞϓϦ։ൃ @FromAtom
͜ͷൃදͷΰʔϧ
ϥΠϒ৴ྲྀߦͬͯΔ͠ɺ ͏ͪϥΠϒ৴Δͧʂʂʂ
ͱɺͳͬͨͱ͖ʹࠔΒͳ͍Α͏ʹ
εΫϦʔϯ৴ػೳ
εΫϦʔϯ৴ػೳͱ εϚʔτϑΥϯͷը໘ΛϥΠϒ৴Ͱ͖Δػೳ
• εϚϗ୯ମͰϥΠϒ৴͕Մೳ • ήʔϜ࣮گ͓ֆ͔͖ͷ৴ʹར༻͞Ε͍ͯΔ • ࠷ۙVTuberք۾Ͱར༻͞Ε͍ͯΔ εΫϦʔϯ৴ػೳͱ
ReplayKitͱεΫϦʔϯ৴
• ReplayKitΛར༻͢ΔͱεΫϦʔϯͷऩLIVE৴͕؆୯ʹͰ͖Δ • ϚΠΫԻΞϓϦ͕ग़ྗ͍ͯ͠ΔԻऔಘͰ͖Δ • ͨͩ͠ɺiOSͷόʔδϣϯʹΑͬͯͰ͖Δ͜ͱ͕ҧ͏ ReplayKitͱεΫϦʔϯ৴
• iOS 9+ ‣ ΈࠐΜͩΞϓϦͷը໘ΛըͰ͖Δ ‣ ৴Ͱ͖ͳ͍ • iOS 10+
‣ Broadcast Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ • iOS 11+ ‣ ༷ʑͳը໘͕৴ɾըͰ͖Δ ‣ ίϯτϩʔϧηϯλʔ͔Βը໘ըͰ͖Δͷͱಉ͡Έ ReplayKitͱεΫϦʔϯ৴
ࠓ͔Β࡞ΔͳΒ ࠷ͰiOS 11+
͡Ό͋ReplayKitʹ͍ͭͯௐͯΈΑ͏
None
None
5͔݅͠هࣄ͕Ͷ͐ʂ
ͳͷͰɺͦΕͧΕͷ࡞ΓํΛղઆ
• ࣮ػͰ͔͠σόοάͰ͖ͳ͍ ‣ ΧϝϥPush௨ͱಉ͡ • iOSඪ४ͷը໘ऩQuickTimeܦ༝Ͱͷը໘ըͱڝ߹͢Δ ‣ σϞͷࡱӨΛ͢Δલʹؾ͍ͮͯযΒͳ͍Α͏ʹ͠·͠ΐ͏ ‣ Pixel3ͰiPhoneXΛࡱӨ͢ΔʑΛա͝͠·ͨ͠
ͪΐͬͱͨ͠ҙ
ΞϓϦͷը໘ΛըͰ͖Δ iOS 9+
ը։࢝
ըऴྃ
iOS 10+ ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ
• Broadcast Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ • ৴Λ͍ͨ͠ΞϓϦͱ৴͢ΔΞϓϦͷํͰ࣮͕ඞཁ iOS10Ͱͷը໘৴
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮ -৴ऴྃ-
৴͢ΔΞϓϦͰͷ࣮ • Broadcast Upload ExtensionΛ࣮͢Δ • ৄ͘͜͠ͷޙ iOS11+
ͷͱҰॹʹ͠·͢
iOS 11+ Ͳͷը໘৴ɾըͰ͖Δ
• iOS11+ ͔ΒͰ͖ΔΑ͏ʹͳͬͨࣄ ‣ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ৴ iOS11+Ͱͷը໘৴
• ࣮ͨ͠ΞϓϦͰը໘ͷΩϟϓνϟ͕Ͱ͖Δ • ը໘ͷCMSampleBufferΛऔಘͯ͠ըɾ৴͕Մೳʹ • ϚΠΫԻΞϓϦԻऔಘՄೳ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ
࣮
• ίϯτϩʔϧηϯλʔͷ˕͔ΒબͰ͖Δ • iOS9+ͱҧ͍ը໘શମ͕ऩͰ͖Δ ‣ ௨ऩ͞ΕΔͷͰҙ ίϯτϩʔϧηϯλʔܦ༝ͷ৴
iOS 11+ ͍͜͠ͷͰҰ୴ཧ
• iOS 9+ ‣ ΈࠐΜͩΞϓϦͷը໘ΛըͰ͖Δ͕৴Ͱ͖ͳ͍ • iOS 10+ ‣ Broadcast
Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ • iOS 11+ ‣ Broadcast Upload ExtensionΛܦ༝͠ͳ͍ը໘ͷऩɾը ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ৴ ReplayKitͱεΫϦʔϯ৴
iOS 11+ ͜ΕͰReplayKitͰ Ͱ͖Δ͜ͱͷઆ໌ྃ
iOS 11+ Broadcast Upload Extension
• ࠓճ͜ͷExtensionͷ • Լهͷ2ͭʹ֘ ‣ iOS10+ - ଞͷΞϓϦܦ༝Ͱ৴Ͱ͖Δ ‣ iOS11+
- ίϯτϩʔϧηϯλʔܦ༝Ͱ৴Ͱ͖Δ Broadcast Upload Extension
࣮खॱ
࣮खॱ
࣮खॱ
࣮खॱ
iOS 11+ SetupUI Extensionͬͯͳʹʁʁ
SetupUI Extensionͱ
SetupUI Extensionͱ
• RPBroadcastActivityViewControllerܦ༝Ͱදࣔ͞ΕΔ • ͜ͷಋઢ͕ෆཁͳ߹Target͔Βআ͢ΕΑ͍ SetupUI Extensionͱ
࣮खॱ • ίʔυ͚ͩͰUIΛ࡞Δ߹ "BroadcastSetupViewController" Λ͍͡ΕOK ‣ ͋ͱόγόγίʔυΛॻ͍ͯͶ •
StoryboardϕʔεͰUIΛ࡞Γ͍ͨ߹·ͣStoryboardΛ࡞Γ·͠ΐ͏
࣮खॱ • ίʔυ͚ͩͰUIΛ࡞Δ߹ "BroadcastSetupViewController" Λ͍͡ΕOK ‣ ͋ͱόγόγίʔυΛॻ͍ͯͶ •
StoryboardϕʔεͰUIΛ࡞Γ͍ͨ߹·ͣStoryboardΛ࡞Γ·͠ΐ͏
࣮खॱ • TARGETS͔ΒSetupUI ExtensionΛબ • GeneralͷDeployment Info͔ΒMain InterfaceΛมߋ
࣮खॱ • InfoλϒΛ։͍ͯ "NSExtension" ͷ "NSExtensionPrincipalClass" Λফ͢ •
"NSExtension" Ͱ "NSExtensionMainStoryboard" ͕͋Δ͜ͱΛ֬ೝ
࣮खॱ • ޙ "BroadcastSetupViewController.swift" ͷαϯϓϧίʔυΛݟͳ͕Β࣮ • ৴༻ͷॲཧBroadcast Upload
ExtensionଆͰ࣮ • SetupUI ExtensionͰLIVE৴ʹඞཁͳใΛೖྗͯ͠Β͏ ‣ e.g. ৴ͷλΠτϧɺίϝϯτɺެ։ൣғ
iOS 11+ Ұ୴ཧ
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ৴͍ͨ͠
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension SetupUI ৴͍ͨ͠
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension SetupUI ৴͍ͨ͠ CMSampleBuffer
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension SetupUI ৴͍ͨ͠ CMSampleBuffer ੈք
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ ৴͍ͨ͠
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ৴͍ͨ͠
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ৴͍ͨ͠ ੈք
iOS 11+ Broadcast Upload Extension
• ॳظঢ়ଶͰϑΝΠϧ2ͭ • "SampleHandler.swift" ʹ࣮Λॻ͍͍ͯ͘ ࣮खॱ
iOS 11+ SampleHandler.swift ʹ ࣮Λॻ͍͍ͯ͘
None
• ৴։࢝࣌ʹݺΕΔ • ϩάΠϯॲཧ • ৴༻ͷԼॲཧ
• ৴͕ϙʔζɾ࠶։͞ΕͨΒݺΕΔ
• broadcastFinishedɿ৴ऴྃ࣌ʹݺΕΔ • finishBroadcastWithErrorɿΤϥʔ͕ൃੜͯ͠ऴྃ͢Δͱ͖ʹݺΕΔ
• ͜͜ͰCMSampleBufferΛड͚औΔ • ͋ͱαʔόαΠυʹૹ৴͢ΕΑ͍
None
• αʔϏε͕࠾༻͍ͯ͠ΔϓϩτίϧʹΑΔ • RTMP(S)͔HLSͳΒHaishinKitΛར༻͢Δͷ͕͓͢͢Ί ‣ https://github.com/shogo4405/HaishinKit.swift • ࠓճWebRTCͷ ‣ ࣌Ӎಊ͞ΜͷSoraΛར༻
‣ Sora iOS SDK͕͋ΔͷͰiOSͪ͜ΒΛར༻ - https://github.com/shiguredo/shiguredo-webrtc-ios CMSampleBufferΛૹΔ
࣮खॱ
• Broadcast Upload ExtensionͰར༻Ͱ͖ΔϝϞϦ50MB·Ͱ • ͑Δͱແ༻ͰKill͞ΕΔ •
Sora iOS SDK͕CMSampleBufferΛVP8Τϯίʔυ͢Δࡍʹ͑Δ ‣ VP8CPUΤϯίʔυͳͷͰϝϞϦΛফඅ͢Δ
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CIImage transformed(by:)
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer
CIImage transformed(by:)
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer
CIImage transformed(by:) SDK
ϦαΠζͨ͠ΒૹΕͨ
• FPSɾը࣭ɾϝϞϦফඅྔͷτϨʔυΦϑ͔Β࠷దղΛࢦ͢ ‣ αʔϏεͷੑ࣭ʹΑͬͯऔࣺબ͕มԽ͢Δ • ΤϯίʔυॲཧҎ֎ͰϝϞϦ͏ͷͰҙ ੍ݶΛ͑ͳ͍αΠζΛ୳Δ
iOS 11+ Ի
• ϚΠΫԻͱΞϓϦԻ • ͳʹ͠ͳ͍ͱϚΠΫԻऔಘͰ͖ͳ͍ • ͜ͷίʔυΛ"broadcastStarted"ʹॻ͚औಘͰ͖Δ Ի
iOS 11+ Sora iOS SDKlibwebrtcͷ্༷ Extension্ͰԻ͕ͱΕͳ͍
• iOS11+ͷRPScreenRecorderΛར༻࣮ͨ͠ͳΒऔಘͰ͖Δ • Broadcast Upload ExtensionͩͱऔಘͰ͖ͳ͍ • libwebrtcΛվ͢Δ͔……ʁ • ࣌Ӎಊ͞ΜղܾࡦΛ୳Γத
• RTMP(S) or HLSͰHaishinKitΛར༻͢ΕऔಘՄೳ Sora iOS SDK with libwebrtc
To Be Continued
iOS 11+ ͔͜͜Βࡉ͔ͳϊϋूͰ͢
• ը໘ʹมԽ͕͋Δͱ͖͚ͩCMSampleBuffer͕ৼͬͯ͘Δ • ղܾࡦ2छྨ ‣ ԻͷมԽʹ߹ΘͤͯલͷϑϨʔϜΛૹΔ - ແԻ৴ͩͱ٧Ή ‣ αʔόαΠυʹHeartbeatΛఆظతʹૹΔ
- ΫϥΠΞϯταΠυͰϑϨʔϜΛੜ͢Δํ๏ - ΫϥΠΞϯταΠυͰAPIΛୟ͚ͩ͘ͰɺαʔόαΠυͰҎલͷϑϨʔϜΛͱʹੜ͢Δ ը໘ʹมԽ͕ແ͍ͱ͖
• ը໘ऩ͞Εͨ͘ͳ͍ίϯςϯπΛӅ͢ ‣ e.g. Netflix, Amazon Prime Video • ࠓɺը໘͕Ωϟϓνϟ͞ΕͯΔ͔ผ͢Δ
• ը໘ऩͷ։࢝ɾऴྃΛݕ͢Δ ը໘ऩΛݕ͍ͨ͠ͱ͖
• print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍ ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘
• print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍ ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘ ͔͜͜ΒSchemeΛมߋ͠·͠ΐ͏
• ※·ͩਂ͘ௐ͍ࠪͯ͠ͳ͍ใ • ը໘ऩதʹGPUΛ͏ΤϯίʔυʢH.264ʣΛ͢Δͱୣ͍߹͏ • ͏·͘ΤϯίʔυͰ͖ͣɺSora iOS SDKͰࠔ͍ͬͯΔϙΠϯτ • ৄ͍͠ํ͍·ͨ͠Β͝ڭ͍ࣔͩ͘͞ʂ
ReplayKitͱH.264Τϯίʔυͷ૬ੑ͕ѱ͍
• εΫϦʔϯ৴ػೳΛ࡞Γ͍ͨͱ͖ReplayKitΛ͍·͠ΐ͏ • iOS11+ͷαϙʔτʹ͢Δͷ͕͓͢͢ΊͰ͢ • Broadcast Upload ExtensionΛ࣮͠·͠ΐ͏ • ϝϞϦ͕50MB͔͑͠ͳ͍ͷͰҙ͠·͠ΐ͏
·ͱΊ