Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019
Search
FromAtom
September 05, 2019
Technology
7
13k
スクリーン配信機能の実装が大変だったので知見をお伝えします / iOSDC2019
iOSDC 2019
Day 0 17:50~
Track A
FromAtom
September 05, 2019
Tweet
Share
More Decks by FromAtom
See All by FromAtom
「UIは英語なのにアプリ内リンクは日本語だ!」を防ぐコツ / pixiv App Night 2024-10-24
fromatom
0
48
『SwiftUIならiOS, macOSの両方で動くエディタアプリが簡単に作れる』 と思ったら大間違いだよ! / pixiv App Night 2024-01-25
fromatom
1
360
君だけのGFMエディタを作ろう! / iOSDC Japan 2023
fromatom
0
1.8k
サポートiOSバージョンを定期的にあげる仕組みづくり / iOSDC Japan 2022
fromatom
1
3.3k
僕たちが 『Appのプライバシーに関する質問への回答』 そして『ATT』に対応するまでの物語 / iOSDC Japan 2021
fromatom
1
3.5k
やってみよう! iOSDCデザインスポンサー! / iOSDC Japan 2021 LT
fromatom
2
1.3k
デバッグメニューのメンテナンスが大変だったので、専用アプリを作りました。 / iOSDC Japan 2020
fromatom
9
6k
🎉 esa 生誕5周年記念パーティー(\( ⁰⊖⁰)/) 🎉 / esa the 5th anniversary
fromatom
1
4.2k
複数のライブ映像を同期再生するのが大変だったので知見をお伝えします / iOSDC Japan 2018
fromatom
8
7.4k
Other Decks in Technology
See All in Technology
AWS SAW(AWS Support Automation Workflows)をもっと広めたい
kazzpapa3
2
170
Why and Why not of enabling swap in Kubernetes
hwchiu
0
440
WHOLENESS, REPAIRING, AND TO HAVE FUN: 全体性、修復、そして楽しむこと
snoozer05
PRO
2
2.9k
AI Builder について
miyakemito
1
120
AWS CDKで大量のパラメータストアを作りたい
y_kotani
1
150
Hotwire光の道とStimulus
nay3
5
2k
Mobbing Practices
kawaguti
PRO
2
290
Comparing Apache Flink and Spark for Modern Stream Data Processing
sharonx
0
170
開発健全性の可視化と開発者体験の改善 ~ Compassでエンジニアに活力と生産性を ~
atlassianjapan
0
170
Aurora_BlueGreenDeploymentsやってみた
tsukasa_ishimaru
1
110
6年の歴史×ペタバイト級のデータ基盤のチームを一体化する開発スタイル
plaidtech
PRO
4
110
LLMアプリをRagasで評価して、Langfuseで可視化しよう!
minorun365
PRO
2
190
Featured
See All Featured
Rebuilding a faster, lazier Slack
samanthasiow
79
8.6k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
355
29k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
27
4.1k
10 Git Anti Patterns You Should be Aware of
lemiorhan
653
59k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
43
6.6k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Building a Modern Day E-commerce SEO Strategy
aleyda
38
6.9k
KATA
mclloyd
29
13k
Build The Right Thing And Hit Your Dates
maggiecrowley
32
2.4k
Unsuck your backbone
ammeep
668
57k
Designing for Performance
lara
604
68k
Agile that works and the tools we love
rasmusluckow
327
21k
Transcript
εΫϦʔϯ৴ػೳͷ ࣮͕େมͩͬͨͷͰ ݟΛ͓͑͠·͢ iOSDC Japan 2019 2019/09/05 17:50~ Track A
pixiv.inc FromAtom
• Swift 1.2 ͔ΒiOSΞϓϦ։ൃͷੈք • ΞϓϦͷςοΫΛϦʔυ͍ͯ͠·͢ • pixiv Sketch /
pixiv Sketch LIVE iOSΞϓϦ։ൃ @FromAtom
͜ͷൃදͷΰʔϧ
ϥΠϒ৴ྲྀߦͬͯΔ͠ɺ ͏ͪϥΠϒ৴Δͧʂʂʂ
ͱɺͳͬͨͱ͖ʹࠔΒͳ͍Α͏ʹ
εΫϦʔϯ৴ػೳ
εΫϦʔϯ৴ػೳͱ εϚʔτϑΥϯͷը໘ΛϥΠϒ৴Ͱ͖Δػೳ
• εϚϗ୯ମͰϥΠϒ৴͕Մೳ • ήʔϜ࣮گ͓ֆ͔͖ͷ৴ʹར༻͞Ε͍ͯΔ • ࠷ۙVTuberք۾Ͱར༻͞Ε͍ͯΔ εΫϦʔϯ৴ػೳͱ
ReplayKitͱεΫϦʔϯ৴
• ReplayKitΛར༻͢ΔͱεΫϦʔϯͷऩLIVE৴͕؆୯ʹͰ͖Δ • ϚΠΫԻΞϓϦ͕ग़ྗ͍ͯ͠ΔԻऔಘͰ͖Δ • ͨͩ͠ɺiOSͷόʔδϣϯʹΑͬͯͰ͖Δ͜ͱ͕ҧ͏ ReplayKitͱεΫϦʔϯ৴
• iOS 9+ ‣ ΈࠐΜͩΞϓϦͷը໘ΛըͰ͖Δ ‣ ৴Ͱ͖ͳ͍ • iOS 10+
‣ Broadcast Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ • iOS 11+ ‣ ༷ʑͳը໘͕৴ɾըͰ͖Δ ‣ ίϯτϩʔϧηϯλʔ͔Βը໘ըͰ͖Δͷͱಉ͡Έ ReplayKitͱεΫϦʔϯ৴
ࠓ͔Β࡞ΔͳΒ ࠷ͰiOS 11+
͡Ό͋ReplayKitʹ͍ͭͯௐͯΈΑ͏
None
None
5͔݅͠هࣄ͕Ͷ͐ʂ
ͳͷͰɺͦΕͧΕͷ࡞ΓํΛղઆ
• ࣮ػͰ͔͠σόοάͰ͖ͳ͍ ‣ ΧϝϥPush௨ͱಉ͡ • iOSඪ४ͷը໘ऩQuickTimeܦ༝Ͱͷը໘ըͱڝ߹͢Δ ‣ σϞͷࡱӨΛ͢Δલʹؾ͍ͮͯযΒͳ͍Α͏ʹ͠·͠ΐ͏ ‣ Pixel3ͰiPhoneXΛࡱӨ͢ΔʑΛա͝͠·ͨ͠
ͪΐͬͱͨ͠ҙ
ΞϓϦͷը໘ΛըͰ͖Δ iOS 9+
ը։࢝
ըऴྃ
iOS 10+ ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ
• Broadcast Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ • ৴Λ͍ͨ͠ΞϓϦͱ৴͢ΔΞϓϦͷํͰ࣮͕ඞཁ iOS10Ͱͷը໘৴
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮
৴͍ͨ͠ΞϓϦଆͰͷ࣮ -৴ऴྃ-
৴͢ΔΞϓϦͰͷ࣮ • Broadcast Upload ExtensionΛ࣮͢Δ • ৄ͘͜͠ͷޙ iOS11+
ͷͱҰॹʹ͠·͢
iOS 11+ Ͳͷը໘৴ɾըͰ͖Δ
• iOS11+ ͔ΒͰ͖ΔΑ͏ʹͳͬͨࣄ ‣ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ৴ iOS11+Ͱͷը໘৴
• ࣮ͨ͠ΞϓϦͰը໘ͷΩϟϓνϟ͕Ͱ͖Δ • ը໘ͷCMSampleBufferΛऔಘͯ͠ըɾ৴͕Մೳʹ • ϚΠΫԻΞϓϦԻऔಘՄೳ RPScreenRecorderΛར༻ͨ͠ը໘ͷΩϟϓνϟ
࣮
• ίϯτϩʔϧηϯλʔͷ˕͔ΒબͰ͖Δ • iOS9+ͱҧ͍ը໘શମ͕ऩͰ͖Δ ‣ ௨ऩ͞ΕΔͷͰҙ ίϯτϩʔϧηϯλʔܦ༝ͷ৴
iOS 11+ ͍͜͠ͷͰҰ୴ཧ
• iOS 9+ ‣ ΈࠐΜͩΞϓϦͷը໘ΛըͰ͖Δ͕৴Ͱ͖ͳ͍ • iOS 10+ ‣ Broadcast
Upload ExtensionΛ࣮͍ͯ͠ΔΞϓϦܦ༝Ͱ৴Ͱ͖Δ • iOS 11+ ‣ Broadcast Upload ExtensionΛܦ༝͠ͳ͍ը໘ͷऩɾը ‣ ίϯτϩʔϧηϯλʔܦ༝ͷ৴ ReplayKitͱεΫϦʔϯ৴
iOS 11+ ͜ΕͰReplayKitͰ Ͱ͖Δ͜ͱͷઆ໌ྃ
iOS 11+ Broadcast Upload Extension
• ࠓճ͜ͷExtensionͷ • Լهͷ2ͭʹ֘ ‣ iOS10+ - ଞͷΞϓϦܦ༝Ͱ৴Ͱ͖Δ ‣ iOS11+
- ίϯτϩʔϧηϯλʔܦ༝Ͱ৴Ͱ͖Δ Broadcast Upload Extension
࣮खॱ
࣮खॱ
࣮खॱ
࣮खॱ
iOS 11+ SetupUI Extensionͬͯͳʹʁʁ
SetupUI Extensionͱ
SetupUI Extensionͱ
• RPBroadcastActivityViewControllerܦ༝Ͱදࣔ͞ΕΔ • ͜ͷಋઢ͕ෆཁͳ߹Target͔Βআ͢ΕΑ͍ SetupUI Extensionͱ
࣮खॱ • ίʔυ͚ͩͰUIΛ࡞Δ߹ "BroadcastSetupViewController" Λ͍͡ΕOK ‣ ͋ͱόγόγίʔυΛॻ͍ͯͶ •
StoryboardϕʔεͰUIΛ࡞Γ͍ͨ߹·ͣStoryboardΛ࡞Γ·͠ΐ͏
࣮खॱ • ίʔυ͚ͩͰUIΛ࡞Δ߹ "BroadcastSetupViewController" Λ͍͡ΕOK ‣ ͋ͱόγόγίʔυΛॻ͍ͯͶ •
StoryboardϕʔεͰUIΛ࡞Γ͍ͨ߹·ͣStoryboardΛ࡞Γ·͠ΐ͏
࣮खॱ • TARGETS͔ΒSetupUI ExtensionΛબ • GeneralͷDeployment Info͔ΒMain InterfaceΛมߋ
࣮खॱ • InfoλϒΛ։͍ͯ "NSExtension" ͷ "NSExtensionPrincipalClass" Λফ͢ •
"NSExtension" Ͱ "NSExtensionMainStoryboard" ͕͋Δ͜ͱΛ֬ೝ
࣮खॱ • ޙ "BroadcastSetupViewController.swift" ͷαϯϓϧίʔυΛݟͳ͕Β࣮ • ৴༻ͷॲཧBroadcast Upload
ExtensionଆͰ࣮ • SetupUI ExtensionͰLIVE৴ʹඞཁͳใΛೖྗͯ͠Β͏ ‣ e.g. ৴ͷλΠτϧɺίϝϯτɺެ։ൣғ
iOS 11+ Ұ୴ཧ
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ৴͍ͨ͠
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension SetupUI ৴͍ͨ͠
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension SetupUI ৴͍ͨ͠ CMSampleBuffer
iOS 10+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension SetupUI ৴͍ͨ͠ CMSampleBuffer ੈք
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ ৴͍ͨ͠
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ৴͍ͨ͠
iOS 11+ͷ߹ ৴͍ͨ͠ΞϓϦ ৴ΞϓϦ Broadcast SetupUI Extension Broadcast
Upload Extension ίϯτϩʔϧηϯλʔ CMSampleBuffer ৴͍ͨ͠ ੈք
iOS 11+ Broadcast Upload Extension
• ॳظঢ়ଶͰϑΝΠϧ2ͭ • "SampleHandler.swift" ʹ࣮Λॻ͍͍ͯ͘ ࣮खॱ
iOS 11+ SampleHandler.swift ʹ ࣮Λॻ͍͍ͯ͘
None
• ৴։࢝࣌ʹݺΕΔ • ϩάΠϯॲཧ • ৴༻ͷԼॲཧ
• ৴͕ϙʔζɾ࠶։͞ΕͨΒݺΕΔ
• broadcastFinishedɿ৴ऴྃ࣌ʹݺΕΔ • finishBroadcastWithErrorɿΤϥʔ͕ൃੜͯ͠ऴྃ͢Δͱ͖ʹݺΕΔ
• ͜͜ͰCMSampleBufferΛड͚औΔ • ͋ͱαʔόαΠυʹૹ৴͢ΕΑ͍
None
• αʔϏε͕࠾༻͍ͯ͠ΔϓϩτίϧʹΑΔ • RTMP(S)͔HLSͳΒHaishinKitΛར༻͢Δͷ͕͓͢͢Ί ‣ https://github.com/shogo4405/HaishinKit.swift • ࠓճWebRTCͷ ‣ ࣌Ӎಊ͞ΜͷSoraΛར༻
‣ Sora iOS SDK͕͋ΔͷͰiOSͪ͜ΒΛར༻ - https://github.com/shiguredo/shiguredo-webrtc-ios CMSampleBufferΛૹΔ
࣮खॱ
• Broadcast Upload ExtensionͰར༻Ͱ͖ΔϝϞϦ50MB·Ͱ • ͑Δͱແ༻ͰKill͞ΕΔ •
Sora iOS SDK͕CMSampleBufferΛVP8Τϯίʔυ͢Δࡍʹ͑Δ ‣ VP8CPUΤϯίʔυͳͷͰϝϞϦΛফඅ͢Δ
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CIImage transformed(by:)
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer
CIImage transformed(by:)
ղܾํ๏ • CMSampleBufferΛॖখ͔ͯ͠ΒSDKʹΘͨ͢ CMSampleBuffer CVPixelBuffer CIImage CMSampleBuffer CVPixelBuffer
CIImage transformed(by:) SDK
ϦαΠζͨ͠ΒૹΕͨ
• FPSɾը࣭ɾϝϞϦফඅྔͷτϨʔυΦϑ͔Β࠷దղΛࢦ͢ ‣ αʔϏεͷੑ࣭ʹΑͬͯऔࣺબ͕มԽ͢Δ • ΤϯίʔυॲཧҎ֎ͰϝϞϦ͏ͷͰҙ ੍ݶΛ͑ͳ͍αΠζΛ୳Δ
iOS 11+ Ի
• ϚΠΫԻͱΞϓϦԻ • ͳʹ͠ͳ͍ͱϚΠΫԻऔಘͰ͖ͳ͍ • ͜ͷίʔυΛ"broadcastStarted"ʹॻ͚औಘͰ͖Δ Ի
iOS 11+ Sora iOS SDKlibwebrtcͷ্༷ Extension্ͰԻ͕ͱΕͳ͍
• iOS11+ͷRPScreenRecorderΛར༻࣮ͨ͠ͳΒऔಘͰ͖Δ • Broadcast Upload ExtensionͩͱऔಘͰ͖ͳ͍ • libwebrtcΛվ͢Δ͔……ʁ • ࣌Ӎಊ͞ΜղܾࡦΛ୳Γத
• RTMP(S) or HLSͰHaishinKitΛར༻͢ΕऔಘՄೳ Sora iOS SDK with libwebrtc
To Be Continued
iOS 11+ ͔͜͜Βࡉ͔ͳϊϋूͰ͢
• ը໘ʹมԽ͕͋Δͱ͖͚ͩCMSampleBuffer͕ৼͬͯ͘Δ • ղܾࡦ2छྨ ‣ ԻͷมԽʹ߹ΘͤͯલͷϑϨʔϜΛૹΔ - ແԻ৴ͩͱ٧Ή ‣ αʔόαΠυʹHeartbeatΛఆظతʹૹΔ
- ΫϥΠΞϯταΠυͰϑϨʔϜΛੜ͢Δํ๏ - ΫϥΠΞϯταΠυͰAPIΛୟ͚ͩ͘ͰɺαʔόαΠυͰҎલͷϑϨʔϜΛͱʹੜ͢Δ ը໘ʹมԽ͕ແ͍ͱ͖
• ը໘ऩ͞Εͨ͘ͳ͍ίϯςϯπΛӅ͢ ‣ e.g. Netflix, Amazon Prime Video • ࠓɺը໘͕Ωϟϓνϟ͞ΕͯΔ͔ผ͢Δ
• ը໘ऩͷ։࢝ɾऴྃΛݕ͢Δ ը໘ऩΛݕ͍ͨ͠ͱ͖
• print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍ ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘
• print("HOGE") ͯ͠Δͷʹίϯιʔϧʹग़ͳ͍ • BreakpointͰࢭ·Βͳ͍ ॳΊͯExtensionͷσόοάΛ͢Δͱ͖ͷ᠘ ͔͜͜ΒSchemeΛมߋ͠·͠ΐ͏
• ※·ͩਂ͘ௐ͍ࠪͯ͠ͳ͍ใ • ը໘ऩதʹGPUΛ͏ΤϯίʔυʢH.264ʣΛ͢Δͱୣ͍߹͏ • ͏·͘ΤϯίʔυͰ͖ͣɺSora iOS SDKͰࠔ͍ͬͯΔϙΠϯτ • ৄ͍͠ํ͍·ͨ͠Β͝ڭ͍ࣔͩ͘͞ʂ
ReplayKitͱH.264Τϯίʔυͷ૬ੑ͕ѱ͍
• εΫϦʔϯ৴ػೳΛ࡞Γ͍ͨͱ͖ReplayKitΛ͍·͠ΐ͏ • iOS11+ͷαϙʔτʹ͢Δͷ͕͓͢͢ΊͰ͢ • Broadcast Upload ExtensionΛ࣮͠·͠ΐ͏ • ϝϞϦ͕50MB͔͑͠ͳ͍ͷͰҙ͠·͠ΐ͏
·ͱΊ