Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
WebRTC動画をトランスコードする / Transcoding video streams ...
Search
Harukasan
PRO
February 05, 2018
Technology
5
1.5k
WebRTC動画をトランスコードする / Transcoding video streams from WebRTC
SkyWay UG Tokyo #2
2018-02-05
Harukasan
PRO
February 05, 2018
Tweet
Share
More Decks by Harukasan
See All by Harukasan
pixivを支える技術 / 技育CAMPアカデミア
harukasan
PRO
3
420
20240401 新卒研修 - ピクシブにおける技術領域
harukasan
PRO
1
730
ピクシブのコンテンツ配信基盤技術 / pixiv TECH SALON
harukasan
PRO
5
5.5k
Goにおける画像ファイル処理 / golang.tokyo #19
harukasan
PRO
7
6.6k
ImageFluxを支えるリモート開発 / 20171202
harukasan
PRO
2
1.8k
YAPC::Fukuoka 前夜祭LT / Yet Another Pawoo Commit logs
harukasan
PRO
0
2.9k
YAPC::Fukuoka lunch session
harukasan
PRO
1
3k
マストドン会議: Pawoo / Mastodon Kaigi2
harukasan
PRO
2
460
大規模Mastodonインスタンスを運用するコツ / Inside Pawoo Mastodon infrastructure
harukasan
PRO
0
3.2k
Other Decks in Technology
See All in Technology
【Developers Summit 2025】プロダクトエンジニアから学ぶ、 ユーザーにより高い価値を届ける技術
niwatakeru
2
890
Larkご案内資料
customercloud
PRO
0
600
事業継続を支える自動テストの考え方
tsuemura
0
300
Developers Summit 2025 浅野卓也(13-B-7 LegalOn Technologies)
legalontechnologies
PRO
0
150
個人開発から公式機能へ: PlaywrightとRailsをつなげた3年の軌跡
yusukeiwaki
10
2.7k
PL900試験から学ぶ Power Platform 基礎知識講座
kumikeyy
0
110
APIファーストで実現する運用性の高い IoT プラットフォーム: SORACOMのアプローチ
soracom
PRO
0
240
サーバーレスアーキテクチャと生成AIの融合 / Serverless Meets Generative AI
_kensh
12
3k
Fintech SREの挑戦 PCI DSS対応をスマートにこなすインフラ戦略/Fintech SRE’s Challenge: Smart Infrastructure Strategies for PCI DSS Compliance
maaaato
0
450
現場で役立つAPIデザイン
nagix
29
10k
5分で紹介する生成AIエージェントとAmazon Bedrock Agents / 5-minutes introduction to generative AI agents and Amazon Bedrock Agents
hideakiaoyagi
0
220
Tech Blogを書きやすい環境づくり
lycorptech_jp
PRO
0
120
Featured
See All Featured
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
193
16k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.6k
How to Think Like a Performance Engineer
csswizardry
22
1.3k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
160
15k
4 Signs Your Business is Dying
shpigford
182
22k
How to train your dragon (web standard)
notwaldorf
90
5.8k
Java REST API Framework Comparison - PWX 2021
mraible
28
8.4k
Writing Fast Ruby
sferik
628
61k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
10
1.3k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
44
9.4k
Mobile First: as difficult as doing things right
swwweet
223
9.3k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
40
2k
Transcript
Transcoding video streams from WebRTC MICHII Shunsuke pixiv Inc. SkyWay
UG Tokyo #2 2018-02-05
MICHII Shunsuke | @harukasan ImageFluxࣄۀ෦ ෦ ϐΫγϒςΫϊϩδʔζגࣜձࣾࣥߦһ ImageFluxͰGoΛॻ͍ͨΓΠϯϑϥपΓΛͬͨΓͱ͔͍Ζ͍Ζ
ࠓͳ͢͜ͱ • WebRTCͰૹΒΕͯ͘ΔH.264/AVCಈըετϦʔϜΛऔΓग़͢ํ๏ • H.264/AVCಈըετϦʔϜͲͷΑ͏ʹૹΒΕͯ͘Δͷ͔
ࠓͳ͞ͳ͍͜ͱ • WebRTCͱͳʹ͔ • WebRTCͰಈըΛετϦʔϛϯά͢Δํ๏ • WebRTC SFU Sora͕ͬͯ͘ΕΔ͜ͱ ଓཱ֬ɺηογϣϯͷཧɺSRTPͷ҉߸Խղআ……
• SkyWayͰͲ͏ͬͯΔͷ͔……ʢRTPΛऔΓग़ͤΕେৎʣ • H.264/AVCҎ֎ͷVP9, HEVC, AV1ͱ͍ͬͨಈըϑΥʔϚοτʹ͍ͭͯ
ͳͥWebRTCಈըΛτϥϯείʔυ͢Δͷ͔ • WebRTCಈըΛHLS৴͢ΔαʔϏεΛ։ൃதʂ https://speakerdeck.com/imageflux/imageflux-live • pixiv Sketch LiveͷόοΫΤϯυͱͯ͠Քಇத • αʔϏεʹڵຯ͋Δํੋඇ͓͕͚͍ͩ͘͞ʂ
ΤϯδχΞืू͍ͯ͠·͢ʂ
ImageFlux LiveͷΈ Transcoder Transcoder Transcoder RTP stream Streamerd Msgpack stream
TS TS TS ৴ऀ WebRTC SFU Sora WebRTC API H.264 bytestreamΛτϥϯείʔυ͢Δ RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ
H.264 bytestreamΛτϥϯείʔυ͢Δ ImageFlux LiveͷΈ Transcoder Transcoder Transcoder RTP stream Msgpack
stream TS TS TS ৴ऀ WebRTC SFU Sora WebRTC API RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ Streamerd
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client ηογϣϯཧ ϦΞϧλΠϜετϦʔϛϯά ଓཱ֬ʢNATӽ͑ʣ
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client ηογϣϯཧ ϦΞϧλΠϜετϦʔϛϯά ଓཱ֬ʢNATӽ͑ʣ
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client SDP SRTP ηογϣϯใ ಈըετϦʔϜ
WebRTCͰૹΒΕͯ͘ΔಈըΛ τϥϯείʔυ͢Δʹ 1. ΫϥΠΞϯτͱηογϣϯใʢSDPʣΛΓͱΓ͢Δ 2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ 3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ 4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ
WebRTCͰૹΒΕͯ͘ΔಈըΛ τϥϯείʔυ͢Δʹ 1. ΫϥΠΞϯτͱηογϣϯใʢSDPʣΛΓͱΓ͢Δ 2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ 3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ 4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ
1, 2WebRTC SFU Sora͕ͬͯ͘Ε·͢
RTP (Realtime Transfer Protocol) RFC3550 RTP: A Transport Protocol for
Real-Time Applications • ԻɺಈըͳͲͷσʔλΛϦΞϧλΠϜʹసૹ͢ΔͨΊͷϓϩτίϧ • UDP্ʹ࣮͞Ε͍ͯΔ • TCPΛ͍ͬͯͳ͍ͷͰɺॱং੍ޚ੍ޚΛRTPϨΠϠͰ࣮Ͱ͖Δ →ݴ͍͑ΕࣗͰ࣮͢Δඞཁ͕͋Δ • SRTPʢͬ͘͟Γݴ͑ʣRTPΛ҉߸Խͨ͠ͷ
࣮ମUDPͷബ͍ϥούʔ • UDPʹബ͍ϔομ͕͍͍ͭͯΔ͚ͩ • ϖΠϩʔυͷූ߸ԽํࣜϖΠϩʔυͷछྨ͝ͱʹҟͳΔ 0 1 2 3 4
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identifier contributing source (CSRC) identifier payload ….
ϖΠϩʔυͷผํ๏ • ϖΠϩʔυͷϑΥʔϚοτpayload typeͰผ͢Δ • payload type7bit͔͠ͳ͘ɺ͙͢ʹΓͳ͘ͳͬͨͷͰɺݱࡏͰ 96-127ΛಈతʹׂΓͯͯ͏ʢRFC3551ʣ • payload
typeͱϑΥʔϚοτͷϚοϐϯάSDPͷattributeͱͯ͠ ΓͱΓ͞ΕΔ • طʹొ͞Ε͍ͯΔͷIANAͰཧ͞Ε͍ͯΔ https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml
ॱংอূ • RTPUDPͰૹΒΕͯ͘ΔͷͰॱ൪อূ͞Ε͍ͯͳ͍ • RTPϔομʹγʔέϯε൪߸ʢsequence numberʣؚ͕·Ε͍ͯΔͷ Ͱ͜ΕΛͬͯιʔτ͢Δඞཁ͕͋Δ • ;ͭ͏ʹ300ݸ͙Β͍ޙΕͯύέοτ͕ඈΜͰ͖ͨΓͯ͠ਏ͍……͕ ͋·ΓͭͱԆͷݪҼʹͳΔͷͰέʔεόΠέʔε
• ͍·ͷ࣮Ͱύέοτ͕ඈΜͩ߹100ݸ͘Β͍ͭΑ͏ʹ͍ͯ͠Δ
ϑΥʔϚοτ͝ͱʹϖΠϩʔυͷܗࣜҟͳΔ • ͦΕͧΕҟͳΔRFCͰඪ४Խ͞Ε͍ͯΔ RFC6184 RTP Payload Format for H.264 Video
RFC7587 RTP Payload Format for the Opus Speech and Audio Codec draft-uberti-payload-vp9-01 RTP Payload Format for VP9 Video
RTP Payload Format for H.264 Video • H.264 NALU (NAL
Unitʣ1͕ͭ1ͭͷUDPύέοτʹؚ·ΕΔ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identifier contributing source (CSRC) identifier NAL Header Payload….
H.264/AVC NAL (Network Abstraction Layer) • H.264/AVCͰNALʢNetwork Abstraction Layerʣ͕ఆٛ͞Ε͍ͯΔ •
NALಈըετϦʔϜΛ͍ΖΜͳϝσΟΞͰૹ͍͢͠Α͏ʹநԽ͠ ͍ͯΔϨΠϠʔ • ؆୯ʹ͍͏ͱɺಈըετϦʔϜΛNAL UnitʢNALUʣͱݺΕΔͻͱ͔ͨ ·Γʹܗ͍ͯ͠Δ
H.264/AVC Bytestream Format (Annex B) • ΄ͱΜͲͷσίʔμH.264/AVC Bytestream Format (Annex
B)Λೖྗͱ ͯ͠͏͚͚ͭΔ • ISO/IEC 14496-10ͷAnnex Bʹࡌ͍ͬͯͨͷͰAnnex B formatͱݺΕΔ • جຊతʹNALUΛ00 00 00 01Ͱܨ͛ΕΑ͍ ͭ·ΓɺRTPͷϖΠϩʔυΛܨ͛Ε͍͍ͩͨΑͦ͞͏ 00 00 00 01 NALU 00 00 00 01 NALU 00 00 00 01 NALU …
ׂύέοτͱ݁߹ύέοτ • UDPͷϑϨʔϜαΠζҰൠతʹ1500byte͔͠ͳ͍ͷͰɺ݁߹ͨ͠Γɺ ׂͯ͠ෳͷύέοτʹೖΕͨΓ͢Δ͜ͱ͕͋Δ • ݁߹ύέοτͱɺׂύέοτNAL Unit typeͷ24-29Ͱఆٛ͞Ε͍ͯΔ NAL Unit
Packet Packet Type Name Section Type Type ------------------------------------------------------------- 1-23 NAL unit Single NAL unit packet 5.6 24 STAP-A Single-time aggregation packet 5.7.1 25 STAP-B Single-time aggregation packet 5.7.1 26 MTAP16 Multi-time aggregation packet 5.7.2 27 MTAP24 Multi-time aggregation packet 5.7.2 28 FU-A Fragmentation unit 5.8 29 FU-B Fragmentation unit 5.8 https://tools.ietf.org/html/rfc6184
ׂͨ͠Γ݁߹ͨ͠Γ͢Δ • ඇΠϯλʔϦʔϒϞʔυͰSTAP-AͱFU-A͕ΘΕΔͷͰ͜ͷ2ݸ͚ͩ ରԠ͢ΕΑ͍ • STAP-A: ෳͷNALUΛαΠζΛઌ಄ʹ͚ͭͯ݁߹ͨ͠ύέοτ • FU-A: ෳͷNALUΛׂ͠ɺ֤ύέοτʹFUϔομΛ͚ͭͨͷ
جຊతʹstart bit (s)͕͍͍ͭͯΔύέοτ͔Βend bit (e)͕͍͍ͭͯΔύ έοτ·Ͱ݁߹͢Δ 0 1 2 3 4 5 6 7 8 S E - Type FUϔομ
RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ 1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢ 2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ 3. ݁߹ύέοτͩͬͨ߹ׂͯ͠ෳͷόΠτετϦʔϜʹ͢Δ
4. ׂύέοτͩͬͨ߹શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬ ͯ1ͭͷNALUΛऔΓग़͢
SDP (Session Description Protocol) RFC4566 SDP: Session Description Protocol •
ηογϣϯใΛදݱ͢ΔͨΊͷϓϩτίϧ • WebRTCΛ͡ΊɺʢWebRTCͷϕʔεʹͳͬͨʣSIPɺRTSPɺϝʔϧͳͲ ͘ΘΕ͍ͯΔ • ಡΈͮΒ͍͜ͱͰ༗໊ʁ
ಡΈͮΒ͍ʁ v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar
i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf
[email protected]
(Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000 https://tools.ietf.org/html/rfc4566
ಡΈͮΒ͍ʁ v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar
i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf
[email protected]
(Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000 https://tools.ietf.org/html/rfc4566
Media Description "m=" m=<media> <port> <proto> <proto> <fmt> … <media>:
ϝσΟΞͷछྨʢaudio, video,…) <port>: ૹ৴ݩϙʔτ <proto>: ϓϩτίϧ <fmt>: ϑΥʔϚοτใ ϝσΟΞͷछྨͱϑΥʔϚοτɺૹ৴ݩϙʔτؚ͕·Ε͍ͯΔ ͋·Γ͑Δใͳ͍
Attributes "a=" a=<attribute>:<value> <attribute>: ଐੑ <value>: ͦͷ • SDPʹΛ֦ு͢ΔͨΊʹ༻ҙ͞Ε͍ͯΔύϥϝʔλ •
WebRTCͰ֦ு͞Ε͍ͯΔ͍͍ͩͨa=ʹೖ͍ͬͯΔ
SDP͔ΒRTPͷpayload type mappingΛऔΓग़͢ a=rtpmap:109 opus/48000/2 a=rtpmap:120 H264/90000 • payload typeͷmappingrtpmapͰࢦఆ͞ΕΔ
• ্ͷྫͩͱɺH.264ͷpayload type120ɺOpus109 • 2൪ͷϑΥʔϚοτ/प/νϟϯωϧ H.264/AVCͩͱ90000KHzͰλΠϜελϯϓ͕ͬͯ͘Δ͜ͱ͕Θ͔Δ
RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ 1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢ 2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ 3. ݁߹ύέοτͩͬͨ߹ׂͯ͠ෳͷόΠτετϦʔϜʹ͢Δ
4. ׂύέοτͩͬͨ߹શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬ ͯ1ͭͷNALUΛऔΓग़͢ ෮श
·ͱΊ • WebRTCಈըετϦʔϜΛτϥϯείʔυ͢Δʹɺ RTPύέοτΛσίʔυͰ͖ΕΑ͍ • αʔόαΠυͰRTPΛऔΓग़͢ʹWebRTC SFU Sora͕ศར • ImageFlux
Live։ൃதͰ͢ʂ