Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
WebRTC動画をトランスコードする / Transcoding video streams ...
Search
Harukasan
PRO
February 05, 2018
Technology
5
1.6k
WebRTC動画をトランスコードする / Transcoding video streams from WebRTC
SkyWay UG Tokyo #2
2018-02-05
Harukasan
PRO
February 05, 2018
Tweet
Share
More Decks by Harukasan
See All by Harukasan
Write your own mrbgem, Create your own device
harukasan
PRO
1
82
PicoRabbit: a Tiny Presentation Device Powered by Ruby
harukasan
PRO
2
420
pixivを支える技術 / 技育CAMPアカデミア
harukasan
PRO
3
510
20240401 新卒研修 - ピクシブにおける技術領域
harukasan
PRO
1
830
ピクシブのコンテンツ配信基盤技術 / pixiv TECH SALON
harukasan
PRO
5
5.7k
Goにおける画像ファイル処理 / golang.tokyo #19
harukasan
PRO
7
6.7k
ImageFluxを支えるリモート開発 / 20171202
harukasan
PRO
2
1.8k
YAPC::Fukuoka 前夜祭LT / Yet Another Pawoo Commit logs
harukasan
PRO
0
3k
YAPC::Fukuoka lunch session
harukasan
PRO
1
3.1k
Other Decks in Technology
See All in Technology
Perlアプリケーションで トレースを実装するまでの 工夫と苦労話
masayoshi
1
410
20250728 MCP, A2A and Multi-Agents in the future
yoshidashingo
1
210
風が吹けばWHOISが使えなくなる~なぜWHOIS・RDAPはサーバー証明書のメール認証に使えなくなったのか~
orangemorishita
15
5.5k
隙間時間で爆速開発! Claude Code × Vibe Coding で作るマニュアル自動生成サービス
akitomonam
3
250
生成AI時代におけるAI・機械学習技術を用いたプロダクト開発の深化と進化 #BetAIDay
layerx
PRO
1
1k
GMOペパボのデータ基盤とデータ活用の現在地 / Current State of GMO Pepabo's Data Infrastructure and Data Utilization
zaimy
3
200
ホリスティックテスティングの右側も大切にする 〜2つの[はか]る〜 / Holistic Testing: Right Side Matters
nihonbuson
PRO
0
580
帳票構造化タスクにおけるLLMファインチューニングの性能評価
yosukeyoshida
1
230
リリース2ヶ月で収益化した話
kent_code3
1
180
オブザーバビリティプラットフォーム開発におけるオブザーバビリティとの向き合い / Hatena Engineer Seminar #34 オブザーバビリティの実現と運用編
arthur1
0
340
みんなのSRE 〜チーム全員でのSRE活動にするための4つの取り組み〜
kakehashi
PRO
2
140
【CEDEC2025】大規模言語モデルを活用したゲーム内会話パートのスクリプト作成支援への取り組み
cygames
PRO
2
770
Featured
See All Featured
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
110
19k
The World Runs on Bad Software
bkeepers
PRO
70
11k
Building Adaptive Systems
keathley
43
2.7k
Designing for humans not robots
tammielis
253
25k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
The Pragmatic Product Professional
lauravandoore
36
6.8k
How GitHub (no longer) Works
holman
314
140k
Practical Orchestrator
shlominoach
190
11k
Done Done
chrislema
185
16k
Making the Leap to Tech Lead
cromwellryan
134
9.5k
A Tale of Four Properties
chriscoyier
160
23k
Reflections from 52 weeks, 52 projects
jeffersonlam
351
21k
Transcript
Transcoding video streams from WebRTC MICHII Shunsuke pixiv Inc. SkyWay
UG Tokyo #2 2018-02-05
MICHII Shunsuke | @harukasan ImageFluxࣄۀ෦ ෦ ϐΫγϒςΫϊϩδʔζגࣜձࣾࣥߦһ ImageFluxͰGoΛॻ͍ͨΓΠϯϑϥपΓΛͬͨΓͱ͔͍Ζ͍Ζ
ࠓͳ͢͜ͱ • WebRTCͰૹΒΕͯ͘ΔH.264/AVCಈըετϦʔϜΛऔΓग़͢ํ๏ • H.264/AVCಈըετϦʔϜͲͷΑ͏ʹૹΒΕͯ͘Δͷ͔
ࠓͳ͞ͳ͍͜ͱ • WebRTCͱͳʹ͔ • WebRTCͰಈըΛετϦʔϛϯά͢Δํ๏ • WebRTC SFU Sora͕ͬͯ͘ΕΔ͜ͱ ଓཱ֬ɺηογϣϯͷཧɺSRTPͷ҉߸Խղআ……
• SkyWayͰͲ͏ͬͯΔͷ͔……ʢRTPΛऔΓग़ͤΕେৎʣ • H.264/AVCҎ֎ͷVP9, HEVC, AV1ͱ͍ͬͨಈըϑΥʔϚοτʹ͍ͭͯ
ͳͥWebRTCಈըΛτϥϯείʔυ͢Δͷ͔ • WebRTCಈըΛHLS৴͢ΔαʔϏεΛ։ൃதʂ https://speakerdeck.com/imageflux/imageflux-live • pixiv Sketch LiveͷόοΫΤϯυͱͯ͠Քಇத • αʔϏεʹڵຯ͋Δํੋඇ͓͕͚͍ͩ͘͞ʂ
ΤϯδχΞืू͍ͯ͠·͢ʂ
ImageFlux LiveͷΈ Transcoder Transcoder Transcoder RTP stream Streamerd Msgpack stream
TS TS TS ৴ऀ WebRTC SFU Sora WebRTC API H.264 bytestreamΛτϥϯείʔυ͢Δ RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ
H.264 bytestreamΛτϥϯείʔυ͢Δ ImageFlux LiveͷΈ Transcoder Transcoder Transcoder RTP stream Msgpack
stream TS TS TS ৴ऀ WebRTC SFU Sora WebRTC API RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ Streamerd
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client ηογϣϯཧ ϦΞϧλΠϜετϦʔϛϯά ଓཱ֬ʢNATӽ͑ʣ
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client ηογϣϯཧ ϦΞϧλΠϜετϦʔϛϯά ଓཱ֬ʢNATӽ͑ʣ
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client SDP SRTP ηογϣϯใ ಈըετϦʔϜ
WebRTCͰૹΒΕͯ͘ΔಈըΛ τϥϯείʔυ͢Δʹ 1. ΫϥΠΞϯτͱηογϣϯใʢSDPʣΛΓͱΓ͢Δ 2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ 3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ 4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ
WebRTCͰૹΒΕͯ͘ΔಈըΛ τϥϯείʔυ͢Δʹ 1. ΫϥΠΞϯτͱηογϣϯใʢSDPʣΛΓͱΓ͢Δ 2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ 3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ 4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ
1, 2WebRTC SFU Sora͕ͬͯ͘Ε·͢
RTP (Realtime Transfer Protocol) RFC3550 RTP: A Transport Protocol for
Real-Time Applications • ԻɺಈըͳͲͷσʔλΛϦΞϧλΠϜʹసૹ͢ΔͨΊͷϓϩτίϧ • UDP্ʹ࣮͞Ε͍ͯΔ • TCPΛ͍ͬͯͳ͍ͷͰɺॱং੍ޚ੍ޚΛRTPϨΠϠͰ࣮Ͱ͖Δ →ݴ͍͑ΕࣗͰ࣮͢Δඞཁ͕͋Δ • SRTPʢͬ͘͟Γݴ͑ʣRTPΛ҉߸Խͨ͠ͷ
࣮ମUDPͷബ͍ϥούʔ • UDPʹബ͍ϔομ͕͍͍ͭͯΔ͚ͩ • ϖΠϩʔυͷූ߸ԽํࣜϖΠϩʔυͷछྨ͝ͱʹҟͳΔ 0 1 2 3 4
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identifier contributing source (CSRC) identifier payload ….
ϖΠϩʔυͷผํ๏ • ϖΠϩʔυͷϑΥʔϚοτpayload typeͰผ͢Δ • payload type7bit͔͠ͳ͘ɺ͙͢ʹΓͳ͘ͳͬͨͷͰɺݱࡏͰ 96-127ΛಈతʹׂΓͯͯ͏ʢRFC3551ʣ • payload
typeͱϑΥʔϚοτͷϚοϐϯάSDPͷattributeͱͯ͠ ΓͱΓ͞ΕΔ • طʹొ͞Ε͍ͯΔͷIANAͰཧ͞Ε͍ͯΔ https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml
ॱংอূ • RTPUDPͰૹΒΕͯ͘ΔͷͰॱ൪อূ͞Ε͍ͯͳ͍ • RTPϔομʹγʔέϯε൪߸ʢsequence numberʣؚ͕·Ε͍ͯΔͷ Ͱ͜ΕΛͬͯιʔτ͢Δඞཁ͕͋Δ • ;ͭ͏ʹ300ݸ͙Β͍ޙΕͯύέοτ͕ඈΜͰ͖ͨΓͯ͠ਏ͍……͕ ͋·ΓͭͱԆͷݪҼʹͳΔͷͰέʔεόΠέʔε
• ͍·ͷ࣮Ͱύέοτ͕ඈΜͩ߹100ݸ͘Β͍ͭΑ͏ʹ͍ͯ͠Δ
ϑΥʔϚοτ͝ͱʹϖΠϩʔυͷܗࣜҟͳΔ • ͦΕͧΕҟͳΔRFCͰඪ४Խ͞Ε͍ͯΔ RFC6184 RTP Payload Format for H.264 Video
RFC7587 RTP Payload Format for the Opus Speech and Audio Codec draft-uberti-payload-vp9-01 RTP Payload Format for VP9 Video
RTP Payload Format for H.264 Video • H.264 NALU (NAL
Unitʣ1͕ͭ1ͭͷUDPύέοτʹؚ·ΕΔ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identifier contributing source (CSRC) identifier NAL Header Payload….
H.264/AVC NAL (Network Abstraction Layer) • H.264/AVCͰNALʢNetwork Abstraction Layerʣ͕ఆٛ͞Ε͍ͯΔ •
NALಈըετϦʔϜΛ͍ΖΜͳϝσΟΞͰૹ͍͢͠Α͏ʹநԽ͠ ͍ͯΔϨΠϠʔ • ؆୯ʹ͍͏ͱɺಈըετϦʔϜΛNAL UnitʢNALUʣͱݺΕΔͻͱ͔ͨ ·Γʹܗ͍ͯ͠Δ
H.264/AVC Bytestream Format (Annex B) • ΄ͱΜͲͷσίʔμH.264/AVC Bytestream Format (Annex
B)Λೖྗͱ ͯ͠͏͚͚ͭΔ • ISO/IEC 14496-10ͷAnnex Bʹࡌ͍ͬͯͨͷͰAnnex B formatͱݺΕΔ • جຊతʹNALUΛ00 00 00 01Ͱܨ͛ΕΑ͍ ͭ·ΓɺRTPͷϖΠϩʔυΛܨ͛Ε͍͍ͩͨΑͦ͞͏ 00 00 00 01 NALU 00 00 00 01 NALU 00 00 00 01 NALU …
ׂύέοτͱ݁߹ύέοτ • UDPͷϑϨʔϜαΠζҰൠతʹ1500byte͔͠ͳ͍ͷͰɺ݁߹ͨ͠Γɺ ׂͯ͠ෳͷύέοτʹೖΕͨΓ͢Δ͜ͱ͕͋Δ • ݁߹ύέοτͱɺׂύέοτNAL Unit typeͷ24-29Ͱఆٛ͞Ε͍ͯΔ NAL Unit
Packet Packet Type Name Section Type Type ------------------------------------------------------------- 1-23 NAL unit Single NAL unit packet 5.6 24 STAP-A Single-time aggregation packet 5.7.1 25 STAP-B Single-time aggregation packet 5.7.1 26 MTAP16 Multi-time aggregation packet 5.7.2 27 MTAP24 Multi-time aggregation packet 5.7.2 28 FU-A Fragmentation unit 5.8 29 FU-B Fragmentation unit 5.8 https://tools.ietf.org/html/rfc6184
ׂͨ͠Γ݁߹ͨ͠Γ͢Δ • ඇΠϯλʔϦʔϒϞʔυͰSTAP-AͱFU-A͕ΘΕΔͷͰ͜ͷ2ݸ͚ͩ ରԠ͢ΕΑ͍ • STAP-A: ෳͷNALUΛαΠζΛઌ಄ʹ͚ͭͯ݁߹ͨ͠ύέοτ • FU-A: ෳͷNALUΛׂ͠ɺ֤ύέοτʹFUϔομΛ͚ͭͨͷ
جຊతʹstart bit (s)͕͍͍ͭͯΔύέοτ͔Βend bit (e)͕͍͍ͭͯΔύ έοτ·Ͱ݁߹͢Δ 0 1 2 3 4 5 6 7 8 S E - Type FUϔομ
RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ 1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢ 2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ 3. ݁߹ύέοτͩͬͨ߹ׂͯ͠ෳͷόΠτετϦʔϜʹ͢Δ
4. ׂύέοτͩͬͨ߹શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬ ͯ1ͭͷNALUΛऔΓग़͢
SDP (Session Description Protocol) RFC4566 SDP: Session Description Protocol •
ηογϣϯใΛදݱ͢ΔͨΊͷϓϩτίϧ • WebRTCΛ͡ΊɺʢWebRTCͷϕʔεʹͳͬͨʣSIPɺRTSPɺϝʔϧͳͲ ͘ΘΕ͍ͯΔ • ಡΈͮΒ͍͜ͱͰ༗໊ʁ
ಡΈͮΒ͍ʁ v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar
i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf
[email protected]
(Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000 https://tools.ietf.org/html/rfc4566
ಡΈͮΒ͍ʁ v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar
i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf
[email protected]
(Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000 https://tools.ietf.org/html/rfc4566
Media Description "m=" m=<media> <port> <proto> <proto> <fmt> … <media>:
ϝσΟΞͷछྨʢaudio, video,…) <port>: ૹ৴ݩϙʔτ <proto>: ϓϩτίϧ <fmt>: ϑΥʔϚοτใ ϝσΟΞͷछྨͱϑΥʔϚοτɺૹ৴ݩϙʔτؚ͕·Ε͍ͯΔ ͋·Γ͑Δใͳ͍
Attributes "a=" a=<attribute>:<value> <attribute>: ଐੑ <value>: ͦͷ • SDPʹΛ֦ு͢ΔͨΊʹ༻ҙ͞Ε͍ͯΔύϥϝʔλ •
WebRTCͰ֦ு͞Ε͍ͯΔ͍͍ͩͨa=ʹೖ͍ͬͯΔ
SDP͔ΒRTPͷpayload type mappingΛऔΓग़͢ a=rtpmap:109 opus/48000/2 a=rtpmap:120 H264/90000 • payload typeͷmappingrtpmapͰࢦఆ͞ΕΔ
• ্ͷྫͩͱɺH.264ͷpayload type120ɺOpus109 • 2൪ͷϑΥʔϚοτ/प/νϟϯωϧ H.264/AVCͩͱ90000KHzͰλΠϜελϯϓ͕ͬͯ͘Δ͜ͱ͕Θ͔Δ
RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ 1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢ 2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ 3. ݁߹ύέοτͩͬͨ߹ׂͯ͠ෳͷόΠτετϦʔϜʹ͢Δ
4. ׂύέοτͩͬͨ߹શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬ ͯ1ͭͷNALUΛऔΓग़͢ ෮श
·ͱΊ • WebRTCಈըετϦʔϜΛτϥϯείʔυ͢Δʹɺ RTPύέοτΛσίʔυͰ͖ΕΑ͍ • αʔόαΠυͰRTPΛऔΓग़͢ʹWebRTC SFU Sora͕ศར • ImageFlux
Live։ൃதͰ͢ʂ