Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
WebRTC動画をトランスコードする / Transcoding video streams ...
Search
Harukasan
PRO
February 05, 2018
Technology
5
1.6k
WebRTC動画をトランスコードする / Transcoding video streams from WebRTC
SkyWay UG Tokyo #2
2018-02-05
Harukasan
PRO
February 05, 2018
Tweet
Share
More Decks by Harukasan
See All by Harukasan
PicoRabbit: a Tiny Presentation Device Powered by Ruby
harukasan
PRO
2
300
pixivを支える技術 / 技育CAMPアカデミア
harukasan
PRO
3
470
20240401 新卒研修 - ピクシブにおける技術領域
harukasan
PRO
1
790
ピクシブのコンテンツ配信基盤技術 / pixiv TECH SALON
harukasan
PRO
5
5.6k
Goにおける画像ファイル処理 / golang.tokyo #19
harukasan
PRO
7
6.6k
ImageFluxを支えるリモート開発 / 20171202
harukasan
PRO
2
1.8k
YAPC::Fukuoka 前夜祭LT / Yet Another Pawoo Commit logs
harukasan
PRO
0
3k
YAPC::Fukuoka lunch session
harukasan
PRO
1
3.1k
マストドン会議: Pawoo / Mastodon Kaigi2
harukasan
PRO
2
490
Other Decks in Technology
See All in Technology
テストって楽しい!開発を加速させるテストの魅力 / Testing is Fun! The Fascinating of Testing to Accelerate Development
aiandrox
0
160
Oracle Cloud Infrastructure:2025年4月度サービス・アップデート
oracle4engineer
PRO
0
380
Microsoft の SSE の現在地
skmkzyk
0
300
Part1 GitHubってなんだろう?その1
tomokusaba
2
540
3D生成AIのための画像生成
kosukeito
2
600
LangfuseではじめるAIアプリのLLMトレーシング
codenote
0
100
データベース04: SQL (1/3) 単純質問 & 集約演算
trycycle
PRO
0
720
AIにおけるソフトウェアテスト_ver1.00
fumisuke
1
360
genspark_presentation.pdf
haruki_uiru
1
210
Compose におけるパスワード自動入力とパスワード保存
tonionagauzzi
0
200
2025-04-24 "Manga AI Understanding & Localization" Furukawa Arata (CyberAgent, Inc)
ornew
2
340
Aspire をカスタマイズしよう & Aspire 9.2
nenonaninu
0
380
Featured
See All Featured
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.6k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.8k
Visualization
eitanlees
146
16k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
5
560
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
23
2.7k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
Thoughts on Productivity
jonyablonski
69
4.6k
Making the Leap to Tech Lead
cromwellryan
133
9.3k
The Cult of Friendly URLs
andyhume
78
6.3k
Reflections from 52 weeks, 52 projects
jeffersonlam
349
20k
Build your cross-platform service in a week with App Engine
jlugia
230
18k
Transcript
Transcoding video streams from WebRTC MICHII Shunsuke pixiv Inc. SkyWay
UG Tokyo #2 2018-02-05
MICHII Shunsuke | @harukasan ImageFluxࣄۀ෦ ෦ ϐΫγϒςΫϊϩδʔζגࣜձࣾࣥߦһ ImageFluxͰGoΛॻ͍ͨΓΠϯϑϥपΓΛͬͨΓͱ͔͍Ζ͍Ζ
ࠓͳ͢͜ͱ • WebRTCͰૹΒΕͯ͘ΔH.264/AVCಈըετϦʔϜΛऔΓग़͢ํ๏ • H.264/AVCಈըετϦʔϜͲͷΑ͏ʹૹΒΕͯ͘Δͷ͔
ࠓͳ͞ͳ͍͜ͱ • WebRTCͱͳʹ͔ • WebRTCͰಈըΛετϦʔϛϯά͢Δํ๏ • WebRTC SFU Sora͕ͬͯ͘ΕΔ͜ͱ ଓཱ֬ɺηογϣϯͷཧɺSRTPͷ҉߸Խղআ……
• SkyWayͰͲ͏ͬͯΔͷ͔……ʢRTPΛऔΓग़ͤΕେৎʣ • H.264/AVCҎ֎ͷVP9, HEVC, AV1ͱ͍ͬͨಈըϑΥʔϚοτʹ͍ͭͯ
ͳͥWebRTCಈըΛτϥϯείʔυ͢Δͷ͔ • WebRTCಈըΛHLS৴͢ΔαʔϏεΛ։ൃதʂ https://speakerdeck.com/imageflux/imageflux-live • pixiv Sketch LiveͷόοΫΤϯυͱͯ͠Քಇத • αʔϏεʹڵຯ͋Δํੋඇ͓͕͚͍ͩ͘͞ʂ
ΤϯδχΞืू͍ͯ͠·͢ʂ
ImageFlux LiveͷΈ Transcoder Transcoder Transcoder RTP stream Streamerd Msgpack stream
TS TS TS ৴ऀ WebRTC SFU Sora WebRTC API H.264 bytestreamΛτϥϯείʔυ͢Δ RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ
H.264 bytestreamΛτϥϯείʔυ͢Δ ImageFlux LiveͷΈ Transcoder Transcoder Transcoder RTP stream Msgpack
stream TS TS TS ৴ऀ WebRTC SFU Sora WebRTC API RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ Streamerd
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client ηογϣϯཧ ϦΞϧλΠϜετϦʔϛϯά ଓཱ֬ʢNATӽ͑ʣ
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client ηογϣϯཧ ϦΞϧλΠϜετϦʔϛϯά ଓཱ֬ʢNATӽ͑ʣ
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client SDP SRTP ηογϣϯใ ಈըετϦʔϜ
WebRTCͰૹΒΕͯ͘ΔಈըΛ τϥϯείʔυ͢Δʹ 1. ΫϥΠΞϯτͱηογϣϯใʢSDPʣΛΓͱΓ͢Δ 2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ 3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ 4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ
WebRTCͰૹΒΕͯ͘ΔಈըΛ τϥϯείʔυ͢Δʹ 1. ΫϥΠΞϯτͱηογϣϯใʢSDPʣΛΓͱΓ͢Δ 2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ 3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ 4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ
1, 2WebRTC SFU Sora͕ͬͯ͘Ε·͢
RTP (Realtime Transfer Protocol) RFC3550 RTP: A Transport Protocol for
Real-Time Applications • ԻɺಈըͳͲͷσʔλΛϦΞϧλΠϜʹసૹ͢ΔͨΊͷϓϩτίϧ • UDP্ʹ࣮͞Ε͍ͯΔ • TCPΛ͍ͬͯͳ͍ͷͰɺॱং੍ޚ੍ޚΛRTPϨΠϠͰ࣮Ͱ͖Δ →ݴ͍͑ΕࣗͰ࣮͢Δඞཁ͕͋Δ • SRTPʢͬ͘͟Γݴ͑ʣRTPΛ҉߸Խͨ͠ͷ
࣮ମUDPͷബ͍ϥούʔ • UDPʹബ͍ϔομ͕͍͍ͭͯΔ͚ͩ • ϖΠϩʔυͷූ߸ԽํࣜϖΠϩʔυͷछྨ͝ͱʹҟͳΔ 0 1 2 3 4
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identifier contributing source (CSRC) identifier payload ….
ϖΠϩʔυͷผํ๏ • ϖΠϩʔυͷϑΥʔϚοτpayload typeͰผ͢Δ • payload type7bit͔͠ͳ͘ɺ͙͢ʹΓͳ͘ͳͬͨͷͰɺݱࡏͰ 96-127ΛಈతʹׂΓͯͯ͏ʢRFC3551ʣ • payload
typeͱϑΥʔϚοτͷϚοϐϯάSDPͷattributeͱͯ͠ ΓͱΓ͞ΕΔ • طʹొ͞Ε͍ͯΔͷIANAͰཧ͞Ε͍ͯΔ https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml
ॱংอূ • RTPUDPͰૹΒΕͯ͘ΔͷͰॱ൪อূ͞Ε͍ͯͳ͍ • RTPϔομʹγʔέϯε൪߸ʢsequence numberʣؚ͕·Ε͍ͯΔͷ Ͱ͜ΕΛͬͯιʔτ͢Δඞཁ͕͋Δ • ;ͭ͏ʹ300ݸ͙Β͍ޙΕͯύέοτ͕ඈΜͰ͖ͨΓͯ͠ਏ͍……͕ ͋·ΓͭͱԆͷݪҼʹͳΔͷͰέʔεόΠέʔε
• ͍·ͷ࣮Ͱύέοτ͕ඈΜͩ߹100ݸ͘Β͍ͭΑ͏ʹ͍ͯ͠Δ
ϑΥʔϚοτ͝ͱʹϖΠϩʔυͷܗࣜҟͳΔ • ͦΕͧΕҟͳΔRFCͰඪ४Խ͞Ε͍ͯΔ RFC6184 RTP Payload Format for H.264 Video
RFC7587 RTP Payload Format for the Opus Speech and Audio Codec draft-uberti-payload-vp9-01 RTP Payload Format for VP9 Video
RTP Payload Format for H.264 Video • H.264 NALU (NAL
Unitʣ1͕ͭ1ͭͷUDPύέοτʹؚ·ΕΔ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identifier contributing source (CSRC) identifier NAL Header Payload….
H.264/AVC NAL (Network Abstraction Layer) • H.264/AVCͰNALʢNetwork Abstraction Layerʣ͕ఆٛ͞Ε͍ͯΔ •
NALಈըετϦʔϜΛ͍ΖΜͳϝσΟΞͰૹ͍͢͠Α͏ʹநԽ͠ ͍ͯΔϨΠϠʔ • ؆୯ʹ͍͏ͱɺಈըετϦʔϜΛNAL UnitʢNALUʣͱݺΕΔͻͱ͔ͨ ·Γʹܗ͍ͯ͠Δ
H.264/AVC Bytestream Format (Annex B) • ΄ͱΜͲͷσίʔμH.264/AVC Bytestream Format (Annex
B)Λೖྗͱ ͯ͠͏͚͚ͭΔ • ISO/IEC 14496-10ͷAnnex Bʹࡌ͍ͬͯͨͷͰAnnex B formatͱݺΕΔ • جຊతʹNALUΛ00 00 00 01Ͱܨ͛ΕΑ͍ ͭ·ΓɺRTPͷϖΠϩʔυΛܨ͛Ε͍͍ͩͨΑͦ͞͏ 00 00 00 01 NALU 00 00 00 01 NALU 00 00 00 01 NALU …
ׂύέοτͱ݁߹ύέοτ • UDPͷϑϨʔϜαΠζҰൠతʹ1500byte͔͠ͳ͍ͷͰɺ݁߹ͨ͠Γɺ ׂͯ͠ෳͷύέοτʹೖΕͨΓ͢Δ͜ͱ͕͋Δ • ݁߹ύέοτͱɺׂύέοτNAL Unit typeͷ24-29Ͱఆٛ͞Ε͍ͯΔ NAL Unit
Packet Packet Type Name Section Type Type ------------------------------------------------------------- 1-23 NAL unit Single NAL unit packet 5.6 24 STAP-A Single-time aggregation packet 5.7.1 25 STAP-B Single-time aggregation packet 5.7.1 26 MTAP16 Multi-time aggregation packet 5.7.2 27 MTAP24 Multi-time aggregation packet 5.7.2 28 FU-A Fragmentation unit 5.8 29 FU-B Fragmentation unit 5.8 https://tools.ietf.org/html/rfc6184
ׂͨ͠Γ݁߹ͨ͠Γ͢Δ • ඇΠϯλʔϦʔϒϞʔυͰSTAP-AͱFU-A͕ΘΕΔͷͰ͜ͷ2ݸ͚ͩ ରԠ͢ΕΑ͍ • STAP-A: ෳͷNALUΛαΠζΛઌ಄ʹ͚ͭͯ݁߹ͨ͠ύέοτ • FU-A: ෳͷNALUΛׂ͠ɺ֤ύέοτʹFUϔομΛ͚ͭͨͷ
جຊతʹstart bit (s)͕͍͍ͭͯΔύέοτ͔Βend bit (e)͕͍͍ͭͯΔύ έοτ·Ͱ݁߹͢Δ 0 1 2 3 4 5 6 7 8 S E - Type FUϔομ
RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ 1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢ 2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ 3. ݁߹ύέοτͩͬͨ߹ׂͯ͠ෳͷόΠτετϦʔϜʹ͢Δ
4. ׂύέοτͩͬͨ߹શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬ ͯ1ͭͷNALUΛऔΓग़͢
SDP (Session Description Protocol) RFC4566 SDP: Session Description Protocol •
ηογϣϯใΛදݱ͢ΔͨΊͷϓϩτίϧ • WebRTCΛ͡ΊɺʢWebRTCͷϕʔεʹͳͬͨʣSIPɺRTSPɺϝʔϧͳͲ ͘ΘΕ͍ͯΔ • ಡΈͮΒ͍͜ͱͰ༗໊ʁ
ಡΈͮΒ͍ʁ v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar
i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf e=j.doe@example.com (Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000 https://tools.ietf.org/html/rfc4566
ಡΈͮΒ͍ʁ v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar
i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf e=j.doe@example.com (Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000 https://tools.ietf.org/html/rfc4566
Media Description "m=" m=<media> <port> <proto> <proto> <fmt> … <media>:
ϝσΟΞͷछྨʢaudio, video,…) <port>: ૹ৴ݩϙʔτ <proto>: ϓϩτίϧ <fmt>: ϑΥʔϚοτใ ϝσΟΞͷछྨͱϑΥʔϚοτɺૹ৴ݩϙʔτؚ͕·Ε͍ͯΔ ͋·Γ͑Δใͳ͍
Attributes "a=" a=<attribute>:<value> <attribute>: ଐੑ <value>: ͦͷ • SDPʹΛ֦ு͢ΔͨΊʹ༻ҙ͞Ε͍ͯΔύϥϝʔλ •
WebRTCͰ֦ு͞Ε͍ͯΔ͍͍ͩͨa=ʹೖ͍ͬͯΔ
SDP͔ΒRTPͷpayload type mappingΛऔΓग़͢ a=rtpmap:109 opus/48000/2 a=rtpmap:120 H264/90000 • payload typeͷmappingrtpmapͰࢦఆ͞ΕΔ
• ্ͷྫͩͱɺH.264ͷpayload type120ɺOpus109 • 2൪ͷϑΥʔϚοτ/प/νϟϯωϧ H.264/AVCͩͱ90000KHzͰλΠϜελϯϓ͕ͬͯ͘Δ͜ͱ͕Θ͔Δ
RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ 1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢ 2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ 3. ݁߹ύέοτͩͬͨ߹ׂͯ͠ෳͷόΠτετϦʔϜʹ͢Δ
4. ׂύέοτͩͬͨ߹શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬ ͯ1ͭͷNALUΛऔΓग़͢ ෮श
·ͱΊ • WebRTCಈըετϦʔϜΛτϥϯείʔυ͢Δʹɺ RTPύέοτΛσίʔυͰ͖ΕΑ͍ • αʔόαΠυͰRTPΛऔΓग़͢ʹWebRTC SFU Sora͕ศར • ImageFlux
Live։ൃதͰ͢ʂ