Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
WebRTC動画をトランスコードする / Transcoding video streams from WebRTC
Search
Harukasan
PRO
February 05, 2018
Technology
5
1.4k
WebRTC動画をトランスコードする / Transcoding video streams from WebRTC
SkyWay UG Tokyo #2
2018-02-05
Harukasan
PRO
February 05, 2018
Tweet
Share
More Decks by Harukasan
See All by Harukasan
20240401 新卒研修 - ピクシブにおける技術領域
harukasan
PRO
1
530
ピクシブのコンテンツ配信基盤技術 / pixiv TECH SALON
harukasan
PRO
5
5.1k
Goにおける画像ファイル処理 / golang.tokyo #19
harukasan
PRO
7
6.2k
ImageFluxを支えるリモート開発 / 20171202
harukasan
PRO
2
1.7k
YAPC::Fukuoka 前夜祭LT / Yet Another Pawoo Commit logs
harukasan
PRO
0
2.8k
YAPC::Fukuoka lunch session
harukasan
PRO
1
2.8k
マストドン会議: Pawoo / Mastodon Kaigi2
harukasan
PRO
2
410
大規模Mastodonインスタンスを運用するコツ / Inside Pawoo Mastodon infrastructure
harukasan
PRO
0
2.9k
JPEG streaming in Go
harukasan
PRO
2
2.4k
Other Decks in Technology
See All in Technology
IaCジェネレーターとBedrockで詳細設計書を生成してみた
tsukasa_ishimaru
4
750
「スニダン」開発組織の構造に込めた意図 ~組織作りはパッションや政治ではない!~
rinchsan
4
590
web-application-security
matsuihidetoshi
1
180
Cypress or Playwright?
rainerhahnekamp
0
150
ワールドカフェI /チューターを改良する / World Café I and Improving the Tutors
ks91
PRO
0
140
障害対応をちょっとずつよくしていくための 演習の作りかた
heleeen
1
340
EM完全に理解した と思ったけど、 やっぱり何も分からなかった話 / EM Night Fukuoka #1
hirutas
0
120
Delivering Millions of Messages within seconds @ Duolingo
pelelgrino
0
360
ゼロから始めるVue.jsコミュニティ貢献 / first-vuejs-community-contribution-link-and-motivation
lmi
1
140
今年のRubyKaigiはProfiler Year🤘
osyoyu
0
210
本当のAWS基礎
toru_kubota
1
550
【SORACOM UG 東海】あらゆるモノがつながる社会へ、IoT と SORACOM
soracom
PRO
1
110
Featured
See All Featured
GitHub's CSS Performance
jonrohan
1025
450k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
62
14k
WebSockets: Embracing the real-time Web
robhawkes
59
7k
Build your cross-platform service in a week with App Engine
jlugia
226
17k
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3.1k
Building a Scalable Design System with Sketch
lauravandoore
457
32k
Facilitating Awesome Meetings
lara
43
5.6k
How to name files
jennybc
65
93k
Mobile First: as difficult as doing things right
swwweet
217
8.6k
In The Pink: A Labor of Love
frogandcode
138
21k
Automating Front-end Workflow
addyosmani
1357
200k
Web development in the modern age
philhawksworth
203
10k
Transcript
Transcoding video streams from WebRTC MICHII Shunsuke pixiv Inc. SkyWay
UG Tokyo #2 2018-02-05
MICHII Shunsuke | @harukasan ImageFluxࣄۀ෦ ෦ ϐΫγϒςΫϊϩδʔζגࣜձࣾࣥߦһ ImageFluxͰGoΛॻ͍ͨΓΠϯϑϥपΓΛͬͨΓͱ͔͍Ζ͍Ζ
ࠓͳ͢͜ͱ • WebRTCͰૹΒΕͯ͘ΔH.264/AVCಈըετϦʔϜΛऔΓग़͢ํ๏ • H.264/AVCಈըετϦʔϜͲͷΑ͏ʹૹΒΕͯ͘Δͷ͔
ࠓͳ͞ͳ͍͜ͱ • WebRTCͱͳʹ͔ • WebRTCͰಈըΛετϦʔϛϯά͢Δํ๏ • WebRTC SFU Sora͕ͬͯ͘ΕΔ͜ͱ ଓཱ֬ɺηογϣϯͷཧɺSRTPͷ҉߸Խղআ……
• SkyWayͰͲ͏ͬͯΔͷ͔……ʢRTPΛऔΓग़ͤΕେৎʣ • H.264/AVCҎ֎ͷVP9, HEVC, AV1ͱ͍ͬͨಈըϑΥʔϚοτʹ͍ͭͯ
ͳͥWebRTCಈըΛτϥϯείʔυ͢Δͷ͔ • WebRTCಈըΛHLS৴͢ΔαʔϏεΛ։ൃதʂ https://speakerdeck.com/imageflux/imageflux-live • pixiv Sketch LiveͷόοΫΤϯυͱͯ͠Քಇத • αʔϏεʹڵຯ͋Δํੋඇ͓͕͚͍ͩ͘͞ʂ
ΤϯδχΞืू͍ͯ͠·͢ʂ
ImageFlux LiveͷΈ Transcoder Transcoder Transcoder RTP stream Streamerd Msgpack stream
TS TS TS ৴ऀ WebRTC SFU Sora WebRTC API H.264 bytestreamΛτϥϯείʔυ͢Δ RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ
H.264 bytestreamΛτϥϯείʔυ͢Δ ImageFlux LiveͷΈ Transcoder Transcoder Transcoder RTP stream Msgpack
stream TS TS TS ৴ऀ WebRTC SFU Sora WebRTC API RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ Streamerd
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client ηογϣϯཧ ϦΞϧλΠϜετϦʔϛϯά ଓཱ֬ʢNATӽ͑ʣ
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client ηογϣϯཧ ϦΞϧλΠϜετϦʔϛϯά ଓཱ֬ʢNATӽ͑ʣ
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client SDP SRTP ηογϣϯใ ಈըετϦʔϜ
WebRTCͰૹΒΕͯ͘ΔಈըΛ τϥϯείʔυ͢Δʹ 1. ΫϥΠΞϯτͱηογϣϯใʢSDPʣΛΓͱΓ͢Δ 2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ 3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ 4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ
WebRTCͰૹΒΕͯ͘ΔಈըΛ τϥϯείʔυ͢Δʹ 1. ΫϥΠΞϯτͱηογϣϯใʢSDPʣΛΓͱΓ͢Δ 2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ 3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ 4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ
1, 2WebRTC SFU Sora͕ͬͯ͘Ε·͢
RTP (Realtime Transfer Protocol) RFC3550 RTP: A Transport Protocol for
Real-Time Applications • ԻɺಈըͳͲͷσʔλΛϦΞϧλΠϜʹసૹ͢ΔͨΊͷϓϩτίϧ • UDP্ʹ࣮͞Ε͍ͯΔ • TCPΛ͍ͬͯͳ͍ͷͰɺॱং੍ޚ੍ޚΛRTPϨΠϠͰ࣮Ͱ͖Δ →ݴ͍͑ΕࣗͰ࣮͢Δඞཁ͕͋Δ • SRTPʢͬ͘͟Γݴ͑ʣRTPΛ҉߸Խͨ͠ͷ
࣮ମUDPͷബ͍ϥούʔ • UDPʹബ͍ϔομ͕͍͍ͭͯΔ͚ͩ • ϖΠϩʔυͷූ߸ԽํࣜϖΠϩʔυͷछྨ͝ͱʹҟͳΔ 0 1 2 3 4
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identifier contributing source (CSRC) identifier payload ….
ϖΠϩʔυͷผํ๏ • ϖΠϩʔυͷϑΥʔϚοτpayload typeͰผ͢Δ • payload type7bit͔͠ͳ͘ɺ͙͢ʹΓͳ͘ͳͬͨͷͰɺݱࡏͰ 96-127ΛಈతʹׂΓͯͯ͏ʢRFC3551ʣ • payload
typeͱϑΥʔϚοτͷϚοϐϯάSDPͷattributeͱͯ͠ ΓͱΓ͞ΕΔ • طʹొ͞Ε͍ͯΔͷIANAͰཧ͞Ε͍ͯΔ https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml
ॱংอূ • RTPUDPͰૹΒΕͯ͘ΔͷͰॱ൪อূ͞Ε͍ͯͳ͍ • RTPϔομʹγʔέϯε൪߸ʢsequence numberʣؚ͕·Ε͍ͯΔͷ Ͱ͜ΕΛͬͯιʔτ͢Δඞཁ͕͋Δ • ;ͭ͏ʹ300ݸ͙Β͍ޙΕͯύέοτ͕ඈΜͰ͖ͨΓͯ͠ਏ͍……͕ ͋·ΓͭͱԆͷݪҼʹͳΔͷͰέʔεόΠέʔε
• ͍·ͷ࣮Ͱύέοτ͕ඈΜͩ߹100ݸ͘Β͍ͭΑ͏ʹ͍ͯ͠Δ
ϑΥʔϚοτ͝ͱʹϖΠϩʔυͷܗࣜҟͳΔ • ͦΕͧΕҟͳΔRFCͰඪ४Խ͞Ε͍ͯΔ RFC6184 RTP Payload Format for H.264 Video
RFC7587 RTP Payload Format for the Opus Speech and Audio Codec draft-uberti-payload-vp9-01 RTP Payload Format for VP9 Video
RTP Payload Format for H.264 Video • H.264 NALU (NAL
Unitʣ1͕ͭ1ͭͷUDPύέοτʹؚ·ΕΔ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identifier contributing source (CSRC) identifier NAL Header Payload….
H.264/AVC NAL (Network Abstraction Layer) • H.264/AVCͰNALʢNetwork Abstraction Layerʣ͕ఆٛ͞Ε͍ͯΔ •
NALಈըετϦʔϜΛ͍ΖΜͳϝσΟΞͰૹ͍͢͠Α͏ʹநԽ͠ ͍ͯΔϨΠϠʔ • ؆୯ʹ͍͏ͱɺಈըετϦʔϜΛNAL UnitʢNALUʣͱݺΕΔͻͱ͔ͨ ·Γʹܗ͍ͯ͠Δ
H.264/AVC Bytestream Format (Annex B) • ΄ͱΜͲͷσίʔμH.264/AVC Bytestream Format (Annex
B)Λೖྗͱ ͯ͠͏͚͚ͭΔ • ISO/IEC 14496-10ͷAnnex Bʹࡌ͍ͬͯͨͷͰAnnex B formatͱݺΕΔ • جຊతʹNALUΛ00 00 00 01Ͱܨ͛ΕΑ͍ ͭ·ΓɺRTPͷϖΠϩʔυΛܨ͛Ε͍͍ͩͨΑͦ͞͏ 00 00 00 01 NALU 00 00 00 01 NALU 00 00 00 01 NALU …
ׂύέοτͱ݁߹ύέοτ • UDPͷϑϨʔϜαΠζҰൠతʹ1500byte͔͠ͳ͍ͷͰɺ݁߹ͨ͠Γɺ ׂͯ͠ෳͷύέοτʹೖΕͨΓ͢Δ͜ͱ͕͋Δ • ݁߹ύέοτͱɺׂύέοτNAL Unit typeͷ24-29Ͱఆٛ͞Ε͍ͯΔ NAL Unit
Packet Packet Type Name Section Type Type ------------------------------------------------------------- 1-23 NAL unit Single NAL unit packet 5.6 24 STAP-A Single-time aggregation packet 5.7.1 25 STAP-B Single-time aggregation packet 5.7.1 26 MTAP16 Multi-time aggregation packet 5.7.2 27 MTAP24 Multi-time aggregation packet 5.7.2 28 FU-A Fragmentation unit 5.8 29 FU-B Fragmentation unit 5.8 https://tools.ietf.org/html/rfc6184
ׂͨ͠Γ݁߹ͨ͠Γ͢Δ • ඇΠϯλʔϦʔϒϞʔυͰSTAP-AͱFU-A͕ΘΕΔͷͰ͜ͷ2ݸ͚ͩ ରԠ͢ΕΑ͍ • STAP-A: ෳͷNALUΛαΠζΛઌ಄ʹ͚ͭͯ݁߹ͨ͠ύέοτ • FU-A: ෳͷNALUΛׂ͠ɺ֤ύέοτʹFUϔομΛ͚ͭͨͷ
جຊతʹstart bit (s)͕͍͍ͭͯΔύέοτ͔Βend bit (e)͕͍͍ͭͯΔύ έοτ·Ͱ݁߹͢Δ 0 1 2 3 4 5 6 7 8 S E - Type FUϔομ
RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ 1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢ 2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ 3. ݁߹ύέοτͩͬͨ߹ׂͯ͠ෳͷόΠτετϦʔϜʹ͢Δ
4. ׂύέοτͩͬͨ߹શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬ ͯ1ͭͷNALUΛऔΓग़͢
SDP (Session Description Protocol) RFC4566 SDP: Session Description Protocol •
ηογϣϯใΛදݱ͢ΔͨΊͷϓϩτίϧ • WebRTCΛ͡ΊɺʢWebRTCͷϕʔεʹͳͬͨʣSIPɺRTSPɺϝʔϧͳͲ ͘ΘΕ͍ͯΔ • ಡΈͮΒ͍͜ͱͰ༗໊ʁ
ಡΈͮΒ͍ʁ v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar
i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf
[email protected]
(Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000 https://tools.ietf.org/html/rfc4566
ಡΈͮΒ͍ʁ v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar
i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf
[email protected]
(Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000 https://tools.ietf.org/html/rfc4566
Media Description "m=" m=<media> <port> <proto> <proto> <fmt> … <media>:
ϝσΟΞͷछྨʢaudio, video,…) <port>: ૹ৴ݩϙʔτ <proto>: ϓϩτίϧ <fmt>: ϑΥʔϚοτใ ϝσΟΞͷछྨͱϑΥʔϚοτɺૹ৴ݩϙʔτؚ͕·Ε͍ͯΔ ͋·Γ͑Δใͳ͍
Attributes "a=" a=<attribute>:<value> <attribute>: ଐੑ <value>: ͦͷ • SDPʹΛ֦ு͢ΔͨΊʹ༻ҙ͞Ε͍ͯΔύϥϝʔλ •
WebRTCͰ֦ு͞Ε͍ͯΔ͍͍ͩͨa=ʹೖ͍ͬͯΔ
SDP͔ΒRTPͷpayload type mappingΛऔΓग़͢ a=rtpmap:109 opus/48000/2 a=rtpmap:120 H264/90000 • payload typeͷmappingrtpmapͰࢦఆ͞ΕΔ
• ্ͷྫͩͱɺH.264ͷpayload type120ɺOpus109 • 2൪ͷϑΥʔϚοτ/प/νϟϯωϧ H.264/AVCͩͱ90000KHzͰλΠϜελϯϓ͕ͬͯ͘Δ͜ͱ͕Θ͔Δ
RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ 1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢ 2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ 3. ݁߹ύέοτͩͬͨ߹ׂͯ͠ෳͷόΠτετϦʔϜʹ͢Δ
4. ׂύέοτͩͬͨ߹શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬ ͯ1ͭͷNALUΛऔΓग़͢ ෮श
·ͱΊ • WebRTCಈըετϦʔϜΛτϥϯείʔυ͢Δʹɺ RTPύέοτΛσίʔυͰ͖ΕΑ͍ • αʔόαΠυͰRTPΛऔΓग़͢ʹWebRTC SFU Sora͕ศར • ImageFlux
Live։ൃதͰ͢ʂ