SkyWay UG Tokyo #2 2018-02-05
Transcoding video streams from WebRTCMICHII Shunsuke pixiv Inc.SkyWay UG Tokyo #2 2018-02-05
View Slide
MICHII Shunsuke | @harukasanImageFluxࣄۀ෦ ෦ ϐΫγϒςΫϊϩδʔζגࣜձࣾࣥߦһImageFluxͰGoΛॻ͍ͨΓΠϯϑϥपΓΛͬͨΓͱ͔͍Ζ͍Ζ
ࠓͳ͢͜ͱ• WebRTCͰૹΒΕͯ͘ΔH.264/AVCಈըετϦʔϜΛऔΓग़͢ํ๏• H.264/AVCಈըετϦʔϜͲͷΑ͏ʹૹΒΕͯ͘Δͷ͔
ࠓͳ͞ͳ͍͜ͱ• WebRTCͱͳʹ͔• WebRTCͰಈըΛετϦʔϛϯά͢Δํ๏• WebRTC SFU Sora͕ͬͯ͘ΕΔ͜ͱ ଓཱ֬ɺηογϣϯͷཧɺSRTPͷ҉߸Խղআ……• SkyWayͰͲ͏ͬͯΔͷ͔……ʢRTPΛऔΓग़ͤΕେৎʣ• H.264/AVCҎ֎ͷVP9, HEVC, AV1ͱ͍ͬͨಈըϑΥʔϚοτʹ͍ͭͯ
ͳͥWebRTCಈըΛτϥϯείʔυ͢Δͷ͔• WebRTCಈըΛHLS৴͢ΔαʔϏεΛ։ൃதʂ https://speakerdeck.com/imageflux/imageflux-live• pixiv Sketch LiveͷόοΫΤϯυͱͯ͠Քಇத• αʔϏεʹڵຯ͋Δํੋඇ͓͕͚͍ͩ͘͞ʂ ΤϯδχΞืू͍ͯ͠·͢ʂ
ImageFlux LiveͷΈTranscoderTranscoderTranscoderRTP streamStreamerdMsgpack streamTSTSTS৴ऀ WebRTC SFU SoraWebRTCAPIH.264 bytestreamΛτϥϯείʔυ͢ΔRTPετϦʔϜΛσίʔυͯ͠H.264 bytestream formatʹͯ͠msgpackʹͭΊΔ
H.264 bytestreamΛτϥϯείʔυ͢ΔImageFlux LiveͷΈTranscoderTranscoderTranscoderRTP streamMsgpack streamTSTSTS৴ऀ WebRTC SFU SoraWebRTCAPIRTPετϦʔϜΛσίʔυͯ͠H.264 bytestream formatʹͯ͠msgpackʹͭΊΔStreamerd
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ClientWebRTCClientηογϣϯཧϦΞϧλΠϜετϦʔϛϯάଓཱ֬ʢNATӽ͑ʣ
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ClientWebRTCClientSDPSRTPηογϣϯใಈըετϦʔϜ
WebRTCͰૹΒΕͯ͘ΔಈըΛ τϥϯείʔυ͢Δʹ1. ΫϥΠΞϯτͱηογϣϯใʢSDPʣΛΓͱΓ͢Δ2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ
WebRTCͰૹΒΕͯ͘ΔಈըΛ τϥϯείʔυ͢Δʹ1. ΫϥΠΞϯτͱηογϣϯใʢSDPʣΛΓͱΓ͢Δ2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ1, 2WebRTC SFU Sora͕ͬͯ͘Ε·͢
RTP (Realtime Transfer Protocol)RFC3550 RTP: A Transport Protocol for Real-Time Applications• ԻɺಈըͳͲͷσʔλΛϦΞϧλΠϜʹసૹ͢ΔͨΊͷϓϩτίϧ• UDP্ʹ࣮͞Ε͍ͯΔ• TCPΛ͍ͬͯͳ͍ͷͰɺॱং੍ޚ੍ޚΛRTPϨΠϠͰ࣮Ͱ͖Δ →ݴ͍͑ΕࣗͰ࣮͢Δඞཁ͕͋Δ• SRTPʢͬ͘͟Γݴ͑ʣRTPΛ҉߸Խͨ͠ͷ
࣮ମUDPͷബ͍ϥούʔ• UDPʹബ͍ϔομ͕͍͍ͭͯΔ͚ͩ• ϖΠϩʔυͷූ߸ԽํࣜϖΠϩʔυͷछྨ͝ͱʹҟͳΔ0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31V P X CC M payload type sequence numbertimestampsynchronization source (SSRC) identifiercontributing source (CSRC) identifierpayload ….
ϖΠϩʔυͷผํ๏• ϖΠϩʔυͷϑΥʔϚοτpayload typeͰผ͢Δ• payload type7bit͔͠ͳ͘ɺ͙͢ʹΓͳ͘ͳͬͨͷͰɺݱࡏͰ 96-127ΛಈతʹׂΓͯͯ͏ʢRFC3551ʣ• payload typeͱϑΥʔϚοτͷϚοϐϯάSDPͷattributeͱͯ͠ ΓͱΓ͞ΕΔ• طʹొ͞Ε͍ͯΔͷIANAͰཧ͞Ε͍ͯΔ https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml
ॱংอূ• RTPUDPͰૹΒΕͯ͘ΔͷͰॱ൪อূ͞Ε͍ͯͳ͍• RTPϔομʹγʔέϯε൪߸ʢsequence numberʣؚ͕·Ε͍ͯΔͷͰ͜ΕΛͬͯιʔτ͢Δඞཁ͕͋Δ• ;ͭ͏ʹ300ݸ͙Β͍ޙΕͯύέοτ͕ඈΜͰ͖ͨΓͯ͠ਏ͍……͕ ͋·ΓͭͱԆͷݪҼʹͳΔͷͰέʔεόΠέʔε• ͍·ͷ࣮Ͱύέοτ͕ඈΜͩ߹100ݸ͘Β͍ͭΑ͏ʹ͍ͯ͠Δ
ϑΥʔϚοτ͝ͱʹϖΠϩʔυͷܗࣜҟͳΔ• ͦΕͧΕҟͳΔRFCͰඪ४Խ͞Ε͍ͯΔRFC6184 RTP Payload Format for H.264 VideoRFC7587 RTP Payload Format for the Opus Speech and Audio Codecdraft-uberti-payload-vp9-01 RTP Payload Format for VP9 Video
RTP Payload Format for H.264 Video• H.264 NALU (NAL Unitʣ1͕ͭ1ͭͷUDPύέοτʹؚ·ΕΔ0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31V P X CC M payload type sequence numbertimestampsynchronization source (SSRC) identifiercontributing source (CSRC) identifierNAL Header Payload….
H.264/AVC NAL (Network Abstraction Layer)• H.264/AVCͰNALʢNetwork Abstraction Layerʣ͕ఆٛ͞Ε͍ͯΔ• NALಈըετϦʔϜΛ͍ΖΜͳϝσΟΞͰૹ͍͢͠Α͏ʹநԽ͍ͯ͠ΔϨΠϠʔ• ؆୯ʹ͍͏ͱɺಈըετϦʔϜΛNAL UnitʢNALUʣͱݺΕΔͻͱ͔ͨ·Γʹܗ͍ͯ͠Δ
H.264/AVC Bytestream Format (Annex B)• ΄ͱΜͲͷσίʔμH.264/AVC Bytestream Format (Annex B)Λೖྗͱͯ͠͏͚͚ͭΔ• ISO/IEC 14496-10ͷAnnex Bʹࡌ͍ͬͯͨͷͰAnnex B formatͱݺΕΔ• جຊతʹNALUΛ00 00 00 01Ͱܨ͛ΕΑ͍ ͭ·ΓɺRTPͷϖΠϩʔυΛܨ͛Ε͍͍ͩͨΑͦ͞͏00 00 00 01 NALU 00 00 00 01 NALU 00 00 00 01 NALU …
ׂύέοτͱ݁߹ύέοτ• UDPͷϑϨʔϜαΠζҰൠతʹ1500byte͔͠ͳ͍ͷͰɺ݁߹ͨ͠Γɺׂͯ͠ෳͷύέοτʹೖΕͨΓ͢Δ͜ͱ͕͋Δ• ݁߹ύέοτͱɺׂύέοτNAL Unit typeͷ24-29Ͱఆٛ͞Ε͍ͯΔNAL Unit Packet Packet Type Name SectionType Type-------------------------------------------------------------1-23 NAL unit Single NAL unit packet 5.624 STAP-A Single-time aggregation packet 5.7.125 STAP-B Single-time aggregation packet 5.7.126 MTAP16 Multi-time aggregation packet 5.7.227 MTAP24 Multi-time aggregation packet 5.7.228 FU-A Fragmentation unit 5.829 FU-B Fragmentation unit 5.8https://tools.ietf.org/html/rfc6184
ׂͨ͠Γ݁߹ͨ͠Γ͢Δ• ඇΠϯλʔϦʔϒϞʔυͰSTAP-AͱFU-A͕ΘΕΔͷͰ͜ͷ2ݸ͚ͩରԠ͢ΕΑ͍• STAP-A: ෳͷNALUΛαΠζΛઌ಄ʹ͚ͭͯ݁߹ͨ͠ύέοτ• FU-A: ෳͷNALUΛׂ͠ɺ֤ύέοτʹFUϔομΛ͚ͭͨͷ جຊతʹstart bit (s)͕͍͍ͭͯΔύέοτ͔Βend bit (e)͕͍͍ͭͯΔύέοτ·Ͱ݁߹͢Δ0 1 2 3 4 5 6 7 8S E - TypeFUϔομ
RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ3. ݁߹ύέοτͩͬͨ߹ׂͯ͠ෳͷόΠτετϦʔϜʹ͢Δ4. ׂύέοτͩͬͨ߹શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬͯ1ͭͷNALUΛऔΓग़͢
SDP (Session Description Protocol)RFC4566 SDP: Session Description Protocol• ηογϣϯใΛදݱ͢ΔͨΊͷϓϩτίϧ• WebRTCΛ͡ΊɺʢWebRTCͷϕʔεʹͳͬͨʣSIPɺRTSPɺϝʔϧͳͲ͘ΘΕ͍ͯΔ• ಡΈͮΒ͍͜ͱͰ༗໊ʁ
ಡΈͮΒ͍ʁv=0o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5s=SDP Seminari=A Seminar on the session description protocolu=http://www.example.com/seminars/sdp.pdf[email protected] (Jane Doe)c=IN IP4 224.2.17.12/127t=2873397496 2873404696a=recvonlym=audio 49170 RTP/AVP 0m=video 51372 RTP/AVP 99a=rtpmap:99 h263-1998/90000https://tools.ietf.org/html/rfc4566
Media Description "m="m= …: ϝσΟΞͷछྨʢaudio, video,…) : ૹ৴ݩϙʔτ : ϓϩτίϧ : ϑΥʔϚοτใϝσΟΞͷछྨͱϑΥʔϚοτɺૹ৴ݩϙʔτؚ͕·Ε͍ͯΔ ͋·Γ͑Δใͳ͍
Attributes "a="a=:: ଐੑ : ͦͷ• SDPʹΛ֦ு͢ΔͨΊʹ༻ҙ͞Ε͍ͯΔύϥϝʔλ• WebRTCͰ֦ு͞Ε͍ͯΔ͍͍ͩͨa=ʹೖ͍ͬͯΔ
SDP͔ΒRTPͷpayload type mappingΛऔΓग़͢a=rtpmap:109 opus/48000/2 a=rtpmap:120 H264/90000• payload typeͷmappingrtpmapͰࢦఆ͞ΕΔ• ্ͷྫͩͱɺH.264ͷpayload type120ɺOpus109• 2൪ͷϑΥʔϚοτ/प/νϟϯωϧ H.264/AVCͩͱ90000KHzͰλΠϜελϯϓ͕ͬͯ͘Δ͜ͱ͕Θ͔Δ
RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ3. ݁߹ύέοτͩͬͨ߹ׂͯ͠ෳͷόΠτετϦʔϜʹ͢Δ4. ׂύέοτͩͬͨ߹શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬͯ1ͭͷNALUΛऔΓग़͢෮श
·ͱΊ• WebRTCಈըετϦʔϜΛτϥϯείʔυ͢Δʹɺ RTPύέοτΛσίʔυͰ͖ΕΑ͍• αʔόαΠυͰRTPΛऔΓग़͢ʹWebRTC SFU Sora͕ศར• ImageFlux Live։ൃதͰ͢ʂ
harukasan
tkikuchi1002
coconala_engineer
palark
cybozuinsideout
you
upura
yamamuteki
chiroito
biwashi
nayuts
smasato
jakevdp
skipperchong
ammeep
notwaldorf
quramy
deanohume
michaelherold
phodgson
denniskardys
cherdarchuk
eileencodes
speakerdeck