SkyWay UG Tokyo #2 2018-02-05
Transcoding video streams from WebRTCMICHII Shunsuke pixiv Inc.SkyWay UG Tokyo #2 2018-02-05
View Slide
MICHII Shunsuke | @harukasanImageFluxࣄۀ෦ ෦ ϐΫγϒςΫϊϩδʔζגࣜձࣾࣥߦһImageFluxͰGoΛॻ͍ͨΓΠϯϑϥपΓΛͬͨΓͱ͔͍Ζ͍Ζ
ࠓͳ͢͜ͱ• WebRTCͰૹΒΕͯ͘ΔH.264/AVCಈըετϦʔϜΛऔΓग़͢ํ๏• H.264/AVCಈըετϦʔϜͲͷΑ͏ʹૹΒΕͯ͘Δͷ͔
ࠓͳ͞ͳ͍͜ͱ• WebRTCͱͳʹ͔• WebRTCͰಈըΛετϦʔϛϯά͢Δํ๏• WebRTC SFU Sora͕ͬͯ͘ΕΔ͜ͱ ଓཱ֬ɺηογϣϯͷཧɺSRTPͷ҉߸Խղআ……• SkyWayͰͲ͏ͬͯΔͷ͔……ʢRTPΛऔΓग़ͤΕେৎʣ• H.264/AVCҎ֎ͷVP9, HEVC, AV1ͱ͍ͬͨಈըϑΥʔϚοτʹ͍ͭͯ
ͳͥWebRTCಈըΛτϥϯείʔυ͢Δͷ͔• WebRTCಈըΛHLS৴͢ΔαʔϏεΛ։ൃதʂ https://speakerdeck.com/imageflux/imageflux-live• pixiv Sketch LiveͷόοΫΤϯυͱͯ͠Քಇத• αʔϏεʹڵຯ͋Δํੋඇ͓͕͚͍ͩ͘͞ʂ ΤϯδχΞืू͍ͯ͠·͢ʂ
ImageFlux LiveͷΈTranscoderTranscoderTranscoderRTP streamStreamerdMsgpack streamTSTSTS৴ऀ WebRTC SFU SoraWebRTCAPIH.264 bytestreamΛτϥϯείʔυ͢ΔRTPετϦʔϜΛσίʔυͯ͠H.264 bytestream formatʹͯ͠msgpackʹͭΊΔ
H.264 bytestreamΛτϥϯείʔυ͢ΔImageFlux LiveͷΈTranscoderTranscoderTranscoderRTP streamMsgpack streamTSTSTS৴ऀ WebRTC SFU SoraWebRTCAPIRTPετϦʔϜΛσίʔυͯ͠H.264 bytestream formatʹͯ͠msgpackʹͭΊΔStreamerd
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ClientWebRTCClientηογϣϯཧϦΞϧλΠϜετϦʔϛϯάଓཱ֬ʢNATӽ͑ʣ
WebRTCͰԿΛ௨৴͍ͯ͠Δͷ͔ClientWebRTCClientSDPSRTPηογϣϯใಈըετϦʔϜ
WebRTCͰૹΒΕͯ͘ΔಈըΛ τϥϯείʔυ͢Δʹ1. ΫϥΠΞϯτͱηογϣϯใʢSDPʣΛΓͱΓ͢Δ2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ
WebRTCͰૹΒΕͯ͘ΔಈըΛ τϥϯείʔυ͢Δʹ1. ΫϥΠΞϯτͱηογϣϯใʢSDPʣΛΓͱΓ͢Δ2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ1, 2WebRTC SFU Sora͕ͬͯ͘Ε·͢
RTP (Realtime Transfer Protocol)RFC3550 RTP: A Transport Protocol for Real-Time Applications• ԻɺಈըͳͲͷσʔλΛϦΞϧλΠϜʹసૹ͢ΔͨΊͷϓϩτίϧ• UDP্ʹ࣮͞Ε͍ͯΔ• TCPΛ͍ͬͯͳ͍ͷͰɺॱং੍ޚ੍ޚΛRTPϨΠϠͰ࣮Ͱ͖Δ →ݴ͍͑ΕࣗͰ࣮͢Δඞཁ͕͋Δ• SRTPʢͬ͘͟Γݴ͑ʣRTPΛ҉߸Խͨ͠ͷ
࣮ମUDPͷബ͍ϥούʔ• UDPʹബ͍ϔομ͕͍͍ͭͯΔ͚ͩ• ϖΠϩʔυͷූ߸ԽํࣜϖΠϩʔυͷछྨ͝ͱʹҟͳΔ0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31V P X CC M payload type sequence numbertimestampsynchronization source (SSRC) identifiercontributing source (CSRC) identifierpayload ….
ϖΠϩʔυͷผํ๏• ϖΠϩʔυͷϑΥʔϚοτpayload typeͰผ͢Δ• payload type7bit͔͠ͳ͘ɺ͙͢ʹΓͳ͘ͳͬͨͷͰɺݱࡏͰ 96-127ΛಈతʹׂΓͯͯ͏ʢRFC3551ʣ• payload typeͱϑΥʔϚοτͷϚοϐϯάSDPͷattributeͱͯ͠ ΓͱΓ͞ΕΔ• طʹొ͞Ε͍ͯΔͷIANAͰཧ͞Ε͍ͯΔ https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml
ॱংอূ• RTPUDPͰૹΒΕͯ͘ΔͷͰॱ൪อূ͞Ε͍ͯͳ͍• RTPϔομʹγʔέϯε൪߸ʢsequence numberʣؚ͕·Ε͍ͯΔͷͰ͜ΕΛͬͯιʔτ͢Δඞཁ͕͋Δ• ;ͭ͏ʹ300ݸ͙Β͍ޙΕͯύέοτ͕ඈΜͰ͖ͨΓͯ͠ਏ͍……͕ ͋·ΓͭͱԆͷݪҼʹͳΔͷͰέʔεόΠέʔε• ͍·ͷ࣮Ͱύέοτ͕ඈΜͩ߹100ݸ͘Β͍ͭΑ͏ʹ͍ͯ͠Δ
ϑΥʔϚοτ͝ͱʹϖΠϩʔυͷܗࣜҟͳΔ• ͦΕͧΕҟͳΔRFCͰඪ४Խ͞Ε͍ͯΔRFC6184 RTP Payload Format for H.264 VideoRFC7587 RTP Payload Format for the Opus Speech and Audio Codecdraft-uberti-payload-vp9-01 RTP Payload Format for VP9 Video
RTP Payload Format for H.264 Video• H.264 NALU (NAL Unitʣ1͕ͭ1ͭͷUDPύέοτʹؚ·ΕΔ0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31V P X CC M payload type sequence numbertimestampsynchronization source (SSRC) identifiercontributing source (CSRC) identifierNAL Header Payload….
H.264/AVC NAL (Network Abstraction Layer)• H.264/AVCͰNALʢNetwork Abstraction Layerʣ͕ఆٛ͞Ε͍ͯΔ• NALಈըετϦʔϜΛ͍ΖΜͳϝσΟΞͰૹ͍͢͠Α͏ʹநԽ͍ͯ͠ΔϨΠϠʔ• ؆୯ʹ͍͏ͱɺಈըετϦʔϜΛNAL UnitʢNALUʣͱݺΕΔͻͱ͔ͨ·Γʹܗ͍ͯ͠Δ
H.264/AVC Bytestream Format (Annex B)• ΄ͱΜͲͷσίʔμH.264/AVC Bytestream Format (Annex B)Λೖྗͱͯ͠͏͚͚ͭΔ• ISO/IEC 14496-10ͷAnnex Bʹࡌ͍ͬͯͨͷͰAnnex B formatͱݺΕΔ• جຊతʹNALUΛ00 00 00 01Ͱܨ͛ΕΑ͍ ͭ·ΓɺRTPͷϖΠϩʔυΛܨ͛Ε͍͍ͩͨΑͦ͞͏00 00 00 01 NALU 00 00 00 01 NALU 00 00 00 01 NALU …
ׂύέοτͱ݁߹ύέοτ• UDPͷϑϨʔϜαΠζҰൠతʹ1500byte͔͠ͳ͍ͷͰɺ݁߹ͨ͠Γɺׂͯ͠ෳͷύέοτʹೖΕͨΓ͢Δ͜ͱ͕͋Δ• ݁߹ύέοτͱɺׂύέοτNAL Unit typeͷ24-29Ͱఆٛ͞Ε͍ͯΔNAL Unit Packet Packet Type Name SectionType Type-------------------------------------------------------------1-23 NAL unit Single NAL unit packet 5.624 STAP-A Single-time aggregation packet 5.7.125 STAP-B Single-time aggregation packet 5.7.126 MTAP16 Multi-time aggregation packet 5.7.227 MTAP24 Multi-time aggregation packet 5.7.228 FU-A Fragmentation unit 5.829 FU-B Fragmentation unit 5.8https://tools.ietf.org/html/rfc6184
ׂͨ͠Γ݁߹ͨ͠Γ͢Δ• ඇΠϯλʔϦʔϒϞʔυͰSTAP-AͱFU-A͕ΘΕΔͷͰ͜ͷ2ݸ͚ͩରԠ͢ΕΑ͍• STAP-A: ෳͷNALUΛαΠζΛઌ಄ʹ͚ͭͯ݁߹ͨ͠ύέοτ• FU-A: ෳͷNALUΛׂ͠ɺ֤ύέοτʹFUϔομΛ͚ͭͨͷ جຊతʹstart bit (s)͕͍͍ͭͯΔύέοτ͔Βend bit (e)͕͍͍ͭͯΔύέοτ·Ͱ݁߹͢Δ0 1 2 3 4 5 6 7 8S E - TypeFUϔομ
RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ3. ݁߹ύέοτͩͬͨ߹ׂͯ͠ෳͷόΠτετϦʔϜʹ͢Δ4. ׂύέοτͩͬͨ߹શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬͯ1ͭͷNALUΛऔΓग़͢
SDP (Session Description Protocol)RFC4566 SDP: Session Description Protocol• ηογϣϯใΛදݱ͢ΔͨΊͷϓϩτίϧ• WebRTCΛ͡ΊɺʢWebRTCͷϕʔεʹͳͬͨʣSIPɺRTSPɺϝʔϧͳͲ͘ΘΕ͍ͯΔ• ಡΈͮΒ͍͜ͱͰ༗໊ʁ
ಡΈͮΒ͍ʁv=0o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5s=SDP Seminari=A Seminar on the session description protocolu=http://www.example.com/seminars/sdp.pdf[email protected] (Jane Doe)c=IN IP4 224.2.17.12/127t=2873397496 2873404696a=recvonlym=audio 49170 RTP/AVP 0m=video 51372 RTP/AVP 99a=rtpmap:99 h263-1998/90000https://tools.ietf.org/html/rfc4566
Media Description "m="m= …: ϝσΟΞͷछྨʢaudio, video,…) : ૹ৴ݩϙʔτ : ϓϩτίϧ : ϑΥʔϚοτใϝσΟΞͷछྨͱϑΥʔϚοτɺૹ৴ݩϙʔτؚ͕·Ε͍ͯΔ ͋·Γ͑Δใͳ͍
Attributes "a="a=:: ଐੑ : ͦͷ• SDPʹΛ֦ு͢ΔͨΊʹ༻ҙ͞Ε͍ͯΔύϥϝʔλ• WebRTCͰ֦ு͞Ε͍ͯΔ͍͍ͩͨa=ʹೖ͍ͬͯΔ
SDP͔ΒRTPͷpayload type mappingΛऔΓग़͢a=rtpmap:109 opus/48000/2 a=rtpmap:120 H264/90000• payload typeͷmappingrtpmapͰࢦఆ͞ΕΔ• ্ͷྫͩͱɺH.264ͷpayload type120ɺOpus109• 2൪ͷϑΥʔϚοτ/प/νϟϯωϧ H.264/AVCͩͱ90000KHzͰλΠϜελϯϓ͕ͬͯ͘Δ͜ͱ͕Θ͔Δ
RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ3. ݁߹ύέοτͩͬͨ߹ׂͯ͠ෳͷόΠτετϦʔϜʹ͢Δ4. ׂύέοτͩͬͨ߹શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬͯ1ͭͷNALUΛऔΓग़͢෮श
·ͱΊ• WebRTCಈըετϦʔϜΛτϥϯείʔυ͢Δʹɺ RTPύέοτΛσίʔυͰ͖ΕΑ͍• αʔόαΠυͰRTPΛऔΓग़͢ʹWebRTC SFU Sora͕ศར• ImageFlux Live։ൃதͰ͢ʂ
harukasan
sasakitomohiro
nnstt1
dzeyelid
yumayabe
hayaok3
mmclsntr
yuya4
tenforward
optim
cyberagentdevelopers
spinscale
yoyakoba
cassininazir
jmmastey
chriscoyier
lemiorhan
addyosmani
jponch
akmur
shpigford
aki_iinuma
tanoku
afnizarnur
mza