WebRTC動画をトランスコードする / Transcoding video streams from WebRTC

by Harukasan

Slide 1

Slide 1 text

Transcoding video streams  from WebRTC MICHII Shunsuke pixiv Inc. SkyWay UG Tokyo #2  2018-02-05

Slide 2

Slide 2 text

MICHII Shunsuke | @harukasan ImageFluxࣄۀ෦ ෦௕  ϐΫγϒςΫϊϩδʔζגࣜձࣾࣥߦ໾һ ImageFluxͰ͸GoΛॻ͍ͨΓΠϯϑϥपΓΛ΍ͬͨΓͱ͔͍Ζ͍Ζ

Slide 3

Slide 3 text

ࠓ೔͸ͳ͢͜ͱ • WebRTCͰૹΒΕͯ͘ΔH.264/AVCಈըετϦʔϜΛऔΓग़͢ํ๏ • H.264/AVCಈըετϦʔϜ͸ͲͷΑ͏ʹૹΒΕͯ͘Δͷ͔

Slide 4

Slide 4 text

ࠓ೔͸ͳ͞ͳ͍͜ͱ • WebRTCͱ͸ͳʹ͔ • WebRTCͰಈըΛετϦʔϛϯά͢Δํ๏ • WebRTC SFU Sora͕΍ͬͯ͘ΕΔ͜ͱ  ઀ଓཱ֬ɺηογϣϯͷ؅ཧɺSRTPͷ҉߸Խղআ…… • SkyWayͰͲ͏΍ͬͯ΍Δͷ͔……ʢRTPΛऔΓग़ͤΕ͹େৎ෉ʣ • H.264/AVCҎ֎ͷVP9, HEVC, AV1ͱ͍ͬͨಈըϑΥʔϚοτʹ͍ͭͯ

Slide 5

Slide 5 text

ͳͥWebRTCಈըΛτϥϯείʔυ͢Δͷ͔ • WebRTCಈըΛHLS഑৴͢ΔαʔϏεΛ։ൃதʂ  https://speakerdeck.com/imageﬂux/imageﬂux-live • pixiv Sketch LiveͷόοΫΤϯυͱͯ͠Քಇத • αʔϏεʹڵຯ͋Δํ͸ੋඇ͓੠͕͚͍ͩ͘͞ʂ  ΤϯδχΞ΋ืू͍ͯ͠·͢ʂ

Slide 6

Slide 6 text

ImageFlux Liveͷ࢓૊Έ Transcoder Transcoder Transcoder RTP stream Streamerd Msgpack  stream TS TS TS ഑৴ऀ WebRTC SFU  Sora WebRTC API H.264 bytestreamΛτϥϯείʔυ͢Δ RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ

Slide 7

Slide 7 text

H.264 bytestreamΛτϥϯείʔυ͢Δ ImageFlux Liveͷ࢓૊Έ Transcoder Transcoder Transcoder RTP stream Msgpack  stream TS TS TS ഑৴ऀ WebRTC SFU  Sora WebRTC API RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ Streamerd

Slide 8

Slide 8 text

WebRTCͰ͸ԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client ηογϣϯ؅ཧ ϦΞϧλΠϜετϦʔϛϯά ઀ଓཱ֬ʢNATӽ͑ʣ

Slide 9

Slide 9 text

WebRTCͰ͸ԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client ηογϣϯ؅ཧ ϦΞϧλΠϜετϦʔϛϯά ઀ଓཱ֬ʢNATӽ͑ʣ

Slide 10

Slide 10 text

WebRTCͰ͸ԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client SDP SRTP ηογϣϯ৘ใ ಈըετϦʔϜ

Slide 11

Slide 11 text

WebRTCͰૹΒΕͯ͘ΔಈըΛ  τϥϯείʔυ͢Δʹ͸ 1. ΫϥΠΞϯτͱηογϣϯ৘ใʢSDPʣΛ΍ΓͱΓ͢Δ 2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ 3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ 4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ

Slide 12

Slide 12 text

WebRTCͰૹΒΕͯ͘ΔಈըΛ  τϥϯείʔυ͢Δʹ͸ 1. ΫϥΠΞϯτͱηογϣϯ৘ใʢSDPʣΛ΍ΓͱΓ͢Δ 2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ 3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ 4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ 1, 2͸WebRTC SFU Sora͕΍ͬͯ͘Ε·͢

Slide 13

Slide 13 text

RTP (Realtime Transfer Protocol) RFC3550  RTP: A Transport Protocol for Real-Time Applications • Ի੠ɺಈըͳͲͷσʔλΛϦΞϧλΠϜʹసૹ͢ΔͨΊͷϓϩτίϧ • UDP্ʹ࣮૷͞Ε͍ͯΔ • TCPΛ࢖͍ͬͯͳ͍ͷͰɺॱং੍ޚ΍᫔᫓੍ޚΛRTPϨΠϠͰ࣮૷Ͱ͖Δ  →ݴ͍׵͑Ε͹ࣗ෼Ͱ࣮૷͢Δඞཁ͕͋Δ • SRTP͸ʢͬ͘͟Γݴ͑͹ʣRTPΛ҉߸Խͨ͠΋ͷ

Slide 14

Slide 14 text

࣮ମ͸UDPͷബ͍ϥούʔ • UDPʹബ͍ϔομ͕͍͍ͭͯΔ͚ͩ • ϖΠϩʔυͷූ߸Խํࣜ͸ϖΠϩʔυͷछྨ͝ͱʹҟͳΔ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identiﬁer contributing source (CSRC) identiﬁer payload  ….

Slide 15

Slide 15 text

ϖΠϩʔυͷ൑ผํ๏ • ϖΠϩʔυͷϑΥʔϚοτ͸payload typeͰ൑ผ͢Δ • payload type͸7bit͔͠ͳ͘ɺ͙͢ʹ଍Γͳ͘ͳͬͨͷͰɺݱࡏͰ͸  96-127ΛಈతʹׂΓ౰ͯͯ࢖͏ʢRFC3551ʣ • payload typeͱϑΥʔϚοτͷϚοϐϯά͸SDPͷattributeͱͯ͠  ΍ΓͱΓ͞ΕΔ • طʹొ࿥͞Ε͍ͯΔͷ͸IANAͰ؅ཧ͞Ε͍ͯΔ  https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml

Slide 16

Slide 16 text

ॱংอূ • RTP͸UDPͰૹΒΕͯ͘ΔͷͰॱ൪อূ͞Ε͍ͯͳ͍ • RTPϔομʹ͸γʔέϯε൪߸ʢsequence numberʣؚ͕·Ε͍ͯΔͷ Ͱ͜ΕΛ࢖ͬͯιʔτ͢Δඞཁ͕͋Δ • ;ͭ͏ʹ300ݸ͙Β͍ޙΕͯύέοτ͕ඈΜͰ͖ͨΓͯ͠ਏ͍……͕  ͋·Γ଴ͭͱ஗ԆͷݪҼʹͳΔͷͰέʔεόΠέʔε • ͍·ͷ࣮૷Ͱ͸ύέοτ͕ඈΜͩ৔߹͸100ݸ͘Β͍଴ͭΑ͏ʹ͍ͯ͠Δ

Slide 17

Slide 17 text

ϑΥʔϚοτ͝ͱʹϖΠϩʔυͷܗࣜ͸ҟͳΔ • ͦΕͧΕҟͳΔRFCͰඪ४Խ͞Ε͍ͯΔ RFC6184  RTP Payload Format for H.264 Video RFC7587  RTP Payload Format for the Opus Speech and Audio Codec draft-uberti-payload-vp9-01  RTP Payload Format for VP9 Video

Slide 18

Slide 18 text

RTP Payload Format for H.264 Video • H.264 NALU (NAL Unitʣ1͕ͭ1ͭͷUDPύέοτʹؚ·ΕΔ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identiﬁer contributing source (CSRC) identiﬁer NAL Header Payload….

Slide 19

Slide 19 text

H.264/AVC NAL (Network Abstraction Layer) • H.264/AVCͰ͸NALʢNetwork Abstraction Layerʣ͕ఆٛ͞Ε͍ͯΔ • NAL͸ಈըετϦʔϜΛ͍ΖΜͳϝσΟΞͰ఻ૹ͠΍͍͢Α͏ʹந৅Խ͠ ͍ͯΔϨΠϠʔ • ؆୯ʹ͍͏ͱɺಈըετϦʔϜΛNAL UnitʢNALUʣͱݺ͹ΕΔͻͱ͔ͨ ·Γʹ੔ܗ͍ͯ͠Δ

Slide 20

Slide 20 text

H.264/AVC Bytestream Format (Annex B) • ΄ͱΜͲͷσίʔμ͸H.264/AVC Bytestream Format (Annex B)Λೖྗͱ ͯ͠͏͚͚ͭΔ • ISO/IEC 14496-10ͷAnnex Bʹࡌ͍ͬͯͨͷͰAnnex B formatͱݺ͹ΕΔ • جຊతʹ͸NALUΛ00 00 00 01Ͱܨ͛Ε͹Α͍  ͭ·ΓɺRTPͷϖΠϩʔυΛܨ͛Ε͹͍͍ͩͨΑͦ͞͏ 00 00 00 01 NALU 00 00 00 01 NALU 00 00 00 01 NALU …

Slide 21

Slide 21 text

෼ׂύέοτͱ݁߹ύέοτ • UDPͷϑϨʔϜαΠζ͸Ұൠతʹ͸1500byte͔͠ͳ͍ͷͰɺ݁߹ͨ͠Γɺ ෼ׂͯ͠ෳ਺ͷύέοτʹೖΕͨΓ͢Δ͜ͱ͕͋Δ • ݁߹ύέοτͱɺ෼ׂύέοτ͸NAL Unit typeͷ24-29Ͱఆٛ͞Ε͍ͯΔ NAL Unit Packet Packet Type Name Section Type Type ------------------------------------------------------------- 1-23 NAL unit Single NAL unit packet 5.6 24 STAP-A Single-time aggregation packet 5.7.1 25 STAP-B Single-time aggregation packet 5.7.1 26 MTAP16 Multi-time aggregation packet 5.7.2 27 MTAP24 Multi-time aggregation packet 5.7.2 28 FU-A Fragmentation unit 5.8 29 FU-B Fragmentation unit 5.8 https://tools.ietf.org/html/rfc6184

Slide 22

Slide 22 text

෼ׂͨ͠Γ݁߹ͨ͠Γ͢Δ • ඇΠϯλʔϦʔϒϞʔυͰ͸STAP-AͱFU-A͕࢖ΘΕΔͷͰ͜ͷ2ݸ͚ͩ ରԠ͢Ε͹Α͍ • STAP-A: ෳ਺ͷNALUΛαΠζΛઌ಄ʹ͚ͭͯ݁߹ͨ͠ύέοτ • FU-A: ෳ਺ͷNALUΛ෼ׂ͠ɺ֤ύέοτʹFUϔομΛ͚ͭͨ΋ͷ  جຊతʹ͸start bit (s)͕͍͍ͭͯΔύέοτ͔Βend bit (e)͕͍͍ͭͯΔύ έοτ·Ͱ݁߹͢Δ 0 1 2 3 4 5 6 7 8 S E - Type FUϔομ

Slide 23

Slide 23 text

RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ͸ 1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢ 2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ 3. ݁߹ύέοτͩͬͨ৔߹͸෼ׂͯ͠ෳ਺ͷόΠτετϦʔϜʹ͢Δ 4. ෼ׂύέοτͩͬͨ৔߹͸શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬ ͯ1ͭͷNALUΛऔΓग़͢

Slide 24

Slide 24 text

SDP (Session Description Protocol) RFC4566  SDP: Session Description Protocol • ηογϣϯ৘ใΛදݱ͢ΔͨΊͷϓϩτίϧ • WebRTCΛ͸͡ΊɺʢWebRTCͷϕʔεʹͳͬͨʣSIPɺRTSPɺϝʔϧͳͲ ޿͘࢖ΘΕ͍ͯΔ • ಡΈͮΒ͍͜ͱͰ༗໊ʁ

Slide 25

Slide 25 text

ಡΈͮΒ͍ʁ v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf [email protected] (Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000 https://tools.ietf.org/html/rfc4566

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Media Description "m=" m= … : ϝσΟΞͷछྨʢaudio, video,…)  : ૹ৴ݩϙʔτ  : ϓϩτίϧ  : ϑΥʔϚοτ৘ใ ϝσΟΞͷछྨͱϑΥʔϚοτɺૹ৴ݩϙʔτؚ͕·Ε͍ͯΔ  ͋·Γ࢖͑Δ৘ใ͸ͳ͍

Slide 28

Slide 28 text

Attributes "a=" a=: : ଐੑ  : ͦͷ஋ • SDPʹ஋Λ֦ு͢ΔͨΊʹ༻ҙ͞Ε͍ͯΔύϥϝʔλ • WebRTCͰ֦ு͞Ε͍ͯΔ஋͸͍͍ͩͨa=ʹೖ͍ͬͯΔ

Slide 29

Slide 29 text

SDP͔ΒRTPͷpayload type mappingΛऔΓग़͢ a=rtpmap:109 opus/48000/2  a=rtpmap:120 H264/90000 • payload typeͷmapping͸rtpmapͰࢦఆ͞ΕΔ • ্ͷྫͩͱɺH.264ͷpayload type͸120ɺOpus͸109 • 2൪໨ͷ஋͸ϑΥʔϚοτ/प೾਺/νϟϯωϧ਺  H.264/AVCͩͱ90000KHzͰλΠϜελϯϓ͕΍ͬͯ͘Δ͜ͱ͕Θ͔Δ

Slide 30

Slide 30 text

Slide 31

Slide 31 text

·ͱΊ • WebRTCಈըετϦʔϜΛτϥϯείʔυ͢Δʹ͸ɺ  RTPύέοτΛσίʔυͰ͖Ε͹Α͍ • αʔόαΠυͰRTPΛऔΓग़͢ʹ͸WebRTC SFU Sora͕ศར • ImageFlux Live։ൃதͰ͢ʂ