WebRTC動画をトランスコードする / Transcoding video streams from WebRTC

124da56a613b15fa980427533e4e3839?s=47 Harukasan
February 05, 2018

WebRTC動画をトランスコードする / Transcoding video streams from WebRTC

SkyWay UG Tokyo #2
2018-02-05

124da56a613b15fa980427533e4e3839?s=128

Harukasan

February 05, 2018
Tweet

Transcript

  1. 4.

    ࠓ೔͸ͳ͞ͳ͍͜ͱ • WebRTCͱ͸ͳʹ͔ • WebRTCͰಈըΛετϦʔϛϯά͢Δํ๏ • WebRTC SFU Sora͕΍ͬͯ͘ΕΔ͜ͱ
 ઀ଓཱ֬ɺηογϣϯͷ؅ཧɺSRTPͷ҉߸Խղআ……

    • SkyWayͰͲ͏΍ͬͯ΍Δͷ͔……ʢRTPΛऔΓग़ͤΕ͹େৎ෉ʣ • H.264/AVCҎ֎ͷVP9, HEVC, AV1ͱ͍ͬͨಈըϑΥʔϚοτʹ͍ͭͯ
  2. 6.

    ImageFlux Liveͷ࢓૊Έ Transcoder Transcoder Transcoder RTP stream Streamerd Msgpack
 stream

    TS TS TS ഑৴ऀ WebRTC SFU
 Sora WebRTC API H.264 bytestreamΛτϥϯείʔυ͢Δ RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ
  3. 7.

    H.264 bytestreamΛτϥϯείʔυ͢Δ ImageFlux Liveͷ࢓૊Έ Transcoder Transcoder Transcoder RTP stream Msgpack


    stream TS TS TS ഑৴ऀ WebRTC SFU
 Sora WebRTC API RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ Streamerd
  4. 13.

    RTP (Realtime Transfer Protocol) RFC3550
 RTP: A Transport Protocol for

    Real-Time Applications • Ի੠ɺಈըͳͲͷσʔλΛϦΞϧλΠϜʹసૹ͢ΔͨΊͷϓϩτίϧ • UDP্ʹ࣮૷͞Ε͍ͯΔ • TCPΛ࢖͍ͬͯͳ͍ͷͰɺॱং੍ޚ΍᫔᫓੍ޚΛRTPϨΠϠͰ࣮૷Ͱ͖Δ
 →ݴ͍׵͑Ε͹ࣗ෼Ͱ࣮૷͢Δඞཁ͕͋Δ • SRTP͸ʢͬ͘͟Γݴ͑͹ʣRTPΛ҉߸Խͨ͠΋ͷ
  5. 14.

    ࣮ମ͸UDPͷബ͍ϥούʔ • UDPʹബ͍ϔομ͕͍͍ͭͯΔ͚ͩ • ϖΠϩʔυͷූ߸Խํࣜ͸ϖΠϩʔυͷछྨ͝ͱʹҟͳΔ 0 1 2 3 4

    5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identifier contributing source (CSRC) identifier payload
 ….
  6. 15.

    ϖΠϩʔυͷ൑ผํ๏ • ϖΠϩʔυͷϑΥʔϚοτ͸payload typeͰ൑ผ͢Δ • payload type͸7bit͔͠ͳ͘ɺ͙͢ʹ଍Γͳ͘ͳͬͨͷͰɺݱࡏͰ͸
 96-127ΛಈతʹׂΓ౰ͯͯ࢖͏ʢRFC3551ʣ • payload

    typeͱϑΥʔϚοτͷϚοϐϯά͸SDPͷattributeͱͯ͠
 ΍ΓͱΓ͞ΕΔ • طʹొ࿥͞Ε͍ͯΔͷ͸IANAͰ؅ཧ͞Ε͍ͯΔ
 https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml
  7. 17.

    ϑΥʔϚοτ͝ͱʹϖΠϩʔυͷܗࣜ͸ҟͳΔ • ͦΕͧΕҟͳΔRFCͰඪ४Խ͞Ε͍ͯΔ RFC6184
 RTP Payload Format for H.264 Video

    RFC7587
 RTP Payload Format for the Opus Speech and Audio Codec draft-uberti-payload-vp9-01
 RTP Payload Format for VP9 Video
  8. 18.

    RTP Payload Format for H.264 Video • H.264 NALU (NAL

    Unitʣ1͕ͭ1ͭͷUDPύέοτʹؚ·ΕΔ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identifier contributing source (CSRC) identifier NAL Header Payload….
  9. 19.

    H.264/AVC NAL (Network Abstraction Layer) • H.264/AVCͰ͸NALʢNetwork Abstraction Layerʣ͕ఆٛ͞Ε͍ͯΔ •

    NAL͸ಈըετϦʔϜΛ͍ΖΜͳϝσΟΞͰ఻ૹ͠΍͍͢Α͏ʹந৅Խ͠ ͍ͯΔϨΠϠʔ • ؆୯ʹ͍͏ͱɺಈըετϦʔϜΛNAL UnitʢNALUʣͱݺ͹ΕΔͻͱ͔ͨ ·Γʹ੔ܗ͍ͯ͠Δ
  10. 20.

    H.264/AVC Bytestream Format (Annex B) • ΄ͱΜͲͷσίʔμ͸H.264/AVC Bytestream Format (Annex

    B)Λೖྗͱ ͯ͠͏͚͚ͭΔ • ISO/IEC 14496-10ͷAnnex Bʹࡌ͍ͬͯͨͷͰAnnex B formatͱݺ͹ΕΔ • جຊతʹ͸NALUΛ00 00 00 01Ͱܨ͛Ε͹Α͍
 ͭ·ΓɺRTPͷϖΠϩʔυΛܨ͛Ε͹͍͍ͩͨΑͦ͞͏ 00 00 00 01 NALU 00 00 00 01 NALU 00 00 00 01 NALU …
  11. 21.

    ෼ׂύέοτͱ݁߹ύέοτ • UDPͷϑϨʔϜαΠζ͸Ұൠతʹ͸1500byte͔͠ͳ͍ͷͰɺ݁߹ͨ͠Γɺ ෼ׂͯ͠ෳ਺ͷύέοτʹೖΕͨΓ͢Δ͜ͱ͕͋Δ • ݁߹ύέοτͱɺ෼ׂύέοτ͸NAL Unit typeͷ24-29Ͱఆٛ͞Ε͍ͯΔ NAL Unit

    Packet Packet Type Name Section Type Type ------------------------------------------------------------- 1-23 NAL unit Single NAL unit packet 5.6 24 STAP-A Single-time aggregation packet 5.7.1 25 STAP-B Single-time aggregation packet 5.7.1 26 MTAP16 Multi-time aggregation packet 5.7.2 27 MTAP24 Multi-time aggregation packet 5.7.2 28 FU-A Fragmentation unit 5.8 29 FU-B Fragmentation unit 5.8 https://tools.ietf.org/html/rfc6184
  12. 24.

    SDP (Session Description Protocol) RFC4566
 SDP: Session Description Protocol •

    ηογϣϯ৘ใΛදݱ͢ΔͨΊͷϓϩτίϧ • WebRTCΛ͸͡ΊɺʢWebRTCͷϕʔεʹͳͬͨʣSIPɺRTSPɺϝʔϧͳͲ ޿͘࢖ΘΕ͍ͯΔ • ಡΈͮΒ͍͜ͱͰ༗໊ʁ
  13. 25.

    ಡΈͮΒ͍ʁ v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar

    i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf e=j.doe@example.com (Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000 https://tools.ietf.org/html/rfc4566
  14. 26.

    ಡΈͮΒ͍ʁ v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar

    i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf e=j.doe@example.com (Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000 https://tools.ietf.org/html/rfc4566
  15. 27.

    Media Description "m=" m=<media> <port> <proto> <proto> <fmt> … <media>:

    ϝσΟΞͷछྨʢaudio, video,…)
 <port>: ૹ৴ݩϙʔτ
 <proto>: ϓϩτίϧ
 <fmt>: ϑΥʔϚοτ৘ใ ϝσΟΞͷछྨͱϑΥʔϚοτɺૹ৴ݩϙʔτؚ͕·Ε͍ͯΔ
 ͋·Γ࢖͑Δ৘ใ͸ͳ͍
  16. 29.

    SDP͔ΒRTPͷpayload type mappingΛऔΓग़͢ a=rtpmap:109 opus/48000/2
 a=rtpmap:120 H264/90000 • payload typeͷmapping͸rtpmapͰࢦఆ͞ΕΔ

    • ্ͷྫͩͱɺH.264ͷpayload type͸120ɺOpus͸109 • 2൪໨ͷ஋͸ϑΥʔϚοτ/प೾਺/νϟϯωϧ਺
 H.264/AVCͩͱ90000KHzͰλΠϜελϯϓ͕΍ͬͯ͘Δ͜ͱ͕Θ͔Δ