WebRTC動画をトランスコードする / Transcoding video streams from WebRTC

Transcript

1. Transcoding video streams
 from WebRTC MICHII Shunsuke pixiv Inc. SkyWay

   UG Tokyo #2
 2018-02-05

2. MICHII Shunsuke | @harukasan ImageFluxࣄۀ෦ ෦௕
 ϐΫγϒςΫϊϩδʔζגࣜձࣾࣥߦ໾һ ImageFluxͰ͸GoΛॻ͍ͨΓΠϯϑϥपΓΛ΍ͬͨΓͱ͔͍Ζ͍Ζ

3. ࠓ೔͸ͳ͢͜ͱ • WebRTCͰૹΒΕͯ͘ΔH.264/AVCಈըετϦʔϜΛऔΓग़͢ํ๏ • H.264/AVCಈըετϦʔϜ͸ͲͷΑ͏ʹૹΒΕͯ͘Δͷ͔

4. ࠓ೔͸ͳ͞ͳ͍͜ͱ • WebRTCͱ͸ͳʹ͔ • WebRTCͰಈըΛετϦʔϛϯά͢Δํ๏ • WebRTC SFU Sora͕΍ͬͯ͘ΕΔ͜ͱ
 ઀ଓཱ֬ɺηογϣϯͷ؅ཧɺSRTPͷ҉߸Խղআ……

   • SkyWayͰͲ͏΍ͬͯ΍Δͷ͔……ʢRTPΛऔΓग़ͤΕ͹େৎ෉ʣ • H.264/AVCҎ֎ͷVP9, HEVC, AV1ͱ͍ͬͨಈըϑΥʔϚοτʹ͍ͭͯ

5. ͳͥWebRTCಈըΛτϥϯείʔυ͢Δͷ͔ • WebRTCಈըΛHLS഑৴͢ΔαʔϏεΛ։ൃதʂ
 https://speakerdeck.com/imageflux/imageflux-live • pixiv Sketch LiveͷόοΫΤϯυͱͯ͠Քಇத • αʔϏεʹڵຯ͋Δํ͸ੋඇ͓੠͕͚͍ͩ͘͞ʂ


   ΤϯδχΞ΋ืू͍ͯ͠·͢ʂ

6. ImageFlux Liveͷ࢓૊Έ Transcoder Transcoder Transcoder RTP stream Streamerd Msgpack
 stream

   TS TS TS ഑৴ऀ WebRTC SFU
 Sora WebRTC API H.264 bytestreamΛτϥϯείʔυ͢Δ RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ

7. H.264 bytestreamΛτϥϯείʔυ͢Δ ImageFlux Liveͷ࢓૊Έ Transcoder Transcoder Transcoder RTP stream Msgpack


   stream TS TS TS ഑৴ऀ WebRTC SFU
 Sora WebRTC API RTPετϦʔϜΛσίʔυͯ͠ H.264 bytestream formatʹͯ͠ msgpackʹͭΊΔ Streamerd

8. WebRTCͰ͸ԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client ηογϣϯ؅ཧ ϦΞϧλΠϜετϦʔϛϯά ઀ଓཱ֬ʢNATӽ͑ʣ

9. WebRTCͰ͸ԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client ηογϣϯ؅ཧ ϦΞϧλΠϜετϦʔϛϯά ઀ଓཱ֬ʢNATӽ͑ʣ

10. WebRTCͰ͸ԿΛ௨৴͍ͯ͠Δͷ͔ Client WebRTC Client SDP SRTP ηογϣϯ৘ใ ಈըετϦʔϜ

11. WebRTCͰૹΒΕͯ͘ΔಈըΛ
 τϥϯείʔυ͢Δʹ͸ 1. ΫϥΠΞϯτͱηογϣϯ৘ใʢSDPʣΛ΍ΓͱΓ͢Δ 2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ 3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ 4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ

12. WebRTCͰૹΒΕͯ͘ΔಈըΛ
 τϥϯείʔυ͢Δʹ͸ 1. ΫϥΠΞϯτͱηογϣϯ৘ใʢSDPʣΛ΍ΓͱΓ͢Δ 2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ 3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ 4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ

    1, 2͸WebRTC SFU Sora͕΍ͬͯ͘Ε·͢

13. RTP (Realtime Transfer Protocol) RFC3550
 RTP: A Transport Protocol for

    Real-Time Applications • Ի੠ɺಈըͳͲͷσʔλΛϦΞϧλΠϜʹసૹ͢ΔͨΊͷϓϩτίϧ • UDP্ʹ࣮૷͞Ε͍ͯΔ • TCPΛ࢖͍ͬͯͳ͍ͷͰɺॱং੍ޚ΍᫔᫓੍ޚΛRTPϨΠϠͰ࣮૷Ͱ͖Δ
 →ݴ͍׵͑Ε͹ࣗ෼Ͱ࣮૷͢Δඞཁ͕͋Δ • SRTP͸ʢͬ͘͟Γݴ͑͹ʣRTPΛ҉߸Խͨ͠΋ͷ

14. ࣮ମ͸UDPͷബ͍ϥούʔ • UDPʹബ͍ϔομ͕͍͍ͭͯΔ͚ͩ • ϖΠϩʔυͷූ߸Խํࣜ͸ϖΠϩʔυͷछྨ͝ͱʹҟͳΔ 0 1 2 3 4

    5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identifier contributing source (CSRC) identifier payload
 ….

15. ϖΠϩʔυͷ൑ผํ๏ • ϖΠϩʔυͷϑΥʔϚοτ͸payload typeͰ൑ผ͢Δ • payload type͸7bit͔͠ͳ͘ɺ͙͢ʹ଍Γͳ͘ͳͬͨͷͰɺݱࡏͰ͸
 96-127ΛಈతʹׂΓ౰ͯͯ࢖͏ʢRFC3551ʣ • payload

    typeͱϑΥʔϚοτͷϚοϐϯά͸SDPͷattributeͱͯ͠
 ΍ΓͱΓ͞ΕΔ • طʹొ࿥͞Ε͍ͯΔͷ͸IANAͰ؅ཧ͞Ε͍ͯΔ
 https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml

16. ॱংอূ • RTP͸UDPͰૹΒΕͯ͘ΔͷͰॱ൪อূ͞Ε͍ͯͳ͍ • RTPϔομʹ͸γʔέϯε൪߸ʢsequence numberʣؚ͕·Ε͍ͯΔͷ Ͱ͜ΕΛ࢖ͬͯιʔτ͢Δඞཁ͕͋Δ • ;ͭ͏ʹ300ݸ͙Β͍ޙΕͯύέοτ͕ඈΜͰ͖ͨΓͯ͠ਏ͍……͕
 ͋·Γ଴ͭͱ஗ԆͷݪҼʹͳΔͷͰέʔεόΠέʔε

    • ͍·ͷ࣮૷Ͱ͸ύέοτ͕ඈΜͩ৔߹͸100ݸ͘Β͍଴ͭΑ͏ʹ͍ͯ͠Δ

17. ϑΥʔϚοτ͝ͱʹϖΠϩʔυͷܗࣜ͸ҟͳΔ • ͦΕͧΕҟͳΔRFCͰඪ४Խ͞Ε͍ͯΔ RFC6184
 RTP Payload Format for H.264 Video

    RFC7587
 RTP Payload Format for the Opus Speech and Audio Codec draft-uberti-payload-vp9-01
 RTP Payload Format for VP9 Video

18. RTP Payload Format for H.264 Video • H.264 NALU (NAL

    Unitʣ1͕ͭ1ͭͷUDPύέοτʹؚ·ΕΔ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 V P X CC M payload type sequence number timestamp synchronization source (SSRC) identifier contributing source (CSRC) identifier NAL Header Payload….

19. H.264/AVC NAL (Network Abstraction Layer) • H.264/AVCͰ͸NALʢNetwork Abstraction Layerʣ͕ఆٛ͞Ε͍ͯΔ •

    NAL͸ಈըετϦʔϜΛ͍ΖΜͳϝσΟΞͰ఻ૹ͠΍͍͢Α͏ʹந৅Խ͠ ͍ͯΔϨΠϠʔ • ؆୯ʹ͍͏ͱɺಈըετϦʔϜΛNAL UnitʢNALUʣͱݺ͹ΕΔͻͱ͔ͨ ·Γʹ੔ܗ͍ͯ͠Δ

20. H.264/AVC Bytestream Format (Annex B) • ΄ͱΜͲͷσίʔμ͸H.264/AVC Bytestream Format (Annex

    B)Λೖྗͱ ͯ͠͏͚͚ͭΔ • ISO/IEC 14496-10ͷAnnex Bʹࡌ͍ͬͯͨͷͰAnnex B formatͱݺ͹ΕΔ • جຊతʹ͸NALUΛ00 00 00 01Ͱܨ͛Ε͹Α͍
 ͭ·ΓɺRTPͷϖΠϩʔυΛܨ͛Ε͹͍͍ͩͨΑͦ͞͏ 00 00 00 01 NALU 00 00 00 01 NALU 00 00 00 01 NALU …

21. ෼ׂύέοτͱ݁߹ύέοτ • UDPͷϑϨʔϜαΠζ͸Ұൠతʹ͸1500byte͔͠ͳ͍ͷͰɺ݁߹ͨ͠Γɺ ෼ׂͯ͠ෳ਺ͷύέοτʹೖΕͨΓ͢Δ͜ͱ͕͋Δ • ݁߹ύέοτͱɺ෼ׂύέοτ͸NAL Unit typeͷ24-29Ͱఆٛ͞Ε͍ͯΔ NAL Unit

    Packet Packet Type Name Section Type Type ------------------------------------------------------------- 1-23 NAL unit Single NAL unit packet 5.6 24 STAP-A Single-time aggregation packet 5.7.1 25 STAP-B Single-time aggregation packet 5.7.1 26 MTAP16 Multi-time aggregation packet 5.7.2 27 MTAP24 Multi-time aggregation packet 5.7.2 28 FU-A Fragmentation unit 5.8 29 FU-B Fragmentation unit 5.8 https://tools.ietf.org/html/rfc6184

22. ෼ׂͨ͠Γ݁߹ͨ͠Γ͢Δ • ඇΠϯλʔϦʔϒϞʔυͰ͸STAP-AͱFU-A͕࢖ΘΕΔͷͰ͜ͷ2ݸ͚ͩ ରԠ͢Ε͹Α͍ • STAP-A: ෳ਺ͷNALUΛαΠζΛઌ಄ʹ͚ͭͯ݁߹ͨ͠ύέοτ • FU-A: ෳ਺ͷNALUΛ෼ׂ͠ɺ֤ύέοτʹFUϔομΛ͚ͭͨ΋ͷ


    جຊతʹ͸start bit (s)͕͍͍ͭͯΔύέοτ͔Βend bit (e)͕͍͍ͭͯΔύ έοτ·Ͱ݁߹͢Δ 0 1 2 3 4 5 6 7 8 S E - Type FUϔομ

23. RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ͸ 1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢ 2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ 3. ݁߹ύέοτͩͬͨ৔߹͸෼ׂͯ͠ෳ਺ͷόΠτετϦʔϜʹ͢Δ

    4. ෼ׂύέοτͩͬͨ৔߹͸શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬ ͯ1ͭͷNALUΛऔΓग़͢

24. SDP (Session Description Protocol) RFC4566
 SDP: Session Description Protocol •

    ηογϣϯ৘ใΛදݱ͢ΔͨΊͷϓϩτίϧ • WebRTCΛ͸͡ΊɺʢWebRTCͷϕʔεʹͳͬͨʣSIPɺRTSPɺϝʔϧͳͲ ޿͘࢖ΘΕ͍ͯΔ • ಡΈͮΒ͍͜ͱͰ༗໊ʁ

25. ಡΈͮΒ͍ʁ v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar

    i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf e=j.doe@example.com (Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000 https://tools.ietf.org/html/rfc4566

26. ಡΈͮΒ͍ʁ v=0 o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5 s=SDP Seminar

    i=A Seminar on the session description protocol u=http://www.example.com/seminars/sdp.pdf e=j.doe@example.com (Jane Doe) c=IN IP4 224.2.17.12/127 t=2873397496 2873404696 a=recvonly m=audio 49170 RTP/AVP 0 m=video 51372 RTP/AVP 99 a=rtpmap:99 h263-1998/90000 https://tools.ietf.org/html/rfc4566

27. Media Description "m=" m=<media> <port> <proto> <proto> <fmt> … <media>:

    ϝσΟΞͷछྨʢaudio, video,…)
 <port>: ૹ৴ݩϙʔτ
 <proto>: ϓϩτίϧ
 <fmt>: ϑΥʔϚοτ৘ใ ϝσΟΞͷछྨͱϑΥʔϚοτɺૹ৴ݩϙʔτؚ͕·Ε͍ͯΔ
 ͋·Γ࢖͑Δ৘ใ͸ͳ͍

28. Attributes "a=" a=<attribute>:<value> <attribute>: ଐੑ
 <value>: ͦͷ஋ • SDPʹ஋Λ֦ு͢ΔͨΊʹ༻ҙ͞Ε͍ͯΔύϥϝʔλ •

    WebRTCͰ֦ு͞Ε͍ͯΔ஋͸͍͍ͩͨa=ʹೖ͍ͬͯΔ

29. SDP͔ΒRTPͷpayload type mappingΛऔΓग़͢ a=rtpmap:109 opus/48000/2
 a=rtpmap:120 H264/90000 • payload typeͷmapping͸rtpmapͰࢦఆ͞ΕΔ

    • ্ͷྫͩͱɺH.264ͷpayload type͸120ɺOpus͸109 • 2൪໨ͷ஋͸ϑΥʔϚοτ/प೾਺/νϟϯωϧ਺
 H.264/AVCͩͱ90000KHzͰλΠϜελϯϓ͕΍ͬͯ͘Δ͜ͱ͕Θ͔Δ

30. RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ͸ 1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢ 2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ 3. ݁߹ύέοτͩͬͨ৔߹͸෼ׂͯ͠ෳ਺ͷόΠτετϦʔϜʹ͢Δ

    4. ෼ׂύέοτͩͬͨ৔߹͸શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬ ͯ1ͭͷNALUΛऔΓग़͢ ෮श

31. ·ͱΊ • WebRTCಈըετϦʔϜΛτϥϯείʔυ͢Δʹ͸ɺ
 RTPύέοτΛσίʔυͰ͖Ε͹Α͍ • αʔόαΠυͰRTPΛऔΓग़͢ʹ͸WebRTC SFU Sora͕ศར • ImageFlux

    Live։ൃதͰ͢ʂ