Upgrade to Pro — share decks privately, control downloads, hide ads and more …

WebRTC動画をトランスコードする / Transcoding video streams from WebRTC

Harukasan
February 05, 2018

WebRTC動画をトランスコードする / Transcoding video streams from WebRTC

SkyWay UG Tokyo #2
2018-02-05

Harukasan

February 05, 2018
Tweet

More Decks by Harukasan

Other Decks in Technology

Transcript

  1. Transcoding video streams

    from WebRTC
    MICHII Shunsuke pixiv Inc.
    SkyWay UG Tokyo #2

    2018-02-05

    View Slide

  2. MICHII Shunsuke | @harukasan
    ImageFluxࣄۀ෦ ෦௕

    ϐΫγϒςΫϊϩδʔζגࣜձࣾࣥߦ໾һ

    ImageFluxͰ͸GoΛॻ͍ͨΓΠϯϑϥपΓΛ΍ͬͨΓͱ͔͍Ζ͍Ζ

    View Slide

  3. ࠓ೔͸ͳ͢͜ͱ
    • WebRTCͰૹΒΕͯ͘ΔH.264/AVCಈըετϦʔϜΛऔΓग़͢ํ๏

    • H.264/AVCಈըετϦʔϜ͸ͲͷΑ͏ʹૹΒΕͯ͘Δͷ͔

    View Slide

  4. ࠓ೔͸ͳ͞ͳ͍͜ͱ
    • WebRTCͱ͸ͳʹ͔

    • WebRTCͰಈըΛετϦʔϛϯά͢Δํ๏

    • WebRTC SFU Sora͕΍ͬͯ͘ΕΔ͜ͱ

    ઀ଓཱ֬ɺηογϣϯͷ؅ཧɺSRTPͷ҉߸Խղআ……

    • SkyWayͰͲ͏΍ͬͯ΍Δͷ͔……ʢRTPΛऔΓग़ͤΕ͹େৎ෉ʣ

    • H.264/AVCҎ֎ͷVP9, HEVC, AV1ͱ͍ͬͨಈըϑΥʔϚοτʹ͍ͭͯ

    View Slide

  5. ͳͥWebRTCಈըΛτϥϯείʔυ͢Δͷ͔
    • WebRTCಈըΛHLS഑৴͢ΔαʔϏεΛ։ൃதʂ

    https://speakerdeck.com/imageflux/imageflux-live

    • pixiv Sketch LiveͷόοΫΤϯυͱͯ͠Քಇத

    • αʔϏεʹڵຯ͋Δํ͸ੋඇ͓੠͕͚͍ͩ͘͞ʂ

    ΤϯδχΞ΋ืू͍ͯ͠·͢ʂ

    View Slide

  6. ImageFlux Liveͷ࢓૊Έ
    Transcoder
    Transcoder
    Transcoder
    RTP stream
    Streamerd
    Msgpack

    stream
    TS
    TS
    TS
    ഑৴ऀ WebRTC SFU

    Sora
    WebRTC
    API
    H.264 bytestreamΛτϥϯείʔυ͢Δ
    RTPετϦʔϜΛσίʔυͯ͠
    H.264 bytestream formatʹͯ͠
    msgpackʹͭΊΔ

    View Slide

  7. H.264 bytestreamΛτϥϯείʔυ͢Δ
    ImageFlux Liveͷ࢓૊Έ
    Transcoder
    Transcoder
    Transcoder
    RTP stream
    Msgpack

    stream
    TS
    TS
    TS
    ഑৴ऀ WebRTC SFU

    Sora
    WebRTC
    API
    RTPετϦʔϜΛσίʔυͯ͠
    H.264 bytestream formatʹͯ͠
    msgpackʹͭΊΔ
    Streamerd

    View Slide

  8. WebRTCͰ͸ԿΛ௨৴͍ͯ͠Δͷ͔
    Client
    WebRTC
    Client
    ηογϣϯ؅ཧ
    ϦΞϧλΠϜετϦʔϛϯά
    ઀ଓཱ֬ʢNATӽ͑ʣ

    View Slide

  9. WebRTCͰ͸ԿΛ௨৴͍ͯ͠Δͷ͔
    Client
    WebRTC
    Client
    ηογϣϯ؅ཧ
    ϦΞϧλΠϜετϦʔϛϯά
    ઀ଓཱ֬ʢNATӽ͑ʣ

    View Slide

  10. WebRTCͰ͸ԿΛ௨৴͍ͯ͠Δͷ͔
    Client
    WebRTC
    Client
    SDP
    SRTP
    ηογϣϯ৘ใ
    ಈըετϦʔϜ

    View Slide

  11. WebRTCͰૹΒΕͯ͘ΔಈըΛ

    τϥϯείʔυ͢Δʹ͸
    1. ΫϥΠΞϯτͱηογϣϯ৘ใʢSDPʣΛ΍ΓͱΓ͢Δ

    2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ

    3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ

    4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ

    View Slide

  12. WebRTCͰૹΒΕͯ͘ΔಈըΛ

    τϥϯείʔυ͢Δʹ͸
    1. ΫϥΠΞϯτͱηογϣϯ৘ใʢSDPʣΛ΍ΓͱΓ͢Δ

    2. ҉߸Խ͞Ε͍ͯΔετϦʔϜʢSRTPʣΛ҉߸Խղআ͢Δ

    3. ಈըετϦʔϜʢRTPʣΛσίʔυ͢Δ

    4. औΓग़ͨ͠H.264ετϦʔϜΛτϥϯείʔυ͢Δ
    1, 2͸WebRTC SFU Sora͕΍ͬͯ͘Ε·͢

    View Slide

  13. RTP (Realtime Transfer Protocol)
    RFC3550

    RTP: A Transport Protocol for Real-Time Applications
    • Ի੠ɺಈըͳͲͷσʔλΛϦΞϧλΠϜʹసૹ͢ΔͨΊͷϓϩτίϧ
    • UDP্ʹ࣮૷͞Ε͍ͯΔ
    • TCPΛ࢖͍ͬͯͳ͍ͷͰɺॱং੍ޚ΍᫔᫓੍ޚΛRTPϨΠϠͰ࣮૷Ͱ͖Δ

    →ݴ͍׵͑Ε͹ࣗ෼Ͱ࣮૷͢Δඞཁ͕͋Δ
    • SRTP͸ʢͬ͘͟Γݴ͑͹ʣRTPΛ҉߸Խͨ͠΋ͷ

    View Slide

  14. ࣮ମ͸UDPͷബ͍ϥούʔ
    • UDPʹബ͍ϔομ͕͍͍ͭͯΔ͚ͩ

    • ϖΠϩʔυͷූ߸Խํࣜ͸ϖΠϩʔυͷछྨ͝ͱʹҟͳΔ
    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
    V P X CC M payload type sequence number
    timestamp
    synchronization source (SSRC) identifier
    contributing source (CSRC) identifier
    payload

    ….

    View Slide

  15. ϖΠϩʔυͷ൑ผํ๏
    • ϖΠϩʔυͷϑΥʔϚοτ͸payload typeͰ൑ผ͢Δ

    • payload type͸7bit͔͠ͳ͘ɺ͙͢ʹ଍Γͳ͘ͳͬͨͷͰɺݱࡏͰ͸

    96-127ΛಈతʹׂΓ౰ͯͯ࢖͏ʢRFC3551ʣ

    • payload typeͱϑΥʔϚοτͷϚοϐϯά͸SDPͷattributeͱͯ͠

    ΍ΓͱΓ͞ΕΔ

    • طʹొ࿥͞Ε͍ͯΔͷ͸IANAͰ؅ཧ͞Ε͍ͯΔ

    https://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml

    View Slide

  16. ॱংอূ
    • RTP͸UDPͰૹΒΕͯ͘ΔͷͰॱ൪อূ͞Ε͍ͯͳ͍

    • RTPϔομʹ͸γʔέϯε൪߸ʢsequence numberʣؚ͕·Ε͍ͯΔͷ
    Ͱ͜ΕΛ࢖ͬͯιʔτ͢Δඞཁ͕͋Δ

    • ;ͭ͏ʹ300ݸ͙Β͍ޙΕͯύέοτ͕ඈΜͰ͖ͨΓͯ͠ਏ͍……͕

    ͋·Γ଴ͭͱ஗ԆͷݪҼʹͳΔͷͰέʔεόΠέʔε

    • ͍·ͷ࣮૷Ͱ͸ύέοτ͕ඈΜͩ৔߹͸100ݸ͘Β͍଴ͭΑ͏ʹ͍ͯ͠Δ

    View Slide

  17. ϑΥʔϚοτ͝ͱʹϖΠϩʔυͷܗࣜ͸ҟͳΔ
    • ͦΕͧΕҟͳΔRFCͰඪ४Խ͞Ε͍ͯΔ

    RFC6184

    RTP Payload Format for H.264 Video

    RFC7587

    RTP Payload Format for the Opus Speech and Audio Codec

    draft-uberti-payload-vp9-01

    RTP Payload Format for VP9 Video

    View Slide

  18. RTP Payload Format for H.264 Video
    • H.264 NALU (NAL Unitʣ1͕ͭ1ͭͷUDPύέοτʹؚ·ΕΔ
    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
    V P X CC M payload type sequence number
    timestamp
    synchronization source (SSRC) identifier
    contributing source (CSRC) identifier
    NAL Header Payload….

    View Slide

  19. H.264/AVC NAL (Network Abstraction Layer)
    • H.264/AVCͰ͸NALʢNetwork Abstraction Layerʣ͕ఆٛ͞Ε͍ͯΔ

    • NAL͸ಈըετϦʔϜΛ͍ΖΜͳϝσΟΞͰ఻ૹ͠΍͍͢Α͏ʹந৅Խ͠
    ͍ͯΔϨΠϠʔ

    • ؆୯ʹ͍͏ͱɺಈըετϦʔϜΛNAL UnitʢNALUʣͱݺ͹ΕΔͻͱ͔ͨ
    ·Γʹ੔ܗ͍ͯ͠Δ

    View Slide

  20. H.264/AVC Bytestream Format (Annex B)
    • ΄ͱΜͲͷσίʔμ͸H.264/AVC Bytestream Format (Annex B)Λೖྗͱ
    ͯ͠͏͚͚ͭΔ

    • ISO/IEC 14496-10ͷAnnex Bʹࡌ͍ͬͯͨͷͰAnnex B formatͱݺ͹ΕΔ

    • جຊతʹ͸NALUΛ00 00 00 01Ͱܨ͛Ε͹Α͍

    ͭ·ΓɺRTPͷϖΠϩʔυΛܨ͛Ε͹͍͍ͩͨΑͦ͞͏
    00 00 00 01 NALU 00 00 00 01 NALU 00 00 00 01 NALU …

    View Slide

  21. ෼ׂύέοτͱ݁߹ύέοτ
    • UDPͷϑϨʔϜαΠζ͸Ұൠతʹ͸1500byte͔͠ͳ͍ͷͰɺ݁߹ͨ͠Γɺ
    ෼ׂͯ͠ෳ਺ͷύέοτʹೖΕͨΓ͢Δ͜ͱ͕͋Δ

    • ݁߹ύέοτͱɺ෼ׂύέοτ͸NAL Unit typeͷ24-29Ͱఆٛ͞Ε͍ͯΔ
    NAL Unit Packet Packet Type Name Section
    Type Type
    -------------------------------------------------------------
    1-23 NAL unit Single NAL unit packet 5.6
    24 STAP-A Single-time aggregation packet 5.7.1
    25 STAP-B Single-time aggregation packet 5.7.1
    26 MTAP16 Multi-time aggregation packet 5.7.2
    27 MTAP24 Multi-time aggregation packet 5.7.2
    28 FU-A Fragmentation unit 5.8
    29 FU-B Fragmentation unit 5.8
    https://tools.ietf.org/html/rfc6184

    View Slide

  22. ෼ׂͨ͠Γ݁߹ͨ͠Γ͢Δ
    • ඇΠϯλʔϦʔϒϞʔυͰ͸STAP-AͱFU-A͕࢖ΘΕΔͷͰ͜ͷ2ݸ͚ͩ
    ରԠ͢Ε͹Α͍

    • STAP-A: ෳ਺ͷNALUΛαΠζΛઌ಄ʹ͚ͭͯ݁߹ͨ͠ύέοτ

    • FU-A: ෳ਺ͷNALUΛ෼ׂ͠ɺ֤ύέοτʹFUϔομΛ͚ͭͨ΋ͷ

    جຊతʹ͸start bit (s)͕͍͍ͭͯΔύέοτ͔Βend bit (e)͕͍͍ͭͯΔύ
    έοτ·Ͱ݁߹͢Δ
    0 1 2 3 4 5 6 7 8
    S E - Type
    FUϔομ

    View Slide

  23. RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ͸
    1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢

    2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ

    3. ݁߹ύέοτͩͬͨ৔߹͸෼ׂͯ͠ෳ਺ͷόΠτετϦʔϜʹ͢Δ

    4. ෼ׂύέοτͩͬͨ৔߹͸શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬ
    ͯ1ͭͷNALUΛऔΓग़͢

    View Slide

  24. SDP (Session Description Protocol)
    RFC4566

    SDP: Session Description Protocol
    • ηογϣϯ৘ใΛදݱ͢ΔͨΊͷϓϩτίϧ
    • WebRTCΛ͸͡ΊɺʢWebRTCͷϕʔεʹͳͬͨʣSIPɺRTSPɺϝʔϧͳͲ
    ޿͘࢖ΘΕ͍ͯΔ
    • ಡΈͮΒ͍͜ͱͰ༗໊ʁ

    View Slide

  25. ಡΈͮΒ͍ʁ
    v=0
    o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5
    s=SDP Seminar
    i=A Seminar on the session description protocol
    u=http://www.example.com/seminars/sdp.pdf
    [email protected] (Jane Doe)
    c=IN IP4 224.2.17.12/127
    t=2873397496 2873404696
    a=recvonly
    m=audio 49170 RTP/AVP 0
    m=video 51372 RTP/AVP 99
    a=rtpmap:99 h263-1998/90000
    https://tools.ietf.org/html/rfc4566

    View Slide

  26. ಡΈͮΒ͍ʁ
    v=0
    o=jdoe 2890844526 2890842807 IN IP4 10.47.16.5
    s=SDP Seminar
    i=A Seminar on the session description protocol
    u=http://www.example.com/seminars/sdp.pdf
    [email protected] (Jane Doe)
    c=IN IP4 224.2.17.12/127
    t=2873397496 2873404696
    a=recvonly
    m=audio 49170 RTP/AVP 0
    m=video 51372 RTP/AVP 99
    a=rtpmap:99 h263-1998/90000
    https://tools.ietf.org/html/rfc4566

    View Slide

  27. Media Description "m="
    m= …
    : ϝσΟΞͷछྨʢaudio, video,…)

    : ૹ৴ݩϙʔτ

    : ϓϩτίϧ

    : ϑΥʔϚοτ৘ใ
    ϝσΟΞͷछྨͱϑΥʔϚοτɺૹ৴ݩϙʔτؚ͕·Ε͍ͯΔ

    ͋·Γ࢖͑Δ৘ใ͸ͳ͍

    View Slide

  28. Attributes "a="
    a=:
    : ଐੑ

    : ͦͷ஋
    • SDPʹ஋Λ֦ு͢ΔͨΊʹ༻ҙ͞Ε͍ͯΔύϥϝʔλ
    • WebRTCͰ֦ு͞Ε͍ͯΔ஋͸͍͍ͩͨa=ʹೖ͍ͬͯΔ

    View Slide

  29. SDP͔ΒRTPͷpayload type mappingΛऔΓग़͢
    a=rtpmap:109 opus/48000/2

    a=rtpmap:120 H264/90000
    • payload typeͷmapping͸rtpmapͰࢦఆ͞ΕΔ

    • ্ͷྫͩͱɺH.264ͷpayload type͸120ɺOpus͸109

    • 2൪໨ͷ஋͸ϑΥʔϚοτ/प೾਺/νϟϯωϧ਺

    H.264/AVCͩͱ90000KHzͰλΠϜελϯϓ͕΍ͬͯ͘Δ͜ͱ͕Θ͔Δ

    View Slide

  30. RTP͔ΒH.264 byte streamΛσίʔυ͢Δʹ͸
    1. SDPΛಡΜͰpayload typeͱϑΥʔϚοτͷϚοϐϯάΛऔΓग़͢

    2. RTPύέοτΛόοϑΝϦϯάͯ͠ɺγʔέϯε൪߸ॱʹฒͼସ͑Δ

    3. ݁߹ύέοτͩͬͨ৔߹͸෼ׂͯ͠ෳ਺ͷόΠτετϦʔϜʹ͢Δ

    4. ෼ׂύέοτͩͬͨ৔߹͸શύέοτ͕όοϑΝϦϯά͞ΕΔͷΛ·ͬ
    ͯ1ͭͷNALUΛऔΓग़͢
    ෮श

    View Slide

  31. ·ͱΊ
    • WebRTCಈըετϦʔϜΛτϥϯείʔυ͢Δʹ͸ɺ

    RTPύέοτΛσίʔυͰ͖Ε͹Α͍

    • αʔόαΠυͰRTPΛऔΓग़͢ʹ͸WebRTC SFU Sora͕ศར

    • ImageFlux Live։ൃதͰ͢ʂ

    View Slide