Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Build WebRTC MCU on Browser

mganeko
August 13, 2016

Build WebRTC MCU on Browser

Build MCU for WebRTC on Web browser.
Using WebRTC, Canvas, Web Audio.
Original presentation was made at WebRTC meetup Tokyo #11.
Youtube https://youtu.be/0RYWw1nPxio?t=50m41s

mganeko

August 13, 2016
Tweet

More Decks by mganeko

Other Decks in Technology

Transcript

  1. WebRTC Meetup Tokyo #11
    Build WebRTC MCU on Browser
    INFOCOM CORPORATION
    Masashi Ganeko
    @massie_g
    Aug. 8th, 2016
    1

    View full-size slide

  2. About me
    • Masashi Ganeko / @massie_g
    – working for a research team of INFOCOM CORPORATION
    • Writing a series of WebRTC on HTML5Experts.jp (HTML5 tech site)
    – https://html5experts.jp/series/webrtc2016/
    • English Presentation for WebRTC (2013, 2016)
    – https://speakerdeck.com/mganeko
    • Japanese Presentation for WebRTC (2013-2016)
    – http://www.slideshare.net/mganeko
    2

    View full-size slide

  3. Topic of today
    • MCU : Multipoint Control Unit
    – Used for Video Conference
    • Web Browser
    – WebRTC, Canvas, Web Audio
    3

    View full-size slide

  4. P2P vs. MCU
    4
    Browser
    A
    Browser
    B
    Browser
    D
    Browser
    C
    P2P
    • No media server
    • Browser side:
    • CPU load: Very High
    • Network load: Very High
    Browser
    A
    Browser
    B
    Browser
    D
    Browser
    C
    MCU
    Mix
    video / audio
    MCU
    • Use MCU server, CPU load: Super High
    • Browser side: very low CPU/Network load
    Good for
    Mobile

    View full-size slide

  5. Past trial:WebRTC Meetup Tokyo #4
    • Absurd trial with WebRTC +α (Oct. 2014)
    – (3) mix video absurd way
    – http://www.slideshare.net/mganeko/webrtc-meetup4-lt/24 (Page 21 - )
    5
    HDMI
    Capture
    unit
    Mix video with HDMI capture

    View full-size slide

  6. It is possible with only browser at 2016
    • Canvas captureStream()
    – Firefox 43- , Chrome 51-
    • Handle remote audio with Web Audio API
    – Firefox (around)40-, Chrome 49-
    6

    View full-size slide

  7. Browser MCU DEMO
    • YouTube - WebRTC Meetup Tokyo #11
    – https://youtu.be/0RYWw1nPxio?t=54m4s
    – 54:04 to 55:22
    7

    View full-size slide

  8. Browser MCU DEMO
    8
    Grid mix

    View full-size slide

  9. Inside MCU Server:Video
    9
    RTCPeerConnection A
    MediaStream

    element
    RTCPeerConnection D
    MediaStream
    element
    drawImage()
    Keep drawing with
    requestAnimationFrame()

    element
    drawImage()

    View full-size slide

  10. Inside MCU Server:Video
    10
    RTCPeerConnection A
    RTCPeerConnection D
    element
    MediaStream
    captureStream(fps)

    View full-size slide

  11. Inside MCU Server:Video
    11
    RTCPeerConnection A
    MediaStream

    element
    RTCPeerConnection D
    MediaStream
    element
    MediaStream
    drawImage()
    Keep drawing with
    requestAnimationFrame()

    element
    captureStream(fps)
    drawImage()

    View full-size slide

  12. Process of MCU Server:Video
    • Get Remote MediaStream through RTCPeerConnection
    – play in element
    • Get context of Canvas by getContext('2d')
    • Keep drawing with window.requestAnimationFrame()
    – Copy image of each video element to canvas, with drawImage()
    • Capture MadiaStream from Canvas, with captureStream(fps)
    – Pass it to RTCPeerConnection.addStream(), and send it to remote peer
    12

    View full-size slide

  13. Tricky points of MCU Server:Video
    • Using window.requestAnimationFrame()
    – Will not be called when the window/tab is completely hidden
    – Animation will stop if the window is minimized or other hidden
    by other tab
    • https://youtu.be/0RYWw1nPxio?t=56m44s to 57:17
    • Even using window.setInterval()
    – The interval will be much longer than specified, the window/tab
    is completely hidden
    13
    DEMO

    View full-size slide

  14. Tricky points of MCU Server:Audio
    • One mixed video is enough for all members
    • Audio should be prepared differently for each member
    – It is hand to speak, if your voice will come back with short delay
    – Maybe familiar case with WebRTC application developers
    – If it is not familiar to you, please try yourself
    • https://mganeko.github.io/webrtcexpjp/basic2016/camera_mic.html
    • Please use headphone, and laud volume
    • Click [Start] button
    • It is so hard to speak
    14
    DEMO

    View full-size slide

  15. Inside MCU Server:Audio
    • Prepare "minus one" Audio
    – Mix audio of all members, except the user
    • Need many combination of audio
    – 4 combination for 4 members
    – N combination for N members
    15
    Browser
    A
    Browser
    B
    Browser
    D
    Browser
    C
    MCU
    Mixing Audio
    is hassle
    Audio is much more
    difficult than Video

    View full-size slide

  16. Inside MCU Server:Audio
    16
    RTCPeerConnection A
    MediaStream
    create with AudioContext .
    createMediaStreamSource()
    MediaStreamAudioSourceNode

    View full-size slide

  17. Inside MCU Server:Audio
    17
    RTCPeerConnection D
    MediaStream
    MediaStreamAudioSourceNode MediaStreamAudioSourceNode
    MediaStreamAudioSourceNode
    MediaStreamAudioDestinationNode
    create with AudioContext .
    createMediaStreamDestination()
    mix

    View full-size slide

  18. Inside MCU Server:Audio
    18
    RTCPeerConnection A
    MediaStream
    RTCPeerConnection D
    MediaStream
    create with AudioContext .
    createMediaStreamSource()
    MediaStreamAudioSourceNode MediaStreamAudioSourceNode
    MediaStreamAudioSourceNode
    MediaStreamAudioDestinationNode
    create with AudioContext .
    createMediaStreamDestination()
    mix

    View full-size slide

  19. Process of MCU Servder:Audio
    • Get remote MediaStream through RTCPeerConnection
    • Create MediaStreamAudioSourceNode of Web Audio
    – Convert MediaStream to Web Audio node
    • Create MediaStreamAudioDestinationNode of Web Audio
    – Mix SourceNode of other members: sourceNode.connect(destinationNode);
    • Get MediaStream
    – Pass to RTCPeerConnection. addStream(), and send to remote peer
    • * repeat for all members
    19

    View full-size slide

  20. Inside of MCU Server:Video & Audio
    20
    RTCPeerConnection A
    RTCPeerConnection D
    MediaStream
    (Video A+B+C+D)
    MediaStream
    (Audio B+C+D)
    RTCPeerConnection B
    RTCPeerConnection C
    MediaStream
    (Audio A+C+D)
    MediaStream
    (Audio A+B+D)
    MediaStream
    (Audio A+B+C)
    Multi-Stream with
    Video only MediaStream
    and Audio only MediaStream

    View full-size slide

  21. Browser MCU DEMO 2
    • Canvas is So flexible !
    • YouTube - WebRTC Meetup Tokyo #11
    – https://youtu.be/0RYWw1nPxio?t=1h35s
    – 1:00:35 to 1:01:29
    21

    View full-size slide

  22. Browser MCU DEMO 2
    22
    Stripe Variable Zoom

    View full-size slide

  23. Browser MCU DEMO 3
    • 3D with Canvas + WebGL
    • YouTube - WebRTC Meetup Tokyo #11
    – https://youtu.be/0RYWw1nPxio?t=1h1m51s
    – 1:01:51 to 1:02:21
    23

    View full-size slide

  24. Browser MCU DEMO 3
    24
    3D with WebGL

    View full-size slide

  25. Browser MCU DEMO 3
    25
    Stripe with 7 members Grid with 17 members
    Frame was dropped, in case of over 16 members
    Too heavy load, even for core-i7 CPU

    View full-size slide

  26. Browser MCU DEMO 4
    26
    • Record mix video/audio with MediaRecorder
    • YouTube - WebRTC Meetup Tokyo #11
    – https://youtu.be/0RYWw1nPxio?t=1h3m47s
    – 1:03:47 to 1:04:33

    View full-size slide

  27. Inside of MCU Server: Recording
    27
    MediaStream
    (Video A+B+C+D)
    element Same as sending to remote peers
    Web Audio API
    MediaStream
    (Audio A+B+C+D)
    Mix all audio
    for recording
    videoTracks[] MediaStreamTrack
    MediaStreamTrack
    aidioTracks[]
    MediaStream
    new MediaStream()
    MediaStream.addTrack()
    MediaStream.addTrack()
    MediaRecorder
    WebM

    View full-size slide

  28. MCU vs. SFU
    28

    View full-size slide

  29. SFU: Selective Forwarding Unit
    29
    Browser
    A
    Browser
    B
    Browser
    D
    Browser
    C
    SFU
    Dispatch
    audio/video
    SFU
    • User SFU server, CPU load: Low
    • Browser side: low CPU load
    • Browser side: network load: middle
    Able to choose
    display layout
    on member side ◎
    Browser
    A
    Browser
    B
    Browser
    D
    Browser
    C
    MCU
    Mix
    video / audio
    MCU
    • Use MCU server, CPU load: Super High
    • Browser side: very low CPU/Network load

    View full-size slide

  30. Browser MCU DEMO 5
    • It is NOT possible to choose display layout on member
    side, with MCU
    • ... NO, it is possible!
    • YouTube - WebRTC Meetup Tokyo #11
    – https://youtu.be/0RYWw1nPxio?t=1h6m49s
    – 1:06:49 to 1:07:25
    30

    View full-size slide

  31. Browser MCU DEMO 5
    31
    Zooming on User/Member side

    View full-size slide

  32. Inside of zooming on MCU member
    32
    MediaStream
    (Video A+B+C+D)
    hidden

    element
    drawImage()
    drawImage()
    drawImage()
    drawImage()
    It is possible to change layout with Canvas in member side
    (It is possible to use even multiple canvas)

    View full-size slide

  33. Appendix:painful point of MCU
    • Adjusting timing is painful for commercial MCU
    – Adjust timing for video of all members
    – Adjust timing of video and audio (lip sync)
    • For adjusting timing, it is need to wait for frame or block
    – But, impossible to wait forever
    – So it is necessary to timeout and send alternative frame/block
    • Sending Black frame, or repeat last frame
    • Sending silent block
    • Browser MCU does not care about adjusting timing
    33

    View full-size slide

  34. Appendix:MCU Sample page
    • Start page (not http, but https)
    – https://mcu.talkin.info/
    • Prepare a room as MCU server
    – Type room name
    – Click [Prepare a Room as MCU Server]
    – Wait for connection, or click [Re-Connect]
    • Join the room
    – Type room name, then click [Join to Room]
    – Click [Start video] to use camera/microphone
    – Click [Connect]
    34

    View full-size slide

  35. Conclusion
    • Potential of WebRTC + Canvas is unlimited
    – It is very powerful toy
    – It is possible to send local video file
    • http://qiita.com/massie_g/items/5a6c4b69374d5997dc37
    • But, CPU load of server side is super heavy
    – It is not scalable
    • Maybe useful, if user can take care of server side browser
    • Original Japanese Slide is on SlideShare
    – http://www.slideshare.net/mganeko/webrtc-build-mcu-on-browser
    35

    View full-size slide

  36. Thank you!
    36

    View full-size slide