Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Build WebRTC MCU on Browser

2836c80ba2ab0c0a0fcd820c713b106d?s=47 mganeko
August 13, 2016

Build WebRTC MCU on Browser

Build MCU for WebRTC on Web browser.
Using WebRTC, Canvas, Web Audio.
Original presentation was made at WebRTC meetup Tokyo #11.
Youtube https://youtu.be/0RYWw1nPxio?t=50m41s

2836c80ba2ab0c0a0fcd820c713b106d?s=128

mganeko

August 13, 2016
Tweet

Transcript

  1. WebRTC Meetup Tokyo #11 Build WebRTC MCU on Browser INFOCOM

    CORPORATION Masashi Ganeko @massie_g Aug. 8th, 2016 1
  2. About me • Masashi Ganeko / @massie_g – working for

    a research team of INFOCOM CORPORATION • Writing a series of WebRTC on HTML5Experts.jp (HTML5 tech site) – https://html5experts.jp/series/webrtc2016/ • English Presentation for WebRTC (2013, 2016) – https://speakerdeck.com/mganeko • Japanese Presentation for WebRTC (2013-2016) – http://www.slideshare.net/mganeko 2
  3. Topic of today • MCU : Multipoint Control Unit –

    Used for Video Conference • Web Browser – WebRTC, Canvas, Web Audio 3
  4. P2P vs. MCU 4 Browser A Browser B Browser D

    Browser C P2P • No media server • Browser side: • CPU load: Very High • Network load: Very High Browser A Browser B Browser D Browser C MCU Mix video / audio MCU • Use MCU server, CPU load: Super High • Browser side: very low CPU/Network load Good for Mobile
  5. Past trial:WebRTC Meetup Tokyo #4 • Absurd trial with WebRTC

    +α (Oct. 2014) – (3) mix video absurd way – http://www.slideshare.net/mganeko/webrtc-meetup4-lt/24 (Page 21 - ) 5 HDMI Capture unit Mix video with HDMI capture
  6. It is possible with only browser at 2016 • Canvas

    captureStream() – Firefox 43- , Chrome 51- • Handle remote audio with Web Audio API – Firefox (around)40-, Chrome 49- 6
  7. Browser MCU DEMO • YouTube - WebRTC Meetup Tokyo #11

    – https://youtu.be/0RYWw1nPxio?t=54m4s – 54:04 to 55:22 7
  8. Browser MCU DEMO 8 Grid mix

  9. Inside MCU Server:Video 9 RTCPeerConnection A MediaStream <video> element RTCPeerConnection

    D MediaStream <canvas> element drawImage() Keep drawing with requestAnimationFrame() <video> element drawImage()
  10. Inside MCU Server:Video 10 RTCPeerConnection A RTCPeerConnection D <canvas> element

    MediaStream captureStream(fps)
  11. Inside MCU Server:Video 11 RTCPeerConnection A MediaStream <video> element RTCPeerConnection

    D MediaStream <canvas> element MediaStream drawImage() Keep drawing with requestAnimationFrame() <video> element captureStream(fps) drawImage()
  12. Process of MCU Server:Video • Get Remote MediaStream through RTCPeerConnection

    – play in <video> element • Get context of Canvas by getContext('2d') • Keep drawing with window.requestAnimationFrame() – Copy image of each video element to canvas, with drawImage() • Capture MadiaStream from Canvas, with captureStream(fps) – Pass it to RTCPeerConnection.addStream(), and send it to remote peer 12
  13. Tricky points of MCU Server:Video • Using window.requestAnimationFrame() – Will

    not be called when the window/tab is completely hidden – Animation will stop if the window is minimized or other hidden by other tab • https://youtu.be/0RYWw1nPxio?t=56m44s to 57:17 • Even using window.setInterval() – The interval will be much longer than specified, the window/tab is completely hidden 13 DEMO
  14. Tricky points of MCU Server:Audio • One mixed video is

    enough for all members • Audio should be prepared differently for each member – It is hand to speak, if your voice will come back with short delay – Maybe familiar case with WebRTC application developers – If it is not familiar to you, please try yourself • https://mganeko.github.io/webrtcexpjp/basic2016/camera_mic.html • Please use headphone, and laud volume • Click [Start] button • It is so hard to speak 14 DEMO
  15. Inside MCU Server:Audio • Prepare "minus one" Audio – Mix

    audio of all members, except the user • Need many combination of audio – 4 combination for 4 members – N combination for N members 15 Browser A Browser B Browser D Browser C MCU Mixing Audio is hassle Audio is much more difficult than Video
  16. Inside MCU Server:Audio 16 RTCPeerConnection A MediaStream create with AudioContext

    . createMediaStreamSource() MediaStreamAudioSourceNode
  17. Inside MCU Server:Audio 17 RTCPeerConnection D MediaStream MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode

    MediaStreamAudioDestinationNode create with AudioContext . createMediaStreamDestination() mix
  18. Inside MCU Server:Audio 18 RTCPeerConnection A MediaStream RTCPeerConnection D MediaStream

    create with AudioContext . createMediaStreamSource() MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioDestinationNode create with AudioContext . createMediaStreamDestination() mix
  19. Process of MCU Servder:Audio • Get remote MediaStream through RTCPeerConnection

    • Create MediaStreamAudioSourceNode of Web Audio – Convert MediaStream to Web Audio node • Create MediaStreamAudioDestinationNode of Web Audio – Mix SourceNode of other members: sourceNode.connect(destinationNode); • Get MediaStream – Pass to RTCPeerConnection. addStream(), and send to remote peer • * repeat for all members 19
  20. Inside of MCU Server:Video & Audio 20 RTCPeerConnection A RTCPeerConnection

    D MediaStream (Video A+B+C+D) MediaStream (Audio B+C+D) RTCPeerConnection B RTCPeerConnection C MediaStream (Audio A+C+D) MediaStream (Audio A+B+D) MediaStream (Audio A+B+C) Multi-Stream with Video only MediaStream and Audio only MediaStream
  21. Browser MCU DEMO 2 • Canvas is So flexible !

    • YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=1h35s – 1:00:35 to 1:01:29 21
  22. Browser MCU DEMO 2 22 Stripe Variable Zoom

  23. Browser MCU DEMO 3 • 3D with Canvas + WebGL

    • YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=1h1m51s – 1:01:51 to 1:02:21 23
  24. Browser MCU DEMO 3 24 3D with WebGL

  25. Browser MCU DEMO 3 25 Stripe with 7 members Grid

    with 17 members Frame was dropped, in case of over 16 members Too heavy load, even for core-i7 CPU
  26. Browser MCU DEMO 4 26 • Record mix video/audio with

    MediaRecorder • YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=1h3m47s – 1:03:47 to 1:04:33
  27. Inside of MCU Server: Recording 27 MediaStream (Video A+B+C+D) <canvas>

    element Same as sending to remote peers Web Audio API MediaStream (Audio A+B+C+D) Mix all audio for recording videoTracks[] MediaStreamTrack MediaStreamTrack aidioTracks[] MediaStream new MediaStream() MediaStream.addTrack() MediaStream.addTrack() MediaRecorder WebM
  28. MCU vs. SFU 28

  29. SFU: Selective Forwarding Unit 29 Browser A Browser B Browser

    D Browser C SFU Dispatch audio/video SFU • User SFU server, CPU load: Low • Browser side: low CPU load • Browser side: network load: middle Able to choose display layout on member side ◎ Browser A Browser B Browser D Browser C MCU Mix video / audio MCU • Use MCU server, CPU load: Super High • Browser side: very low CPU/Network load
  30. Browser MCU DEMO 5 • It is NOT possible to

    choose display layout on member side, with MCU • ... NO, it is possible! • YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=1h6m49s – 1:06:49 to 1:07:25 30
  31. Browser MCU DEMO 5 31 Zooming on User/Member side

  32. Inside of zooming on MCU member 32 MediaStream (Video A+B+C+D)

    hidden <video> element drawImage() drawImage() drawImage() drawImage() It is possible to change layout with Canvas in member side (It is possible to use even multiple canvas)
  33. Appendix:painful point of MCU • Adjusting timing is painful for

    commercial MCU – Adjust timing for video of all members – Adjust timing of video and audio (lip sync) • For adjusting timing, it is need to wait for frame or block – But, impossible to wait forever – So it is necessary to timeout and send alternative frame/block • Sending Black frame, or repeat last frame • Sending silent block • Browser MCU does not care about adjusting timing 33
  34. Appendix:MCU Sample page • Start page (not http, but https)

    – https://mcu.talkin.info/ • Prepare a room as MCU server – Type room name – Click [Prepare a Room as MCU Server] – Wait for connection, or click [Re-Connect] • Join the room – Type room name, then click [Join to Room] – Click [Start video] to use camera/microphone – Click [Connect] 34
  35. Conclusion • Potential of WebRTC + Canvas is unlimited –

    It is very powerful toy – It is possible to send local video file • http://qiita.com/massie_g/items/5a6c4b69374d5997dc37 • But, CPU load of server side is super heavy – It is not scalable • Maybe useful, if user can take care of server side browser • Original Japanese Slide is on SlideShare – http://www.slideshare.net/mganeko/webrtc-build-mcu-on-browser 35
  36. Thank you! 36