Share
Embed
Build WebRTC MCU on Browser
Build MCU for WebRTC on Web browser.
Using WebRTC, Canvas, Web Audio.
Original presentation was made at WebRTC meetup Tokyo #11.
Youtube https://youtu.be/0RYWw1nPxio?t=50m41s
mganeko
August 13, 2016
Transcript
36 slides
-
WebRTC Meetup Tokyo #11 Build WebRTC MCU on Browser INFOCOM
CORPORATION Masashi Ganeko @massie_g Aug. 8th, 2016 1 -
About me • Masashi Ganeko / @massie_g – working for
a research team of INFOCOM CORPORATION • Writing a series of WebRTC on HTML5Experts.jp (HTML5 tech site) – https://html5experts.jp/series/webrtc2016/ • English Presentation for WebRTC (2013, 2016) – https://speakerdeck.com/mganeko • Japanese Presentation for WebRTC (2013-2016) – http://www.slideshare.net/mganeko 2 -
Topic of today • MCU : Multipoint Control Unit –
Used for Video Conference • Web Browser – WebRTC, Canvas, Web Audio 3 -
P2P vs. MCU 4 Browser A Browser B Browser D
Browser C P2P • No media server • Browser side: • CPU load: Very High • Network load: Very High Browser A Browser B Browser D Browser C MCU Mix video / audio MCU • Use MCU server, CPU load: Super High • Browser side: very low CPU/Network load Good for Mobile -
Past trial:WebRTC Meetup Tokyo #4 • Absurd trial with WebRTC
+α (Oct. 2014) – (3) mix video absurd way – http://www.slideshare.net/mganeko/webrtc-meetup4-lt/24 (Page 21 - ) 5 HDMI Capture unit Mix video with HDMI capture -
It is possible with only browser at 2016 • Canvas
captureStream() – Firefox 43- , Chrome 51- • Handle remote audio with Web Audio API – Firefox (around)40-, Chrome 49- 6 -
Browser MCU DEMO • YouTube - WebRTC Meetup Tokyo #11
– https://youtu.be/0RYWw1nPxio?t=54m4s – 54:04 to 55:22 7 -
Inside MCU Server:Video 9 RTCPeerConnection A MediaStream <video> element RTCPeerConnection
D MediaStream <canvas> element drawImage() Keep drawing with requestAnimationFrame() <video> element drawImage() -
Inside MCU Server:Video 10 RTCPeerConnection A RTCPeerConnection D <canvas> element
MediaStream captureStream(fps) -
Inside MCU Server:Video 11 RTCPeerConnection A MediaStream <video> element RTCPeerConnection
D MediaStream <canvas> element MediaStream drawImage() Keep drawing with requestAnimationFrame() <video> element captureStream(fps) drawImage() -
Process of MCU Server:Video • Get Remote MediaStream through RTCPeerConnection
– play in <video> element • Get context of Canvas by getContext('2d') • Keep drawing with window.requestAnimationFrame() – Copy image of each video element to canvas, with drawImage() • Capture MadiaStream from Canvas, with captureStream(fps) – Pass it to RTCPeerConnection.addStream(), and send it to remote peer 12 -
Tricky points of MCU Server:Video • Using window.requestAnimationFrame() – Will
not be called when the window/tab is completely hidden – Animation will stop if the window is minimized or other hidden by other tab • https://youtu.be/0RYWw1nPxio?t=56m44s to 57:17 • Even using window.setInterval() – The interval will be much longer than specified, the window/tab is completely hidden 13 DEMO -
Tricky points of MCU Server:Audio • One mixed video is
enough for all members • Audio should be prepared differently for each member – It is hand to speak, if your voice will come back with short delay – Maybe familiar case with WebRTC application developers – If it is not familiar to you, please try yourself • https://mganeko.github.io/webrtcexpjp/basic2016/camera_mic.html • Please use headphone, and laud volume • Click [Start] button • It is so hard to speak 14 DEMO -
Inside MCU Server:Audio • Prepare "minus one" Audio – Mix
audio of all members, except the user • Need many combination of audio – 4 combination for 4 members – N combination for N members 15 Browser A Browser B Browser D Browser C MCU Mixing Audio is hassle Audio is much more difficult than Video -
Inside MCU Server:Audio 16 RTCPeerConnection A MediaStream create with AudioContext
. createMediaStreamSource() MediaStreamAudioSourceNode -
Inside MCU Server:Audio 17 RTCPeerConnection D MediaStream MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode
MediaStreamAudioDestinationNode create with AudioContext . createMediaStreamDestination() mix -
Inside MCU Server:Audio 18 RTCPeerConnection A MediaStream RTCPeerConnection D MediaStream
create with AudioContext . createMediaStreamSource() MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioDestinationNode create with AudioContext . createMediaStreamDestination() mix -
Process of MCU Servder:Audio • Get remote MediaStream through RTCPeerConnection
• Create MediaStreamAudioSourceNode of Web Audio – Convert MediaStream to Web Audio node • Create MediaStreamAudioDestinationNode of Web Audio – Mix SourceNode of other members: sourceNode.connect(destinationNode); • Get MediaStream – Pass to RTCPeerConnection. addStream(), and send to remote peer • * repeat for all members 19 -
Inside of MCU Server:Video & Audio 20 RTCPeerConnection A RTCPeerConnection
D MediaStream (Video A+B+C+D) MediaStream (Audio B+C+D) RTCPeerConnection B RTCPeerConnection C MediaStream (Audio A+C+D) MediaStream (Audio A+B+D) MediaStream (Audio A+B+C) Multi-Stream with Video only MediaStream and Audio only MediaStream -
Browser MCU DEMO 2 • Canvas is So flexible !
• YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=1h35s – 1:00:35 to 1:01:29 21 -
Browser MCU DEMO 3 • 3D with Canvas + WebGL
• YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=1h1m51s – 1:01:51 to 1:02:21 23 -
Browser MCU DEMO 3 25 Stripe with 7 members Grid
with 17 members Frame was dropped, in case of over 16 members Too heavy load, even for core-i7 CPU -
Browser MCU DEMO 4 26 • Record mix video/audio with
MediaRecorder • YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=1h3m47s – 1:03:47 to 1:04:33 -
Inside of MCU Server: Recording 27 MediaStream (Video A+B+C+D) <canvas>
element Same as sending to remote peers Web Audio API MediaStream (Audio A+B+C+D) Mix all audio for recording videoTracks[] MediaStreamTrack MediaStreamTrack aidioTracks[] MediaStream new MediaStream() MediaStream.addTrack() MediaStream.addTrack() MediaRecorder WebM -
SFU: Selective Forwarding Unit 29 Browser A Browser B Browser
D Browser C SFU Dispatch audio/video SFU • User SFU server, CPU load: Low • Browser side: low CPU load • Browser side: network load: middle Able to choose display layout on member side ◎ Browser A Browser B Browser D Browser C MCU Mix video / audio MCU • Use MCU server, CPU load: Super High • Browser side: very low CPU/Network load -
Browser MCU DEMO 5 • It is NOT possible to
choose display layout on member side, with MCU • ... NO, it is possible! • YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=1h6m49s – 1:06:49 to 1:07:25 30 -
Inside of zooming on MCU member 32 MediaStream (Video A+B+C+D)
hidden <video> element drawImage() drawImage() drawImage() drawImage() It is possible to change layout with Canvas in member side (It is possible to use even multiple canvas) -
Appendix:painful point of MCU • Adjusting timing is painful for
commercial MCU – Adjust timing for video of all members – Adjust timing of video and audio (lip sync) • For adjusting timing, it is need to wait for frame or block – But, impossible to wait forever – So it is necessary to timeout and send alternative frame/block • Sending Black frame, or repeat last frame • Sending silent block • Browser MCU does not care about adjusting timing 33 -
Appendix:MCU Sample page • Start page (not http, but https)
– https://mcu.talkin.info/ • Prepare a room as MCU server – Type room name – Click [Prepare a Room as MCU Server] – Wait for connection, or click [Re-Connect] • Join the room – Type room name, then click [Join to Room] – Click [Start video] to use camera/microphone – Click [Connect] 34 -
Conclusion • Potential of WebRTC + Canvas is unlimited –
It is very powerful toy – It is possible to send local video file • http://qiita.com/massie_g/items/5a6c4b69374d5997dc37 • But, CPU load of server side is super heavy – It is not scalable • Maybe useful, if user can take care of server side browser • Original Japanese Slide is on SlideShare – http://www.slideshare.net/mganeko/webrtc-build-mcu-on-browser 35