Build MCU for WebRTC on Web browser.
Using WebRTC, Canvas, Web Audio.
Original presentation was made at WebRTC meetup Tokyo #11.
Youtube https://youtu.be/0RYWw1nPxio?t=50m41s
About me • Masashi Ganeko / @massie_g – working for a research team of INFOCOM CORPORATION • Writing a series of WebRTC on HTML5Experts.jp (HTML5 tech site) – https://html5experts.jp/series/webrtc2016/ • English Presentation for WebRTC (2013, 2016) – https://speakerdeck.com/mganeko • Japanese Presentation for WebRTC (2013-2016) – http://www.slideshare.net/mganeko 2
P2P vs. MCU 4 Browser A Browser B Browser D Browser C P2P • No media server • Browser side: • CPU load: Very High • Network load: Very High Browser A Browser B Browser D Browser C MCU Mix video / audio MCU • Use MCU server, CPU load: Super High • Browser side: very low CPU/Network load Good for Mobile
Past trial：WebRTC Meetup Tokyo #4 • Absurd trial with WebRTC ＋α (Oct. 2014) – (3) mix video absurd way – http://www.slideshare.net/mganeko/webrtc-meetup4-lt/24 (Page 21 - ) 5 HDMI Capture unit Mix video with HDMI capture
It is possible with only browser at 2016 • Canvas captureStream() – Firefox 43- , Chrome 51- • Handle remote audio with Web Audio API – Firefox (around)40-, Chrome 49- 6
Process of MCU Server：Video • Get Remote MediaStream through RTCPeerConnection – play in element • Get context of Canvas by getContext('2d') • Keep drawing with window.requestAnimationFrame() – Copy image of each video element to canvas, with drawImage() • Capture MadiaStream from Canvas, with captureStream(fps) – Pass it to RTCPeerConnection.addStream(), and send it to remote peer 12
Tricky points of MCU Server：Video • Using window.requestAnimationFrame() – Will not be called when the window/tab is completely hidden – Animation will stop if the window is minimized or other hidden by other tab • https://youtu.be/0RYWw1nPxio?t=56m44s to 57:17 • Even using window.setInterval() – The interval will be much longer than specified, the window/tab is completely hidden 13 DEMO
Tricky points of MCU Server：Audio • One mixed video is enough for all members • Audio should be prepared differently for each member – It is hand to speak, if your voice will come back with short delay – Maybe familiar case with WebRTC application developers – If it is not familiar to you, please try yourself • https://mganeko.github.io/webrtcexpjp/basic2016/camera_mic.html • Please use headphone, and laud volume • Click [Start] button • It is so hard to speak 14 DEMO
Inside MCU Server：Audio • Prepare "minus one" Audio – Mix audio of all members, except the user • Need many combination of audio – 4 combination for 4 members – N combination for N members 15 Browser A Browser B Browser D Browser C MCU Mixing Audio is hassle Audio is much more difficult than Video
Process of MCU Servder：Audio • Get remote MediaStream through RTCPeerConnection • Create MediaStreamAudioSourceNode of Web Audio – Convert MediaStream to Web Audio node • Create MediaStreamAudioDestinationNode of Web Audio – Mix SourceNode of other members: sourceNode.connect(destinationNode); • Get MediaStream – Pass to RTCPeerConnection. addStream(), and send to remote peer • * repeat for all members 19
Inside of MCU Server：Video & Audio 20 RTCPeerConnection A RTCPeerConnection D MediaStream (Video A+B+C+D) MediaStream (Audio B+C+D) RTCPeerConnection B RTCPeerConnection C MediaStream (Audio A+C+D) MediaStream (Audio A+B+D) MediaStream (Audio A+B+C) Multi-Stream with Video only MediaStream and Audio only MediaStream
Inside of MCU Server: Recording 27 MediaStream (Video A+B+C+D) element Same as sending to remote peers Web Audio API MediaStream (Audio A+B+C+D) Mix all audio for recording videoTracks[] MediaStreamTrack MediaStreamTrack aidioTracks[] MediaStream new MediaStream() MediaStream.addTrack() MediaStream.addTrack() MediaRecorder WebM
SFU: Selective Forwarding Unit 29 Browser A Browser B Browser D Browser C SFU Dispatch audio/video SFU • User SFU server, CPU load: Low • Browser side: low CPU load • Browser side: network load: middle Able to choose display layout on member side ◎ Browser A Browser B Browser D Browser C MCU Mix video / audio MCU • Use MCU server, CPU load: Super High • Browser side: very low CPU/Network load
Browser MCU DEMO 5 • It is NOT possible to choose display layout on member side, with MCU • ... NO, it is possible! • YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=1h6m49s – 1:06:49 to 1:07:25 30
Inside of zooming on MCU member 32 MediaStream (Video A+B+C+D) hidden
element drawImage() drawImage() drawImage() drawImage() It is possible to change layout with Canvas in member side (It is possible to use even multiple canvas)
Appendix：painful point of MCU • Adjusting timing is painful for commercial MCU – Adjust timing for video of all members – Adjust timing of video and audio (lip sync) • For adjusting timing, it is need to wait for frame or block – But, impossible to wait forever – So it is necessary to timeout and send alternative frame/block • Sending Black frame, or repeat last frame • Sending silent block • Browser MCU does not care about adjusting timing 33
Appendix：MCU Sample page • Start page (not http, but https) – https://mcu.talkin.info/ • Prepare a room as MCU server – Type room name – Click [Prepare a Room as MCU Server] – Wait for connection, or click [Re-Connect] • Join the room – Type room name, then click [Join to Room] – Click [Start video] to use camera/microphone – Click [Connect] 34
Conclusion • Potential of WebRTC + Canvas is unlimited – It is very powerful toy – It is possible to send local video file • http://qiita.com/massie_g/items/5a6c4b69374d5997dc37 • But, CPU load of server side is super heavy – It is not scalable • Maybe useful, if user can take care of server side browser • Original Japanese Slide is on SlideShare – http://www.slideshare.net/mganeko/webrtc-build-mcu-on-browser 35
mganeko
carta_engineering
toshiaizawa
jacopen
makamaka
dskst
masashi_sutou
takahashikazutaro
chiroruxx
tomohisa
safie_recruit
oogfranz
sat
bkeepers
brad_frost
geeforr
afnizarnur
sachag
lauravandoore
62gerente
chrislema
mclloyd
tammielis
smashingmag
destraynor