Presentation for Nodefest Tokyo 2017
Building WebRTC MCU with Node.js and Headless browser.
Mix video and audio with Canvas and WebAudio, without window/desktop environment
research team of INFOCOM CORPORATION • Staff of WebRTC Meetup Tokyo • Staff of WebRTC Beginners Tokyo • WebRTC related presentaRons – hSps://speakerdeck.com/mganeko 2
Node.js with WebRTC DataChannel (in Japanese) – hSps://www.slideshare.net/mganeko/nodejs-with-webrtc-datachannel • Nodefest Tokyo 2014 – Using Kurento Media Server with Node.js (in Japanese) – hSp://www.slideshare.net/mganeko/nodekurento • Nodefest Tokyo 2013 – Let's Begin WebRTC – hSps://speakerdeck.com/mganeko/lets-begin-webrtc 3
– MCU: mixing video/audio in server • It is possible to build MCU with Web Browser • Manipulate video/audio as you like – with Node.js x Headless Chrome 4
– Audio – Data • Open standard … anyone can use – W3C WebRTC Working Group ... API – IETF RTCWEB Working Group … Protocol – Core library is open source: libwebrtc • Designed for Web browsers, and other web connected devices • Easy to combine with other Web technologies 10101110100…
connecRon between browsers • SFU (SelecRve Forwarding Unit) – Server dispatchs video/audio streams • MCU (MulRpoint Control Unit) – Mix audio/audio, and re-encode in Server side 8
NO encode / transcode in server • Good – CPU load of browser/client: low – Upstream network load: low – SFU server CPU load: low • Bad • Upstream network load: High
• Good – CPU load of browser/client: lowest – Upstream/downstream network load: lowest – → Good for mobile clients • Bad • Server CPU load: Super High • Video layout is fixed Browser A Browser B Browser D Browser C MCU Mix video / audio
in server – Compression for each connecRon – Different CODEC for each client PC Chrome PC Firefox Android Chrome iOS Safari MCU VP9 VP9 VP9 VP9 VP8 VP8 H.264 H.264 H.264 only
• looks not so acRve, amer acquisiRon by Twillio • My slides about Node.js x Kurento Media Server (in Japanese) – hSps://www.slideshare.net/mganeko/nodekurento – Licode … hSp://lynckia.com/licode/ • Products – Dialogic PowerMedia XMS 15
RTCPeerConnecRon D MediaStream <canvas> element drawImage() Keep drawing with requestAnimaRonFrame() <video> element drawImage() MediaStream Get video with captureStream(fps)
MediaStream through RTCPeerConnecRon – play in <video> element • Get context of Canvas by getContext('2d') • Keep drawing with window.requestAnimaRonFrame() – Copy image of each video element to canvas, with drawImage() • Capture MadiaStream from Canvas, with captureStream(fps) – Pass it to RTCPeerConnecRon.addStream(), and send it to remote peer 23
not be called when the window/tab is completely hidden – AnimaRon will stop if the window is minimized or other hidden by other tab • Even using window.setInterval() – The interval will be much longer than specified, the window/tab is completely hidden 25
enough for all members • Audio should be prepared differently for each member – It is hand to speak, if your voice will come back with short delay – Maybe familiar case with WebRTC applicaRon developers 27
audio of all members, except the user • Need many combinaRon of audio – 4 combinaRon for 4 members – N combinaRon for N members 28 Browser A Browser B Browser D Browser C MCU Mixing Audio is hassle Audio is much more difficult than Video
MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioDesRnaRonNode Mix Repeat for N paSern RTCPeerConnecRon A RTCPeerConnecRon B RTCPeerConnecRon C
MediaStream through RTCPeerConnecRon • Create MediaStreamAudioSourceNode of Web Audio – Convert MediaStream to Web Audio node • Create MediaStreamAudioDesRnaRonNode of Web Audio – Mix SourceNode of other members: sourceNode.connect(desRnaRonNode); • Get MediaStream – Pass to RTCPeerConnecRon. addStream(), and send to remote peer • * repeat for all members 32
connecRons • Signaling process is necessary before P2P – Exchange IP address, ports, and CODEC condiRons • No standard for signaling – Different between Apps / Services → Separate mixing core from peer handling 35 Browser A Browser B Browser MCU Signaling Server message message message media media
URL – * "--remote-debugging-port" is necessary (Nov 15 2017) – * no more "--disable-gpu" from Chrome 61 • Work without Window/Window System – render video – draw Canvas – capture MediaStream from Canvas • AnimaRon event fires with requestAnimaRonFrame() 38
DevTools Protocol – chrome-remote-interface • hSps://github.com/cyrus-and/chrome-remote-interface – puppeteer • hSps://github.com/GoogleChrome/puppeteer • Simple implementaRon for Browser MCU series – MCU browser is connected to Node.js server over WebSocket/Socket.io • It already have connecRon for signaling • The connecRon is used for command from Node.js server – No page transiRon (Kind of Single Page App), so keep connected 39
execute as child process let proc = childProcess.execFile(path, args); • Stop // Not possible to terminate from inside of Chrome proc.kill('SIGKILL'); 40
each Room – Mix video/audio for the Room separately 46 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling /MCU Server Room1 Room2
Server – Get higher, when a new member join the room – Get much higher, when a new room added • Auto scale out is needed to correspond 47 ObservaRon on vultr Spec - CPU: 2 vCore - RAM: 4GB CPU load - IniRal (no room) … CPU 3% - 1room, 2members … CPU 200%, +300MB - 1room, 4members … CPU 200%, +50MB - 2room, 4+2members … CPU 200%,+300MB
each Room automaRcally – VM instance of Container 48 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling Server Room1 Room2 MCU Server MCU Server Invoke Invoke → High server cost, difficult to control
owner provide MCU resource with a browser or a container 49 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling Server Room1 Room2 Browser MCU Docker Start by hand Start by hand
MCU Service（mulRple room) HTML for MCU HTML for members Signaling Server Node.js + ws + express Test server hSps://meeRn.biz Chrome Browser MCU Core Video/Audio mix (js) P2P handling (js) PeerConnecRon P2P handling (js) PeerConnecRon Member browser Chrome/Firefox/ Safari Member browser Chrome/Firefox/ Safari Headless Chrome
– Core library … hSps://github.com/mganeko/browser_mcu_core • Sample for single room server… Browser MCU Server – hSps://github.com/mganeko/browser_mcu_server • Container for single room server and headless browser – hSps://github.com/mganeko/browser_mcu_pack – hSps://hub.docker.com/r/mganeko/browser_mcu_pack/ • Trial server … hSps://meeRn.biz/ – MCU … hSps://meeRn.biz/mulR_mcu.html 52 Most useful
Bad point of MCU: Super high CPU load in server – Good point of MCU: very preferable for mobile clients • It is possible to build MCU on Web browser • To make use of browser MCU, separaRon of heavy load components is necessary • Node.js x Headless browser … extend potenRal of Node.js – Realize Video / Audio manipulaRon in server side 53
mganeko
lmi
iotengineer22
msato
ryder472
.jpeg){kind=avatar}
arthur1
sansantech
shujisado
salaboy
naospon
bbqallstars
4su_para
sonatard
stephaniewalter
mraible
holman
chriscoyier
hatefulcrawdad
tenderlove
ddemaree
sachag
hannesfritz
lara
phodgson
sstephenson
