Presentation for Nodefest Tokyo 2017
Building WebRTC MCU with Node.js and Headless browser.
Mix video and audio with Canvas and WebAudio, without window/desktop environment
About Me • Masashi Ganeko / @massie_g • Working for research team of INFOCOM CORPORATION • Staff of WebRTC Meetup Tokyo • Staff of WebRTC Beginners Tokyo • WebRTC related presentaRons – hSps://speakerdeck.com/mganeko 2
Sessions in past Nodefest • Nodefest Tokyo 2016 LT – Node.js with WebRTC DataChannel (in Japanese) – hSps://www.slideshare.net/mganeko/nodejs-with-webrtc-datachannel • Nodefest Tokyo 2014 – Using Kurento Media Server with Node.js (in Japanese) – hSp://www.slideshare.net/mganeko/nodekurento • Nodefest Tokyo 2013 – Let's Begin WebRTC – hSps://speakerdeck.com/mganeko/lets-begin-webrtc 3
Topic of today • WebRTC topology is not only Peer-to-Peer – MCU: mixing video/audio in server • It is possible to build MCU with Web Browser • Manipulate video/audio as you like – with Node.js x Headless Chrome 4
What is WebRTC • Web Real-Rme CommunicaRon for – Video – Audio – Data • Open standard … anyone can use – W3C WebRTC Working Group ... API – IETF RTCWEB Working Group … Protocol – Core library is open source: libwebrtc • Designed for Web browsers, and other web connected devices • Easy to combine with other Web technologies 10101110100…
WebRTC Topology • Peer to Peer … Basic – Direct connecRon between browsers • SFU (SelecRve Forwarding Unit) – Server dispatchs video/audio streams • MCU (MulRpoint Control Unit) – Mix audio/audio, and re-encode in Server side 8
P2P • Basic topology – Full mesh connecRon between browsers • Good – NO media server • Bad – CPU load: Super High – Network load: Super High Browser A Browser B Browser D Browser C
Browser A Why super high CPU load in P2P send send send But this: encode for each receiver Browser A Original video encoded send send send Original video encoded encoded encoded
SFU: SelecRve Forwarding Unit • Dispatch video/audio in server – NO encode / transcode in server • Good – CPU load of browser/client: low – Upstream network load: low – SFU server CPU load: low • Bad • Upstream network load: High
MCU (MulRpoint Control Unit) • Mix video/audio in server side • Good – CPU load of browser/client: lowest – Upstream/downstream network load: lowest – → Good for mobile clients • Bad • Server CPU load: Super High • Video layout is fixed Browser A Browser B Browser D Browser C MCU Mix video / audio
MCU for mobile • Mix video/audio in server • Re-encode/transcode in server – Compression for each connecRon – Different CODEC for each client PC Chrome PC Firefox Android Chrome iOS Safari MCU VP9 VP9 VP9 VP9 VP8 VP8 H.264 H.264 H.264 only
MCU OSS and products • OSS – Kureto … hSp://www.kurento.org/ • looks not so acRve, amer acquisiRon by Twillio • My slides about Node.js x Kurento Media Server (in Japanese) – hSps://www.slideshare.net/mganeko/nodekurento – Licode … hSp://lynckia.com/licode/ • Products – Dialogic PowerMedia XMS 15
What MCU does • Receive Video/Audio from clients • Mix Videos • Mix Audios • Send back Video/Audio to clients 18 WebRTC PeerConnecRon WebRTC PeerConnecRon Canvas WebAudio I can do with Browser !
Steps to mix video in Browser MCU • Get Remote MediaStream through RTCPeerConnecRon – play in element • Get context of Canvas by getContext('2d') • Keep drawing with window.requestAnimaRonFrame() – Copy image of each video element to canvas, with drawImage() • Capture MadiaStream from Canvas, with captureStream(fps) – Pass it to RTCPeerConnecRon.addStream(), and send it to remote peer 23
Tricky points of MCU Server：Video • Using window.requestAnimaRonFrame() – Will not be called when the window/tab is completely hidden – AnimaRon will stop if the window is minimized or other hidden by other tab • Even using window.setInterval() – The interval will be much longer than specified, the window/tab is completely hidden 25
Tricky points of MCU Server：Audio • One mixed video is enough for all members • Audio should be prepared differently for each member – It is hand to speak, if your voice will come back with short delay – Maybe familiar case with WebRTC applicaRon developers 27
Inside MCU Server：Audio • Prepare "minus one" Audio – Mix audio of all members, except the user • Need many combinaRon of audio – 4 combinaRon for 4 members – N combinaRon for N members 28 Browser A Browser B Browser D Browser C MCU Mixing Audio is hassle Audio is much more difficult than Video
Inside MCU Server：Audio 30 RTCPeerConnecRon D MediaStream MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioDesRnaRonNode mix RTCPeerConnecRon A Create with AudioContext . createMediaStreamDesRnaRon() RTCPeerConnecRon B RTCPeerConnecRon C
Inside MCU Server：Audio 31 RTCPeerConnecRon A MediaStream RTCPeerConnecRon D MediaStream MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioDesRnaRonNode Mix Repeat for N paSern RTCPeerConnecRon A RTCPeerConnecRon B RTCPeerConnecRon C
Steps to mix audio in Browser MCU • Get remote MediaStream through RTCPeerConnecRon • Create MediaStreamAudioSourceNode of Web Audio – Convert MediaStream to Web Audio node • Create MediaStreamAudioDesRnaRonNode of Web Audio – Mix SourceNode of other members: sourceNode.connect(desRnaRonNode); • Get MediaStream – Pass to RTCPeerConnecRon. addStream(), and send to remote peer • * repeat for all members 32
Whole structure • Browser MCU is communicaRng over P2P WebRTC connecRons • Signaling process is necessary before P2P – Exchange IP address, ports, and CODEC condiRons • No standard for signaling – Different between Apps / Services → Separate mixing core from peer handling 35 Browser A Browser B Browser MCU Signaling Server message message message media media
Headless Chrome • Chrome 59 ～ – chrome --headless --remote-debugging-port=9222 URL – * "--remote-debugging-port" is necessary (Nov 15 2017) – * no more "--disable-gpu" from Chrome 61 • Work without Window/Window System – render video – draw Canvas – capture MediaStream from Canvas • AnimaRon event fires with requestAnimaRonFrame() 38
Control of Headless Chrome • Node.js modules … using Chrome DevTools Protocol – chrome-remote-interface • hSps://github.com/cyrus-and/chrome-remote-interface – puppeteer • hSps://github.com/GoogleChrome/puppeteer • Simple implementaRon for Browser MCU series – MCU browser is connected to Node.js server over WebSocket/Socket.io • It already have connecRon for signaling • The connecRon is used for command from Node.js server – No page transiRon (Kind of Single Page App), so keep connected 39
PrimiRve control of Headless Chrome • Invoke from Node.js // execute as child process let proc = childProcess.execFile(path, args); • Stop // Not possible to terminate from inside of Chrome proc.kill('SIGKILL'); 40
Run in Docker container • Headless chrome will run in Docker container • Made a container image, include Node.js and Headless Chrome – Single room – Browser MCU Pack (Dockerfile) • hSps://github.com/mganeko/browser_mcu_pack – base： node:latest • Clone hSps://github.com/mganeko/browser_mcu_server • Install Chrome stable • Container image on docker hub: mganeko/browser_mcu_pack – hSps://hub.docker.com/r/mganeko/browser_mcu_pack/ 42
Support mulRple video chat rooms • Run browser MUC for each Room – Mix video/audio for the Room separately 46 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling /MCU Server Room1 Room2
Bad point of MCU • Super High CPU load in Server – Get higher, when a new member join the room – Get much higher, when a new room added • Auto scale out is needed to correspond 47 ObservaRon on vultr Spec - CPU: 2 vCore - RAM: 4GB CPU load - IniRal (no room) … CPU 3% - 1room, 2members … CPU 200%, +300MB - 1room, 4members … CPU 200%, +50MB - 2room, 4+2members … CPU 200%,+300MB
Hot to auto scale out? • Invoke MCU Server for each Room automaRcally – VM instance of Container 48 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling Server Room1 Room2 MCU Server MCU Server Invoke Invoke → High server cost, difficult to control
Make it simple: Room owner iniRaRve scale out • Room owner provide MCU resource with a browser or a container 49 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling Server Room1 Room2 Browser MCU Docker Start by hand Start by hand
Browser MCU docker Browser MCU Series on Web 50 Browser MCU Service（mulRple room) HTML for MCU HTML for members Signaling Server Node.js + ws + express Test server hSps://meeRn.biz Chrome Browser MCU Core Video/Audio mix (js) P2P handling (js) PeerConnecRon P2P handling (js) PeerConnecRon Member browser Chrome/Firefox/ Safari Member browser Chrome/Firefox/ Safari Headless Chrome
Try Browser MCU Series • Browser MCU Series – hSps://github.com/mganeko/browser_mcu – Core library … hSps://github.com/mganeko/browser_mcu_core • Sample for single room server… Browser MCU Server – hSps://github.com/mganeko/browser_mcu_server • Container for single room server and headless browser – hSps://github.com/mganeko/browser_mcu_pack – hSps://hub.docker.com/r/mganeko/browser_mcu_pack/ • Trial server … hSps://meeRn.biz/ – MCU … hSps://meeRn.biz/mulR_mcu.html 52 Most useful
Conclusion • WebRTC has many topologies, not only P2P – Bad point of MCU: Super high CPU load in server – Good point of MCU: very preferable for mobile clients • It is possible to build MCU on Web browser • To make use of browser MCU, separaRon of heavy load components is necessary • Node.js x Headless browser … extend potenRal of Node.js – Realize Video / Audio manipulaRon in server side 53
mganeko
lmi
chiroruxx
iseki
konboi
sbtechnight
kentosuzuki
xibuka
oracle4engineer
mapbox_jp
nenonaninu
jacopen
udzura
marcduiker
trishagee
aleyda
yeseniaperezcruz
mthomps
keavy
marktimemedia
tmm1
smashingmag
shlominoach
jlugia
ufuk