Share
Embed
Node.js x Chrome headless for WebRTC MCU
Presentation for Nodefest Tokyo 2017
Building WebRTC MCU with Node.js and Headless browser.
Mix video and audio with Canvas and WebAudio, without window/desktop environment
mganeko
November 26, 2017
Transcript
54 slides
-
Node.js x Chrome headless = WebRTC MCU Nodefest 2017 2017.11.25
INFOCOM CORPORATION Masashi Ganeko @massie_g 1 -
About Me • Masashi Ganeko / @massie_g • Working for
research team of INFOCOM CORPORATION • Staff of WebRTC Meetup Tokyo • Staff of WebRTC Beginners Tokyo • WebRTC related presentaRons – hSps://speakerdeck.com/mganeko 2 -
Sessions in past Nodefest • Nodefest Tokyo 2016 LT –
Node.js with WebRTC DataChannel (in Japanese) – hSps://www.slideshare.net/mganeko/nodejs-with-webrtc-datachannel • Nodefest Tokyo 2014 – Using Kurento Media Server with Node.js (in Japanese) – hSp://www.slideshare.net/mganeko/nodekurento • Nodefest Tokyo 2013 – Let's Begin WebRTC – hSps://speakerdeck.com/mganeko/lets-begin-webrtc 3 -
Topic of today • WebRTC topology is not only Peer-to-Peer
– MCU: mixing video/audio in server • It is possible to build MCU with Web Browser • Manipulate video/audio as you like – with Node.js x Headless Chrome 4 -
What is WebRTC • Web Real-Rme CommunicaRon for – Video
– Audio – Data • Open standard … anyone can use – W3C WebRTC Working Group ... API – IETF RTCWEB Working Group … Protocol – Core library is open source: libwebrtc • Designed for Web browsers, and other web connected devices • Easy to combine with other Web technologies 10101110100… -
WebRTC enabled OS 7 Supported in Mac OS X /
iOS Safari 11 (H.264 only) NEW! NEW! -
WebRTC Topology • Peer to Peer … Basic – Direct
connecRon between browsers • SFU (SelecRve Forwarding Unit) – Server dispatchs video/audio streams • MCU (MulRpoint Control Unit) – Mix audio/audio, and re-encode in Server side 8 -
P2P • Basic topology – Full mesh connecRon between browsers
• Good – NO media server • Bad – CPU load: Super High – Network load: Super High Browser A Browser B Browser D Browser C -
Why super high CPU load in P2P Browser A Original
video encoded send send send Not This (Same encoded data for all receivers) -
Browser A Why super high CPU load in P2P send
send send But this: encode for each receiver Browser A Original video encoded send send send Original video encoded encoded encoded -
SFU: SelecRve Forwarding Unit • Dispatch video/audio in server –
NO encode / transcode in server • Good – CPU load of browser/client: low – Upstream network load: low – SFU server CPU load: low • Bad • Upstream network load: High -
MCU (MulRpoint Control Unit) • Mix video/audio in server side
• Good – CPU load of browser/client: lowest – Upstream/downstream network load: lowest – → Good for mobile clients • Bad • Server CPU load: Super High • Video layout is fixed Browser A Browser B Browser D Browser C MCU Mix video / audio -
MCU for mobile • Mix video/audio in server • Re-encode/transcode
in server – Compression for each connecRon – Different CODEC for each client PC Chrome PC Firefox Android Chrome iOS Safari MCU VP9 VP9 VP9 VP9 VP8 VP8 H.264 H.264 H.264 only -
MCU OSS and products • OSS – Kureto … hSp://www.kurento.org/
• looks not so acRve, amer acquisiRon by Twillio • My slides about Node.js x Kurento Media Server (in Japanese) – hSps://www.slideshare.net/mganeko/nodekurento – Licode … hSp://lynckia.com/licode/ • Products – Dialogic PowerMedia XMS 15 -
What MCU does • Receive Video/Audio from clients • Mix
Videos • Mix Audios • Send back Video/Audio to clients 17 -
What MCU does • Receive Video/Audio from clients • Mix
Videos • Mix Audios • Send back Video/Audio to clients 18 WebRTC PeerConnecRon WebRTC PeerConnecRon Canvas WebAudio I can do with Browser ! -
Inside of Browser MCU:Video 20 RTCPeerConnecRon A MediaStream <video> element
RTCPeerConnecRon D MediaStream <canvas> element drawImage() Keep drawing with requestAnimaRonFrame() <video> element drawImage() -
Inside of Browser MCU:Video 21 RTCPeerConnecRon A RTCPeerConnecRon D <canvas>
element MediaStream Get video with captureStream(fps) -
Inside of Browser MCU:Video 22 RTCPeerConnecRon A MediaStream <video> element
RTCPeerConnecRon D MediaStream <canvas> element drawImage() Keep drawing with requestAnimaRonFrame() <video> element drawImage() MediaStream Get video with captureStream(fps) -
Steps to mix video in Browser MCU • Get Remote
MediaStream through RTCPeerConnecRon – play in <video> element • Get context of Canvas by getContext('2d') • Keep drawing with window.requestAnimaRonFrame() – Copy image of each video element to canvas, with drawImage() • Capture MadiaStream from Canvas, with captureStream(fps) – Pass it to RTCPeerConnecRon.addStream(), and send it to remote peer 23 -
DEMO1: Mixing video with Browser MCU • Start node server
• Open Browser for MCU • Open client browser and connect • Demo case of Browser MCU hidden 24 -
Tricky points of MCU Server:Video • Using window.requestAnimaRonFrame() – Will
not be called when the window/tab is completely hidden – AnimaRon will stop if the window is minimized or other hidden by other tab • Even using window.setInterval() – The interval will be much longer than specified, the window/tab is completely hidden 25 -
Tricky points of MCU Server:Audio • One mixed video is
enough for all members • Audio should be prepared differently for each member – It is hand to speak, if your voice will come back with short delay – Maybe familiar case with WebRTC applicaRon developers 27 -
Inside MCU Server:Audio • Prepare "minus one" Audio – Mix
audio of all members, except the user • Need many combinaRon of audio – 4 combinaRon for 4 members – N combinaRon for N members 28 Browser A Browser B Browser D Browser C MCU Mixing Audio is hassle Audio is much more difficult than Video -
Inside MCU Server:Audio 29 RTCPeerConnecRon A MediaStream MediaStreamAudioSourceNode Create with
AudioContext . createMediaStreamSource() -
Inside MCU Server:Audio 30 RTCPeerConnecRon D MediaStream MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode
MediaStreamAudioDesRnaRonNode mix RTCPeerConnecRon A Create with AudioContext . createMediaStreamDesRnaRon() RTCPeerConnecRon B RTCPeerConnecRon C -
Inside MCU Server:Audio 31 RTCPeerConnecRon A MediaStream RTCPeerConnecRon D MediaStream
MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioDesRnaRonNode Mix Repeat for N paSern RTCPeerConnecRon A RTCPeerConnecRon B RTCPeerConnecRon C -
Steps to mix audio in Browser MCU • Get remote
MediaStream through RTCPeerConnecRon • Create MediaStreamAudioSourceNode of Web Audio – Convert MediaStream to Web Audio node • Create MediaStreamAudioDesRnaRonNode of Web Audio – Mix SourceNode of other members: sourceNode.connect(desRnaRonNode); • Get MediaStream – Pass to RTCPeerConnecRon. addStream(), and send to remote peer • * repeat for all members 32 -
Inside of MCU Server:Video & Audio 33 RTCPeerConnecRon A RTCPeerConnecRon
D videoTrack MediaStream (Audio B+C+D) RTCPeerConnecRon B RTCPeerConnecRon C MediaStream (Audio A+C+D) MediaStream (Audio A+B+D) MediaStream (Audio A+B+C) Combine videoTrack and audioTrack to one MediaStream addTrack() addTrack() addTrack() addTrack() MediaStream (Video A+B+C+D) -
Whole structure • Browser MCU is communicaRng over P2P WebRTC
connecRons • Signaling process is necessary before P2P – Exchange IP address, ports, and CODEC condiRons • No standard for signaling – Different between Apps / Services → Separate mixing core from peer handling 35 Browser A Browser B Browser MCU Signaling Server message message message media media -
Browser MCU Series • hSps://github.com/mganeko/browser_mcu • MCU Core … Mixing
of Video/Audio – No peer handling • P2P handling – CreaRng PeerConnecRon, exchanging SDP/ICE candidate • Sending message – over WebSocket / Socket.io • Signaling Server – Node.js – hSp/hSps server + WebSocket/socket.io server 36 Browser MCU Core Video/Audio mix (js) Browser MCU Server Sample code browser_mcu_core browser_mcu_server -
Browser MCU Series 37 Browser MCU Server(single room) HTML for
MCU HTML for members Signaling Server Node.js + ws + express Headless Chrome Browser MCU Core Video/Audio mix (js) P2P handling (js) PeerConnecRon P2P handling (js) PeerConnecRon load invoke Member browser Chrome/Firefox/ Safari Member browser Chrome/Firefox/ Safari -
Headless Chrome • Chrome 59 ~ – chrome --headless --remote-debugging-port=9222
URL – * "--remote-debugging-port" is necessary (Nov 15 2017) – * no more "--disable-gpu" from Chrome 61 • Work without Window/Window System – render video – draw Canvas – capture MediaStream from Canvas • AnimaRon event fires with requestAnimaRonFrame() 38 -
Control of Headless Chrome • Node.js modules … using Chrome
DevTools Protocol – chrome-remote-interface • hSps://github.com/cyrus-and/chrome-remote-interface – puppeteer • hSps://github.com/GoogleChrome/puppeteer • Simple implementaRon for Browser MCU series – MCU browser is connected to Node.js server over WebSocket/Socket.io • It already have connecRon for signaling • The connecRon is used for command from Node.js server – No page transiRon (Kind of Single Page App), so keep connected 39 -
PrimiRve control of Headless Chrome • Invoke from Node.js //
execute as child process let proc = childProcess.execFile(path, args); • Stop // Not possible to terminate from inside of Chrome proc.kill('SIGKILL'); 40 -
DEMO2 (mulR room, auto start chrome) • Start node server
• Connect member page → start Headless Chrome – connect – ps –ef | grep Chrome • (when all member disconnected → Chrome will quit) • (connect member again → start headless Chrome) • Kill Chrome → video will stop 41 -
Run in Docker container • Headless chrome will run in
Docker container • Made a container image, include Node.js and Headless Chrome – Single room – Browser MCU Pack (Dockerfile) • hSps://github.com/mganeko/browser_mcu_pack – base: node:latest • Clone hSps://github.com/mganeko/browser_mcu_server • Install Chrome stable • Container image on docker hub: mganeko/browser_mcu_pack – hSps://hub.docker.com/r/mganeko/browser_mcu_pack/ 42 -
Browser MCU Series 43 Browser MCU Server (single room) HTML
for MCU HTML for members Signaling Server Node.js + ws + express Browser MCU Pack (docker iamge) Headless Chrome Browser MCU Core Video/Audio mix (js) P2P handling (js) PeerConnecRon P2P handling (js) PeerConnecRon load invoke Member browser Chrome/Firefox/ Safari Member browser Chrome/Firefox/ Safari -
DEMO (Single room, Docker container) • Start container – docker
pull mganeko/browser_mcu_pack – docker run -d -p 3000:3000 --cap-add=SYS_ADMIN mganeko/ browser_mcu_pack • Connect as Member – Open hSp://localhost:3000/ – Click [StartVideo], click [Connect] • When stop container → video will stop – docker stop xxxxxxxxxx 44 skip -
Support mulRple video chat rooms • Run browser MUC for
each Room – Mix video/audio for the Room separately 46 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling /MCU Server Room1 Room2 -
Bad point of MCU • Super High CPU load in
Server – Get higher, when a new member join the room – Get much higher, when a new room added • Auto scale out is needed to correspond 47 ObservaRon on vultr Spec - CPU: 2 vCore - RAM: 4GB CPU load - IniRal (no room) … CPU 3% - 1room, 2members … CPU 200%, +300MB - 1room, 4members … CPU 200%, +50MB - 2room, 4+2members … CPU 200%,+300MB -
Hot to auto scale out? • Invoke MCU Server for
each Room automaRcally – VM instance of Container 48 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling Server Room1 Room2 MCU Server MCU Server Invoke Invoke → High server cost, difficult to control -
Make it simple: Room owner iniRaRve scale out • Room
owner provide MCU resource with a browser or a container 49 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling Server Room1 Room2 Browser MCU Docker Start by hand Start by hand -
Browser MCU docker Browser MCU Series on Web 50 Browser
MCU Service(mulRple room) HTML for MCU HTML for members Signaling Server Node.js + ws + express Test server hSps://meeRn.biz Chrome Browser MCU Core Video/Audio mix (js) P2P handling (js) PeerConnecRon P2P handling (js) PeerConnecRon Member browser Chrome/Firefox/ Safari Member browser Chrome/Firefox/ Safari Headless Chrome -
DEMO3 (mulRple room server, container MCU) • Start MCU browser,
click [Open Room] – hSps://meeRn.biz/mulR_mcu.html?room=node2017 • Or, start container for MCU – docker pull mganeko/browser_mcu_docker – docker run -d -p 9222:9222 --cap-add=SYS_ADMIN mganeko/ browser_mcu_docker node2017 • Start browser for members (including iPad Safari 11) – hSps://meeRn.biz/?room=node2017 51 -
Try Browser MCU Series • Browser MCU Series – hSps://github.com/mganeko/browser_mcu
– Core library … hSps://github.com/mganeko/browser_mcu_core • Sample for single room server… Browser MCU Server – hSps://github.com/mganeko/browser_mcu_server • Container for single room server and headless browser – hSps://github.com/mganeko/browser_mcu_pack – hSps://hub.docker.com/r/mganeko/browser_mcu_pack/ • Trial server … hSps://meeRn.biz/ – MCU … hSps://meeRn.biz/mulR_mcu.html 52 Most useful -
Conclusion • WebRTC has many topologies, not only P2P –
Bad point of MCU: Super high CPU load in server – Good point of MCU: very preferable for mobile clients • It is possible to build MCU on Web browser • To make use of browser MCU, separaRon of heavy load components is necessary • Node.js x Headless browser … extend potenRal of Node.js – Realize Video / Audio manipulaRon in server side 53 -
Thank you! 54 hSps://github.com/mganeko/browser_mcu Series Please Join: WebRTC Meetup Tokyo
WebRTC Begginers Tokyo