Node.js x Chrome headless for WebRTC MCU

Presentation for Nodefest Tokyo 2017
Building WebRTC MCU with Node.js and Headless browser.
Mix video and audio with Canvas and WebAudio, without window/desktop environment

Transcript

1. Node.js x Chrome headless = WebRTC MCU Nodefest 2017 2017.11.25 INFOCOM CORPORATION Masashi Ganeko @massie_g 1
2. About Me •  Masashi Ganeko / @massie_g •  Working for research team of INFOCOM CORPORATION •  Staff of WebRTC Meetup Tokyo •  Staff of WebRTC Beginners Tokyo •  WebRTC related presentaRons –  hSps://speakerdeck.com/mganeko 2
3. Sessions in past Nodefest •  Nodefest Tokyo 2016 LT –  Node.js with WebRTC DataChannel (in Japanese) –  hSps://www.slideshare.net/mganeko/nodejs-with-webrtc-datachannel •  Nodefest Tokyo 2014 –  Using Kurento Media Server with Node.js (in Japanese) –  hSp://www.slideshare.net/mganeko/nodekurento •  Nodefest Tokyo 2013 –  Let's Begin WebRTC –  hSps://speakerdeck.com/mganeko/lets-begin-webrtc 3
4. Topic of today •  WebRTC topology is not only Peer-to-Peer – MCU: mixing video/audio in server •  It is possible to build MCU with Web Browser •  Manipulate video/audio as you like – with Node.js x Headless Chrome 4
5. 1. WebRTC and MCU Why using MCU ? 5
6. What is WebRTC •  Web Real-Rme CommunicaRon for –  Video –  Audio –  Data •  Open standard … anyone can use –  W3C WebRTC Working Group ... API –  IETF RTCWEB Working Group … Protocol –  Core library is open source: libwebrtc •  Designed for Web browsers, and other web connected devices •  Easy to combine with other Web technologies 10101110100…
7. WebRTC enabled OS 7 Supported in Mac OS X / iOS Safari 11 (H.264 only) NEW! NEW!
8. WebRTC Topology •  Peer to Peer … Basic –  Direct connecRon between browsers •  SFU (SelecRve Forwarding Unit) –  Server dispatchs video/audio streams •  MCU (MulRpoint Control Unit) –  Mix audio/audio, and re-encode in Server side 8
9. P2P •  Basic topology –  Full mesh connecRon between browsers •  Good –  NO media server •  Bad –  CPU load: Super High –  Network load: Super High Browser A Browser B Browser D Browser C
10. Why super high CPU load in P2P Browser A Original video encoded send send send Not This （Same encoded data for all receivers）
11. Browser A Why super high CPU load in P2P send send send But this: encode for each receiver Browser A Original video encoded send send send Original video encoded encoded encoded
12. SFU: SelecRve Forwarding Unit •  Dispatch video/audio in server –  NO encode / transcode in server •  Good –  CPU load of browser/client: low –  Upstream network load: low –  SFU server CPU load: low •  Bad •  Upstream network load: High
13. MCU (MulRpoint Control Unit) •  Mix video/audio in server side •  Good –  CPU load of browser/client: lowest –  Upstream/downstream network load: lowest –  →　Good for mobile clients •  Bad •  Server CPU load: Super High •  Video layout is fixed Browser A Browser B Browser D Browser C MCU Mix video / audio
14. MCU for mobile •  Mix video/audio in server •  Re-encode/transcode in server –  Compression for each connecRon –  Different CODEC for each client PC Chrome PC Firefox Android Chrome iOS Safari MCU VP9 VP9 VP9 VP9 VP8 VP8 H.264 H.264 H.264 only
15. MCU OSS and products •  OSS –  Kureto … hSp://www.kurento.org/ •  looks not so acRve, amer acquisiRon by Twillio •  My slides about Node.js x Kurento Media Server (in Japanese) –  hSps://www.slideshare.net/mganeko/nodekurento –  Licode … hSp://lynckia.com/licode/ •  Products –  Dialogic PowerMedia XMS 15
16. 2. MCU Core with Browser MCU Core processing for Video/Audio 16
17. What MCU does •  Receive Video/Audio from clients •  Mix Videos •  Mix Audios •  Send back Video/Audio to clients 17
18. What MCU does •  Receive Video/Audio from clients •  Mix Videos •  Mix Audios •  Send back Video/Audio to clients 18 WebRTC PeerConnecRon WebRTC PeerConnecRon Canvas WebAudio I can do with Browser !
19. Video mixing 19
20. Inside of Browser MCU：Video 20 RTCPeerConnecRon A MediaStream <video> element RTCPeerConnecRon D MediaStream <canvas> element drawImage() Keep drawing with requestAnimaRonFrame() <video> element drawImage()
21. Inside of Browser MCU：Video 21 RTCPeerConnecRon A RTCPeerConnecRon D <canvas> element MediaStream Get video with captureStream(fps)
22. Inside of Browser MCU：Video 22 RTCPeerConnecRon A MediaStream <video> element RTCPeerConnecRon D MediaStream <canvas> element drawImage() Keep drawing with requestAnimaRonFrame() <video> element drawImage() MediaStream Get video with captureStream(fps)
23. Steps to mix video in Browser MCU •  Get Remote MediaStream through RTCPeerConnecRon –  play in <video> element •  Get context of Canvas by getContext('2d') •  Keep drawing with window.requestAnimaRonFrame() –  Copy image of each video element to canvas, with drawImage() •  Capture MadiaStream from Canvas, with captureStream(fps) –  Pass it to RTCPeerConnecRon.addStream(), and send it to remote peer 23
24. DEMO1： Mixing video with Browser MCU •  Start node server •  Open Browser for MCU •  Open client browser and connect •  Demo case of Browser MCU hidden 24
25. Tricky points of MCU Server：Video •  Using window.requestAnimaRonFrame() –  Will not be called when the window/tab is completely hidden –  AnimaRon will stop if the window is minimized or other hidden by other tab •  Even using window.setInterval() –  The interval will be much longer than specified, the window/tab is completely hidden 25
26. Audio mixing 26
27. Tricky points of MCU Server：Audio •  One mixed video is enough for all members •  Audio should be prepared differently for each member –  It is hand to speak, if your voice will come back with short delay –  Maybe familiar case with WebRTC applicaRon developers 27
28. Inside MCU Server：Audio •  Prepare "minus one" Audio –  Mix audio of all members, except the user •  Need many combinaRon of audio –  4 combinaRon for 4 members –  N combinaRon for N members 28 Browser A Browser B Browser D Browser C MCU Mixing Audio is hassle Audio is much more difficult than Video
29. Inside MCU Server：Audio 29 RTCPeerConnecRon A MediaStream MediaStreamAudioSourceNode Create with AudioContext . createMediaStreamSource()
30. Inside MCU Server：Audio 30 RTCPeerConnecRon D MediaStream MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioDesRnaRonNode mix RTCPeerConnecRon A Create with AudioContext . createMediaStreamDesRnaRon() RTCPeerConnecRon B RTCPeerConnecRon C
31. Inside MCU Server：Audio 31 RTCPeerConnecRon A MediaStream RTCPeerConnecRon D MediaStream MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioDesRnaRonNode Mix Repeat for N paSern RTCPeerConnecRon A RTCPeerConnecRon B RTCPeerConnecRon C
32. Steps to mix audio in Browser MCU •  Get remote MediaStream through RTCPeerConnecRon •  Create MediaStreamAudioSourceNode of Web Audio –  Convert MediaStream to Web Audio node •  Create MediaStreamAudioDesRnaRonNode of Web Audio –  Mix SourceNode of other members: sourceNode.connect(desRnaRonNode); •  Get MediaStream –  Pass to RTCPeerConnecRon. addStream(), and send to remote peer •  * repeat for all members 32
33. Inside of MCU Server：Video & Audio 33 RTCPeerConnecRon A RTCPeerConnecRon D videoTrack MediaStream (Audio B+C+D) RTCPeerConnecRon B RTCPeerConnecRon C MediaStream (Audio A+C+D) MediaStream (Audio A+B+D) MediaStream (Audio A+B+C) Combine videoTrack and audioTrack to one MediaStream addTrack() addTrack() addTrack() addTrack() MediaStream (Video A+B+C+D)
34. 3. MCU Server with Node.js + headless Browser 34
35. Whole structure •  Browser MCU is communicaRng over P2P WebRTC connecRons •  Signaling process is necessary before P2P –  Exchange IP address, ports, and CODEC condiRons •  No standard for signaling –  Different between Apps / Services → Separate mixing core from peer handling 35 Browser A Browser B Browser MCU Signaling Server message message message media media
36. Browser MCU Series •  hSps://github.com/mganeko/browser_mcu •  MCU Core … Mixing of Video/Audio –  No peer handling •  P2P handling –  CreaRng PeerConnecRon, exchanging SDP/ICE candidate •  Sending message –  over WebSocket / Socket.io •  Signaling Server –  Node.js –  hSp/hSps server + WebSocket/socket.io server 36 Browser MCU Core Video/Audio mix (js) Browser MCU Server Sample code browser_mcu_core browser_mcu_server
37. Browser MCU Series 37 Browser MCU Server(single room) HTML for MCU HTML for members Signaling Server Node.js + ws + express Headless Chrome Browser MCU Core Video/Audio mix (js) P2P handling (js) PeerConnecRon P2P handling (js) PeerConnecRon load invoke Member browser Chrome/Firefox/ Safari Member browser Chrome/Firefox/ Safari
38. Headless Chrome •  Chrome 59 ～ –  chrome --headless --remote-debugging-port=9222 URL –  * "--remote-debugging-port" is necessary (Nov 15 2017) –  * no more "--disable-gpu" from Chrome 61 •  Work without Window/Window System –  render video –  draw Canvas –  capture MediaStream from Canvas •  AnimaRon event fires with requestAnimaRonFrame() 38
39. Control of Headless Chrome •  Node.js modules … using Chrome DevTools Protocol –  chrome-remote-interface •  hSps://github.com/cyrus-and/chrome-remote-interface –  puppeteer •  hSps://github.com/GoogleChrome/puppeteer •  Simple implementaRon for Browser MCU series –  MCU browser is connected to Node.js server over WebSocket/Socket.io •  It already have connecRon for signaling •  The connecRon is used for command from Node.js server –  No page transiRon (Kind of Single Page App), so keep connected 39
40. PrimiRve control of Headless Chrome •  Invoke from Node.js // execute as child process let proc = childProcess.execFile(path, args); •  Stop // Not possible to terminate from inside of Chrome proc.kill('SIGKILL'); 40
41. DEMO2 (mulR room, auto start chrome) •  Start node server •  Connect member page → start Headless Chrome –  connect –  ps –ef | grep Chrome •  （when all member disconnected　→ Chrome will quit） •  （connect member again → start headless Chrome） •  Kill Chrome → video will stop 41
42. Run in Docker container •  Headless chrome will run in Docker container •  Made a container image, include Node.js and Headless Chrome –  Single room –  Browser MCU Pack (Dockerfile) •  hSps://github.com/mganeko/browser_mcu_pack –  base： node:latest •  Clone hSps://github.com/mganeko/browser_mcu_server　 •  Install Chrome stable •  Container image on docker hub: mganeko/browser_mcu_pack –  hSps://hub.docker.com/r/mganeko/browser_mcu_pack/ 42
43. Browser MCU Series 43 Browser MCU Server (single room) HTML for MCU HTML for members Signaling Server Node.js + ws + express Browser MCU Pack (docker iamge) Headless Chrome Browser MCU Core Video/Audio mix (js) P2P handling (js) PeerConnecRon P2P handling (js) PeerConnecRon load invoke Member browser Chrome/Firefox/ Safari Member browser Chrome/Firefox/ Safari
44. DEMO (Single room, Docker container) •  Start container –  docker pull mganeko/browser_mcu_pack –  docker run -d -p 3000:3000 --cap-add=SYS_ADMIN mganeko/ browser_mcu_pack •  Connect as Member –  Open hSp://localhost:3000/ –  Click [StartVideo], click [Connect] •  When stop container → video will stop –  docker stop xxxxxxxxxx 44 skip
45. 4. Scale out for MulRple rooms 45
46. Support mulRple video chat rooms •  Run browser MUC for each Room –  Mix video/audio for the Room separately 46 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling /MCU Server Room1 Room2
47. Bad point of MCU •  Super High CPU load in Server –  Get higher, when a new member join the room –  Get much higher, when a new room added •  Auto scale out is needed to correspond 47 ObservaRon on vultr Spec -  CPU: 2 vCore -  RAM: 4GB CPU load -  IniRal (no room) … CPU 3% -  1room, 2members … CPU 200%, +300MB -  1room, 4members … CPU 200%, +50MB -  2room, 4+2members … CPU 200%,+300MB
48. Hot to auto scale out? •  Invoke MCU Server for each Room automaRcally –  VM instance of Container 48 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling Server Room1 Room2 MCU Server MCU Server Invoke Invoke → High server cost, difficult to control
49. Make it simple: Room owner iniRaRve scale out •  Room owner provide MCU resource with a browser or a container 49 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling Server Room1 Room2 Browser MCU Docker Start by hand Start by hand
50. Browser MCU docker Browser MCU Series on Web 50 Browser MCU Service（mulRple room) HTML for MCU HTML for members Signaling Server Node.js + ws + express Test server hSps://meeRn.biz Chrome Browser MCU Core Video/Audio mix (js) P2P handling (js) PeerConnecRon P2P handling (js) PeerConnecRon Member browser Chrome/Firefox/ Safari Member browser Chrome/Firefox/ Safari Headless Chrome
51. DEMO3 (mulRple room server, container MCU) •  Start MCU browser, click [Open Room] –  hSps://meeRn.biz/mulR_mcu.html?room=node2017 •  Or, start container for MCU –  docker pull mganeko/browser_mcu_docker –  docker run -d -p 9222:9222 --cap-add=SYS_ADMIN mganeko/ browser_mcu_docker node2017 •  Start browser for members (including iPad Safari 11） –  hSps://meeRn.biz/?room=node2017 51
52. Try Browser MCU Series •  Browser MCU Series –  hSps://github.com/mganeko/browser_mcu –  Core library … hSps://github.com/mganeko/browser_mcu_core •  Sample for single room server… Browser MCU Server –  hSps://github.com/mganeko/browser_mcu_server •  Container for single room server and headless browser –  hSps://github.com/mganeko/browser_mcu_pack –  hSps://hub.docker.com/r/mganeko/browser_mcu_pack/ •  Trial server … hSps://meeRn.biz/ –  MCU … hSps://meeRn.biz/mulR_mcu.html 52 Most useful
53. Conclusion •  WebRTC has many topologies, not only P2P –  Bad point of MCU: Super high CPU load in server –  Good point of MCU: very preferable for mobile clients •  It is possible to build MCU on Web browser •  To make use of browser MCU, separaRon of heavy load components is necessary •  Node.js x Headless browser … extend potenRal of Node.js –  Realize Video / Audio manipulaRon in server side 53
54. Thank you! 54 hSps://github.com/mganeko/browser_mcu Series Please Join: WebRTC Meetup Tokyo WebRTC Begginers Tokyo