Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Node.js x Chrome headless for WebRTC MCU

2836c80ba2ab0c0a0fcd820c713b106d?s=47 mganeko
November 26, 2017

Node.js x Chrome headless for WebRTC MCU

Presentation for Nodefest Tokyo 2017
Building WebRTC MCU with Node.js and Headless browser.
Mix video and audio with Canvas and WebAudio, without window/desktop environment

2836c80ba2ab0c0a0fcd820c713b106d?s=128

mganeko

November 26, 2017
Tweet

Transcript

  1. Node.js x Chrome headless = WebRTC MCU Nodefest 2017 2017.11.25

    INFOCOM CORPORATION Masashi Ganeko @massie_g 1
  2. About Me •  Masashi Ganeko / @massie_g •  Working for

    research team of INFOCOM CORPORATION •  Staff of WebRTC Meetup Tokyo •  Staff of WebRTC Beginners Tokyo •  WebRTC related presentaRons –  hSps://speakerdeck.com/mganeko 2
  3. Sessions in past Nodefest •  Nodefest Tokyo 2016 LT – 

    Node.js with WebRTC DataChannel (in Japanese) –  hSps://www.slideshare.net/mganeko/nodejs-with-webrtc-datachannel •  Nodefest Tokyo 2014 –  Using Kurento Media Server with Node.js (in Japanese) –  hSp://www.slideshare.net/mganeko/nodekurento •  Nodefest Tokyo 2013 –  Let's Begin WebRTC –  hSps://speakerdeck.com/mganeko/lets-begin-webrtc 3
  4. Topic of today •  WebRTC topology is not only Peer-to-Peer

    – MCU: mixing video/audio in server •  It is possible to build MCU with Web Browser •  Manipulate video/audio as you like – with Node.js x Headless Chrome 4
  5. 1. WebRTC and MCU Why using MCU ? 5

  6. What is WebRTC •  Web Real-Rme CommunicaRon for –  Video

    –  Audio –  Data •  Open standard … anyone can use –  W3C WebRTC Working Group ... API –  IETF RTCWEB Working Group … Protocol –  Core library is open source: libwebrtc •  Designed for Web browsers, and other web connected devices •  Easy to combine with other Web technologies 10101110100…
  7. WebRTC enabled OS 7 Supported in Mac OS X /

    iOS Safari 11 (H.264 only) NEW! NEW!
  8. WebRTC Topology •  Peer to Peer … Basic –  Direct

    connecRon between browsers •  SFU (SelecRve Forwarding Unit) –  Server dispatchs video/audio streams •  MCU (MulRpoint Control Unit) –  Mix audio/audio, and re-encode in Server side 8
  9. P2P •  Basic topology –  Full mesh connecRon between browsers

    •  Good –  NO media server •  Bad –  CPU load: Super High –  Network load: Super High Browser A Browser B Browser D Browser C
  10. Why super high CPU load in P2P Browser A Original

    video encoded send send send Not This (Same encoded data for all receivers)
  11. Browser A Why super high CPU load in P2P send

    send send But this: encode for each receiver Browser A Original video encoded send send send Original video encoded encoded encoded
  12. SFU: SelecRve Forwarding Unit •  Dispatch video/audio in server – 

    NO encode / transcode in server •  Good –  CPU load of browser/client: low –  Upstream network load: low –  SFU server CPU load: low •  Bad •  Upstream network load: High
  13. MCU (MulRpoint Control Unit) •  Mix video/audio in server side

    •  Good –  CPU load of browser/client: lowest –  Upstream/downstream network load: lowest –  → Good for mobile clients •  Bad •  Server CPU load: Super High •  Video layout is fixed Browser A Browser B Browser D Browser C MCU Mix video / audio
  14. MCU for mobile •  Mix video/audio in server •  Re-encode/transcode

    in server –  Compression for each connecRon –  Different CODEC for each client PC Chrome PC Firefox Android Chrome iOS Safari MCU VP9 VP9 VP9 VP9 VP8 VP8 H.264 H.264 H.264 only
  15. MCU OSS and products •  OSS –  Kureto … hSp://www.kurento.org/

    •  looks not so acRve, amer acquisiRon by Twillio •  My slides about Node.js x Kurento Media Server (in Japanese) –  hSps://www.slideshare.net/mganeko/nodekurento –  Licode … hSp://lynckia.com/licode/ •  Products –  Dialogic PowerMedia XMS 15
  16. 2. MCU Core with Browser MCU Core processing for Video/Audio

    16
  17. What MCU does •  Receive Video/Audio from clients •  Mix

    Videos •  Mix Audios •  Send back Video/Audio to clients 17
  18. What MCU does •  Receive Video/Audio from clients •  Mix

    Videos •  Mix Audios •  Send back Video/Audio to clients 18 WebRTC PeerConnecRon WebRTC PeerConnecRon Canvas WebAudio I can do with Browser !
  19. Video mixing 19

  20. Inside of Browser MCU:Video 20 RTCPeerConnecRon A MediaStream <video> element

    RTCPeerConnecRon D MediaStream <canvas> element drawImage() Keep drawing with requestAnimaRonFrame() <video> element drawImage()
  21. Inside of Browser MCU:Video 21 RTCPeerConnecRon A RTCPeerConnecRon D <canvas>

    element MediaStream Get video with captureStream(fps)
  22. Inside of Browser MCU:Video 22 RTCPeerConnecRon A MediaStream <video> element

    RTCPeerConnecRon D MediaStream <canvas> element drawImage() Keep drawing with requestAnimaRonFrame() <video> element drawImage() MediaStream Get video with captureStream(fps)
  23. Steps to mix video in Browser MCU •  Get Remote

    MediaStream through RTCPeerConnecRon –  play in <video> element •  Get context of Canvas by getContext('2d') •  Keep drawing with window.requestAnimaRonFrame() –  Copy image of each video element to canvas, with drawImage() •  Capture MadiaStream from Canvas, with captureStream(fps) –  Pass it to RTCPeerConnecRon.addStream(), and send it to remote peer 23
  24. DEMO1: Mixing video with Browser MCU •  Start node server

    •  Open Browser for MCU •  Open client browser and connect •  Demo case of Browser MCU hidden 24
  25. Tricky points of MCU Server:Video •  Using window.requestAnimaRonFrame() –  Will

    not be called when the window/tab is completely hidden –  AnimaRon will stop if the window is minimized or other hidden by other tab •  Even using window.setInterval() –  The interval will be much longer than specified, the window/tab is completely hidden 25
  26. Audio mixing 26

  27. Tricky points of MCU Server:Audio •  One mixed video is

    enough for all members •  Audio should be prepared differently for each member –  It is hand to speak, if your voice will come back with short delay –  Maybe familiar case with WebRTC applicaRon developers 27
  28. Inside MCU Server:Audio •  Prepare "minus one" Audio –  Mix

    audio of all members, except the user •  Need many combinaRon of audio –  4 combinaRon for 4 members –  N combinaRon for N members 28 Browser A Browser B Browser D Browser C MCU Mixing Audio is hassle Audio is much more difficult than Video
  29. Inside MCU Server:Audio 29 RTCPeerConnecRon A MediaStream MediaStreamAudioSourceNode Create with

    AudioContext . createMediaStreamSource()
  30. Inside MCU Server:Audio 30 RTCPeerConnecRon D MediaStream MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode

    MediaStreamAudioDesRnaRonNode mix RTCPeerConnecRon A Create with AudioContext . createMediaStreamDesRnaRon() RTCPeerConnecRon B RTCPeerConnecRon C
  31. Inside MCU Server:Audio 31 RTCPeerConnecRon A MediaStream RTCPeerConnecRon D MediaStream

    MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioDesRnaRonNode Mix Repeat for N paSern RTCPeerConnecRon A RTCPeerConnecRon B RTCPeerConnecRon C
  32. Steps to mix audio in Browser MCU •  Get remote

    MediaStream through RTCPeerConnecRon •  Create MediaStreamAudioSourceNode of Web Audio –  Convert MediaStream to Web Audio node •  Create MediaStreamAudioDesRnaRonNode of Web Audio –  Mix SourceNode of other members: sourceNode.connect(desRnaRonNode); •  Get MediaStream –  Pass to RTCPeerConnecRon. addStream(), and send to remote peer •  * repeat for all members 32
  33. Inside of MCU Server:Video & Audio 33 RTCPeerConnecRon A RTCPeerConnecRon

    D videoTrack MediaStream (Audio B+C+D) RTCPeerConnecRon B RTCPeerConnecRon C MediaStream (Audio A+C+D) MediaStream (Audio A+B+D) MediaStream (Audio A+B+C) Combine videoTrack and audioTrack to one MediaStream addTrack() addTrack() addTrack() addTrack() MediaStream (Video A+B+C+D)
  34. 3. MCU Server with Node.js + headless Browser 34

  35. Whole structure •  Browser MCU is communicaRng over P2P WebRTC

    connecRons •  Signaling process is necessary before P2P –  Exchange IP address, ports, and CODEC condiRons •  No standard for signaling –  Different between Apps / Services → Separate mixing core from peer handling 35 Browser A Browser B Browser MCU Signaling Server message message message media media
  36. Browser MCU Series •  hSps://github.com/mganeko/browser_mcu •  MCU Core … Mixing

    of Video/Audio –  No peer handling •  P2P handling –  CreaRng PeerConnecRon, exchanging SDP/ICE candidate •  Sending message –  over WebSocket / Socket.io •  Signaling Server –  Node.js –  hSp/hSps server + WebSocket/socket.io server 36 Browser MCU Core Video/Audio mix (js) Browser MCU Server Sample code browser_mcu_core browser_mcu_server
  37. Browser MCU Series 37 Browser MCU Server(single room) HTML for

    MCU HTML for members Signaling Server Node.js + ws + express Headless Chrome Browser MCU Core Video/Audio mix (js) P2P handling (js) PeerConnecRon P2P handling (js) PeerConnecRon load invoke Member browser Chrome/Firefox/ Safari Member browser Chrome/Firefox/ Safari
  38. Headless Chrome •  Chrome 59 ~ –  chrome --headless --remote-debugging-port=9222

    URL –  * "--remote-debugging-port" is necessary (Nov 15 2017) –  * no more "--disable-gpu" from Chrome 61 •  Work without Window/Window System –  render video –  draw Canvas –  capture MediaStream from Canvas •  AnimaRon event fires with requestAnimaRonFrame() 38
  39. Control of Headless Chrome •  Node.js modules … using Chrome

    DevTools Protocol –  chrome-remote-interface •  hSps://github.com/cyrus-and/chrome-remote-interface –  puppeteer •  hSps://github.com/GoogleChrome/puppeteer •  Simple implementaRon for Browser MCU series –  MCU browser is connected to Node.js server over WebSocket/Socket.io •  It already have connecRon for signaling •  The connecRon is used for command from Node.js server –  No page transiRon (Kind of Single Page App), so keep connected 39
  40. PrimiRve control of Headless Chrome •  Invoke from Node.js //

    execute as child process let proc = childProcess.execFile(path, args); •  Stop // Not possible to terminate from inside of Chrome proc.kill('SIGKILL'); 40
  41. DEMO2 (mulR room, auto start chrome) •  Start node server

    •  Connect member page → start Headless Chrome –  connect –  ps –ef | grep Chrome •  (when all member disconnected → Chrome will quit) •  (connect member again → start headless Chrome) •  Kill Chrome → video will stop 41
  42. Run in Docker container •  Headless chrome will run in

    Docker container •  Made a container image, include Node.js and Headless Chrome –  Single room –  Browser MCU Pack (Dockerfile) •  hSps://github.com/mganeko/browser_mcu_pack –  base: node:latest •  Clone hSps://github.com/mganeko/browser_mcu_server  •  Install Chrome stable •  Container image on docker hub: mganeko/browser_mcu_pack –  hSps://hub.docker.com/r/mganeko/browser_mcu_pack/ 42
  43. Browser MCU Series 43 Browser MCU Server (single room) HTML

    for MCU HTML for members Signaling Server Node.js + ws + express Browser MCU Pack (docker iamge) Headless Chrome Browser MCU Core Video/Audio mix (js) P2P handling (js) PeerConnecRon P2P handling (js) PeerConnecRon load invoke Member browser Chrome/Firefox/ Safari Member browser Chrome/Firefox/ Safari
  44. DEMO (Single room, Docker container) •  Start container –  docker

    pull mganeko/browser_mcu_pack –  docker run -d -p 3000:3000 --cap-add=SYS_ADMIN mganeko/ browser_mcu_pack •  Connect as Member –  Open hSp://localhost:3000/ –  Click [StartVideo], click [Connect] •  When stop container → video will stop –  docker stop xxxxxxxxxx 44 skip
  45. 4. Scale out for MulRple rooms 45

  46. Support mulRple video chat rooms •  Run browser MUC for

    each Room –  Mix video/audio for the Room separately 46 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling /MCU Server Room1 Room2
  47. Bad point of MCU •  Super High CPU load in

    Server –  Get higher, when a new member join the room –  Get much higher, when a new room added •  Auto scale out is needed to correspond 47 ObservaRon on vultr Spec -  CPU: 2 vCore -  RAM: 4GB CPU load -  IniRal (no room) … CPU 3% -  1room, 2members … CPU 200%, +300MB -  1room, 4members … CPU 200%, +50MB -  2room, 4+2members … CPU 200%,+300MB
  48. Hot to auto scale out? •  Invoke MCU Server for

    each Room automaRcally –  VM instance of Container 48 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling Server Room1 Room2 MCU Server MCU Server Invoke Invoke → High server cost, difficult to control
  49. Make it simple: Room owner iniRaRve scale out •  Room

    owner provide MCU resource with a browser or a container 49 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling Server Room1 Room2 Browser MCU Docker Start by hand Start by hand
  50. Browser MCU docker Browser MCU Series on Web 50 Browser

    MCU Service(mulRple room) HTML for MCU HTML for members Signaling Server Node.js + ws + express Test server hSps://meeRn.biz Chrome Browser MCU Core Video/Audio mix (js) P2P handling (js) PeerConnecRon P2P handling (js) PeerConnecRon Member browser Chrome/Firefox/ Safari Member browser Chrome/Firefox/ Safari Headless Chrome
  51. DEMO3 (mulRple room server, container MCU) •  Start MCU browser,

    click [Open Room] –  hSps://meeRn.biz/mulR_mcu.html?room=node2017 •  Or, start container for MCU –  docker pull mganeko/browser_mcu_docker –  docker run -d -p 9222:9222 --cap-add=SYS_ADMIN mganeko/ browser_mcu_docker node2017 •  Start browser for members (including iPad Safari 11) –  hSps://meeRn.biz/?room=node2017 51
  52. Try Browser MCU Series •  Browser MCU Series –  hSps://github.com/mganeko/browser_mcu

    –  Core library … hSps://github.com/mganeko/browser_mcu_core •  Sample for single room server… Browser MCU Server –  hSps://github.com/mganeko/browser_mcu_server •  Container for single room server and headless browser –  hSps://github.com/mganeko/browser_mcu_pack –  hSps://hub.docker.com/r/mganeko/browser_mcu_pack/ •  Trial server … hSps://meeRn.biz/ –  MCU … hSps://meeRn.biz/mulR_mcu.html 52 Most useful
  53. Conclusion •  WebRTC has many topologies, not only P2P – 

    Bad point of MCU: Super high CPU load in server –  Good point of MCU: very preferable for mobile clients •  It is possible to build MCU on Web browser •  To make use of browser MCU, separaRon of heavy load components is necessary •  Node.js x Headless browser … extend potenRal of Node.js –  Realize Video / Audio manipulaRon in server side 53
  54. Thank you! 54 hSps://github.com/mganeko/browser_mcu Series Please Join: WebRTC Meetup Tokyo

    WebRTC Begginers Tokyo