Tweet
Tweet

Slide 1

Slide 1 text

Node.js x Chrome headless = WebRTC MCU Nodefest 2017 2017.11.25 INFOCOM CORPORATION Masashi Ganeko @massie_g 1

Slide 2

Slide 2 text

About Me •  Masashi Ganeko / @massie_g •  Working for research team of INFOCOM CORPORATION •  Staff of WebRTC Meetup Tokyo •  Staff of WebRTC Beginners Tokyo •  WebRTC related presentaRons –  hSps://speakerdeck.com/mganeko 2

Slide 3

Slide 3 text

Sessions in past Nodefest •  Nodefest Tokyo 2016 LT –  Node.js with WebRTC DataChannel (in Japanese) –  hSps://www.slideshare.net/mganeko/nodejs-with-webrtc-datachannel •  Nodefest Tokyo 2014 –  Using Kurento Media Server with Node.js (in Japanese) –  hSp://www.slideshare.net/mganeko/nodekurento •  Nodefest Tokyo 2013 –  Let's Begin WebRTC –  hSps://speakerdeck.com/mganeko/lets-begin-webrtc 3

Slide 4

Slide 4 text

Topic of today •  WebRTC topology is not only Peer-to-Peer – MCU: mixing video/audio in server •  It is possible to build MCU with Web Browser •  Manipulate video/audio as you like – with Node.js x Headless Chrome 4

Slide 5

Slide 5 text

1. WebRTC and MCU Why using MCU ? 5

Slide 6

Slide 6 text

What is WebRTC •  Web Real-Rme CommunicaRon for –  Video –  Audio –  Data •  Open standard … anyone can use –  W3C WebRTC Working Group ... API –  IETF RTCWEB Working Group … Protocol –  Core library is open source: libwebrtc •  Designed for Web browsers, and other web connected devices •  Easy to combine with other Web technologies 10101110100…

Slide 7

Slide 7 text

WebRTC enabled OS 7 Supported in Mac OS X / iOS Safari 11 (H.264 only) NEW! NEW!

Slide 8

Slide 8 text

WebRTC Topology •  Peer to Peer … Basic –  Direct connecRon between browsers •  SFU (SelecRve Forwarding Unit) –  Server dispatchs video/audio streams •  MCU (MulRpoint Control Unit) –  Mix audio/audio, and re-encode in Server side 8

Slide 9

Slide 9 text

P2P •  Basic topology –  Full mesh connecRon between browsers •  Good –  NO media server •  Bad –  CPU load: Super High –  Network load: Super High Browser A Browser B Browser D Browser C

Slide 10

Slide 10 text

Why super high CPU load in P2P Browser A Original video encoded send send send Not This （Same encoded data for all receivers）

Slide 11

Slide 11 text

Browser A Why super high CPU load in P2P send send send But this: encode for each receiver Browser A Original video encoded send send send Original video encoded encoded encoded

Slide 12

Slide 12 text

SFU: SelecRve Forwarding Unit •  Dispatch video/audio in server –  NO encode / transcode in server •  Good –  CPU load of browser/client: low –  Upstream network load: low –  SFU server CPU load: low •  Bad •  Upstream network load: High

Slide 13

Slide 13 text

MCU (MulRpoint Control Unit) •  Mix video/audio in server side •  Good –  CPU load of browser/client: lowest –  Upstream/downstream network load: lowest –  →　Good for mobile clients •  Bad •  Server CPU load: Super High •  Video layout is fixed Browser A Browser B Browser D Browser C MCU Mix video / audio

Slide 14

Slide 14 text

MCU for mobile •  Mix video/audio in server •  Re-encode/transcode in server –  Compression for each connecRon –  Different CODEC for each client PC Chrome PC Firefox Android Chrome iOS Safari MCU VP9 VP9 VP9 VP9 VP8 VP8 H.264 H.264 H.264 only

Slide 15

Slide 15 text

MCU OSS and products •  OSS –  Kureto … hSp://www.kurento.org/ •  looks not so acRve, amer acquisiRon by Twillio •  My slides about Node.js x Kurento Media Server (in Japanese) –  hSps://www.slideshare.net/mganeko/nodekurento –  Licode … hSp://lynckia.com/licode/ •  Products –  Dialogic PowerMedia XMS 15

Slide 16

Slide 16 text

2. MCU Core with Browser MCU Core processing for Video/Audio 16

Slide 17

Slide 17 text

What MCU does •  Receive Video/Audio from clients •  Mix Videos •  Mix Audios •  Send back Video/Audio to clients 17

Slide 18

Slide 18 text

What MCU does •  Receive Video/Audio from clients •  Mix Videos •  Mix Audios •  Send back Video/Audio to clients 18 WebRTC PeerConnecRon WebRTC PeerConnecRon Canvas WebAudio I can do with Browser !

Slide 19

Slide 19 text

Video mixing 19

Slide 20

Slide 20 text

Inside of Browser MCU：Video 20 RTCPeerConnecRon A MediaStream element RTCPeerConnecRon D MediaStream element drawImage() Keep drawing with requestAnimaRonFrame() element drawImage()

Slide 21

Slide 21 text

Inside of Browser MCU：Video 21 RTCPeerConnecRon A RTCPeerConnecRon D element MediaStream Get video with captureStream(fps)

Slide 22

Slide 22 text

Inside of Browser MCU：Video 22 RTCPeerConnecRon A MediaStream element RTCPeerConnecRon D MediaStream element drawImage() Keep drawing with requestAnimaRonFrame() element drawImage() MediaStream Get video with captureStream(fps)

Slide 23

Slide 23 text

Steps to mix video in Browser MCU •  Get Remote MediaStream through RTCPeerConnecRon –  play in element •  Get context of Canvas by getContext('2d') •  Keep drawing with window.requestAnimaRonFrame() –  Copy image of each video element to canvas, with drawImage() •  Capture MadiaStream from Canvas, with captureStream(fps) –  Pass it to RTCPeerConnecRon.addStream(), and send it to remote peer 23

Slide 24

Slide 24 text

DEMO1： Mixing video with Browser MCU •  Start node server •  Open Browser for MCU •  Open client browser and connect •  Demo case of Browser MCU hidden 24

Slide 25

Slide 25 text

Tricky points of MCU Server：Video •  Using window.requestAnimaRonFrame() –  Will not be called when the window/tab is completely hidden –  AnimaRon will stop if the window is minimized or other hidden by other tab •  Even using window.setInterval() –  The interval will be much longer than specified, the window/tab is completely hidden 25

Slide 26

Slide 26 text

Audio mixing 26

Slide 27

Slide 27 text

Tricky points of MCU Server：Audio •  One mixed video is enough for all members •  Audio should be prepared differently for each member –  It is hand to speak, if your voice will come back with short delay –  Maybe familiar case with WebRTC applicaRon developers 27

Slide 28

Slide 28 text

Inside MCU Server：Audio •  Prepare "minus one" Audio –  Mix audio of all members, except the user •  Need many combinaRon of audio –  4 combinaRon for 4 members –  N combinaRon for N members 28 Browser A Browser B Browser D Browser C MCU Mixing Audio is hassle Audio is much more difficult than Video

Slide 29

Slide 29 text

Inside MCU Server：Audio 29 RTCPeerConnecRon A MediaStream MediaStreamAudioSourceNode Create with AudioContext . createMediaStreamSource()

Slide 30

Slide 30 text

Inside MCU Server：Audio 30 RTCPeerConnecRon D MediaStream MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioDesRnaRonNode mix RTCPeerConnecRon A Create with AudioContext . createMediaStreamDesRnaRon() RTCPeerConnecRon B RTCPeerConnecRon C

Slide 31

Slide 31 text

Inside MCU Server：Audio 31 RTCPeerConnecRon A MediaStream RTCPeerConnecRon D MediaStream MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioDesRnaRonNode Mix Repeat for N paSern RTCPeerConnecRon A RTCPeerConnecRon B RTCPeerConnecRon C

Slide 32

Slide 32 text

Steps to mix audio in Browser MCU •  Get remote MediaStream through RTCPeerConnecRon •  Create MediaStreamAudioSourceNode of Web Audio –  Convert MediaStream to Web Audio node •  Create MediaStreamAudioDesRnaRonNode of Web Audio –  Mix SourceNode of other members: sourceNode.connect(desRnaRonNode); •  Get MediaStream –  Pass to RTCPeerConnecRon. addStream(), and send to remote peer •  * repeat for all members 32

Slide 33

Slide 33 text

Inside of MCU Server：Video & Audio 33 RTCPeerConnecRon A RTCPeerConnecRon D videoTrack MediaStream (Audio B+C+D) RTCPeerConnecRon B RTCPeerConnecRon C MediaStream (Audio A+C+D) MediaStream (Audio A+B+D) MediaStream (Audio A+B+C) Combine videoTrack and audioTrack to one MediaStream addTrack() addTrack() addTrack() addTrack() MediaStream (Video A+B+C+D)

Slide 34

Slide 34 text

3. MCU Server with Node.js + headless Browser 34

Slide 35

Slide 35 text

Whole structure •  Browser MCU is communicaRng over P2P WebRTC connecRons •  Signaling process is necessary before P2P –  Exchange IP address, ports, and CODEC condiRons •  No standard for signaling –  Different between Apps / Services → Separate mixing core from peer handling 35 Browser A Browser B Browser MCU Signaling Server message message message media media

Slide 36

Slide 36 text

Browser MCU Series •  hSps://github.com/mganeko/browser_mcu •  MCU Core … Mixing of Video/Audio –  No peer handling •  P2P handling –  CreaRng PeerConnecRon, exchanging SDP/ICE candidate •  Sending message –  over WebSocket / Socket.io •  Signaling Server –  Node.js –  hSp/hSps server + WebSocket/socket.io server 36 Browser MCU Core Video/Audio mix (js) Browser MCU Server Sample code browser_mcu_core browser_mcu_server

Slide 37

Slide 37 text

Browser MCU Series 37 Browser MCU Server(single room) HTML for MCU HTML for members Signaling Server Node.js + ws + express Headless Chrome Browser MCU Core Video/Audio mix (js) P2P handling (js) PeerConnecRon P2P handling (js) PeerConnecRon load invoke Member browser Chrome/Firefox/ Safari Member browser Chrome/Firefox/ Safari

Slide 38

Slide 38 text

Headless Chrome •  Chrome 59 ～ –  chrome --headless --remote-debugging-port=9222 URL –  * "--remote-debugging-port" is necessary (Nov 15 2017) –  * no more "--disable-gpu" from Chrome 61 •  Work without Window/Window System –  render video –  draw Canvas –  capture MediaStream from Canvas •  AnimaRon event fires with requestAnimaRonFrame() 38

Slide 39

Slide 39 text

Control of Headless Chrome •  Node.js modules … using Chrome DevTools Protocol –  chrome-remote-interface •  hSps://github.com/cyrus-and/chrome-remote-interface –  puppeteer •  hSps://github.com/GoogleChrome/puppeteer •  Simple implementaRon for Browser MCU series –  MCU browser is connected to Node.js server over WebSocket/Socket.io •  It already have connecRon for signaling •  The connecRon is used for command from Node.js server –  No page transiRon (Kind of Single Page App), so keep connected 39

Slide 40

Slide 40 text

PrimiRve control of Headless Chrome •  Invoke from Node.js // execute as child process let proc = childProcess.execFile(path, args); •  Stop // Not possible to terminate from inside of Chrome proc.kill('SIGKILL'); 40

Slide 41

Slide 41 text

DEMO2 (mulR room, auto start chrome) •  Start node server •  Connect member page → start Headless Chrome –  connect –  ps –ef | grep Chrome •  （when all member disconnected　→ Chrome will quit） •  （connect member again → start headless Chrome） •  Kill Chrome → video will stop 41

Slide 42

Slide 42 text

Run in Docker container •  Headless chrome will run in Docker container •  Made a container image, include Node.js and Headless Chrome –  Single room –  Browser MCU Pack (Dockerfile) •  hSps://github.com/mganeko/browser_mcu_pack –  base： node:latest •  Clone hSps://github.com/mganeko/browser_mcu_server　 •  Install Chrome stable •  Container image on docker hub: mganeko/browser_mcu_pack –  hSps://hub.docker.com/r/mganeko/browser_mcu_pack/ 42

Slide 43

Slide 43 text

Browser MCU Series 43 Browser MCU Server (single room) HTML for MCU HTML for members Signaling Server Node.js + ws + express Browser MCU Pack (docker iamge) Headless Chrome Browser MCU Core Video/Audio mix (js) P2P handling (js) PeerConnecRon P2P handling (js) PeerConnecRon load invoke Member browser Chrome/Firefox/ Safari Member browser Chrome/Firefox/ Safari

Slide 44

Slide 44 text

DEMO (Single room, Docker container) •  Start container –  docker pull mganeko/browser_mcu_pack –  docker run -d -p 3000:3000 --cap-add=SYS_ADMIN mganeko/ browser_mcu_pack •  Connect as Member –  Open hSp://localhost:3000/ –  Click [StartVideo], click [Connect] •  When stop container → video will stop –  docker stop xxxxxxxxxx 44 skip

Slide 45

Slide 45 text

4. Scale out for MulRple rooms 45

Slide 46

Slide 46 text

Support mulRple video chat rooms •  Run browser MUC for each Room –  Mix video/audio for the Room separately 46 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling /MCU Server Room1 Room2

Slide 47

Slide 47 text

Bad point of MCU •  Super High CPU load in Server –  Get higher, when a new member join the room –  Get much higher, when a new room added •  Auto scale out is needed to correspond 47 ObservaRon on vultr Spec -  CPU: 2 vCore -  RAM: 4GB CPU load -  IniRal (no room) … CPU 3% -  1room, 2members … CPU 200%, +300MB -  1room, 4members … CPU 200%, +50MB -  2room, 4+2members … CPU 200%,+300MB

Slide 48

Slide 48 text

Hot to auto scale out? •  Invoke MCU Server for each Room automaRcally –  VM instance of Container 48 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling Server Room1 Room2 MCU Server MCU Server Invoke Invoke → High server cost, difficult to control

Slide 49

Slide 49 text

Make it simple: Room owner iniRaRve scale out •  Room owner provide MCU resource with a browser or a container 49 Browser A Browser B Browser MCU Signaling Server Browser C Browser D Browser MCU Web/Signaling Server Room1 Room2 Browser MCU Docker Start by hand Start by hand

Slide 50

Slide 50 text

Browser MCU docker Browser MCU Series on Web 50 Browser MCU Service（mulRple room) HTML for MCU HTML for members Signaling Server Node.js + ws + express Test server hSps://meeRn.biz Chrome Browser MCU Core Video/Audio mix (js) P2P handling (js) PeerConnecRon P2P handling (js) PeerConnecRon Member browser Chrome/Firefox/ Safari Member browser Chrome/Firefox/ Safari Headless Chrome

Slide 51

Slide 51 text

DEMO3 (mulRple room server, container MCU) •  Start MCU browser, click [Open Room] –  hSps://meeRn.biz/mulR_mcu.html?room=node2017 •  Or, start container for MCU –  docker pull mganeko/browser_mcu_docker –  docker run -d -p 9222:9222 --cap-add=SYS_ADMIN mganeko/ browser_mcu_docker node2017 •  Start browser for members (including iPad Safari 11） –  hSps://meeRn.biz/?room=node2017 51

Slide 52

Slide 52 text

Try Browser MCU Series •  Browser MCU Series –  hSps://github.com/mganeko/browser_mcu –  Core library … hSps://github.com/mganeko/browser_mcu_core •  Sample for single room server… Browser MCU Server –  hSps://github.com/mganeko/browser_mcu_server •  Container for single room server and headless browser –  hSps://github.com/mganeko/browser_mcu_pack –  hSps://hub.docker.com/r/mganeko/browser_mcu_pack/ •  Trial server … hSps://meeRn.biz/ –  MCU … hSps://meeRn.biz/mulR_mcu.html 52 Most useful

Slide 53

Slide 53 text

Conclusion •  WebRTC has many topologies, not only P2P –  Bad point of MCU: Super high CPU load in server –  Good point of MCU: very preferable for mobile clients •  It is possible to build MCU on Web browser •  To make use of browser MCU, separaRon of heavy load components is necessary •  Node.js x Headless browser … extend potenRal of Node.js –  Realize Video / Audio manipulaRon in server side 53

Slide 54

Slide 54 text

Thank you! 54 hSps://github.com/mganeko/browser_mcu Series Please Join: WebRTC Meetup Tokyo WebRTC Begginers Tokyo