Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Node.js x Chrome headless for WebRTC MCU

mganeko
November 26, 2017

Node.js x Chrome headless for WebRTC MCU

Presentation for Nodefest Tokyo 2017
Building WebRTC MCU with Node.js and Headless browser.
Mix video and audio with Canvas and WebAudio, without window/desktop environment

mganeko

November 26, 2017
Tweet

More Decks by mganeko

Other Decks in Technology

Transcript

  1. Node.js x Chrome headless
    = WebRTC MCU
    Nodefest 2017
    2017.11.25
    INFOCOM CORPORATION
    Masashi Ganeko @massie_g
    1

    View full-size slide

  2. About Me
    •  Masashi Ganeko / @massie_g
    •  Working for research team of INFOCOM CORPORATION
    •  Staff of WebRTC Meetup Tokyo
    •  Staff of WebRTC Beginners Tokyo
    •  WebRTC related presentaRons
    –  hSps://speakerdeck.com/mganeko
    2

    View full-size slide

  3. Sessions in past Nodefest
    •  Nodefest Tokyo 2016 LT
    –  Node.js with WebRTC DataChannel (in Japanese)
    –  hSps://www.slideshare.net/mganeko/nodejs-with-webrtc-datachannel
    •  Nodefest Tokyo 2014
    –  Using Kurento Media Server with Node.js (in Japanese)
    –  hSp://www.slideshare.net/mganeko/nodekurento
    •  Nodefest Tokyo 2013
    –  Let's Begin WebRTC
    –  hSps://speakerdeck.com/mganeko/lets-begin-webrtc
    3

    View full-size slide

  4. Topic of today
    •  WebRTC topology is not only Peer-to-Peer
    – MCU: mixing video/audio in server
    •  It is possible to build MCU with Web Browser
    •  Manipulate video/audio as you like
    – with Node.js x Headless Chrome
    4

    View full-size slide

  5. 1. WebRTC and MCU
    Why using MCU ?
    5

    View full-size slide

  6. What is WebRTC
    •  Web Real-Rme CommunicaRon for
    –  Video
    –  Audio
    –  Data
    •  Open standard … anyone can use
    –  W3C WebRTC Working Group ... API
    –  IETF RTCWEB Working Group … Protocol
    –  Core library is open source: libwebrtc
    •  Designed for Web browsers, and other web connected devices
    •  Easy to combine with other Web technologies
    10101110100…

    View full-size slide

  7. WebRTC enabled OS
    7
    Supported in
    Mac OS X / iOS
    Safari 11
    (H.264 only)
    NEW!
    NEW!

    View full-size slide

  8. WebRTC Topology
    •  Peer to Peer … Basic
    –  Direct connecRon between browsers
    •  SFU (SelecRve Forwarding Unit)
    –  Server dispatchs video/audio streams
    •  MCU (MulRpoint Control Unit)
    –  Mix audio/audio, and re-encode in Server side
    8

    View full-size slide

  9. P2P
    •  Basic topology
    –  Full mesh connecRon between
    browsers
    •  Good
    –  NO media server
    •  Bad
    –  CPU load: Super High
    –  Network load: Super High
    Browser
    A
    Browser
    B
    Browser
    D
    Browser
    C

    View full-size slide

  10. Why super high CPU load in P2P
    Browser A
    Original video
    encoded
    send
    send
    send
    Not This
    (Same encoded data
    for all receivers)

    View full-size slide

  11. Browser A
    Why super high CPU load in P2P
    send send
    send
    But this: encode for each receiver
    Browser A
    Original video
    encoded
    send
    send
    send
    Original video
    encoded
    encoded
    encoded

    View full-size slide

  12. SFU: SelecRve Forwarding Unit
    •  Dispatch video/audio in server
    –  NO encode / transcode in server
    •  Good
    –  CPU load of browser/client: low
    –  Upstream network load: low
    –  SFU server CPU load: low
    •  Bad
    •  Upstream network load: High

    View full-size slide

  13. MCU (MulRpoint Control Unit)
    •  Mix video/audio in server side
    •  Good
    –  CPU load of browser/client: lowest
    –  Upstream/downstream network load: lowest
    –  → Good for mobile clients
    •  Bad
    •  Server CPU load: Super High
    •  Video layout is fixed
    Browser
    A
    Browser
    B
    Browser
    D
    Browser
    C
    MCU
    Mix
    video /
    audio

    View full-size slide

  14. MCU for mobile
    •  Mix video/audio in server
    •  Re-encode/transcode in server
    –  Compression for each connecRon
    –  Different CODEC for each client
    PC
    Chrome
    PC
    Firefox
    Android
    Chrome
    iOS
    Safari
    MCU
    VP9
    VP9
    VP9
    VP9
    VP8
    VP8
    H.264
    H.264
    H.264 only

    View full-size slide

  15. MCU OSS and products
    •  OSS
    –  Kureto … hSp://www.kurento.org/
    •  looks not so acRve, amer acquisiRon by Twillio
    •  My slides about Node.js x Kurento Media Server (in Japanese)
    –  hSps://www.slideshare.net/mganeko/nodekurento
    –  Licode … hSp://lynckia.com/licode/
    •  Products
    –  Dialogic PowerMedia XMS
    15

    View full-size slide

  16. 2. MCU Core with Browser
    MCU Core processing
    for Video/Audio
    16

    View full-size slide

  17. What MCU does
    •  Receive Video/Audio from clients
    •  Mix Videos
    •  Mix Audios
    •  Send back Video/Audio to clients
    17

    View full-size slide

  18. What MCU does
    •  Receive Video/Audio from clients
    •  Mix Videos
    •  Mix Audios
    •  Send back Video/Audio to clients
    18
    WebRTC
    PeerConnecRon
    WebRTC
    PeerConnecRon
    Canvas
    WebAudio
    I can do with Browser !

    View full-size slide

  19. Video mixing
    19

    View full-size slide

  20. Inside of Browser MCU:Video
    20
    RTCPeerConnecRon A
    MediaStream

    element
    RTCPeerConnecRon D
    MediaStream
    element
    drawImage()
    Keep drawing with
    requestAnimaRonFrame()

    element
    drawImage()

    View full-size slide

  21. Inside of Browser MCU:Video
    21
    RTCPeerConnecRon A
    RTCPeerConnecRon D
    element
    MediaStream
    Get video with captureStream(fps)

    View full-size slide

  22. Inside of Browser MCU:Video
    22
    RTCPeerConnecRon A
    MediaStream

    element
    RTCPeerConnecRon D
    MediaStream
    element
    drawImage()
    Keep drawing with
    requestAnimaRonFrame()

    element
    drawImage()
    MediaStream
    Get video with captureStream(fps)

    View full-size slide

  23. Steps to mix video in Browser MCU
    •  Get Remote MediaStream through RTCPeerConnecRon
    –  play in element
    •  Get context of Canvas by getContext('2d')
    •  Keep drawing with window.requestAnimaRonFrame()
    –  Copy image of each video element to canvas, with drawImage()
    •  Capture MadiaStream from Canvas, with captureStream(fps)
    –  Pass it to RTCPeerConnecRon.addStream(), and send it to remote peer
    23

    View full-size slide

  24. DEMO1: Mixing video with Browser MCU
    •  Start node server
    •  Open Browser for MCU
    •  Open client browser and connect
    •  Demo case of Browser MCU hidden
    24

    View full-size slide

  25. Tricky points of MCU Server:Video
    •  Using window.requestAnimaRonFrame()
    –  Will not be called when the window/tab is completely hidden
    –  AnimaRon will stop if the window is minimized or other hidden
    by other tab
    •  Even using window.setInterval()
    –  The interval will be much longer than specified, the window/tab
    is completely hidden
    25

    View full-size slide

  26. Audio mixing
    26

    View full-size slide

  27. Tricky points of MCU Server:Audio
    •  One mixed video is enough for all members
    •  Audio should be prepared differently for each member
    –  It is hand to speak, if your voice will come back with short
    delay
    –  Maybe familiar case with WebRTC applicaRon developers
    27

    View full-size slide

  28. Inside MCU Server:Audio
    •  Prepare "minus one" Audio
    –  Mix audio of all members, except the user
    •  Need many combinaRon of audio
    –  4 combinaRon for 4 members
    –  N combinaRon for N members
    28
    Browser
    A
    Browser
    B
    Browser
    D
    Browser
    C
    MCU
    Mixing Audio
    is hassle
    Audio is much more
    difficult than Video

    View full-size slide

  29. Inside MCU Server:Audio
    29
    RTCPeerConnecRon A
    MediaStream
    MediaStreamAudioSourceNode
    Create with AudioContext .
    createMediaStreamSource()

    View full-size slide

  30. Inside MCU Server:Audio
    30
    RTCPeerConnecRon D
    MediaStream
    MediaStreamAudioSourceNode MediaStreamAudioSourceNode
    MediaStreamAudioSourceNode
    MediaStreamAudioDesRnaRonNode
    mix
    RTCPeerConnecRon A
    Create with AudioContext .
    createMediaStreamDesRnaRon()
    RTCPeerConnecRon B RTCPeerConnecRon C

    View full-size slide

  31. Inside MCU Server:Audio
    31
    RTCPeerConnecRon A
    MediaStream
    RTCPeerConnecRon D
    MediaStream
    MediaStreamAudioSourceNode MediaStreamAudioSourceNode
    MediaStreamAudioSourceNode
    MediaStreamAudioDesRnaRonNode
    Mix
    Repeat for
    N paSern
    RTCPeerConnecRon A
    RTCPeerConnecRon B RTCPeerConnecRon C

    View full-size slide

  32. Steps to mix audio in Browser MCU
    •  Get remote MediaStream through RTCPeerConnecRon
    •  Create MediaStreamAudioSourceNode of Web Audio
    –  Convert MediaStream to Web Audio node
    •  Create MediaStreamAudioDesRnaRonNode of Web Audio
    –  Mix SourceNode of other members: sourceNode.connect(desRnaRonNode);
    •  Get MediaStream
    –  Pass to RTCPeerConnecRon. addStream(), and send to remote peer
    •  * repeat for all members
    32

    View full-size slide

  33. Inside of MCU Server:Video & Audio
    33
    RTCPeerConnecRon A
    RTCPeerConnecRon D
    videoTrack
    MediaStream
    (Audio B+C+D)
    RTCPeerConnecRon B
    RTCPeerConnecRon C
    MediaStream
    (Audio A+C+D)
    MediaStream
    (Audio A+B+D)
    MediaStream
    (Audio A+B+C)
    Combine
    videoTrack and audioTrack
    to one MediaStream
    addTrack()
    addTrack()
    addTrack()
    addTrack()
    MediaStream
    (Video A+B+C+D)

    View full-size slide

  34. 3. MCU Server with Node.js +
    headless Browser
    34

    View full-size slide

  35. Whole structure
    •  Browser MCU is communicaRng over P2P
    WebRTC connecRons
    •  Signaling process is necessary before P2P
    –  Exchange IP address, ports, and CODEC
    condiRons
    •  No standard for signaling
    –  Different between Apps / Services
    → Separate mixing core from peer handling
    35
    Browser
    A
    Browser
    B
    Browser MCU
    Signaling
    Server
    message
    message
    message
    media media

    View full-size slide

  36. Browser MCU Series
    •  hSps://github.com/mganeko/browser_mcu
    •  MCU Core … Mixing of Video/Audio
    –  No peer handling
    •  P2P handling
    –  CreaRng PeerConnecRon, exchanging SDP/ICE candidate
    •  Sending message
    –  over WebSocket / Socket.io
    •  Signaling Server
    –  Node.js
    –  hSp/hSps server + WebSocket/socket.io server
    36
    Browser MCU Core
    Video/Audio mix (js)
    Browser MCU
    Server
    Sample
    code
    browser_mcu_core
    browser_mcu_server

    View full-size slide

  37. Browser MCU Series
    37
    Browser MCU Server(single room)
    HTML for MCU
    HTML for members
    Signaling Server
    Node.js + ws + express
    Headless Chrome
    Browser MCU Core
    Video/Audio mix (js)
    P2P handling (js)
    PeerConnecRon
    P2P handling (js)
    PeerConnecRon
    load invoke
    Member browser
    Chrome/Firefox/
    Safari
    Member browser
    Chrome/Firefox/
    Safari

    View full-size slide

  38. Headless Chrome
    •  Chrome 59 ~
    –  chrome --headless --remote-debugging-port=9222 URL
    –  * "--remote-debugging-port" is necessary (Nov 15 2017)
    –  * no more "--disable-gpu" from Chrome 61
    •  Work without Window/Window System
    –  render video
    –  draw Canvas
    –  capture MediaStream from Canvas
    •  AnimaRon event fires with requestAnimaRonFrame()
    38

    View full-size slide

  39. Control of Headless Chrome
    •  Node.js modules … using Chrome DevTools Protocol
    –  chrome-remote-interface
    •  hSps://github.com/cyrus-and/chrome-remote-interface
    –  puppeteer
    •  hSps://github.com/GoogleChrome/puppeteer
    •  Simple implementaRon for Browser MCU series
    –  MCU browser is connected to Node.js server over WebSocket/Socket.io
    •  It already have connecRon for signaling
    •  The connecRon is used for command from Node.js server
    –  No page transiRon (Kind of Single Page App), so keep connected
    39

    View full-size slide

  40. PrimiRve control of Headless Chrome
    •  Invoke from Node.js
    // execute as child process
    let proc = childProcess.execFile(path, args);
    •  Stop
    // Not possible to terminate from inside of Chrome
    proc.kill('SIGKILL');
    40

    View full-size slide

  41. DEMO2 (mulR room, auto start chrome)
    •  Start node server
    •  Connect member page → start Headless Chrome
    –  connect
    –  ps –ef | grep Chrome
    •  (when all member disconnected → Chrome will quit)
    •  (connect member again → start headless Chrome)
    •  Kill Chrome → video will stop
    41

    View full-size slide

  42. Run in Docker container
    •  Headless chrome will run in Docker container
    •  Made a container image, include Node.js and Headless Chrome
    –  Single room
    –  Browser MCU Pack (Dockerfile)
    •  hSps://github.com/mganeko/browser_mcu_pack
    –  base: node:latest
    •  Clone hSps://github.com/mganeko/browser_mcu_server 
    •  Install Chrome stable
    •  Container image on docker hub: mganeko/browser_mcu_pack
    –  hSps://hub.docker.com/r/mganeko/browser_mcu_pack/
    42

    View full-size slide

  43. Browser MCU Series
    43
    Browser MCU Server (single room)
    HTML for MCU
    HTML for members
    Signaling Server
    Node.js + ws + express
    Browser MCU Pack (docker iamge)
    Headless Chrome
    Browser MCU Core
    Video/Audio mix (js)
    P2P handling (js)
    PeerConnecRon
    P2P handling (js)
    PeerConnecRon
    load invoke
    Member browser
    Chrome/Firefox/
    Safari
    Member browser
    Chrome/Firefox/
    Safari

    View full-size slide

  44. DEMO (Single room, Docker container)
    •  Start container
    –  docker pull mganeko/browser_mcu_pack
    –  docker run -d -p 3000:3000 --cap-add=SYS_ADMIN mganeko/
    browser_mcu_pack
    •  Connect as Member
    –  Open hSp://localhost:3000/
    –  Click [StartVideo], click [Connect]
    •  When stop container → video will stop
    –  docker stop xxxxxxxxxx
    44
    skip

    View full-size slide

  45. 4. Scale out for MulRple rooms
    45

    View full-size slide

  46. Support mulRple video chat rooms
    •  Run browser MUC for each Room
    –  Mix video/audio for the Room separately
    46
    Browser
    A
    Browser
    B
    Browser MCU Signaling
    Server
    Browser
    C
    Browser
    D
    Browser MCU
    Web/Signaling
    /MCU Server
    Room1 Room2

    View full-size slide

  47. Bad point of MCU
    •  Super High CPU load in Server
    –  Get higher, when a new member join
    the room
    –  Get much higher, when a new room
    added
    •  Auto scale out is needed to
    correspond
    47
    ObservaRon on vultr
    Spec
    -  CPU: 2 vCore
    -  RAM: 4GB
    CPU load
    -  IniRal (no room) … CPU 3%
    -  1room, 2members … CPU 200%, +300MB
    -  1room, 4members … CPU 200%, +50MB
    -  2room, 4+2members … CPU 200%,+300MB

    View full-size slide

  48. Hot to auto scale out?
    •  Invoke MCU Server for each Room automaRcally
    –  VM instance of Container
    48
    Browser
    A
    Browser
    B
    Browser MCU Signaling
    Server
    Browser
    C
    Browser
    D
    Browser MCU
    Web/Signaling
    Server
    Room1 Room2
    MCU Server MCU Server
    Invoke
    Invoke
    → High server cost, difficult to control

    View full-size slide

  49. Make it simple: Room owner iniRaRve scale out
    •  Room owner provide MCU resource with a browser or a container
    49
    Browser
    A
    Browser
    B
    Browser MCU Signaling
    Server
    Browser
    C
    Browser
    D
    Browser MCU
    Web/Signaling
    Server
    Room1 Room2
    Browser MCU
    Docker
    Start by hand
    Start by hand

    View full-size slide

  50. Browser MCU docker
    Browser MCU Series on Web
    50
    Browser MCU Service(mulRple room)
    HTML for MCU
    HTML for members
    Signaling Server
    Node.js + ws + express
    Test server hSps://meeRn.biz
    Chrome
    Browser MCU Core
    Video/Audio mix (js)
    P2P handling (js)
    PeerConnecRon
    P2P handling (js)
    PeerConnecRon
    Member browser
    Chrome/Firefox/
    Safari
    Member browser
    Chrome/Firefox/
    Safari
    Headless Chrome

    View full-size slide

  51. DEMO3 (mulRple room server, container MCU)
    •  Start MCU browser, click [Open Room]
    –  hSps://meeRn.biz/mulR_mcu.html?room=node2017
    •  Or, start container for MCU
    –  docker pull mganeko/browser_mcu_docker
    –  docker run -d -p 9222:9222 --cap-add=SYS_ADMIN mganeko/
    browser_mcu_docker node2017
    •  Start browser for members (including iPad Safari 11)
    –  hSps://meeRn.biz/?room=node2017
    51

    View full-size slide

  52. Try Browser MCU Series
    •  Browser MCU Series
    –  hSps://github.com/mganeko/browser_mcu
    –  Core library … hSps://github.com/mganeko/browser_mcu_core
    •  Sample for single room server… Browser MCU Server
    –  hSps://github.com/mganeko/browser_mcu_server
    •  Container for single room server and headless browser
    –  hSps://github.com/mganeko/browser_mcu_pack
    –  hSps://hub.docker.com/r/mganeko/browser_mcu_pack/
    •  Trial server … hSps://meeRn.biz/
    –  MCU … hSps://meeRn.biz/mulR_mcu.html
    52
    Most useful

    View full-size slide

  53. Conclusion
    •  WebRTC has many topologies, not only P2P
    –  Bad point of MCU: Super high CPU load in server
    –  Good point of MCU: very preferable for mobile clients
    •  It is possible to build MCU on Web browser
    •  To make use of browser MCU, separaRon of heavy load
    components is necessary
    •  Node.js x Headless browser … extend potenRal of Node.js
    –  Realize Video / Audio manipulaRon in server side
    53

    View full-size slide

  54. Thank you!
    54
    hSps://github.com/mganeko/browser_mcu Series
    Please Join:
    WebRTC Meetup Tokyo
    WebRTC Begginers Tokyo

    View full-size slide