About me • Masashi Ganeko / @massie_g – working for a research team of INFOCOM CORPORATION • Writing a series of WebRTC on HTML5Experts.jp (HTML5 tech site) – https://html5experts.jp/series/webrtc2016/ • English Presentation for WebRTC (2013-2017) – https://speakerdeck.com/mganeko • Japanese Presentation for WebRTC (2013-2017) – http://www.slideshare.net/mganeko 2
Topic of today • Playback WebRTC Meetup Tokyo #13 – SFU : Selective forwarding Unit – mediasoup : Open Source WebRTC SFU module for Node.js • New Topics for #15 – Updates of mediasoup v0.3 → v1.2 – New sample codes for v1.2 • https://github.com/mganeko/mediasoup_sample 3
P2P vs. SFU: Selective Forwarding Unit 5 Browser A Browser B Browser D Browser C P2P • No media server • Browser side: • CPU load: Very High • Network load: Very High 5 Browser A Browser B Browser D Browser C SFU Dispatch audio/video SFU • User SFU server, CPU load: Low • Browser side: low CPU load • Browser side: network load: middle Able to choose display layout on member side ◎
MCU vs. SFU 6 Browser A Browser B Browser D Browser C SFU Dispatch audio/video SFU • User SFU server, CPU load: Low • Browser side: low CPU load • Browser side: network load: middle Able to choose display layout on member side ◎ Browser A Browser B Browser D Browser C MCU Mix video / audio MCU • Use MCU server, CPU load: Super High • Browser side: very low CPU/Network load
What is mediasoup ? • Open Source WebRTC SFU – Web Site https://mediasoup.org/ – GitHub https://github.com/ibc/mediasoup/ → https://github.com/versatica/mediasoup • Not a Server App, but components for a Server – Instead of creating yet another opinionated server, mediasoup is a Node.js module which can be integrated into a larger application or made standalone with just a few lines of JavaScript. • Provided as Node.js module – You have to prepare Web Server and Signaling method 8
What is mediasoup ? (2) • Minimum … Only Media (No DataChannel), No Codecs • Powerful … Main part is build with C++, on libuv – Node.js invokes child worker process – Node.js and worker communicate via interprocess communication • JavaScript API for Node.js – WebRTC 1.0 … RTCPeerConnection, using promise – ORTC … Transport, RtpSender, RtpReceiver 9
DEMO of mediasoup sample 10 • Multiple participants video chat sample – Video and Audio – Looks same as P2P, but … • if you stop SFU Sever process, then Remote video/audio will stop • Source codes on GitHub – https://github.com/mganeko/mediasoup_sample • multi participants, 1 way streaming, SSL support
How to install mediasoup • Preparation – Node.js v4.8.0 or higher – Linux / Mac OS X (Windows not supported yet) – Python 2.x (NOT Python 3.x !!) • Explicit local python version is needed, when using pyenv / virtualenv • $ pyenv local 2.7.12 – make – gcc with g++ , or clang with C++11 • Installation – $ npm install mediasoup • C++ compile may take a few minutes 11
Object structure of mediasoup 12 Server Room Room Room Peer Peer Peer Transport Transport RtpSender RtpSender RtpSender RtpReceiver RtpReceiver RtpReceiver ・1 Server has Many Rooms ・1 Room has Many Peers ・1 Peer has multiple RtpSender ・1 Peer has multiple RtpReceiver ・1 Peer has multiple のTransport Correspond to MediaTrack (video/audio) Correspond to MediaTrack Correspond to UDP/TCP port 1 Transport is used by - multiple RtpSender - multiple RtpReceiver BUNDLE, rtcp-mux multi-stream
Inside of mediasoup Room 13 Room Peer RtpReceiver RtpSender Browser Browser Peer RtpReceiver RtpSender Peer Browser RtpReceiver RtpSender RtpSender RtpSender RtpSender Streams are connected automatically in the room Transport Transport Transport
Rough Sketch to use let server = mediasoup.Server(); let peer1; let receiver1; return server.createRoom(roomOptions) .then( (room) => { peer1 = room.Peer('alice'); return peer.setCapabilities(peerCapabilities); }) .then(() => { return peer.createTransport({ tcp: false }); }) .then((transport) => { rtpReceiver1 = peer1.RtpReceiver('audio', transport); }) 15 • Compose objects step by step • Use Promise • Do not create RtpSender expressly • Will be created automatically, passed by event (I do not understand well ..) Quite complex
How to send “Peer Capabilities” • Peer Capabilities – Available CODECs – Available RTCP controls • Getting Capabilities of browser – ORTC, WebRTC 1.0 Spec has getCapabilities(), but not implemented yet • RTCRtpSender.getCapabilities(kind), • RTCRtpReceiver.getCapabilities(kind) – BUT, there are in SDP • Steps in Browser side – Get SDP with pc.createOffer({ offerToReceiveAudio: false, offerToReceiveVideo: false}) – DO NOT CALL pc.setLocalDescription() … P2P initiation will not start – Send SDP to mediasoup as “Capabilities” 21
Limitation of RTCPeerConnection (v0.3) • Need initial Offer from Remote (Browser) – Currently, the mediasoup implementation of RTCPeerConnection requires that the initial offer comes from the remote endpoint, • After onnegotiationneeded() event, create Offer in mediasoup • Not allowed to re-creating Offer in Remote (Browser) – Add/Remove of Remote stream/track is not supported • It seems like Plan B support is tentative. (maybe my sample was wrong ..) 24 Limitation of RTCPeerConnection, NOT limitation of mediasoup itself. It must be possible if you compose Transport, RtpReceiver, SDP manually.
Changes of RTCPeerConnection (v1.2) • Set capabilities of browser at first • Offer is always from mediasoup side / mediasoup do not accept Offer • When Add/Remove of Remote stream/track after connecting – Request re-negotiation to mediasoup somehow – Re-create Offer in mediasoup, and send to browser – Create Answer in browser , and send back to meidasoup • Plan B support for Chrome – Specify “usePlanB” option while creating PeerConnection object – Support Mix of Chrome / Firefox in the same room 25
Tricky point of Vanilla ICE • Using Vanilla ICE in my sample code – DO NOT send SDP right after createAnswer() from browser – Send Answer SDP after collecting all ICE candidates • When re-create Answer in browser – Send Answer SDP right away – onicecandidate() WILL NOT be fired on Chrome • all ICE candidates are collected, already • No events will be fired, after re- createAnswer() – onicecandidate() with empty candiate will be fired on Firefox 26
1 way streaming • 1 to N streaming (1 Publisher - N Subscriber) – Use option for creating Offer for Subscriber at mediasoup side • pc.createOffer({ offerToReceiveAudio:0, offerToReceiveVideo:0}) • Strick control for “1” to N – MUST exclude 2nd Publisher with application code – If you allow 2nd Publisher, the become “2 to N” • Bi-directional between publishers • ontrack()/onaddtream() will be fired in publishers 28 * It is possible to stop stream between publishers, if use RtpSender directly
mediasoup 1 way streaming (1 to N) 29 Browser Publisher 1 room Browser Subscriber A Browser Subscriber B Browser Subscriber C PeerConnection PeerConnection PeerConnection PeerConnection
mediasoup When 2nd Publisher come (real) 31 Browser Publisher 1 room Browser Subscriber A Browser Subscriber B Browser Subscriber C PeerConnection PeerConnection PeerConnection PeerConnection Browser Publisher 2 PeerConnection * It is possible to stop stream between publishers, if use RtpSender directly
Other limitation of mediasoup • mediasoup should be outside of NAT/Firewall. Need global IP address. – mediasoup does not look for STUN/TURN server • Browser inside of NAT/Firewall is OK – When Browoser looks for STUN/TURN server, it is possible to connect. 32 mediasoup Browser STUN Browser TURN NAT Firewall
Conclusion • Trends of WebRTC is changing as P2P → MCU → SFU • mediasoup v1.2 is stable enough to use – First choice if you want to built in SFU in your application. • If you want high functional Server, there are commercial products (such as Siguredo Sora, tokbox, twillio) • This slides on speakerdeck – https://speakerdeck.com/mganeko/playing-with-oss-webrtc-sfu-meidasoup-update-for-v1-dot-2 • Sample source on GitHub – https://github.com/mganeko/mediasoup_sample 33
mganeko
arthur1
tatematsu_san
sat
tarao
motok1
megabitsenmzq
udzura
nttcom
daina0321
sakaik
visional_engineering_and_design
uzulla
addyosmani
holman
lauravandoore
keithpitt
jennybc
maltzj
davidbonilla
dotmariusz
hannesfritz
wjessup
andyhume
marcelosomers