Playing with OSS WebRTC SFU meidasoup (update for v1.2)

English translation of the slides for WebRTC Meetup Tokyo 15
Un-offical document for how to use mediasoup v1.2 from your Node.js App

Transcript

1. Playing with OSS SFU meidasoup (Update for v1.2) WebRTC Meetup Tokyo #15 INFOCOM CORPORATION Masashi Ganeko @massie_g June 13th, 2017 1
2. About me • Masashi Ganeko / @massie_g – working for a research team of INFOCOM CORPORATION • Writing a series of WebRTC on HTML5Experts.jp (HTML5 tech site) – https://html5experts.jp/series/webrtc2016/ • English Presentation for WebRTC (2013-2017) – https://speakerdeck.com/mganeko • Japanese Presentation for WebRTC (2013-2017) – http://www.slideshare.net/mganeko 2
3. Topic of today • Playback WebRTC Meetup Tokyo #13 – SFU : Selective forwarding Unit – mediasoup : Open Source WebRTC SFU module for Node.js • New Topics for #15 – Updates of mediasoup v0.3 → v1.2 – New sample codes for v1.2 • https://github.com/mganeko/mediasoup_sample 3
4. Playback WebRTC Meetup Tokyo #13 4
5. P2P vs. SFU: Selective Forwarding Unit 5 Browser A Browser B Browser D Browser C P2P • No media server • Browser side: • CPU load: Very High • Network load: Very High 5 Browser A Browser B Browser D Browser C SFU Dispatch audio/video SFU • User SFU server, CPU load: Low • Browser side: low CPU load • Browser side: network load: middle Able to choose display layout on member side ◎
6. MCU vs. SFU 6 Browser A Browser B Browser D Browser C SFU Dispatch audio/video SFU • User SFU server, CPU load: Low • Browser side: low CPU load • Browser side: network load: middle Able to choose display layout on member side ◎ Browser A Browser B Browser D Browser C MCU Mix video / audio MCU • Use MCU server, CPU load: Super High • Browser side: very low CPU/Network load
7. Encoding and Encrypting 7 Browser Thanks to いらすとや http://www.irasutoya.com/ Browser P2P Encode Encrypt Transmit Decrypt Decode TURN Browser TURN Browser SFU MCU Browser SFU Browser Browser MCU Browser
8. What is mediasoup ? • Open Source WebRTC SFU – Web Site https://mediasoup.org/ – GitHub https://github.com/ibc/mediasoup/ → https://github.com/versatica/mediasoup • Not a Server App, but components for a Server – Instead of creating yet another opinionated server, mediasoup is a Node.js module which can be integrated into a larger application or made standalone with just a few lines of JavaScript. • Provided as Node.js module – You have to prepare Web Server and Signaling method 8
9. What is mediasoup ? (2) • Minimum … Only Media (No DataChannel), No Codecs • Powerful … Main part is build with C++, on libuv – Node.js invokes child worker process – Node.js and worker communicate via interprocess communication • JavaScript API for Node.js – WebRTC 1.0 … RTCPeerConnection, using promise – ORTC … Transport, RtpSender, RtpReceiver 9
10. DEMO of mediasoup sample 10 • Multiple participants video chat sample – Video and Audio – Looks same as P2P, but … • if you stop SFU Sever process, then Remote video/audio will stop • Source codes on GitHub – https://github.com/mganeko/mediasoup_sample • multi participants, 1 way streaming, SSL support
11. How to install mediasoup • Preparation – Node.js v4.8.0 or higher – Linux / Mac OS X (Windows not supported yet) – Python 2.x (NOT Python 3.x !!) • Explicit local python version is needed, when using pyenv / virtualenv • $ pyenv local 2.7.12 – make – gcc with g++ , or clang with C++11 • Installation – $ npm install mediasoup • C++ compile may take a few minutes 11
12. Object structure of mediasoup 12 Server Room Room Room Peer Peer Peer Transport Transport RtpSender RtpSender RtpSender RtpReceiver RtpReceiver RtpReceiver ・1 Server has Many Rooms ・1 Room has Many Peers ・1 Peer has multiple RtpSender ・1 Peer has multiple RtpReceiver ・1 Peer has multiple のTransport Correspond to MediaTrack (video/audio) Correspond to MediaTrack Correspond to UDP/TCP port 1 Transport is used by - multiple RtpSender - multiple RtpReceiver BUNDLE, rtcp-mux multi-stream
13. Inside of mediasoup Room 13 Room Peer RtpReceiver RtpSender Browser Browser Peer RtpReceiver RtpSender Peer Browser RtpReceiver RtpSender RtpSender RtpSender RtpSender Streams are connected automatically in the room Transport Transport Transport
14. worker process 14 Default : Core number Case of 5 Peers in 1 Room All peers in the same Room are assigned to the same worker
15. Rough Sketch to use let server = mediasoup.Server(); let peer1; let receiver1; return server.createRoom(roomOptions) .then( (room) => { peer1 = room.Peer('alice'); return peer.setCapabilities(peerCapabilities); }) .then(() => { return peer.createTransport({ tcp: false }); }) .then((transport) => { rtpReceiver1 = peer1.RtpReceiver('audio', transport); }) 15 • Compose objects step by step • Use Promise • Do not create RtpSender expressly • Will be created automatically, passed by event (I do not understand well ..) Quite complex
16. Another Object structure 16 Server Room Room Room Peer Peer Peer RTCPeerConnection • 1 RTCPeerConnection corresponds to 1 Peer • Same interface as Browser Offer/Answer RTCPeerConnection RTCPeerConnection
17. Updates in mediasoup v1.2 (at meetup #15) 17
18. Changes for v1.x • v1.x • The first Offer is created by mediasoup (SFU side) • v0.3 – The first Offer is created by browser (client side) 18
19. Flow with RTCPeerConenction (v0.3) 19 Browser A RTCPeerConnection mediasoup RTCPeerConnection A Browser B send offer SDP send offer SDP send answer SDP send answer SDP onnegotiationneeded() fired onnegotiationneeded() fired RTCPeerConnection B RTCPeerConnection send answer SDP onaddstream() / ontrack() fired send offer SDP send offer SDP send answer SDP onaddstream() / ontrack() fired createOffer() createAnswer() createOffer() createAnswer() createOffer() createAnswer() createOffer() createAnswer()
20. Flow with RTCPeerConenction (v1.2) 20 Browser A RTCPeerConnection mediasoup RTCPeerConnection A Browser B send “Peer Capabilities” send offer SDP send answer SDP send offer SDP onnegotiationneeded() fired RTCPeerConnection B RTCPeerConnection send answer SDP onaddstream() / ontrack() fired onaddstream() / ontrack() fired createOffer() createOffer() createAnswer() createOffer() createAnswer() createAnswer() send “Peer Capabilities” send answer SDP send offer SDP
21. How to send “Peer Capabilities” • Peer Capabilities – Available CODECs – Available RTCP controls • Getting Capabilities of browser – ORTC, WebRTC 1.0 Spec has getCapabilities(), but not implemented yet • RTCRtpSender.getCapabilities(kind), • RTCRtpReceiver.getCapabilities(kind) – BUT, there are in SDP • Steps in Browser side – Get SDP with pc.createOffer({ offerToReceiveAudio: false, offerToReceiveVideo: false}) – DO NOT CALL pc.setLocalDescription() … P2P initiation will not start – Send SDP to mediasoup as “Capabilities” 21
22. Capabilities information in SDP 22 m=video 50083 UDP/TLS/RTP/SAVPF 96 98 100 102 127 97 99 101 125 a=mid:video a=rtcp-mux a=rtcp-rsize a=rtpmap:96 VP8/90000 a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtpmap:98 VP9/90000 a=rtcp-fb:98 ccm fir a=rtcp-fb:98 nack a=rtcp-fb:98 nack pli a=rtcp-fb:98 goog-remb a=rtcp-fb:98 transport-cc a=rtpmap:100 H264/90000 a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=rtcp-fb:100 goog-remb a=rtcp-fb:100 transport-cc a=fmtp:100 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f a=rtpmap:102 red/90000 a=rtpmap:127 ulpfec/90000 a=rtpmap:97 rtx/90000 a=fmtp:97 apt=96 a=rtpmap:99 rtx/90000 a=fmtp:99 apt=98 a=rtpmap:101 rtx/90000 a=fmtp:101 apt=100 a=rtpmap:125 rtx/90000 a=fmtp:125 apt=102
23. Flow of sending Peer Capabilities (v1.2) 23 Browser A RTCPeerConnection mediasoup RTCPeerConnection A send “Peer Capabilities” SDP send offer SDP createOffer() createAnswer() setLocalDescription() setCapabilities() createOffer() setRemoteDescription() setLocalDescription() setLocalDescription() send answer SDP setRemoteDescription()
24. Limitation of RTCPeerConnection (v0.3) • Need initial Offer from Remote (Browser) – Currently, the mediasoup implementation of RTCPeerConnection requires that the initial offer comes from the remote endpoint, • After onnegotiationneeded() event, create Offer in mediasoup • Not allowed to re-creating Offer in Remote (Browser) – Add/Remove of Remote stream/track is not supported • It seems like Plan B support is tentative. (maybe my sample was wrong ..) 24 Limitation of RTCPeerConnection, NOT limitation of mediasoup itself. It must be possible if you compose Transport, RtpReceiver, SDP manually.
25. Changes of RTCPeerConnection (v1.2) • Set capabilities of browser at first • Offer is always from mediasoup side / mediasoup do not accept Offer • When Add/Remove of Remote stream/track after connecting – Request re-negotiation to mediasoup somehow – Re-create Offer in mediasoup, and send to browser – Create Answer in browser , and send back to meidasoup • Plan B support for Chrome – Specify “usePlanB” option while creating PeerConnection object – Support Mix of Chrome / Firefox in the same room 25
26. Tricky point of Vanilla ICE • Using Vanilla ICE in my sample code – DO NOT send SDP right after createAnswer() from browser – Send Answer SDP after collecting all ICE candidates • When re-create Answer in browser – Send Answer SDP right away – onicecandidate() WILL NOT be fired on Chrome • all ICE candidates are collected, already • No events will be fired, after re- createAnswer() – onicecandidate() with empty candiate will be fired on Firefox 26
27. Tricky point of Vanilla ICE 27 Browser A RTCPeerConnection mediasoup RTCPeerConnection A Browser B send “Peer Capabilities” send offer SDP send answer SDP send offer SDP onnegotiationneeded() fired RTCPeerConnection B RTCPeerConnection send answer SDP onaddstream() / ontrack() fired onaddstream() / ontrack() fired createOffer() createOffer() createAnswer() createOffer() createAnswer() createAnswer() send “Peer Capabilities” send answer SDP send offer SDP Wait for collecting all ICE candidates DO NOT wait for candidates
28. 1 way streaming • 1 to N streaming (1 Publisher - N Subscriber) – Use option for creating Offer for Subscriber at mediasoup side • pc.createOffer({ offerToReceiveAudio:0, offerToReceiveVideo:0}) • Strick control for “1” to N – MUST exclude 2nd Publisher with application code – If you allow 2nd Publisher, the become “2 to N” • Bi-directional between publishers • ontrack()/onaddtream() will be fired in publishers 28 * It is possible to stop stream between publishers, if use RtpSender directly
29. mediasoup 1 way streaming (1 to N) 29 Browser Publisher 1 room Browser Subscriber A Browser Subscriber B Browser Subscriber C PeerConnection PeerConnection PeerConnection PeerConnection
30. mediasoup When 2nd Publisher come (expect) 30 Browser Publisher 1 room Browser Subscriber A Browser Subscriber B Browser Subscriber C PeerConnection PeerConnection PeerConnection PeerConnection Browser Publisher 2 PeerConnection
31. mediasoup When 2nd Publisher come (real) 31 Browser Publisher 1 room Browser Subscriber A Browser Subscriber B Browser Subscriber C PeerConnection PeerConnection PeerConnection PeerConnection Browser Publisher 2 PeerConnection * It is possible to stop stream between publishers, if use RtpSender directly
32. Other limitation of mediasoup • mediasoup should be outside of NAT/Firewall. Need global IP address. – mediasoup does not look for STUN/TURN server • Browser inside of NAT/Firewall is OK – When Browoser looks for STUN/TURN server, it is possible to connect. 32 mediasoup Browser STUN Browser TURN NAT Firewall
33. Conclusion • Trends of WebRTC is changing as P2P → MCU → SFU • mediasoup v1.2 is stable enough to use – First choice if you want to built in SFU in your application. • If you want high functional Server, there are commercial products (such as Siguredo Sora, tokbox, twillio) • This slides on speakerdeck – https://speakerdeck.com/mganeko/playing-with-oss-webrtc-sfu-meidasoup-update-for-v1-dot-2 • Sample source on GitHub – https://github.com/mganeko/mediasoup_sample 33
34. Thank you! 34