Tweet
Tweet

Slide 1

Slide 1 text

WebRTC Meetup Tokyo #11 Build WebRTC MCU on Browser INFOCOM CORPORATION Masashi Ganeko @massie_g Aug. 8th, 2016 1

Slide 2

Slide 2 text

About me • Masashi Ganeko / @massie_g – working for a research team of INFOCOM CORPORATION • Writing a series of WebRTC on HTML5Experts.jp (HTML5 tech site) – https://html5experts.jp/series/webrtc2016/ • English Presentation for WebRTC (2013, 2016) – https://speakerdeck.com/mganeko • Japanese Presentation for WebRTC (2013-2016) – http://www.slideshare.net/mganeko 2

Slide 3

Slide 3 text

Topic of today • MCU : Multipoint Control Unit – Used for Video Conference • Web Browser – WebRTC, Canvas, Web Audio 3

Slide 4

Slide 4 text

P2P vs. MCU 4 Browser A Browser B Browser D Browser C P2P • No media server • Browser side: • CPU load: Very High • Network load: Very High Browser A Browser B Browser D Browser C MCU Mix video / audio MCU • Use MCU server, CPU load: Super High • Browser side: very low CPU/Network load Good for Mobile

Slide 5

Slide 5 text

Past trial：WebRTC Meetup Tokyo #4 • Absurd trial with WebRTC ＋α (Oct. 2014) – (3) mix video absurd way – http://www.slideshare.net/mganeko/webrtc-meetup4-lt/24 (Page 21 - ) 5 HDMI Capture unit Mix video with HDMI capture

Slide 6

Slide 6 text

It is possible with only browser at 2016 • Canvas captureStream() – Firefox 43- , Chrome 51- • Handle remote audio with Web Audio API – Firefox (around)40-, Chrome 49- 6

Slide 7

Slide 7 text

Browser MCU DEMO • YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=54m4s – 54:04 to 55:22 7

Slide 8

Slide 8 text

Browser MCU DEMO 8 Grid mix

Slide 9

Slide 9 text

Inside MCU Server：Video 9 RTCPeerConnection A MediaStream element RTCPeerConnection D MediaStream element drawImage() Keep drawing with requestAnimationFrame() element drawImage()

Slide 10

Slide 10 text

Inside MCU Server：Video 10 RTCPeerConnection A RTCPeerConnection D element MediaStream captureStream(fps)

Slide 11

Slide 11 text

Inside MCU Server：Video 11 RTCPeerConnection A MediaStream element RTCPeerConnection D MediaStream element MediaStream drawImage() Keep drawing with requestAnimationFrame() element captureStream(fps) drawImage()

Slide 12

Slide 12 text

Process of MCU Server：Video • Get Remote MediaStream through RTCPeerConnection – play in element • Get context of Canvas by getContext('2d') • Keep drawing with window.requestAnimationFrame() – Copy image of each video element to canvas, with drawImage() • Capture MadiaStream from Canvas, with captureStream(fps) – Pass it to RTCPeerConnection.addStream(), and send it to remote peer 12

Slide 13

Slide 13 text

Tricky points of MCU Server：Video • Using window.requestAnimationFrame() – Will not be called when the window/tab is completely hidden – Animation will stop if the window is minimized or other hidden by other tab • https://youtu.be/0RYWw1nPxio?t=56m44s to 57:17 • Even using window.setInterval() – The interval will be much longer than specified, the window/tab is completely hidden 13 DEMO

Slide 14

Slide 14 text

Tricky points of MCU Server：Audio • One mixed video is enough for all members • Audio should be prepared differently for each member – It is hand to speak, if your voice will come back with short delay – Maybe familiar case with WebRTC application developers – If it is not familiar to you, please try yourself • https://mganeko.github.io/webrtcexpjp/basic2016/camera_mic.html • Please use headphone, and laud volume • Click [Start] button • It is so hard to speak 14 DEMO

Slide 15

Slide 15 text

Inside MCU Server：Audio • Prepare "minus one" Audio – Mix audio of all members, except the user • Need many combination of audio – 4 combination for 4 members – N combination for N members 15 Browser A Browser B Browser D Browser C MCU Mixing Audio is hassle Audio is much more difficult than Video

Slide 16

Slide 16 text

Inside MCU Server：Audio 16 RTCPeerConnection A MediaStream create with AudioContext . createMediaStreamSource() MediaStreamAudioSourceNode

Slide 17

Slide 17 text

Inside MCU Server：Audio 17 RTCPeerConnection D MediaStream MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioDestinationNode create with AudioContext . createMediaStreamDestination() mix

Slide 18

Slide 18 text

Inside MCU Server：Audio 18 RTCPeerConnection A MediaStream RTCPeerConnection D MediaStream create with AudioContext . createMediaStreamSource() MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioSourceNode MediaStreamAudioDestinationNode create with AudioContext . createMediaStreamDestination() mix

Slide 19

Slide 19 text

Process of MCU Servder：Audio • Get remote MediaStream through RTCPeerConnection • Create MediaStreamAudioSourceNode of Web Audio – Convert MediaStream to Web Audio node • Create MediaStreamAudioDestinationNode of Web Audio – Mix SourceNode of other members: sourceNode.connect(destinationNode); • Get MediaStream – Pass to RTCPeerConnection. addStream(), and send to remote peer • * repeat for all members 19

Slide 20

Slide 20 text

Inside of MCU Server：Video & Audio 20 RTCPeerConnection A RTCPeerConnection D MediaStream (Video A+B+C+D) MediaStream (Audio B+C+D) RTCPeerConnection B RTCPeerConnection C MediaStream (Audio A+C+D) MediaStream (Audio A+B+D) MediaStream (Audio A+B+C) Multi-Stream with Video only MediaStream and Audio only MediaStream

Slide 21

Slide 21 text

Browser MCU DEMO 2 • Canvas is So flexible ! • YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=1h35s – 1:00:35 to 1:01:29 21

Slide 22

Slide 22 text

Browser MCU DEMO 2 22 Stripe Variable Zoom

Slide 23

Slide 23 text

Browser MCU DEMO 3 • 3D with Canvas + WebGL • YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=1h1m51s – 1:01:51 to 1:02:21 23

Slide 24

Slide 24 text

Browser MCU DEMO 3 24 3D with WebGL

Slide 25

Slide 25 text

Browser MCU DEMO 3 25 Stripe with 7 members Grid with 17 members Frame was dropped, in case of over 16 members Too heavy load, even for core-i7 CPU

Slide 26

Slide 26 text

Browser MCU DEMO 4 26 • Record mix video/audio with MediaRecorder • YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=1h3m47s – 1:03:47 to 1:04:33

Slide 27

Slide 27 text

Inside of MCU Server: Recording 27 MediaStream (Video A+B+C+D) element Same as sending to remote peers Web Audio API MediaStream (Audio A+B+C+D) Mix all audio for recording videoTracks[] MediaStreamTrack MediaStreamTrack aidioTracks[] MediaStream new MediaStream() MediaStream.addTrack() MediaStream.addTrack() MediaRecorder WebM

Slide 28

Slide 28 text

MCU vs. SFU 28

Slide 29

Slide 29 text

SFU: Selective Forwarding Unit 29 Browser A Browser B Browser D Browser C SFU Dispatch audio/video SFU • User SFU server, CPU load: Low • Browser side: low CPU load • Browser side: network load: middle Able to choose display layout on member side ◎ Browser A Browser B Browser D Browser C MCU Mix video / audio MCU • Use MCU server, CPU load: Super High • Browser side: very low CPU/Network load

Slide 30

Slide 30 text

Browser MCU DEMO 5 • It is NOT possible to choose display layout on member side, with MCU • ... NO, it is possible! • YouTube - WebRTC Meetup Tokyo #11 – https://youtu.be/0RYWw1nPxio?t=1h6m49s – 1:06:49 to 1:07:25 30

Slide 31

Slide 31 text

Browser MCU DEMO 5 31 Zooming on User/Member side

Slide 32

Slide 32 text

Inside of zooming on MCU member 32 MediaStream (Video A+B+C+D) hidden element drawImage() drawImage() drawImage() drawImage() It is possible to change layout with Canvas in member side (It is possible to use even multiple canvas)

Slide 33

Slide 33 text

Appendix：painful point of MCU • Adjusting timing is painful for commercial MCU – Adjust timing for video of all members – Adjust timing of video and audio (lip sync) • For adjusting timing, it is need to wait for frame or block – But, impossible to wait forever – So it is necessary to timeout and send alternative frame/block • Sending Black frame, or repeat last frame • Sending silent block • Browser MCU does not care about adjusting timing 33

Slide 34

Slide 34 text

Appendix：MCU Sample page • Start page (not http, but https) – https://mcu.talkin.info/ • Prepare a room as MCU server – Type room name – Click [Prepare a Room as MCU Server] – Wait for connection, or click [Re-Connect] • Join the room – Type room name, then click [Join to Room] – Click [Start video] to use camera/microphone – Click [Connect] 34

Slide 35

Slide 35 text

Conclusion • Potential of WebRTC + Canvas is unlimited – It is very powerful toy – It is possible to send local video file • http://qiita.com/massie_g/items/5a6c4b69374d5997dc37 • But, CPU load of server side is super heavy – It is not scalable • Maybe useful, if user can take care of server side browser • Original Japanese Slide is on SlideShare – http://www.slideshare.net/mganeko/webrtc-build-mcu-on-browser 35

Slide 36

Slide 36 text

Thank you! 36