Slide 1

Slide 1 text

Tokyo Node Festival 2013 Let’s begin WebRTC 2013.10.26 INFOCOM CORPORATION https://www.infocom.co.jp/english/aboutus/group/index.html 1

Slide 2

Slide 2 text

References • This document is referring following web sites. – WebRTC official site • http://www.webrtc.org/ – HTML5 ROCKS • http://www.html5rocks.com/en/tutorials/webrtc/basics/ – WebRTC for Beginners Muaz Khan • https://www.webrtc-experiment.com/docs/webrtc-for-beginners.html • https://github.com/muaz-khan/WebRTC-Experiment – WebRTC Conference 2013 Atlanta • http://www.webrtcworld.com/conference/west/default.aspx • http://images.tmcnet.com/expo/webrtc-conference/pdfs/WebRTC%20SG13AtlantaRe- CapSml.pdf – Socket.io • Official http://socket.io/ Japanese http://jxck.github.io/socket.io/ – And so many blogs. • All rights of each product is reserved by original right holder. – ®, TM are omitted. 2

Slide 3

Slide 3 text

What’s WebRTC • Web Real-Time Communication – Frameworks for realizing Real-Time Communication on Web Browser – A Part of HTML5 • Many technologies are combined – Do you know what are included? 3

Slide 4

Slide 4 text

WebRTC samples • apprtc https://apprtc.appspot.com/ • Cube Slam https://www.cubeslam.com/ • Blog entries – http://t3n.de/news/cube-slam-pong-klon-live-video-473325/ – http://japanese.engadget.com/2013/06/12/google-cube-slam- webrtc-pong/ 4 WebRTC here

Slide 5

Slide 5 text

Why so interested in WebRTC? Because WebRTC is disruptive. Career fixed telephone mobile phone (TV broadcast) method Over The Top Skype, WebEx (YouTube, USTREAM) Web Browser WebRTC can speak to world- wide (with cost) for users can speak to world- wide with low cost no application, free to use only careers with infrastructure can participate market independent of careers, few venders can participate without special infrastructure, everyone can participate none for venders limited API, limited combination fully programmable, use as component single use usage user can combine built in to products / services call center, e-commerce, information site, etc.

Slide 6

Slide 6 text

Combination VS. Built in Combination  making estimate sheet with WORD and Calculator Total: 568,900 Automatic calculation Built in  making estimate sheet with EXCEL Total: 568,900 Calculate sum by hand, and type into WORD by hand

Slide 7

Slide 7 text

WebRTC elements • User Media – Camera – Microphone – Screen capture • Stream (MediaStream) • P2P communication (RTCPeerConnection) • Data transfer (DataChannel) • Related API, HTML features – JavaScript (most important) – Video, Audio – WebScoket – WebAudio API – Canvas – WebGL – …, etc. 7

Slide 8

Slide 8 text

User Media WebRTC Basic, Use camera 8

Slide 9

Slide 9 text

DEMO • Various camera (for Chrome) – http://localhost/rtc/camera.html 9

Slide 10

Slide 10 text

10 ←Mirror ← Press, Expand Upside-down, Rotate Black-and-white, Sepia * Combination with CSS3

Slide 11

Slide 11 text

Self camera (Chrome) WebRTC Camera var video1 = document.getElementById('video1'); navigator.webkitGetUserMedia({audio:false, video:true}, function(stream) { // success video1.src = window.webkitURL.createObjectURL(stream); }, function(err) { // error console.log(err); } ); 11

Slide 12

Slide 12 text

Various camera Sepia camera Black and white Thin camera Heavy camera Mirror camera Upside-down Lean camera 12

Slide 13

Slide 13 text

2 video tag and MediaStream (in mirror camera) 13 1 MediaStream can connect to several outputs. MediaStream : output = 1 : n

Slide 14

Slide 14 text

WebRTC ready browsers • Windows / Mac – Chrome 26 - (Camera, microphone. Configuration for screen capture) – FireFox 22 - (Camera, microphone. NO screen capture) – Opera 15 - (Camera, microphone. NO screen capture) *different prefix for JavaScript Chrome: webkit, FireFox: moz • Android – Chrome 29 - (Camera, microphone. NO screen capture) • iOS – Not supported yet – Google says, it is on progress. coming soon. • At WebRTC Conference 2013 Atlanta • * I hope it will be include in Chrome for iOS near future. 14

Slide 15

Slide 15 text

WebRTC communication types • standalone – microphone capture → speaker – camera capture → view in screen – screen capture – audio recording – video recording – Image capture (with Canvas) 15

Slide 16

Slide 16 text

DEMO • 10 seconds video message (for Chrome) – Store video, audio, capture on local memory – Upload to server – Streaming playback 16

Slide 17

Slide 17 text

17

Slide 18

Slide 18 text

CAUTION!! 18 Be careful !

Slide 19

Slide 19 text

Howling WebRTC Camera var video1 = document.getElementById('video1'); navigator.webkitGetUserMedia({audio:true, video:true}, function(stream) { // success video1.src = window.webkitURL.createObjectURL(stream); }, function(err) { // error console.log(err); } ); 19

Slide 20

Slide 20 text

How howling happens (local) 20 Most of samples are implemented like this  Headphone is necessary

Slide 21

Slide 21 text

Howling prevention (local) 21

Slide 22

Slide 22 text

WebRTC Security Access permission 22

Slide 23

Slide 23 text

Communication Video chat with WebRTC 1 to 1, 1 pair 1 to 1, multiple room n to n, multiple room 23

Slide 24

Slide 24 text

Communication • RTCPeerConnection – Transfer root for MediaStream ( video and audio ) – P2P (Peer to Peer)  browser to browser – use UDP/IP 24 RTCPeerConnection RTCPeerConnection UDP/IP

Slide 25

Slide 25 text

Before starting P2P communication • Have to know IP address of each other • Have to know UPD port number to use – UDP port is decided dynamically • Negotiation process is necessary, before RTCPeerConnection communication – This process is called as “Signaling” 25 RTCPeerConnection RTCPeerConnection UDP/IP IP address of each other UDP port number to use

Slide 26

Slide 26 text

Signaling before P2P • Broker is needed, which is known by both browser – → Signaling server is prepared usually • No standard of signaling protocol – Your own way • WebSocket (TCP/IP) • Ajax (HTTP, HTTPS) – Existing protocol • SIP(for VoIP) with WebSocket(TCP/IP) • XMPP(for IM)with WebSocket(TCP/IP) 26

Slide 27

Slide 27 text

Signaling before P2P • Broker is needed, which is known by both browser – → Signaling server is prepared usually • No standard of signaling protocol – Your own way • WebSocket (TCP/IP) • Ajax (HTTP, HTTPS) – Existing protocol • SIP(for VoIP) with WebSocket(TCP/IP) • XMPP(for IM)with WebSocket(TCP/IP) 27 Node.js ( + socket.io )

Slide 28

Slide 28 text

Exchanged information while signaling • Session Description Protocol (SDP) – Information of session contents, such as media type (video, audio), codec, etc. – IP address, port number – P2P data transfer protocol → Secure RTP is used in WebRTC – Band width – Session property (name, id, active time, etc.) • Interactive Connectivity Establishment (ICE) – Lists of possible transfer roots • Direct P2P • Using STUN to go through NAT port mapping • Using TRUN, with packet relay server – Shortest (less hop) root will be chosen 28

Slide 29

Slide 29 text

Process of signaling : SDP (0) 29 PeerConnection socket Application Signaling Server PeerConnection socket Application connect() connect connect() connect

Slide 30

Slide 30 text

Process of signaling : SDP (1) 30 PeerConnection socket Application Signaling Server PeerConnection socket Application connect() connect connect() connect createOffer() offer sdp setLocalDescription(sdp) send(sdp) send sdp send sdp onMessage(sdp) setRemoteDescription(sdp)

Slide 31

Slide 31 text

Process of signaling : SDP (2) 31 PeerConnection socket Application Signaling Server PeerConnection socket Application connect() connect connect() connect createOffer() offer sdp setLocalDescription(sdp) send(sdp) send sdp send sdp onMessage(sdp) setRemoteDescription(sdp) createAnswer() answer sdp send sdp send(sdp) send sdp onMessage(sdp) setRemoteDescription(sdp) setLocalDescription(sdp)

Slide 32

Slide 32 text

Process of signaling : ICE (1) 32 PeerConnection socket Application Signaling Server PeerConnection socket Application send(ice) send ice send ice onMessage(ice) addIceCandidate(ice) onIceCandidate(ice)

Slide 33

Slide 33 text

Process of signaling : ICE (2) 33 PeerConnection socket Application Signaling Server PeerConnection socket Application onIceCandidate(ice) send(ice) send ice send ice onMessage(ice) addIceCandidate(ice) send ice send(ice) send ice onMessage(ice) addIceCandidate(ice) onIceCandidate(ice)

Slide 34

Slide 34 text

Process of signaling : ICE (3) 34 PeerConnection socket Application Signaling Server PeerConnection socket Application onIceCandidate(ice) send(ice) send ice send ice onMessage(ice) addIceCandidate(ice) send ice send(ice) send ice onMessage(ice) addIceCandidate(ice) onIceCandidate(ice) onIceCandidate() : end of candidate onIceCandidate() : end of candidate P2P stream transfer

Slide 35

Slide 35 text

WebRTC communication types (1) • bidirectional communication – 1 to 1 audio – 1 to 1 video (& audio) – 1 to 1 screen capture (& audio) 35 Single tenant Only 1 pair on 1 server

Slide 36

Slide 36 text

DEMO • WebRTC 1 to 1 Video Chat (for Chrome) – Server is running on Node.js + socket.io – Refer Simple WebRTC • http://simplewebrtc.com/ 36

Slide 37

Slide 37 text

1 to 1 signaling server (node.js) 37 var io = require('socket.io').listen(9001); io.sockets.on('connection', function(socket) { socket.on('message', function(message) { socket.broadcast.emit('message', message); }); socket.on('disconnect', function() { socket.broadcast.emit('user disconnected'); }); }); signaling

Slide 38

Slide 38 text

1 to 1 browser (1) JavaScript socket.io 38 var socket = io.connect('http://localhost:9001/'); socket.on('connect', onChannelOpened) .on('message', onMessage); function onChannelOpened(evt) { } function onMessage(evt) { if (evt.type === 'offer') { sendAnswer(evt); // receive offer, send answer } else if (evt.type === 'answer') { peerConn.setRemoteDescription(new RTCSessionDescription(evt)); // receive answer } else if (evt.type === 'candidate') { var candidate = new RTCIceCandidate({sdpMLineIndex:evt.sdpMLineIndex, sdpMid:evt.sdpMid, candidate:evt.candidate}); peerConn.addIceCandidate(candidate); // set and send candidate } else if (evt.type === 'bye') { stop(); } } Signaling SDP Signaling ICE

Slide 39

Slide 39 text

1 to 1 browser (2) JavaScript connection handling 39 // start communication function connect() { sendOffer(); } // end of communication function hangUp() { socket.json.send({type: "bye"}); stop(); } function stop() { peerConn.close(); peerConn = null; }

Slide 40

Slide 40 text

1 to 1 browser (3) JavaScript SDP offer / answer 40 var peerConn = null; // RTCPeerConnection function sendOffer() { peerConn = prepareNewConnection(); peerConn.createOffer(function (sessionDescription) { // in case of success peerConn.setLocalDescription(sessionDescription); socket.json.send(sessionDescription); }, function () { // in case of error console.log("Create Offer failed"); }, mediaConstraints); } function sendAnswer(evt) { peerConn = prepareNewConnection(); peerConn.setRemoteDescription(new RTCSessionDescription(evt)); peerConn.createAnswer(function (sessionDescription) { // in case of success peerConn.setLocalDescription(sessionDescription); socket.json.send(sessionDescription); }, function () { // in case of error console.log("Create Answer failed"); }, mediaConstraints); } Signaling SDP Signaling SDP

Slide 41

Slide 41 text

1 to 1 browser (4) JavaScript RTCPeerConnection 41 function prepareNewConnection() { var pc_config = {"iceServers":[]}; var peer = new webkitRTCPeerConnection(pc_config); peer.onicecandidate = function (evt) { if (evt.candidate) { socket.json.send({type: "candidate", sdpMLineIndex: evt.candidate.sdpMLineIndex, sdpMid: evt.candidate.sdpMid, candidate: evt.candidate.candidate}); } else { } }; peer.addStream(localStream); peer.addEventListener("addstream", onRemoteStreamAdded, false); peer.addEventListener("removestream", onRemoteStreamRemoved, false) function onRemoteStreamAdded(event) { remoteVideo.src = window.webkitURL.createObjectURL(event.stream); // set stream to video element } function onRemoteStreamRemoved(event) { remoteVideo.src = ""; // clear stream of video element } return peer; } Signaling ICE

Slide 42

Slide 42 text

1 to 1 browser (5) JavaScript Video handling 42 var localVideo = document.getElementById('local-video'); var remoteVideo = document.getElementById('remote-video'); var localStream = null; function startVideo() { navigator.webkitGetUserMedia({video: true, audio: true}, successCallback, errorCallback); function successCallback(stream) { localStream = stream; localVideo.src = window.webkitURL.createObjectURL(stream); localVideo.play(); } function errorCallback(error) { console.error('An error occurred: [CODE ' + error.code + ']'); return; } } function stopVideo() { localVideo.src = ""; localStream.stop(); }

Slide 43

Slide 43 text

Question about Socket.io • Socket.io https://github.com/LearnBoost/socket.io-spec – Socket.IO aims to bring a WebSocket-like API to many browsers and devices • with some specific features to help with the creation of real-world real-time applications and games – Multiple transport support (old user agents, mobile browsers, etc.). 'websocket' , 'flashsocket' , 'htmlfile' , 'xhr-polling' , 'jsonp-polling‘ – Multiple sockets under the same connection (namespaces). • Why do I need to write like this? – var socket = io.connect('http://localhost:9001/'); • I want to use WebScoket, so why not like this? – io.connect('localhost:9001/'); 43

Slide 44

Slide 44 text

Answer about Socket.io • Socket.io Protocol – https://github.com/LearnBoost/socket.io-spec – A Socket.IO client first decides on a transport to utilize to connect. – A simple HTTP(or HTTPS) handshake takes place at the beginning of a Socket.IO connection. – The handshake, if successful, results in the client receiving: • A session id that will be given for the transport to open connections. • A number of seconds within which a heartbeat is expected (heartbeat timeout) • A number of seconds after the transport connection is closed when the socket is considered disconnected if the transport connection is not reopened (close timeout). 44

Slide 45

Slide 45 text

WebRTC communication types (2) • bidirectional communication – 1 to 1 audio – 1 to 1 video (& audio) – 1 to 1 screen capture (& audio) 45 Multi-tenant (multiple rooms) use room (join() / leave()) of Socket.io

Slide 46

Slide 46 text

1 to 1 multi room browser-side JavaScript 46 function onChannelOpened(evt) { console.log('Channel opened.'); channelReady = true; var roomname = getRoomName(); socket.emit('enter', roomname); } function getRoomName() { // for example, set ?roomname in URL var url = document.location.href; var args = url.split('?'); if (args.length > 1) { var room = args[1]; if (room != "") { return room; } } return "_defaultroom"; }

Slide 47

Slide 47 text

1 to 1 multi room server-side (node.js) 47 socket.on('enter', function(roomname) { socket.set('roomname', roomname); socket.join(roomname); }); function emitMessage(type, message) { var roomname; socket.get('roomname', function(err, _room) { roomname = _room; }); if (roomname) { socket.broadcast.to(roomname).emit(type, message); } else { socket.broadcast.emit(type, message); } } socket.on('message', function(message) { emitMessage('message', message); }); socket.on('disconnect', function() { emitMessage('user disconnected'); });

Slide 48

Slide 48 text

CAUTION!! 48 Trap 1

Slide 49

Slide 49 text

ICE handshake process – ideal case 49 User A User B send offer sdp setRemoteDescription(sdp) setRemoteDescription(sdp) send answer sdp send ice addIceCandidate(ice) send ice addIceCandidate(ice) send ice addIceCandidate(ice) send ice addIceCandidate(ice) end of candidate end of candidate P2P stream transfer createNewPeer createNewPeer Exchange Ice in turn

Slide 50

Slide 50 text

ICE handshake process – expected case 50 User A User B send offer sdp setRemoteDescription(sdp) setRemoteDescription(sdp) send answer sdp send ice addIceCandidate(ice) send ice addIceCandidate(ice) send ice addIceCandidate(ice) send ice addIceCandidate(ice) end of candidate end of candidate P2P stream transfer createNewPeer createNewPeer send Ice successive from one side send Ice successive from the other side

Slide 51

Slide 51 text

ICE handshake process – unexpected case 51 User A User B send offer sdp setRemoteDescription(sdp) setRemoteDescription(sdp) send answer sdp send ice addIceCandidate(ice) send ice addIceCandidate(ice) send ice addIceCandidate(ice) send ice addIceCandidate(ice) end of candidate end of candidate P2P stream transfer createNewPeer createNewPeer Accept ICE before create Answer SDP This is not an error case. This is acceptable.

Slide 52

Slide 52 text

WebRTC communication types (3) • bidirectional communication – 1 to 1 audio – 1 to 1 video (& audio) – 1 to 1 screen capture (& audio) – n to n audio – n to n video (& audio) – n to n screen capture (& audio) 52 Multiple people in 1 room, multiple rooms

Slide 53

Slide 53 text

Demo • WebRTC 4 people Video Chat (for Chrome) 53

Slide 54

Slide 54 text

CAUTION!! 54 Trap 2

Slide 55

Slide 55 text

1 to 1 signaling process 55 User A User B send offer sdp (broadcast) setRemoteDescription(sdp) setRemoteDescription(sdp) send answer sdp (broadcast) send ice addIceCandidate(ice) send ice addIceCandidate(ice) send ice addIceCandidate(ice) send ice addIceCandidate(ice) end of candidate end of candidate P2P stream transfer createNewPeer createNewPeer

Slide 56

Slide 56 text

Simple expand for n to n: SDP corrupt 56 User A User B send offer sdp (broadcast) setRemoteDescription(sdp) setRemoteDescription(sdp) send answer sdp User C setRemoteDescription(sdp) send offer sdp (broadcast) send answer sdp createNewPeer createNewPeer createNewPeer no peer! → createNewPeer setRemoteDescription(sdp) send offer sdp corrupt! OK OK OK

Slide 57

Slide 57 text

n to n signaling: split broadcast from SDP 57 User A User B send “response” setRemoteDescription(sdp) setRemoteDescription(sdp) send answer sdp User C setRemoteDescription(sdp) send “call” (broadcast) send answer sdp createNewPeer createNewPeer createNewPeer setRemoteDescription(sdp) send offer sdp send offer sdp OK OK send “response” createNewPeer OK setRemoteDescription(sdp) OK send “call” (broadcast)

Slide 58

Slide 58 text

4 people browser-side (1) JavaScript 58 function call() { if (! isLocalStreamStarted()) return; socket.json.send({type: "call"}); } function onMessage(evt) { var id = evt.from; var target = evt.sendto; var conn = getConnection(id); if (evt.type === 'call') { if (! isLocalStreamStarted()) return; if (conn) return; // already connected if (isConnectPossible()) { socket.json.send({type: "response", sendto: id }); } else { console.warn('max connections. so ignore call'); } } else if (evt.type === 'response') { sendOffer(id); return; } }

Slide 59

Slide 59 text

4 people browser-side (2) JavaScript 59 function sendOffer(id) { var conn = getConnection(id); if (!conn) { conn = prepareNewConnection(id); } conn.peerconnection.createOffer(function (sessionDescription) { // in case of success conn.iceReady = true; conn.peerconnection.setLocalDescription(sessionDescription); sessionDescription.sendto = conn.id; socket.json.send(sessionDescription); }, function () { // in case of error console.log("Create Offer failed"); }, mediaConstraints); conn.iceReady = true; }

Slide 60

Slide 60 text

4 people browser-side (3) JavaScript 60 function sendAnswer(id, evt) { var conn = getConnection(id); if (! conn) { conn = prepareNewConnection(id); } conn.peerconnection.setRemoteDescription(new RTCSessionDescription(evt)); conn.peerconnection.createAnswer(function (sessionDescription) { // in case of success conn.iceReady = true; conn.peerconnection.setLocalDescription(sessionDescription); sessionDescription.sendto = id; socket.json.send(sessionDescription); }, function () { // in case of error console.log("Create Answer failed"); }, mediaConstraints); conn.iceReady = true; }

Slide 61

Slide 61 text

4 people browser-side (4) JavaScript 61 function prepareNewConnection(id) { var pc_config = {"iceServers":[]}; var peer = null; peer = new webkitRTCPeerConnection(pc_config); var conn = new Connection(); conn.id = id; conn.peerconnection = peer; peer.id = id; addConnection(id, conn); // send any ice candidates to the other peer peer.onicecandidate = function (evt) { if (evt.candidate) { socket.json.send({type: "candidate", sendto: conn.id, sdpMLineIndex: evt.candidate.sdpMLineIndex, sdpMid: evt.candidate.sdpMid, candidate: evt.candidate.candidate}); } else { conn.established = true; // end of candidates } }; …

Slide 62

Slide 62 text

4 people browser-side (5) JavaScript 62 var MAX_CONNECTION_COUNT = 3; var connections = {}; // Connection hash function Connection() { // Connection Class var self = this; var id = ""; // socket.id of partner var peerconnection = null; // RTCPeerConnection instance var established = false; // is Already Established var iceReady = false; } function getConnection(id) { var con = null; con = connections[id]; return con; } function addConnection(id, connection) { connections[id] = connection; } getConnectionCount(), isConnectPossible(), deleteConnection(id), …, etc.

Slide 63

Slide 63 text

4 people server-side (node.js) 63 socket.on('message', function(message) { // set message sender message.from = socket.id; // send to specific target var target = message.sendto; if (target) { io.sockets.socket(target).emit('message', message); return; } // broadcast in room emitMessage('message', message); });

Slide 64

Slide 64 text

WebRTC communication types (4) • one-way communication – 1 → 1 audio – 1 → 1 video (& audio) – 1 → 1 screen capture (& audio) – 1 → n audio – 1 → n video (& audio) – 1 → n screen capture (& audio) 64

Slide 65

Slide 65 text

Advantage / disadvantage of P2P • Advantage – No server transfer, no overhead – Without high spec sever • running on Raspberry Pi • Disadvantage – Communication path glows exponentially, for many participants • 2 node → 1 path • 3 node → 3 path • 4 node → 6 path • 5 node → 10 path • 6 node → 15 path 65 P2P Server-client P2P for 4 Server-client for 4

Slide 66

Slide 66 text

66 It takes 6 hours to build node.js …

Slide 67

Slide 67 text

Difficult problem 1 67

Slide 68

Slide 68 text

Direct P2p in same network 68 Dynamic UDP port (50000~65535)

Slide 69

Slide 69 text

IP addr / UDP port exchange via STUN Then P2P over NAT 69

Slide 70

Slide 70 text

Packet relay via TURN server (go through Firewall) 70

Slide 71

Slide 71 text

Packet relay via TURN server (go through Firewall) 71 Not able to work yet. Still under survey.

Slide 72

Slide 72 text

Question • Is there anyone who success to run TURN server by yourself? – http://code.google.com/p/rfc5766-turn-server/ • How to configure firewall, NAT, Proxy ?? 72 Please tell me, any information or advice!

Slide 73

Slide 73 text

Difficult problem 2 73

Slide 74

Slide 74 text

Howling prevention (local) 74

Slide 75

Slide 75 text

…still happens with partner Howling with partner 75 Auto mute, or auto band filter should be needed. Is it possible in JavaScript? (I don’t know)

Slide 76

Slide 76 text

Question • Do you know any example or library of JavaScript? – To subtract sound waveform – To invert sound waveform and add • Something like noise canceling headphone 76 Please tell me, any information or advice!

Slide 77

Slide 77 text

Conclusion • WebRTC is powerful – Expand web browser potential. Camera, microphone, screen capture. – Combination with other HTML5 features • Canvas, CSS, WebAudio API, WebGL • WebRTC is flexible – Local, 1 to 1, n to n – Bidirectional, single-way • WebRTC is disruptive – Any one can participate • WebRTC is programmable – Built into your own products / services 77

Slide 78

Slide 78 text

WebRTC has no limit. Your imagination will define limit. END 78 https://github.com/mganeko/mgrtc

Slide 79

Slide 79 text

Appendix: socket.io tips • Get client information on server • IP address ip = socket.handshake.headers['x-forwarded-for'] || socket.handshake.address.address; • Transport (websocket, ajax …) tr = io.transports[socket.id].name; 79

Slide 80

Slide 80 text

Appendix: Infocom Group 80 http://www.infocom.co.jp https://www.infocom.co.jp/english/aboutus/group/index.html

Slide 81

Slide 81 text

Thank you! 81 END