Lessons from the experimental edge of technology

When asked if I wanted to build a Chrome Experiment: a thing that shows off some of the new technology coming to browsers - of course I said yes! Then promptly I bit off the biggest bite of tech I could get my hands on: WebRTC, two way video, peer connection API, orientation events, CSS3 animations, Three.js: generally a lot of tech I didn't really understand!

This talk is about the experience of working with WebRTC for video & data streaming and creating a game with embedded video. How some of the problems I encountered blew my schedule to pieces and perhaps how I could have done things differently.

Transcript

1. Lessons on the experimental edge of web technology Remy Sharp / @rem http://www.flickr.com/photos/spaztacular/3537911581/
2. Experiment... Google IO... Head Shots.
3. Caveat
4. Two tranches of work 1. Real-time communication - WebRTC 2. Graphics and performance
5. WebRTC... aaaahhahh.
6. Don't make me write code • Peer.js • SimpleWebRTC • easyRTC • WebRTC.io
8. My Ed209
9. Three simple tasks 1. Get a room (aka pin for my game) 2. Listen and action remote connections 3. Add my video stream
10. // `pin` is the unique game id // basically a room name rtc.connect('wss:localhost:8000', pin); 1. Get a room
11. rtc.on('add remote stream', function(stream, id) { // `stream` is the video stream // 'remote' refers to a video element on the page // with the id of 'remote' rtc.attachStream(stream, 'remote'); // in the actual game we'd trigger an event to // notify that the remote stream has been added, // and we'd start playing }); 2. Listen for join
12. // this will give the video & audio stream // to any peers rtc.createStream({ 'video': true, 'audio': true }, function(stream) { // since this game doesn't show own our // face, we do nothing with the stream }); 3. Add my stream
13. The lesson Libraries are good (when you're fumbling) Libraries are bad (when shit don't work)
14. Angry cat say: WHY YOU SEND ALL TEH DATA?
15. navigator.getUserMedia({ 'video': { 'mandatory': { maxWidth: 160, maxHeight: 90 } } }, function(stream) { video.src = URL.createObjectURL(stream); // will be true when stream is running // in video elem console.log(video.videoWidth === 160); });
16. The lesson Only serve what you need. (an oldie, but a goodie)
17. It's not all client side :(
18. var express = require('express'), app = express(), server = require('http').createServer(app), webRTC = require('webrtc.io').listen(server); server.listen(8000); Bind webrtc.io. Bosh.
19. Data: RTCDataChannel 1. Reliable: true / TCP* 2. Reliable: false / UDP
20. function dataChannelOpen() { for (var id in rtc.dataChannels) { // note: string based ready states // (rather than usual ints) if (rtc.dataChannels[id].readyState == 'open') { return true; } } return false; } Test for open channel
21. Streaming shit loads of event data • If you can: avoid it • Head Shots only sends orientation event change • Peer only gets a single message
22. Mechanics 1.Open Websocket for game discovery 2.Connect video & audio stream 3.Test if data channel is "open" 4.Allow game to start...
23. How it originally worked 1. Welcome screen with join & start button 2. Start - automatically created a new game (now in waiting mode) 3. Join - prompted the user for a pin code 4. Waiting - this used a hanging XHR request 5. Play - wherein both players prompted for access to their camera
24. But...connected games would instantly disconnect. W.T.F. ?
25. Paper cup phone example
26. SPA
27. The lesson SPA can save you. SPA for persistance.
28. Add SSL
29. Graphics
30. •Started with DOM, but 3D sucked •Moved to Three.js with help from @seb_ly 3D
32. •CSS3 Renderer performed... •But caused entire chunks to vanish •Weekend later, bug fixed in Chrome Beta Fixing 3D
33. •Render background once, separately from moving foreground •Only one video (not two) •rAF, not setInterval Back to 30fps+
40. Playing in the future
41. Crazy latest tech also means latest standards. •Micro DOM & event library: min.js •Extensive use of HTML5's classList & dataset •Micro data binding: bind.js •Super minimal xhr
42. // `pin` is the unique game id var $ = document.querySelectorAll.bind(document); Node.prototype.on = Node.prototype.addEventListener; $('#somelink')[0].on('touchstart', handleTouch); Less is $$$$ https://github.com/remy/min.js
43. $('#gameover').dataset.winner = winner; $('#gameover').classList.add('showWinner'); Less is...HTML5 https://github.com/remy/min.js
44. $.on('gameReady', startTheFunk); // Later that day... if (connected) { $.trigger('gameReady'); } Less is...pubsub https://github.com/remy/min.js
45. var callback; if (typeof mapping[path.join('.')] === 'function') { callback = mapping[path.join('.')]; } else { callback = function (value) { if (callback.elements) { forEach.call(callback.elements, function (element) { element.innerHTML = value; }); } }; callback.elements = $(mapping[path.join('.')] || '☺'); } Object.defineProperty(target, key, { set: function (v) { value = v; if (callback) { callback(value); } }, get: function () { return value; } }); target[key] = value; Data binding ~40 lines
46. var callback; if (typeof mapping[path.join('.')] === 'function') { callback = mapping[path.join('.')]; } else { callback = function (value) { if (callback.elements) { forEach.call(callback.elements, function (element) { element.innerHTML = value; }); } }; callback.elements = $(mapping[path.join('.')] || '☺'); } Object.defineProperty(target, key, { set: function (v) { value = v; if (callback) { callback(value); } }, get: function () { return value; } }); target[key] = value; Setter/getter action
47. var callback; if (typeof mapping[path.join('.')] === 'function') { callback = mapping[path.join('.')]; } else { callback = function (value) { if (callback.elements) { forEach.call(callback.elements, function (element) { element.innerHTML = value; }); } }; callback.elements = $(mapping[path.join('.')] || '☺'); } Object.defineProperty(target, key, { set: function (v) { value = v; if (callback) { callback(value); } }, get: function () { return value; } }); target[key] = value; innerHTML setting
48. var callback; if (typeof mapping[path.join('.')] === 'function') { callback = mapping[path.join('.')]; } else { callback = function (value) { if (callback.elements) { forEach.call(callback.elements, function (element) { element.innerHTML = value; }); } }; callback.elements = $(mapping[path.join('.')] || '☺'); } Object.defineProperty(target, key, { set: function (v) { value = v; if (callback) { callback(value); } }, get: function () { return value; } }); target[key] = value; Initialise
49. game = new Bind({ me: { // original object score: 10 } }, { 'me.score': '#myscore' }); // DOM updates game.me.score = 20; https://github.com/remy/bind Binding
50. game = new Bind(result.data, { 'me.score': '#myscore', 'them.score': function (value) { $.trigger('theirScore', value); }, 'turn': function (myturn) { turnEl.dataset.turn = myturn; $.trigger('myturn', myturn); } }); https://github.com/remy/bind
51. The biggies •Newness of WebRTC ate way more time than I anticipated •Google engineers are there to help •Maybe the horse came before the cart?
52. Is WebRTC ready for prime time? •I'm not so sure •Support and interop is being aggressively worked on...but Microsoft...Apple? •Mobile looking hopeful •Give it 12 months* * Check expiry date
54. http://io13webrtc.appspot.com
55. Live here: headshots.leftlogic.com Code: github.com/leftlogic/headshots Yes, I know the source is a bit of a mess! Remy Sharp / @rem Have a pop.