Lessons from the experimental edge of technology

Lessons from the experimental edge of technology

When asked if I wanted to build a Chrome Experiment: a thing that shows off some of the new technology coming to browsers - of course I said yes! Then promptly I bit off the biggest bite of tech I could get my hands on: WebRTC, two way video, peer connection API, orientation events, CSS3 animations, Three.js: generally a lot of tech I didn't really understand!

This talk is about the experience of working with WebRTC for video & data streaming and creating a game with embedded video. How some of the problems I encountered blew my schedule to pieces and perhaps how I could have done things differently.

C8b387c489181844b3ffc704fadc0f14?s=128

Remy Sharp

May 29, 2013
Tweet

Transcript

  1. Lessons on the experimental edge of web technology Remy Sharp

    / @rem http://www.flickr.com/photos/spaztacular/3537911581/
  2. Experiment... Google IO... Head Shots.

  3. Caveat

  4. Two tranches of work 1. Real-time communication - WebRTC 2.

    Graphics and performance
  5. WebRTC... aaaahhahh.

  6. Don't make me write code • Peer.js • SimpleWebRTC •

    easyRTC • WebRTC.io
  7. None
  8. My Ed209

  9. Three simple tasks 1. Get a room (aka pin for

    my game) 2. Listen and action remote connections 3. Add my video stream
  10. // `pin` is the unique game id // basically a

    room name rtc.connect('wss:localhost:8000', pin); 1. Get a room
  11. rtc.on('add remote stream', function(stream, id) { // `stream` is the

    video stream // 'remote' refers to a video element on the page // with the id of 'remote' rtc.attachStream(stream, 'remote'); // in the actual game we'd trigger an event to // notify that the remote stream has been added, // and we'd start playing }); 2. Listen for join
  12. // this will give the video & audio stream //

    to any peers rtc.createStream({ 'video': true, 'audio': true }, function(stream) { // since this game doesn't show own our // face, we do nothing with the stream }); 3. Add my stream
  13. The lesson Libraries are good (when you're fumbling) Libraries are

    bad (when shit don't work)
  14. Angry cat say: WHY YOU SEND ALL TEH DATA?

  15. navigator.getUserMedia({ 'video': { 'mandatory': { maxWidth: 160, maxHeight: 90 }

    } }, function(stream) { video.src = URL.createObjectURL(stream); // will be true when stream is running // in video elem console.log(video.videoWidth === 160); });
  16. The lesson Only serve what you need. (an oldie, but

    a goodie)
  17. It's not all client side :(

  18. var express = require('express'), app = express(), server = require('http').createServer(app),

    webRTC = require('webrtc.io').listen(server); server.listen(8000); Bind webrtc.io. Bosh.
  19. Data: RTCDataChannel 1. Reliable: true / TCP* 2. Reliable: false

    / UDP
  20. function dataChannelOpen() { for (var id in rtc.dataChannels) { //

    note: string based ready states // (rather than usual ints) if (rtc.dataChannels[id].readyState == 'open') { return true; } } return false; } Test for open channel
  21. Streaming shit loads of event data • If you can:

    avoid it • Head Shots only sends orientation event change • Peer only gets a single message
  22. Mechanics 1.Open Websocket for game discovery 2.Connect video & audio

    stream 3.Test if data channel is "open" 4.Allow game to start...
  23. How it originally worked 1. Welcome screen with join &

    start button 2. Start - automatically created a new game (now in waiting mode) 3. Join - prompted the user for a pin code 4. Waiting - this used a hanging XHR request 5. Play - wherein both players prompted for access to their camera
  24. But...connected games would instantly disconnect. W.T.F. ?

  25. Paper cup phone example

  26. SPA

  27. The lesson SPA can save you. SPA for persistance.

  28. Add SSL

  29. Graphics

  30. •Started with DOM, but 3D sucked •Moved to Three.js with

    help from @seb_ly 3D
  31. None
  32. •CSS3 Renderer performed... •But caused entire chunks to vanish •Weekend

    later, bug fixed in Chrome Beta Fixing 3D
  33. •Render background once, separately from moving foreground •Only one video

    (not two) •rAF, not setInterval Back to 30fps+
  34. None
  35. None
  36. None
  37. None
  38. None
  39. None
  40. Playing in the future

  41. Crazy latest tech also means latest standards. •Micro DOM &

    event library: min.js •Extensive use of HTML5's classList & dataset •Micro data binding: bind.js •Super minimal xhr
  42. // `pin` is the unique game id var $ =

    document.querySelectorAll.bind(document); Node.prototype.on = Node.prototype.addEventListener; $('#somelink')[0].on('touchstart', handleTouch); Less is $$$$ https://github.com/remy/min.js
  43. $('#gameover').dataset.winner = winner; $('#gameover').classList.add('showWinner'); Less is...HTML5 https://github.com/remy/min.js

  44. $.on('gameReady', startTheFunk); // Later that day... if (connected) { $.trigger('gameReady');

    } Less is...pubsub https://github.com/remy/min.js
  45. var callback; if (typeof mapping[path.join('.')] === 'function') { callback =

    mapping[path.join('.')]; } else { callback = function (value) { if (callback.elements) { forEach.call(callback.elements, function (element) { element.innerHTML = value; }); } }; callback.elements = $(mapping[path.join('.')] || '☺'); } Object.defineProperty(target, key, { set: function (v) { value = v; if (callback) { callback(value); } }, get: function () { return value; } }); target[key] = value; Data binding ~40 lines
  46. var callback; if (typeof mapping[path.join('.')] === 'function') { callback =

    mapping[path.join('.')]; } else { callback = function (value) { if (callback.elements) { forEach.call(callback.elements, function (element) { element.innerHTML = value; }); } }; callback.elements = $(mapping[path.join('.')] || '☺'); } Object.defineProperty(target, key, { set: function (v) { value = v; if (callback) { callback(value); } }, get: function () { return value; } }); target[key] = value; Setter/getter action
  47. var callback; if (typeof mapping[path.join('.')] === 'function') { callback =

    mapping[path.join('.')]; } else { callback = function (value) { if (callback.elements) { forEach.call(callback.elements, function (element) { element.innerHTML = value; }); } }; callback.elements = $(mapping[path.join('.')] || '☺'); } Object.defineProperty(target, key, { set: function (v) { value = v; if (callback) { callback(value); } }, get: function () { return value; } }); target[key] = value; innerHTML setting
  48. var callback; if (typeof mapping[path.join('.')] === 'function') { callback =

    mapping[path.join('.')]; } else { callback = function (value) { if (callback.elements) { forEach.call(callback.elements, function (element) { element.innerHTML = value; }); } }; callback.elements = $(mapping[path.join('.')] || '☺'); } Object.defineProperty(target, key, { set: function (v) { value = v; if (callback) { callback(value); } }, get: function () { return value; } }); target[key] = value; Initialise
  49. game = new Bind({ me: { // original object score:

    10 } }, { 'me.score': '#myscore' }); // DOM updates game.me.score = 20; https://github.com/remy/bind Binding
  50. game = new Bind(result.data, { 'me.score': '#myscore', 'them.score': function (value)

    { $.trigger('theirScore', value); }, 'turn': function (myturn) { turnEl.dataset.turn = myturn; $.trigger('myturn', myturn); } }); https://github.com/remy/bind
  51. The biggies •Newness of WebRTC ate way more time than

    I anticipated •Google engineers are there to help •Maybe the horse came before the cart?
  52. Is WebRTC ready for prime time? •I'm not so sure

    •Support and interop is being aggressively worked on...but Microsoft...Apple? •Mobile looking hopeful •Give it 12 months* * Check expiry date
  53. None
  54. http://io13webrtc.appspot.com

  55. Live here: headshots.leftlogic.com Code: github.com/leftlogic/headshots Yes, I know the source

    is a bit of a mess! Remy Sharp / @rem Have a pop.