Slide 1

Slide 1 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland A SO YOU WANT TO BUILD WEBSOCKET ABSTRACTION Friday, October 18, 13

Slide 2

Slide 2 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland     3rd-Eden 3rdEden arnout.kazemier arnoutkazemier Friday, October 18, 13

Slide 3

Slide 3 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland  Friday, October 18, 13

Slide 4

Slide 4 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland  Friday, October 18, 13

Slide 5

Slide 5 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland primus transforming the way we use real-time Friday, October 18, 13

Slide 6

Slide 6 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland open source is flawed Friday, October 18, 13

Slide 7

Slide 7 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland open source is flawed Friday, October 18, 13

Slide 8

Slide 8 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland Friday, October 18, 13

Slide 9

Slide 9 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland Friday, October 18, 13

Slide 10

Slide 10 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland Friday, October 18, 13

Slide 11

Slide 11 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland Friday, October 18, 13

Slide 12

Slide 12 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland Friday, October 18, 13

Slide 13

Slide 13 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland 1 line of code to switch between excluding adding a framework to your package.json Friday, October 18, 13

Slide 14

Slide 14 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland 1 line of code to switch between excluding adding a framework to your package.json different frameworks build-in socket.io, engine.io, sockjs, websockets, browserchannel, but allows addition third party 5 Friday, October 18, 13

Slide 15

Slide 15 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland 1 line of code to switch between excluding adding a framework to your package.json different frameworks build-in socket.io, engine.io, sockjs, websockets, browserchannel, but allows addition third party 5 stars, but growing steady star it on http://github.com/primus/primus 279 Friday, October 18, 13

Slide 16

Slide 16 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland 1 amazing plugin interface keeps the core light, while maximizing developer flexibility Friday, October 18, 13

Slide 17

Slide 17 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland 1 amazing plugin interface keeps the core light, while maximizing developer flexibility different ways to stay connected online/offline detection, client side pings combined with different reconnect strategies 5 Friday, October 18, 13

Slide 18

Slide 18 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland 1 amazing plugin interface keeps the core light, while maximizing developer flexibility different ways to stay connected online/offline detection, client side pings combined with different reconnect strategies 5 reasons on why you should use it fixes bugs in real-time frameworks, fixes websocket crashes, it has a cool name, node.js stream interface, an api that never changes ∞ Friday, October 18, 13

Slide 19

Slide 19 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland MADE WITH ♥ oh, and it’s used in production at nodejitsu and observe.it <3 Friday, October 18, 13

Slide 20

Slide 20 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland ◅ ▻ var Primus = require(‘primus’) , http = require(‘http’); // Create a HTTP server, attach Primus, specify transformer & done var server = http.createServer() , primus = new Primus(server, { transformer: ‘websockets’ }); primus.on(‘connection’, function (spark) { spark.write(‘foo’); spark.write({ json: ‘also works’ }); spark.on(‘data’, function (data) { console.log(‘data’, data); }); // Sparks are streams, so pipe data. require(‘fs’).createReadStream(__dirname +’/index.js’).pipe(spark); }); server.listen(80); the server Friday, October 18, 13

Slide 21

Slide 21 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland ◅ ▻ var primus = new Primus(‘http://localhost:8080?foo=bar', { // Options to further customize Primus. }); primus.on(‘data’, function (data) { console.log(‘received’, data); }); primus.on(‘end’, function () { console.log(‘connection end’); }); primus.write({ sending: 1337 }); primus.write(‘same syntax as node’); the client Friday, October 18, 13

Slide 22

Slide 22 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland WHAT COULD BE YOUR REASON? Friday, October 18, 13

Slide 23

Slide 23 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland Friday, October 18, 13

Slide 24

Slide 24 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland scalability Friday, October 18, 13

Slide 25

Slide 25 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland better performance scalability Friday, October 18, 13

Slide 26

Slide 26 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland high availability better performance scalability Friday, October 18, 13

Slide 27

Slide 27 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland high availability better performance scalability NIH Friday, October 18, 13

Slide 28

Slide 28 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland high availability debug better performance scalability NIH Friday, October 18, 13

Slide 29

Slide 29 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland high availability debug better performance scalability NIH latency Friday, October 18, 13

Slide 30

Slide 30 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland high availability debug better performance scalability NIH OMG WHY ARE YOU USING COFFEESCRIPT latency Friday, October 18, 13

Slide 31

Slide 31 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland high availability debug better performance scalability NIH MAINTENANCE OMG WHY ARE YOU USING COFFEESCRIPT latency Friday, October 18, 13

Slide 32

Slide 32 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland high availability debug better performance biz integration scalability NIH MAINTENANCE OMG WHY ARE YOU USING COFFEESCRIPT latency Friday, October 18, 13

Slide 33

Slide 33 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland high availability missing critical features debug better performance biz integration scalability NIH MAINTENANCE OMG WHY ARE YOU USING COFFEESCRIPT latency Friday, October 18, 13

Slide 34

Slide 34 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland high availability missing critical features debug better performance biz integration scalability NIH MAINTENANCE OMG WHY ARE YOU USING COFFEESCRIPT latency freedom Friday, October 18, 13

Slide 35

Slide 35 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland THE abstraction requirements Friday, October 18, 13

Slide 36

Slide 36 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland Friday, October 18, 13

Slide 37

Slide 37 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland message encoding Friday, October 18, 13

Slide 38

Slide 38 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland message encoding data transfer Friday, October 18, 13

Slide 39

Slide 39 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland AUTHENTICATION & HANDSHAKING message encoding data transfer Friday, October 18, 13

Slide 40

Slide 40 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland AUTHENTICATION & HANDSHAKINGEXTENDABLE BY PLUGINS message encoding data transfer Friday, October 18, 13

Slide 41

Slide 41 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland work in hostile user environments AUTHENTICATION & HANDSHAKINGEXTENDABLE BY PLUGINS message encoding data transfer Friday, October 18, 13

Slide 42

Slide 42 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland reliable connections work in hostile user environments AUTHENTICATION & HANDSHAKINGEXTENDABLE BY PLUGINS message encoding data transfer Friday, October 18, 13

Slide 43

Slide 43 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland AUTOMATIC RECONNECT reliable connections work in hostile user environments AUTHENTICATION & HANDSHAKINGEXTENDABLE BY PLUGINS message encoding data transfer Friday, October 18, 13

Slide 44

Slide 44 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland AUTOMATIC RECONNECT reliable connections no packet loss work in hostile user environments AUTHENTICATION & HANDSHAKINGEXTENDABLE BY PLUGINS message encoding data transfer Friday, October 18, 13

Slide 45

Slide 45 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland AUTOMATIC RECONNECT no high latency reliable connections no packet loss work in hostile user environments AUTHENTICATION & HANDSHAKINGEXTENDABLE BY PLUGINS message encoding data transfer Friday, October 18, 13

Slide 46

Slide 46 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland AUTOMATIC RECONNECT no high latency reliable connections no packet loss work in hostile user environments AUTHENTICATION & HANDSHAKINGEXTENDABLE BY PLUGINS message encoding data transfer no code bloat Friday, October 18, 13

Slide 47

Slide 47 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland AUTOMATIC RECONNECT NETWORK CONDITION HEARTBEATS no high latency reliable connections no packet loss work in hostile user environments AUTHENTICATION & HANDSHAKINGEXTENDABLE BY PLUGINS message encoding data transfer no code bloat Friday, October 18, 13

Slide 48

Slide 48 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland AUTOMATIC RECONNECT NETWORK CONDITION HEARTBEATS no high latency reliable connections no packet loss CROSS BROWSER PLATFORM DOMAIN / / work in hostile user environments AUTHENTICATION & HANDSHAKINGEXTENDABLE BY PLUGINS message encoding data transfer no code bloat Friday, October 18, 13

Slide 49

Slide 49 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland FYI real-time is hard Friday, October 18, 13

Slide 50

Slide 50 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland FYI real-time is an adventure Friday, October 18, 13

Slide 51

Slide 51 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland FYI real-time is an adventure Friday, October 18, 13

Slide 52

Slide 52 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland metrics should be the first component you build knowledge is power, but only for those who use it Friday, October 18, 13

Slide 53

Slide 53 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland data transfer gzip ajax and websocket frames keeps the core light, while maximizing developer flexibility Friday, October 18, 13

Slide 54

Slide 54 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland data transfer proxies and security software mangle headers again, use secure connections to obfuscate the connection from this software, use regular expressions or direct gzip detection Accept-EncodXng: gzip, deflate X-cept-Encoding: gzip, deflate XXXXXXXXXXXXXXX: XXXXXXXXXXXXX ---------------: ------------- ~~~~~~~~~~~~~~~: ~~~~~~~~~~~~~ source: http://developer.yahoo.com/blogs/ydn/pushing-beyond-gzipping-25601.html Friday, October 18, 13

Slide 55

Slide 55 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland stop inventing protocols just use JSON keeps the core light, while maximizing developer flexibility Friday, October 18, 13

Slide 56

Slide 56 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland prevent packet drops use packet/message numbering no open source framework implements this, but has been solved for ages Friday, October 18, 13

Slide 57

Slide 57 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland xhr jsonp handshake connection authorization generate a custom connection id which should be used for each request / Friday, October 18, 13

Slide 58

Slide 58 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland customize handshake response retrieving initial state from server or other data that is needed to give the connection it’s initial state and saves an extra poll request Friday, October 18, 13

Slide 59

Slide 59 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland connect with query strings sending initial state to server again, to save inital request, this can save a extra server request Friday, October 18, 13

Slide 60

Slide 60 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland fix multiple tabs use multiple subdomains / hosts no open source framework implements this, but has been solved for ages this can also be used for load balancing when done correctly Friday, October 18, 13

Slide 61

Slide 61 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland ping pong heartbeat server side ping over come maximum connection holding limitations Friday, October 18, 13

Slide 62

Slide 62 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland ping pong heartbeat server side ping over come maximum connection holding limitations Friday, October 18, 13

Slide 63

Slide 63 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland ping pong heartbeat client side ping detect connection drops, like wifi changes or 3G to wifi upgrades etc Friday, October 18, 13

Slide 64

Slide 64 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland ping pong heartbeat leverage native pings when available websockets have ping/pong protocols which are optimised for this, no api for the browser :( because they didn’t find it useful enough and that it should be implemented by browser vendors (which didn’t implement it) Friday, October 18, 13

Slide 65

Slide 65 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland ping pong heartbeat make the messages useful send metrics, messages, something to make these ping/pongs more effecient Friday, October 18, 13

Slide 66

Slide 66 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland support multiple transports JSONP-polling, long polling, HTMLFile, EventSource WebSockets and think outside the box allow custom transports maybe SPDY push script files? Datachannels? Friday, October 18, 13

Slide 67

Slide 67 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland support multiple transports force HTTPS and upgrade them correctly while handling port upgrades / downgrades for WebSockets Friday, October 18, 13

Slide 68

Slide 68 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland focus on cross domain use postMessage / iframes to transform same domain in to cross domain Friday, October 18, 13

Slide 69

Slide 69 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland iframe all the things their here to assist you, instead of haunt you sandbox your library with the most under estimated technology currently available on the web actually... Friday, October 18, 13

Slide 70

Slide 70 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland be extremely paranoid one single bug can take down your connection watch my previous talks about websuckets and their issues - enough to fill a single talk Friday, October 18, 13

Slide 71

Slide 71 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland be extremely paranoid no-cache and random query string EVERYTHING iOS6 caches POST requests, query strings prevent GET caching as well Friday, October 18, 13

Slide 72

Slide 72 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland warning node.js specific slides ahead ⚠ Friday, October 18, 13

Slide 73

Slide 73 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland build for failure crash only design & service based architecture this in my opinion the easiest to scale and node crashes a lot Friday, October 18, 13

Slide 74

Slide 74 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland make it a stream no flow control library can beat stream#pipe yes, I know, streams are slow… but usability & maintenance wins over performance every day ™ Friday, October 18, 13

Slide 75

Slide 75 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland prevent gc at all cost stop creating new objects, pool & reuse this is a serious pain point for node.js and real-time.. you might even be better of with erlang Friday, October 18, 13

Slide 76

Slide 76 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland     3rd-Eden 3rdEden arnout.kazemier arnoutkazemier Friday, October 18, 13

Slide 77

Slide 77 text

Arnout Kazemier, founder observe.it Realtimeconf 2013, Portland thanks building your own abstraction is rewarding take this journey and start an adventure Friday, October 18, 13