SocketStream - Speaker Deck

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

A web framework for single page apps

Slide 3

Slide 3 text

Created by Owen Barnes @temporalwave ! ! (my boss at a former company)

Slide 4

Slide 4 text

About me Paul Jensen @paulbjensen ! ! (I’m the new lead developer)

Slide 5

Slide 5 text

Where to begin?

Slide 6

Slide 6 text

Why Single Page Apps?

Slide 7

Slide 7 text

With a traditional web app, the user has to refresh the page to see new information

Slide 8

Slide 8 text

Time Server Client

Slide 9

Slide 9 text

Time Server Client GET/football/live HTTP/1.1

Slide 10

Slide 10 text

Time Server Client HTTP/1.1 200 OK

Slide 11

Slide 11 text

Time Server Client 20 seconds later…

Slide 12

Slide 12 text

Time Server Client I wonder what the latest score is… Let’s reload the page

Slide 13

Slide 13 text

Time Server Client GET/football/live HTTP/1.1

Slide 14

Slide 14 text

Time Server Client HTTP/1.1 304 Not Modified

Slide 15

Slide 15 text

Time Server Client

Slide 16

Slide 16 text

Time Server Client The user had to press F5 to get at any new information

Slide 17

Slide 17 text

Time Server Client Even though there was no new information, the server still had to serve the HTTP request

Slide 18

Slide 18 text

This is not a fun experience

Slide 19

Slide 19 text

How do we make this better?

Slide 20

Slide 20 text

Time Server Client

Slide 21

Slide 21 text

We could use AJAX to update the page

Slide 22

Slide 22 text

We’d save the user having to press the F5 key

Slide 23

Slide 23 text

What else can we do?

Slide 24

Slide 24 text

Time Server Client

Slide 25

Slide 25 text

Optimise the response

Slide 26

Slide 26 text

GZIP the response data, and …

Slide 27

Slide 27 text

Avoid sending data we already have on the client

Slide 28

Slide 28 text

We could also separate the HTML from the data

Slide 29

Slide 29 text

Reuse the HTML on the client

Slide 30

Slide 30 text

…and use the server to provide you with just data

Slide 31

Slide 31 text

And the web site becomes a client Server Web App Native App API User

Slide 32

Slide 32 text

The server is just an API

Slide 33

Slide 33 text

A beautiful separation of concerns

Slide 34

Slide 34 text

Overview • The server becomes a REST API serving JSON • HTML compilation is done on the client • As a result, less processing & bandwidth is consumed by the server

Slide 35

Slide 35 text

Why Realtime?

Slide 36

Slide 36 text

Time Server Client

Slide 37

Slide 37 text

Polling the server every [n] seconds for new data is redundant

Slide 38

Slide 38 text

There has to be a better way

Slide 39

Slide 39 text

What if the server could send its client(s) new data as soon as it came about?

Slide 40

Slide 40 text

We can, thanks to WebSockets

Slide 41

Slide 41 text

WebSockets allows data to be sent both ways

Slide 42

Slide 42 text

Time Server Client

Slide 43

Slide 43 text

Time Server Client Goal

Slide 44

Slide 44 text

Time Server Client Goal The server sends a message to the client that an action has occurred

Slide 45

Slide 45 text

We eliminate the need to poll the server for new data

Slide 46

Slide 46 text

Overview • We can replace AJAX polling with WebSockets, and provide a better user experience as a result • We also remove the need to make redundant polling requests back to the server. • We use WebSockets for sending/receiving JSON

Slide 47

Slide 47 text

Single Page Apps + The Realtime Web

Slide 48

Slide 48 text

There are many ways to build this kind of app

Slide 49

Slide 49 text

You could build it mostly from scratch, and use Express + Socket.io

Slide 50

Slide 50 text

Or alternatively, you could use a web framework like Meteor or Firebase

Slide 51

Slide 51 text

SocketStream is somewhere in-between these 2 approaches

Slide 52

Slide 52 text

It provides tools to help with building realtime single page apps...

Slide 53

Slide 53 text

... Whilst trying not to restrict what technologies you can use with your app

Slide 54

Slide 54 text

For example, we don't provide an ORM. Instead, you choose the database & the ORM

Slide 55

Slide 55 text

Also, we don't mandate using a specific client-side framework ! You can use BackBone, Angular, Ember, or something else, that is entirely your choice.

Slide 56

Slide 56 text

What we focus on instead are these things:

Slide 57

Slide 57 text

Building RPC APIs Live Reload WebSocket Management Minifying CSS/JS for production use Client-side code organisation Building PubSub APIs HTML / CSS / JS code preprocessing Session Management Building custom APIs on top of WS HTML Templates Web Workers Connect middleware compatibility

Slide 58

Slide 58 text

I'll run through each of these, 1-by-1. But first, let's look at how to use SocketStream

Slide 59

Slide 59 text

Getting started npm install -g socketstream ! socketstream new my_app

Slide 60

Slide 60 text

Getting started Success! Created app 'my_app' with: ! ✓ Basic chat demo (-m for minimal install) ✓ Javascript example code (-c if you prefer CoffeeScript) ✓ Plain HTML for views (-j if you prefer Jade) Next, run the following commands: ! cd my_app [sudo] npm install ! To start your app: ! node app.js 

Slide 61

Slide 61 text

Here's what the initial file/ folder structure looks like

Slide 62

Slide 62 text

Slide 63

Slide 63 text

Client code is organised into 5 sub-folders

Slide 64

Slide 64 text

Client side code organisation • CODE stores client side JavaScript files and libraries • CSS stores CSS files • STATIC stores public files like images, font files, and other assets • TEMPLATES stores HTML templates for the single page app to render on the client • VIEWS stores HTML files that are rendered from the server for the initial page

Slide 65

Slide 65 text

Those sub-folders have sub- folders, but are optional

Slide 66

Slide 66 text

This is how we load them // My SocketStream 0.3 app ! var http = require('http'), ss = require('socketstream'); ! // Define a single-page client called 'main' ss.client.define('main', { view: 'app.html', css: ['libs/reset.css', 'app.styl'], code: ['libs/jquery.min.js', 'app'], tmpl: '*' }); ! // Serve this client on the root URL ss.http.route('/', function(req, res){ res.serveClient('main'); });

Slide 67

Slide 67 text

// My SocketStream 0.3 app ! var http = require('http'), ss = require('socketstream'); ! // Define a single-page client called 'main' ss.client.define('main', { view: 'app.html', css: ['libs/reset.css', 'app.styl'], code: ['libs/jquery.min.js', 'app'], tmpl: '*' }); ! // Serve this client on the root URL ss.http.route('/', function(req, res){ res.serveClient('main'); });

Slide 68

Slide 68 text

Slide 69

Slide 69 text

Slide 70

Slide 70 text

Slide 71

Slide 71 text

Slide 72

Slide 72 text

Slide 73

Slide 73 text

SocketStream uses Browserify to handle requiring JS files

Slide 74

Slide 74 text

Browserify allows us to use a Node.js style of requiring JS files

Slide 75

Slide 75 text

// This file automatically gets called first by SocketStream and must always exist ! // Make 'ss' available to all modules and the browser console window.ss = require('socketstream'); ! ss.server.on('disconnect', function(){ console.log('Connection down :-('); }); ! ss.server.on('reconnect', function(){ console.log('Connection back up :-)'); }); ! ss.server.on('ready', function(){ ! // Wait for the DOM to finish loading jQuery(function(){ // Load app require('/app'); ! }); ! });

Slide 76

Slide 76 text

Slide 77

Slide 77 text

Slide 78

Slide 78 text

Over the years, developers have come up with new languages to generate HTML, CSS, and JavaScript

Slide 79

Slide 79 text

SocketStream allows developers to use these code preprocessors in their apps

Slide 80

Slide 80 text

Adding a preprocessor is simple // Code Formatters ss.client.formatters.add(require('ss-stylus'));

Slide 81

Slide 81 text

For Javascript • SS-COFFEE - supports using CoffeeScript • SS-GORILLA - supports using GorillaScript

Slide 82

Slide 82 text

For CSS • SS-STYLUS - supports using Stylus • SS-LESS - supports using Less

Slide 83

Slide 83 text

For HTML Views • SS-JADE - supports using Jade

Slide 84

Slide 84 text

For HTML Templating • SS-HOGAN - supports using Twitter's Hogan.js • SS-COFFEEKUP - supports using CoffeeKup

Slide 85

Slide 85 text

Setting a Templating engine // Use server-side compiled Hogan (Mustache) templates. Others engines available ss.client.templateEngine.use(require('ss-hogan'));

Slide 86

Slide 86 text

Slide 87

Slide 87 text

Having to press F5 to reload the page in order to view changes to HTML/CSS/JS...

Slide 88

Slide 88 text

... is not a fun experience

Slide 89

Slide 89 text

In development mode, SocketStream will watch the client files for changes, and reload the page when they occur

Slide 90

Slide 90 text

In the case of CSS, SocketStream will apply the changes without reloading the page

Slide 91

Slide 91 text

Slide 92

Slide 92 text

Client-side HTML templates are made available to the browser via the ss.tmpl object

Slide 93

Slide 93 text

No content

Slide 94

Slide 94 text

Slide 95

Slide 95 text

When you're building a single page app, you'll have a lot of JS files, and maybe a few CSS files

Slide 96

Slide 96 text

But serving a HTML page with lots of these files can take time, and is inefficient

Slide 97

Slide 97 text

SocketStream provides a way to concatenate, minify, and GZip these files into 1 JS and 1 CSS file

Slide 98

Slide 98 text

This saves bytes being transferred, as well as reducing the number of HTTP requests you make

Slide 99

Slide 99 text

Also, you can tell SocketStream to load these files from a CDN

Slide 100

Slide 100 text

Setting a Templating engine // Minimize and pack assets if you type: SS_ENV=production node app.js if (ss.env === 'production') ss.client.packAssets();

Slide 101

Slide 101 text

Slide 102

Slide 102 text

Web Workers are handy for intensive client-side JS operations

Slide 103

Slide 103 text

SocketStream provides support for using Web Workers in your app

Slide 104

Slide 104 text

First, create a folder

Slide 105

Slide 105 text

Next, put your web worker files in that folder

Slide 106

Slide 106 text

Then, load the worker in a client code file, and enjoy

Slide 107

Slide 107 text

Slide 108

Slide 108 text

SocketStream uses Connect middleware to support HTTP features

Slide 109

Slide 109 text

SocketStream uses the following middleware by default: • compress - for GZipping assets • cookieParser - for handling user tracking • favicon - for serving a favicon.ico ﬁle • session - for handling sessions • static - for serving static assets

Slide 110

Slide 110 text

SocketStream uses the following middleware by default: • compress middleware is loaded ﬁrst, before all other middleware • static middleware is loaded last, after all other middleware

Slide 111

Slide 111 text

SocketStream provides a way to load custom middleware into the connect stack

Slide 112

Slide 112 text

ss.http.middleware.prepend() ss.http.middleware.append()

Slide 113

Slide 113 text

This allows you to use all of the connect middleware out there today, i.e. EveryAuth

Slide 114

Slide 114 text

Slide 115

Slide 115 text

We use connect’s session middleware, so authentication can be done with either EveryAuth, PassportJS, or you can roll your own.

Slide 116

Slide 116 text

We also recommend using connect-redis

Slide 117

Slide 117 text

Both HTTP and WebSocket interfaces can get/set the session data

Slide 118

Slide 118 text

Via HTTP // app.js ss.http.router.on('/updateSession', function(req, res) { req.session.myVar = 4321; res.end('req.session.myVar has been updated to', req.session.myVar); });

Slide 119

Slide 119 text

Via WebSockets

Slide 120

Slide 120 text

Slide 121

Slide 121 text

RPC is a common pattern for clients requesting data from the server

Slide 122

Slide 122 text

SocketStream provides a way to construct RPC APIs with flexibility

Slide 123

Slide 123 text

No content

Slide 124

Slide 124 text

No content

Slide 125

Slide 125 text

Slide 126

Slide 126 text

PubSub is a great pattern for Single Page Apps

Slide 127

Slide 127 text

SocketStream handles this in various ways:

Slide 128

Slide 128 text

1 - Publishing to everyone viewing the app right now ss.publish.all('newMessage', message); // Broadcast the message to everyone Server // Listen out for newMessage events coming from the server ss.event.on('newMessage', function(message) { // do something with the message }); Client

Slide 129

Slide 129 text

2 - Sending to private channels // in a /server/rpc file after calling req.use('session') middleware ! req.session.channel.subscribe('disney') ! req.session.channel.unsubscribe('kids') ! req.session.channel.reset() // unsubscribes the session from every channel ! req.session.channel.list() // shows what channels are subscribed to Server (subscribe/unsubscribe the session )

Slide 130

Slide 130 text

2 - Sending to private channels // in a /server/rpc file ss.publish.channel('disney', 'chatMessage', {from: 'jerry', message: 'Has anyone seen Tom?'}); Server (publish to channel) // in a /client/code file ss.event.on('chatMessage', function(msg, channelName){ console.log('The following message was sent to the ' + channelName + ' channel:', msg); }); Client (receive channel message)

Slide 131

Slide 131 text

3 - Sending to users // in a /server/rpc file ss.publish.user('fred', 'specialOffer', 'Here is a special offer just for you!'); Server

Slide 132

Slide 132 text

4 - Sending to a browser tab // in a /server/rpc file ss.publish.socketId('254987654324567', 'justForMe', 'Just for one tab'); Server

Slide 133

Slide 133 text

Slide 134

Slide 134 text

On top of RPC and PubSub, SocketStream provides you with a way to create custom request responders

Slide 135

Slide 135 text

Request Response is basically a WebSocket message handler

Slide 136

Slide 136 text

It allows you to write message handling for games, where every byte matters

Slide 137

Slide 137 text

Slide 138

Slide 138 text

WebSockets are not immortal…

Slide 139

Slide 139 text

They are mangled by mobile networks…

Slide 140

Slide 140 text

Blocked by firewalls…

Slide 141

Slide 141 text

Routed to dead ends by proxy servers

Slide 142

Slide 142 text

And severed by train tunnels

Slide 143

Slide 143 text

Also, browser support for WebSockets isn’t guaranteed

Slide 144

Slide 144 text

You need a transport strategy

Slide 145

Slide 145 text

Originally, SocketStream used Socket.io

Slide 146

Slide 146 text

But Socket.io asserted that if a browser supported WebSockets, then it would work

Slide 147

Slide 147 text

They learned from this, by building Engine.io

Slide 148

Slide 148 text

I created the transport wrapper for Engine.io in SocketStream for Bechtel & Dashku

Slide 149

Slide 149 text

And designed it to reconnect the client when severed

Slide 150

Slide 150 text

Months later, it made it’s way into SocketStream’s core.

Slide 151

Slide 151 text

SocketStream let’s you use this, alongside SockJS

Slide 152

Slide 152 text

…and that is SocketStream in a nutshell. Whew!

Slide 153

Slide 153 text

Let’s look at some SocketStream apps in the wild

Slide 154

Slide 154 text

Hollow hollowdocumentary.com

Slide 155

Slide 155 text

Vmux vmux.co

Slide 156

Slide 156 text

Dashku dashku.com

Slide 157

Slide 157 text

SocketStream plugins

Slide 158

Slide 158 text

SS-BACKBONE

Slide 159

Slide 159 text

SS-ANGULAR

Slide 160

Slide 160 text

SS-CUCUMBER

Slide 161

Slide 161 text

Tips for deploying SocketStream in production

Slide 162

Slide 162 text

1 - Check your server’s ulimit configuration (This can bite you hard)

Slide 163

Slide 163 text

I learned this when Dashku went #1 on Hacker News in 45min

Slide 164

Slide 164 text

2 - Use HTTPS, but handle it at the load balancer level rather than at the app level

Slide 165

Slide 165 text

HTTPS helps to improve the stability of WebSocket connections, especially on mobile devices

Slide 166

Slide 166 text

But Node’s HTTPS implementation is noticeably slower than using HAProxy or Nginx

Slide 167

Slide 167 text

Where is SocketStream going next?

Slide 168

Slide 168 text

We’re in the process of getting SocketStream’s test coverage up

Slide 169

Slide 169 text

We’re also trying to close some ancient bugs

Slide 170

Slide 170 text

We also need better documentation

Slide 171

Slide 171 text

We’re giving the web site an overhaul

Slide 172

Slide 172 text

And we want to document how SocketStream’s internals function, to help build 0.4