Socket.io & realtime & MongoDB tailable cursors, Oh my Ben Wen - VP MongoLab - [email protected] - @benwen - http://linkedin.com/in/benwen JS.Everywhere() 10.24.2012 An API example with Node.js and capped collections 

Motivation: The client-server environment has changed •  Web 1.0 –  Single client environment: web browser –  Server sends PRESENTATION to client (HTML) –  Server controls “page flow” of application •  Web 2.0 –  Multiple heterogeneous client environments •  Web browser with rich client functionality (HTML5 / AJAX) •  Mobile web •  Mobile apps: iPhone (Objective-C), Android (Java), etc… •  Browser plug-ins •  TV (it’s coming) •  Flash (it’s going) –  Mashups: client receives data from multiple servers –  Server sends DATA to the client (JSON, XML (yuck), RSS, KML) –  Client and server more de-coupled –  ALL IN “SOFT” REALTIME 

Client is getting thicker (smarter) server is getting thinner app server browser HTML (presentation) database web browser mobile browser database web 1.0 web 2.0 app server a lot of presentation logic / app flow very little presentation logic JSON, XML, RSS, KML (data) apps other Very rich and dynamic (HTML5 / CSS3) Lots of presentation logic 

Your new job: make APIs app server browser database web browser mobile browser database web 1.0 web 2.0 app server a lot of presentation logic / app flow very little presentation logic apps other Very rich and dynamic (HTML5 / CSS3) Lots of presentation logic BUILD THIS! HTML (presentation) JSON, XML, RSS, KML (data) 

We want a stack optimized for moving JSON other web services JSON JSON JSON node mongodb AND IN “REALTIME”! (psst, visit nodestack.org) 

An example app 

On multiple Platforms as a Service •  Heroku: http://tractorpush.herokuapp.com/ –  Standard bearer PaaS •  AppFog: http://tractorpush.aws.af.cm/ –  Multi-cloud PaaS. Acquired nodester.com •  Nodejitsu: http://tractorpush.jit.su/ –  Node.js dedicated PaaS •  Windows Azure: http://tractorpush.azurewebsites.net/ –  Globally distributed PaaS •  (working on others… ) 

Tractor Push Architecture 

why node.js? JavaScript is everywhere :) node 

Node.js: What is it? •  The Javascript part: V8 engine •  The Node part: –  single threaded event loop –  non-blocking I/O •  Stunning results –  with 1 process, 1 core, < 1GB RAM –  can push Gbps of traffic, with 100k’s of connections 

Threaded vs. Evented servers Threaded •  One thread per request •  Creating each thread is expensive •  There is a max # threads Evented •  Single threaded •  Event loop •  Non-blocking I/O •  Can handle lots of open connections 

What is Node.js great at? •  Great for I/O bound services and APIs •  Perfect for real-time, push / long-polling •  Great for server to server 

Remember your new job: API makin database app server JSON BUILD THIS! 

It’s about JSON {….} the right data structure -> expressiveness -> productivity (less code, less work) 

Socket.io •  Realtime cross-browser ‘push’ communication •  Clients listen on event types •  Multiple underlying transports –  WebSockets: ‘Upgrade’ of HTTP for full-duplex communication –  Flash Sockets –  AJAX Long polling –  Etc. (multipart streaming, iframe, JSONP) •  Perfect for Node.js’s asynchronous nature •  MIT License 

WebSockets •  Heroku: http://tractorpush.herokuapp.com/ –  No WebSockets •  AppFog: http://tractorpush.aws.af.cm/ –  No WebSockets •  Nodejitsu: http://tractorpush.jit.su/ –  YES! WebSockets •  Windows Azure: http://tractorpush.azurewebsites.net/ –  No WebSockets 

why mongodb? mongodb A JSON database (ok actually binary JSON aka BSON) 

How many of you have done this? ID SSN_LAST F_NAME L_NAME PHONE ALT_PHONES 1 “4347” “Bob” “Smith” “4155164347” “4158983787, 4156563987, 4159878787” 2 “9489” “Mary” “Jones” “6502132333” “6503134421, 6509872736” 3 “9898” “David” “Bass” “6504143451” “6503134421, 6509876228, 6502334998” 4 … and felt guilty… but convinced yourself you won’t need to search on this column anyway… 

Forget big data (for the moment) , this is a data structure thing {
 _id: 1234,
 author: { name: “Bob Davis”, email : [email protected] },
 post: “In these troubled times I like to …“,
 date: { $date: “2010-07-12 13:23UTC” },
 location: [ -121.2322, 42.1223222 ],
 rating: 2.2, 
 comments: [
 { user: “[email protected]”,
 upVotes: 22,
 downVotes: 14, 
 " text: “Great point! I agree” },
 { user: “[email protected]”,
 upVotes: 421,
 downVotes: 22, 
 " text: “You are a moron” } 
 ],
 tags: [ “Politics”, “Virginia” ]
 }" 

Contrast with the relational model for blog post really the best way to persist this? 

{
 _id: 1234,
 author: { name: “Bob Davis”, email : [email protected] },
 post: “In these troubled times I like to …“,
 date: { $date: “2010-07-12 13:23UTC” },
 location: [ -121.2322, 42.1223222 ],
 rating: 2.2, 
 comments: [
 { user: “[email protected]”,
 upVotes: 22,
 downVotes: 14, 
 " text: “Great point! I agree” },
 { user: “[email protected]”,
 upVotes: 421,
 downVotes: 22, 
 " text: “You are a moron” } 
 ],
 tags: [ “Politics”, “Virginia” ]
 }" This is involves less brain damage… 

MongoDB is great for APIs •  JSON-based document (object) database •  Flexible query language •  Atomic updates •  Indexes •  Geospatial awareness (geo apis are cool!) •  Ok, and it scales…. –  Replica-set clusters –  Auto-sharding –  ** BUT THIS IS NOT THE TOPIC OF THE TALK ** 

MongoDB capped collections •  Allows for tailable cursors –  block when out of data, instead of issuing an exception •  Fixed in size (‘capped’) •  Very high performance •  Maintain insertion order •  Not shardable 

Sample document: ‘array’ 

Sample document ‘complex’ 

Server side 

Client side 

Questions? 

More info •  Nodestack –  http://nodestack.org •  MongoDB / 10gen –  http://mongodb.org –  http://10gen.com •  Node.js / Joyent –  http://nodejs.org –  http://joyent.com •  MongoLab –  http://mongolab.com –  [email protected] •  Tractorpush Code –  Server: •  https://github.com/mongolab/ tractorpush-server –  Writer •  https://github.com/mongolab/ tractorpush-writer-ruby –  Doc •  https://devcenter.heroku.com/ articles/realtime-polyglot-app- node-ruby-mongodb-socketio –  Presentation •  https://speakerdeck.com/ benwen/jseverywhere2012 

Thank You Ben Wen - VP MongoLab - [email protected] - @benwen - http://linkedin.com/in/benwen JS.Everywhere() 10.24.2012