jseverywhere2012

Programmers are building and designing Web APIs, often before the Web or mobile experience. Real-time APIs are now becoming more prevalent with the acceptance of WebSockets and Node.js. In this talk, we will walk through a Socket.io ‘push’ application that uses MongoDB capped collections and a simple Javascript client. The code is running at tractorpush.herokuapp.com and is available on Github at https://github.com/mongolab/tractorpush-server

Transcript

1. Socket.io & realtime & MongoDB tailable cursors, Oh my Ben Wen - VP MongoLab - ben@mongolab.com - @benwen - http://linkedin.com/in/benwen JS.Everywhere() 10.24.2012 An API example with Node.js and capped collections
2. Motivation: The client-server environment has changed •  Web 1.0 –  Single client environment: web browser –  Server sends PRESENTATION to client (HTML) –  Server controls “page flow” of application •  Web 2.0 –  Multiple heterogeneous client environments •  Web browser with rich client functionality (HTML5 / AJAX) •  Mobile web •  Mobile apps: iPhone (Objective-C), Android (Java), etc… •  Browser plug-ins •  TV (it’s coming) •  Flash (it’s going) –  Mashups: client receives data from multiple servers –  Server sends DATA to the client (JSON, XML (yuck), RSS, KML) –  Client and server more de-coupled –  ALL IN “SOFT” REALTIME
3. Client is getting thicker (smarter) server is getting thinner app server browser HTML (presentation) database web browser mobile browser database web 1.0 web 2.0 app server a lot of presentation logic / app flow very little presentation logic JSON, XML, RSS, KML (data) apps other Very rich and dynamic (HTML5 / CSS3) Lots of presentation logic
4. Your new job: make APIs app server browser database web browser mobile browser database web 1.0 web 2.0 app server a lot of presentation logic / app flow very little presentation logic apps other Very rich and dynamic (HTML5 / CSS3) Lots of presentation logic BUILD THIS! HTML (presentation) JSON, XML, RSS, KML (data)
5. We want a stack optimized for moving JSON other web services JSON JSON JSON node mongodb AND IN “REALTIME”! (psst, visit nodestack.org)
6. An example app
7. On multiple Platforms as a Service •  Heroku: http://tractorpush.herokuapp.com/ –  Standard bearer PaaS •  AppFog: http://tractorpush.aws.af.cm/ –  Multi-cloud PaaS. Acquired nodester.com •  Nodejitsu: http://tractorpush.jit.su/ –  Node.js dedicated PaaS •  Windows Azure: http://tractorpush.azurewebsites.net/ –  Globally distributed PaaS •  (working on others… )
8. Tractor Push Architecture
9. why node.js? JavaScript is everywhere :) node
10. Node.js: What is it? •  The Javascript part: V8 engine •  The Node part: –  single threaded event loop –  non-blocking I/O •  Stunning results –  with 1 process, 1 core, < 1GB RAM –  can push Gbps of traffic, with 100k’s of connections
11. Threaded vs. Evented servers Threaded •  One thread per request •  Creating each thread is expensive •  There is a max # threads Evented •  Single threaded •  Event loop •  Non-blocking I/O •  Can handle lots of open connections
12. What is Node.js great at? •  Great for I/O bound services and APIs •  Perfect for real-time, push / long-polling •  Great for server to server
13. Remember your new job: API makin database app server JSON BUILD THIS!
14. It’s about JSON {….} the right data structure -> expressiveness -> productivity (less code, less work)
15. Socket.io •  Realtime cross-browser ‘push’ communication •  Clients listen on event types •  Multiple underlying transports –  WebSockets: ‘Upgrade’ of HTTP for full-duplex communication –  Flash Sockets –  AJAX Long polling –  Etc. (multipart streaming, iframe, JSONP) •  Perfect for Node.js’s asynchronous nature •  MIT License
16. WebSockets •  Heroku: http://tractorpush.herokuapp.com/ –  No WebSockets •  AppFog: http://tractorpush.aws.af.cm/ –  No WebSockets •  Nodejitsu: http://tractorpush.jit.su/ –  YES! WebSockets •  Windows Azure: http://tractorpush.azurewebsites.net/ –  No WebSockets
17. why mongodb? mongodb A JSON database (ok actually binary JSON aka BSON)
18. How many of you have done this? ID SSN_LAST F_NAME L_NAME PHONE ALT_PHONES 1 “4347” “Bob” “Smith” “4155164347” “4158983787, 4156563987, 4159878787” 2 “9489” “Mary” “Jones” “6502132333” “6503134421, 6509872736” 3 “9898” “David” “Bass” “6504143451” “6503134421, 6509876228, 6502334998” 4 … and felt guilty… but convinced yourself you won’t need to search on this column anyway…
19. Forget big data (for the moment) , this is a data structure thing {
 _id: 1234,
 author: { name: “Bob Davis”, email : b@d.com },
 post: “In these troubled times I like to …“,
 date: { $date: “2010-07-12 13:23UTC” },
 location: [ -121.2322, 42.1223222 ],
 rating: 2.2, 
 comments: [
 { user: “jgs32@hotmail.com”,
 upVotes: 22,
 downVotes: 14, 
 " text: “Great point! I agree” },
 { user: “holly.davidson@gmail.com”,
 upVotes: 421,
 downVotes: 22, 
 " text: “You are a moron” } 
 ],
 tags: [ “Politics”, “Virginia” ]
 }"
20. Contrast with the relational model for blog post really the best way to persist this?
21. {
 _id: 1234,
 author: { name: “Bob Davis”, email : b@d.com },
 post: “In these troubled times I like to …“,
 date: { $date: “2010-07-12 13:23UTC” },
 location: [ -121.2322, 42.1223222 ],
 rating: 2.2, 
 comments: [
 { user: “jgs32@hotmail.com”,
 upVotes: 22,
 downVotes: 14, 
 " text: “Great point! I agree” },
 { user: “holly.davidson@gmail.com”,
 upVotes: 421,
 downVotes: 22, 
 " text: “You are a moron” } 
 ],
 tags: [ “Politics”, “Virginia” ]
 }" This is involves less brain damage…
22. MongoDB is great for APIs •  JSON-based document (object) database •  Flexible query language •  Atomic updates •  Indexes •  Geospatial awareness (geo apis are cool!) •  Ok, and it scales…. –  Replica-set clusters –  Auto-sharding –  ** BUT THIS IS NOT THE TOPIC OF THE TALK **
23. MongoDB capped collections •  Allows for tailable cursors –  block when out of data, instead of issuing an exception •  Fixed in size (‘capped’) •  Very high performance •  Maintain insertion order •  Not shardable
24. Sample document: ‘array’
25. Sample document ‘complex’
26. Server side
27. Client side
28. Questions?
29. More info •  Nodestack –  http://nodestack.org •  MongoDB / 10gen –  http://mongodb.org –  http://10gen.com •  Node.js / Joyent –  http://nodejs.org –  http://joyent.com •  MongoLab –  http://mongolab.com –  will@mongolab.com •  Tractorpush Code –  Server: •  https://github.com/mongolab/ tractorpush-server –  Writer •  https://github.com/mongolab/ tractorpush-writer-ruby –  Doc •  https://devcenter.heroku.com/ articles/realtime-polyglot-app- node-ruby-mongodb-socketio –  Presentation •  https://speakerdeck.com/ benwen/jseverywhere2012
30. Thank You Ben Wen - VP MongoLab - ben@mongolab.com - @benwen - http://linkedin.com/in/benwen JS.Everywhere() 10.24.2012