An Introduction to the node.js Mongo Driver - Christian Kvalheim

MongoUK 2011

Transcript

1. An introduction to the node.js mongo driver Christian Amor Kvalheim Tuesday, October 11, 2011
2. • I Work at Xing, based in Barcelona • Author of the node.js mongodb driver • find me on • web: http://christiankvalheim.com • twitter: christkv • github: christkv • email: christkv@gmail.com Who am I Tuesday, October 11, 2011
3. • Basic NodeJS info • A basic CRUD application using node.js + express + mongodb • Interesting parts of the driver What’s for dinner https://github.com/christkv/mongodb-presentation Code ON Tuesday, October 11, 2011
4. • High Concurrency low latency • Evented I/O • Asynchronous programming platform • Single-threaded, no thread and context swaps • A really fast javascript platform + Tuesday, October 11, 2011
5. • Fantastic combination • Both do Javascript • Both do JSON Natively • Both are asynchronous • Modern web is Javascript + JSON • Avoid mental context switching + Tuesday, October 11, 2011
6. • Build • realtime web apps • high throughput and concurrency apps • analytics/chat/cms you name it • Scale your application with less hardware and resources + Tuesday, October 11, 2011
7. Tuesday, October 11, 2011
8. How to get started Tuesday, October 11, 2011
9. How to get started • Install node.js (from source or package) Tuesday, October 11, 2011
10. How to get started • Install node.js (from source or package) • Install npm (node package manager) Tuesday, October 11, 2011
11. How to get started • Install node.js (from source or package) • Install npm (node package manager) • Create an empty directory Tuesday, October 11, 2011
12. How to get started • Install node.js (from source or package) • Install npm (node package manager) • Create an empty directory • npm install mongodb Tuesday, October 11, 2011
13. How to get started • Install node.js (from source or package) • Install npm (node package manager) • Create an empty directory • npm install mongodb • npm install express Tuesday, October 11, 2011
14. How to get started • Install node.js (from source or package) • Install npm (node package manager) • Create an empty directory • npm install mongodb • npm install express • npm install jade Tuesday, October 11, 2011
15. var express = require('express'), Db = require('mongodb').Db, Server = require('mongodb').Server, Connection = require('mongodb').Connection; var host = 'localhost'; var port = Connection.DEFAULT_PORT; var db = new Db('node-mongo-examples', new Server(host, port, {}), {native_parser:false}); var app = express.createServer(); app.get('/', function(req, res){ res.send('Hello World'); }); db.open(function(err, db) { if(err) throw err app.listen(8124); }); Tuesday, October 11, 2011
16. POST // Create method app.post('/location', function(req, res) { geoCodeDecorateObject(req.body.address, {description:req.body.description}, function(err, object) { db.collection('locations', function(err, collection) { // Insert doc collection.insert(object, {safe:true}, function(err, result) { // Fetch all docs for rendering of list collection.find({}).toArray(function(err, items) { res.render('./basic.jade', {locals: {locations:items}}); }) }); }); }); }); Tuesday, October 11, 2011
17. Safe or not • Mongo Insert/Update/Delete are async • 2nd call to lastError required to check for the success of the operation • safe option ensures the second error call • you can also run the driver in strict mode Tuesday, October 11, 2011
18. DELETE // Delete method app.del('/location', function(req, res) { var id = ObjectID.createFromHexString(req.body.id); db.collection('locations', function(err, collection) { collection.remove({_id:id}, {safe:true}, function(err, numberOfDeletedRecords) { // Fetch all docs for rendering of list collection.find({}).toArray(function(err, items) { res.render('./basic.jade', {locals: {locations:items}}); }) }) }); }); Tuesday, October 11, 2011
19. GET // Get method app.get('/location', function(req, res) { var id = ObjectID.createFromHexString(req.body.id); db.collection('locations', function(err, collection) { collection.findOne({_id:id}, function(err, item) { // Fetch all docs for rendering of list collection.find({}).toArray(function(err, items) { res.render('./basic.jade', {locals: {locations:items, location:item}}); }) }) }); }); Tuesday, October 11, 2011
20. PUT // Update method app.put('/location', function(req, res) { var id = ObjectID.createFromHexString(req.body.id); db.collection('locations', function(err, collection) { collection.findOne({_id:id}, function(err, object) { object.description = req.body.description; object.address = req.body.address; geoCodeDecorateObject(req.body.address, object, function(err, object) { collection.update({_id:object._id}, object, {safe:true}, function(err, numberOfUpdatedObjects) { // Fetch all docs for rendering of list collection.find({}).toArray(function(err, items) { res.render('./basic.jade', {locals: {locations:items}}); }) }) }) }); }); }); Tuesday, October 11, 2011
21. Geo Encoding & Search Tuesday, October 11, 2011
22. Adding an index db.open(function(err, db) { if(err) throw err // !!! CHANGE db.ensureIndex("locations", {loc:"2d"}, function(err, result) { if(err) throw err app.listen(8124); }); }); Tuesday, October 11, 2011
23. var geoCodeDecorateObject = function(address, object, callback) { var googleGeoCodeApi = {host: 'maps.googleapis.com', port: 80,path: '/maps/api/geocode/json?sensor=false&address=' + escape(address), method: 'GET' }; var clientReq = http.get(googleGeoCodeApi, function(clientRes) { var data = []; clientRes.on('data', function(chunk) { data.push(chunk.toString()); }); clientRes.on('end', function() { var googleObject = JSON.parse(data.join('')); object.address = address; object.geodata = googleObject.results.pop(); // !!! CHANGE object.loc = {long:object.geodata.geometry.location.lng, lat:object.geodata.geometry.location.lat}; callback(null, object); }); }); clientReq.end(); } Tuesday, October 11, 2011
24. Tuesday, October 11, 2011
25. SEARCH // Search option app.post('/search', function(req, res) { geoCodeDecorateObject(req.body.address, {}, function(err, object) { // Unpack geo object var long = object.geodata.geometry.location.lng; var lat = object.geodata.geometry.location.lat db.collection('locations', function(err, collection) { collection.find({loc : {'$near': [long, lat], '$maxDistance': parseFloat(req.body.distance)/69}}).toArray(function(err, geoItems) { // Fetch all docs for rendering of list collection.find({}).toArray(function(err, items) { res.render('./basic_6.jade', {locals: {locations:items, results:geoItems}}); }) }); }); }); }); Tuesday, October 11, 2011
26. // Search option app.post('/search', function(req, res) { geoCodeDecorateObject(req.body.address, {}, function(err, object) { // Unpack geo object var long = object.geodata.geometry.location.lng; var lat = object.geodata.geometry.location.lat db.collection('locations', function(err, collection) { collection.find({loc : {'$near': [long, lat], '$maxDistance': parseFloat(req.body.distance)/69}}).toArray(function(err, geoItems) { // Fetch all docs for rendering of list collection.find({}).toArray(function(err, items) { res.render('./basic_6.jade', {locals: {locations:items, results:geoItems}}); }) }); }); }); }); Picking it apart Tuesday, October 11, 2011
27. Other features Tuesday, October 11, 2011
28. Other features •Cursors •Grid FS Tuesday, October 11, 2011
29. Cursors Tuesday, October 11, 2011
30. Cursors • toArray Function • each Function • streamRecords Function Tuesday, October 11, 2011
31. toArray collection.find().toArray(function(err, documents){ test.deepEqual([1, 2, 3], documents[0].b); // Let's close the db test.done(); }); Tuesday, October 11, 2011
32. each collection.find({}, {'sort':[['age', 1]]}).each(function(err, item){ if(item != null) { // Do work } else { // Finished } }); Tuesday, October 11, 2011
33. streamRecords var stream = collection.find({}, {'limit' :3}).streamRecords(); stream.on('end', function() { // No more results in the pipe }); stream.on('data',function(data){ // Item }); Tuesday, October 11, 2011
34. streamRecords var stream = collection.find({}, {'limit' :3}).streamRecords(); stream.on('end', function() { // No more results in the pipe }); stream.on('data',function(data){ // Item }); PREFERRED Tuesday, October 11, 2011
35. Grid FS • Write and read a file • Stream a file Tuesday, October 11, 2011
36. Write a file var gridStore = new GridStore(client, 'test_gs_writing_file', 'w'); gridStore.open(function(err, gridStore) { gridStore.writeFile('./test_gs_weird_bug.png', function(err, fileObject) { }); }); GridStore.read(client, 'test_gs_writing_file', function(err, fileData) { }); }); Tuesday, October 11, 2011
37. Stream a file var gridStore = new GridStore(client, "test_gs_read_stream", "r"); gridStore.open(function(err, gs) { var stream = gs.stream(true); stream.on("data", function(chunk) { // Received a chunk of data }); stream.on("end", function() { // Finished streaming }); }); Tuesday, October 11, 2011
38. Stream a file var gridStore = new GridStore(client, "test_gs_read_stream", "r"); gridStore.open(function(err, gs) { var stream = gs.stream(true); stream.on("data", function(chunk) { // Received a chunk of data }); stream.on("end", function() { // Finished streaming }); }); PREFERRED Tuesday, October 11, 2011
39. Tuesday, October 11, 2011
40. Typical starting problems for(var i = 0; i < 100; i++) { collection.insert({'i':i}); } collection.count(function(err, count) { // count is > 0 <= 100 }); Tuesday, October 11, 2011
41. Typical starting problems for(var i = 0; i < 100; i++) { collection.insert({'i':i}, {safe:true}, function(err, result) { // Do something }); } ?????????????????????????????? collection.count(function(err, count) { // count is > 0 <= 100 }); Tuesday, October 11, 2011
42. Typical starting problems for(var i = 0; i < 100; i++) { collection.insert({'i':i}, {safe:true}, function(err, result) { // Do something }); } ?????????????????????????????? collection.count(function(err, count) { // count is > 0 <= 100 }); Many possible solutions Tuesday, October 11, 2011
43. My Chosen Solution Step( function insert() { var group = this.group(); for(var i = 0; i < 100; i++) { collection.insert({'i':i}, {safe:true}, group()); } }, function finished() { collection.count(function(err, count) { // Count is 100 :) }); } ) https://github.com/creationix/step Tuesday, October 11, 2011
44. Overview of driver • Supports nearly all of the Mongo DB features • GridFS • Replicasets for scaling up • Support for MongoDB 2.0 • JS/C++ BSON Parser Tuesday, October 11, 2011
45. What’s coming • Full support for tags in replicasets • New BSON C++ BSON parser • Raw mode • New faster socket handler code • JS Instance serialization (EXPERIMENTAL) Tuesday, October 11, 2011
46. Raw Mode • Passthrough raw BSON objects to web client (avoid overhead of parsing) • BSON JS parser will be reworked to work in browser • For analytics or other situations where you don’t do “post-processing” on server before the client Tuesday, October 11, 2011
47. JS Instance Serialization (EXPERIMENTAL) var Car = function(name) { this.name = name; } Car.prototype.ack = function(callback) { var http = require('http'); // callback to main service http.get({host:'api.carservice.com', port: 80, path: '/car/ack'}, function(res) { callback(null, res); }) } Tuesday, October 11, 2011
48. What have we learnt • How to connect to a Mongodb server • How to write a simple CRUD location app in node.js with Mongodb • Other cool parts of the driver • How to avoid the first pitfall of using nodeJS Tuesday, October 11, 2011
49. DONE Q&A Code on https://github.com/christkv/mongodb-presentation Tuesday, October 11, 2011