Building a Mobile Backend API with Node.js: Better Than Expected

In late 2011, OmniTI began working on a project to build a large scale, mobile driven service that would provide loyalty programs for TV viewing. Our primary role in that process was focusing on running the web API’s that would glue together all of the backend infrastructure. While it was still unproven (to us) at the time, we decided to pick Node.js for this project; we subsequently built—and now manage—that service. So, what do you learn when you throw a million users at a new technology just to see how it works?

Co-presented with @bdunavant at Surge 2013.

Transcript

1. Building a Mobile Backend API with Node.js Better Than Expected or My Name is Brian and I use NodeJS Saturday, September 14, 13
2. Who is OmniTI? OmniTI is the leading Web Scalability and Performance provider utilizing a cross-disciplinary, "devops style" approach. • Broad Solution Stack (OS, Database, Web Design and Development) • Solution provider to 10% of the Alexa Top 100 • Do “Big Internet” stuff Saturday, September 14, 13
3. Brian Bickerton Brian Dunavant @Tau_Zero @bdunavant Saturday, September 14, 13
4. Business Model http://www.digital-literacies.com/wp-content/uploads/2011/09/couch-potato-cat.jpeg Saturday, September 14, 13
5. Service Oriented Arch • Highly Scalable • Fault Tolerant • Graceful degradation Saturday, September 14, 13
6. JSend • Open standard • Simple. Light-weight. • Everyone knows what to expect. http://labs.omniti.com/labs/jsend { status : "success", data : { "post" : { "id" : 1, "title" : "A blog post", "body" : "Some useful content" } } } Saturday, September 14, 13
7. Basic Backend services • Audio Matching = C • Search = Solr • User Storage = Riak • Analytical Storage = Postgres • Data Warehouse = Hadoop • RTA = Esper • Business Ledger = Java Saturday, September 14, 13
8. Saturday, September 14, 13
9. One API to query them all Take the responses and bind them One API to rule them all One API to find them http://www.hi-wallpapers.com/uploads/image/201301/12/1357942144.jpg Saturday, September 14, 13
10. ? Saturday, September 14, 13
11. ? ಠ_ಠ Saturday, September 14, 13
12. • April, 2011 • Node was about v0.4.6 - still very young • Controversial - “Node.js is Cancer” Saturday, September 14, 13
13. Why not node? • It’s not C • Single-threaded • Barrier of entry to JS as server code • It was still very young • Very few debugging tools at the time • dtrace • console.log() • Confusing stack traces Saturday, September 14, 13
14. Saturday, September 14, 13
15. Okay, so why then? • Expressive - Fast to develop/modify • Almost everyone knows some JS • Single threaded forces some scalable patterns • Smaller technology stack • Fast fixing prod errors. No compiling. Saturday, September 14, 13
16. http://24.media.tumblr.com/tumblr_lmtqdlwT0P1qiib0qo1_500.gif vs http://gifs.gifbin.com/092011/1317747782_badmington_juggling__10_shuttlecocks.gif Synchronous IO Asynchronous IO Saturday, September 14, 13
17. Total Packages: 39 323 4 242 889 downloads in the last day 24 669 137 downloads in the last week 83 507 007 downloads in the last month APNS Saturday, September 14, 13
18. Of course, there were bumps along the way Saturday, September 14, 13
19. node-uuid Time: 7.354733784s Collisions in prod libuuid Time: 0.652528982s No collisions in prod Iterations: 1,000,000 UUID Saturday, September 14, 13
20. node-postgres a lot better after some patching http://www.globalanimal.org/wp-content/uploads/2011/11/Elephant-gets-prosthetic-leg-400x295.jpg Saturday, September 14, 13
21. • riak-js • sailthru-node-client • node-mailer • node-c2dm • statsd http://thesouthinmymouth.files.wordpress.com/2010/03/dscn1312.jpg Saturday, September 14, 13
22. Timeouts are easy… right? Saturday, September 14, 13
23. ClientRequest.prototype.setTimeout = function() { this._deferToConnect('setTimeout', arguments); }; v0.6.10 Saturday, September 14, 13
24. ClientRequest.prototype.setTimeout = function(msecs, callback) { if (callback) this.once('timeout', callback); var self = this; function emitTimeout() { self.emit('timeout'); self.destroy(new Error('timeout')); } if (this.socket && this.socket.writable) { this.socket.setTimeout(msecs, emitTimeout); return; } if (this.socket) { this.socket.once('connect', function() { this.setTimeout(msecs, emitTimeout); }); return; } this.once('socket', function(sock) { this.setTimeout(msecs, emitTimeout); }); }; v0.6.17 Saturday, September 14, 13
25. Current ClientRequest.prototype.setTimeout = function(msecs, callback) { if (callback) this.once('timeout', callback); var self = this; function emitTimeout() { self.emit('timeout'); } if (this.socket && this.socket.writable) { if (this.timeoutCb) this.socket.setTimeout(0, this.timeoutCb); this.timeoutCb = emitTimeout; this.socket.setTimeout(msecs, emitTimeout); return; } // Set timeoutCb so that it'll get cleaned up on request end this.timeoutCb = emitTimeout; if (this.socket) { var sock = this.socket; this.socket.once('connect', function() { sock.setTimeout(msecs, emitTimeout); }); return; } this.once('socket', function(sock) { sock.setTimeout(msecs, emitTimeout); }); }; Saturday, September 14, 13
26. HTTP Agent • Agent is node’s way of connection pooling for hosts • Use a separate agent for each backend. • This is critical if different backends are on the same IP/host. • Allows custom settings for each backend pool • Don’t use core agent if you want to actually keep-alive a socket. Saturday, September 14, 13
27. Package Management { "name": "pg", "version": "2.6.1", "description": "PostgreSQL client - pure javascript & libpq with the same API", "keywords": ["postgres","pg","libpq","postgre","database","rdbms"], "homepage": "http://github.com/brianc/node-postgres", "repository": { "type": "git", "url": "git://github.com/brianc/node-postgres.git" }, "author": "Brian Carlson <brian.m.carlson@gmail.com>", "main": "./lib", "dependencies": { "generic-pool": "2.0.3", "buffer-writer": "1.0.0" }, "devDependencies": { "jshint": "1.1.0", "semver": "~1.1.4" }, "scripts": { "test": "make test-travis connectionString=postgres://postgres@localhost:5432/ postgres", "install": "node-gyp rebuild || (exit 0)" }, "engines": { "node": ">= 0.8.0" } } Saturday, September 14, 13
28. Why to Shrinkwrap Installed on January 1 node_modules |-- A @ 2.5.0 (installs 2.5.0) | `-- C @ >= 1.0.0 (installs 1.0.1) `-- B @ 0.1.2 (installs 0.1.2) |-- D @ * (installs 2.4.1) `-- E @ 1.5.1 (installs 1.5.1) Installed on July 1 node_modules |-- A @ 2.5.0 (installs 2.5.0) | `-- C @ >= 1.0.0 (installs 2.1.1) `-- B @ 0.1.2 (installs 0.1.2) |-- D @ * (installs 3.0.7) `-- E @ 1.5.1 (installs 1.5.1) Saturday, September 14, 13
29. Load balancing • Several ways to do it in node.js • Early days you had modules like fugue • Now core has Clustering • All of this is silly in a production environment • There are 65,535 ports, why only use one? Saturday, September 14, 13
30. Namespacing • Every process can handle any request • Single bug could take down every process • Functional groupings of endpoints in load balancer traffic groups • No two groupings completely overlap the same machines Saturday, September 14, 13
31. No Downtime Deploys • Checkout staging code, start up a process • Run test suite against to verify no errors • Merge to production branch, Tag • Rolling deploy Saturday, September 14, 13
32. • You don’t know if it works unless it’s being properly monitored • In a Service Based Arch, there’s lots of things to go wrong • Properly assigned blame praise Deep Monitoring Saturday, September 14, 13
33. Saturday, September 14, 13
34. Saturday, September 14, 13
35. http://www.lipstiq.com/wp-content/uploads/2013/07/minions-film.jpg Monitoring all the minions Saturday, September 14, 13
36. Results • Over 3M registered users • 100% Public API uptime since launch • Over 4x capacity more than highest peaks seen Saturday, September 14, 13
37. Thank You! Questions? Saturday, September 14, 13