Writing Real Programs … with JavaScript!? An introduction to Node.JS Andrew Montalenti

Act I, Functional JavaScript Act II, Meet Node.JS

Act I, JavaScript: the beast with two backs • JS is a cool programming language! – Functional – Weakly and dynamically typed – Function scope and closure – Has a >99% desktop install base (no kidding!) • JS is not a real programming language! – No compilation units, modules and namespaces – No classes or object-orientation built-in – No standard library (no kidding!) – Incoherent implementation standard

JS boring details in one slide • Number: double-precision 64-bit IEEE 754, e.g. 0 or 0.5 • String: stream of 16-bit Unicode characters, e.g. “hello, world!” • Boolean: true or false • object: {a: b} - like Python dictionary + attribute (dot notation) syntax • Function: function() {} • Array: [a, b] – implemented as hash, includes length property • Date: supports basic date/time representation and parsing from strings • RegExp: created using /regex/ syntax; efficient RegEx impl • Null: null - deliberate non-value • Undefined: undefined - uninitialized value no I/O – object literals – common C-style if, for, switch/case, while statements – uncommon C-style do/while, tertiary statement, and short-circuiting – var keyword – string concatenation and splicing –

What makes JavaScript cool? Our first example is a map/reduce implementation written with JS functions. 1. we build an iterator function called each which is found in most JS frameworks 2. we implement map and reduce using each 3. we implement a test function that maps pow2 function (x*x) to each element in a list, and then reduces it via addition. Equivalent to the following Python: sum(x*x for x in items)

The iterator function function each(items, fn) { for (var i = 0; i < items.length; i++) { var item = items[i]; fn.apply(item, []); } };

The mapper function map(fn, items) { var results = []; each(items, function() { var val = fn.apply(this, []); results.push(val) }); return results; };

The reducer function reduce(items, fn) { var result; var first = true; var that = this; each(items, function() { if (first) { result = this; first = false; return; // continues } result = fn.apply(that, [result, this]); }); return result; };

Functional programming in JS function test_map_reduce() { var adder = function(i1, i2) { return i1 + i2; }; var pow2 = function() { return this * this; }; var input = [1, 2, 3, 4, 5, 6]; console.log("input: " + input); var mapped = map(pow2, input); console.log("mapped: " + mapped); var reduced = reduce(mapped, adder); console.log("reduced: " + reduced); };

It works! >>> test_map_reduce(); input: 1,2,3,4,5,6 mapped: 1,4,9,16,25,36 reduced: 91

Grokking Functional JS Functions are first-class objects, but they have some quirks. Three declaration forms: • function statements • anonymous function expressions • named function expressions (exotic) The keyword this is used for functional calling context, controlled by fn.apply()

Declaration Forms function add1(i1, i2) { return i1 + i2; } var add2 = function(i1, i2) { return i1 + i2; } var add3 = function add3(i1, i2) { return i1 + i2; }

Calling context function each(items, fn) { for (var i = 0; i < items.length; i++) { var item = items[i]; fn.apply(item, []); } } each([0, 1], function() { console.log(this); }); function context this fn.apply( item , []);

Closure: obvious and complex Because JavaScript can treat function definitions as expressions, they can be used almost anywhere in your program. Common locations • Nested within other functions • As values in object literals • As arguments to other functions Closure allows functions to access the values of variables outside the scope of the function definition, even after that variable has gone out of scope.

Common Function Locations function nested_argument() { var arr = []; each(results, function() { // do something with arr }); } function obj_literal() { var obj = {}; return { "do": function() { /* do something with obj */ } }; }

anonymous function Closure function map(fn, items) { var results = []; each(items, function() { var val = fn.apply(this, []); results.push(val) }); return results; } “closes over” outer variables [1, 2, 3] results results

Closure Stack function stack() { var stack = []; var pusher = function(item) { stack.push(item); }; var popper = function() { return stack.pop(); }; var peeker = function() { return stack[stack.length-1]; } var obj = { "push": pusher, "pop": popper, "peek": peeker }; return obj; // upon return, stack not in scope } Each is an example of closure

OO JS? >>> var s = stack() Object >>> s.push(0) >>> s.peek() 0 >>> s.peek() 0 >>> s.push(1) >>> s.peek() 1 >>> s.pop() 1 >>> s.pop() 0 >>> s.pop() >>> s.pop()

Closure Book

What hath we wrought?!

JavaScript OO vs. Java OO “Java is to JavaScript what Car is to Carpet.” - Chris Heilmann

Classic Stack function Stack() { var stack = []; this.push = function(item) { stack.push(item); }; this.pop = function() { return stack.pop(); }; this.peek = function() { return stack[stack.length-1]; }; };

Classic Stack, improved var Stack = function() { this.stack = []; }; Stack.prototype.push = function(item) { this.stack.push(item); }; Stack.prototype.pop = function() { return this.stack.pop(); }; Stack.prototype.peek = function() { return this.stack[stack.length-1]; };

Objects and calling context call_method this obj.call_method(); call_method this var fn = obj.call_method; fn();

The new keyword >>> var s = new Stack() Object >>> s.push(0) >>> s.peek() 0 >>> s.peek() 0 >>> s.push(1) >>> s.peek() 1 >>> s.pop() 1 >>> s.pop() 0 >>> s.pop() >>> s.pop() What the heck is the new keyword doing?

The new keyword >>> var s = {}; >>> Stack.apply(s, []); >>> s.peek() 0 >>> s.peek() 0 >>> s.push(1) >>> s.peek() 1 >>> s.pop() 1 >>> s.pop() 0 >>> s.pop() >>> s.pop() Yep, that’s all.

Spot the bug >>> var s = Stack(); >>> s.push(0); TypeError: s is undefined . . . source=with(_FirebugCommandLine) . . . {s.push(0)\n}; >>> window.push(0); >>> window.pop(); 0 Ugh!

Now, for the bad parts 1. In JavaScript, variables are global by default. 2. The keyword this is bound to “the global object” by default, which in browsers is usually window. 3. Since functions meant to be used with the new keyword can be called without the new keyword, functions that rely on new/this for OO programming are often accidentally misused. 4. Since this changes based on calling context AND “normal” fn() invocation does not alter calling context, this is a regular source of bugs.

What can we do? – Define only a SINGLE GLOBAL for your application, which acts as a namespace – Be highly COGNIZANT OF this, and DOCUMENT when you rely on it – Name all functions meant to be used with new with CamelCase, e.g. Stack – Use the MODULE PATTERN and CLOSURE to organize your code – Follow OO patterns and use a FRAMEWORK – Use jslint to check our source code

Module Pattern var NS = function () { // create namespace // private module data var VERSION = “1.2”; // public module classes, functions, and variables var m = {}; m.DateParser = function() { this.parse = function() {}; }; m.parse_date = function(s) { var dp = new m.DateParser(); return dp.parse(s); }; return m; }(); // invoke function to hide private data

Using a module >>> NS.parse_date(“2009-09-01”); DateParser(year=2009, month=09, day=01) >>> new NS.DateParser().parse(“2009-09-01”); DateParser(year=2009, month=09, day=01)

No Standard Library • JavaScript desperately needs a standard library • Heavy development is underway to achieve this: – chironjs / narwhal – CommonJS – ECMAScript standard – For server-side, Node.JS

My pragmatic client-side JS stack These are libraries I use every day for real programming with JS: – jQuery: CSS-style DOM selector, animations, event binding, HTTP requests – Sometimes, ExtJS: Observable, ISO8601 DateTime, OO, XTemplate, Store, DomHelper, JSON, Forms, TaskRunner, and a slew of extensible UI components – Underscore.js: functional primitives – DateJS: extended date parsing and manipulation – BlackbirdJS: cross-browser client-side logging – Bootstrap (new!): standard UI components for HTML5

My Pragmatic Dev Env These are the tools I use on a daily basis for doing JS prototyping and development: – Firebug and Live HTTP Headers – Webstorm – node, especially node-static

ACT I, Fin This ends our look at the JavaScript language. Now, on to Node.JS.

Useful Resources My favorite JavaScript reference documents: http://delicious.com/amontalenti/my:jsref Other questions? @amontalenti

No content

No content

Act II, Node.JS

No content

Why Server-Side JS (SSJS)? • The Grand Unified Stack: JavaScript front to back, baby. • The rise of Rich Internet Applications (RIAs) has led to a smaller role for servers anyway. • Re-use between client/server (e.g. form validation logic, utilities, object models) may de-duplicate code • Single syntax may reduce programmer context switching

Why SSJS has failed thus far? • Netscape LiveWire – Tied to a commercial vendor – JavaScript was not even adopted yet • Rhino – Open source – No community (identity crisis) – Poor performance – JavaScript was not cool yet

Node.JS: right place, right time • And, the right tool. • Node.JS – Open source – Library to build HTTP servers – Riding asynchronous wave (epoll) – Riding realtime wave (COMET, Websockets) – JavaScript finally cool, thanks to AJAX/RIAs – Strong community, and growing

Revenge of JS • “Despite JavaScript's astonishing shortcomings, deep down, in its core, it got something very right. • JavaScript was the world's most misunderstood programming language. • Its obvious defects, its unfashionable programming model, intentional mispositioning at its introduction, and its ridiculous name caused it to be rejected as unworthy by most knowledgeable programmers. But Ajax gave JavaScript a second chance.

It’s better to be lucky • Because JavaScript is the language of the web browser, and because the web browser has become the dominant application delivery system, and because JavaScript isn't too bad, JavaScript has become the World's Most Popular Programming Language. Its popularity is growing. It is now being embedded in other applications and contexts. JavaScript has become important. • It is better to be lucky than smart.” – Douglas Crockford

Does coolness matter?

What Node.JS isn’t… • Django/Rails – Pieces of Django/Rails are implemented by npm modules, but no unified “web stack” yet • Rhino/Spidermonkey – Node.JS is one layer up from these (think Python/Ruby standard library, not the interpreter itself). – But… Node.JS uses a different interpreter, too.

What Node.JS is… • A standard library for JavaScript • An asynchronous server library (epoll) – Twisted – Tornado – EventMachine • Because programmable async servers were the reason Node.JS was written, its standard library has an async bent.

Node.JS Architecture

OK, but what is Node.JS, really? • It’s clear devs want it to become as powerful as Django/Rails for web app development – express module is starting to provide routing and controllers – mongodb and redis are popular data stores for Node – Template languages are ported every day

The promise: Node can simplify the web stack • Simplify the backend with JSON-based, async write/read data stores • Simplify the middle tier with clientside / serverside code sharing. • Simplify the frontend, with server-backed data bindings • Simplify deployment, since Node.JS runs directly in production.

Aside: my intro to epoll “The C10K Problem”

YAGNI, right?

Node.JS is programmable nginx

I/O is our biggest bottleneck

HTTP Hello, World var http = require('http'); var server = http.createServer(function(req, rsp) { rsp.writeHead(200, {"Content-Type": "text/plain"}); rsp.end("Hello World\n"); }); server.listen(8000); console.log("Server running at :8000");

Express Hello, World var app = require('express').createServer() app.get('/', function(req, rsp) { rsp.writeHead(200, {"Content-Type": "text/plain"}); rsp.end("Hello World\n"); }); app.listen(8000); console.log("Server running at :8000");

UNIX Hello, World var fs = require('fs'); var lazy = require('lazy'); lazy(fs.createReadStream("/usr/share/dict/words")) .lines .forEach(function(line) { var line = line.toString(); if (line.indexOf("hello") !== -1 || line.indexOf("world") !== -1) { console.log(line); } });

Client/Server Hello, World (1) var html = require('fs’).readFileSync(…), srv = require('http’).createServer(function(req, res){ res.end(html); }); server.listen(8080); var nowjs = require("now"); var everyone = nowjs.initialize(server); everyone.now.distributeMessage = function(message){ everyone.now.receiveMessage(this.now.name, message); };

Client/Server Hello, World (2) $(document).ready(function(){ now.receiveMessage = function(name, message){ $("#messages") .append("
" + name + ": " + message); } $("#send-button").click(function(){ now.distributeMessage("Hello, world"); }); now.name = prompt("What's your name?", ""); });

Notice a trend? • Node.JS relies heavily on what you might call the Hollywood Principle • “Don’t call us, we’ll call you” • Node.JS achieves this through a heavy use of callbacks and errbacks • Luckily, we now know Functional JS!

Why callbacks? • In Node.JS, everything runs in parallel, except your code. • Let’s say you register two callbacks in Express. – Node.JS registers those callbacks, and goes to sleep. – Request comes in, dispatches it to callback – meanwhile, any other requests wait in line. – As long as your callback doesn’t block, performance is guaranteed.

OK, let’s break it • Let’s see what actually happens if one of our callbacks takes a long time to respond.

Broken World var app = require('express').createServer() var i = 1; app.get('/', function(req, rsp) { if (i === 3) { console.log("spin needlessly"); while (1) {} } rsp.writeHead(200, {"Content-Type": "text/plain"}); rsp.end("Hello World " + i + "\n"); i++; }); app.listen(8000); console.log("Server running at :8000");

Enough Hello, World. How about Real World? • There is a Node.JS sweet spot right now. • Real-time Analytics – Low latency, high concurrency HTTP/UDP – JSON is a natural fit for data exchange – Client-side “pixel” can be synced w/ server- side pixel parser – Websockets natural fit for viewing live data – MongoDB a natural fit for data store • We’ll look at three examples.

No content

No content

StatsD

Parse.ly real-world use case • Parse.ly uses Node.JS for what we call “dynamic JavaScript configuration” • Client-side JavaScript code makes a JSON-P call to a “config server”, which returns JavaScript configuration settings • Allows us to load custom JS for individual publishers, or even individuals • Run A/B tests, do sampling, etc. • It’s perfect for this (esp. with Redis)

My eye is also on Node.IO • Web crawling framework built with Node • Borrows from jQuery for selectors • Best possible DOM comprehension possible with JavaScript – No more “real world HTML” and “valid XHTML” mismatch problems • Evented system fits nicely. I use Scrapy in the Python world for this now.

More use cases born every day • Chat applications • Collaboration tools • JSON API servers • Telephony systems

ACT II, Fin • That’s it, a whirlwind tour through old and weary functional JavaScript and its ambitious younger brother, Node.JS. • To follow me: – @amontalenti – gplus.to/amontalenti – linkedin.com/in/andrewmontalenti – andrew@parsely.com – http://pixelmonkey.org