Data In Your Browser?! Data In Your Browser?! Tuesday, October 9, 12

All rights reserved to Pete Karl Irene Ros @ireneros, irene@bocoup.com http://bocoup.com | http://ireneros.com | http://github.com/iros Irene Ros @ireneros, irene@bocoup.com http://bocoup.com | http://ireneros.com | http://github.com/iros Tuesday, October 9, 12

Bocoup | http://bocoup.com What do you do with your data? Create Read Update Delete or “why would you be manipulating your data to begin with” Tuesday, October 9, 12

Bocoup | http://bocoup.com What do you do with your data? Create Read Update Delete Filtering Computation or “why would you be manipulating your data to begin with” Tuesday, October 9, 12

Bocoup | http://bocoup.com What do you do with your data? Create Read Update Delete Filtering Computation or “why would you be manipulating your data to begin with” AWESOME ZONE Tuesday, October 9, 12

Why Client-Side?! Why Client-Side?! Yey, Browsers! They can actually Handle it! Less Load on the Server: More free resources not taken up by computation and data crunching Less Caching? Less storage When computations are user- specific Tuesday, October 9, 12

How do we treat data? Rows of records Good for iteration (doing something with each record) Bad for figuring out something about a specific property Tuesday, October 9, 12

var data = [ {"name":"Abe Sapien","type":"hero","age":57}, {"name":"Abin Sur","type":"hero","age":58}, {"name":"Adam Monroe","type":"hero","age":59}, {"name":"Adam Strange","type":"hero","age":60}, {"name":"Agent 13","type":"hero","age":61} ]; Tuesday, October 9, 12

How do we treat data? Column-wise (table-ish) Good for figuring out something about a specific property Bad for iteration (doing something with each record) Tuesday, October 9, 12

var data = [ [ "Abe Sapien", "Abin Sur", "Adam Monroe", "Adam Strange", "Agent 13" ], [ "hero", "hero", "hero", "hero", "hero" ], [ 57, 58, 59, 60, 61 ] ]; Tuesday, October 9, 12

Follow Along! Follow Along! https://github.com/iros/clientsidedata *There are more libraries explored there than this presentation covers. Party on. * * Tuesday, October 9, 12

Follow Along! Follow Along! https://github.com/iros/clientsidedata *There are more libraries explored there than this presentation covers. Party on. * * Tuesday, October 9, 12

{ "name": "Batman", "type": "hero", "id": 1339970999, "intelligence": 1.0, "strength": 0.28, "speed": 0.27, "durability": 0.42, "power": 0.37, "combat": 1.0, "Alternate_Identities": "Batman", "Aliases": "Insider, Matches Malone", "Nicknames": "The Dark Knight, The Caped Crusader, The Worlds Greatest ...", "Place_of_birth": "Crest Hill, Bristol Township; Gotham County", "First_appearance": "Detective Comics #27 (May, 1939)", "Alignment": "good", "Gender": "male", "Race": "", "Height_cm": "188", "Height_ft": "6'2", "Weight_lb": "210", "Weight_km": "95", "Eye_color": "blue", "Hair_color": "black", "Occupation": "Businessman", "Base": "Batcave, Stately Wayne Manor, Gotham City; Hall of Justice, Justice League Watchtower", "Relatives": "Dr. Thomas Wayne (father, deceased), Martha Wayne (mother, deceased), Philip Wayne (uncle, foster father, deceased), Alfred Pennyworth (butler, foster father), Dick Grayson (Nightwing - adopted son), Jason Todd (Robin II - adopted son, deceased), Tim Drake (Robin III - protégé)", "Group_Affiliation": "Batman Family, Batman Incorporated, Justice League, Outsiders, Wayne Enterprises, Club of Heroes", "Friends": "", "Enemies": "", "superpower_Ability_Shift": 0, "superpower_Absorption": 0, "superpower_Accuracy": 1, "superpower_Adaptation": 0, "superpower_Aerokinesis": 0, ... Tuesday, October 9, 12

Heroes Villains data/just_heroes.json data/just_villains.json Tuesday, October 9, 12

Heroes Villains data/just_heroes.json data/just_villains.json data/all.json Tuesday, October 9, 12

Rows of Data! Tuesday, October 9, 12

Bocoup | http://bocoup.com TaffyDB ‣ Database-like (records are json objs) ‣ Rich selection language ‣ Supports saving to LocalStorage ‣ Extensible ‣ No events (for updates, new rows etc.) ‣ “templating” support: var row = products({ item : 2 }) .supplant("" + "{name}" + "{price}" + ""); http://www.taffydb.com Tuesday, October 9, 12

Bocoup | http://bocoup.com // create a database of heroes heroes = TAFFY(heroes); // find all the male & female heroes with a height var male_heroes = heroes({ Gender : "male", type : "hero", Height_cm : { "!is" : "NA" } }), female_heroes = heroes({ Gender : "female", type : "hero", Height_cm : { "!is" : "NA" } }), // find all heroes that are not male and are not female but have a height unknown_gender_heroes = heroes({ type : "hero", Height_cm : { "!is" : "NA" } }, [{ Gender : { "!is" : "male" }}, { Gender : { "!is" : "female" }}] ); Tuesday, October 9, 12

Are Abilities Related?! intelligence power durability speed combat http://jsbin.com/avovem/2/ clientsidedata/code/taffy-db/competencies.html strength Tuesday, October 9, 12

Columns of Data! Tuesday, October 9, 12

Bocoup | http://bocoup.com Crossfilter ‣ Extremely fast ‣ Uses Typed arrays ‣ Doesn’t do so well with incomplete or messy data... http://square.github.com/crossfilter/ Tuesday, October 9, 12

Bocoup | http://bocoup.com // pass the data through the crossfilter var heroes_crossfilter = crossfilter(heroes); // dimension based on intelligence var intelligence_dim = heroes_crossfilter.dimension(function(hero) { return hero.intelligence; }); // find all records that are in the upper quadrant var lower_quad_intelligence_filter = intelligence_dim.filter([0, 0.25]), lower_quad_intelligence = lower_quad_intelligence_filter.top(Infinity); // clear filter intelligence_dim.filterAll(); Tuesday, October 9, 12

Bocoup | http://bocoup.com Typed Arrays - why? ‣ Some operations are faster ‣ Quick manipulation of raw binary data ‣ Initialization to 0 by default ‣ You can share buffers ‣ Different views on the same data ‣ Smaller memory footprint (if you know the dimensions of your data!) ‣ Browser support? http://caniuse.com/#search=typed arrays ‣ Perf tests: http://jsperf.com/typed-arrays-vs-arrays/6 http://jsperf.com/typed-array-iteration Tuesday, October 9, 12

Bocoup | http://bocoup.com var buffer = new ArrayBuffer(5), totalUint8Array = new Uint8Array(buffer); //8-bit unsigned integer totalUint8Array[0] = NaN; totalUint8Array[1] = undefined; totalUint8Array[2] = null; totalUint8Array[3] = “cat”; totalUint8Array[4] = 258; console.log(totalUint8Array); >> [0, 0, 0, 0, 2] Typed Arrays - why not? ‣ NaN, Undefined, Null, strings turn to 0. ‣ Data Surprises http://jsfiddle.net/EXNtv/2/ Tuesday, October 9, 12

Bocoup | http://bocoup.com intelligence power durability speed combat How are competencies Distributed? http://jsbin.com/izefeh/1 clientsidedata/code/crossfilter/visual.html Tuesday, October 9, 12

Bocoup | http://bocoup.com The Miso Project - Dataset This library was written jointly by Bocoup & The Guardian - I am biased. Ping myself or Alex Graul anytime for help. Tuesday, October 9, 12

Bocoup | http://bocoup.com Fetch Parse Compute Filter Derive Tuesday, October 9, 12

Bocoup | http://bocoup.com Miso Dataset ‣ Node.js + Client-side ‣ Built-in importers and parsers for common data sources & structures ‣ Evented Views ‣ Common math functions ‣ Common derived dataset functions http://misoproject.com/dataset @misoprojec on github add remove update Tuesday, October 9, 12

Bocoup | http://bocoup.com Tuesday, October 9, 12

Bocoup | http://bocoup.com // Instantiate a new Miso Dataset object and point it // at our trusty hero database. var heroes = new Miso.Dataset({ url: 'http://localhost:8000/data/just_heroes.json' // or data : yourJSONObjectArray.... }); // fetch data heroes.fetch().then(function() { console.log("Hero #:" + heroes.length); }) ; Tuesday, October 9, 12

Bocoup | http://bocoup.com // Instantiate a new Miso Dataset object and point it // at our trusty hero database. var heroes = new Miso.Dataset({ url: 'http://localhost:8000/data/just_heroes.json' // or data : yourJSONObjectArray.... }); // fetch data heroes.fetch().then(function() { console.log("Hero #:" + heroes.length); }) ; >> Hero #: 570 Tuesday, October 9, 12

Bocoup | http://bocoup.com // Instantiate a new Miso Dataset object and point it // at our trusty hero database. var heroes = new Miso.Dataset({ url: 'http://localhost:8000/data/just_heroes.json' // or data : yourJSONObjectArray.... }); // fetch data heroes.fetch().then(function() { console.log("Hero #:" + heroes.length); }) ; >> Hero #: 570 // Get all the hair color counts // and then sort them by the actual resulting count. var heroes_by_haircolor = heroes .countBy("Hair_color") .sort(function(row1, row2) { if (row1.count > row2.count) { return -1; } if (row1.count < row2.count) { return 1; } return 0; }); heroes_by_haircolor.each(function(row) { print(row); }); clientsidedata/code/miso.dataset/basic.html Tuesday, October 9, 12

Bocoup | http://bocoup.com // Instantiate a new Miso Dataset object and point it // at our trusty hero database. var heroes = new Miso.Dataset({ url: 'http://localhost:8000/data/just_heroes.json' // or data : yourJSONObjectArray.... }); // fetch data heroes.fetch().then(function() { console.log("Hero #:" + heroes.length); }) ; >> Hero #: 570 Top hair colors for all heroes and villains •(156) •Black(112) •Brown(63) •Blond(57) •Red(42) •None(22) •Bald(16) •White(15) •Blonde(11) •Green(6) // Get all the hair color counts // and then sort them by the actual resulting count. var heroes_by_haircolor = heroes .countBy("Hair_color") .sort(function(row1, row2) { if (row1.count > row2.count) { return -1; } if (row1.count < row2.count) { return 1; } return 0; }); heroes_by_haircolor.each(function(row) { print(row); }); clientsidedata/code/miso.dataset/basic.html Tuesday, October 9, 12

Bocoup | http://bocoup.com ALL THE SUPER POWERS! clientsidedata/code/miso.dataset/superpowers.html Tuesday, October 9, 12

Bocoup | http://bocoup.com Conclusions! ‣ Rows or Columns? ‣ Mix and Match ‣ Beware of iterating too much ‣ Beware of premature optimization ‣ Going to search a lot? Cache it. ‣ Beware of over-caching - keys take space too ‣ Only using a subset of your data? Filter it, then act on it. ‣ Clean up your events ‣ Not everything can be client-side Tuesday, October 9, 12

Bocoup | http://bocoup.com Irene Ros irene@bocoup.com @ireneros github.com/iros github.com/misoproject Questions? Things! Write with us: Gaming + JavaScript? http://buildnewgames.com/ Interested in app control flow management? Contact me for early access to scene.js/storyboard.js (name pending) Tuesday, October 9, 12

Same array of records, altered Object 1. a: 10 2. b: 20 3. sum: 30 4. __proto__: Object Object 1. a: 50 2. b: 60 3. sum: 110 4. __proto__: Object var data = [{ a: 10, b :20 }, { a : 50, b: 60 }]; _.each(data, function(row) { row.sum = row.a + row.b; }); >> New array [10, 40, 5340, 60] _.map([1,4,534,6], function(value) { return value * 10; }); >> 1 _.min([1,4,3,5]) Single Value >> _.inject( [1,2,3,4], function(memo, value) { return memo + value; }, 0 ); >> 10 Tuesday, October 9, 12

Who’s Taller? Heroes or Villains? http://jsbin.com/afupoh/11/ clientsidedata/code/underscore/visual.html Tuesday, October 9, 12

underscore.js vs lodash.js ‣ Drop in replacement. ‣ But different behavior! ‣ Features: ‣ AMD Support ‣ Lazy Binding ‣ _.forOwn - iterating over object’s own properties (no more fetching _.keys first.) ‣ _.clone - has deep cloning! ‣ _.merge - deep _.extend ‣ _.countBy ‣ Custom builds (mobile, content security policy, etc.) ‣ * Faster http://lodash.com/benchmarks Tuesday, October 9, 12