Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Roll Your Own Realtime Visualizations

Roll Your Own Realtime Visualizations

Humans are excellent at recognizing patterns. When we look at something familiar, we notice at a glance when something is unusual. The standard time-series graphs that we use to monitor our systems are immensely useful, but seeing that data in real time allows our pattern recognition to really shine.

I’ll talk about ways to get realtime data out of your system and discuss some of the options for visualizing that data. I’ll share some of the visualizations we’ve built for LivingSocial. If you’ve ever thought “I really wish I could see what my visitors are doing”, this talk is for you.

Ben Bleything

June 30, 2012
Tweet

More Decks by Ben Bleything

Other Decks in Programming

Transcript

  1. Pattern recognition It deosn't mttaer waht oredr the ltteers in

    a wrod are, the olny ipromatnt tihng is taht the frist and lsat ltteres are in the rghit pclae. Excerpted from O’Reilly’s Designing Data Visualizations
  2. Things brains hate :03:10 0.7 :03:20 0.6 :03:30 0.4 :03:40

    0.3 :03:50 0.9 :04:00 0.9 :04:10 0.7 :04:20 1.0 :04:30 0.7 :04:40 0.5 :04:50 0.3 :05:00 1.2 :05:10 0.7 11:05:20 0.4 11:05:30 0.5 11:05:40 0.7 11:05:50 0.7 11:06:00 0.4 11:06:10 0.5 11:06:20 0.6 11:06:30 0.3 11:06:40 1.0 11:06:50 1.0 11:07:00 0.5 11:07:10 1.1 11:07:20 0.4 11:07:30 0.6 11:07:40 0.8 11:07:50 1.0 11:08:00 0.5 11:08:10 0.4 11:08:20 0.8 11:08:30 0.6 11:08:40 0.8 11:08:50 0.4 11:09:00 1.0 11:09:10 0.7 11:09:20 1.2 11:09:30 1.1 11:09:40 11:09:50 11:10:00 11:10:10 11:10:20 11:10:30 11:10:40 11:10:50 11:11:00 11:11:10 11:11:20 11:11:30 11:11:40 :(
  3. But ben! this session is supposed to be about “real-time

    visualizations” but all you’ve talked about so far is something something brain and static time-series graphs and i didn’t come here to hear about that crap! it’s
  4. design your viz • pick your technology • HTML •

    d3 • Processing • from scratch • visual and information design
  5. Visual design • Edward Tufte • The Visual Display of

    Quantitative Information • Envisioning Information • Visual Explanations • Beautiful Evidence • Julie Steele & Noah Iliinsky • Beautiful Visualization
  6. getting data out • redis pub/sub • message queue •

    raw socket connections • log tailing just use redis (queues are okay too) meh
  7. redis consumer var redis_client = require("redis"), redis = redis_client.createClient(6379); redis.on("pmessage",

    function(pattern, channel, data) { return console.log("" + channel + ": " + data); }); redis.on("ready", function() { return redis.psubscribe("realtime:purchase"); }); redis.on("error", function() { return console.log("redis error; retrying"); });
  8. But ben, what about this “processing” you were talking about

    earlier? it sounded kinda cool even though it’s java, and besides i’m no good at javascript and I reaaally want to write some awesome things so maybe
  9. processing import processing.opengl.*; Director director; Driver script; PImage background; String

    timestamp; Vector bloops; color mask; Clock clock; void setup() { // to render in glorious 1080p, change this to true boolean render_huge = false; // to render in 640x320 (for vimeo), change this to true boolean render_tiny = false; // to render in 1024x512 (for projection)
  10. processing boolean render_projector = true; // to save a quicktime

    movie, change this to true boolean render_movie = false; // to render real-time purchases, set this to true. boolean realtime = true; int width, height; if( render_huge ) { width = 1920; height = 1080; background = loadImage("world_1080.png"); } else if( render_tiny ) { width = 640; height = 320; background = loadImage("world_640.png");
  11. processing } else if( render_projector ) { width = 1024;

    height = 512; background = loadImage("world_1024.png"); } else { width = 1280; height = 720; background = loadImage("world_720.png"); } size( width, height ); frameRate(30); background(0); smooth(); mask = color(0,0,0,145); bloops = new Vector();
  12. processing director = new Director( this, "viz.mov", render_movie ); int

    clockSize = width / 20; int clockY = (int) (height - clockSize * 2.2); if( realtime ) { script = new RedisDriver( "localhost", 6379 ); clock = new Clock( 0, width / 2, clockY, clockSize, true ); } else { script = new ScriptDriver("mapping.script"); clock = new Clock( 0, width / 2, clockY, clockSize, false ); } } // Line format: //
  13. processing // unixtime|lat1,long1@lat2,long2|lat1,long1@lat2,long2|... // 1327097733|42.17,[email protected],-178.68| 38.11,[email protected],-125.34 // String parseLine( String

    line ) { // if the line is null then something is weird and we should skip it if( line == null ) { println("null");return null; } // details is an array of [timestamp, coord_pair, coord_pair, ...] String[] details = split(line,'|'); String timestamp = subset(details, 0, 1)[0]; // if the line ends with a pipe, it means we only got a timestamp if( line.endsWith("|") ) { return timestamp; }
  14. processing // pairs is an array of lat,long@lat,long entries String[]

    pairs = subset(details, 1); for( int i = 0; i < pairs.length; i++ ) { Bloop bloop = new Bloop( pairs[i] ); bloops.add( bloop ); } return timestamp; } void renderBloops() { int bloop_count = bloops.size(); println( "rendering " + bloop_count + " bloops" ); for( Enumeration e = bloops.elements() ; e.hasMoreElements() ; ) { Bloop bloop = (Bloop) e.nextElement();
  15. processing if( bloop.finished() ) { bloops.remove( bloop ); } else

    { bloop.draw(); } } } void draw() { String ts = parseLine( script.read() ); if( script.finished() && bloops.isEmpty() ) { println("done"); director.finish(); noLoop(); } else { if( ts != null ) { timestamp = ts; }
  16. processing // clear screen image( background, 0, 0 ); try

    { clock.drawAt( Integer.parseInt(timestamp) ); } catch( java.lang.NumberFormatException ex ) { // no-op } fill(mask); stroke(mask); rect(0,0,width,height); renderBloops(); } director.addFrame(); }