stopwatch - let the data tell a story

talk at the Portland Data Visualization group [1] on an internal tool I built for troubleshooting CloudFoundry installations across five datacenters on three continents.

[1]: https://groups.google.com/forum/?fromgroups#!forum/pdx-visualization

The type family used is Adobe's free Source Code Pro: http://store1.adobe.com/cfusion/store/html/index.cfm?event=displayFontPackage&code=1959

Transcript

1. harnessing the power of the eye to help direct troubleshooting efforts across a distributed & service-oriented architecture stopwatch (will have a better name before open-sourcing. maybe Hermes or BatmanSonar or something)
2. Matthew Lyon @mattly AppFog Platform as a Service by and for developers
3. PHP Fog our first product started as a prototype
4. symptoms of a problem lots of things are randomly very slow or fail without a seeming connection to each other except that it tends to happen at the same time
5. troubleshooting statsd/graphite + easy to store data + stores right data - hard to get at data - complex API - wasn’t helping me pinpoint the problem - doesn’t highlight relationships tailing log files + got a lot of raw data + helped understand the interaction between system components - cost sanity and time
6. 1 month finding root cause gitolite on EBS 1 day writing code to replace gitolite at my in-laws. in Yakima, Washington. on Thanksgiving. 1 month convincing people I was right before high-risk deploy
7. open-source Platform-as-a-Service toolkit created by VMWare launched with Ruby, Java, Node.js runtimes we contributed PHP runtime support it now also runs Python and Erlang (and if you wanted perl, it wouldn’t be hard to add) they built it to run on vSphere we run it on AWS and others cloudfoundry
8. cloudfoundry
9. we run five of those on three continents
10. Hey it’s really slow right now. can you take a look? Sunday, 7:04am *sigh* on it.
11. observe & measure hey, there’s (one of the) problem(s)!
12. numbers are great, but you suck at stats especially if you’re not aware you do unless perhaps you’re German
13. averages lie especially in comparisons failures times what you want is the distribution
14. the human eye can quickly make sense of a lot of data but percentiles don’t tell the whole story and summaries lie too
16. break out by facets appRegistry: resolve service via database dispatcher: import service on rackspace (oh and guess what? this one failed)
17. <demo>
18. we run the largest installations of cloudfoundry some bugs only manifest at the edges the only one run as a pay-for service on the public cloud (that is, AWS, Rackspace Cloud, etc)
19. if you can’t measure from the inside then observe it from the outside cf’s deploy mechanism had timeout issues particularly with AWS/ELB and large apps
20. make it obvious that something is wrong from across the room EBS... again
21. the site is unresponsive? failures gonna propagate
22. cloudfoundry academy, lesson 1: uncaught exceptions will kill you
23. summarize into buckets to help find the pain points
24. drill into the buckets to figure out what’s wrong in this case, the culprit was a new edge case in creating java apps
25. staying on top of network problems
27. AWS US-East is having problems... again with EBS... again this is the AWS outage around Thanksgiving 2012 that took down half the internet
31. a quick tour of how I learned to draw
32. Tufte’s three principles of data density: 1. Above all else, show the data 2. Maximize the data-to-ink ratio 3. Erase non-data ink basically, make every pixel mean something
33. d3 + lots of good tools for simple data + almost an all-in-one solution + many prefab “layouts” + fluent interface (ie, jQuery) ? uses svg - data “joins” are a little weird - fluent interface isn’t always predictable svg inserts a dom node per shape if you’ve got >50k data points, consider...
34. canvas + single dom-node - quickdraw-like API - lack of comprehensive docs - renders to pixels, no zooming also, blurry on Retina (first world problem, I know) - doesn’t necessarily give performance gains performance-tuning rendering changing render contexts is expensive
35. invert data by style use beginPath() and fill() sparingly (yes, this is coffeescript)