This talk was given at Bocoup's OpenVis Conf in Boston.
Abe Stanway@abestanwayData visualizationin the trenches
View Slide
1.5 Billion page views$117 Million of goods sold950 thousand users
1.5 Billion page views$117 Million of goods sold950 thousand users(in December)
TextWe practice continuous deployment.
de • ploy /diˈploi/VerbTo release your code for the world to see,hopefully without breaking the Internet
250+ committers,everyone deploys.
Day one:Deploy.
30+ DEPLOYS A DAY
Text“30 deploys a day? Is that safe?”
TextYes, with the proper tooling.
TextEvery engineer must have a finger onthe pulse of the system.
TextHow do you make an entire web stack“consumable” to a handful ofengineers?
TextMore informationQuickly consumableMore abstraction
p
TextReal time error logging
TextNo abstraction. Fluffy information.Easy to consume.
“Not all things thatbreak throw errors.”- Oscar Wilde
1. ssh to server2. poke around for the log files3. try to remember what they mean4. try to scroll back in time to findwhen they started acting up.5. repeat
TextNo abstraction. Fluffy information.
Text...but hard to get at. Lots offriction means not easilyconsumable.
TextBump up a layer of abstraction.
1. create a graph2. look at the graph3. ??4. profit!!
StatsD
StatsD::increment(“foo.bar”)
If it moves, graph it!
If it doesn’t move, graph it anyway(it might make a run for it)
TextSome abstraction. Denser information.
Text...still hard to get at en masse.
DASHBOARDS!
Hang out with the dashboards after you push.
[1358731200, 20][1358731200, 20][1358731200, 20][1358731200, 20][1358731200, 20][1358731200, 20][1358731200, 20][1358731200, 20][1358731200, 60]
DASHBOARDS x 250000 !
“...but there are also unknownunknowns - there are things wedo not know we don’t know.”
TextSlightly denser information isnegated by the deluge.
TextThe majority remainsunconsumable.
SKYLINE
A real timeanomaly detectionsystem
TextVery abstract, harder tounderstand, but the effectiveinformation density is massive.
TextConsumption is also increasedby outsourcing it to the machine.
TextTrust becomes an issue.
So you found an anomaly.
MAYBE THERE ARE OTHERS
Oculus: a metricscorrelation system
TextA good tool adds “touch” to the system.
TextMore “touch” means moreintimacy with the stack.
TextA delicate balance between insight,consumptive capacity, and actionability.
TextMore abstraction leads to less intimacy...
Text...but greater information density.
@abestanway[email protected]Abe StanwayData EngineerThanks!
astanway
shohata
chobishiba
tmikov2023
sanposhiho
hschwentner
ivargrimstad
maimux2x
iimokoy
iuchimasatoshi
quramy
tomokusaba
skanehira
bcantrill
cassininazir
jonrohan
dotmariusz
skipperchong
jensimmons
yeseniaperezcruz
eitanlees
michaelherold
dougneiner
thoeni
shpigford