Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Data Visualization in the Trenches
Abe Stanway
May 17, 2013
Programming
5
500
Data Visualization in the Trenches
This talk was given at Bocoup's OpenVis Conf in Boston.
Abe Stanway
May 17, 2013
Tweet
Share
More Decks by Abe Stanway
See All by Abe Stanway
Building Data Driven Organizations
astanway
1
190
MOM! My algorithms SUCK
astanway
15
2.4k
A Deep Dive into Monitoring with Skyline
astanway
6
1.6k
Bring the Noise: Continuously Deploying Under a Hailstorm of Metrics
astanway
34
7.1k
Gifs as Language
astanway
2
520
Your API is a Product
astanway
3
750
Zen and the Art of Writing Commit Logs
astanway
3
610
Other Decks in Programming
See All in Programming
TokyoR#103_DataProcessing
kilometer
0
530
新卒でサービス立ち上げから Hasuraを使って3年経った振り返り
yutorin
0
230
Rによる大規模データの処理
s_uryu
2
640
AWSにおける標的型Bot対策
hacomono
0
420
PHPDocにおける配列の型定義を少し知る
shimabox
1
140
AWSとCPUのムフフな関係
cmdemura
0
470
tidy_rpart
bk_18
0
590
ITエンジニア特化型Q&Aサイトteratailを 言語、DB、クラウドなど フルリプレイスした話
leveragestech
0
410
Enumを自動で網羅的にテストしてみた
estie
0
1.3k
Spring BootとKubernetesで実現する今どきのDevOps入門
xblood
0
350
ポケモンで学ぶiOS 16弾丸ツアー 🚅
giginet
PRO
1
610
Hono v3 - Do Everything, Run Anywhere, But Small, And Faster
yusukebe
4
130
Featured
See All Featured
Git: the NoSQL Database
bkeepers
PRO
419
60k
Making Projects Easy
brettharned
102
4.8k
Code Review Best Practice
trishagee
50
11k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
236
1.1M
4 Signs Your Business is Dying
shpigford
171
20k
Build your cross-platform service in a week with App Engine
jlugia
221
17k
How New CSS Is Changing Everything About Graphic Design on the Web
jensimmons
214
12k
What's new in Ruby 2.0
geeforr
336
30k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
217
21k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
22
1.7k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
349
27k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
31
20k
Transcript
Abe Stanway @abestanway Data visualization in the trenches
None
1.5 Billion page views $117 Million of goods sold 950
thousand users
1.5 Billion page views $117 Million of goods sold 950
thousand users (in December)
Text We practice continuous deployment.
de • ploy /diˈploi/ Verb To release your code for
the world to see, hopefully without breaking the Internet
250+ committers, everyone deploys.
Day one: Deploy.
None
30+ DEPLOYS A DAY
Text “30 deploys a day? Is that safe?”
Text Yes, with the proper tooling.
Text Every engineer must have a finger on the pulse
of the system.
Text How do you make an entire web stack “consumable”
to a handful of engineers?
Text More information Quickly consumable More abstraction
p
None
Text Real time error logging
Text No abstraction. Fluffy information. Easy to consume.
“Not all things that break throw errors.” - Oscar Wilde
1. ssh to server 2. poke around for the log
files 3. try to remember what they mean 4. try to scroll back in time to find when they started acting up. 5. repeat
Text No abstraction. Fluffy information.
Text ...but hard to get at. Lots of friction means
not easily consumable.
Text Bump up a layer of abstraction.
1. create a graph 2. look at the graph 3.
?? 4. profit!!
StatsD
StatsD::increment(“foo.bar”)
If it moves, graph it!
If it doesn’t move, graph it anyway (it might make
a run for it)
None
Text Some abstraction. Denser information.
Text ...still hard to get at en masse.
Text Bump up a layer of abstraction.
DASHBOARDS!
Hang out with the dashboards after you push.
None
[1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 20]
[1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 60]
DASHBOARDS x 250000 !
None
“...but there are also unknown unknowns - there are things
we do not know we don’t know.”
Text Slightly denser information is negated by the deluge.
Text The majority remains unconsumable.
Text Bump up a layer of abstraction.
SKYLINE
A real time anomaly detection system
None
Text Very abstract, harder to understand, but the effective information
density is massive.
Text Consumption is also increased by outsourcing it to the
machine.
Text Trust becomes an issue.
So you found an anomaly.
MAYBE THERE ARE OTHERS
Oculus: a metrics correlation system
None
Text A good tool adds “touch” to the system.
Text More “touch” means more intimacy with the stack.
Text A delicate balance between insight, consumptive capacity, and actionability.
Text More abstraction leads to less intimacy...
Text ...but greater information density.
@abestanway
[email protected]
Abe Stanway Data Engineer Thanks!