Slide 1

Slide 1 text

Abe Stanway @abestanway Data visualization in the trenches

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

1.5 Billion page views $117 Million of goods sold 950 thousand users

Slide 4

Slide 4 text

1.5 Billion page views $117 Million of goods sold 950 thousand users (in December)

Slide 5

Slide 5 text

Text We practice continuous deployment.

Slide 6

Slide 6 text

de • ploy /diˈploi/ Verb To release your code for the world to see, hopefully without breaking the Internet

Slide 7

Slide 7 text

250+ committers, everyone deploys.

Slide 8

Slide 8 text

Day one: Deploy.

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

30+ DEPLOYS A DAY

Slide 11

Slide 11 text

Text “30 deploys a day? Is that safe?”

Slide 12

Slide 12 text

Text Yes, with the proper tooling.

Slide 13

Slide 13 text

Text Every engineer must have a finger on the pulse of the system.

Slide 14

Slide 14 text

Text How do you make an entire web stack “consumable” to a handful of engineers?

Slide 15

Slide 15 text

Text More information Quickly consumable More abstraction

Slide 16

Slide 16 text

p

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

Text Real time error logging

Slide 19

Slide 19 text

Text No abstraction. Fluffy information. Easy to consume.

Slide 20

Slide 20 text

“Not all things that break throw errors.” - Oscar Wilde

Slide 21

Slide 21 text

1. ssh to server 2. poke around for the log files 3. try to remember what they mean 4. try to scroll back in time to find when they started acting up. 5. repeat

Slide 22

Slide 22 text

Text No abstraction. Fluffy information.

Slide 23

Slide 23 text

Text ...but hard to get at. Lots of friction means not easily consumable.

Slide 24

Slide 24 text

Text Bump up a layer of abstraction.

Slide 25

Slide 25 text

1. create a graph 2. look at the graph 3. ?? 4. profit!!

Slide 26

Slide 26 text

StatsD

Slide 27

Slide 27 text

StatsD::increment(“foo.bar”)

Slide 28

Slide 28 text

If it moves, graph it!

Slide 29

Slide 29 text

If it doesn’t move, graph it anyway (it might make a run for it)

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

Text Some abstraction. Denser information.

Slide 32

Slide 32 text

Text ...still hard to get at en masse.

Slide 33

Slide 33 text

Text Bump up a layer of abstraction.

Slide 34

Slide 34 text

DASHBOARDS!

Slide 35

Slide 35 text

Hang out with the dashboards after you push.

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

[1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 60]

Slide 38

Slide 38 text

DASHBOARDS x 250000 !

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

“...but there are also unknown unknowns - there are things we do not know we don’t know.”

Slide 41

Slide 41 text

Text Slightly denser information is negated by the deluge.

Slide 42

Slide 42 text

Text The majority remains unconsumable.

Slide 43

Slide 43 text

Text Bump up a layer of abstraction.

Slide 44

Slide 44 text

SKYLINE

Slide 45

Slide 45 text

A real time anomaly detection system

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

Text Very abstract, harder to understand, but the effective information density is massive.

Slide 48

Slide 48 text

Text Consumption is also increased by outsourcing it to the machine.

Slide 49

Slide 49 text

Text Trust becomes an issue.

Slide 50

Slide 50 text

So you found an anomaly.

Slide 51

Slide 51 text

MAYBE THERE ARE OTHERS

Slide 52

Slide 52 text

Oculus: a metrics correlation system

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

Text A good tool adds “touch” to the system.

Slide 55

Slide 55 text

Text More “touch” means more intimacy with the stack.

Slide 56

Slide 56 text

Text A delicate balance between insight, consumptive capacity, and actionability.

Slide 57

Slide 57 text

Text More abstraction leads to less intimacy...

Slide 58

Slide 58 text

Text ...but greater information density.

Slide 59

Slide 59 text

@abestanway [email protected] Abe Stanway Data Engineer Thanks!