Data Visualization in the Trenches

Transcript

1. 1.

   Abe Stanway @abestanway Data visualization in the trenches

2. 2.

   None
3. 3.

   1.5 Billion page views $117 Million of goods sold 950

   thousand users
4. 4.

   1.5 Billion page views $117 Million of goods sold 950

   thousand users (in December)
5. 5.

   Text We practice continuous deployment.

6. 6.

   de • ploy /diˈploi/ Verb To release your code for

   the world to see, hopefully without breaking the Internet
7. 7.

   250+ committers, everyone deploys.

8. 8.

   Day one: Deploy.

9. 9.

   None
10. 10.

    30+ DEPLOYS A DAY

11. 11.

    Text “30 deploys a day? Is that safe?”

12. 12.

    Text Yes, with the proper tooling.

13. 13.

    Text Every engineer must have a finger on the pulse

    of the system.
14. 14.

    Text How do you make an entire web stack “consumable”

    to a handful of engineers?
15. 15.

    Text More information Quickly consumable More abstraction

16. 16.

    p

17. 17.

    None
18. 18.

    Text Real time error logging

19. 19.

    Text No abstraction. Fluffy information. Easy to consume.

20. 20.

    “Not all things that break throw errors.” - Oscar Wilde

21. 21.

    1. ssh to server 2. poke around for the log

    files 3. try to remember what they mean 4. try to scroll back in time to find when they started acting up. 5. repeat
22. 22.

    Text No abstraction. Fluffy information.

23. 23.

    Text ...but hard to get at. Lots of friction means

    not easily consumable.
24. 24.

    Text Bump up a layer of abstraction.

25. 25.

    1. create a graph 2. look at the graph 3.

    ?? 4. profit!!
26. 26.

    StatsD

27. 27.

    StatsD::increment(“foo.bar”)

28. 28.

    If it moves, graph it!

29. 29.

    If it doesn’t move, graph it anyway (it might make

    a run for it)
30. 30.

    None
31. 31.

    Text Some abstraction. Denser information.

32. 32.

    Text ...still hard to get at en masse.

33. 33.

    Text Bump up a layer of abstraction.

34. 34.

    DASHBOARDS!

35. 35.

    Hang out with the dashboards after you push.

36. 36.

    None
37. 37.

    [1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 20]

    [1358731200, 20] [1358731200, 20] [1358731200, 20] [1358731200, 60]
38. 38.

    DASHBOARDS x 250000 !

39. 39.

    None
40. 40.

    “...but there are also unknown unknowns - there are things

    we do not know we don’t know.”
41. 41.

    Text Slightly denser information is negated by the deluge.

42. 42.

    Text The majority remains unconsumable.

43. 43.

    Text Bump up a layer of abstraction.

44. 44.

    SKYLINE

45. 45.

    A real time anomaly detection system

46. 46.

    None
47. 47.

    Text Very abstract, harder to understand, but the effective information

    density is massive.
48. 48.

    Text Consumption is also increased by outsourcing it to the

    machine.
49. 49.

    Text Trust becomes an issue.

50. 50.

    So you found an anomaly.

51. 51.

    MAYBE THERE ARE OTHERS

52. 52.

    Oculus: a metrics correlation system

53. 53.

    None
54. 54.

    Text A good tool adds “touch” to the system.

55. 55.

    Text More “touch” means more intimacy with the stack.

56. 56.

    Text A delicate balance between insight, consumptive capacity, and actionability.

57. 57.

    Text More abstraction leads to less intimacy...

58. 58.

    Text ...but greater information density.

59. 59.

    @abestanway abe@etsy.com Abe Stanway Data Engineer Thanks!