Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Data Visualization in the Trenches

Data Visualization in the Trenches

This talk was given at Bocoup's OpenVis Conf in Boston.

Abe Stanway

May 17, 2013
Tweet

More Decks by Abe Stanway

Other Decks in Programming

Transcript

  1. Abe Stanway
    @abestanway
    Data visualization
    in the trenches

    View Slide

  2. View Slide

  3. 1.5 Billion page views
    $117 Million of goods sold
    950 thousand users

    View Slide

  4. 1.5 Billion page views
    $117 Million of goods sold
    950 thousand users
    (in December)

    View Slide

  5. Text
    We practice continuous deployment.

    View Slide

  6. de • ploy /diˈploi/
    Verb
    To release your code for the world to see,
    hopefully without breaking the Internet

    View Slide

  7. 250+ committers,
    everyone deploys.

    View Slide

  8. Day one:
    Deploy.

    View Slide

  9. View Slide

  10. 30+ DEPLOYS A DAY

    View Slide

  11. Text
    “30 deploys a day? Is that safe?”

    View Slide

  12. Text
    Yes, with the proper tooling.

    View Slide

  13. Text
    Every engineer must have a finger on
    the pulse of the system.

    View Slide

  14. Text
    How do you make an entire web stack
    “consumable” to a handful of
    engineers?

    View Slide

  15. Text
    More information
    Quickly consumable
    More abstraction

    View Slide

  16. p

    View Slide

  17. View Slide

  18. Text
    Real time error logging

    View Slide

  19. Text
    No abstraction. Fluffy information.
    Easy to consume.

    View Slide

  20. “Not all things that
    break throw errors.”
    - Oscar Wilde

    View Slide

  21. 1. ssh to server
    2. poke around for the log files
    3. try to remember what they mean
    4. try to scroll back in time to find
    when they started acting up.
    5. repeat

    View Slide

  22. Text
    No abstraction. Fluffy information.

    View Slide

  23. Text
    ...but hard to get at. Lots of
    friction means not easily
    consumable.

    View Slide

  24. Text
    Bump up a layer of abstraction.

    View Slide

  25. 1. create a graph
    2. look at the graph
    3. ??
    4. profit!!

    View Slide

  26. StatsD

    View Slide

  27. StatsD::increment(“foo.bar”)

    View Slide

  28. If it moves, graph it!

    View Slide

  29. If it doesn’t move, graph it anyway
    (it might make a run for it)

    View Slide

  30. View Slide

  31. Text
    Some abstraction. Denser information.

    View Slide

  32. Text
    ...still hard to get at en masse.

    View Slide

  33. Text
    Bump up a layer of abstraction.

    View Slide

  34. DASHBOARDS!

    View Slide

  35. Hang out with the dashboards after you push.

    View Slide

  36. View Slide

  37. [1358731200, 20]
    [1358731200, 20]
    [1358731200, 20]
    [1358731200, 20]
    [1358731200, 20]
    [1358731200, 20]
    [1358731200, 20]
    [1358731200, 20]
    [1358731200, 60]

    View Slide

  38. DASHBOARDS x 250000 !

    View Slide

  39. View Slide

  40. “...but there are also unknown
    unknowns - there are things we
    do not know we don’t know.”

    View Slide

  41. Text
    Slightly denser information is
    negated by the deluge.

    View Slide

  42. Text
    The majority remains
    unconsumable.

    View Slide

  43. Text
    Bump up a layer of abstraction.

    View Slide

  44. SKYLINE

    View Slide

  45. A real time
    anomaly detection
    system

    View Slide

  46. View Slide

  47. Text
    Very abstract, harder to
    understand, but the effective
    information density is massive.

    View Slide

  48. Text
    Consumption is also increased
    by outsourcing it to the machine.

    View Slide

  49. Text
    Trust becomes an issue.

    View Slide

  50. So you found an anomaly.

    View Slide

  51. MAYBE THERE ARE OTHERS

    View Slide

  52. Oculus: a metrics
    correlation system

    View Slide

  53. View Slide

  54. Text
    A good tool adds “touch” to the system.

    View Slide

  55. Text
    More “touch” means more
    intimacy with the stack.

    View Slide

  56. Text
    A delicate balance between insight,
    consumptive capacity, and actionability.

    View Slide

  57. Text
    More abstraction leads to less intimacy...

    View Slide

  58. Text
    ...but greater information density.

    View Slide

  59. @abestanway
    [email protected]
    Abe Stanway
    Data Engineer
    Thanks!

    View Slide