Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Data Visualization in the Trenches

Data Visualization in the Trenches

This talk was given at Bocoup's OpenVis Conf in Boston.

Abe Stanway

May 17, 2013
Tweet

More Decks by Abe Stanway

Other Decks in Programming

Transcript

  1. Abe Stanway
    @abestanway
    Data visualization
    in the trenches

    View full-size slide

  2. 1.5 Billion page views
    $117 Million of goods sold
    950 thousand users

    View full-size slide

  3. 1.5 Billion page views
    $117 Million of goods sold
    950 thousand users
    (in December)

    View full-size slide

  4. Text
    We practice continuous deployment.

    View full-size slide

  5. de • ploy /diˈploi/
    Verb
    To release your code for the world to see,
    hopefully without breaking the Internet

    View full-size slide

  6. 250+ committers,
    everyone deploys.

    View full-size slide

  7. Day one:
    Deploy.

    View full-size slide

  8. 30+ DEPLOYS A DAY

    View full-size slide

  9. Text
    “30 deploys a day? Is that safe?”

    View full-size slide

  10. Text
    Yes, with the proper tooling.

    View full-size slide

  11. Text
    Every engineer must have a finger on
    the pulse of the system.

    View full-size slide

  12. Text
    How do you make an entire web stack
    “consumable” to a handful of
    engineers?

    View full-size slide

  13. Text
    More information
    Quickly consumable
    More abstraction

    View full-size slide

  14. Text
    Real time error logging

    View full-size slide

  15. Text
    No abstraction. Fluffy information.
    Easy to consume.

    View full-size slide

  16. “Not all things that
    break throw errors.”
    - Oscar Wilde

    View full-size slide

  17. 1. ssh to server
    2. poke around for the log files
    3. try to remember what they mean
    4. try to scroll back in time to find
    when they started acting up.
    5. repeat

    View full-size slide

  18. Text
    No abstraction. Fluffy information.

    View full-size slide

  19. Text
    ...but hard to get at. Lots of
    friction means not easily
    consumable.

    View full-size slide

  20. Text
    Bump up a layer of abstraction.

    View full-size slide

  21. 1. create a graph
    2. look at the graph
    3. ??
    4. profit!!

    View full-size slide

  22. StatsD::increment(“foo.bar”)

    View full-size slide

  23. If it moves, graph it!

    View full-size slide

  24. If it doesn’t move, graph it anyway
    (it might make a run for it)

    View full-size slide

  25. Text
    Some abstraction. Denser information.

    View full-size slide

  26. Text
    ...still hard to get at en masse.

    View full-size slide

  27. Text
    Bump up a layer of abstraction.

    View full-size slide

  28. Hang out with the dashboards after you push.

    View full-size slide

  29. [1358731200, 20]
    [1358731200, 20]
    [1358731200, 20]
    [1358731200, 20]
    [1358731200, 20]
    [1358731200, 20]
    [1358731200, 20]
    [1358731200, 20]
    [1358731200, 60]

    View full-size slide

  30. DASHBOARDS x 250000 !

    View full-size slide

  31. “...but there are also unknown
    unknowns - there are things we
    do not know we don’t know.”

    View full-size slide

  32. Text
    Slightly denser information is
    negated by the deluge.

    View full-size slide

  33. Text
    The majority remains
    unconsumable.

    View full-size slide

  34. Text
    Bump up a layer of abstraction.

    View full-size slide

  35. A real time
    anomaly detection
    system

    View full-size slide

  36. Text
    Very abstract, harder to
    understand, but the effective
    information density is massive.

    View full-size slide

  37. Text
    Consumption is also increased
    by outsourcing it to the machine.

    View full-size slide

  38. Text
    Trust becomes an issue.

    View full-size slide

  39. So you found an anomaly.

    View full-size slide

  40. MAYBE THERE ARE OTHERS

    View full-size slide

  41. Oculus: a metrics
    correlation system

    View full-size slide

  42. Text
    A good tool adds “touch” to the system.

    View full-size slide

  43. Text
    More “touch” means more
    intimacy with the stack.

    View full-size slide

  44. Text
    A delicate balance between insight,
    consumptive capacity, and actionability.

    View full-size slide

  45. Text
    More abstraction leads to less intimacy...

    View full-size slide

  46. Text
    ...but greater information density.

    View full-size slide

  47. @abestanway
    [email protected]
    Abe Stanway
    Data Engineer
    Thanks!

    View full-size slide