Building Data Driven Organizations

6601d82cf1b6776afd9c31f3d18294c3?s=47 Abe Stanway
September 13, 2014

Building Data Driven Organizations

Given at IT Weekend 2014 in Kiev

6601d82cf1b6776afd9c31f3d18294c3?s=128

Abe Stanway

September 13, 2014
Tweet

Transcript

  1. @AbeStanway BUILDING A DATA DRIVEN ORGANIZATION

  2. 1. why 2. how

  3. 1. why 2. how

  4. “DATA IS THE NEW GOLD”

  5. Predict the future!

  6. Retain Customers!

  7. Grow the business!

  8. Recommend content!

  9. Drive Engagement!

  10. unclear paths to $$$

  11. IN IT, It’s clear.

  12. Data are Dollars

  13. . IT Working = +$$$ IT Not Working = -$$$

    . .
  14. How do you know if your IT is working right

    now?
  15. How do you know if you are earning money right

    now?
  16. KPIs. What are they?

  17. Etsy: Literally a Money per second Graph

  18. Planet Labs: Literally an Images per day graph

  19. What are the Kpis for kips?

  20. $ per second items bought per second page requests per

    second database queries per second memcache hits per second fread() per second
  21. If you do not have the data about your infrastructure,

    it is already broken. LEsson:
  22. None
  23. Test driven development -> data driven development

  24. Without data, you are flying blind

  25. How do you know you’re hitting your goals?

  26. How do you know if You’re making the right ones

    in the first place?
  27. How do you know if you’re still in business?

  28. How do you even know what planet you live on?

  29. Assumptions are death

  30. You need data, yo.

  31. 1. why 2. how

  32. 1. collect 2. analyze 3. ??? 4. Profit!

  33. 1. collect 2. analyze 3. ACT 4. Profit!

  34. data that cannot be acted upon should not be analyzed.

  35. None
  36. You are running a business, not an art museum

  37. You are Trying to Win the market, not a fields

    medal
  38. This can be disappointing

  39. Data SCientist?

  40. Data Scientist? Realist.

  41. Find a way to Align your employees intellectual curiosity With

    your Real business needs. LEssoN:
  42. Train your organization

  43. you need a data culture.

  44. “It’s not shipped until it’s monitored”

  45. “If you are not looking at Dashboards, you are not

    doing your job”
  46. Building instrumentation and watching dashboards are hard And Time consuming

  47. App code -> statsD -> Graphite -> Dashboards -> Insights

    by hand by hand by hand by hand by hand
  48. Developers just want to code

  49. Let’s automate

  50. Which is easier to automate? Insights or data collection?

  51. Insights are sexy and fun

  52. Collection is hard And unsexy

  53. Collection is hard And Boring

  54. Collection is hard And unsexy

  55. We’re on track to have excellent automated insights

  56. anomaly detection

  57. App code -> statsD -> Graphite -> Dashboards -> Insights

    by hand by hand by hand AUTOMATIC! AUTOMATIC!
  58. (…if only we had the data)

  59. How do we automate data collection?

  60. currently have ganglia, New relic, collectD, etc

  61. NOT WHAT WE NEED

  62. they provide data about your raw machines, not your CUSTOM

    DEVELOPED TECHNOLOGY And Application level logic
  63. Healthy servers don’t make you money. Healthy services do.

  64. enter LARIMAR

  65. Full disclosure: this is my new PROJECT ! we’re going

    to talk about it because i’m pretty excited and the beta is opening up soon.
  66. LARIMAR uses raw machine metrics to infer App level architecture

    and inform developers about problems
  67. A service: cpu resources disk io PCAP data ports Used

    syscalls
  68. A service: cpu resources disk io PCAP data ports Used

    syscalls service fingerprint MACHINE LEARNING
  69. A service: cpu resources disk io PCAP data ports Used

    syscalls ABNORMAL BEHAVIOR MACHINE LEARNING
  70. a system: service service service service service MACHINE LEARNING graphical

    system fingerprint
  71. a system: service service service service service MACHINE LEARNING Abnormal,

    holistic system behavior
  72. Larimar automates both analysis And Relevant data collection

  73. so your developers can focus on coding and acting on

    insights
  74. No configuration!

  75. App code -> statsD -> Graphite -> Dashboards -> Insights

    by hand AUTOMATIC! AUTOMATIC! AUTOMATIC! AUTOMATIC!
  76. 1. collect 2. analyze 3. ACT 4. Profit!

  77. organizational shifts are still needed to inspire ACTION on Data

  78. but ACTION is easier to inspire when there is lots

    of data and lots of insight everywhere
  79. Create a culture where your developers create these kinds of

    tools
  80. When a data driven mindset is the default, tools will

    build themselves.
  81. Thanks! @abestanway ! ! larimar.io @larimarhq