Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Boston 2013 - Session - Aaron Quint

Monitorama
March 28, 2013
410

Boston 2013 - Session - Aaron Quint

Monitorama

March 28, 2013
Tweet

Transcript

  1. A litte bit of @aq • Expert Eater • Experienced

    Ruby and JS Developer Wednesday, May 29, 13
  2. A litte bit of @aq • Expert Eater • Experienced

    Ruby and JS Developer • Growing Student of Operations Wednesday, May 29, 13
  3. A litte bit of @aq • Expert Eater • Experienced

    Ruby and JS Developer • Growing Student of Operations • Beginner Distributed Systems Maintainer Wednesday, May 29, 13
  4. And let us get back to Shaving Yaks. correlation can

    narrow our work Wednesday, May 29, 13
  5. 0 1.5 3 4.5 6 0 225 450 675 900

    CPU vs Response Time Wednesday, May 29, 13
  6. 0 1.5 3 4.5 6 0 225 450 675 900

    CPU vs Response Time Wednesday, May 29, 13
  7. 1 data = [ 2 [100, 0.7], 3 [125, 0.5],

    4 [150, 1], 5 [300, 2.1], 6 [500, 3.4], 7 [900, 6] 8 ] 9 10 x, y = data.transpose 11 n = data.size 12 x_mean = x.reduce(:+) / n 13 y_mean = y.reduce(:+) / n 14 x_stddev = Math.sqrt(x.inject {|sum, i| sum + (i - x_mean)**2 } / (n - 1).to_f) 15 y_stddev = Math.sqrt(y.inject {|sum, i| sum + (i - y_mean)**2 } / (n - 1).to_f) 16 z_x = x.collect {|i| (i - x_mean) / x_stddev } 17 z_y = y.collect {|i| (i - y_mean) / y_stddev } 18 pearsons = z_x.zip(z_y).collect {|x| x[0] * x[1] }.reduce(:+) / n 19 # => 0.9265763490538744 Wednesday, May 29, 13
  8. PEarson • Close to absolute 1 = probably correlated samples

    • Could be applied to moving averages? Wednesday, May 29, 13
  9. PEarson • Close to absolute 1 = probably correlated samples

    • Could be applied to moving averages? • Could we pull it into a graphite function? (Hackathon anyone?) Wednesday, May 29, 13
  10. LIMITS OF Mathematical correlation • Requires known inputs and assumptions

    • Suggestion of correlation, not proof Wednesday, May 29, 13
  11. LIMITS OF Mathematical correlation • Requires known inputs and assumptions

    • Suggestion of correlation, not proof • Needs a large amount of knowledge of the data set to make decisions Wednesday, May 29, 13
  12. LIMITS OF VISUAL CORRELATION • Takes a good eye •

    Hard to see the signal through the noise Wednesday, May 29, 13
  13. LIMITS OF VISUAL CORRELATION • Takes a good eye •

    Hard to see the signal through the noise • Doesn’t really account for domino events Wednesday, May 29, 13
  14. LIMITS OF VISUAL CORRELATION • Takes a good eye •

    Hard to see the signal through the noise • Doesn’t really account for domino events • Good for trends but not as much for events Wednesday, May 29, 13
  15. Each person uses their unique knowledge of the situation to

    point out unique data points. RASHoMONING Wednesday, May 29, 13
  16. LIMITS OF EMOTIONAL correlation • Provides a trail not an

    answer • Depends on having a team of people Wednesday, May 29, 13
  17. LIMITS OF EMOTIONAL correlation • Provides a trail not an

    answer • Depends on having a team of people • Many ideas, needs a “judge” Wednesday, May 29, 13
  18. LIMITS OF EMOTIONAL correlation • Provides a trail not an

    answer • Depends on having a team of people • Many ideas, needs a “judge” • HUMANS (Hence Rashomoning) Wednesday, May 29, 13
  19. Hotpot = Chef, Sensu, Graphite, (Logstash) Simply align disparate sources

    of data TO VISUALLY CORRELATE Wednesday, May 29, 13
  20. Math to filter out noise. USe PEARSONS to pull out

    potentially related data Wednesday, May 29, 13
  21. Have the ability to easily divide datasets by “cohorts” cohort

    analysis for processes/nodes Wednesday, May 29, 13