Correlation: The Next Frontier

Correlation: The Next Frontier

My talk from #monitorama 2013 with ideas about how to apply different types of correlation to our data

F04bfa14141dca6713f0d9caa763e26b?s=128

Aaron Quint

March 28, 2013
Tweet

Transcript

  1. 2.
  2. 4.
  3. 5.
  4. 9.

    A litte bit of @aq • Expert Eater • Experienced

    Ruby and JS Developer • Growing Student of Operations
  5. 10.

    A litte bit of @aq • Expert Eater • Experienced

    Ruby and JS Developer • Growing Student of Operations • Beginner Distributed Systems Maintainer
  6. 13.
  7. 15.
  8. 17.
  9. 18.
  10. 23.
  11. 24.
  12. 33.

    0 1.5 3 4.5 6 0 225 450 675 900

    CPU vs Response Time
  13. 34.

    0 1.5 3 4.5 6 0 225 450 675 900

    CPU vs Response Time
  14. 35.
  15. 36.

    1 data = [ 2 [100, 0.7], 3 [125, 0.5],

    4 [150, 1], 5 [300, 2.1], 6 [500, 3.4], 7 [900, 6] 8 ] 9 10 x, y = data.transpose 11 n = data.size 12 x_mean = x.reduce(:+) / n 13 y_mean = y.reduce(:+) / n 14 x_stddev = Math.sqrt(x.inject {|sum, i| sum + (i - x_mean)**2 } / (n - 1).to_f) 15 y_stddev = Math.sqrt(y.inject {|sum, i| sum + (i - y_mean)**2 } / (n - 1).to_f) 16 z_x = x.collect {|i| (i - x_mean) / x_stddev } 17 z_y = y.collect {|i| (i - y_mean) / y_stddev } 18 pearsons = z_x.zip(z_y).collect {|x| x[0] * x[1] }.reduce(:+) / n 19 # => 0.9265763490538744
  16. 37.
  17. 39.

    PEarson • Close to absolute 1 = probably correlated samples

    • Could be applied to moving averages?
  18. 40.

    PEarson • Close to absolute 1 = probably correlated samples

    • Could be applied to moving averages? • Could we pull it into a graphite function? (Hackathon anyone?)
  19. 44.

    LIMITS OF Mathematical correlation • Requires known inputs and assumptions

    • Suggestion of correlation, not proof • Needs a large amount of knowledge of the data set to make decisions
  20. 49.

    LIMITS OF VISUAL CORRELATION • Takes a good eye •

    Hard to see the signal through the noise
  21. 50.

    LIMITS OF VISUAL CORRELATION • Takes a good eye •

    Hard to see the signal through the noise • Doesn’t really account for domino events
  22. 51.

    LIMITS OF VISUAL CORRELATION • Takes a good eye •

    Hard to see the signal through the noise • Doesn’t really account for domino events • Good for trends but not as much for events
  23. 54.

    Each person uses their unique knowledge of the situation to

    point out unique data points. RASHoMONING
  24. 57.

    LIMITS OF EMOTIONAL correlation • Provides a trail not an

    answer • Depends on having a team of people
  25. 58.

    LIMITS OF EMOTIONAL correlation • Provides a trail not an

    answer • Depends on having a team of people • Many ideas, needs a “judge”
  26. 59.

    LIMITS OF EMOTIONAL correlation • Provides a trail not an

    answer • Depends on having a team of people • Many ideas, needs a “judge” • HUMANS (Hence Rashomoning)
  27. 64.
  28. 67.
  29. 73.