Monitorama
March 28, 2013
410

# Boston 2013 - Session - Aaron Quint

March 28, 2013

## Transcript

1. ### CORRELATION: THE NEXT FRONTIER monitorama 2013 / boston / @aq

Wednesday, May 29, 13

29, 13
8. ### A litte bit of @aq • Expert Eater • Experienced

Ruby and JS Developer Wednesday, May 29, 13
9. ### A litte bit of @aq • Expert Eater • Experienced

Ruby and JS Developer • Growing Student of Operations Wednesday, May 29, 13
10. ### A litte bit of @aq • Expert Eater • Experienced

Ruby and JS Developer • Growing Student of Operations • Beginner Distributed Systems Maintainer Wednesday, May 29, 13

13

May 29, 13

May 29, 13
22. ### PURE DATA BASIC INFERENCES AND CORRELATIONS THE FUCKING MATRIX whoa

Wednesday, May 29, 13

26. ### PURE DATA BASIC INFERENCES AND CORRELATIONS PREDICTIVE AND DIRECT RELATIONSHIPS

Wednesday, May 29, 13

May 29, 13

30. ### And let us get back to Shaving Yaks. correlation can

narrow our work Wednesday, May 29, 13

13
32. ### Say that 5 times fast. PEARSON Product moment correlation coefﬁcient

Wednesday, May 29, 13
33. ### 0 1.5 3 4.5 6 0 225 450 675 900

CPU vs Response Time Wednesday, May 29, 13
34. ### 0 1.5 3 4.5 6 0 225 450 675 900

CPU vs Response Time Wednesday, May 29, 13

36. ### 1 data = [ 2 [100, 0.7], 3 [125, 0.5],

4 [150, 1], 5 [300, 2.1], 6 [500, 3.4], 7 [900, 6] 8 ] 9 10 x, y = data.transpose 11 n = data.size 12 x_mean = x.reduce(:+) / n 13 y_mean = y.reduce(:+) / n 14 x_stddev = Math.sqrt(x.inject {|sum, i| sum + (i - x_mean)**2 } / (n - 1).to_f) 15 y_stddev = Math.sqrt(y.inject {|sum, i| sum + (i - y_mean)**2 } / (n - 1).to_f) 16 z_x = x.collect {|i| (i - x_mean) / x_stddev } 17 z_y = y.collect {|i| (i - y_mean) / y_stddev } 18 pearsons = z_x.zip(z_y).collect {|x| x[0] * x[1] }.reduce(:+) / n 19 # => 0.9265763490538744 Wednesday, May 29, 13

38. ### PEarson • Close to absolute 1 = probably correlated samples

Wednesday, May 29, 13
39. ### PEarson • Close to absolute 1 = probably correlated samples

• Could be applied to moving averages? Wednesday, May 29, 13
40. ### PEarson • Close to absolute 1 = probably correlated samples

• Could be applied to moving averages? • Could we pull it into a graphite function? (Hackathon anyone?) Wednesday, May 29, 13

42. ### LIMITS OF Mathematical correlation • Requires known inputs and assumptions

Wednesday, May 29, 13
43. ### LIMITS OF Mathematical correlation • Requires known inputs and assumptions

• Suggestion of correlation, not proof Wednesday, May 29, 13
44. ### LIMITS OF Mathematical correlation • Requires known inputs and assumptions

• Suggestion of correlation, not proof • Needs a large amount of knowledge of the data set to make decisions Wednesday, May 29, 13

29, 13

May 29, 13
49. ### LIMITS OF VISUAL CORRELATION • Takes a good eye •

Hard to see the signal through the noise Wednesday, May 29, 13
50. ### LIMITS OF VISUAL CORRELATION • Takes a good eye •

Hard to see the signal through the noise • Doesn’t really account for domino events Wednesday, May 29, 13
51. ### LIMITS OF VISUAL CORRELATION • Takes a good eye •

Hard to see the signal through the noise • Doesn’t really account for domino events • Good for trends but not as much for events Wednesday, May 29, 13

29, 13

54. ### Each person uses their unique knowledge of the situation to

point out unique data points. RASHoMONING Wednesday, May 29, 13

57. ### LIMITS OF EMOTIONAL correlation • Provides a trail not an

answer • Depends on having a team of people Wednesday, May 29, 13
58. ### LIMITS OF EMOTIONAL correlation • Provides a trail not an

answer • Depends on having a team of people • Many ideas, needs a “judge” Wednesday, May 29, 13
59. ### LIMITS OF EMOTIONAL correlation • Provides a trail not an

answer • Depends on having a team of people • Many ideas, needs a “judge” • HUMANS (Hence Rashomoning) Wednesday, May 29, 13

May 29, 13

13

65. ### Hotpot = Chef, Sensu, Graphite, (Logstash) Simply align disparate sources

of data TO VISUALLY CORRELATE Wednesday, May 29, 13

68. ### Math to ﬁlter out noise. USe PEARSONS to pull out

potentially related data Wednesday, May 29, 13
69. ### Have the ability to easily divide datasets by “cohorts” cohort

analysis for processes/nodes Wednesday, May 29, 13
70. ### “Node notes”. Document everything. Treat personal and institutional knowledge as

data Wednesday, May 29, 13
71. ### By making more data available to everyone. Make emotional correlation

less EMO Wednesday, May 29, 13

May 29, 13

74. ### TAKE the correlations and let the machine turn them into

decisions Wednesday, May 29, 13

29, 13

13