Modern Radiology for Distributed Systems

Transcript

1. Modern Radiology for Distributed Systems Dietrich Featherston @d2fn Thursday, October

   11, 12

2. This is a talk about monitoring Thursday, October 11, 12

3. But not just any kind of monitoring Non-invasive monitoring Thursday,

   October 11, 12

4. non-invasive monitoring measures taken to describe the state of a

   system with minimal changes to the system being monitored Thursday, October 11, 12

5. Insight Invasiveness Radiographic Imagery Thursday, October 11, 12

6. preventative care measures taken to prevent diseases or injuries rather

   than curing them or treating their symptoms Thursday, October 11, 12

7. Non-invasive monitoring techniques focus primarily on host-based metrics Why is

   this a problem? Thursday, October 11, 12

8. Because applications are distributed Thursday, October 11, 12

9. Information emitted about nodes in the network n Information emitted

   about edges in the network n² Network size Thursday, October 11, 12

10. We analyze cell-structure because we can’t envision the whole organism

    We react to disease and injury because we lack preventative care Thursday, October 11, 12

11. We lack preventative care for applications because our non-invasive monitoring

    techniques are growing less and less meaningful Thursday, October 11, 12

12. Radiology is useful in illuminating non-invasive monitoring of distributed systems

    Thursday, October 11, 12

13. Thursday, October 11, 12

14. Thursday, October 11, 12

15. Thursday, October 11, 12

16. Context is everything Thursday, October 11, 12

17. How do we use context? Thursday, October 11, 12

18. Context Your Big Dumb Data !!! Thursday, October 11, 12

19. Human brain + med school Radiographic Imagery Diagnoses Thursday, October

    11, 12

20. Signal Processing VLA Output E.T. Thursday, October 11, 12

21. Network Data Application Behavior Application Topology Signal Processing Expert Brain

    Thursday, October 11, 12

22. dimensions (11) epoch seconds epoch minutes epoch hours node id

    source ip source port dest ip dest port interface country network/asn measurements (8) egress packets egress octets ingress packets ingress octets retransmits errors app-rtt handshake-rtt Thursday, October 11, 12

23. Case Study #1 GC-Death of a distributed JVM application Thursday,

    October 11, 12

24. Thursday, October 11, 12

25. Case Study #2 Symptoms: - Latent Riak handoff - Cluster

    throughput bottoming out Thursday, October 11, 12

26. Thursday, October 11, 12

27. busy_dist_port Thursday, October 11, 12

28. +zdbbl 8192 Thursday, October 11, 12

29. Thursday, October 11, 12

30. Case Study #3 Bringing a dead riak node back online

    Thursday, October 11, 12

31. Thursday, October 11, 12

32. Thursday, October 11, 12

33. Thursday, October 11, 12

34. Case Study #4 Retransmits 10% of total network throughput Thursday,

    October 11, 12

35. Thursday, October 11, 12

36. var put: HttpPut = null try { // ... put

    data } catch { case e: Exception => // ... handle exception } finally { if(put != null) { put.abort() } } Thursday, October 11, 12

37. var put: HttpPut = null try { // ... put

    data } catch { case e: Exception => // ... handle exception } finally { if(put != null) { put.abort() } } Thursday, October 11, 12

38. abort public void abort() Description copied from interface: HttpUriRequest Aborts

    execution of the request. Source: http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/client/methods/HttpRequestBase.html#abort() THANKS Thursday, October 11, 12

39. 129 public void abort() { 130 ClientConnectionRequest localRequest; 131 ConnectionReleaseTrigger

    localTrigger; 132 133 this.abortLock.lock(); 134 try { 135 if (this.aborted) { 136 return; 137 } 138 this.aborted = true; 139 140 localRequest = connRequest; 141 localTrigger = releaseTrigger; 142 } finally { 143 this.abortLock.unlock(); 144 } 145 146 // Trigger the callbacks outside of the lock, to prevent 147 // deadlocks in the scenario where the callbacks have 148 // their own locks that may be used while calling 149 // setReleaseTrigger or setConnectionRequest. 150 if (localRequest != null) { 151 localRequest.abortRequest(); 152 } 153 if (localTrigger != null) { 154 try { 155 localTrigger.abortConnection(); 156 } catch (IOException ex) { 157 // ignore 158 } 159 } 160 } Thursday, October 11, 12

40. Thursday, October 11, 12

41. augmented intelligence precedes artificial intelligence Thursday, October 11, 12

42. 1895 Wilhelm Röntgen discovers X-Rays First medical use of x-rays

    in human imaging takes place one month later Thursday, October 11, 12

43. 1895 Wilhelm Röntgen discovers X-Rays First medical use of x-rays

    in human imaging takes place one month later 1905 First English text on chest radiography Thursday, October 11, 12

44. 1895 Wilhelm Röntgen discovers X-Rays First medical use of x-rays

    in human imaging takes place one month later 1920 1905 First English text on chest radiography Society of Radiographers formed Thursday, October 11, 12

45. Recognition of radiology as a formal medical discipline was a

    cultural problem, not a technology problem http://www.bshr.org.uk/page13.html Thursday, October 11, 12

46. If you want to talk to me about the query

    language used to ask questions of the network data we collect at Boundary talk to me after or hit me up on twitter. @d2fn github.com/dietrichf Thursday, October 11, 12

47. Find 45 minutes of total traffic seen on meters 1,

    2, 226, & 301 starting 18 hours ago broken down by peer ip retain top 10 by the ratio of retransmits to packets get volume_1s_meter_ip [ meter in {1, 2, 226, 301}; epochMillis from -18h for 45m; ] categorize sum(ingress) as ingress, sum(egress) as egress, sum(ingressPackets + egressPackets) as packets, sum(retransmits) as retransmits, mean(appRttUsec/1000) as appRttMs by epochMillis, ip retain top 10 per epochMillis on retransmits/packets Thursday, October 11, 12

48. Find 45 minutes of total traffic seen on meters 1,

    2, 226, & 301 starting 18 hours ago broken down by peer ip retain top 10 by the ratio of retransmits to packets get volume_1s_meter_ip [ meter in {1, 2, 226, 301}; epochMillis from -18h for 45m; ] categorize sum(ingress) as ingress, sum(egress) as egress, sum(ingressPackets + egressPackets) as packets, sum(retransmits) as retransmits, mean(appRttUsec/1000) as appRttMs by epochMillis, ip retain top 10 per epochMillis on retransmits/packets Thursday, October 11, 12

49. Find 45 minutes of total traffic seen on meters 1,

    2, 226, & 301 starting 18 hours ago broken down by peer ip retain top 10 by the ratio of retransmits to packets get volume_1s_meter_ip [ meter in {1, 2, 226, 301}; epochMillis from -18h for 45m; ] categorize sum(ingress) as ingress, sum(egress) as egress, sum(ingressPackets + egressPackets) as packets, sum(retransmits) as retransmits, mean(appRttUsec/1000) as appRttMs by epochMillis, ip retain top 10 per epochMillis on retransmits/packets Thursday, October 11, 12

50. Find 45 minutes of total traffic seen on meters 1,

    2, 226, & 301 starting 18 hours ago broken down by peer ip retain top 10 by the ratio of retransmits to packets get volume_1s_meter_ip [ meter in {1, 2, 226, 301}; epochMillis from -18h for 45m; ] categorize sum(ingress) as ingress, sum(egress) as egress, sum(ingressPackets + egressPackets) as packets, sum(retransmits) as retransmits, mean(appRttUsec/1000) as appRttMs by epochMillis, ip retain top 10 per epochMillis on retransmits/packets Thursday, October 11, 12

51. Find 45 minutes of total traffic seen on meters 1,

    2, 226, & 301 starting 18 hours ago broken down by peer ip retain top 10 by the ratio of retransmits to packets get volume_1s_meter_ip [ meter in {1, 2, 226, 301}; epochMillis from -18h for 45m; ] categorize sum(ingress) as ingress, sum(egress) as egress, sum(ingressPackets + egressPackets) as packets, sum(retransmits) as retransmits, mean(appRttUsec/1000) as appRttMs by epochMillis, ip retain top 10 per epochMillis on retransmits/packets Thursday, October 11, 12

52. Find 45 minutes of total traffic seen on meters 1,

    2, 226, & 301 starting 18 hours ago broken down by peer ip retain top 10 by the ratio of retransmits to packets get volume_1s_meter_ip [ meter in {1, 2, 226, 301}; epochMillis from -18h for 45m; ] categorize sum(ingress) as ingress, sum(egress) as egress, sum(ingressPackets + egressPackets) as packets, sum(retransmits) as retransmits, mean(appRttUsec/1000) as appRttMs by epochMillis, ip retain top 10 per epochMillis on retransmits/packets Thursday, October 11, 12

53. Find 45 minutes of total traffic seen on meters 1,

    2, 226, & 301 starting 18 hours ago broken down by peer ip retain top 10 by the ratio of retransmits to packets get volume_1s_meter_ip [ meter in {1, 2, 226, 301}; epochMillis from -18h for 45m; ] categorize sum(ingress) as ingress, sum(egress) as egress, sum(ingressPackets + egressPackets) as packets, sum(retransmits) as retransmits, mean(appRttUsec/1000) as appRttMs by epochMillis, ip retain top 10 per epochMillis on retransmits/packets Thursday, October 11, 12

54. Find 45 minutes of total traffic seen on meters 1,

    2, 226, & 301 starting 18 hours ago broken down by peer ip retain top 10 by the ratio of retransmits to packets get volume_1s_meter_ip [ meter in {1, 2, 226, 301}; epochMillis from -18h for 45m; ] categorize sum(ingress) as ingress, sum(egress) as egress, sum(ingressPackets + egressPackets) as packets, sum(retransmits) as retransmits, mean(appRttUsec/1000) as appRttMs by epochMillis, ip retain top 10 per epochMillis on retransmits/packets Thursday, October 11, 12