When monitoring becomes analytics

As systems get larger, monitoring your servers becomes a thankless (and often useless) task. What becomes more valuable is thinking about your monitoring task as an analytics challenge. What services do I care about? How do I abstract out of servers into thinking about the user experience? How can I relate infrastructure performance to business questions? These questions are the next frontier for devops analytics. We’ll cover these kinds of questions and more in this session, including:

• Moving to monitoring systems instead of servers
• Applying data science and statistics across operational information
• New ways to explore complex system data
• Bringing together metrics and events for a unified look at your system

We’ll not only take a look at a new approach to thinking about your data, but also new underlying technology to analyze operations data. We’ll cover streaming, in-memory computing, and dataflow.

Transcript

1. When monitoring becomes analytics Chris & Mike
2. What we’re going to talk about • How to simply structure infrastructure to collect metrics at scale • Metrics collection and analysis at scale • Understanding metrics using streaming analytics • Joining in additional context to data streams
3. ◦ Delivering Faster Better ▪ Fast, Efficient Web Delivery at Global Scale ▪ DDoS Protection and SSL Delivery @ Scale ▪ Simplified Workflows to Optimize App Acceleration
4. Landscape Limelight Networks • Top tier CDN. We have a video platform & object storage on top • 10 tbps egress with our own international backbone • Global footprint with 100+ data centers Environment • Technology company, 24/7 engineering presence • Diverse, talented teams with variety of skills - systems, networking, & software ◦ Massive FreeBSD Edge ◦ Several flavors of *nix’s
6. Dataflow is like assembling legos to create streaming analytics. Built on Dataflow
7. Networks, Systems, and Services Deliver Faster Better
9. Operational overview Simplicity is prerequisite for reliability. — Edsger W. Dijkstra
10. Operational overview Poll or push Segmentation resilience and re-assignment Multiple storage options and/or strategies Analysis and alerting
11. Operational overview Poll or push Segmentation resilience and re-assignment Multiple storage options and/or strategies Analysis and alerting Intuitive UI?
12. The basics Metric :: Timestamp :: Value
13. The basics Metric :: Timestamp :: Value $ PORT=4242 $ SERVER=collector.llnw.net $ echo "net.if.out_em0_bytes_ 1356998400 91912520 fqdn=cdn388.sea. llnw.net proxy=SEA01" | nc ${SERVER} ${PORT}; $ echo "app.5min.client_http.hit_kbytes_out 1356998420 26786.90 fqdn=cdn388.sea.llnw.net port=80" | nc ${SERVER} ${PORT};
18. Challenge Context and situational awareness when performing analysis (or changes)
19. http://tinyurl.com/events-meet-metrics
20. http://tinyurl.com/events-meet-metrics
21. http://tinyurl.com/events-meet-metrics Doh!
22. Operational challenge • “Collect everything” is not always possible • Complex to perform aggregation
24. Information theory is a branch of applied mathematics, electrical engineering, and computer science involving the quantification of information. Information theory was developed by Claude E. Shannon to find fundamental limits on signal processing operations such as compressing data and on reliably storing and communicating data.
28. Wat?
29. Challenge(s) Avoid changing namespaces and categories to fit the system Opportunity Consistent naming/schema offer interfaces and natural places to join data together
30. http://tinyurl.com/deep-troubleshooting
31. http://tinyurl.com/deep-troubleshooting
32. Analysis tools Seasonality Back testing thresholds and alerting
33. http://tinyurl.com/anomaly-detection
34. http://tinyurl.com/anomaly-detection
35. Takeaways and lessons • Tiered and resilient infrastructure to flexibly collect metrics at scale • Don’t let backend systems dictate metrics namespace and categorization • Freedom to perform aggregates, transforms, and ad-hoc analytics • Situational awareness
36. Thank you
37. Chat with us! booth # 625 demmer@jut.io @jut_inc booth # 509 cchristensen@llnw.com @imetchrischris