PSConf Asia 2018 - Monitoring 101 With Windows and PowerShell

The talk starts with some basic monitoring concepts and then applies these concepts to Microsoft Active Directory using PowerShell. Datadog is used as the monitoring tool.

Transcript

1. Matthew Hodgkins | Principal Engineer | @MattHodge | https://hodgkins.io
3. Agenda. • Why is monitoring important? • What does bad monitoring look like? • Concepts • Datadog • Monitoring Active Directory Demo • Good monitoring examples
4. Why. • Downtime costs are growing (smiles and $) • More complicated infrastructure • Who wants to work at night?
5. How do I know I have bad monitoring?
6. If your application... • Needs customers to call so you know it's broken • Has no alerts defined • Is giving your team alert fatigue • Has unmonitored dependencies
8. If you need to... • Create a ticket to have something monitored • Log into a server and open log files or event logs • Manually add servers to your monitoring system
11. The whole point. Are the things I own doing the business-y thing that they are supposed to be doing?
12. Concepts. • Logs vs Metrics • Causes vs Symptoms • Golden Metrics • Dependencies • Service Level Objectives
13. Logs vs Metrics. Metrics Logs • Trends over time • Retention • Smarter alerting • A single measure • Flexibility of what to store • Troubleshooting • Auditing • Correlating ID’s
14. Logs vs Metrics - Alerting.
15. Logs vs Metrics - Alerting.
16. Logs vs Metrics - Alerting.
17. Logs vs Metrics. • Combine them where possible • Alert on the metrics, dig into the logs
18. Causes vs Symptoms. • Causes have a technical basis • Symptoms have a basis in business
20. Causes vs Symptoms. • Add metrics and logs for causes • Alert on the symptom so you can always catch a problem • Helps reduce alert fatigue
21. Causes vs Symptoms - Example App.
22. Causes vs Symptoms - Examples. Bad Good Number of PHP Processes > 100 Search success rate > 99% Messages in Queue > 2000 Review added in last 60 minutes MSSQL CPU Usage > 90% Products page 95th percentile load time < 2 seconds Processed > 100,000 messages Pricing data updated < 24 hours ago
23. Golden Metrics. • Get these metrics first • SRE Book (Latency, Traffic, Errors, Saturation) • USE (Utilization, Saturation, Errors) • RED (Rate, Error, Duration)
24. Types of Metrics - Gauge.
25. Types of Metrics - Counter.
26. Types of Metrics - Histogram.
27. Dependencies. • You’re probably going to have them • Measure them from your applications perspective • Stop with “it feels slow”
28. Service Level Objectives. • A target range for a metric • Set expectations for your service • Your applications should have them defined • Base on golden metrics • Should be discussed with your stakeholders
30. Concepts. • Logs vs Metrics ✅ • Causes vs Symptoms ✅ • Golden Metrics ✅ • Dependencies ✅ • Service Level Objectives ✅
32. Datadog. • Cloud based • Single binary to install an agent • Dashboarding, alerting, metrics, logs • Simple • Free for 5 instances
33. Monitoring Active Directory. • Causes ◦ Basic Metrics + Windows Services
34. Demo 1 - Install Datadog.
35. Active Directory - Potential Causes. • Basic System Metrics • Windows Services
36. Demo 1a - Basic System Metrics.
37. Demo 2 - Monitor Windows Services.
38. Active Directory - SLO. • Traffic ◦ LDAP Searches / second
39. Demo 3 - Active Directory Integration.
40. Active Directory - SLO. • Errors ◦ Login success rate %
41. SLO - Login Success Rate %. • No in-built counters for this • Two Windows Event Logs: ◦ 4624 - Account was successfully logged on ◦ 4625 - Account failed to log on • So how do we get this to Datadog?
43. DogstatsD. • Datadog agent listens on UDP 8125 • Accepts strings like this: active_directory.login_events:23|c|#event_id:4624 active_directory.login_events:47|c|#event_id:4625
44. Event Logs to Datadog Metric.
45. Demo 4 - PowerShell Integration.
46. Endless Options with PowerShell. • Count number of domain administrators • Track dcdiag Errors • Monitor replication latency per DC • Account creation / lockouts
47. Active Directory - SLO’s. • Latency ◦ ATQ Request Latency • Saturation ◦ ATQ Outstanding Requests
48. ATQ. https://blogs.technet.microsoft.com/askds/2014/10/24/understanding-atq-performance-counters-yet-another-twist-in-the-world-of-tlas/
49. Demo 5 - Custom WMI.
50. Make It Pretty.
52. Application Landscape
53. Drilldowns to possible causes
54. Knowing before 3rd parties
55. Alerts on Business-y Things (SLO’s)
56. Summary. • Why is monitoring important? ✅ • What does bad monitoring look like? ✅ • Concepts ✅ • Datadog ✅ • Monitoring Active Directory Demo ✅ • Good monitoring examples ✅
57. Matthew Hodgkins | @MattHodge | https://hodgkins.io