Collecting the right data to monitor your infrastructure

In the world of containers and microservices, where your infrastructure consists of thousands of containers that are changing continuously, monitoring your infrastructure become a challenge. At the beginning, you collect the logs from all your servers to help you understand when there is a failure in your system, but logs are not always available, think of when the service is down. To prevent a failure, you would need to monitor the status of your services, and the health of the server where the service is running. In a distributed environment where the microservices communicate between them via APIs, it’s important to be able to visualize the traffic exchanged between your microservices for troubleshooting purposes. This talk will present how you can use the open source tools and in particular the Elastic Beats to offer a broad visibility into your network by collecting different kinds of operational data from all your services into a central point in Elasticsearch, and then build Dashboards with Kibana.

Transcript

1. Collecting the right data to monitor your infrastructure by Monica Sarbu
2. 2 About me Monica Sarbu Team lead, Beats Working at Elastic for 2 years Building monitoring tools for 10+ years Likes traveling Starting Beats 4 years ago Enjoys writing code @monicasarbu
3. Beats is a family of lightweight shippers that collect and ship all kinds of operational data to Elasticsearch
4. The Beats family 4 Packetbeat Network data libbeat Beats library Filebeat Log files Winlogbeat Windows Event Logs Heartbeat Uptime monitoring +40 community Beats Metricbeat Metrics
5. Beats, multiple use cases Se Security Uptime monitoring Service monitoring Application monitoring System monitoring Network monitoring
6. Use case: application monitoring
7. Application logs • install Filebeat on the application servers to forward the logs in a raw format • easy when application writes logs in json format, otherwise it requires log parsing • Parsing can be done with Ingest Node plugin of Elasticsearch or Logstash by defining Grok patterns
8. Application metrics • Logs can be a source of metrics • instrument your application to get metrics
9. host Instrument using Prometheus 9 Your app push metricbeat pull
10. host JMX monitoring with Jolokia 10 metricbeat pull push Your app JVM
11. host Profiling Go programs 11 Your app push metricbeat pull
13. Windows application event logs • collect them with Winlogbeat https://en.wikipedia.org/wiki/Event_Viewer
14. Application running in Kubernetes • Each Beat can use the kubernetes processor to enrich events with metadata coming from the Kubernetes Pod from which the event originated like: pod name, namespace, labels
15. Application running in Cloud • Each Beat can use the add_cloud_metadata processor to add cloud metadata in the exported events
16. Use case: service monitoring
17. Service logs • out-of-the-box experience with Filebeat modules $ filebeat -setup -modules=apache2 $ filebeat -setup -modules=apache2,nginx
19. Service metrics • using Metricbeat modules MySQL Memcache PHP-FPM CEPH Zoo keeper Docker Apache Kafka HAProxy HTTP Redis Couchbase NGINX Postgres Kubernetes Vsphere Dropwizard
20. Docker metrics • use Metricbeat to collect metrics about your Docker containers either interrogating directly the Docker API or using cgroup.
22. Kubernetes metrics • use the kubernetes module in Metricbeat to get details about the running containers and the available pods
24. Time Series Visual Builder • Kibana plugin 5.4+ • specific to display Metricbeat data
26. Use case: system monitoring
27. System metrics • collect periodically metrics by installing Metricbeat on your system • Metricbeat system module: - CPU usage - Load - Memory usage - Desk IO - Filesystem - Network stats - Per process information - Per core information
29. Windows system event logs • collect them with Winlogbeat https://i-msdn.sec.s-msft.com/dynimg/IC500861.png
30. Use case: network monitoring
31. Monitor traffic exchanged between servers • by installing Packetbeat on your servers, you can decode the messages exchanged between them http mysql redis
32. Use case: uptime monitoring
33. Heartbeat - Ping all the things 33 host Your app OS TCP/TLS connect ICMP ping HTTP/S request
34. • Round Trip Times: • dns_resolve • icmp • tcp_connect • socks5_connect • tls_handshake • http Heartbeat metrics 34
35. Using a Logging and Monitoring system to discover security breaches as soon as possible
36. auth logs • SSH logins (password or publickey, IP, GeoIP) • failed sudo attempts • useradd / groupadd
38. Windows logons • logon failure events from the Security event log
40. auditd • watch file accesses • new network connections • new processes
41. Running processes • find suspicious commands
42. Network connections • outbound connections initiated by a process • processes in listening mode
43. DNS tunneling • Packetbeat monitors the DNS traffic to look for a high number of unique hostnames for a domain https://www.elastic.co/blog/detecting_dns_tunnels_with_packetbeat_and_watcher
44. In conclusion
45. metrics logs events dashboards machine learning reporting alerts Elasticsearch One storage for everything
46. Thank you!
47. Q&A We’re hiring!