Elastic{ON} 2018 - What's Brewing in Beats

Transcript

1. Team Lead @Beats @monicasarbu What’s Brewing in Beats Monica Sarbu Tech Lead @Beats @tudor_g Tudor Golubenco
2. The History Beats family libbeat Packetbeat
3. The History libbeat Packetbeat Filebeat Winlogbeat Beats family
4. The History libbeat Packetbeat Filebeat Winlogbeat Metricbeat Heartbeat Beats family
5. The History libbeat Packetbeat Filebeat Winlogbeat Metricbeat Heartbeat Auditbeat Beats family
6. The History Beats family libbeat Community Beat Metricbeat Community Beat Community Beats +60
7. Growing Beats community 7 50M Cumulative downloads 3 Years
8. Docker & Kubernetes
9. 9 With containers architecture, everything is a moving target
11. ✓ pod ✓ node ✓ system ✓ container ✓ event ✓ volume Monitor Kubernetes cluster 11 ✓ state_container ✓ state_deployment ✓ state_node ✓ state_pod ✓ state_replicated Via the Kubernetes module in Metricbeat
12. Monitor services running inside containers 12 Metricbeat Filebeat Node n Logs Metrics Nginx
13. Logs, metrics, APM traces 13 API server pod watcher Pod start/stop events 418a913c7076 ……………… c626cfdf3861 ……………… e5563a7cb80e ……………… 73de79be045c ……………… updates Docker logs enrich enriched events Enhanced with Kubernetes metadata add_kubernetes_metadata
14. Autodiscover 14 Watch Docker or Kubernetes events and react to changes metricbeat.autodiscover: providers: - type: kubernetes host: ${HOSTNAME} templates: - condition.contains: kubernetes.container.name: nginx config: - module: nginx period: 10s metricsets: ["stubstatus"] hosts: ["${data.host}:8080"]
15. Monitor applications instrumented with Prometheus 15 Node 1 Metricbeat Node 2 Metricbeat Node n Metricbeat App App App pull pull pull
16. Kubernetes deployment 16 Deploy Filebeat or Metricbeat as DaemonSets Node 1 Metricbeat Filebeat Node 2 Metricbeat Filebeat Node n Metricbeat Filebeat Filebeat DaemonSet Metricbeat DaemonSet
17. 17 Kubernetes, Docker, and Containers at Elastic Carlos Pérez-Aradros Software Engineer, Beats Thu 1 Mar, 10:30-11:15 Salon 1-7 Tyler Langlois Infrastructure Engineer
18. 18 Monitoring Anything and Everything with Beats at eBay Vijay Samuel Senior Software Engineer @eBay Wed 28 Feb, 13:30-14:15 Salon 1-7
19. Curated UI for Kubernetes 19 Visualise the cluster and group by nodes or namespaces or pods
20. Infra UI demo by Chris Cowan
21. Auditbeat
22. Auditbeat modules 22 Auditd File integrity Watch your systems from the OS layer
23. Auditbeat: Linux kernel auditing 23 Auditd Like auditd, but perfectly integrated with the Elastic stack • Indexes directly into Elasticsearch • Correlates kernel audit events • Resolves user IDs to user names
24. Auditbeat: file integrity 24 File integrity Index file hashes and watch changes • Performs an initial scan of all files • Computes hashes of the watched files • Watches for file changes • Linux, macOs, Windows
25. Equifax data breach 25 What if they had Auditbeat installed?
26. Auditbeat demo
27. Why Auditbeat? 27 • Detects short lived processes and connections • Works on older kernels (2.6+) • Doesn’t require a kernel module
28. Recent & Next in Beats
29. Xpack (Basic) Beats Monitoring (6.2) 29
30. Xpack (Basic) Beats Monitoring (6.2) 30
31. Central Configuration (6.x) 31 BoF: Beats monitoring and central configuration @Thursday 9:30
32. Secrets Keystore (6.2) 32 filebeat modules enable system filebeat keystore add cloud.auth filebeat -e -E ‘cloud.auth=${cloud.auth}’ \ -E ‘cloud.id=…’
33. Add Data UI (6.2) 33
34. Add Data UI (6.2) 34
35. Spooling on disk (6.3) 35 Disk queue PublishEvent () Output memqueue batch ACK ACK ACK libbeat
36. • Runs as an AWS Lambda function • Collects Cloudwatch Logs, Cloudtrail logs, logs from S3 or Kinesis Serverless shippers (6.x) 36
37. Learn more at Elastic{ON}
38. 38 Build Your Own Filebeat Module Noémi Ványi Software Engineer, Beats Wed 28 Feb, 13:55-14:15 Golden Gate C
39. 39 Questions?
40. www.elastic.co
41. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 41 Please attribute Elastic with a link to elastic.co