Elastic{ON} 2018 - What's Brewing in Beats

Slide 1

Slide 1 text

Team Lead @Beats @monicasarbu What’s Brewing in Beats Monica Sarbu Tech Lead @Beats @tudor_g Tudor Golubenco

Slide 2

Slide 2 text

The History Beats family libbeat Packetbeat

Slide 3

Slide 3 text

The History libbeat Packetbeat Filebeat Winlogbeat Beats family

Slide 4

Slide 4 text

The History libbeat Packetbeat Filebeat Winlogbeat Metricbeat Heartbeat Beats family

Slide 5

Slide 5 text

The History libbeat Packetbeat Filebeat Winlogbeat Metricbeat Heartbeat Auditbeat Beats family

Slide 6

Slide 6 text

The History Beats family libbeat Community Beat Metricbeat Community Beat Community Beats +60

Slide 7

Slide 7 text

Growing Beats community 7 50M Cumulative downloads 3 Years

Slide 8

Slide 8 text

Docker & Kubernetes

Slide 9

Slide 9 text

9 With containers architecture, everything is a moving target

Slide 10

Slide 10 text

Slide 11

Slide 11 text

✓ pod ✓ node ✓ system ✓ container ✓ event ✓ volume Monitor Kubernetes cluster 11 ✓ state_container ✓ state_deployment ✓ state_node ✓ state_pod ✓ state_replicated Via the Kubernetes module in Metricbeat

Slide 12

Slide 12 text

Monitor services running inside containers 12 Metricbeat Filebeat Node n Logs Metrics Nginx

Slide 13

Slide 13 text

Logs, metrics, APM traces 13 API server pod watcher Pod start/stop events 418a913c7076 ……………… c626cfdf3861 ……………… e5563a7cb80e ……………… 73de79be045c ……………… updates Docker logs enrich enriched events Enhanced with Kubernetes metadata add_kubernetes_metadata

Slide 14

Slide 14 text

Autodiscover 14 Watch Docker or Kubernetes events and react to changes metricbeat.autodiscover: providers: - type: kubernetes host: ${HOSTNAME} templates: - condition.contains: kubernetes.container.name: nginx config: - module: nginx period: 10s metricsets: ["stubstatus"] hosts: ["${data.host}:8080"]

Slide 15

Slide 15 text

Monitor applications instrumented with Prometheus 15 Node 1 Metricbeat Node 2 Metricbeat Node n Metricbeat App App App pull pull pull

Slide 16

Slide 16 text

Kubernetes deployment 16 Deploy Filebeat or Metricbeat as DaemonSets Node 1 Metricbeat Filebeat Node 2 Metricbeat Filebeat Node n Metricbeat Filebeat Filebeat DaemonSet Metricbeat DaemonSet

Slide 17

Slide 17 text

17 Kubernetes, Docker, and Containers at Elastic Carlos Pérez-Aradros Software Engineer, Beats Thu 1 Mar, 10:30-11:15 Salon 1-7 Tyler Langlois Infrastructure Engineer

Slide 18

Slide 18 text

18 Monitoring Anything and Everything with Beats at eBay Vijay Samuel Senior Software Engineer @eBay Wed 28 Feb, 13:30-14:15 Salon 1-7

Slide 19

Slide 19 text

Curated UI for Kubernetes 19 Visualise the cluster and group by nodes or namespaces or pods

Slide 20

Slide 20 text

Infra UI demo by Chris Cowan

Slide 21

Slide 21 text

Auditbeat

Slide 22

Slide 22 text

Auditbeat modules 22 Auditd File integrity Watch your systems from the OS layer

Slide 23

Slide 23 text

Auditbeat: Linux kernel auditing 23 Auditd Like auditd, but perfectly integrated with the Elastic stack • Indexes directly into Elasticsearch • Correlates kernel audit events • Resolves user IDs to user names

Slide 24

Slide 24 text

Auditbeat: file integrity 24 File integrity Index file hashes and watch changes • Performs an initial scan of all files • Computes hashes of the watched files • Watches for file changes • Linux, macOs, Windows

Slide 25

Slide 25 text

Equifax data breach 25 What if they had Auditbeat installed?

Slide 26

Slide 26 text

Auditbeat demo

Slide 27

Slide 27 text

Why Auditbeat? 27 • Detects short lived processes and connections • Works on older kernels (2.6+) • Doesn’t require a kernel module

Slide 28

Slide 28 text

Recent & Next in Beats

Slide 29

Slide 29 text

Xpack (Basic) Beats Monitoring (6.2) 29

Slide 30

Slide 30 text

Xpack (Basic) Beats Monitoring (6.2) 30

Slide 31

Slide 31 text

Central Configuration (6.x) 31 BoF: Beats monitoring and central configuration @Thursday 9:30

Slide 32

Slide 32 text

Secrets Keystore (6.2) 32 filebeat modules enable system filebeat keystore add cloud.auth filebeat -e -E ‘cloud.auth=${cloud.auth}’ \ -E ‘cloud.id=…’

Slide 33

Slide 33 text

Add Data UI (6.2) 33

Slide 34

Slide 34 text

Add Data UI (6.2) 34

Slide 35

Slide 35 text

Spooling on disk (6.3) 35 Disk queue PublishEvent () Output memqueue batch ACK ACK ACK libbeat

Slide 36

Slide 36 text

• Runs as an AWS Lambda function • Collects Cloudwatch Logs, Cloudtrail logs, logs from S3 or Kinesis Serverless shippers (6.x) 36

Slide 37

Slide 37 text

Learn more at Elastic{ON}

Slide 38

Slide 38 text

38 Build Your Own Filebeat Module Noémi Ványi Software Engineer, Beats Wed 28 Feb, 13:55-14:15 Golden Gate C

Slide 39

Slide 39 text

39 Questions?

Slide 40

Slide 40 text

www.elastic.co

Slide 41

Slide 41 text

Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 41 Please attribute Elastic with a link to elastic.co