Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What's Brewing in Beats?

Elastic Co
March 07, 2017

What's Brewing in Beats?

Beats are a family of lightweight shippers that send data from edge machines to Elasticsearch. They started with a single Beat – Packetbeat – for network data, and have since expanded to four additional Elastic Beats for log files, metrics, windows event logs, and availability monitoring, and more than 30 community-created Beats for all kinds of operational data.

In this session, Monica and Tudor will walk you through the latest Beats and their features. You will learn about the new Filebeat modules, which simplify the collection and parsing of common log files down to a single command. You will also see Heartbeat in action, our newest Beat responsible for uptime monitoring.

Tudor Golubenco l Beats Creator & Tech Lead l Elastic
Monica Sarbu l Beats Creator & Team Lead l Elastic

Elastic Co

March 07, 2017
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. What’s brewin’ in Beats Monica Sarbu @monicasarbu Tudor Golubenco @tudor_g

  2. 2 The road so far

  3. Recap of our road so far 3 Packetbeat Network data

  4. Recap of our road so far 4 Packetbeat Network data

    libbeat Beats library
  5. Recap of our road so far 5 Packetbeat Network data

    libbeat Beats library Topbeat System statistics
  6. Recap of our road so far 6 Packetbeat Network data

    libbeat Beats library Filebeat Log files Topbeat System statistics
  7. Recap of our road so far 7 Packetbeat Network data

    libbeat Beats library Filebeat Log files Winlogbeat Windows Event Logs Topbeat System statistics
  8. Recap of our road so far 8 Packetbeat Network data

    libbeat Beats library Filebeat Log files Winlogbeat Windows Event Logs Topbeat System statistics +40 community Beats
  9. Recap of our road so far 9 Packetbeat Network data

    libbeat Beats library Filebeat Log files Winlogbeat Windows Event Logs +40 community Beats Metricbeat Metrics
  10. Recap of our road so far 10 Packetbeat Network data

    libbeat Beats library Filebeat Log files Winlogbeat Windows Event Logs Heartbeat Uptime monitoring +40 community Beats Metricbeat Metrics
  11. What have we learned

  12. Out-of-the-box experience For metrics and network traffic 12

  13. What about logs? 13 Configure Filebeat paths & multiline Tune

    the Elasticsearch template Write Logstash Grok patterns Create Kibana dashboards
  14. What about logs? 14 Configure Filebeat paths & multiline Tune

    the Elasticsearch template Write Logstash Grok patterns Create Kibana dashboards very powerful, but complex ☹
  15. Grok patterns in Ingest Node 15 Create Kibana dashboards Write

    Ingest Grok patterns and Elasticsearch template I N G E S T Configure Filebeat paths & multiline
  16. Grok patterns in Ingest Node 16 Create Kibana dashboards Write

    Ingest Grok patterns and Elasticsearch template I N G E S T Configure Filebeat paths & multiline duplicated effort ☹
  17. Filebeat modules 17 Filebeat configuration Ingest pipelines Elasticsearch template Kibana

    dashboards
  18. Visualize your Apache logs in minutes 18 $ filebeat -setup

    -modules=apache2
  19. … and more to come 19 Filebeat configuration Ingest pipelines

    Elasticsearch template Kibana dashboards ML jobs Watcher alerts
  20. Modules … modules everywhere 20 MySQL Packetbeat MySQL Filebeat MySQL

    Metricbeat queries slow queries stats
  21. Demo

  22. Recently in Metricbeat

  23. Metricbeat modules 23 MySQL Memcache PHP-FPM CEPH Zoo keeper Golang

    Docker Apache Kafka HAProxy System Redis Couchbase NGINX Postgres Prometheus Jolokia
  24. What about monitoring your apps?

  25. host Instrument using Prometheus 25 Your app push metricbeat pull

  26. host JMX monitoring with Jolokia 26 metricbeat pull push Your

    app JVM
  27. host Profiling Go programs 27 Your app push metricbeat pull

  28. What if your application is running in Docker?

  29. • cgroup metrics • Docker API Docker metrics 29

  30. Run Metricbeat in a container 30 App1 App2 App3 Host

  31. Configuration reloading 31 /etc/metricbeat/conf.d/ metricbeat App1 App2 App3

  32. Monitor network connections You can now see who is talking

    with whom and how much 32
  33. What if your application is running in the cloud?

  34. Add cloud metadata add_cloud_metadata Beat processor 34

  35. I just called … to say … HTTP GET Heartbeat

  36. Heartbeat - Ping all the things 36 host Your app

    OS TCP/TLS connect ICMP ping HTTP/S request
  37. • Round Trip Times: • resolve • icmp • tcp_connect

    • socks5_connect • tls_handshake • http Heartbeat metrics 37
  38. Many to many 38

  39. Heartbeat - Ping all IPs behind a DNS name 39

    h ELB
  40. Coming up next

  41. Because the Beats also need a bit of monitoring love

    Central monitoring for Beats 41 production monitoring forwards monitoring data Specialized monitoring UI
  42. Make upgrades easier 42 filebeat-2017.01.14 Filebeat 5.x: filebeat-6.0.0-2017.01.14 Filebeat 6.0:

  43. Improve Metricbeat dashboards

  44. Future themes

  45. 45 Enable more security use cases Photo credit: https://www.flickr.com/photos/charbelakhras/2535524197

  46. 46 MOAR MODULES

  47. 47 Add a touch of magic

  48. 48 De-duplicate

  49. Central configuration management 49

  50. More Beats topics on Wednesday and Thursday At Elastic{ON}

  51. 51 Andrew Monitoring Docker with Metricbeat Wednesday, 4:40 pm Spotlight

    Theater Be there to learn how to monitor your containers with Beats
  52. 52 Nicolas & Steffen Ship Your Own Data: Tailoring Beats

    to Your Use Case Thursday, 11am Stage B Be there to learn how to extend Beats
  53. 53 Chris BoF: Kibana Visualizations Wednesday, 1:15 pm Spotlight Theater

    Demo Time series visualization builder in the Birds of a Feather session
  54. 54 Thanks! Meet us at the AMA booth Sept 2016