Ship Your Own Data: Tailoring Beats to Your Use Case

Beats is a little bit like LEGO: You can use each Beat itself as a building block to cover your needs, but at the same time each Beat consists of different reusable and extendable elements. This makes it possible for developers and operators to combine and extend Beats in different ways.

In this talk, Nicolas and Steffen will introduce developers, operators, and Beats users to the internals of the Elastic Beats. The knowledge gained will help with making informed decisions on how to extend Beats to deal with your particular use case if it’s not fully-covered by the existing Beats features.

Nicolas Ruflin l Software Engineer l Elastic
Steffen Siering l Software Engineer l Elastic

Transcript

1. Elastic 09.03.2017 @urso, @ruflin Ship your own data Tailoring Beats to your use cases Steffen Siering & Nicolas Ruflin
2. Agenda 2 1 Intro: Beats and the Elastic Stack 2 Beats Internals 3 Demo 4 Beats Framework 5 What's next?
3. Beats 3 Logstash Beats Log Files Metrics Wire Data more Elasticsearch Kafka Redis Messaging Queue
4. Beats are shippers that collect and ship all kinds of data to Elasticsearch
5. • Filebeat: Logs • Metricbeat: Metrics • Packetbeat: Transaction logs + Flow metrics • Winlogbeat: Windows events • Heartbeat: Network stats Which data are we shipping so far? 5 +40 community Beats
6. We want MORE data!
7. 7 • Beats is like Lego blocks • Learn the building blocks • Build your own • Extend existing beat
8. Beats extensions points 8 Processors Outputs Protocol Analyzer Packetbeat Modules Filebeat Modules / Metricsets Metricbeat Monitors Heartbeat Libbeat Community Beat
9. No Go No Problem
10. Beats Internals How to Extend Beats
11. Agenda 11 Beats Internals 1 libbeat: Processors and Publisher Pipeline 2 Go Modules 3 fields.yml 4 Beats in Detail
12. Beats Internals Libbeat
13. • Framework Beats are build upon • Processors • Outputs • Publisher Pipeline • Configuration • Logging • Internal Metrics Features - libbeat 13
14. Publisher Pipeline - libbeat 14 flush Processors PublishEvent enqueue Buffer Publisher Pipeline
15. Publisher Pipeline - libbeat 15 Work Queue Retry Queue Worker flush signal send ACK Processors PublishEvent enqueue signal Buffer Publisher Pipeline Output
16. Publisher Pipeline - libbeat 16 • PublishEvent(s) Options: • Guaranteed = do not drop events • Sync = blocking publish • Signal = (async) Signal Callback Work Queue Retry Queue Worker flush signal send ACK Processors PublishEvent enqueue signal Buffer Work Queue Retry Queue Worker flush signal send ACK Processors PublishEvent enqueue signal Buffer Publisher Pipeline Output
17. Processors - libbeat 17 type Processor interface { Run(event common.MapStr) (common.MapStr, error) String() string }
18. Outputs - libbeat 18 type Outputer interface { PublishEvent(sig op.Signaler, opts Options, data Data) error Close() error } type BulkOutputer interface { Outputer BulkPublish(sig op.Signaler, opts Options, data []Data) error }
19. Beats Internals Go Modules / (Static) Plugins
20. Configuration - Modules 20 Registry
21. • Modules register in package `init`. • Beats import Module Packages at build time Configuration - Modules 21 Registry Module register
22. • Modules register in package `init`. • Beats import Module Packages at build time Configuration - Modules 22 Registry Module register Loader query common.Config
23. • Modules register in package `init`. • Beats import Module Packages at build time • Loader instantiates Module based on config Configuration - Modules 23 Registry Module register Loader query instantiate common.Config
24. Configuration - Modules 24 # dictionary style modules output: elasticsearch: hosts: ['localhost:9200'] <module name>: <module settings> # list style modules metricbeat.modules: - module: <module name> <module settings>
25. Configuration - Modules 25 type moduleConfig struct { … } var defaultConfig = moduleConfig{ … } func init() { processors.RegisterPlugin(“name”, New) }
26. Configuration - Modules 26 func New(cfg *common.Config) (processors.Processor, error) { config := defaultConfig if err := cfg.Unpack(&config); err != nil { return nil, err } … }
27. Main-Import-Trick 27 package main import ( “os" “github.com/elastic/beats/libbeat/beat" “github.com/elastic/beats/metricbeat/beater” // load my plugins _ “my/module/package/name” ) func main() { if err := beat.Run(“mymetricbeat”, "", beater.New); err != nil { os.Exit(1) } }
28. Beats Internals fields.yml
29. • For all Events generated by Beats we provide: • Elasticsearch index template • Kibana index pattern • Documentation! ‣ common definition and documentation in fields.yml Introduction - fields.yml 29
30. Syntax - fields.yml 30 field ::= name: <field name> type: <type> [format: <format>] description: <docstring> [fields: <fields list>] # 'type' must be group fields list ::= - <field> - <field> ... type ::= group, long, keyword format ::= bytes, percent, ...
31. Example - fields.yml 31 name: memory type: group description: > `memory` contains local memory stats. fields: - name: total type: long format: bytes description: > Total memory
32. Beats Internals Filebeat
33. Filebeat 33 Crawler Prospector Harvester Spooler Publisher Registrar Libbeat Filesystem
34. Filebeat 34 Crawler Harvester Spooler Publisher Registrar Libbeat Filesystem Prospector manage
35. Filebeat 35 Crawler Spooler Publisher Registrar Libbeat Prospector Harvester manage scan Filesystem
36. Filebeat 36 Crawler Publisher Registrar Libbeat Prospector Harvester Filesystem Spooler send Events read send Events
37. Filebeat 37 Crawler Registrar Filesystem Spooler Harvester Publisher Libbeat flush publish
38. Filebeat 38 Crawler Filesystem Harvester Publisher Libbeat Spooler Registrar ACK update sync State
39. Filebeat 39 Crawler Prospector Harvester Spooler Publisher Registrar Libbeat Filesystem manage manage send Events flush publish ACK update sync State read scan send Events
40. Filebeat Modules 40 Crawler Prospector Harvester Spooler Publisher Registrar Libbeat Filesystem manage manage send Events flush publish ACK sync State read scan send Events Modules {Configuration} configure configure configure Elasticsearch Ingest Pipeline update
41. • Pre-build set of configurations • Simplify: Collecting, Parsing, Visualization • Modules per Service • Modules combine multiple Filesets Filebeat Modules 41 Module Elasticsearch Ingest Pipeline Kibana Filebeat Fileset Fileset
42. Filebeat Modules 42 module/{module}/{fileset} ├── manifest.yml ├── config │ └── {fileset}.yml ├── ingest │ └── pipeline.json ├── _meta │ └── fields.yml └── test
43. Filebeat Modules 43 module_version: "1.0" var: - name: paths default: - /var/log/nginx/access.log* os.darwin: - /usr/local/var/log/nginx/access.log* os.windows: - c:/programdata/nginx/logs/*access.log* - name: pipeline default: ingest/pipeline.json ingest_pipeline: {{ .pipeline }} prospector: config/nginx-access.yml nginx/access/manifest.yml
44. • Source: Log Files • Extension points: Filebeat Modules (5.3) • Use as framework: not yet • Publisher Guarantees: send at least once • On back-pressure: wait Filebeat - Summary 44
45. Beats Internals Winlogbeat
46. Winglogbeat 46 Main EventLog start publish ACK Registrar Filesystem update sync State Poll API
47. • Source: Windows Event Log API / Event Logging API • Extension points: none • Use as framework: not yet • Publisher Guarantees: send at least once • On back-pressure: wait Winlogbeat - Summary 47
48. Beats Internals Metricbeat
49. Metricbeat 49 Modules Metricsets Libbeat System Apache Docker Redis Custom… Metricbeat Module Reloader
50. • Module • group Metricsets • matches Service Type • provide common helper functions for Metricsets • Metricset = Service Metrics Type • Name matches Service API name Metricbeat Modules 50 Module API pull parse normalize _meta/fields.yml _meta/docs.asciidoc _meta/kibana _meta/config.yml _meta/docs.asciidoc _meta/fields.yml Metricset Metricset
51. Metricset API 51 type EventFetcher interface { MetricSet Fetch() (common.MapStr, error) }
52. Metricset API 52 type EventsFetcher interface { MetricSet Fetch() ([]common.MapStr, error) }
53. • Source: Poll Services • Extension points: Modules and Metricsets • Use as framework: YES • Publisher Guarantees: Drop after max_retries (default 3) • On back-pressure: wait Metricbeat - Summary 53
54. Beats Internals Heartbeat
55. Heartbeat 55 Scheduler Monitors Resolve Tasks Ping Tasks Monitor Reloader Heartbeat Libbeat DNS HTTP ICMP TCP
56. Heartbeat Monitor API 56 type Job interface { Name() string TaskRunner } type TaskRunner interface { Run() (common.MapStr, []TaskRunner, error) }
57. • Source: Ping Services/Hosts • Extension points: Monitors • Use as framework: YES • Publisher Guarantees: Drop after max_retries (default 3) • On back-pressure: wait Heartbeat - Summary 57
58. Beats Internals Packetbeat
59. Packetbeat 59 Libbeat Network Sniffer Decoder IP layer TCP UDP Packet Analyzers Transaction Publisher Flows Publisher Flows Parse Correlate Publish Transport
60. Packetbeat 60 Libbeat Network Sniffer Decoder IP layer TCP UDP Packet Analyzers Transaction Publisher Parse Correlate Publish Transport • Protocol Analyzer log transactions Flows Publisher Flows
61. Packetbeat 61 Libbeat Network Sniffer Decoder IP layer TCP UDP Packet Analyzers Transaction Publisher Flows Publisher Flows Parse Correlate Publish Transport • Protocol Analyzer log transactions • Flows collect connection metrics
62. TCP Plugin Interface 62 type TCPPlugin interface { Parse(pkt *Packet, address *common.TCPTuple, dir uint8, state ProtocolData) ProtocolData // common protocol settings getter GetPorts() []int ConnectionTimeout() time.Duration // connection FIN/packet loss ReceivedFin(…) ProtocolData GapInStream(…) (state ProtocolData, drop bool) }
63. • Code generator: beats/packetbeat/scripts • contains README with sample Tutorial • Example: Cassandra by @medcl Packet Analyzers 63 Parse Correlate Publish Connection State Packet Connection State Message store uncorrelated messages in connection state
 or transaction table Create and Report Transaction Log Event Messages TCP module Packet Analyzer store parsing state in connection state New/Updated/Dropped Connection State Packetbeat
64. • Code generator: beats/packetbeat/scripts • contains README with sample Tutorial • Example: Cassandra by @medcl
 
 • Flow support for metrics in packet analyzer modules planned in #3444 Packet Analyzers 64 Parse Correlate Publish Connection State Packet Connection State Message store uncorrelated messages in connection state
 or transaction table Create and Report Transaction Log Event Messages TCP module Packet Analyzer store parsing state in connection state New/Updated/Dropped Connection State Packetbeat
65. • Source: Network Packets • Extension points: Protocol Analyzers • Use as framework: YES • Publisher Guarantees: Drop after max_retries (default 3) • On back-pressure: • Transactions: drop • Flows: wait Packetbeat - Summary 65
66. Recap 66 Processors Outputs Protocol Analyzer Packetbeat Modules Filebeat Modules / Metricsets Metricbeat Monitors Heartbeat Libbeat Community Beat
67. Summary 67 FILEBEAT
 WINLOGBEAT PACKETBEAT METRICBEAT
 HEARTBEAT Source Log Files
 Windows Event Logs Network Packets Poll Services Extension Points Filebeat Modules Protocol Analyzers Modules/Metricsets Use as Framework No (not yet) Yes Yes Publisher Guarantees send-at-least-once Drop after N retries
 (default: 3) Drop after N retries
 (default: 3) Back-pressure handling Wait Transactions: Drop
 Flows: Wait Wait
68. Demo: Build a Metricbeat Metricset
69. Raw beats vs module beat vs fork 69 RAW BEAT MODULE BEAT BEATS FORK Reason / Motivation own data model / logic own module / experiment contribute back Complexity / Flexibilty high medium low / medium Maintenance Costs moderate low depends Contribute Back community list community list / PR PR Packaging yes yes yes Which one fits you best?
70. • Packaging (zip, deb, rpm, ...) • Test Framework • Documentation • ES Index Templates • Kibana (Dashboards, Index Patterns) Batteries included 70
71. • More generators • Filebeat Prospector Types? • Improve Heartbeat extendibility • Pipeline unification • Simplify, Simplify What's next? 71
72. • Go 1.8 adds plugin support (Linux only + requires cgo) • Beats 5.3 adds experimental go plugins: • source code only • not available in 5.3 binaries Go Plugins (experimental) 72 func init() { mymodules.Register(“name”, New) } var Bundle = myplugins.Plugin(“name”, New)
73. 73 •Lets ship your data •The sky is the limit •Let the beats flow
74. • Birds of Feather session at 12:45 • AMA Booth • discuss.elastic.co More Questions? 74
75. www.elastic.co
76. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 76 Please attribute Elastic with a link to elastic.co