Before you reinvent the wheel... • Is there an existing plugin? • Can I extend an existing plugin? • Has anyone (outside the Logstash community) solved this problem? • Can I piggy-back on that? • Depends on the license, of course • Is this going to be internal/private or shared/open-source? • Will this be better as a filter or a codec? • Am I reproducing existing functionality? 16
Examples • Dedicated plugins vs. codecs for Netflow • Add a grok pattern vs. write a dedicated plugin • TCP/UDP inputs and outputs already exist. • Write a codec if sending/receiving data via TCP/UDP. • http and http_poller input plugins already exist. • Use these if your project sends http, or provides data via http • JDBC input plugin vs. custom plugin 17
For a new plugin, start with the examples • One of each kind for you to choose from • https://github.com/logstash-plugins/logstash-input-example/ • https://github.com/logstash-plugins/logstash-filter-example/ • https://github.com/logstash-plugins/logstash-output-example/ • https://github.com/logstash-plugins/logstash-codec-example/ 21
Create a GitHub repo for your new plugin Each Logstash plugin lives in its own GitHub repository. To create a new repository for your plugin: 1. Log in to GitHub. 2. Click the Repositories tab. You’ll see a list of other repositories you’ve forked or contributed to. 3. Click the green New button in the upper right. 22
Create a GitHub repo for your new plugin 4. Specify the following settings for your new repo: • Repository name — a unique name of the form logstash-TYPE- pluginname. • Public or Private — your choice, but the repository must be Public if you want to submit it as an official plugin. • Initialize this repository with a README — enables you to immediately clone the repository to your computer. 5. Click Create Repository. 23
More repository prep • Clone your repository • Clone the example repository that is of the same type • Delete the .git directory from the example repository • Delete everything in your repository, except the .git directory • cp -R everything in the example repository you cloned to your newly wiped plugin directory. • Rename the following files to fit your plugin type and name: • logstash-TYPE-example.gemspec • example.rb • example_spec.rb 24
Coming Soon: generate This will be much more simple... 26 $ bin/logstash-plugin generate --help Usage: bin/logstash-plugin generate [OPTIONS] Options: --type TYPE Type of the plugin {input, filter, codec, output}s --name PLUGIN Name of the new plugin --path PATH Location where the plugin skeleton will be created (default: "${PWD}") -h, --help print help
Plugin Anatomy The "bones" 28 # encoding: utf-8 require "logstash/inputs/base" require "logstash/namespace" require "stud/interval" require "socket" # for Socket.gethostname # Add any asciidoc formatted documentation here # Generate a repeating message. # # This plugin is intended only as an example. class LogStash::Inputs::Example < LogStash::Inputs::Base
Plugin Anatomy Configuration & Auto-documentation 29 # This is where your plugin name goes config_name "example" # If undefined, Logstash will complain, even if codec is unused. default :codec, "plain" # The message string to use in the event. config :message, :validate => :string, :default => "Hello World!" # Set how frequently messages should be sent. # # The default, `1`, means send a message every second. config :interval, :validate => :number, :default => 1
Plugin Anatomy Filter config example 32 # Setting the config_name here is required. This is how you # configure this filter from your Logstash config. # # filter { # example { message => "My message..." } # } config_name "example" # Replace the message with this value. config :new_message, :validate => :string, :default => "Hello World!"
Plugin Anatomy Filter method example 33 public def filter(event) if @message # Replace the event message with our message as configured # in the config file. # OLD WAY: event["message"] = @new_message # New WAY below: event.set('message') = @new_message end # filter_matched should go in the last line of our code filter_matched(event) end # def filter end # class LogStash::Filters::Example
Plugin Anatomy Output method example 34 public # Takes an array of events def multi_receive(events) # code goes here. # Be sure to loop through all events in the array end # def multi_receive public # Needed for logstash < 2.2 compatibility # Takes events one at a time def receive(event) # code goes here. end # def receive
Plugin Anatomy Output threading consideration 35 # If declared logstash will only allow a single instance of this plugin # to exist, regardless of how many CPU cores logstash detects. This is best # used in cases like the File output, where separate threads writing to a # single file would only cause problems. # # respond_to? check needed for backwards compatibility with < 2.2 Logstashes declare_workers_not_supported! if self.respond_to? (:declare_workers_not_supported!)
Plugin Anatomy Output threading consideration 36 # If declared threadsafe logstash will only ever create one # instance of this plugin per pipeline. # That instance will be shared across all workers # It is up to the plugin author to correctly write concurrent code! # # respond_to? check needed for backwards compatibility with < 2.2 Logstashes declare_threadsafe! if self.respond_to?(:declare_threadsafe!)
Publish your plugin • https://www.elastic.co/guide/en/logstash/current/submitting-plugin.html • Go solo (self-publish to RubyGems), or • Submit to the logstash-plugins repository on GitHub • Acceptance Guidelines • Code Review • Must include tests • Submit an issue at https://github.com/elastic/logstash 41
untergeek
munetoshi
hollycummins
track3jyo
fendo181
ivargrimstad
koba04
mozu1206
shioyama
konpay
totokit4
hpgrahsl
robhawkes
chrislema
dougneiner
bryan
addyosmani
trishagee
aarron
chriscoyier
ammeep
andyhume
akosma
reverentgeek