Save 37% off PRO during our Black Friday Sale! »

Writing Elastic Stack Plugins

Writing Elastic Stack Plugins

These slides were presented at OpenWest 2017:

The Elastic Stack consists of a few systems: Elasticsearch, Beats, Kibana, Logstash. In this talk, you will learn how to build plugins that span these products to form a cohesive user experience for custom analytic applications built on top of the stack. We will walk through what it looks like to
write Logstash plugin, Elasticsearch plugins, beats, and Kibana plugins.


Elastic Co

July 13, 2017


  1. @talevy Extending The Elastic Stack Tal Levy, Software Engineer

  2. The Elastic Stack 2 Kibana Elasticsearch Beats Logstash Security Alerting

    Monitoring Reporting X-Pack Graph Elastic Cloud
  3. - Shay Banon { } { “tagline”: “You Know, for

    Search” }
  4. 4 • Originally built on Lucene for text-based searching •

    Lucene and Elasticsearch work together to provide new storage formats and data types specific for numeric and keyword metrics. • Distributed More than search
  5. Fast, Efficient, and Memory Friendly 5 Respectful of your time

    Indices are sharded and distributed by default Datetime data-types use columnar-based storage for efficient querying Instant queries: requests to data that is unchanged is efficiently cached. Strong query language for searching, sorting, and bucketing by any dimension Easy index management across time and categories
  6. 6 • Beats is the platform for single-purpose data shippers

    • They install as lightweight agents and send data from hundreds or thousands of machines to Logstash or Elasticsearch. The Lightweight Data Shipper
  7. 7 • Logstash is an open source, server-side data processing

    pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.” (Ours is Elasticsearch, naturally.) • Derive structure from unstructured data with grok Collect, Parse, Transform
  8. 8 • Originally built for time-based log operations • Grew

    to be the aggregations visualizer of Elasticsearch • Ecosystem grew to include Timelion What You See Is What You Get
  9. The Elastic Stack Architecture 9

  10. 10 From Full-Text Search

  11. 11 To Log Search

  12. 12 To Analytics

  13. 13 To Time Series Composer

  14. {Coming Soon}

  15. > SELECT * FROM emp.emp; birth_date | emp_no | first_name

    | hire_date | last_name ---------------+---------------+---------------+---------------+----------- 1959-12-03 |10003 |Parto |1986-08-28 |Bamford 1953-04-20 |10006 |Anneke |1989-06-02 |Preusig # Full-text search > SELECT * FROM emp.emp WHERE QUERY('Baek fox'); birth_date | emp_no | first_name | hire_date | last_name ---------------+---------------+---------------+---------------+----------- 1957-12-03 |10080 |Premal | 1985-11-19 |Baek # Time-based filtering > SELECT last_name l, first_name f FROM emp.emp WHERE year(hire_date) < 1990 LIMIT 5; l | f ---------------+--------------- Genin |Berni Bamford |Parto SQL
  16. Extension Points

  17. Elasticsearch

  18. Elasticsearch & its Architecture 18 Node1 Node2 Node3 Index JSON

  19. Elasticsearch & its Architecture 19 Node1 Index JSON • API

    • Ingest • Analysis • Storage
  20. • Plugins are a way to enhance the core Elasticsearch

    functionality in a custom manner. They range from adding custom mapping types, custom analyzers, native scripts, custom discovery and more. • Plugins contain JAR files, but may also contain scripts and config files, and must be installed on every node in the cluster. After installation, each node must be restarted before the plugin becomes visible. • Written in Java Elasticsearch Plugins 20
  21. • Many plugins are built and maintained by Elasticsearch in

    the core distribution • Issues and bug reports can be reported on the Elasticsearch Github Repo Core Plugins 21 • This category identifies plugins that are external to the Elasticsearch project. • Issues and bug reports can usually be reported on the community plugin’s web site. Community Contributed Plugins
  22. Easy Plugin Management allows diverse customization! 22 bin/elasticsearch-plugin install [plugin_name]

  23. • Analysis Plugins • ICU Analysis, Japanese, Ukrainian, Hebrew •

    Discovery Plugins • EC2, Azure, GCE, File-Based • Ingest Plugins • Attachment, Geoip, User-Agent • Scripting Plugins • Javascript, Python, Ruby • Many More! Types of Plugins 23
  24. Logstash

  25. Logstash & its Architecture 25 Inputs Filters Outputs

  26. Logstash Plugins 26 EVERYTHING IS A PLUGIN! inputs? EVERYTHING IS

    A PLUGIN! filters? EVERYTHING IS A PLUGIN! outputs? [[ Written in JRuby ]]
  27. • Inputs • Beats, Elasticsearch, JDBC, Redis, S3, Twitter, Kafka…

    many more! • Filters • Grok, JSON, Anonymize, KV, JDBC, Elasticsearch… many more! • Outputs • Elasticsearch, Kafka, S3, Redis, Zabbix, PagerDuty… many more! Types of Plugins 27
  28. Easy Plugin Management allows diverse customization! 28 bin/logstash-plugin install [plugin_name]

  29. Beats

  30. Beats & its Architecture 30

  31. • Two Components • A component that collects the data

    • A publisher that sends the data somewhere (already implemented) • Written in Go Beats Plugins 31
  32. The Beater Interface & Run 32 // Once the beat

    is fully configured, the Run() method is invoked. The // Run()-method implements the beat its run-loop. Once the Run()-method returns, // the beat shuts down. // // The Stop() method is invoked the first time (and only the first time) a // shutdown signal is received. The Stop()-method normally will stop the Run()-loop, // such that the beat can gracefully shutdown. type Beater interface { // The main event loop. This method should block until signalled to stop by an // invocation of the Stop() method. Run(b *Beat) error // Stop is invoked to signal that the Run method should finish its execution. // It will be invoked at most once. Stop() }
  33. Types of Plugins 33

  34. Library builds executable 34 make ./my_custom_beat

  35. Kibana

  36. {Coming Soon} Machine Learning Anomaly Detection

  37. {Coming Soon} Graph Connecting Your Data

  38. Easy Plugin Management allows diverse customization! 38 bin/kibana-plugin install [plugin_name]

  39. Demo: Elasticsearch Ingest Plugin

  40. • We’ll write an Ingest Plugin that reads text from

    one field and maps the words to their spanish translations and writes the translated text into a new field • We’ll use a cookiecutter template to start our project Build a Spanish Translator Plugin 40
  41. Ingest Pipeline 41 PUT _ingest/pipeline/translate { "description": "translate", "processors": [

    { "translate": { "field": "content", "target_field": "spanish_content" } } ] }
  42. Indexing with a Pipeline 42 POST spanish_example/message/1?pipeline=translate { "content": "an

    orange is green" }
  43. • Kibana Plugin Template: • Elasticsearch Plugins: •

    Ingest Plugin Cookiecutter: ingest-processor • Logstash Plugins: logstash.html • Beats Plugins: • Elastic Discussion Forums: Resources 43
  44. Go Customize It!

  45. Thanks!