Writing Elastic Stack Plugins

These slides were presented at OpenWest 2017: https://www.openwest.org/custom/description.php?id=107

The Elastic Stack consists of a few systems: Elasticsearch, Beats, Kibana, Logstash. In this talk, you will learn how to build plugins that span these products to form a cohesive user experience for custom analytic applications built on top of the stack. We will walk through what it looks like to
write Logstash plugin, Elasticsearch plugins, beats, and Kibana plugins.

Transcript

1. @talevy Extending The Elastic Stack Tal Levy, Software Engineer
2. The Elastic Stack 2 Kibana Elasticsearch Beats Logstash Security Alerting Monitoring Reporting X-Pack Graph Elastic Cloud
3. - Shay Banon { } { “tagline”: “You Know, for Search” }
4. 4 • Originally built on Lucene for text-based searching • Lucene and Elasticsearch work together to provide new storage formats and data types specific for numeric and keyword metrics. • Distributed More than search
5. Fast, Efficient, and Memory Friendly 5 Respectful of your time Indices are sharded and distributed by default Datetime data-types use columnar-based storage for efficient querying Instant queries: requests to data that is unchanged is efficiently cached. Strong query language for searching, sorting, and bucketing by any dimension Easy index management across time and categories
6. 6 • Beats is the platform for single-purpose data shippers • They install as lightweight agents and send data from hundreds or thousands of machines to Logstash or Elasticsearch. The Lightweight Data Shipper
7. 7 • Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.” (Ours is Elasticsearch, naturally.) • Derive structure from unstructured data with grok Collect, Parse, Transform
8. 8 • Originally built for time-based log operations • Grew to be the aggregations visualizer of Elasticsearch • Ecosystem grew to include Timelion What You See Is What You Get
9. The Elastic Stack Architecture 9
10. 10 From Full-Text Search
11. 11 To Log Search
12. 12 To Analytics
13. 13 To Time Series Composer
14. {Coming Soon}
15. > SELECT * FROM emp.emp; birth_date | emp_no | first_name | hire_date | last_name ---------------+---------------+---------------+---------------+----------- 1959-12-03 |10003 |Parto |1986-08-28 |Bamford 1953-04-20 |10006 |Anneke |1989-06-02 |Preusig # Full-text search > SELECT * FROM emp.emp WHERE QUERY('Baek fox'); birth_date | emp_no | first_name | hire_date | last_name ---------------+---------------+---------------+---------------+----------- 1957-12-03 |10080 |Premal | 1985-11-19 |Baek # Time-based filtering > SELECT last_name l, first_name f FROM emp.emp WHERE year(hire_date) < 1990 LIMIT 5; l | f ---------------+--------------- Genin |Berni Bamford |Parto SQL
16. Extension Points
17. Elasticsearch
18. Elasticsearch & its Architecture 18 Node1 Node2 Node3 Index JSON
19. Elasticsearch & its Architecture 19 Node1 Index JSON • API • Ingest • Analysis • Storage
20. • Plugins are a way to enhance the core Elasticsearch functionality in a custom manner. They range from adding custom mapping types, custom analyzers, native scripts, custom discovery and more. • Plugins contain JAR files, but may also contain scripts and config files, and must be installed on every node in the cluster. After installation, each node must be restarted before the plugin becomes visible. • Written in Java Elasticsearch Plugins 20
21. • Many plugins are built and maintained by Elasticsearch in the core distribution • Issues and bug reports can be reported on the Elasticsearch Github Repo Core Plugins 21 • This category identifies plugins that are external to the Elasticsearch project. • Issues and bug reports can usually be reported on the community plugin’s web site. Community Contributed Plugins
22. Easy Plugin Management allows diverse customization! 22 bin/elasticsearch-plugin install [plugin_name]
23. • Analysis Plugins • ICU Analysis, Japanese, Ukrainian, Hebrew • Discovery Plugins • EC2, Azure, GCE, File-Based • Ingest Plugins • Attachment, Geoip, User-Agent • Scripting Plugins • Javascript, Python, Ruby • Many More! Types of Plugins 23
24. Logstash
25. Logstash & its Architecture 25 Inputs Filters Outputs
26. Logstash Plugins 26 EVERYTHING IS A PLUGIN! inputs? EVERYTHING IS A PLUGIN! filters? EVERYTHING IS A PLUGIN! outputs? [[ Written in JRuby ]]
27. • Inputs • Beats, Elasticsearch, JDBC, Redis, S3, Twitter, Kafka… many more! • Filters • Grok, JSON, Anonymize, KV, JDBC, Elasticsearch… many more! • Outputs • Elasticsearch, Kafka, S3, Redis, Zabbix, PagerDuty… many more! Types of Plugins 27
28. Easy Plugin Management allows diverse customization! 28 bin/logstash-plugin install [plugin_name]
29. Beats
30. Beats & its Architecture 30
31. • Two Components • A component that collects the data • A publisher that sends the data somewhere (already implemented) • Written in Go Beats Plugins 31
32. The Beater Interface & Run 32 // Once the beat is fully configured, the Run() method is invoked. The // Run()-method implements the beat its run-loop. Once the Run()-method returns, // the beat shuts down. // // The Stop() method is invoked the first time (and only the first time) a // shutdown signal is received. The Stop()-method normally will stop the Run()-loop, // such that the beat can gracefully shutdown. type Beater interface { // The main event loop. This method should block until signalled to stop by an // invocation of the Stop() method. Run(b *Beat) error // Stop is invoked to signal that the Run method should finish its execution. // It will be invoked at most once. Stop() }
33. Types of Plugins 33
34. Library builds executable 34 make ./my_custom_beat
35. Kibana
36. {Coming Soon} Machine Learning Anomaly Detection
37. {Coming Soon} Graph Connecting Your Data
38. Easy Plugin Management allows diverse customization! 38 bin/kibana-plugin install [plugin_name]
39. Demo: Elasticsearch Ingest Plugin
40. • We’ll write an Ingest Plugin that reads text from one field and maps the words to their spanish translations and writes the translated text into a new field • We’ll use a cookiecutter template to start our project Build a Spanish Translator Plugin 40
41. Ingest Pipeline 41 PUT _ingest/pipeline/translate { "description": "translate", "processors": [ { "translate": { "field": "content", "target_field": "spanish_content" } } ] }
42. Indexing with a Pipeline 42 POST spanish_example/message/1?pipeline=translate { "content": "an orange is green" }
43. • Kibana Plugin Template: https://github.com/elastic/template-kibana-plugin/ • Elasticsearch Plugins: https://www.elastic.co/guide/en/elasticsearch/plugins/5.5/intro.html • Ingest Plugin Cookiecutter: https://github.com/spinscale/cookiecutter-elasticsearch- ingest-processor • Logstash Plugins: https://www.elastic.co/guide/en/logstash/current/contributing-to- logstash.html • Beats Plugins: https://www.elastic.co/guide/en/beats/libbeat/current/new-beat.html • Elastic Discussion Forums: https://discuss.elastic.co/ Resources 43
44. Go Customize It!
45. Thanks!