What’s the Latest in Logstash

Elastic's Jordan Sissel and Suyog Rao update the attendees about all of the new greatness in Logstash at Elastic{ON}16, February 18, 2016, in San Francisco.

1. ‹#› Suyog Rao (@suyograo) Jordan Sissel (@jordansissel) The Latest In Logstash
2. ‹#› Since last year…
3. Elastic Data Ingest Systems 3 2009 Logstash Packet Beat Beats Ingest Node 2012 2013 2015 2016 (aka: lumberjack) Stage C @ 2:15pm Yesterday @ 5:00pm logstash-forwarder
4. The long road to 1.5.0 • 1.4.2 released June 2014 • 1.5.0 released May 2015 • 1.5.0 took a long time. Let’s talk about that. 4
5. 1.5: the plugin manager 5
6. 1.5: better release tooling 6
7. 1.5: automated testing of code changes 7
9. 1.5: way more tests! 9
10. Faster release pace • Since 1.5.0, we have done 15 releases (1.4.x, 1.5.x, 2.0.x, 2.1.x, 2.2.x) • 1.4.2 to 1.5.0: 1 release in 10 months • 1.5.0 to 2.2.1: 15 releases in 9 months. • 16x more main releases 10
11. 11 Events 0 1000000 2000000 3000000 4000000 5000000 6000000 7000000 8000000 Logstash Releases 1.4.4 1.5.0 2.0.0 2.1.0 2.2.0 2.3.0 Apache Parsing Complex Syslog
12. ‹#› Delivery Guarantees Take it to the next level.
13. Delivery - Today 13 Transit Problems Data Loss? CPU contention No :) Network fault No :) Output slow No :) Machine Problems Data Loss? kill -9 Yes :( Power loss Yes :( Logstash crash Yes :(
14. The Logstash Pipeline (v1.0 to v2.1) 14 Input I F O Filter Output Fixed-size Blocking Queue Fixed-size Blocking Queue
15. I F O Acknowledging Events (available already) 15 1) Read 2) Enqueue 3) In the queue :) 4) Got it!
16. I F O Too many queues! (Logstash v2.2.0) 16
17. Batching (Logstash v2.2.0) 17 Publish 1 :: Consume 1 Publish N :: Consume N Publish 1 :: Consume 1 I F O
18. Java for Serialization (soon) 18 I F O
19. I F O Output Acknowledgements (soon) 19 1) Read 2) Send 3) Success! 4) Got it!
20. Persisted Queue (soon) 20 I F O … … … … …
21. Dead Letter Queue (soon) 21 I F O Success … … … Failure … …
22. Delivery - Future (Logstash 5.x) 22 Transit Problems Data Loss? CPU contention No :) Network fault No :) Output slow No :) Machine Problems Data Loss? kill -9 No :) Power loss No :) Logstash crash No :)
23. ‹#› Logstash Deep Dive — Stage C @ 3:15pm with Colin and Andrew
24. ‹#› Logstash Management Let’s make it easier.
25. ‹#› Monitoring
26. 26 What’s the throughput of Logstash? Which filter is adding to latency? CPU is high, where is the hotspot? Is there a processing backpressure?
28. 28 Next: Monitoring UI
29. 29 Road To Runtime Configuration
30. Step 1: Config Reloading Previously: Any config change made to file required a process restart Feedback loop for development/ testing slow Processing pipeline must be long living 30 Why? File watched for changes Current Pipeline stopped Config Validated New Pipeline started - no process restart How?
32. 32 role: frontend-logs 1 Create a role, upload config Step 2: Centralised Config
33. 33 Node Management hello can you hear me ? Hello from the other side
34. Elasticsearch Management Plugin 34 GET /_logstash/${role}/config/current POST /_logstash/${role}/config/_rollback GET /_logstash/${role}/config/${version_number} GET /_logstash/${role}/nodes/${node_name} GET /_logstash
35. Whats next? Versioning and Rollback Config Validation Track nodes in Elasticsearch 35 Management UI Pipeline Config Viewer Clustered Metrics Load Distribution High Availability Centralized Config Clustering 5.x timeframe
36. ‹#› Plugins Ecosystem
37. Plugins 40+ inputs, 80+ filters 203 Plugins: State of the Union 37 8000 Commits Maintainers and Core Team 1900 Releases RubyGems.org
38. More ways to get data into Elastic Stack 38 Salesforce • Pull in Salesforce Objects using SOQL HTTP • Poll any HTTP endpoint repeatedly • Monitor your website • HTTP input to get data directly JDBC • Sync DB with ES • Schedule queries • Save state of last run
39. ‹#› Community Maintainers Program
40. Plugins: Pulse, Open Issues 40
41. ‹#› 41 Avishai Ish-Shalom Björn Puttman Fabien Baligand Joe Lawson Magnus Bäck Philippe Weber Jurgens du Toit Russell Savage