Elastic{ON} 2018: Latest in Logstash

Much has happened since 5.0. Persistent queues, pipeline viewer (x-ray vision, basically) and the ability to run multiple pipelines at the same time for different use cases, and a move to the latest version of JRuby — it's all laying the foundation for even more goodness to come. See where the Logstash roadmap is headed and what to expect next.

Transcript

1. Elastic February 28, 2018 @jordansissel / @andrewvc What’s the Latest in Logstash Jordan Sissel and Andrew Cholakian
2. Thank you to our community You complete the picture
3. In the last year, 1,413 Logstashers have helped us with 9,237 issues, comments, and pull requests to our logstash-plugins repository. 864 pull requests were opened. The Logstash Plugins Community 3
4. In the last year, 76 Logstashers have helped us with 1,131 issues, comments, and pull requests to our logstash-plugins repository. 179 pull requests were opened. The Logstash Core Community 4
5. Triaging in Two Parts
6. We couldn’t do it without you!
7. The Latest
8. queue.type: persisted # (v5.4) input input input filter filter filter output output Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY queue Theme: Don’t lose data.
9. WHAT IF WE CAN’T DELIVER SOMETHING? Theme: Don’t lose data.
11. Simple things should be simple
12. bin/logstash --modules netflow
14. bin/logstash --modules arcsight
16. bin/logstash --modules <MORE?!>
17. 17 A Tale of Two* Pipelines *(or more)
18. 1 8 input { beats { port => 3444 tag => apache } tcp { port => 4222 tag => firewall } } filter { if "apache" in [tags] { dissect { ... } } else if "firewall" in [tags] { grok { ... } } } output { if "apache" in [tags] { elasticsearch { ... } } else if "firewall" in [tags] { tcp { ... } } } TCP BEATS DISSECT GROK TCP ES
19. 1 9 input { beats { port => 3444 tag => apache } tcp { port => 4222 tag => firewall } } filter { if "apache" in [tags] { dissect { ... } } else if "firewall" in [tags] { grok { ... } } } output { if "apache" in [tags] { elasticsearch { ... } } else if "firewall" in [tags] { tcp { ... } } } input { beats { port => 3444 tag => apache } tcp { port => 4222 tag => firewall } } filter { if "apache" in [tags] { dissect { ... } } else if "firewall" in [tags] { grok { ... } } } output { if "apache" in [tags] { elasticsearch { ... } } else if "firewall" in [tags] { tcp { ... } } }
20. 2 0 input { beats { port => 3444 tag => apache } } filter { dissect { ... } } output { elasticsearch { ... } } input { tcp { port => 4222 tag => firewall } } filter { grok { ... } } output { tcp { ... } } Multiple Pipelines (v6.0) BEATS DISSECT ES TCP GROK TCP
21. 2 1
22. X-Pack Central Management (v6.0) ------- ----------- -------- - ------ --- ----- ---- ---- --- --- --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- --- --- --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- --- --- --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- Three Logstash instances. Doing the same thing. Let’s simplify this.
23. X-Pack Central Management (v6.0) ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- One configuration source.
25. 2 5
26. WITH GREAT POWER COMES GREAT
27. --experimental-java-execution (v6.1) 27 0: new #121 3: dup 4: iconst_2 5: fconst_1 6: invokespecial #122 9: astore_1 10: aload_1 11: ldc #47 13: aload_0 14: getfield #6 17: invokeinterface #52, 3 filter { if “debug” in [tags] { drop { } } grok { match => { … } } } if event.getField(“tags”)... { drop.execute(event); if event.isCancelled() { return; } } grok.execute(event); Compile bytecode Load Configuration Transform to Java
28. --experimental-java-execution (v6.1) 28 “Compile” code = config.compile() pipeline = eval(code) loop do batch = queue.pop() pipeline.execute(batch) end Load Configuration filter { if “debug” in [tags] { drop { } } grok { match => { … } } } Transform to Ruby if event.get(“tags”).include?( drop.execute(event) return if event.cancelled? end grok.execute(event) Ruby
29. Logstash 6.2: Protect credentials with the keystore logstash-keystore (Logstash 6.2) % logstash-keystore create % logstash-keystore add es_password # use es_password in the pipeline: output { elasticsearch { hosts => … user => “elastic” password => “${es_password}” } }
30. Upcoming Features
31. Quick Hits
32. Visual Pipeline Builder
33. More Security Modules (SIEM, IDS, Firewalls, etc.)
34. SNMP Poller (It’s not a trap!)
35. Centralized Management Improvements
36. Node Groups
37. How we do it today 3 7 xpack.management.enabled: true xpack.management.pipeline.id: ["apache", "cloudwatch_logs"] LS1 LS2 LS3 apache cloudwatch logs
38. Where we Want to Go 3 8 xpack.management.enabled: true xpack.management.pipeline.node_groups: ["webserver_logs”, “security_logs”] LS1 LS2 LS3 apache logs cloudwatch logs webserver_logs security_logs nginx logs azure activity logs
39. Pipeline Settings Will Also Be Supported!
40. Ancillary Files in Config Management
41. Config File Grok Patterns GeoIP Database Without Config Management
42. Config File Grok Patterns GeoIP Database Config Management Today
43. Config File Grok Patterns GeoIP Database Config Management in the Future
44. Java Plugin API
45. • Allow people to use additional JVM languages to develop plugins • Enable performance optimization where required Java Plugin API Goals 4 5
46. • We’re starting with a low level API, optimized for performance • Will add sugar on top • We plan to support the current plugin API indefinitely • Some esoteric APIs, like flush, may go away Java Plugin API Plan 4 6
47. Logstash -> Logstash: Making it a better story
48. Currently, things are a little ugly
49. How we do it today 4 9 output { lumbjerjack { … } } input { beats { … } } ? ? ?
50. Interpipeline Communications
51. - pipeline.id: senderone config.string: "input { generator { message => huhx } } output { internal { send_to => [foo] } }" - pipeline.id: sendertwo config.string: "input { generator { message => whutx } } output { internal { send_to => [foo] } }" - pipeline.id: out config.string: " input { internal { address => foo } } output { stdout { codec => json_lines } }" What it looks like 5 1 senderone sendertwo out
52. Share a Port 5 2 Beats Ingest + Routing Enrich Weblogs + Output Enrich ETL Logs + Output Enrich Metrics + Output
53. Buffer as Needed 5 3 PQ PQ Logs Ingest Output to Elasticsearch Output to S3
54. Put it Together 5 4 PQ PQ PQ Beats Ingest and Route Enrich Weblogs Enrich ETL Logs Enrich Metrics + Output to Metrics ES Cluster Output to Logging ES Cluster
55. Language Improvements on the Horizons
56. • Is not null checks • Array/field reference conflation (is [foo] an array, or a field reference?) • What else does the lang need? • Can we experiment with new languages? The Language Can Move Forward 5 6
57. An Ephemeral World Or, how I learned to stop worrying about my disk and embrace ephemeral storage
58. An Ephemeral World Part 1: End to End ACKs
59. Resiliency today isn’t as easy as it could be
60. 60 You need to think about disks if you use the PQ
61. 61 You need to be able to tolerate data loss if you use the in- memory queue
62. What if we could give you the best of both worlds?
63. End-to-End (E2E) ACKs solve this problem
64. We currently buffer at each stage
65. Each event must be persisted, then acknowledged
66. With E2E ACKs, Nodes Can Fully Fail
67. Where we can replay or delay asynchronously acknowledge, we can skip persistence 6 7
68. • TCP, UDP, and Syslog protocol offer no facility to replay • The best strategy here is to store ASAP on Logstash, and then try to get it elsewhere ASAP Where we can’t replay, the PQ is still best 6 8
69. • Currently in progress • Will likely require rewrites to input plugins to be efficient • Only works for things that are replayable • Luckily, a lot of things are replayable E2E ACK Summary 6 9
70. An Ephemeral World Part 2: Distributed Plugin State/Execution
71. Problem 1: Plugins store metadata on disk
72. This all must be backed up
73. Problem 2: Inputs cannot share work
74. Today, Logstashes cannot coordinate their access to a service like S3 Active Failover
75. • Users must implement backups of metadata • Scaling past one box requires manual partitioning, or is not possible • Failover is tricky, and involves restoring backed up metadata Why Local State is Irritating 7 5
76. Solution: Centralized State/Distributed Exec
77. Local File Metadata Will Still Work! Active Failover
78. But This is Way Cooler Indexing cluster Management Cluster Management Data
79. • Plugin state kept in ES • Leader election through ES • Task assignment through ES • Still in design phase, we have a PoC in progress How it Works 7 9
80. Problem 3: The DLQ is Local
81. Solution: Create an ES DLQ
82. • Pitch in on PRs • Report bugs • Review PRs We Need Your Help! 8 2
83. Thank You!