Elastic{ON} 2018: Latest in Logstash

Dd9d954997353b37b4c2684f478192d3?s=47 Elastic Co
March 01, 2018

Elastic{ON} 2018: Latest in Logstash

Much has happened since 5.0. Persistent queues, pipeline viewer (x-ray vision, basically) and the ability to run multiple pipelines at the same time for different use cases, and a move to the latest version of JRuby — it's all laying the foundation for even more goodness to come. See where the Logstash roadmap is headed and what to expect next.

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

March 01, 2018
Tweet

Transcript

  1. Elastic February 28, 2018 @jordansissel / @andrewvc What’s the Latest

    in Logstash Jordan Sissel and Andrew Cholakian
  2. Thank you to our community You complete the picture

  3. In the last year, 1,413 Logstashers have helped us with

    9,237 issues, comments, and pull requests to our logstash-plugins repository. 864 pull requests were opened. The Logstash Plugins Community 3
  4. In the last year, 76 Logstashers have helped us with

    1,131 issues, comments, and pull requests to our logstash-plugins repository. 179 pull requests were opened. The Logstash Core Community 4
  5. Triaging in Two Parts

  6. We couldn’t do it without you!

  7. The Latest

  8. queue.type: persisted # (v5.4) input input input filter filter filter

    output output Icons made by Freepik from www.flaticon.com is licensed by CC 3.0 BY queue Theme: Don’t lose data.
  9. WHAT IF WE CAN’T DELIVER SOMETHING? Theme: Don’t lose data.

  10. None
  11. Simple things should be simple

  12. bin/logstash --modules netflow

  13. None
  14. bin/logstash --modules arcsight

  15. None
  16. bin/logstash --modules <MORE?!>

  17. 17 A Tale of Two* Pipelines *(or more)

  18. 1 8 input { beats { port => 3444 tag

    => apache } tcp { port => 4222 tag => firewall } } filter { if "apache" in [tags] { dissect { ... } } else if "firewall" in [tags] { grok { ... } } } output { if "apache" in [tags] { elasticsearch { ... } } else if "firewall" in [tags] { tcp { ... } } } TCP BEATS DISSECT GROK TCP ES
  19. 1 9 input { beats { port => 3444 tag

    => apache } tcp { port => 4222 tag => firewall } } filter { if "apache" in [tags] { dissect { ... } } else if "firewall" in [tags] { grok { ... } } } output { if "apache" in [tags] { elasticsearch { ... } } else if "firewall" in [tags] { tcp { ... } } } input { beats { port => 3444 tag => apache } tcp { port => 4222 tag => firewall } } filter { if "apache" in [tags] { dissect { ... } } else if "firewall" in [tags] { grok { ... } } } output { if "apache" in [tags] { elasticsearch { ... } } else if "firewall" in [tags] { tcp { ... } } }
  20. 2 0 input { beats { port => 3444 tag

    => apache } } filter { dissect { ... } } output { elasticsearch { ... } } input { tcp { port => 4222 tag => firewall } } filter { grok { ... } } output { tcp { ... } } Multiple Pipelines (v6.0) BEATS DISSECT ES TCP GROK TCP
  21. 2 1

  22. X-Pack Central Management (v6.0) ------- ----------- -------- - ------ ---

    ----- ---- ---- --- --- --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- --- --- --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- --- --- --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- Three Logstash instances. Doing the same thing. Let’s simplify this.
  23. X-Pack Central Management (v6.0) ------- ----------- -------- - ------ ---

    ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- ------- ----------- -------- - ------ --- ----- ---- ---- ------ --- ------------ -- -- - -- ---- - -- --- -- - ---- - --- --- One configuration source.
  24. None
  25. 2 5

  26. WITH GREAT POWER COMES GREAT

  27. --experimental-java-execution (v6.1) 27 0: new #121 3: dup 4: iconst_2

    5: fconst_1 6: invokespecial #122 9: astore_1 10: aload_1 11: ldc #47 13: aload_0 14: getfield #6 17: invokeinterface #52, 3 filter { if “debug” in [tags] { drop { } } grok { match => { … } } } if event.getField(“tags”)... { drop.execute(event); if event.isCancelled() { return; } } grok.execute(event); Compile bytecode Load Configuration Transform to Java
  28. --experimental-java-execution (v6.1) 28 “Compile” code = config.compile() pipeline = eval(code)

    loop do batch = queue.pop() pipeline.execute(batch) end Load Configuration filter { if “debug” in [tags] { drop { } } grok { match => { … } } } Transform to Ruby if event.get(“tags”).include?( drop.execute(event) return if event.cancelled? end grok.execute(event) Ruby
  29. Logstash 6.2: Protect credentials with the keystore logstash-keystore (Logstash 6.2)

    % logstash-keystore create % logstash-keystore add es_password # use es_password in the pipeline: output { elasticsearch { hosts => … user => “elastic” password => “${es_password}” } }
  30. Upcoming Features

  31. Quick Hits

  32. Visual Pipeline Builder

  33. More Security Modules (SIEM, IDS, Firewalls, etc.)

  34. SNMP Poller (It’s not a trap!)

  35. Centralized Management Improvements

  36. Node Groups

  37. How we do it today 3 7 xpack.management.enabled: true xpack.management.pipeline.id:

    ["apache", "cloudwatch_logs"] LS1 LS2 LS3 apache cloudwatch logs
  38. Where we Want to Go 3 8 xpack.management.enabled: true xpack.management.pipeline.node_groups:

    ["webserver_logs”, “security_logs”] LS1 LS2 LS3 apache logs cloudwatch logs webserver_logs security_logs nginx logs azure activity logs
  39. Pipeline Settings Will Also Be Supported!

  40. Ancillary Files in Config Management

  41. Config File Grok Patterns GeoIP Database Without Config Management

  42. Config File Grok Patterns GeoIP Database Config Management Today

  43. Config File Grok Patterns GeoIP Database Config Management in the

    Future
  44. Java Plugin API

  45. • Allow people to use additional JVM languages to develop

    plugins • Enable performance optimization where required Java Plugin API Goals 4 5
  46. • We’re starting with a low level API, optimized for

    performance • Will add sugar on top • We plan to support the current plugin API indefinitely • Some esoteric APIs, like flush, may go away Java Plugin API Plan 4 6
  47. Logstash -> Logstash: Making it a better story

  48. Currently, things are a little ugly

  49. How we do it today 4 9 output { lumbjerjack

    { … } } input { beats { … } } ? ? ?
  50. Interpipeline Communications

  51. - pipeline.id: senderone config.string: "input { generator { message =>

    huhx } } output { internal { send_to => [foo] } }" - pipeline.id: sendertwo config.string: "input { generator { message => whutx } } output { internal { send_to => [foo] } }" - pipeline.id: out config.string: " input { internal { address => foo } } output { stdout { codec => json_lines } }" What it looks like 5 1 senderone sendertwo out
  52. Share a Port 5 2 Beats Ingest + Routing Enrich

    Weblogs + Output Enrich ETL Logs + Output Enrich Metrics + Output
  53. Buffer as Needed 5 3 PQ PQ Logs Ingest Output

    to Elasticsearch Output to S3
  54. Put it Together 5 4 PQ PQ PQ Beats Ingest

    and Route Enrich Weblogs Enrich ETL Logs Enrich Metrics + Output to Metrics ES Cluster Output to Logging ES Cluster
  55. Language Improvements on the Horizons

  56. • Is not null checks • Array/field reference conflation (is

    [foo] an array, or a field reference?) • What else does the lang need? • Can we experiment with new languages? The Language Can Move Forward 5 6
  57. An Ephemeral World Or, how I learned to stop worrying

    about my disk and embrace ephemeral storage
  58. An Ephemeral World Part 1: End to End ACKs

  59. Resiliency today isn’t as easy as it could be

  60. 60 You need to think about disks if you use

    the PQ
  61. 61 You need to be able to tolerate data loss

    if you use the in- memory queue
  62. What if we could give you the best of both

    worlds?
  63. End-to-End (E2E) ACKs solve this problem

  64. We currently buffer at each stage

  65. Each event must be persisted, then acknowledged

  66. With E2E ACKs, Nodes Can Fully Fail

  67. Where we can replay or delay asynchronously acknowledge, we can

    skip persistence 6 7
  68. • TCP, UDP, and Syslog protocol offer no facility to

    replay • The best strategy here is to store ASAP on Logstash, and then try to get it elsewhere ASAP Where we can’t replay, the PQ is still best 6 8
  69. • Currently in progress • Will likely require rewrites to

    input plugins to be efficient • Only works for things that are replayable • Luckily, a lot of things are replayable E2E ACK Summary 6 9
  70. An Ephemeral World Part 2: Distributed Plugin State/Execution

  71. Problem 1: Plugins store metadata on disk

  72. This all must be backed up

  73. Problem 2: Inputs cannot share work

  74. Today, Logstashes cannot coordinate their access to a service like

    S3 Active Failover
  75. • Users must implement backups of metadata • Scaling past

    one box requires manual partitioning, or is not possible • Failover is tricky, and involves restoring backed up metadata Why Local State is Irritating 7 5
  76. Solution: Centralized State/Distributed Exec

  77. Local File Metadata Will Still Work! Active Failover

  78. But This is Way Cooler Indexing cluster Management Cluster Management

    Data
  79. • Plugin state kept in ES • Leader election through

    ES • Task assignment through ES • Still in design phase, we have a PoC in progress How it Works 7 9
  80. Problem 3: The DLQ is Local

  81. Solution: Create an ES DLQ

  82. • Pitch in on PRs • Report bugs • Review

    PRs We Need Your Help! 8 2
  83. Thank You!