Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What's the Latest in Logstash?

Dd9d954997353b37b4c2684f478192d3?s=47 Elastic Co
March 08, 2017

What's the Latest in Logstash?

Logstash team and tech leads address questions like: Can we go faster? What is the persistent queue? How do I monitor Logstash? What is the future of Logstash configuration? How does the team keep systadmins and DevOps in mind when working on the product? And what's in store for the Logstash UI?

Andrew Cholakian l Software Engineer l Elastic
Suyog Rao l Logstash Team Lead l Elastic
Jordan Sissel l Logstash Creator & Tech Lead l Elastic

Dd9d954997353b37b4c2684f478192d3?s=128

Elastic Co

March 08, 2017
Tweet

Transcript

  1. Andrew Cholakian Suyog Rao Jordan Sissel cool stuff: 2017

  2. It’s time for numbers!

  3. across 228 plugins creating 1080 releases 3131 changes by 210

    humans PLUGINS
  4. creating 24 releases 2021 changes by 72 humans LOGSTASH

  5. theme: resisting data loss

  6. input filter output Q

  7. input filter output queue memory

  8. None
  9. kernel: Out of memory: Kill process 15334 (java)

  10. surviving (temporary) machine failures Persisted Queue

  11. input filter output disk 1 2 3 4 1 2

    3
  12. "queue": { "events": 340190, "type": "persisted", "capacity": { "page_capacity_in_bytes": 262144000,

    "max_queue_size_in_bytes": 1073741824, "max_unread_events": 0 }, "data": { "path": "/home/jls/build/logstash-5.2.2/data/queue", "free_space_in_bytes": 4822740992, "storage_type": "ext4" } queue stats
  13. 13 Impact on Performance

  14. Basic Configuration 14 # Enable the persisted queue queue.type: persisted

    # Maximum size on disk of the queue queue.max_bytes: 4gb
  15. Durability Configuration 15 # Checkpoint every 123 writes queue.checkpoint.writes: 123

    # Checkpoint every 456 events processed and ack’d queue.checkpoint.acks: 456
  16. Management and Monitoring

  17. GET http://localhost:9600/ { "host" : "coffee", "version" : "5.2.2", "http_address"

    : "127.0.0.1:9600", "id" : "a882e2c6-3cf7-410d-93ea-f2d2a9bc2474", "name" : "coffee", "build_date" : "2017-02-24T17:46:55Z", "build_sha" : "57984d20eb28b0df40a59077c600ec1a399d46f5", "build_snapshot" : false }
  18. GET http://localhost:9600/_node/stats/jvm … "jvm" : { "threads" : { "count"

    : 6, "peak_count" : 6 }, "mem" : { "heap_used_in_bytes" : 301445688, "heap_used_percent" : 14, "heap_committed_in_bytes" : 519045120, …
  19. GET http://localhost:9600/_node/stats/process … "process" : { "open_file_descriptors" : 56, "peak_open_file_descriptors"

    : 57, "max_file_descriptors" : 10240, "mem" : { "total_virtual_in_bytes" : 5224431616 }, "cpu" : { "total_in_millis" : 47890212000, …
  20. GET http://localhost:9600/_node/stats/pipeline … "plugins" : { "outputs" : [ {

    "id" : "92ddb0615336293c1757cac81c3bebfa19985e68-2", "events" : { "duration_in_millis" : 104, "in" : 29, "out" : 29 }, "name" : "stdout" …
  21. PUT http://localhost:9600/_node/logging { "logger.logstash.outputs.elasticsearch" : "DEBUG" }

  22. 22

  23. 23 bin/logstash-plugin install x-pack

  24. perhaps, nicer than grok The Dissect Filter

  25. 144.23.4.1 - - [13/Mar/2016:02:38:26 -0400] "GET /fancy.html HTTP/1.1" 200 6146

    "-" "Mozilla/5.0 (X11; Linux x86_64; rv:51.0) Gecko/ 20100101 Firefox/51.0" logs
  26. (?<a0>(?<a1>(?<a2>\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?| Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b) +(?<a3>(?:(?:0[1-9])|(?: [12][0-9])|(?:3[01])|[1-9])) (?<a4>(?!<[0-9])(?<a5>(?:2[0123]|[01][0-9])):(?<a6>(?:[0-5][0-9]))(?::(? <a7>(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9]))) (?<a8>(?:(?<a9>\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?: \.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b))|(?<a10>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.] (?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]| [0-1]?[0-9]{1,2}))(?![0-9]))))

    (?<a11>(?<a12>(?:[\w._/%-]+))(?:\[(?<a13>\b(?:[1-9][0-9]*)\b)\])?): (? <a14>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?: 25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])):(?<a15>(?:[+-]?(?: [0-9]+))) \[(?<a16>(?<a17>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))/(?<a18>\b(?:Jan(?:uary)?| Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b)/(?<a19>[0-9]+):(?<a20>(?!<[0-9])(?<a21>(?:2[0123]| [01][0-9])):(?<a22>(?:[0-5][0-9]))(?::(?<a23>(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9])).(?<a24>(?:[+-]? (?:[0-9]+))))\] (?<a25>\S+) (?<a26>\S+)/(?<a27>\S+) (?<a28>(?:[+-]?(?:[0-9]+)))/(?<a29>(?:[+-]?(?: [0-9]+)))/(?<a30>(?:[+-]?(?:[0-9]+)))/(?<a31>(?:[+-]?(?:[0-9]+)))/(?<a32>\S+) (?<a33>(?:[+-]?(?:[0-9]+))) (?<a34>\S+) (?<a35>.*?) (?<a36>.*?) (?<a37>\S+) (?<a38>(?:[+-]?(?:[0-9]+)))/(?<a39>(?:[+-]?(?: [0-9]+)))/(?<a40>(?:[+-]?(?:[0-9]+)))/(?<a41>(?:[+-]?(?:[0-9]+)))/(?<a42>\S+) (?<a43>(?:[+-]?(?:[0-9]+)))/ (?<a44>(?:[+-]?(?:[0-9]+))) \{(?<a45>(?<a46>.*?))\} \{(?<a47>(?<a48>.*?))\} "(?<a49>\b\w+\b) (? <a50>(?<a51>(?:/[A-Za-z0-9$.+!*'(){},~:;=#%_-]*)+)(?:(?<a52>\?[A-Za-z0-9$.+!*'(){},~#%&/=:;_-]*))?) HTTP/(?<a53>(?:(?<a54>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))") regex
  27. %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[% {HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int})

    %{QS:referrer} %{QS:agent} grok
  28. %{clientip} %{ident} %{auth} [%{timestamp}] "%{request}" % {response} %{bytes} "%{referrer}" "%{agent}"

    dissect
  29. %{clientip} %{ident} %{auth} [%{timestamp}] "%{request}" %{response} %{bytes} "%{referrer}" "%{agent}" %{IPORHOST:clientip}

    %{USER:ident} %{USER:auth} \[% {HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent} GROK DISSECT
  30. 30 Pipeline Throughput

  31. { "name": “ grok" "events": { "duration_in_millis": 36640, "in": 401003,

    "out": 401002 } } { "name": " dissect" "events": { "duration_in_millis": 18206, "in": 1111072, "out": 1111072 } } 91 µs per event 16 µs per event (5x faster)
  32. support all the versions! Kafka plugins

  33. 59 kafka plugin releases since Elastic{ON} 2016

  34. Kafka has lots of releases… 0.8, 0.9, 0.10.0.x, 0.10.1.x

  35. Pipeline Visualization and Beyond Andrew Cholakian (@andrewvc)

  36. Logstash performance problems? 36

  37. 37 What are my plugins even doing?

  38. 38 Are they running as fast as they could be?

  39. 39 What does this config even do?

  40. 40 Finally, there is a solution!

  41. The Logstash Pipeline Visualizer

  42. The Logstash Threading Model 42 Input Codec Queue Codec Input

    Filters Outputs Batcher Filters Outputs Batcher Filters Outputs Batcher Input Thread Input Thread Pipeline Worker Thread Pipeline Worker Thread Pipeline Worker Thread
  43. Demo 1

  44. Logstash Intermediate Representation (LIR)

  45. Current (pre-LIR) pipeline 45 Config File 1 2 3 TreeTop

    IR Ruby Code
  46. How it Works in LIR 46 Config File 1 2

    3 4 5 TreeTop IR Imperative IR (LIR) DAG IR (LIR) Graph JSON Format Ruby Code Byte Code Etc.
  47. 47 LIR Config File Formats Execution Metrics Correlation A Central

    Hub for Logstash Execution
  48. 48 LIR Logstash Config Language Execution Metrics Correlation New Config

    File Formats! Ingest Pipeline JSON?
  49. 49 LIR Config File Formats Ruby Exec Metrics Correlation Multiple

    Execution Engines Java Exec
  50. 50 LIR Config File Formats Execution Pipeline Visualizer Better Diagnostic

    Tools Step Debugger?
  51. 51 Roadmap 1. Enable LIR in core: In progress! 2.

    Pipeline Visualizer: In progress! 3. Java Pipeline Execution: Next 4. Visual Pipeline Editor: Next 5. New config languages: Planning Source: Gray Arial10pt
  52. Suyog Rao (@suyograo) Centralised Management

  53. 53 • Elasticsearch as a remote config store • Manage

    configurations via UI • Group multiple Logstash under roles • Simple alternative to puppet, chef Centralised Management
  54. Demo 2

  55. Pipelines: Text Based 55

  56. 56 Pipelines: Visual Builder beats (filebeat) grok (parseApache) stdin (testStdin)

    date (apacheDate) false true condition (tag == _grokparsefailure)
  57. Sub-Title Plugins

  58. 58

  59. 59

  60. 60

  61. JDBC Lookup 61 SKU PRODUCT STATE 99888555 plum california 78782234

    apple michigan 34559099 blueberry new mexico 77837890 avocado california 34559099 pear california 14239596 strawberry new mexico
  62. 62

  63. Creating Custom Plugins 63

  64. 64

  65. 65 • Ease of management • More integrations • Resiliency

    and performance Conclusion
  66. 66 More Questions? Visit us at the AMA

  67. www.elastic.co

  68. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/

    Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 68 Please attribute Elastic with a link to elastic.co