What's the Latest in Logstash?

Logstash team and tech leads address questions like: Can we go faster? What is the persistent queue? How do I monitor Logstash? What is the future of Logstash configuration? How does the team keep systadmins and DevOps in mind when working on the product? And what's in store for the Logstash UI?

Andrew Cholakian l Software Engineer l Elastic
Suyog Rao l Logstash Team Lead l Elastic
Jordan Sissel l Logstash Creator & Tech Lead l Elastic

Transcript

1. Andrew Cholakian Suyog Rao Jordan Sissel cool stuff: 2017
2. It’s time for numbers!
3. across 228 plugins creating 1080 releases 3131 changes by 210 humans PLUGINS
4. creating 24 releases 2021 changes by 72 humans LOGSTASH
5. theme: resisting data loss
6. input filter output Q
7. input filter output queue memory
9. kernel: Out of memory: Kill process 15334 (java)
10. surviving (temporary) machine failures Persisted Queue
11. input filter output disk 1 2 3 4 1 2 3
12. "queue": { "events": 340190, "type": "persisted", "capacity": { "page_capacity_in_bytes": 262144000, "max_queue_size_in_bytes": 1073741824, "max_unread_events": 0 }, "data": { "path": "/home/jls/build/logstash-5.2.2/data/queue", "free_space_in_bytes": 4822740992, "storage_type": "ext4" } queue stats
13. 13 Impact on Performance
14. Basic Configuration 14 # Enable the persisted queue queue.type: persisted # Maximum size on disk of the queue queue.max_bytes: 4gb
15. Durability Configuration 15 # Checkpoint every 123 writes queue.checkpoint.writes: 123 # Checkpoint every 456 events processed and ack’d queue.checkpoint.acks: 456
16. Management and Monitoring
17. GET http://localhost:9600/ { "host" : "coffee", "version" : "5.2.2", "http_address" : "127.0.0.1:9600", "id" : "a882e2c6-3cf7-410d-93ea-f2d2a9bc2474", "name" : "coffee", "build_date" : "2017-02-24T17:46:55Z", "build_sha" : "57984d20eb28b0df40a59077c600ec1a399d46f5", "build_snapshot" : false }
18. GET http://localhost:9600/_node/stats/jvm … "jvm" : { "threads" : { "count" : 6, "peak_count" : 6 }, "mem" : { "heap_used_in_bytes" : 301445688, "heap_used_percent" : 14, "heap_committed_in_bytes" : 519045120, …
19. GET http://localhost:9600/_node/stats/process … "process" : { "open_file_descriptors" : 56, "peak_open_file_descriptors" : 57, "max_file_descriptors" : 10240, "mem" : { "total_virtual_in_bytes" : 5224431616 }, "cpu" : { "total_in_millis" : 47890212000, …
20. GET http://localhost:9600/_node/stats/pipeline … "plugins" : { "outputs" : [ { "id" : "92ddb0615336293c1757cac81c3bebfa19985e68-2", "events" : { "duration_in_millis" : 104, "in" : 29, "out" : 29 }, "name" : "stdout" …
21. PUT http://localhost:9600/_node/logging { "logger.logstash.outputs.elasticsearch" : "DEBUG" }
23. 23 bin/logstash-plugin install x-pack
24. perhaps, nicer than grok The Dissect Filter
25. 144.23.4.1 - - [13/Mar/2016:02:38:26 -0400] "GET /fancy.html HTTP/1.1" 200 6146 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:51.0) Gecko/ 20100101 Firefox/51.0" logs
26. (?<a0>(?<a1>(?<a2>\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?| Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b) +(?<a3>(?:(?:0[1-9])|(?: [12][0-9])|(?:3[01])|[1-9])) (?<a4>(?!<[0-9])(?<a5>(?:2[0123]|[01][0-9])):(?<a6>(?:[0-5][0-9]))(?::(? <a7>(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9]))) (?<a8>(?:(?<a9>\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?: \.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b))|(?<a10>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.] (?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]| [0-1]?[0-9]{1,2}))(?![0-9])))) (?<a11>(?<a12>(?:[\w._/%-]+))(?:\[(?<a13>\b(?:[1-9][0-9]*)\b)\])?): (? <a14>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?: 25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])):(?<a15>(?:[+-]?(?: [0-9]+))) \[(?<a16>(?<a17>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))/(?<a18>\b(?:Jan(?:uary)?| Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?| Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b)/(?<a19>[0-9]+):(?<a20>(?!<[0-9])(?<a21>(?:2[0123]| [01][0-9])):(?<a22>(?:[0-5][0-9]))(?::(?<a23>(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9])).(?<a24>(?:[+-]? (?:[0-9]+))))\] (?<a25>\S+) (?<a26>\S+)/(?<a27>\S+) (?<a28>(?:[+-]?(?:[0-9]+)))/(?<a29>(?:[+-]?(?: [0-9]+)))/(?<a30>(?:[+-]?(?:[0-9]+)))/(?<a31>(?:[+-]?(?:[0-9]+)))/(?<a32>\S+) (?<a33>(?:[+-]?(?:[0-9]+))) (?<a34>\S+) (?<a35>.*?) (?<a36>.*?) (?<a37>\S+) (?<a38>(?:[+-]?(?:[0-9]+)))/(?<a39>(?:[+-]?(?: [0-9]+)))/(?<a40>(?:[+-]?(?:[0-9]+)))/(?<a41>(?:[+-]?(?:[0-9]+)))/(?<a42>\S+) (?<a43>(?:[+-]?(?:[0-9]+)))/ (?<a44>(?:[+-]?(?:[0-9]+))) \{(?<a45>(?<a46>.*?))\} \{(?<a47>(?<a48>.*?))\} "(?<a49>\b\w+\b) (? <a50>(?<a51>(?:/[A-Za-z0-9$.+!*'(){},~:;=#%_-]*)+)(?:(?<a52>\?[A-Za-z0-9$.+!*'(){},~#%&/=:;_-]*))?) HTTP/(?<a53>(?:(?<a54>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))") regex
27. %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[% {HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent} grok
28. %{clientip} %{ident} %{auth} [%{timestamp}] "%{request}" % {response} %{bytes} "%{referrer}" "%{agent}" dissect
29. %{clientip} %{ident} %{auth} [%{timestamp}] "%{request}" %{response} %{bytes} "%{referrer}" "%{agent}" %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[% {HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent} GROK DISSECT
30. 30 Pipeline Throughput
31. { "name": “ grok" "events": { "duration_in_millis": 36640, "in": 401003, "out": 401002 } } { "name": " dissect" "events": { "duration_in_millis": 18206, "in": 1111072, "out": 1111072 } } 91 µs per event 16 µs per event (5x faster)
32. support all the versions! Kafka plugins
33. 59 kafka plugin releases since Elastic{ON} 2016
34. Kafka has lots of releases… 0.8, 0.9, 0.10.0.x, 0.10.1.x
35. Pipeline Visualization and Beyond Andrew Cholakian (@andrewvc)
36. Logstash performance problems? 36
37. 37 What are my plugins even doing?
38. 38 Are they running as fast as they could be?
39. 39 What does this config even do?
40. 40 Finally, there is a solution!
41. The Logstash Pipeline Visualizer
42. The Logstash Threading Model 42 Input Codec Queue Codec Input Filters Outputs Batcher Filters Outputs Batcher Filters Outputs Batcher Input Thread Input Thread Pipeline Worker Thread Pipeline Worker Thread Pipeline Worker Thread
43. Demo 1
44. Logstash Intermediate Representation (LIR)
45. Current (pre-LIR) pipeline 45 Config File 1 2 3 TreeTop IR Ruby Code
46. How it Works in LIR 46 Config File 1 2 3 4 5 TreeTop IR Imperative IR (LIR) DAG IR (LIR) Graph JSON Format Ruby Code Byte Code Etc.
47. 47 LIR Config File Formats Execution Metrics Correlation A Central Hub for Logstash Execution
48. 48 LIR Logstash Config Language Execution Metrics Correlation New Config File Formats! Ingest Pipeline JSON?
49. 49 LIR Config File Formats Ruby Exec Metrics Correlation Multiple Execution Engines Java Exec
50. 50 LIR Config File Formats Execution Pipeline Visualizer Better Diagnostic Tools Step Debugger?
51. 51 Roadmap 1. Enable LIR in core: In progress! 2. Pipeline Visualizer: In progress! 3. Java Pipeline Execution: Next 4. Visual Pipeline Editor: Next 5. New config languages: Planning Source: Gray Arial10pt
52. Suyog Rao (@suyograo) Centralised Management
53. 53 • Elasticsearch as a remote config store • Manage configurations via UI • Group multiple Logstash under roles • Simple alternative to puppet, chef Centralised Management
54. Demo 2
55. Pipelines: Text Based 55
56. 56 Pipelines: Visual Builder beats (filebeat) grok (parseApache) stdin (testStdin) date (apacheDate) false true condition (tag == _grokparsefailure)
57. Sub-Title Plugins
61. JDBC Lookup 61 SKU PRODUCT STATE 99888555 plum california 78782234 apple michigan 34559099 blueberry new mexico 77837890 avocado california 34559099 pear california 14239596 strawberry new mexico
63. Creating Custom Plugins 63
65. 65 • Ease of management • More integrations • Resiliency and performance Conclusion
66. 66 More Questions? Visit us at the AMA
67. www.elastic.co
68. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 68 Please attribute Elastic with a link to elastic.co