Recent enhancements in Fluentd and the future

Fluentd meetup Tokyo 2019

Transcript

1. © 2019 Arm Limited Yuta Iwama Arm Treasure Data Recent enhancements in Fluentd and the future
2. © 2019 Arm Limited 2 About Me: Yuta Iwama • Yuta Iwama (@ganmacs) • Joined Arm Treasure Data in April of 2019 • Software Engineer • Fluentd Team & Main Maintainer of Fluentd
3. © 2019 Arm Limited Recent Enhancements
4. © 2019 Arm Limited Keepalive for out_forward • Reduce the overheads (CPU usage, latency, etc.) of sending events • Introduced keepalive and keepalive_timeout directives • v1.5.0 or later
5. © 2019 Arm Limited 5 out_forward without keepalive per flushing events Normal Security directive Security directive + TLS
6. © 2019 Arm Limited 6 out_forward without keepalive per flushing events Normal Security directive Security directive + TLS Write events Close TCP connection Security handshake Open TCP connection Write events Close TCP connection TLS handshake Security handshake Open TCP connection Write events Close TCP connection Open TCP connection
7. © 2019 Arm Limited 7 out_forward with keepalive per flushing events Normal Security directive Security directive + TLS Write events Close TCP connection Security handshake Open TCP connection Write events Close TCP connection TLS handshake Security handshake Open TCP connection Write events Close TCP connection Open TCP connection Wait with timeout Wait with timeout Wait with timeout
8. © 2019 Arm Limited Implementation of keepalive • out_forward is one of the most complex plugins in Fluentd • It has many features https://docs.fluentd.org/output/forward • Heatbeat – UDP and TCP • Verify connection at startup • Auth with password/username – Security directive on the previous page • ack_response – At least once semantics • Need to implement to go well with these features • SocketCache
9. © 2019 Arm Limited monitor_agent plugin exposed more metrics • Monitoring fluentd itself is an important thing for operating • v1.6.0 or later • emit_records: The total number of emitted records • emit_count: The total number of emit call in output plugin • write_count: The total number of write/try_write call in output plugin • rollback_count: The total number of rollback. rollback happens when write/try_write failed • slow_flush_count: The total number of slow flush. This count will be incremented when buffer flush is longer than slow_flush_log_threshold • flush_time_count: The total time of buffer flush in milliseconds • buffer_stage_length: Current length of staged buffer chunks • buffer_stage_byte_size: Current bytesize of staged buffer chunks • buffer_queue_byte_size: Current bytesize of queued buffer chunks • buffer_available_buffer_space_ratios: Show available space for buffer
10. © 2019 Arm Limited Http Server helper • Provide http server helper to create HTTP server easily • Support all HTTP methods • Use socketry/async-http • Only ruby 2.3 or later • Rewrite monitor_agent with this helper • v1.6.0 or later
11. © 2019 Arm Limited Response JSON with Http Server helper • Http server helper expects an array as return value like below • [${response_status}, ${headers}, ${body}]
12. © 2019 Arm Limited The Future plan
13. © 2019 Arm Limited Our plans • Service discovery plugin • Create new buffer plugin using chunk I/O • Replace cool.io with async-io • Need to next cool.io since it is in maintenance mode • Need to performance check • Drop support of old ruby 2.1 and 2.2 • Drop support of fluent v0.12.x
14. © 2019 Arm Limited Why use Service Discovery • We recommend fluentd is used as 2 tier architecture for high availability Forwarders Aggregators Data Store
15. © 2019 Arm Limited Why use Service Discovery(2) • In microservices architecture, IP and port can change dynamically 10.0.0.1:1234 10.0.0.1:1236 Forwarders Aggregators
16. © 2019 Arm Limited Why use Service Discovery(2) • In microservices architecture, IP and port can change dynamically • e.g. targets can increase/decrease by auto scaling 10.0.0.1:1234 10.0.0.1:1236 NEW!! 10.0.0.1:1235 Forwarders Aggregators
17. © 2019 Arm Limited Why use Service Discovery(2) • In microservices architecture, IP and port can change dynamically • e.g. targets can increase/decrease by auto scaling • load-balance: e.g. send 10% message like canary deploy 10.0.0.1:1234 10.0.0.1:1236 NEW!!(10%) 10.0.0.1:1235 Forwarders Aggregators
18. © 2019 Arm Limited Why use Service Discovery(2) • In microservices architecture, IP and port can change dynamically • e.g. targets can increase/decrease by auto scaling • How to notify all forwarders these changes? – Send all forwarders signals to reload fluentd? 10.0.0.1:1234 10.0.0.1:1236 NEW!! 10.0.0.1:1235 Forwarders Aggregators
19. © 2019 Arm Limited Solution 1: L3 Load Balancer • No need to notify all forwarders • Can’t dispatch requests equally • They don’t know the protocol of forwarding of fluentd and msgpack 10.0.0.1:1234 10.0.0.1:1236 Forwarders Aggregators Load balancer
20. © 2019 Arm Limited Solution 2: DNS Round Robin • No need to notify all forwarders • Can’t dispatch requests with load 10.0.0.1 10.0.0.2 Forwarders Aggregators Domain Name A record sample.fluentd.org 10.0.0.1 sample.fluentd.org 10.0.0.2 Resolve name sample.fluentd.org first
21. © 2019 Arm Limited We plan to implement Service Discovery plugin • Introduce service_discovery directive • Fetch server data periodically • sd_file, sd_dns(srv), sd_consul, sd_http, etc. • Like prometheus *_sd_config 10.0.0.1 10.0.0.2 Forwarders Aggregators Service Name IP weight service1 10.0.0.1:1234 70 service1 10.0.0.2:1234 30 Fetch data Register myself
22. © 2019 Arm Limited Current Fluentd Buffer plugins • Two buffer plugins • memory buffer plugin • file buffer plugin • file buffer plugin has two files(metadata file and content file) per each chunk • Buffer lifecycle – staged -> queued -> flushed – Current implementation needs to handle two file descriptors in each steps
23. © 2019 Arm Limited Chunk I/O • ChunkIO is used in fluent-bit as inner buffer • Implemented in C • File Layout • https://github.com/edsiper/chunkio/tree/2a5b4b6c6bd1aee7a223cad7b35b4d3ee73a 3134#file-layout
24. © 2019 Arm Limited Wrapping up • Released new features • keepalive for out_forward • http_server helper • More metrics for monitor_agent • Plan to implement features for next release • Service discovery plugin • Use chunk I/O as buffer • Replace cool.io with async-io • Plan to stop supporting • Ruby 2.1 and 2.2 • Fluentd v0.12.x