Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Recent enhancements in Fluentd and the future

Recent enhancements in Fluentd and the future

Fluentd meetup Tokyo 2019

8c6d9b9b484e5b3f3fa45b0fbc39dabe?s=128

Yuta Iwama

July 17, 2019
Tweet

Transcript

  1. © 2019 Arm Limited Yuta Iwama Arm Treasure Data Recent

    enhancements in Fluentd and the future
  2. © 2019 Arm Limited 2 About Me: Yuta Iwama •

    Yuta Iwama (@ganmacs) • Joined Arm Treasure Data in April of 2019 • Software Engineer • Fluentd Team & Main Maintainer of Fluentd
  3. © 2019 Arm Limited Recent Enhancements

  4. © 2019 Arm Limited Keepalive for out_forward • Reduce the

    overheads (CPU usage, latency, etc.) of sending events • Introduced keepalive and keepalive_timeout directives • v1.5.0 or later
  5. © 2019 Arm Limited 5 out_forward without keepalive per flushing

    events Normal Security directive Security directive + TLS
  6. © 2019 Arm Limited 6 out_forward without keepalive per flushing

    events Normal Security directive Security directive + TLS Write events Close TCP connection Security handshake Open TCP connection Write events Close TCP connection TLS handshake Security handshake Open TCP connection Write events Close TCP connection Open TCP connection
  7. © 2019 Arm Limited 7 out_forward with keepalive per flushing

    events Normal Security directive Security directive + TLS Write events Close TCP connection Security handshake Open TCP connection Write events Close TCP connection TLS handshake Security handshake Open TCP connection Write events Close TCP connection Open TCP connection Wait with timeout Wait with timeout Wait with timeout
  8. © 2019 Arm Limited Implementation of keepalive • out_forward is

    one of the most complex plugins in Fluentd • It has many features https://docs.fluentd.org/output/forward • Heatbeat – UDP and TCP • Verify connection at startup • Auth with password/username – Security directive on the previous page • ack_response – At least once semantics • Need to implement to go well with these features • SocketCache
  9. © 2019 Arm Limited monitor_agent plugin exposed more metrics •

    Monitoring fluentd itself is an important thing for operating • v1.6.0 or later • emit_records: The total number of emitted records • emit_count: The total number of emit call in output plugin • write_count: The total number of write/try_write call in output plugin • rollback_count: The total number of rollback. rollback happens when write/try_write failed • slow_flush_count: The total number of slow flush. This count will be incremented when buffer flush is longer than slow_flush_log_threshold • flush_time_count: The total time of buffer flush in milliseconds • buffer_stage_length: Current length of staged buffer chunks • buffer_stage_byte_size: Current bytesize of staged buffer chunks • buffer_queue_byte_size: Current bytesize of queued buffer chunks • buffer_available_buffer_space_ratios: Show available space for buffer
  10. © 2019 Arm Limited Http Server helper • Provide http

    server helper to create HTTP server easily • Support all HTTP methods • Use socketry/async-http • Only ruby 2.3 or later • Rewrite monitor_agent with this helper • v1.6.0 or later
  11. © 2019 Arm Limited Response JSON with Http Server helper

    • Http server helper expects an array as return value like below • [${response_status}, ${headers}, ${body}]
  12. © 2019 Arm Limited The Future plan

  13. © 2019 Arm Limited Our plans • Service discovery plugin

    • Create new buffer plugin using chunk I/O • Replace cool.io with async-io • Need to next cool.io since it is in maintenance mode • Need to performance check • Drop support of old ruby 2.1 and 2.2 • Drop support of fluent v0.12.x
  14. © 2019 Arm Limited Why use Service Discovery • We

    recommend fluentd is used as 2 tier architecture for high availability Forwarders Aggregators Data Store
  15. © 2019 Arm Limited Why use Service Discovery(2) • In

    microservices architecture, IP and port can change dynamically 10.0.0.1:1234 10.0.0.1:1236 Forwarders Aggregators
  16. © 2019 Arm Limited Why use Service Discovery(2) • In

    microservices architecture, IP and port can change dynamically • e.g. targets can increase/decrease by auto scaling 10.0.0.1:1234 10.0.0.1:1236 NEW!! 10.0.0.1:1235 Forwarders Aggregators
  17. © 2019 Arm Limited Why use Service Discovery(2) • In

    microservices architecture, IP and port can change dynamically • e.g. targets can increase/decrease by auto scaling • load-balance: e.g. send 10% message like canary deploy 10.0.0.1:1234 10.0.0.1:1236 NEW!!(10%) 10.0.0.1:1235 Forwarders Aggregators
  18. © 2019 Arm Limited Why use Service Discovery(2) • In

    microservices architecture, IP and port can change dynamically • e.g. targets can increase/decrease by auto scaling • How to notify all forwarders these changes? – Send all forwarders signals to reload fluentd? 10.0.0.1:1234 10.0.0.1:1236 NEW!! 10.0.0.1:1235 Forwarders Aggregators
  19. © 2019 Arm Limited Solution 1: L3 Load Balancer •

    No need to notify all forwarders • Can’t dispatch requests equally • They don’t know the protocol of forwarding of fluentd and msgpack 10.0.0.1:1234 10.0.0.1:1236 Forwarders Aggregators Load balancer
  20. © 2019 Arm Limited Solution 2: DNS Round Robin •

    No need to notify all forwarders • Can’t dispatch requests with load 10.0.0.1 10.0.0.2 Forwarders Aggregators Domain Name A record sample.fluentd.org 10.0.0.1 sample.fluentd.org 10.0.0.2 Resolve name sample.fluentd.org first
  21. © 2019 Arm Limited We plan to implement Service Discovery

    plugin • Introduce service_discovery directive • Fetch server data periodically • sd_file, sd_dns(srv), sd_consul, sd_http, etc. • Like prometheus *_sd_config 10.0.0.1 10.0.0.2 Forwarders Aggregators Service Name IP weight service1 10.0.0.1:1234 70 service1 10.0.0.2:1234 30 Fetch data Register myself
  22. © 2019 Arm Limited Current Fluentd Buffer plugins • Two

    buffer plugins • memory buffer plugin • file buffer plugin • file buffer plugin has two files(metadata file and content file) per each chunk • Buffer lifecycle – staged -> queued -> flushed – Current implementation needs to handle two file descriptors in each steps
  23. © 2019 Arm Limited Chunk I/O • ChunkIO is used

    in fluent-bit as inner buffer • Implemented in C • File Layout • https://github.com/edsiper/chunkio/tree/2a5b4b6c6bd1aee7a223cad7b35b4d3ee73a 3134#file-layout
  24. © 2019 Arm Limited Wrapping up • Released new features

    • keepalive for out_forward • http_server helper • More metrics for monitor_agent • Plan to implement features for next release • Service discovery plugin • Use chunk I/O as buffer • Replace cool.io with async-io • Plan to stop supporting • Ruby 2.1 and 2.2 • Fluentd v0.12.x