Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Treasure Data Summer Internship 2016

8c6d9b9b484e5b3f3fa45b0fbc39dabe?s=47 Yuta Iwama
September 30, 2016

Treasure Data Summer Internship 2016

8c6d9b9b484e5b3f3fa45b0fbc39dabe?s=128

Yuta Iwama

September 30, 2016
Tweet

Transcript

  1. Internship Final Report Sep 30, 2016 Yuta Iwama

  2. Who am I • Yuma Iwama (@ganmacs) • Master’s student,

    The University of Tokyo • Research: Programming languages (My theme is extending language syntax) • Group: Chiba Shigeru Group
  3. What I did in summer intern • Add features and

    enhancements to Fluentd v0.14.x
  4. What I did in summer intern • 6 features •

    Counter API (Not merged yet) • Data compression in buffer plugins and forward plugins • New out_file plugin only for <secondary> section • A CLI tool to read dumped event data • Log rotation • `filter_with_time` method in filter plugins • 2 enhancements • Optimizing multiple filter calls • Add event size to options in a forward protocol • Some small tasks
  5. I’ll talk about • Counter API • Data compression in

    buffer plugins and forward plugins • New out_file plugin only for <secondary> section • A CLI tool to read dumped event data • Log rotation • Optimizing multiple filter calls
  6. I’ll talk about • Counter API • Data compression in

    buffer plugins and forward plugins • New out_file plugin only for <secondary> section • A CLI tool to read dumped event data • Log rotation • Optimizing multiple filter calls
  7. Data Compression in buffer plugins and forward plugins

  8. Current buffer plugins and forward plugins in Fluentd • Buffer

    plugins have data as a string (formatted with MessagePack or user custom formats) • Forward plugins send data as a string (format is same as buffer plugins) • Although data is serialized with MessagePack, its footprint is large • Current way consumes many memory resources and bandwidth of the network
  9. New buffer plugins and forward plugins • String data in

    buffer plugins can be compressed • Forward plugins can send and receive compressed data • Things to be able to • Save the bandwidth across the datacenter • Accelerate the transfer speed and save the time • Reduce memory consumptions and costs of IaaS (EC2 , etc.)
  10. Implementation • I used “zlib” in Ruby to implement a

    compression/decompression method • It’s hard to work both compressed version and raw version (To solve this problem, I used `extend` in Ruby not to break existing interface)
  11. New out_file plugin only for <secondary> section

  12. Background • Many users use out_file plugin to dump buffer

    with <secondary> sections when primary buffered output plugins are failing flush • But out_file is too complex and has too many features for such purpose • => We need simple out_file only for <secondary> section just to dump buffer
  13. New plugin: secondary_out_file • Only four attributes • directory: the

    directory dumped data saved • basename: the file name of dumped data (default value is dump.bin) • append: the flushed data is appended to an existing file or not (default false) • compress: The type of the file ( gzip or txt, default is txt) • Users can use this plugin only to set directory<match > @type forward ... <secondary> type secondary_file directory log/secondary/ </secondary> </match>
  14. A CLI tool which is used for reading dump data

  15. Background • Dumped data are created by secondary plugins (e.g.

    secondary_out_file) when primary plugins are failing flush • We can't read dumped data because dumped data is binary format(MessagePack) in most case • => Provide a CLI tool to read dumped data
  16. fluent-binlog-reader • fluent-buinlog-reader is bundled in Fluent • It reads

    dumped data and outputs readable format • Users can use fluent’s formatter plugins as an output format $ fluent-binlog-reader --help Usage: fluent-binlog-reader <command> [<args>] Commands of fluent-binlog-reader: cat : Read files sequentially, writing them to standard output. head : Display the beginning of a text file. format : Display plugins that you can use. See 'fluent-binlog-reader <command> --help' for more information on a specific command.
  17. fluent-binlog-reader $ fluent-binlog-reader head packed.log 2016-08-12T17:24:18+09:00 packed.log {"message":"dummy"} 2016-08-12T17:24:18+09:00 packed.log

    {"message":"dummy"} 2016-08-12T17:24:18+09:00 packed.log {"message":"dummy"} 2016-08-12T17:24:18+09:00 packed.log {"message":"dummy"} 2016-08-12T17:24:18+09:00 packed.log {"message":"dummy"} Default format is json format Using a “csv formatter” to output dumped data $ fluent-binlog-reader cat --formats=csv -e fields=message packed.log "dummy" ... "dummy"
  18. Log rotation

  19. Background • Fluentd can’t do log rotation • As the

    file size of log increases, it becomes difficult to handle the log file. • => Fluentd supports log rotation to keep log files down to a manageable size
  20. Log rotation • Two options • log-rotate-age: The number of

    old log files to keep • log-rotate-size: Maximum log file size • Use serverengine log rotation (Fluentd uses serverengine logger to one’s log)
  21. Optimise multiple filter calls

  22. Background • If users apply multiple filters to incoming events,

    Fluentd creates a lot of EventStream object and calls its add method • => Removing useless instantiations of EventStream and the `add` method calls
  23. Filter 1 1. Create an EventStream object (1 time) 2.

    Apply a filter to each event (5 times) 3. Add a filtered event to an EventStream object (5times) [e1, e2, e3, e4, e5] If 10 filters are applied 1. call 10 times 2. call 50 times 3. call 50 times [e1’, e2’, e3’, e4’, e5’] [e1x, e2x, e3x, e4x, e5x] Filter n Current filters
  24. Filter 1 1. Create an EventStream object (1 time) 2.

    Apply each filters to each event (n * 5 times) 3. Add a filtered event to an EventStream object (5times) [e1, e2, e3, e4, e5] if 10 filters are applied 1. call 1 time 2. call 50 times 3. call 5 times [e1x, e2x, e3x, e4x, e5x] Filter n + Constraint: These filters must not be implemented `filter_stream` method Optimised case
  25. Performance Tool : ruby-prof ProductName: Mac OS X ProductVersion: 10.11.6

    BuildVersion: 15G31 PROCESSOR: 2.7 GHz Intel Core i5 MEMORY: 8 GB 1867 MHz DDR3 /PU PQUJNJ[FE 0QUJNJTFE   1.2 times faster when it is using 10 filters and 1000 events per sec
  26. I’ll talk about • Counter API • Data compression in

    buffer plugins and forward plugins • New out_file plugin only for <secondary> section • A CLI tool to read dumped event data • Log rotation • Optimizing multiple filter calls
  27. Counter API

  28. Motivations • To get metrics of Fluentd itself between processes

    • To provide counter API to 3rd party plugins • It is useful to implement counter plugins (e.g. fluent-plugin-datacounter and fluent- plugin-flowcounter)
  29. What’s the counter • The counter: • A key-value store

    • Used for storing the number of occurrences of a particular event in the specified time • Provides API to users to operate its value(e.g. inc, reset , etc.) • shared between processes
  30. $PVOUFS LFZ WBMVF LFZ  LFZ  LFZ  Counter

    Process 1 inc(key1 => 2) Process 2 reset(key2) What’s the counter (cont.)
  31. $PVOUFS Counter What’s the counter (cont.) LFZ WBMVF LFZ 

    LFZ  LFZ  Process 1 inc(key1 => 2) Process 2 reset(key2)
  32. Implementation • RPC server and client • All operators should

    be thread safe • Cleaning mutex objects for keys
  33. Implementation • RPC server and client • All operators should

    be thread safe • Cleaning mutex objects for keys
  34. RPC server and client • Because the counter is shared

    between processes. We need a server and clients (Store counter values in server and clients manipulate them by RPC ) • I designed RPC server and client for counter • I use cool.io to implement RPC server and client • cool.io is providing a high-performance event framework for Ruby (https://coolio.github.io/)
  35. API to operate counter values • init: create new value

    • reset: reset a counter value • delete: delete a counter value • inc: increment or decrement a counter value • get: fetch a counter value
  36. Implementation • RPC server and client • All operators should

    be thread safe • Cleaning mutex objects for keys
  37. All operations should be a thread safe • Counter works

    in multi threads • You need to get a lock per keys when you change a counter value • Counter stores mutex objects in hash (key_name => mutex_object)
  38. How an inc method works LFZ WBMVF LFZ  Counter

    server client in worker1 inc( key1 => 2) 1. Call an inc method LFZ WBMVF LFZ NVUFYPCK Mutex hash
  39. How an inc method works LFZ WBMVF LFZ  Counter

    server LFZ WBMVF LFZ NVUFYPCK Mutex hash client in worker1 inc( key1 => 2) 1. Call an inc method 2. Get a lock for a mutex hash locked
  40. How an inc method works LFZ WBMVF LFZ  Counter

    server LFZ WBMVF LFZ MPDLFE Mutex hash client in worker1 inc( key1 => 2) 1. Call an inc method 2. Get a lock for a mutex hash 3. Get a lock for a key locked
  41. How an inc method works LFZ WBMVF LFZ  Counter

    server LFZ WBMVF LFZ MPDLFE Mutex hash client in worker1 inc( key1 => 2) 1. Call an inc method 2. Get a lock for a mutex hash 3. Get a lock for a key 4. Unlock a mutex hash
  42. How an inc method works LFZ WBMVF LFZ  Counter

    server LFZ WBMVF LFZ MPDLFE Mutex hash client in worker1 inc( key1 => 2) 1. Call an inc method 2. Get a lock for a mutex hash 3. Get a lock for a key 4. Unlock a mutex hash 5. Change a counter value
  43. How an inc method works LFZ WBMVF LFZ  Counter

    server LFZ WBMVF LFZ VOMPDL Mutex hash client in worker1 inc( key1 => 2) 1. Call an inc method 2. Get a lock for a mutex hash 3. Get a lock for a key 4. Unlock a mutex hash 5. Change a counter value 6. Unlock a key lock
  44. Implementation • RPC server and client • All operators should

    be thread safe • Cleaning mutex objects for keys
  45. Mutex objects for keys • To avoid storing mutex objects

    for all keys, I implement a cleanup thread which removes unused key’s mutex object (like GC) • This thread removes mutex objects which are not used for a certain period
  46. Cleaning up a mutex hash LFZ WBMVF LFZ  Counter

    server LFZ WBMVF LFZ NVUFYPCK Mutex hash • If “key1” are not modified for a long period, “key1” may be unused after this
  47. Cleaning up a mutex hash LFZ WBMVF LFZ NVUFYPCK Mutex

    hash • If “key1” are not modified for a long period, “key1” may be unused after this 1. Start a cleaning thread (once in 15 min)
  48. Cleaning up a mutex hash LFZ WBMVF LFZ NVUFYPCK Mutex

    hash • If “key1” are not modified for a long period, “key1” may be unused after this 1. Start a cleaning thread (once in 15 min) 2. Get a lock for a mutex hash locked
  49. Cleaning up a mutex hash LFZ WBMVF Mutex hash •

    If “key1” are not modified for a long period, “key1” may be unused after this 1. Start a cleaning thread (once in 15 min) 2. Get a lock for a mutex hash 3. Remove a mutex for an unused key locked
  50. Cleaning up a mutex hash LFZ WBMVF Mutex hash •

    If “key1” are not modified for a long period, “key1” may be unused after this 1. Start a cleaning thread (once in 15 min) 2. Get a lock for a mutex hash 3. Remove a mutex for an unused key 4. Try to get a lock for the same key If this thread can’t get a lock
 restore a key-value locked
  51. Cleaning up a mutex hash LFZ WBMVF Mutex hash •

    If “key1” are not modified for a long period, “key1” may be unused after this 1. Start a cleaning thread (once in 15 min) 2. Get a lock for a mutex hash 3. Remove a mutex for an unused key 4. Try to get a lock for the same key 5. Unlock a mutex hash
  52. Summary • Add six features and two enhancements to Fluentd

    v0.14.x • Counter API is not merged yet • Other PRs have been merged
  53. Impression of intern • The hardest thing for me is

    to design about counter API(It takes over 1 week) • I have learned about the development of middleware which is used by many people • I want to became more careful to code written by myself (typo, description, comment etc.)