Treasure Data Summer Internship 2016

1. Internship Final Report Sep 30, 2016 Yuta Iwama
2. Who am I • Yuma Iwama (@ganmacs) • Master’s student, The University of Tokyo • Research: Programming languages (My theme is extending language syntax) • Group: Chiba Shigeru Group
3. What I did in summer intern • Add features and enhancements to Fluentd v0.14.x
4. What I did in summer intern • 6 features • Counter API (Not merged yet) • Data compression in buffer plugins and forward plugins • New out_file plugin only for <secondary> section • A CLI tool to read dumped event data • Log rotation • `filter_with_time` method in filter plugins • 2 enhancements • Optimizing multiple filter calls • Add event size to options in a forward protocol • Some small tasks
5. I’ll talk about • Counter API • Data compression in buffer plugins and forward plugins • New out_file plugin only for <secondary> section • A CLI tool to read dumped event data • Log rotation • Optimizing multiple filter calls
6. I’ll talk about • Counter API • Data compression in buffer plugins and forward plugins • New out_file plugin only for <secondary> section • A CLI tool to read dumped event data • Log rotation • Optimizing multiple filter calls
7. Data Compression in buffer plugins and forward plugins
8. Current buffer plugins and forward plugins in Fluentd • Buffer plugins have data as a string (formatted with MessagePack or user custom formats) • Forward plugins send data as a string (format is same as buffer plugins) • Although data is serialized with MessagePack, its footprint is large • Current way consumes many memory resources and bandwidth of the network
9. New buffer plugins and forward plugins • String data in buffer plugins can be compressed • Forward plugins can send and receive compressed data • Things to be able to • Save the bandwidth across the datacenter • Accelerate the transfer speed and save the time • Reduce memory consumptions and costs of IaaS (EC2 , etc.)
10. Implementation • I used “zlib” in Ruby to implement a compression/decompression method • It’s hard to work both compressed version and raw version (To solve this problem, I used `extend` in Ruby not to break existing interface)
11. New out_file plugin only for <secondary> section
12. Background • Many users use out_file plugin to dump buffer with <secondary> sections when primary buffered output plugins are failing flush • But out_file is too complex and has too many features for such purpose • => We need simple out_file only for <secondary> section just to dump buffer
13. New plugin: secondary_out_file • Only four attributes • directory: the directory dumped data saved • basename: the file name of dumped data (default value is dump.bin) • append: the flushed data is appended to an existing file or not (default false) • compress: The type of the file ( gzip or txt, default is txt) • Users can use this plugin only to set directory<match > @type forward ... <secondary> type secondary_file directory log/secondary/ </secondary> </match>
14. A CLI tool which is used for reading dump data
15. Background • Dumped data are created by secondary plugins (e.g. secondary_out_file) when primary plugins are failing flush • We can't read dumped data because dumped data is binary format(MessagePack) in most case • => Provide a CLI tool to read dumped data
16. fluent-binlog-reader • fluent-buinlog-reader is bundled in Fluent • It reads dumped data and outputs readable format • Users can use fluent’s formatter plugins as an output format $ fluent-binlog-reader --help Usage: fluent-binlog-reader <command> [<args>] Commands of fluent-binlog-reader: cat : Read files sequentially, writing them to standard output. head : Display the beginning of a text file. format : Display plugins that you can use. See 'fluent-binlog-reader <command> --help' for more information on a specific command.
17. fluent-binlog-reader $ fluent-binlog-reader head packed.log 2016-08-12T17:24:18+09:00 packed.log {"message":"dummy"} 2016-08-12T17:24:18+09:00 packed.log {"message":"dummy"} 2016-08-12T17:24:18+09:00 packed.log {"message":"dummy"} 2016-08-12T17:24:18+09:00 packed.log {"message":"dummy"} 2016-08-12T17:24:18+09:00 packed.log {"message":"dummy"} Default format is json format Using a “csv formatter” to output dumped data $ fluent-binlog-reader cat --formats=csv -e fields=message packed.log "dummy" ... "dummy"
18. Log rotation
19. Background • Fluentd can’t do log rotation • As the file size of log increases, it becomes difficult to handle the log file. • => Fluentd supports log rotation to keep log files down to a manageable size
20. Log rotation • Two options • log-rotate-age: The number of old log files to keep • log-rotate-size: Maximum log file size • Use serverengine log rotation (Fluentd uses serverengine logger to one’s log)
21. Optimise multiple filter calls
22. Background • If users apply multiple filters to incoming events, Fluentd creates a lot of EventStream object and calls its add method • => Removing useless instantiations of EventStream and the `add` method calls
23. Filter 1 1. Create an EventStream object (1 time) 2. Apply a filter to each event (5 times) 3. Add a filtered event to an EventStream object (5times) [e1, e2, e3, e4, e5] If 10 filters are applied 1. call 10 times 2. call 50 times 3. call 50 times [e1’, e2’, e3’, e4’, e5’] [e1x, e2x, e3x, e4x, e5x] Filter n Current filters
24. Filter 1 1. Create an EventStream object (1 time) 2. Apply each filters to each event (n * 5 times) 3. Add a filtered event to an EventStream object (5times) [e1, e2, e3, e4, e5] if 10 filters are applied 1. call 1 time 2. call 50 times 3. call 5 times [e1x, e2x, e3x, e4x, e5x] Filter n + Constraint: These filters must not be implemented `filter_stream` method Optimised case
25. Performance Tool : ruby-prof ProductName: Mac OS X ProductVersion: 10.11.6 BuildVersion: 15G31 PROCESSOR: 2.7 GHz Intel Core i5 MEMORY: 8 GB 1867 MHz DDR3 /PU
    PQUJNJ[FE 0QUJNJTFE   1.2 times faster when it is using 10 filters and 1000 events per sec
26. I’ll talk about • Counter API • Data compression in buffer plugins and forward plugins • New out_file plugin only for <secondary> section • A CLI tool to read dumped event data • Log rotation • Optimizing multiple filter calls
27. Counter API
28. Motivations • To get metrics of Fluentd itself between processes • To provide counter API to 3rd party plugins • It is useful to implement counter plugins (e.g. fluent-plugin-datacounter and fluent- plugin-flowcounter)
29. What’s the counter • The counter: • A key-value store • Used for storing the number of occurrences of a particular event in the specified time • Provides API to users to operate its value(e.g. inc, reset , etc.) • shared between processes
30. $PVOUFS LFZ WBMVF LFZ  LFZ  LFZ  Counter Process 1 inc(key1 => 2) Process 2 reset(key2) What’s the counter (cont.)
31. $PVOUFS Counter What’s the counter (cont.) LFZ WBMVF LFZ  LFZ  LFZ  Process 1 inc(key1 => 2) Process 2 reset(key2)
32. Implementation • RPC server and client • All operators should be thread safe • Cleaning mutex objects for keys
33. Implementation • RPC server and client • All operators should be thread safe • Cleaning mutex objects for keys
34. RPC server and client • Because the counter is shared between processes. We need a server and clients (Store counter values in server and clients manipulate them by RPC ) • I designed RPC server and client for counter • I use cool.io to implement RPC server and client • cool.io is providing a high-performance event framework for Ruby (https://coolio.github.io/)
35. API to operate counter values • init: create new value • reset: reset a counter value • delete: delete a counter value • inc: increment or decrement a counter value • get: fetch a counter value
36. Implementation • RPC server and client • All operators should be thread safe • Cleaning mutex objects for keys
37. All operations should be a thread safe • Counter works in multi threads • You need to get a lock per keys when you change a counter value • Counter stores mutex objects in hash (key_name => mutex_object)
38. How an inc method works LFZ WBMVF LFZ  Counter server client in worker1 inc( key1 => 2) 1. Call an inc method LFZ WBMVF LFZ NVUFY
    PCK Mutex hash
39. How an inc method works LFZ WBMVF LFZ  Counter server LFZ WBMVF LFZ NVUFY
    PCK Mutex hash client in worker1 inc( key1 => 2) 1. Call an inc method 2. Get a lock for a mutex hash locked
40. How an inc method works LFZ WBMVF LFZ  Counter server LFZ WBMVF LFZ MPDLFE Mutex hash client in worker1 inc( key1 => 2) 1. Call an inc method 2. Get a lock for a mutex hash 3. Get a lock for a key locked
41. How an inc method works LFZ WBMVF LFZ  Counter server LFZ WBMVF LFZ MPDLFE Mutex hash client in worker1 inc( key1 => 2) 1. Call an inc method 2. Get a lock for a mutex hash 3. Get a lock for a key 4. Unlock a mutex hash
42. How an inc method works LFZ WBMVF LFZ  Counter server LFZ WBMVF LFZ MPDLFE Mutex hash client in worker1 inc( key1 => 2) 1. Call an inc method 2. Get a lock for a mutex hash 3. Get a lock for a key 4. Unlock a mutex hash 5. Change a counter value
43. How an inc method works LFZ WBMVF LFZ  Counter server LFZ WBMVF LFZ VOMPDL Mutex hash client in worker1 inc( key1 => 2) 1. Call an inc method 2. Get a lock for a mutex hash 3. Get a lock for a key 4. Unlock a mutex hash 5. Change a counter value 6. Unlock a key lock
44. Implementation • RPC server and client • All operators should be thread safe • Cleaning mutex objects for keys
45. Mutex objects for keys • To avoid storing mutex objects for all keys, I implement a cleanup thread which removes unused key’s mutex object (like GC) • This thread removes mutex objects which are not used for a certain period
46. Cleaning up a mutex hash LFZ WBMVF LFZ  Counter server LFZ WBMVF LFZ NVUFY
    PCK Mutex hash • If “key1” are not modified for a long period, “key1” may be unused after this
47. Cleaning up a mutex hash LFZ WBMVF LFZ NVUFY
    PCK Mutex hash • If “key1” are not modified for a long period, “key1” may be unused after this 1. Start a cleaning thread (once in 15 min)
48. Cleaning up a mutex hash LFZ WBMVF LFZ NVUFY
    PCK Mutex hash • If “key1” are not modified for a long period, “key1” may be unused after this 1. Start a cleaning thread (once in 15 min) 2. Get a lock for a mutex hash locked
49. Cleaning up a mutex hash LFZ WBMVF Mutex hash • If “key1” are not modified for a long period, “key1” may be unused after this 1. Start a cleaning thread (once in 15 min) 2. Get a lock for a mutex hash 3. Remove a mutex for an unused key locked
50. Cleaning up a mutex hash LFZ WBMVF Mutex hash • If “key1” are not modified for a long period, “key1” may be unused after this 1. Start a cleaning thread (once in 15 min) 2. Get a lock for a mutex hash 3. Remove a mutex for an unused key 4. Try to get a lock for the same key If this thread can’t get a lock
 restore a key-value locked
51. Cleaning up a mutex hash LFZ WBMVF Mutex hash • If “key1” are not modified for a long period, “key1” may be unused after this 1. Start a cleaning thread (once in 15 min) 2. Get a lock for a mutex hash 3. Remove a mutex for an unused key 4. Try to get a lock for the same key 5. Unlock a mutex hash
52. Summary • Add six features and two enhancements to Fluentd v0.14.x • Counter API is not merged yet • Other PRs have been merged
53. Impression of intern • The hardest thing for me is to design about counter API(It takes over 1 week) • I have learned about the development of middleware which is used by many people • I want to became more careful to code written by myself (typo, description, comment etc.)