Introduce log collector in Go into production Tatsuhiko Kubo@cubicdaiya Go Conference 2015 Winter

@cubicdaiya / Tatsuhiko Kubo Site Reliability Engineer @ Mercari, Inc. C Go Lua nginx ngx_small_light, ngx_dynamic_upstream, nginx-build, slackboard, cachectl, gaurun, etc…

No content

https://www.mercari.com/ Mercari - Your Friendly Mobile Marketplace

mercari.go • cubicdaiya/nginx-build • cubicdaiya/cachectl • cubicdaiya/slackboard • mercari/gaurun • etc… (in-house cli and daemon)

Agenda • Casual introduction of log-collectors in Go • fluent-forwarder, fluent-agent-hydra, Heka • In the case of Mercari • Fluentd and fluent-agent-hydra • Road to introduce fluent-agent-hydra into production environment

Casual introduction of log-collectors in Go

fluent-forwarder • https://github.com/fluent/fluentd-forwarder • Light-weight Fluentd forworder • Not provide plugin system • Supports Windows

fluent-agent-hydra • https://github.com/fujiwara/fluent-agent-hydra • Features • in_tail & (in|out)_forward • Handle multiple file per 1 process • Monitoring API • Configuration with TOML • Support LTSV and JSON

Heka • https://github.com/mozilla-services/heka • Event processing / windowed stream operations • Provides Plugin system • User can write plugin in Go or Lua • Custom Input, Output, Encoder, Decoder, Filter • Configuration with TOML

In the case of Mercari

Transfer various logs with Fluentd "QQ 8PSLFS #BUDI ,JCBOB #JH2VFSZ /PSJLSB .BDLFSFM

Fluentd • Fluentd is very flexible and robust log- collector • Plugin systems • Active developers and communities • But performance is not always enough

0QFO3FTUZ 0QFO3FTUZ 0QFO3FTUZ (PPHMF#JH2VFSZ Developer Data Sientist Analyze by SQL send events send events send events Powered by cookpad/puree-(ios|android) utilize events utilize events utilize events PascalʙMercari analysis baseʙ (Before) in_tail & out_forward

0QFO3FTUZ 0QFO3FTUZ 0QFO3FTUZ (PPHMF#JH2VFSZ Developer Data Sientist Analyze by SQL send events send events send events Powered by cookpad/puree-(ios|android) utilize events utilize events utilize events hydra(※) hydra(※) hydra(※) (※) fluent-agent-hydra PascalʙMercari analysis baseʙ (Now) in_tail & out_forward

PascalʙMercari analysis baseʙ • Built with the software blocks below • Puree, OpenResty (ngx_lua), Fluentd, fluent-agent-hydra, Google BigQuery • Aggregate various logs to Google BigQuery • Event in app • A/B Testing • etc…

Why switched to fluent-agent-hydraʁ • Low server resource is required for Pascal • Fluentd had consumed non-negligible amount of CPU resource (50ʙ60% at peak) • Pascal indicates modestly high workload • OpenResty processes a lot of JSONs and outputs various logs • Requests are come from not only device but API-servers also

Why switched to fluent-agent-hydraʁ • fluent-agent-hydra • CPU usage is less than Fluentd • Half as compared to Fluentd in our caseʂ • Enable handling multiple logs per 1 process • Simple

fluent-agent-hydra internal ɿgoroutine monitor out_forwarder in_forwarder watcher & in_tail for file go func() go func() go func() go func() run main() wait signal Some gorouines make more goroutines

goroutines communicates with channel ɿgoroutine monitor out_forwarder in_forwarder watcher & in_tail for file receiver is monitor receiver is out_forwarder

Road to introduce fluent-agent-hydra into production environment

Monitoring fluent-agent-hydra • fluent-agent-hydra provides monitoring APIs • Application stats • current positions for tailing log files • sent amount and bytes per a log • other informations (e.g. error) • System stats • Powered by golang-stats-api-handler

Application Stats DVSMT<.POJUPS)PTU><.POJUPS1PSU>cKRb` { "receiver": null, "servers": [ { "error": "", "alive": true, "address": "host1:24224" }, { "error": "", "alive": true, "address": “host2:24225" } ], "files": { “/var/log/nginx/access.log“: { "error": "", "position": 472132311, "tag": “nginx.access_log” }, … …

System Stats DVSMT<.POJUPS)PTU><.POJUPS1PSU>TZTUFNcKRb` { "gc_pause": [], "gc_pause_per_second": 0, "gc_per_second": 0, "gc_num": 216, "gc_last": 1449345754024961000, "gc_next": 98962453, "heap_objects": 590421, "heap_released": 0, "heap_inuse": 90906624, "heap_idle": 11984896, "heap_sys": 102891520, "cgo_call_num": 1, "gomaxprocs": 8, "goroutine_num": 31, "cpu_num": 8, "go_arch": "amd64", "go_os": "linux", "go_version": "go1.5.1", "time": 1449345783047388700, "memory_alloc": 88036000, "memory_total_alloc": 10373554664, "memory_sys": 111020280, "memory_lookups": 1343, "memory_mallocs": 162497629, "memory_frees": 161907208, "memory_stack": 917504, "heap_alloc": 88036000 }

Alerting to Slack

After switched to fluent-agent-hydra

After switched to fluent-agent-hydra BigQuery error in load operation: Error processing job Field:xxx: Cloud not convert value to integer ( bad value or out of range ) Field:yyy: Cloud not convert value to integer ( bad value or out of range ) Field:xxx: Cloud not convert value to integer ( bad value or out of range ) Field:xxx: Cloud not convert value to integer ( bad value or out of range ) Field:xxx: Cloud not convert value to integer ( bad value or out of range ) … ʂʁ

Bigquery’s demand • Google Bigquery demands fixed table schema and strict data format [ { “name”:”value”, “type”:”INTEGER” }, ] ■ schema.json {“value”:150} ■ valid data foramt ■ invalid data format {“value”:150.0} {“value”:”150”}

Special conversion behavior for numerical value • fluent-agent-hydra treats a numerical value as float64 even if its type is integer • When log-format is JSON • Whyʁ • Because fluent-agent-hydra uses encoding/json and unmarshal JSON into interface values

According to document, https://golang.org/pkg/encoding/json/

According to document, https://golang.org/pkg/encoding/json/

{“value”:150} {“value”:150.0} (In actual, msgpack) (PPHMF#JH2VFSZ 0QFO3FTUZ qVFOUBHFOU IZESB out_forward & out_file in_tail & out_forward output to log file -PH 4FSWFS bq load xxx.log schema.json Error processing job [ { “name”:”value”, “type”:”INTEGER” }, ] schema.json {“value”:150.0}

By the way, • fluent-agent-hydra provides the directive Types for converting type # in config.toml Types = “value:integer”

By the way, • fluent-agent-hydra provides the directive Types for converting type # in config.toml Types = “value:integer” But this was provided for only LTSV at that time…

https://github.com/fujiwara/fluent-agent-hydra/pull/5 Therefore,

https://github.com/fujiwara/fluent-agent-hydra/pull/6 Moreover,

Now • fluent-agent-hydra provides the directive Types for converting type # in config.toml Types = “value:integer” Always convert type to int64 regardless format is LTSV or JSON

https://github.com/fujiwara/fluent-agent-hydra/blob/master/ README.md#about-special-conversion-behavior-for-numerical-value Now

0QFO3FTUZ 0QFO3FTUZ 0QFO3FTUZ (PPHMF#JH2VFSZ Developer Data Sientist Analyze by SQL send events send events send events Powered by cookpad/puree-(ios|android) utilize events utilize events utilize events hydra(※) hydra(※) hydra(※) (※) fluent-agent-hydra PascalʙMercari analysis baseʙ (Now) in_tail & out_forward

Summary • Fluentd is very flexible and robust log-collector • But performance is not always enough • There are some alternatives in Go • fluent-agent-hydra might fit the case below • Want a faster and light-weight log-collector for in_tail & out_forward • But robustness is less than Fluentd • e.g. position file is not supported

We are hiring Go Engineer!

GoͱGCPͰ৽αʔϏε։ൃத https://www.wantedly.com/projects/32278