Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduce log collector in Go into production

Tatsuhiko Kubo
December 06, 2015

Introduce log collector in Go into production

Introduce log collector in Go into production

Tatsuhiko Kubo

December 06, 2015
Tweet

More Decks by Tatsuhiko Kubo

Other Decks in Technology

Transcript

  1. Introduce log collector in Go
    into production
    Tatsuhiko Kubo@cubicdaiya
    Go Conference 2015 Winter

    View full-size slide

  2. @cubicdaiya / Tatsuhiko Kubo
    Site Reliability Engineer @ Mercari, Inc.
    C
    Go
    Lua
    nginx
    ngx_small_light, ngx_dynamic_upstream,
    nginx-build, slackboard, cachectl, gaurun, etc…

    View full-size slide

  3. https://www.mercari.com/
    Mercari - Your Friendly Mobile Marketplace

    View full-size slide

  4. mercari.go
    • cubicdaiya/nginx-build
    • cubicdaiya/cachectl
    • cubicdaiya/slackboard
    • mercari/gaurun
    • etc… (in-house cli and daemon)

    View full-size slide

  5. Agenda
    • Casual introduction of log-collectors in Go
    • fluent-forwarder, fluent-agent-hydra, Heka
    • In the case of Mercari
    • Fluentd and fluent-agent-hydra
    • Road to introduce fluent-agent-hydra into
    production environment

    View full-size slide

  6. Casual introduction of
    log-collectors in Go

    View full-size slide

  7. fluent-forwarder
    • https://github.com/fluent/fluentd-forwarder
    • Light-weight Fluentd forworder
    • Not provide plugin system
    • Supports Windows

    View full-size slide

  8. fluent-agent-hydra
    • https://github.com/fujiwara/fluent-agent-hydra
    • Features
    • in_tail & (in|out)_forward
    • Handle multiple file per 1 process
    • Monitoring API
    • Configuration with TOML
    • Support LTSV and JSON

    View full-size slide

  9. Heka
    • https://github.com/mozilla-services/heka
    • Event processing / windowed stream operations
    • Provides Plugin system
    • User can write plugin in Go or Lua
    • Custom Input, Output, Encoder, Decoder, Filter
    • Configuration with TOML

    View full-size slide

  10. In the case of Mercari

    View full-size slide

  11. Transfer various logs with Fluentd
    "QQ
    8PSLFS
    #BUDI
    ,JCBOB
    #JH2VFSZ
    /PSJLSB
    .BDLFSFM

    View full-size slide

  12. Fluentd
    • Fluentd is very flexible and robust log-
    collector
    • Plugin systems
    • Active developers and communities
    • But performance is not always enough

    View full-size slide

  13. 0QFO3FTUZ
    0QFO3FTUZ
    0QFO3FTUZ
    (PPHMF#JH2VFSZ
    Developer Data Sientist
    Analyze
    by SQL
    send events
    send events
    send events
    Powered by
    cookpad/puree-(ios|android)
    utilize events
    utilize events
    utilize events
    PascalʙMercari analysis baseʙ (Before)
    in_tail & out_forward

    View full-size slide

  14. 0QFO3FTUZ
    0QFO3FTUZ
    0QFO3FTUZ
    (PPHMF#JH2VFSZ
    Developer Data Sientist
    Analyze
    by SQL
    send events
    send events
    send events
    Powered by
    cookpad/puree-(ios|android)
    utilize events
    utilize events
    utilize events
    hydra(※)
    hydra(※)
    hydra(※)
    (※) fluent-agent-hydra
    PascalʙMercari analysis baseʙ (Now)
    in_tail & out_forward

    View full-size slide

  15. PascalʙMercari analysis baseʙ
    • Built with the software blocks below
    • Puree, OpenResty (ngx_lua), Fluentd,
    fluent-agent-hydra, Google BigQuery
    • Aggregate various logs to Google BigQuery
    • Event in app
    • A/B Testing
    • etc…

    View full-size slide

  16. Why switched to fluent-agent-hydraʁ
    • Low server resource is required for Pascal
    • Fluentd had consumed non-negligible amount of
    CPU resource (50ʙ60% at peak)
    • Pascal indicates modestly high workload
    • OpenResty processes a lot of JSONs and
    outputs various logs
    • Requests are come from not only device but
    API-servers also

    View full-size slide

  17. Why switched to fluent-agent-hydraʁ
    • fluent-agent-hydra
    • CPU usage is less than Fluentd
    • Half as compared to Fluentd in our caseʂ
    • Enable handling multiple logs per 1 process
    • Simple

    View full-size slide

  18. fluent-agent-hydra internal
    ɿgoroutine
    monitor
    out_forwarder
    in_forwarder
    watcher & in_tail for file
    go func()
    go func()
    go func()
    go func()
    run main()
    wait signal
    Some gorouines make more goroutines

    View full-size slide

  19. goroutines communicates with channel
    ɿgoroutine
    monitor
    out_forwarder
    in_forwarder
    watcher & in_tail for file
    receiver is monitor
    receiver is out_forwarder

    View full-size slide

  20. Road to introduce fluent-agent-hydra
    into production environment

    View full-size slide

  21. Monitoring fluent-agent-hydra
    • fluent-agent-hydra provides monitoring APIs
    • Application stats
    • current positions for tailing log files
    • sent amount and bytes per a log
    • other informations (e.g. error)
    • System stats
    • Powered by golang-stats-api-handler

    View full-size slide

  22. Application Stats
    DVSMT<.POJUPS)PTU><.POJUPS1PSU>cKRb`
    {
    "receiver": null,
    "servers": [
    {
    "error": "",
    "alive": true,
    "address": "host1:24224"
    },
    {
    "error": "",
    "alive": true,
    "address": “host2:24225"
    }
    ],
    "files": {
    “/var/log/nginx/access.log“: {
    "error": "",
    "position": 472132311,
    "tag": “nginx.access_log”
    },


    View full-size slide

  23. System Stats
    DVSMT<.POJUPS)PTU><.POJUPS1PSU>TZTUFNcKRb`
    {
    "gc_pause": [],
    "gc_pause_per_second": 0,
    "gc_per_second": 0,
    "gc_num": 216,
    "gc_last": 1449345754024961000,
    "gc_next": 98962453,
    "heap_objects": 590421,
    "heap_released": 0,
    "heap_inuse": 90906624,
    "heap_idle": 11984896,
    "heap_sys": 102891520,
    "cgo_call_num": 1,
    "gomaxprocs": 8,
    "goroutine_num": 31,
    "cpu_num": 8,
    "go_arch": "amd64",
    "go_os": "linux",
    "go_version": "go1.5.1",
    "time": 1449345783047388700,
    "memory_alloc": 88036000,
    "memory_total_alloc": 10373554664,
    "memory_sys": 111020280,
    "memory_lookups": 1343,
    "memory_mallocs": 162497629,
    "memory_frees": 161907208,
    "memory_stack": 917504,
    "heap_alloc": 88036000
    }

    View full-size slide

  24. Alerting to Slack

    View full-size slide

  25. After switched to fluent-agent-hydra

    View full-size slide

  26. After switched to fluent-agent-hydra
    BigQuery error in load operation: Error processing job
    Field:xxx: Cloud not convert value to integer ( bad value or out of range )
    Field:yyy: Cloud not convert value to integer ( bad value or out of range )
    Field:xxx: Cloud not convert value to integer ( bad value or out of range )
    Field:xxx: Cloud not convert value to integer ( bad value or out of range )
    Field:xxx: Cloud not convert value to integer ( bad value or out of range )

    ʂʁ

    View full-size slide

  27. Bigquery’s demand
    • Google Bigquery demands fixed table schema
    and strict data format
    [
    {
    “name”:”value”,
    “type”:”INTEGER”
    },
    ]
    ■ schema.json
    {“value”:150}
    ■ valid data foramt
    ■ invalid data format
    {“value”:150.0} {“value”:”150”}

    View full-size slide

  28. Special conversion behavior for numerical value
    • fluent-agent-hydra treats a numerical value
    as float64 even if its type is integer
    • When log-format is JSON
    • Whyʁ
    • Because fluent-agent-hydra uses
    encoding/json and unmarshal JSON into
    interface values

    View full-size slide

  29. According to document,
    https://golang.org/pkg/encoding/json/

    View full-size slide

  30. According to document,
    https://golang.org/pkg/encoding/json/

    View full-size slide

  31. {“value”:150}
    {“value”:150.0}
    (In actual, msgpack)
    (PPHMF#JH2VFSZ
    0QFO3FTUZ
    qVFOUBHFOU
    IZESB
    out_forward & out_file
    in_tail & out_forward
    output to log file
    -PH
    4FSWFS
    bq load xxx.log schema.json
    Error processing job [
    {
    “name”:”value”,
    “type”:”INTEGER”
    },
    ]
    schema.json
    {“value”:150.0}

    View full-size slide

  32. By the way,
    • fluent-agent-hydra provides the directive
    Types for converting type
    # in config.toml
    Types = “value:integer”

    View full-size slide

  33. By the way,
    • fluent-agent-hydra provides the directive
    Types for converting type
    # in config.toml
    Types = “value:integer”
    But this was provided for only LTSV at that time…

    View full-size slide

  34. https://github.com/fujiwara/fluent-agent-hydra/pull/5
    Therefore,

    View full-size slide

  35. https://github.com/fujiwara/fluent-agent-hydra/pull/6
    Moreover,

    View full-size slide

  36. Now
    • fluent-agent-hydra provides the directive
    Types for converting type
    # in config.toml
    Types = “value:integer”
    Always convert type to int64
    regardless format is LTSV or JSON

    View full-size slide

  37. https://github.com/fujiwara/fluent-agent-hydra/blob/master/
    README.md#about-special-conversion-behavior-for-numerical-value
    Now

    View full-size slide

  38. 0QFO3FTUZ
    0QFO3FTUZ
    0QFO3FTUZ
    (PPHMF#JH2VFSZ
    Developer Data Sientist
    Analyze
    by SQL
    send events
    send events
    send events
    Powered by
    cookpad/puree-(ios|android)
    utilize events
    utilize events
    utilize events
    hydra(※)
    hydra(※)
    hydra(※)
    (※) fluent-agent-hydra
    PascalʙMercari analysis baseʙ (Now)
    in_tail & out_forward

    View full-size slide

  39. Summary
    • Fluentd is very flexible and robust log-collector
    • But performance is not always enough
    • There are some alternatives in Go
    • fluent-agent-hydra might fit the case below
    • Want a faster and light-weight log-collector for
    in_tail & out_forward
    • But robustness is less than Fluentd
    • e.g. position file is not supported

    View full-size slide

  40. We are hiring
    Go Engineer!

    View full-size slide

  41. GoͱGCPͰ৽αʔϏε։ൃத
    https://www.wantedly.com/projects/32278

    View full-size slide