Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduce log collector in Go into production

Tatsuhiko Kubo
December 06, 2015

Introduce log collector in Go into production

Introduce log collector in Go into production

Tatsuhiko Kubo

December 06, 2015
Tweet

More Decks by Tatsuhiko Kubo

Other Decks in Technology

Transcript

  1. Introduce log collector in Go
    into production
    Tatsuhiko [email protected]
    Go Conference 2015 Winter

    View Slide

  2. @cubicdaiya / Tatsuhiko Kubo
    Site Reliability Engineer @ Mercari, Inc.
    C
    Go
    Lua
    nginx
    ngx_small_light, ngx_dynamic_upstream,
    nginx-build, slackboard, cachectl, gaurun, etc…

    View Slide

  3. View Slide

  4. https://www.mercari.com/
    Mercari - Your Friendly Mobile Marketplace

    View Slide

  5. mercari.go
    • cubicdaiya/nginx-build
    • cubicdaiya/cachectl
    • cubicdaiya/slackboard
    • mercari/gaurun
    • etc… (in-house cli and daemon)

    View Slide

  6. Agenda
    • Casual introduction of log-collectors in Go
    • fluent-forwarder, fluent-agent-hydra, Heka
    • In the case of Mercari
    • Fluentd and fluent-agent-hydra
    • Road to introduce fluent-agent-hydra into
    production environment

    View Slide

  7. Casual introduction of
    log-collectors in Go

    View Slide

  8. fluent-forwarder
    • https://github.com/fluent/fluentd-forwarder
    • Light-weight Fluentd forworder
    • Not provide plugin system
    • Supports Windows

    View Slide

  9. fluent-agent-hydra
    • https://github.com/fujiwara/fluent-agent-hydra
    • Features
    • in_tail & (in|out)_forward
    • Handle multiple file per 1 process
    • Monitoring API
    • Configuration with TOML
    • Support LTSV and JSON

    View Slide

  10. Heka
    • https://github.com/mozilla-services/heka
    • Event processing / windowed stream operations
    • Provides Plugin system
    • User can write plugin in Go or Lua
    • Custom Input, Output, Encoder, Decoder, Filter
    • Configuration with TOML

    View Slide

  11. In the case of Mercari

    View Slide

  12. Transfer various logs with Fluentd
    "QQ
    8PSLFS
    #BUDI
    ,JCBOB
    #JH2VFSZ
    /PSJLSB
    .BDLFSFM

    View Slide

  13. Fluentd
    • Fluentd is very flexible and robust log-
    collector
    • Plugin systems
    • Active developers and communities
    • But performance is not always enough

    View Slide

  14. 0QFO3FTUZ
    0QFO3FTUZ
    0QFO3FTUZ
    (PPHMF#JH2VFSZ
    Developer Data Sientist
    Analyze
    by SQL
    send events
    send events
    send events
    Powered by
    cookpad/puree-(ios|android)
    utilize events
    utilize events
    utilize events
    PascalʙMercari analysis baseʙ (Before)
    in_tail & out_forward

    View Slide

  15. 0QFO3FTUZ
    0QFO3FTUZ
    0QFO3FTUZ
    (PPHMF#JH2VFSZ
    Developer Data Sientist
    Analyze
    by SQL
    send events
    send events
    send events
    Powered by
    cookpad/puree-(ios|android)
    utilize events
    utilize events
    utilize events
    hydra(※)
    hydra(※)
    hydra(※)
    (※) fluent-agent-hydra
    PascalʙMercari analysis baseʙ (Now)
    in_tail & out_forward

    View Slide

  16. PascalʙMercari analysis baseʙ
    • Built with the software blocks below
    • Puree, OpenResty (ngx_lua), Fluentd,
    fluent-agent-hydra, Google BigQuery
    • Aggregate various logs to Google BigQuery
    • Event in app
    • A/B Testing
    • etc…

    View Slide

  17. Why switched to fluent-agent-hydraʁ
    • Low server resource is required for Pascal
    • Fluentd had consumed non-negligible amount of
    CPU resource (50ʙ60% at peak)
    • Pascal indicates modestly high workload
    • OpenResty processes a lot of JSONs and
    outputs various logs
    • Requests are come from not only device but
    API-servers also

    View Slide

  18. Why switched to fluent-agent-hydraʁ
    • fluent-agent-hydra
    • CPU usage is less than Fluentd
    • Half as compared to Fluentd in our caseʂ
    • Enable handling multiple logs per 1 process
    • Simple

    View Slide

  19. fluent-agent-hydra internal
    ɿgoroutine
    monitor
    out_forwarder
    in_forwarder
    watcher & in_tail for file
    go func()
    go func()
    go func()
    go func()
    run main()
    wait signal
    Some gorouines make more goroutines

    View Slide

  20. goroutines communicates with channel
    ɿgoroutine
    monitor
    out_forwarder
    in_forwarder
    watcher & in_tail for file
    receiver is monitor
    receiver is out_forwarder

    View Slide

  21. Road to introduce fluent-agent-hydra
    into production environment

    View Slide

  22. Monitoring fluent-agent-hydra
    • fluent-agent-hydra provides monitoring APIs
    • Application stats
    • current positions for tailing log files
    • sent amount and bytes per a log
    • other informations (e.g. error)
    • System stats
    • Powered by golang-stats-api-handler

    View Slide

  23. Application Stats
    DVSMT

    View Slide

  24. System Stats
    DVSMT

    View Slide

  25. Alerting to Slack

    View Slide

  26. After switched to fluent-agent-hydra

    View Slide

  27. After switched to fluent-agent-hydra
    BigQuery error in load operation: Error processing job
    Field:xxx: Cloud not convert value to integer ( bad value or out of range )
    Field:yyy: Cloud not convert value to integer ( bad value or out of range )
    Field:xxx: Cloud not convert value to integer ( bad value or out of range )
    Field:xxx: Cloud not convert value to integer ( bad value or out of range )
    Field:xxx: Cloud not convert value to integer ( bad value or out of range )

    ʂʁ

    View Slide

  28. Bigquery’s demand
    • Google Bigquery demands fixed table schema
    and strict data format
    [
    {
    “name”:”value”,
    “type”:”INTEGER”
    },
    ]
    ■ schema.json
    {“value”:150}
    ■ valid data foramt
    ■ invalid data format
    {“value”:150.0} {“value”:”150”}

    View Slide

  29. Special conversion behavior for numerical value
    • fluent-agent-hydra treats a numerical value
    as float64 even if its type is integer
    • When log-format is JSON
    • Whyʁ
    • Because fluent-agent-hydra uses
    encoding/json and unmarshal JSON into
    interface values

    View Slide

  30. According to document,
    https://golang.org/pkg/encoding/json/

    View Slide

  31. According to document,
    https://golang.org/pkg/encoding/json/

    View Slide

  32. {“value”:150}
    {“value”:150.0}
    (In actual, msgpack)
    (PPHMF#JH2VFSZ
    0QFO3FTUZ
    qVFOUBHFOU
    IZESB
    out_forward & out_file
    in_tail & out_forward
    output to log file
    -PH
    4FSWFS
    bq load xxx.log schema.json
    Error processing job [
    {
    “name”:”value”,
    “type”:”INTEGER”
    },
    ]
    schema.json
    {“value”:150.0}

    View Slide

  33. By the way,
    • fluent-agent-hydra provides the directive
    Types for converting type
    # in config.toml
    Types = “value:integer”

    View Slide

  34. By the way,
    • fluent-agent-hydra provides the directive
    Types for converting type
    # in config.toml
    Types = “value:integer”
    But this was provided for only LTSV at that time…

    View Slide

  35. https://github.com/fujiwara/fluent-agent-hydra/pull/5
    Therefore,

    View Slide

  36. https://github.com/fujiwara/fluent-agent-hydra/pull/6
    Moreover,

    View Slide

  37. Now
    • fluent-agent-hydra provides the directive
    Types for converting type
    # in config.toml
    Types = “value:integer”
    Always convert type to int64
    regardless format is LTSV or JSON

    View Slide

  38. https://github.com/fujiwara/fluent-agent-hydra/blob/master/
    README.md#about-special-conversion-behavior-for-numerical-value
    Now

    View Slide

  39. 0QFO3FTUZ
    0QFO3FTUZ
    0QFO3FTUZ
    (PPHMF#JH2VFSZ
    Developer Data Sientist
    Analyze
    by SQL
    send events
    send events
    send events
    Powered by
    cookpad/puree-(ios|android)
    utilize events
    utilize events
    utilize events
    hydra(※)
    hydra(※)
    hydra(※)
    (※) fluent-agent-hydra
    PascalʙMercari analysis baseʙ (Now)
    in_tail & out_forward

    View Slide

  40. Summary
    • Fluentd is very flexible and robust log-collector
    • But performance is not always enough
    • There are some alternatives in Go
    • fluent-agent-hydra might fit the case below
    • Want a faster and light-weight log-collector for
    in_tail & out_forward
    • But robustness is less than Fluentd
    • e.g. position file is not supported

    View Slide

  41. We are hiring
    Go Engineer!

    View Slide

  42. GoͱGCPͰ৽αʔϏε։ൃத
    https://www.wantedly.com/projects/32278

    View Slide