Go in Production at Mackerel.io

Go in Production At Mackerel.io @stanaka Hatena Co., Ltd.

@stanaka / id:stanaka Shinji Tanaka / ాத৻࢘ Hatena Co., Ltd.
We’re Hiring at both KYOTO and TOKYO!

Thank you, Brad! Hatena is a heavy user of: memcached
MogileFS stores a few billions objects meta data: over 2TB in ONE table of MySQL. Anyway it works!

https://mackerel.io/

Role-based merged graph Loadavg comparison Stacking CPU usage

Overview of Mackerel.io Server Cloud A Cloud B mackerel.io User
Send resource usage data View control panels Alert notiﬁcations Agent Server User environments Agent Server Agent Server Agent Server

Overview of Mackerel.io Server Cloud A Cloud B mackerel.io User
Send resource usage data View control panels Alert notiﬁcations User environments Agent Server Agent Server Agent Server Agent Server

Written in Go Open Source Software (Apache2 License) ! Read
information and resource usage of OS/ Hardware/System Post data to REST API of Mackerel.io https://github.com/mackerelio/mackerel-agent

Why Go Need to support various environments ! Easy setup
! Works quietly as possible VmHWM: 6468 kB VmRSS: 5768 kB Cross-compiler Few dependency ! One binary ! Small footprint

Code Highlights in Detail

Periodical Invocation Invoke every 1 minute go func() {
c := time.Tick(1 * time.Second) last := time.Now() ticker <-‐ last // sends tick once at first ! for t := range c { if t.Second() == 0 || t.After(last.Add(1*time.Minute)) { last = t ticker <-‐ t } } }()

Collect metrics Invoke collector go func() { //
Start collectMetrics concurrently // so that it does not prevent runnnig next collectMetrics. sem := make(chan uint, COLLECT_METRICS_WORKER_MAX) for tickedTime := range ticker { sem <-‐ 1 go func() { metricsResult <-‐ agent.collectMetrics(tickedTime) <-‐sem }() } }()

Collect metrics in detail Agent.collectMetics > Agent.generateValues > Generate (cpuusage.go)
func (g *CpuusageGenerator) collectProcStatValues() ([]float64, float64, uint, error) { file, err := os.Open("/proc/stat") func (g *CpuusageGenerator) Generate() (metrics.Values, error) { prevValues, prevTotal, _, err := g.collectProcStatValues() $ cat /proc/stat cpu 7792253 5479 4851396 18056319678 127239 0 146818 2383839 cpu0 5385397 1412 1970781 4509432750 103260 0 136689 876389

Collect metrics (plugin) plugin: outputs to stdout ! exec external
commands func (g *PluginGenerator) collectValues(command string) (metrics.Values, error) { ... var outBuffer bytes.Buffer var errBuffer bytes.Buffer ! cmd := exec.Command("/bin/sh", "-‐c", command) cmd.Stdout = &outBuffer cmd.Stderr = &errBuffer ! err := cmd.Run() {metric name}\t{metric value}\t{epoch seconds}

Parallel Collector go func() { var wg sync.WaitGroup
for _, g := range generators { wg.Add(1) go func(g metrics.Generator) { defer wg.Done() ! values, err := g.Generate() ... processed <-‐ values }(g) } wg.Wait() finish <-‐ true // processed all jobs }()

Wait for all collectors go func() { allValues
:= metrics.Values(make(map[string]float64)) for { select { case values := <-‐processed: allValues.Merge(values) case <-‐finish: result <-‐ allValues return } } }()

Post metrics func loop(ag *agent.Agent, api *mackerel.API, host …) {
metricsResult := ag.Watch() postQueue := make(chan []*mackerel.CreatingMetricsValue, ..) go func() { for values := range postQueue { err := api.PostMetricsValues(values) ... // retry process time.Sleep(METRICS_POST_DEQUEUE_DELAY) } }() ... for { select { case result := <-‐metricsResult: ... // Assemble results postQueue <-‐ creatingValues } } }

Multi architectures Build constrains command/ command.go
command_linux.go # filename-‐based command_windows.go metrics/ linux/ cpu.go # // +build linux (directive-‐based) memory.go ... windows/ cpu.go # // +build window memory.go ...

Multi architectures command/command_linux.go func metricsGenerators(conf config.Config) []metrics.Generator {
generators := []metrics.Generator{ &metricsLinux.Loadavg5Generator{}, &metricsLinux.CpuusageGenerator{Interval: 60}, ... } for _, pluginConfig := range conf.Plugin["metrics"] { generators = append(generators, &metricsLinux.PluginGenerator{pluginConfig}) } ! return generators }

Build (Compile) use Makeﬁle ! ! ! ! embed GITCOMMIT
and VERSION build: deps go build -‐ldflags="\ -‐X github.com/mackerelio/mackerel-‐agent/version.GITCOMMIT `git rev-‐parse -‐-‐short HEAD` \ -‐X github.com/mackerelio/mackerel-‐agent/version.VERSION `git describe -‐-‐tags -‐-‐abbrev=0 | sed 's/^v//' | sed 's/\+.*$ $//'` " \ -‐o build/$(BIN) \ github.com/mackerelio/mackerel-‐agent req.Header.Add("X-‐Revision", version.GITCOMMIT)

Build (Packaging) use Docker to build in clean environment FROM
centos RUN yum update -‐y RUN yum install -‐y rpmdevtools ! RUN mkdir -‐p /rpmbuild ADD ./ /rpmbuild/ RUN chown root:root -‐R /rpmbuild CMD rpmbuild -‐ba /rpmbuild/SPECS/mackerel-‐agent.spec

Daemonize nohup & `&` ! ! follow the recommended method
of Debian  (not implemented yet…) nohup mackerel-‐agent -‐-‐apikey=$APIKEY -‐-‐pidfile=$PIDFILE \  $OTHER_OPTS >>$LOGFILE 2>&1 & start-‐stop-‐daemon -‐-‐start -‐-‐quiet -‐-‐pidfile $PIDFILE \ -‐-‐background -‐-‐exec mackerel-‐agent

Detect Memory leaks import _ "net/http/pprof" go func() {
log.Println(http.ListenAndServe("localhost:6060", nil)) }()

Pros / Cons Pros Channel and goroutine is GREAT Easy
to support various architectures linux, windows, 386, x86_64, ARM… Cons String processing (parsing /proc/…) is annoying..

Go in Production at Mackerel.io

Go in Production at Mackerel.io

Shinji Tanaka

More Decks by Shinji Tanaka

Other Decks in Technology

Featured

Transcript