Slide 1
Slide 1 text
A l t e r n a t i v e F l u e n t d i m p l e m e n t a t i o n
i n G o
i d : s t a n a k a / @ s t a n a k a
D e c e m b e r 1 2 , 2 0 1 3
Slide 2
Slide 2 text
id:stanaka / @stanaka
• Shinji TANAKA
• CTO at Hatena Co., Ltd.
• LTSV .. Labeled Tab-Separated Values
• immutable infrastructure / Docker
Slide 3
Slide 3 text
fluentd in Hatena
Slide 4
Slide 4 text
Go Programming Language
Slide 5
Slide 5 text
Go Programming Language
• 2009 released
• Rob Pike, Ken Thompson, Robert Griesemer
• 2012 Mar … Go 1.0
• 2013 Jun … Go 1.1
• 2013 Dec … Go 1.2
Slide 6
Slide 6 text
about Go #1
• Compiler language
• Static Typed with type inference
• Garbage Collection
• Concurrency model (goroutine, channel)
Slide 7
Slide 7 text
about Go #2
• High performance
• Small footprint
• Embedded support function like build, format, test,
documentation
• Various standard libraries (over 147)
Slide 8
Slide 8 text
Performance of Go
• REST API server benchmark
• Faster than Perl/Ruby
• blog.stanaka.org/entry/2013/10/17/093259
Slide 9
Slide 9 text
Major products by Go
• Docker
• Packer
• Devops guys like Go
Slide 10
Slide 10 text
Go v1.2
• released on Dec 2013
• Pre-emption in the scheduler
• Performance improvement
• Library updates
Slide 11
Slide 11 text
Fluentd Go implementation
• “Proof of Concept” level
• developed by @moriyoshit
• and @stanaka
• https://github.com/moriyoshi/ik
Slide 12
Slide 12 text
ik
Slide 13
Slide 13 text
Current status
• Suit for leaf nodes
• Support only less plugins
• in_forward, out_forward, out_stdout
• only one executable file
Slide 14
Slide 14 text
Configuration
• support v10 conf file.
• v11??
type forward
port 24224
!
type stdout
Slide 15
Slide 15 text
Plugin system
• Need to build whole plugins
• Need register codes for plugins
engine := ik.NewEngine(logger, scoreKeeper)
engine.RegisterInputFactory(plugins.GetForwardInputFactory())
engine.RegisterOutputFactory(plugins.GetStdoutOutputFactory())
engine.RegisterOutputFactory(plugins.GetForwardOutputFactory())
Slide 16
Slide 16 text
in_forward
• treat incoming connection with goroutine
func (in *ForwardInput) Start() error {
in.running = true // XXX: RACE
go func () {
for in.running {
conn, err := in.listener.Accept()
if err != nil {
in.logger.Fatal(err.Error())
continue
}
go newForwardClient(in, in.logger, conn, in.codec).handle()
}
}()
return nil
}
Slide 17
Slide 17 text
out_forward
• Periodically flush with goroutine and channel
func (output *ForwardOutput) run_flush(interval int) {
ticker := time.NewTicker(time.Duration(interval) * time.Second)
go func() {
for {
select {
case <-‐ ticker.C:
output.flush()
}
}
}()
}
Slide 18
Slide 18 text
Installation
• Quite easy
• wget http://repository/bin/ik
Slide 19
Slide 19 text
footprint comparison (trivial)
• RSS size just after execution
• with in_forward, out_forward plugin
RSS
ik (Go) 3224
fluentd (ruby) 19940 *1
*1: size of the bigger process
Slide 20
Slide 20 text
Performance
• “ik” is UNDER development
• not tuned for performance yet
• has primitive buffering (bytes/buffer)
Slide 21
Slide 21 text
Conclusion
• “ik” alternative implementation of fluentd by Go
• small footprint / easy installation
• high performance? (future works)
• Code: https://github.com/moriyoshi/ik