$30 off During Our Annual Pro Sale. View Details »

Alternative Fluentd implementation in Go

Shinji Tanaka
December 13, 2013

Alternative Fluentd implementation in Go

Shinji Tanaka

December 13, 2013
Tweet

More Decks by Shinji Tanaka

Other Decks in Technology

Transcript

  1. A l t e r n a t i v

    e F l u e n t d i m p l e m e n t a t i o n i n G o i d : s t a n a k a / @ s t a n a k a D e c e m b e r 1 2 , 2 0 1 3
  2. id:stanaka / @stanaka • Shinji TANAKA • CTO at Hatena

    Co., Ltd.
 • LTSV .. Labeled Tab-Separated Values • immutable infrastructure / Docker
  3. fluentd in Hatena

  4. Go Programming Language

  5. Go Programming Language • 2009 released • Rob Pike, Ken

    Thompson, Robert Griesemer • 2012 Mar … Go 1.0 • 2013 Jun … Go 1.1 • 2013 Dec … Go 1.2
  6. about Go #1 • Compiler language • Static Typed with

    type inference • Garbage Collection • Concurrency model (goroutine, channel)
  7. about Go #2 • High performance • Small footprint •

    Embedded support function like build, format, test, documentation • Various standard libraries (over 147)
  8. Performance of Go • REST API server benchmark • Faster

    than Perl/Ruby • blog.stanaka.org/entry/2013/10/17/093259
  9. Major products by Go • Docker • Packer • Devops

    guys like Go
  10. Go v1.2 • released on Dec 2013 • Pre-emption in

    the scheduler • Performance improvement • Library updates
  11. Fluentd Go implementation • “Proof of Concept” level • developed

    by @moriyoshit • and @stanaka • https://github.com/moriyoshi/ik
  12. ik

  13. Current status • Suit for leaf nodes • Support only

    less plugins • in_forward, out_forward, out_stdout • only one executable file
  14. Configuration • support v10 conf file. • v11?? <source>  

       type  forward      port  24224   </source>   ! <match  **>      type  stdout   </match>  
  15. Plugin system • Need to build whole plugins • Need

    register codes for plugins engine  :=  ik.NewEngine(logger,  scoreKeeper)   engine.RegisterInputFactory(plugins.GetForwardInputFactory())   engine.RegisterOutputFactory(plugins.GetStdoutOutputFactory())   engine.RegisterOutputFactory(plugins.GetForwardOutputFactory())
  16. in_forward • treat incoming connection with goroutine func  (in  *ForwardInput)

     Start()  error  {      in.running  =  true  //  XXX:  RACE      go  func  ()  {          for  in.running  {              conn,  err  :=  in.listener.Accept()              if  err  !=  nil  {                  in.logger.Fatal(err.Error())                  continue              }              go  newForwardClient(in,  in.logger,  conn,  in.codec).handle()          }      }()      return  nil   }
  17. out_forward • Periodically flush with goroutine and channel func  (output

     *ForwardOutput)  run_flush(interval  int)  {     ticker  :=  time.NewTicker(time.Duration(interval)  *  time.Second)     go  func()  {       for  {         select  {         case  <-­‐  ticker.C:           output.flush()         }       }     }()   }  
  18. Installation • Quite easy • wget http://repository/bin/ik

  19. footprint comparison (trivial) • RSS size just after execution •

    with in_forward, out_forward plugin RSS ik (Go) 3224 fluentd (ruby) 19940 *1 *1: size of the bigger process
  20. Performance • “ik” is UNDER development • not tuned for

    performance yet • has primitive buffering (bytes/buffer)
  21. Conclusion • “ik” alternative implementation of fluentd by Go •

    small footprint / easy installation • high performance? (future works) • Code: https://github.com/moriyoshi/ik