Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Alternative Fluentd implementation in Go
Search
Shinji Tanaka
December 13, 2013
Technology
3
6.2k
Alternative Fluentd implementation in Go
http://github.com/moriyoshi/ik
Shinji Tanaka
December 13, 2013
Tweet
Share
More Decks by Shinji Tanaka
See All by Shinji Tanaka
Mackerel loves Machine Learnings
stanaka
0
3.1k
How to Contribute Go
stanaka
4
18k
Introduction to Mackerel
stanaka
0
1.1k
Upcoming features in 2016, Mackerel
stanaka
0
3.5k
Mackerel's first year and onward
stanaka
0
3k
Mackerel's first year and onward (in English)
stanaka
0
410
Blue-Green Deployment with ECS and monitoring
stanaka
0
620
Mackerel Meetup #4 (in Eng)
stanaka
0
540
Mackerel Meetup #4
stanaka
0
3.3k
Other Decks in Technology
See All in Technology
フルカイテン株式会社 エンジニア向け採用資料
fullkaiten
0
9k
GopherCon Tour 概略
logica0419
2
170
履歴 on Rails: Bitemporal Data Modelで実現する履歴管理/history-on-rails-with-bitemporal-data-model
hypermkt
0
2k
多野優介
tanoyusuke
1
200
動画データのポテンシャルを引き出す! Databricks と AI活用への奮闘記(現在進行形)
databricksjapan
0
140
Optuna DashboardにおけるPLaMo2連携機能の紹介 / PFN LLM セミナー
pfn
PRO
1
840
analysis パッケージの仕組みの上でMulti linter with configを実現する / Go Conference 2025
k1low
1
260
いまさら聞けない ABテスト入門
skmr2348
1
190
Geospatialの世界最前線を探る [2025年版]
dayjournal
3
480
Goに育てられ開発者向けセキュリティ事業を立ち上げた僕が今向き合う、AI × セキュリティの最前線 / Go Conference 2025
flatt_security
0
330
データエンジニアがこの先生きのこるには...?
10xinc
0
430
FastAPIの魔法をgRPC/Connect RPCへ
monotaro
PRO
1
680
Featured
See All Featured
The Power of CSS Pseudo Elements
geoffreycrofte
79
6k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
36
2.5k
Intergalactic Javascript Robots from Outer Space
tanoku
273
27k
Facilitating Awesome Meetings
lara
56
6.6k
The Straight Up "How To Draw Better" Workshop
denniskardys
237
140k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
How STYLIGHT went responsive
nonsquared
100
5.8k
Building a Modern Day E-commerce SEO Strategy
aleyda
43
7.7k
Designing for humans not robots
tammielis
254
25k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
The Language of Interfaces
destraynor
162
25k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
12
1.1k
Transcript
A l t e r n a t i v
e F l u e n t d i m p l e m e n t a t i o n i n G o i d : s t a n a k a / @ s t a n a k a D e c e m b e r 1 2 , 2 0 1 3
id:stanaka / @stanaka • Shinji TANAKA • CTO at Hatena
Co., Ltd. • LTSV .. Labeled Tab-Separated Values • immutable infrastructure / Docker
fluentd in Hatena
Go Programming Language
Go Programming Language • 2009 released • Rob Pike, Ken
Thompson, Robert Griesemer • 2012 Mar … Go 1.0 • 2013 Jun … Go 1.1 • 2013 Dec … Go 1.2
about Go #1 • Compiler language • Static Typed with
type inference • Garbage Collection • Concurrency model (goroutine, channel)
about Go #2 • High performance • Small footprint •
Embedded support function like build, format, test, documentation • Various standard libraries (over 147)
Performance of Go • REST API server benchmark • Faster
than Perl/Ruby • blog.stanaka.org/entry/2013/10/17/093259
Major products by Go • Docker • Packer • Devops
guys like Go
Go v1.2 • released on Dec 2013 • Pre-emption in
the scheduler • Performance improvement • Library updates
Fluentd Go implementation • “Proof of Concept” level • developed
by @moriyoshit • and @stanaka • https://github.com/moriyoshi/ik
ik
Current status • Suit for leaf nodes • Support only
less plugins • in_forward, out_forward, out_stdout • only one executable file
Configuration • support v10 conf file. • v11?? <source>
type forward port 24224 </source> ! <match **> type stdout </match>
Plugin system • Need to build whole plugins • Need
register codes for plugins engine := ik.NewEngine(logger, scoreKeeper) engine.RegisterInputFactory(plugins.GetForwardInputFactory()) engine.RegisterOutputFactory(plugins.GetStdoutOutputFactory()) engine.RegisterOutputFactory(plugins.GetForwardOutputFactory())
in_forward • treat incoming connection with goroutine func (in *ForwardInput)
Start() error { in.running = true // XXX: RACE go func () { for in.running { conn, err := in.listener.Accept() if err != nil { in.logger.Fatal(err.Error()) continue } go newForwardClient(in, in.logger, conn, in.codec).handle() } }() return nil }
out_forward • Periodically flush with goroutine and channel func (output
*ForwardOutput) run_flush(interval int) { ticker := time.NewTicker(time.Duration(interval) * time.Second) go func() { for { select { case <-‐ ticker.C: output.flush() } } }() }
Installation • Quite easy • wget http://repository/bin/ik
footprint comparison (trivial) • RSS size just after execution •
with in_forward, out_forward plugin RSS ik (Go) 3224 fluentd (ruby) 19940 *1 *1: size of the bigger process
Performance • “ik” is UNDER development • not tuned for
performance yet • has primitive buffering (bytes/buffer)
Conclusion • “ik” alternative implementation of fluentd by Go •
small footprint / easy installation • high performance? (future works) • Code: https://github.com/moriyoshi/ik