Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Alternative Fluentd implementation in Go
Search
Shinji Tanaka
December 13, 2013
Technology
3
6.1k
Alternative Fluentd implementation in Go
http://github.com/moriyoshi/ik
Shinji Tanaka
December 13, 2013
Tweet
Share
More Decks by Shinji Tanaka
See All by Shinji Tanaka
Mackerel loves Machine Learnings
stanaka
0
3k
How to Contribute Go
stanaka
4
18k
Introduction to Mackerel
stanaka
0
1k
Upcoming features in 2016, Mackerel
stanaka
0
3.4k
Mackerel's first year and onward
stanaka
0
2.9k
Mackerel's first year and onward (in English)
stanaka
0
390
Blue-Green Deployment with ECS and monitoring
stanaka
0
600
Mackerel Meetup #4 (in Eng)
stanaka
0
500
Mackerel Meetup #4
stanaka
0
3.2k
Other Decks in Technology
See All in Technology
DevinでAI AWSエンジニア製造計画 序章 〜CDKを添えて〜/devin-load-to-aws-engineer
tomoki10
0
190
Global Databaseで実現するマルチリージョン自動切替とBlue/Greenデプロイ
j2yano
0
150
OPENLOGI Company Profile for engineer
hr01
1
20k
EDRの検知の仕組みと検知回避について
chayakonanaika
12
5.2k
Ruby on Railsで持続可能な開発を行うために取り組んでいること
am1157154
3
160
AWSアカウントのセキュリティ自動化、どこまで進める? 最適な設計と実践ポイント
yuobayashi
7
990
DeepSeekとは?何がいいの? - Databricksと学ぶDeepSeek! 〜これからのLLMに備えよ!〜
taka_aki
1
160
AIエージェント入門
minorun365
PRO
32
19k
大規模アジャイルフレームワークから学ぶエンジニアマネジメントの本質
staka121
PRO
3
1.4k
Amazon Aurora のバージョンアップ手法について
smt7174
2
180
Autonomous Database Serverless 技術詳細 / adb-s_technical_detail_jp
oracle4engineer
PRO
17
45k
1行のコードから社会課題の解決へ: EMの探究、事業・技術・組織を紡ぐ実践知 / EM Conf 2025
9ma3r
12
4.4k
Featured
See All Featured
Six Lessons from altMBA
skipperchong
27
3.6k
The World Runs on Bad Software
bkeepers
PRO
67
11k
Docker and Python
trallard
44
3.3k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
27
1.9k
GitHub's CSS Performance
jonrohan
1030
460k
Agile that works and the tools we love
rasmusluckow
328
21k
The Language of Interfaces
destraynor
156
24k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
356
29k
The Pragmatic Product Professional
lauravandoore
32
6.4k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9.3k
BBQ
matthewcrist
87
9.5k
Git: the NoSQL Database
bkeepers
PRO
428
65k
Transcript
A l t e r n a t i v
e F l u e n t d i m p l e m e n t a t i o n i n G o i d : s t a n a k a / @ s t a n a k a D e c e m b e r 1 2 , 2 0 1 3
id:stanaka / @stanaka • Shinji TANAKA • CTO at Hatena
Co., Ltd. • LTSV .. Labeled Tab-Separated Values • immutable infrastructure / Docker
fluentd in Hatena
Go Programming Language
Go Programming Language • 2009 released • Rob Pike, Ken
Thompson, Robert Griesemer • 2012 Mar … Go 1.0 • 2013 Jun … Go 1.1 • 2013 Dec … Go 1.2
about Go #1 • Compiler language • Static Typed with
type inference • Garbage Collection • Concurrency model (goroutine, channel)
about Go #2 • High performance • Small footprint •
Embedded support function like build, format, test, documentation • Various standard libraries (over 147)
Performance of Go • REST API server benchmark • Faster
than Perl/Ruby • blog.stanaka.org/entry/2013/10/17/093259
Major products by Go • Docker • Packer • Devops
guys like Go
Go v1.2 • released on Dec 2013 • Pre-emption in
the scheduler • Performance improvement • Library updates
Fluentd Go implementation • “Proof of Concept” level • developed
by @moriyoshit • and @stanaka • https://github.com/moriyoshi/ik
ik
Current status • Suit for leaf nodes • Support only
less plugins • in_forward, out_forward, out_stdout • only one executable file
Configuration • support v10 conf file. • v11?? <source>
type forward port 24224 </source> ! <match **> type stdout </match>
Plugin system • Need to build whole plugins • Need
register codes for plugins engine := ik.NewEngine(logger, scoreKeeper) engine.RegisterInputFactory(plugins.GetForwardInputFactory()) engine.RegisterOutputFactory(plugins.GetStdoutOutputFactory()) engine.RegisterOutputFactory(plugins.GetForwardOutputFactory())
in_forward • treat incoming connection with goroutine func (in *ForwardInput)
Start() error { in.running = true // XXX: RACE go func () { for in.running { conn, err := in.listener.Accept() if err != nil { in.logger.Fatal(err.Error()) continue } go newForwardClient(in, in.logger, conn, in.codec).handle() } }() return nil }
out_forward • Periodically flush with goroutine and channel func (output
*ForwardOutput) run_flush(interval int) { ticker := time.NewTicker(time.Duration(interval) * time.Second) go func() { for { select { case <-‐ ticker.C: output.flush() } } }() }
Installation • Quite easy • wget http://repository/bin/ik
footprint comparison (trivial) • RSS size just after execution •
with in_forward, out_forward plugin RSS ik (Go) 3224 fluentd (ruby) 19940 *1 *1: size of the bigger process
Performance • “ik” is UNDER development • not tuned for
performance yet • has primitive buffering (bytes/buffer)
Conclusion • “ik” alternative implementation of fluentd by Go •
small footprint / easy installation • high performance? (future works) • Code: https://github.com/moriyoshi/ik