Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Alternative Fluentd implementation in Go
Search
Shinji Tanaka
December 13, 2013
Technology
3
5.9k
Alternative Fluentd implementation in Go
http://github.com/moriyoshi/ik
Shinji Tanaka
December 13, 2013
Tweet
Share
More Decks by Shinji Tanaka
See All by Shinji Tanaka
Mackerel loves Machine Learnings
stanaka
0
2.9k
How to Contribute Go
stanaka
4
18k
Introduction to Mackerel
stanaka
0
1k
Upcoming features in 2016, Mackerel
stanaka
0
3.3k
Mackerel's first year and onward
stanaka
0
2.8k
Mackerel's first year and onward (in English)
stanaka
0
360
Blue-Green Deployment with ECS and monitoring
stanaka
0
570
Mackerel Meetup #4 (in Eng)
stanaka
0
470
Mackerel Meetup #4
stanaka
0
3.1k
Other Decks in Technology
See All in Technology
React Aria で実現する次世代のアクセシビリティ
ryo_manba
4
1.2k
株式会社EventHub・エンジニア採用資料
eventhub
0
2.9k
Jetpack Compose Modifier 徹底解説 / Jetpack Compose Modifier
wiroha
0
200
プロダクトエンジニアを支えるための開発生産性向上施策
tsukakei
0
140
PdMはどのように全てのスピードを上げられるか ~ 非連続進化のための具体的な取り組み ~
sansantech
PRO
4
1.3k
QAに対する超個人的な解釈 / Personal Take on QA
toma_sm
1
100
グイグイ系QAマネージャーの仕事
sadonosake
0
340
Analytics-Backed App Widget Development - Served with Jetpack Glance
miyabigouji
0
610
社内の学びの場・コミュニティ形成とエンジニア同士のリレーションシップ構築/devreljapan2024
nishiuma
3
290
Agile in Automotive Industry, puzzles and lights.
hiranabe
3
1.4k
20240912 JJUGナイトセミナー
mii1004
0
140
Oracle Autonomous Database:サービス概要のご紹介
oracle4engineer
PRO
1
7.1k
Featured
See All Featured
Agile that works and the tools we love
rasmusluckow
327
20k
Learning to Love Humans: Emotional Interface Design
aarron
270
40k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
38
9.2k
Mobile First: as difficult as doing things right
swwweet
221
8.8k
StorybookのUI Testing Handbookを読んだ
zakiyama
26
5.1k
It's Worth the Effort
3n
182
27k
Reflections from 52 weeks, 52 projects
jeffersonlam
346
20k
How GitHub (no longer) Works
holman
310
140k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
25
3.9k
A Modern Web Designer's Workflow
chriscoyier
691
190k
Debugging Ruby Performance
tmm1
72
12k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
227
52k
Transcript
A l t e r n a t i v
e F l u e n t d i m p l e m e n t a t i o n i n G o i d : s t a n a k a / @ s t a n a k a D e c e m b e r 1 2 , 2 0 1 3
id:stanaka / @stanaka • Shinji TANAKA • CTO at Hatena
Co., Ltd. • LTSV .. Labeled Tab-Separated Values • immutable infrastructure / Docker
fluentd in Hatena
Go Programming Language
Go Programming Language • 2009 released • Rob Pike, Ken
Thompson, Robert Griesemer • 2012 Mar … Go 1.0 • 2013 Jun … Go 1.1 • 2013 Dec … Go 1.2
about Go #1 • Compiler language • Static Typed with
type inference • Garbage Collection • Concurrency model (goroutine, channel)
about Go #2 • High performance • Small footprint •
Embedded support function like build, format, test, documentation • Various standard libraries (over 147)
Performance of Go • REST API server benchmark • Faster
than Perl/Ruby • blog.stanaka.org/entry/2013/10/17/093259
Major products by Go • Docker • Packer • Devops
guys like Go
Go v1.2 • released on Dec 2013 • Pre-emption in
the scheduler • Performance improvement • Library updates
Fluentd Go implementation • “Proof of Concept” level • developed
by @moriyoshit • and @stanaka • https://github.com/moriyoshi/ik
ik
Current status • Suit for leaf nodes • Support only
less plugins • in_forward, out_forward, out_stdout • only one executable file
Configuration • support v10 conf file. • v11?? <source>
type forward port 24224 </source> ! <match **> type stdout </match>
Plugin system • Need to build whole plugins • Need
register codes for plugins engine := ik.NewEngine(logger, scoreKeeper) engine.RegisterInputFactory(plugins.GetForwardInputFactory()) engine.RegisterOutputFactory(plugins.GetStdoutOutputFactory()) engine.RegisterOutputFactory(plugins.GetForwardOutputFactory())
in_forward • treat incoming connection with goroutine func (in *ForwardInput)
Start() error { in.running = true // XXX: RACE go func () { for in.running { conn, err := in.listener.Accept() if err != nil { in.logger.Fatal(err.Error()) continue } go newForwardClient(in, in.logger, conn, in.codec).handle() } }() return nil }
out_forward • Periodically flush with goroutine and channel func (output
*ForwardOutput) run_flush(interval int) { ticker := time.NewTicker(time.Duration(interval) * time.Second) go func() { for { select { case <-‐ ticker.C: output.flush() } } }() }
Installation • Quite easy • wget http://repository/bin/ik
footprint comparison (trivial) • RSS size just after execution •
with in_forward, out_forward plugin RSS ik (Go) 3224 fluentd (ruby) 19940 *1 *1: size of the bigger process
Performance • “ik” is UNDER development • not tuned for
performance yet • has primitive buffering (bytes/buffer)
Conclusion • “ik” alternative implementation of fluentd by Go •
small footprint / easy installation • high performance? (future works) • Code: https://github.com/moriyoshi/ik