Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Alternative Fluentd implementation in Go
Search
Shinji Tanaka
December 13, 2013
Technology
3
6k
Alternative Fluentd implementation in Go
http://github.com/moriyoshi/ik
Shinji Tanaka
December 13, 2013
Tweet
Share
More Decks by Shinji Tanaka
See All by Shinji Tanaka
Mackerel loves Machine Learnings
stanaka
0
3k
How to Contribute Go
stanaka
4
18k
Introduction to Mackerel
stanaka
0
1k
Upcoming features in 2016, Mackerel
stanaka
0
3.4k
Mackerel's first year and onward
stanaka
0
2.9k
Mackerel's first year and onward (in English)
stanaka
0
380
Blue-Green Deployment with ECS and monitoring
stanaka
0
590
Mackerel Meetup #4 (in Eng)
stanaka
0
490
Mackerel Meetup #4
stanaka
0
3.1k
Other Decks in Technology
See All in Technology
.NET 最新アップデート ~ AI とクラウド時代のアプリモダナイゼーション
chack411
0
200
いま現場PMのあなたが、 経営と向き合うPMになるために 必要なこと、腹をくくること
hiro93n
9
8k
[IBM TechXchange Dojo]Watson Discoveryとwatsonx.aiでRAGを実現!事例のご紹介+座学②
siyuanzh09
0
120
When Windows Meets Kubernetes…
pichuang
0
310
Bring Your Own Container: When Containers Turn the Key to EDR Bypass/byoc-avtokyo2024
tkmru
0
870
Kotlin Multiplatformのポテンシャル
recruitengineers
PRO
2
160
Reactフレームワークプロダクトを モバイルアプリにして、もっと便利に。 ユーザに価値を届けよう。/React Framework with Capacitor
rdlabo
0
140
0→1事業こそPMは営業すべし / pmconf #落選お披露目 / PM should do sales in zero to one
roki_n_
PRO
1
1.8k
DMMブックスへのTipKit導入
ttyi2
1
120
PaaSの歴史と、 アプリケーションプラットフォームのこれから
jacopen
7
1.5k
色々なAWSサービス名の由来を調べてみた
iriikeita
0
110
Git scrapingで始める継続的なデータ追跡 / Git Scraping
ohbarye
5
510
Featured
See All Featured
Site-Speed That Sticks
csswizardry
3
280
Intergalactic Javascript Robots from Outer Space
tanoku
270
27k
Fashionably flexible responsive web design (full day workshop)
malarkey
406
66k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.6k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
7k
VelocityConf: Rendering Performance Case Studies
addyosmani
327
24k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
26
1.9k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
27
1.5k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
52k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
10
870
Visualization
eitanlees
146
15k
Agile that works and the tools we love
rasmusluckow
328
21k
Transcript
A l t e r n a t i v
e F l u e n t d i m p l e m e n t a t i o n i n G o i d : s t a n a k a / @ s t a n a k a D e c e m b e r 1 2 , 2 0 1 3
id:stanaka / @stanaka • Shinji TANAKA • CTO at Hatena
Co., Ltd. • LTSV .. Labeled Tab-Separated Values • immutable infrastructure / Docker
fluentd in Hatena
Go Programming Language
Go Programming Language • 2009 released • Rob Pike, Ken
Thompson, Robert Griesemer • 2012 Mar … Go 1.0 • 2013 Jun … Go 1.1 • 2013 Dec … Go 1.2
about Go #1 • Compiler language • Static Typed with
type inference • Garbage Collection • Concurrency model (goroutine, channel)
about Go #2 • High performance • Small footprint •
Embedded support function like build, format, test, documentation • Various standard libraries (over 147)
Performance of Go • REST API server benchmark • Faster
than Perl/Ruby • blog.stanaka.org/entry/2013/10/17/093259
Major products by Go • Docker • Packer • Devops
guys like Go
Go v1.2 • released on Dec 2013 • Pre-emption in
the scheduler • Performance improvement • Library updates
Fluentd Go implementation • “Proof of Concept” level • developed
by @moriyoshit • and @stanaka • https://github.com/moriyoshi/ik
ik
Current status • Suit for leaf nodes • Support only
less plugins • in_forward, out_forward, out_stdout • only one executable file
Configuration • support v10 conf file. • v11?? <source>
type forward port 24224 </source> ! <match **> type stdout </match>
Plugin system • Need to build whole plugins • Need
register codes for plugins engine := ik.NewEngine(logger, scoreKeeper) engine.RegisterInputFactory(plugins.GetForwardInputFactory()) engine.RegisterOutputFactory(plugins.GetStdoutOutputFactory()) engine.RegisterOutputFactory(plugins.GetForwardOutputFactory())
in_forward • treat incoming connection with goroutine func (in *ForwardInput)
Start() error { in.running = true // XXX: RACE go func () { for in.running { conn, err := in.listener.Accept() if err != nil { in.logger.Fatal(err.Error()) continue } go newForwardClient(in, in.logger, conn, in.codec).handle() } }() return nil }
out_forward • Periodically flush with goroutine and channel func (output
*ForwardOutput) run_flush(interval int) { ticker := time.NewTicker(time.Duration(interval) * time.Second) go func() { for { select { case <-‐ ticker.C: output.flush() } } }() }
Installation • Quite easy • wget http://repository/bin/ik
footprint comparison (trivial) • RSS size just after execution •
with in_forward, out_forward plugin RSS ik (Go) 3224 fluentd (ruby) 19940 *1 *1: size of the bigger process
Performance • “ik” is UNDER development • not tuned for
performance yet • has primitive buffering (bytes/buffer)
Conclusion • “ik” alternative implementation of fluentd by Go •
small footprint / easy installation • high performance? (future works) • Code: https://github.com/moriyoshi/ik