Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Casual Log Collection and Querying with fluent-...
Search
UENISHI Kota
June 01, 2013
Technology
3
460
Casual Log Collection and Querying with fluent-plugin-riak
My talk at RubyKaigi 2013
http://rubykaigi.org/2013/talk/S70
UENISHI Kota
June 01, 2013
Tweet
Share
More Decks by UENISHI Kota
See All by UENISHI Kota
Storage Systems in Preferred Networks
kuenishi
0
36
Metadata Management in Distributed File Systems
kuenishi
2
510
Behind The Scenes: Cloud Native Storage System for AI
kuenishi
2
400
Apache Ozone behind Simulation and AI Industries
kuenishi
0
380
Distributed Deep Learning with Chainer and Hadoop
kuenishi
3
1.2k
A Few Ways to Accelerate Deep Learning
kuenishi
0
1.1k
Introducing Retz
kuenishi
5
1.2k
Introducing Retz and how to develop practical frameworks
kuenishi
3
740
Formalization and Proof of Distributed Systems (ja)
kuenishi
10
6.4k
Other Decks in Technology
See All in Technology
Goss: Faiss向けの新しい本番環境対応 Goバインディング #coefl_go_jp
bengo4com
0
1.4k
AIエージェント就活入門 - MCPが履歴書になる未来
eltociear
0
280
モバイルアプリ研修
recruitengineers
PRO
2
110
Understanding Go GC #coefl_go_jp
bengo4com
0
1.1k
メルカリIBIS:AIが拓く次世代インシデント対応
0gm
2
520
kintone開発チームの紹介
cybozuinsideout
PRO
0
73k
モダンな現場と従来型の組織——そこに生じる "不整合" を解消してこそチームがパフォーマンスを発揮できる / Team-oriented Organization Design 20250825
mtx2s
5
500
Devinを使ったモバイルアプリ開発 / Mobile app development with Devin
yanzm
0
170
20250818_KGX・One Hokkaidoコラボイベント
tohgeyukihiro
0
130
小さなチーム 大きな仕事 - 個人開発でAIをフル活用する
himaratsu
0
100
広島銀行におけるAWS活用の取り組みについて
masakimori
0
120
2025新卒研修・Webアプリケーションセキュリティ #弁護士ドットコム
bengo4com
3
10k
Featured
See All Featured
Faster Mobile Websites
deanohume
309
31k
Music & Morning Musume
bryan
46
6.7k
A Tale of Four Properties
chriscoyier
160
23k
Making the Leap to Tech Lead
cromwellryan
134
9.5k
Automating Front-end Workflow
addyosmani
1370
200k
Designing for Performance
lara
610
69k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.8k
Build The Right Thing And Hit Your Dates
maggiecrowley
37
2.8k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
KATA
mclloyd
32
14k
GraphQLとの向き合い方2022年版
quramy
49
14k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Transcript
Casual Log Collection and Querying with fluent-plugin-riak @kuenishi from @basho
2013/6/1 RubyKaigi
Who the hell are you? •UENISHI, Kota (@kuenishi) •Basho Japan
KK •devoted to Distributed Systems for ~6 yrs •msgpack-erlang, Jubatus
Casual Log Collection •Aggregate Every Log with Fluentd •Put Them
all into <Some Storage You Like> •Ask your Query to <Some Storage You Like>
Whole Sketch
fluentd: casual log collector http://www.flickr.com/photos/markchadwick/8757802771/ http://www.flickr.com/photos/usdagov/5681152426/ before: logs are scattered
all over the servers in chaos after: all logs flows cleanly via fluentd in order
Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache
Frontend Access logs syslogd App logs System logs Backend Databases
Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache
Frontend Access logs syslogd App logs System logs Backend Databases filter / buffer / routing
Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache
Frontend Access logs syslogd App logs System logs Backend Databases filter / buffer / routing Riak
what’s ? •Distributed Key-Value Store •Focused on •Availability •Scalability •Easy
Operation, ҆ (Sleep)
when Riak? •Hadoop is too much •MongoDB is too small
•Document DB aspect of Riak •put them all into Riak
Not Only KVS •Aspect of Document Database •MapReduce in JavaScript
/ Erlang
Buy it if interested
fluent-plugin-riak JSON
fluent.conf <match apache.**> type riak # define the cluster via
pb ports nodes 192.168.0.1:8087 192.168.0.2:8087 </match>
log everything as JSON { "host":"103.5.142.5", "user":"-", "method":"PUT", "path":"/buckets/moriyoshi/object/riaklogo.png", "code":"200",
"size":"0", "referer":"", "agent":"", "time":"2013-05-27T05:42:09Z", "tag":"riak.cluster2" }, ...
How to Query
Ruby Cluent for Querying irb> q = client.bucket(‘fluentlog’) irb> q
= q.map(“function(v){ return [v]; }”).reduce(“function(values){ return values; }“, :keep => false) irb> r = q.run()
Debug distributed JS http://www.flickr.com/photos/heatsink/110859301/
Any Other Rubyish way? http://www.flickr.com/photos/snazzyshot/5366645175/
ripple
github.com/basho/ripple •a rich Ruby toolkit for Riak, consists of •Riak
client •Riak-sessions •Ripple
http://www.flickr.com/photos/toco/2612055052/
None
None
Mohair: Not Only NoSQL http://www.flickr.com/photos/frank-wouters/2464743512/
JSON { "host":"103.5.142.5", "user":"-", "method":"PUT", "path":"/buckets/moriyoshi/object/riaklogo.png", "code":"200", "size":"0", "referer":"", "agent":"",
"time":"2013-05-27T05:42:09Z", "tag":"riak.cluster2" }, ...
SQL create table apachelogs { host varchar(16), user varchar(256), method
varchar(5), path varchar(1024), code integer, size integer, referer text, agent varchar(1024), time timestamp, tag varchar(1024) }
“Mohair” for Querying > select * from fluentlog \ where
method = “GET” group by host
Converting SQL to MapReduce •SQL -(parslet)-> JS -> Riak mapred
•where sentence is at Map •group by, count(-) is at Reduce
Chef’s Capricious Roadmap •Secondary Index Support •Query Optimization •types: timestamp,
float •nested columns •insert / delete
check it out! github: basho/riak kuenishi/fluent-plugin-riak kuenishi/mohair (kuenishi/fluent-logger-erlang)
Conclusion •NoSQL is not NoSQL any more •put’em all into
Riak via Fluentd •Query via SQL with Mohair •waiting for pull requests
Questions? •
[email protected]
•Riak Meetup (7/10) •Riak SCR (twice in a
month) •ιϑτΣΞσβΠϯ7݄߸(nginx/riak) •σʔλϕʔεΤϯδχΞཆಡຊ