Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Casual Log Collection and Querying with fluent-...
Search
UENISHI Kota
June 01, 2013
Technology
3
420
Casual Log Collection and Querying with fluent-plugin-riak
My talk at RubyKaigi 2013
http://rubykaigi.org/2013/talk/S70
UENISHI Kota
June 01, 2013
Tweet
Share
More Decks by UENISHI Kota
See All by UENISHI Kota
Metadata Management in Distributed File Systems
kuenishi
2
460
Behind The Scenes: Cloud Native Storage System for AI
kuenishi
2
320
Apache Ozone behind Simulation and AI Industries
kuenishi
0
280
Distributed Deep Learning with Chainer and Hadoop
kuenishi
3
1.1k
A Few Ways to Accelerate Deep Learning
kuenishi
0
980
Introducing Retz
kuenishi
5
1k
Introducing Retz and how to develop practical frameworks
kuenishi
3
660
Formalization and Proof of Distributed Systems (ja)
kuenishi
10
6.2k
Mesos Frameworkの作り方 (How to Make Mesos Framework)
kuenishi
7
2.3k
Other Decks in Technology
See All in Technology
全社を巻き込んだ業務オペレーション改善と、それは事業成長に貢献しているのか?を実感した話
marroooon
0
150
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
5
48k
RDS for Db2 データ移行編 - Part2:S3経由のバックアップ・リストアでデータ移行 /20241011-RDSforDb2-dojo
mayumihirano
0
140
WebRTC と AI の組み合わせ
tnoho
0
100
Japan AWS Jr. Championsがお届けする、アウトプットのすすめ
hamijay_cloud
0
210
Amazon ECS & AWS Fargate 今昔物語 / past and present stories of Amazon ECS and AWS Fargate
iselegant
19
4.1k
Bluesky 2019〜2022
yamarten
1
120
フェンリルの SwiftUI の研修を覗いてみる / Fenrir SwiftUI Training
studio_rookery
0
160
実は仲良し?Amplify Gen2と生成AI
mkdev10
0
150
Deep dive into Nuxt Server Components
wattanx
1
1.3k
Application Signalsで始めるSLO ユーザー満足度を数値化する第一歩
niftycorp
PRO
2
160
Transforming Event Attendees into Lifelong Donors: Insights from Claire Axelrad
auctria
PRO
1
130
Featured
See All Featured
Being A Developer After 40
akosma
85
590k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
26
4.1k
A Modern Web Designer's Workflow
chriscoyier
692
190k
Practical Orchestrator
shlominoach
186
10k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
37
1.8k
Learning to Love Humans: Emotional Interface Design
aarron
272
40k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
159
15k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
40
2.1k
Side Projects
sachag
452
42k
4 Signs Your Business is Dying
shpigford
180
21k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
48k
Designing for humans not robots
tammielis
249
25k
Transcript
Casual Log Collection and Querying with fluent-plugin-riak @kuenishi from @basho
2013/6/1 RubyKaigi
Who the hell are you? •UENISHI, Kota (@kuenishi) •Basho Japan
KK •devoted to Distributed Systems for ~6 yrs •msgpack-erlang, Jubatus
Casual Log Collection •Aggregate Every Log with Fluentd •Put Them
all into <Some Storage You Like> •Ask your Query to <Some Storage You Like>
Whole Sketch
fluentd: casual log collector http://www.flickr.com/photos/markchadwick/8757802771/ http://www.flickr.com/photos/usdagov/5681152426/ before: logs are scattered
all over the servers in chaos after: all logs flows cleanly via fluentd in order
Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache
Frontend Access logs syslogd App logs System logs Backend Databases
Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache
Frontend Access logs syslogd App logs System logs Backend Databases filter / buffer / routing
Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache
Frontend Access logs syslogd App logs System logs Backend Databases filter / buffer / routing Riak
what’s ? •Distributed Key-Value Store •Focused on •Availability •Scalability •Easy
Operation, ҆ (Sleep)
when Riak? •Hadoop is too much •MongoDB is too small
•Document DB aspect of Riak •put them all into Riak
Not Only KVS •Aspect of Document Database •MapReduce in JavaScript
/ Erlang
Buy it if interested
fluent-plugin-riak JSON
fluent.conf <match apache.**> type riak # define the cluster via
pb ports nodes 192.168.0.1:8087 192.168.0.2:8087 </match>
log everything as JSON { "host":"103.5.142.5", "user":"-", "method":"PUT", "path":"/buckets/moriyoshi/object/riaklogo.png", "code":"200",
"size":"0", "referer":"", "agent":"", "time":"2013-05-27T05:42:09Z", "tag":"riak.cluster2" }, ...
How to Query
Ruby Cluent for Querying irb> q = client.bucket(‘fluentlog’) irb> q
= q.map(“function(v){ return [v]; }”).reduce(“function(values){ return values; }“, :keep => false) irb> r = q.run()
Debug distributed JS http://www.flickr.com/photos/heatsink/110859301/
Any Other Rubyish way? http://www.flickr.com/photos/snazzyshot/5366645175/
ripple
github.com/basho/ripple •a rich Ruby toolkit for Riak, consists of •Riak
client •Riak-sessions •Ripple
http://www.flickr.com/photos/toco/2612055052/
None
None
Mohair: Not Only NoSQL http://www.flickr.com/photos/frank-wouters/2464743512/
JSON { "host":"103.5.142.5", "user":"-", "method":"PUT", "path":"/buckets/moriyoshi/object/riaklogo.png", "code":"200", "size":"0", "referer":"", "agent":"",
"time":"2013-05-27T05:42:09Z", "tag":"riak.cluster2" }, ...
SQL create table apachelogs { host varchar(16), user varchar(256), method
varchar(5), path varchar(1024), code integer, size integer, referer text, agent varchar(1024), time timestamp, tag varchar(1024) }
“Mohair” for Querying > select * from fluentlog \ where
method = “GET” group by host
Converting SQL to MapReduce •SQL -(parslet)-> JS -> Riak mapred
•where sentence is at Map •group by, count(-) is at Reduce
Chef’s Capricious Roadmap •Secondary Index Support •Query Optimization •types: timestamp,
float •nested columns •insert / delete
check it out! github: basho/riak kuenishi/fluent-plugin-riak kuenishi/mohair (kuenishi/fluent-logger-erlang)
Conclusion •NoSQL is not NoSQL any more •put’em all into
Riak via Fluentd •Query via SQL with Mohair •waiting for pull requests
Questions? •
[email protected]
•Riak Meetup (7/10) •Riak SCR (twice in a
month) •ιϑτΣΞσβΠϯ7݄߸(nginx/riak) •σʔλϕʔεΤϯδχΞཆಡຊ