Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Log Everything with Fluentd
Kentaro Kuribayashi
February 15, 2013
Technology
4
4k
Log Everything with Fluentd
Kentaro Kuribayashi
February 15, 2013
Tweet
Share
More Decks by Kentaro Kuribayashi
See All by Kentaro Kuribayashi
WEB+DB PRESSで特集記事を書く方法 / How to Become an Author of WEB+DB PRESS
kentaro
4
660
GMOペパボが考えるこれからのサービス開発 / Toward The Next Service Development Era
kentaro
1
650
IoTシステムの双方向データフローにおける設計と実装の複雑さを解消する手法の提案 / Master's Thesis Examination
kentaro
0
450
双方向データフローに基づくインテリジェントなIoTシステムを実現するための研究 / My Research Plan for the Doctoral Course
kentaro
0
450
Elixirで構成された3層構造からなるIoTシステムにおける分散機械学習・推論実行基盤へ向けて / Toward a Distributed Machine Learning Infrastructure for IoT Systems in Elixir
kentaro
1
600
IoTシステムの双方向データフローにおける設計と実装の複雑さを解消する手法の提案 / Proposal to Eliminate the Complexity of Design and Implementation in The Bidirectional Dataflow of IoT Systems
kentaro
1
770
Elixirでプログラミング言語を作ろう #tokyoex / Let's Make Your Own Language in Elixir
kentaro
0
580
Multi-Tenant Erlang Distributionを目指して - ElixirConf US 2021参加報告 / Toward The Multi-Tenant Erlang Distribution
kentaro
0
360
Pratipad: A Declarative Framework for Describing Bidirectional Dataflow in IoT Systems with Elixir
kentaro
0
1.3k
Other Decks in Technology
See All in Technology
Oracle Database Technology Night #55 Oracle Autonomous Database 再入門
oracle4engineer
PRO
1
110
Salesforce女子部-権限についてまとめてみたその1
sfggjp
0
180
Poolにおける足を止めないシステム基盤構築
winebarrel
3
630
How We Foster Reliability in Diversity
nari_ex
PRO
8
2.3k
技術広報の役割を定義してみた 2022年春
afroscript
3
2.4k
エンタープライズにおけるSRE立ち上げとNew Relic選定に至った背景とは / SRE Startup and New Relic in the Enterprise
tomoyakitaura
2
140
ニフティでSRE推進活動を始めて取り組んできたこと
niftycorp
2
190
1年間のポストモーテム運用とそこから生まれたツール sre-advisor / SRE NEXT 2022
fujiwara3
6
2.9k
ISUCON で使えるツールを作った
shotakitazawa
0
350
SRE_チーム立ち上げから1年_気づいたら_SRE_っぽくない仕事まで貢献しちゃってる説
bitkey
PRO
0
1.9k
Power Query 日時の変換でちょっと焦ったケース +1 / Power Query Some cases
ishiayaya
0
150
成長を続ける組織でのSRE戦略:プレモーテムによる信頼性の認識共有 SRE Next 2022
niwatakeru
7
2.3k
Featured
See All Featured
Building a Scalable Design System with Sketch
lauravandoore
447
30k
Practical Orchestrator
shlominoach
178
8.6k
Web Components: a chance to create the future
zenorocha
303
40k
The Art of Programming - Codeland 2020
erikaheidi
31
5.8k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
268
11k
Done Done
chrislema
174
14k
What’s in a name? Adding method to the madness
productmarketing
11
1.5k
Reflections from 52 weeks, 52 projects
jeffersonlam
337
17k
A better future with KSS
kneath
225
15k
Visualization
eitanlees
124
11k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
37
3.2k
Web development in the modern age
philhawksworth
197
9.3k
Transcript
Log Everything @kentaro
@kentaro Software engineer Rubyist / Perl Monger Kentaro Kuribayashi paperboy&co.
PHPer
None
Contributions to Fluentd World
out_flatten https://github.com/kentaro/fluent-plugin-flatten out_extract_query_params https://github.com/kentaro/fluent-plugin-extract_query_params out_rewrite https://github.com/kentaro/fluent-plugin-rewrite
None
Log Format
None
• Easy to extend • Easy to parse
Key Description Apache Nginx time Time request is received %t
$time_local vhost Host name %v $host host Remote host %h $remote_addr method Request method %m $method path Request path %U%q $request_uri version HTTP version %H $server_protocol status Response status %>s $status size Response size %b $body_bytes_sent referer Referer %{Referer}i $http_referer ua User-agent %{User-Agent}i $http_user_agent restime Time elapsed for response %D $request_time ustime Time elapsed for upstream response - $upstream_response_time
LogFormat "vhost:%V\ttime:%t\thost:%h\tmethod:%m\tpath:%U%q \tversion:%H\tstatus:%>s\tsize:%b\treferer:%{Referer}i\tua: %{User-Agent}i\trestime:%D" ltsv log_format ltsv "vhost:$host\ttime:$time_local\thost: $remote_addr\tmethod:$request_method\tpath:$request_uri \tversion:$server_protocol\tstatus:$status\tsize:
$body_bytes_sent\treferer:$http_referer\tua: $http_user_agent\trestime:$request_time\tustime: $upstream_response_time"; Apache Nginx
Log Everything to Access Log File
with Log Everything to Access Log File
fluent-plugin-php
None
apache_note()
%{foo}n #=> bar apache_note(‘foo’, ‘bar’) PHP Code Apache Log Format
Flags
• URL Groups • User/Guest • Device • Bot Access
• etc.
Structuralize URLs • Visualization • Analysis • Rough grouping is
enough
Group Path read ^/book buy ^/cart ^/book/\d+/purchase find ^/special ^/label
^/users ^/books ^/authors write ^/admin ^/users/{account}/draft set ^/users/{account}/manage ^/users/{account}/profile ^/users/{account}/account communicate ^/users/{account}/contact ^/users/{account}/reaction top ^/$ ^$
out_rewrite
<match apache.log.**> type rewrite remove_prefix apache.log add_prefix filtered # url
group <rule> key uriGroup pattern ^(.+)$ append_to_tag true fallback other </rule> ...
... # pc/smartphone <rule> key device pattern ^(pc|sp)$ append_to_tag true
</rule> ...
... # user/guest <rule> key loggedIn pattern ^(user|guest)$ append_to_tag true
</rule> </match>
filtered.book.pc.user => { ... } filtered.book.sp.guest => { ... }
filtered.read.pc.guest => { ... } ... uriGroup:book device:pc loggedIn:user ... ... uriGroup:book device:sp loggedIn:guest ... ... uriGroup:read device:pc loggedIn:guest ... Raw Log Lines Filtered Key/Values
None
Profiling
Problem Hard to track results continuously
class Foo { function hoge () { $this->profiler->start(‘method-hoge’); // ...
do something ... $this->profiler->end(‘method-hoge’); } function fuga () { $this->profiler->start(‘method-fuga-foo’); // ... do something ... $this->profiler->end(‘method-fuga-foo’); $this->profiler->start(‘method-fuga-bar’); // ... do something ... $this->profiler->end(‘method-fuga-bar’); } }
public function dispatchLoopShutdown() { $profile_result = Model_Measure::dump(); if (APPLICATION_ENV !==
'production') { Pb_Logger::debug($profile_result); } Pb_Util::httpd_note('profile', json_encode($profile_result)); }
static public function httpd_note($name, $value = '') { if (function_exists('apache_note'))
{ if ($value) { return apache_note($name, $value); } else { // do nothing... } } }
... profile:%{profile}n" ... profile:{\"book.index.get_book\":0.1010639667511, \"book.index.get_author\":0.2032630443573, \"book.index.get_chapters\":0.23988509178162, \"book.index.get_version\":0.039833068847656, \"book.index.check\":0.00014090538024902, \"book.index.get_pager\":0.00022792816162109} Log
Format Raw Log Line
out_flatten
<match test.**> type flatten key foo add_tag_prefix flattened. remove_tag_prefix test.
inner_key value_for_flat_key </match> "test" => { "foo" => '{"bar" : {"qux" : "quux", "hoe" : "poe" }, "baz" : "bazz" }', "hoge" => "fuga" } "flattened.foo.bar.qux" => { "value_for_flat_key" => "quux" } "flattened.foo.bar.hoe" => { "value_for_flat_key" => "poe" } "flattened.foo.baz" => { "value_for_flat_key" => "bazz" } flattened
<match app.httpd.access> type flatten key profile inner_key response_time add_tag_prefix flattened.
remove_tag_prefix app.httpd.access. </store>
None
None
Bonus
out_extract_query_params
<match test.**> type extract_query_params key url add_tag_prefix extracted. only foo,
baz </match> "test" => { "url" => "http://example.com/?foo=bar&baz=qux&hoge=fuga" } "extracted.test" => { "url" => "http://example.com/?foo=bar&baz=qux&hoge=fuga" "foo" => "bar", "baz" => "qux" } Extract Params
<match access_log> type extract_query_params key path add_tag_prefix extracted. only hoge
</match> method:%m path:%U%q version:%H #=> method:GET path:/foo/bar?campaign_id=1 version:HTTP1.1 extracted.access_log => { "method" : "GET", "path" : "/foo/bar?campaign_id=1", "version" : "HTTP1.1", "campaign_id" : "1" } Extract Params Works fine with LTSV
Recap
• Adopt LTSV asap • Log things as much as
possible • My plugins help you