Log Everything with Fluentd

Log Everything with Fluentd

23f4d5d797a91b6d17d627b90b5a42d9?s=128

Kentaro Kuribayashi

February 15, 2013
Tweet

Transcript

  1. 3.
  2. 6.
  3. 8.
  4. 10.

    Key Description Apache Nginx time Time request is received %t

    $time_local vhost Host name %v $host host Remote host %h $remote_addr method Request method %m $method path Request path %U%q $request_uri version HTTP version %H $server_protocol status Response status %>s $status size Response size %b $body_bytes_sent referer Referer %{Referer}i $http_referer ua User-agent %{User-Agent}i $http_user_agent restime Time elapsed for response %D $request_time ustime Time elapsed for upstream response - $upstream_response_time
  5. 15.
  6. 18.
  7. 21.

    Group Path read ^/book buy ^/cart ^/book/\d+/purchase find ^/special ^/label

    ^/users ^/books ^/authors write ^/admin ^/users/{account}/draft set ^/users/{account}/manage ^/users/{account}/profile ^/users/{account}/account communicate ^/users/{account}/contact ^/users/{account}/reaction top ^/$ ^$
  8. 23.

    <match apache.log.**> type rewrite remove_prefix apache.log add_prefix filtered # url

    group <rule> key uriGroup pattern ^(.+)$ append_to_tag true fallback other </rule> ...
  9. 26.

    filtered.book.pc.user => { ... } filtered.book.sp.guest => { ... }

    filtered.read.pc.guest => { ... } ... uriGroup:book device:pc loggedIn:user ... ... uriGroup:book device:sp loggedIn:guest ... ... uriGroup:read device:pc loggedIn:guest ... Raw Log Lines Filtered Key/Values
  10. 27.
  11. 30.

    class Foo { function hoge () { $this->profiler->start(‘method-hoge’); // ...

    do something ... $this->profiler->end(‘method-hoge’); } function fuga () { $this->profiler->start(‘method-fuga-foo’); // ... do something ... $this->profiler->end(‘method-fuga-foo’); $this->profiler->start(‘method-fuga-bar’); // ... do something ... $this->profiler->end(‘method-fuga-bar’); } }
  12. 31.

    public function dispatchLoopShutdown() { $profile_result = Model_Measure::dump(); if (APPLICATION_ENV !==

    'production') { Pb_Logger::debug($profile_result); } Pb_Util::httpd_note('profile', json_encode($profile_result)); }
  13. 32.

    static public function httpd_note($name, $value = '') { if (function_exists('apache_note'))

    { if ($value) { return apache_note($name, $value); } else { // do nothing... } } }
  14. 35.

    <match test.**> type flatten key foo add_tag_prefix flattened. remove_tag_prefix test.

    inner_key value_for_flat_key </match> "test" => { "foo" => '{"bar" : {"qux" : "quux", "hoe" : "poe" }, "baz" : "bazz" }', "hoge" => "fuga" } "flattened.foo.bar.qux" => { "value_for_flat_key" => "quux" } "flattened.foo.bar.hoe" => { "value_for_flat_key" => "poe" } "flattened.foo.baz" => { "value_for_flat_key" => "bazz" } flattened
  15. 37.
  16. 38.
  17. 39.
  18. 41.

    <match test.**> type extract_query_params key url add_tag_prefix extracted. only foo,

    baz </match> "test" => { "url" => "http://example.com/?foo=bar&baz=qux&hoge=fuga" } "extracted.test" => { "url" => "http://example.com/?foo=bar&baz=qux&hoge=fuga" "foo" => "bar", "baz" => "qux" } Extract Params
  19. 42.

    <match access_log> type extract_query_params key path add_tag_prefix extracted. only hoge

    </match> method:%m path:%U%q version:%H #=> method:GET path:/foo/bar?campaign_id=1 version:HTTP1.1 extracted.access_log => { "method" : "GET", "path" : "/foo/bar?campaign_id=1", "version" : "HTTP1.1", "campaign_id" : "1" } Extract Params Works fine with LTSV
  20. 43.
  21. 44.

    • Adopt LTSV asap • Log things as much as

    possible • My plugins help you