Log Everything with Fluentd

Transcript

1. Log Everything @kentaro
2. @kentaro Software engineer Rubyist / Perl Monger Kentaro Kuribayashi paperboy&co. PHPer
4. Contributions to Fluentd World
5. out_flatten https://github.com/kentaro/fluent-plugin-flatten out_extract_query_params https://github.com/kentaro/fluent-plugin-extract_query_params out_rewrite https://github.com/kentaro/fluent-plugin-rewrite
7. Log Format
9. • Easy to extend • Easy to parse
10. Key Description Apache Nginx time Time request is received %t $time_local vhost Host name %v $host host Remote host %h $remote_addr method Request method %m $method path Request path %U%q $request_uri version HTTP version %H $server_protocol status Response status %>s $status size Response size %b $body_bytes_sent referer Referer %{Referer}i $http_referer ua User-agent %{User-Agent}i $http_user_agent restime Time elapsed for response %D $request_time ustime Time elapsed for upstream response - $upstream_response_time
11. LogFormat "vhost:%V\ttime:%t\thost:%h\tmethod:%m\tpath:%U%q \tversion:%H\tstatus:%>s\tsize:%b\treferer:%{Referer}i\tua: %{User-Agent}i\trestime:%D" ltsv log_format ltsv "vhost:$host\ttime:$time_local\thost: $remote_addr\tmethod:$request_method\tpath:$request_uri \tversion:$server_protocol\tstatus:$status\tsize: $body_bytes_sent\treferer:$http_referer\tua: $http_user_agent\trestime:$request_time\tustime: $upstream_response_time"; Apache Nginx
12. Log Everything to Access Log File
13. with Log Everything to Access Log File
14. fluent-plugin-php
16. apache_note()
17. %{foo}n #=> bar apache_note(‘foo’, ‘bar’) PHP Code Apache Log Format
18. Flags
19. • URL Groups • User/Guest • Device • Bot Access • etc.
20. Structuralize URLs • Visualization • Analysis • Rough grouping is enough
21. Group Path read ^/book buy ^/cart ^/book/\d+/purchase find ^/special ^/label ^/users ^/books ^/authors write ^/admin ^/users/{account}/draft set ^/users/{account}/manage ^/users/{account}/profile ^/users/{account}/account communicate ^/users/{account}/contact ^/users/{account}/reaction top ^/$ ^$
22. out_rewrite
23. <match apache.log.**> type rewrite remove_prefix apache.log add_prefix filtered # url group <rule> key uriGroup pattern ^(.+)$ append_to_tag true fallback other </rule> ...
24. ... # pc/smartphone <rule> key device pattern ^(pc|sp)$ append_to_tag true </rule> ...
25. ... # user/guest <rule> key loggedIn pattern ^(user|guest)$ append_to_tag true </rule> </match>
26. filtered.book.pc.user => { ... } filtered.book.sp.guest => { ... } filtered.read.pc.guest => { ... } ... uriGroup:book device:pc loggedIn:user ... ... uriGroup:book device:sp loggedIn:guest ... ... uriGroup:read device:pc loggedIn:guest ... Raw Log Lines Filtered Key/Values
28. Profiling
29. Problem Hard to track results continuously
30. class Foo { function hoge () { $this->profiler->start(‘method-hoge’); // ... do something ... $this->profiler->end(‘method-hoge’); } function fuga () { $this->profiler->start(‘method-fuga-foo’); // ... do something ... $this->profiler->end(‘method-fuga-foo’); $this->profiler->start(‘method-fuga-bar’); // ... do something ... $this->profiler->end(‘method-fuga-bar’); } }
31. public function dispatchLoopShutdown() { $profile_result = Model_Measure::dump(); if (APPLICATION_ENV !== 'production') { Pb_Logger::debug($profile_result); } Pb_Util::httpd_note('profile', json_encode($profile_result)); }
32. static public function httpd_note($name, $value = '') { if (function_exists('apache_note')) { if ($value) { return apache_note($name, $value); } else { // do nothing... } } }
33. ... profile:%{profile}n" ... profile:{\"book.index.get_book\":0.1010639667511, \"book.index.get_author\":0.2032630443573, \"book.index.get_chapters\":0.23988509178162, \"book.index.get_version\":0.039833068847656, \"book.index.check\":0.00014090538024902, \"book.index.get_pager\":0.00022792816162109} Log Format Raw Log Line
34. out_flatten
35. <match test.**> type flatten key foo add_tag_prefix flattened. remove_tag_prefix test. inner_key value_for_flat_key </match> "test" => { "foo" => '{"bar" : {"qux" : "quux", "hoe" : "poe" }, "baz" : "bazz" }', "hoge" => "fuga" } "flattened.foo.bar.qux" => { "value_for_flat_key" => "quux" } "flattened.foo.bar.hoe" => { "value_for_flat_key" => "poe" } "flattened.foo.baz" => { "value_for_flat_key" => "bazz" } flattened
36. <match app.httpd.access> type flatten key profile inner_key response_time add_tag_prefix flattened. remove_tag_prefix app.httpd.access. </store>
39. Bonus
40. out_extract_query_params
41. <match test.**> type extract_query_params key url add_tag_prefix extracted. only foo, baz </match> "test" => { "url" => "http://example.com/?foo=bar&baz=qux&hoge=fuga" } "extracted.test" => { "url" => "http://example.com/?foo=bar&baz=qux&hoge=fuga" "foo" => "bar", "baz" => "qux" } Extract Params
42. <match access_log> type extract_query_params key path add_tag_prefix extracted. only hoge </match> method:%m path:%U%q version:%H #=> method:GET path:/foo/bar?campaign_id=1 version:HTTP1.1 extracted.access_log => { "method" : "GET", "path" : "/foo/bar?campaign_id=1", "version" : "HTTP1.1", "campaign_id" : "1" } Extract Params Works fine with LTSV
43. Recap
44. • Adopt LTSV asap • Log things as much as possible • My plugins help you