Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Log Everything with Fluentd

Log Everything with Fluentd

Kentaro Kuribayashi

February 15, 2013
Tweet

More Decks by Kentaro Kuribayashi

Other Decks in Technology

Transcript

  1. Log Everything
    @kentaro

    View full-size slide

  2. @kentaro
    Software engineer
    Rubyist / Perl Monger
    Kentaro Kuribayashi
    paperboy&co.
    PHPer

    View full-size slide

  3. Contributions
    to
    Fluentd World

    View full-size slide

  4. out_flatten
    https://github.com/kentaro/fluent-plugin-flatten
    out_extract_query_params
    https://github.com/kentaro/fluent-plugin-extract_query_params
    out_rewrite
    https://github.com/kentaro/fluent-plugin-rewrite

    View full-size slide

  5. • Easy to extend
    • Easy to parse

    View full-size slide

  6. Key Description Apache Nginx
    time Time request is received %t $time_local
    vhost Host name %v $host
    host Remote host %h $remote_addr
    method Request method %m $method
    path Request path %U%q $request_uri
    version HTTP version %H $server_protocol
    status Response status %>s $status
    size Response size %b $body_bytes_sent
    referer Referer %{Referer}i $http_referer
    ua User-agent %{User-Agent}i $http_user_agent
    restime Time elapsed for response %D $request_time
    ustime Time elapsed for upstream response - $upstream_response_time

    View full-size slide

  7. LogFormat "vhost:%V\ttime:%t\thost:%h\tmethod:%m\tpath:%U%q
    \tversion:%H\tstatus:%>s\tsize:%b\treferer:%{Referer}i\tua:
    %{User-Agent}i\trestime:%D" ltsv
    log_format ltsv "vhost:$host\ttime:$time_local\thost:
    $remote_addr\tmethod:$request_method\tpath:$request_uri
    \tversion:$server_protocol\tstatus:$status\tsize:
    $body_bytes_sent\treferer:$http_referer\tua:
    $http_user_agent\trestime:$request_time\tustime:
    $upstream_response_time";
    Apache
    Nginx

    View full-size slide

  8. Log Everything
    to
    Access Log File

    View full-size slide

  9. with
    Log Everything
    to
    Access Log File

    View full-size slide

  10. fluent-plugin-php

    View full-size slide

  11. apache_note()

    View full-size slide

  12. %{foo}n #=> bar
    apache_note(‘foo’, ‘bar’)
    PHP Code
    Apache Log Format

    View full-size slide

  13. • URL Groups
    • User/Guest
    • Device
    • Bot Access
    • etc.

    View full-size slide

  14. Structuralize URLs
    • Visualization
    • Analysis
    • Rough grouping is enough

    View full-size slide

  15. Group Path
    read ^/book
    buy ^/cart
    ^/book/\d+/purchase
    find
    ^/special
    ^/label
    ^/users
    ^/books
    ^/authors
    write ^/admin
    ^/users/{account}/draft
    set ^/users/{account}/manage
    ^/users/{account}/profile
    ^/users/{account}/account
    communicate ^/users/{account}/contact
    ^/users/{account}/reaction
    top ^/$
    ^$

    View full-size slide


  16. type rewrite
    remove_prefix apache.log
    add_prefix filtered
    # url group

    key uriGroup
    pattern ^(.+)$
    append_to_tag true
    fallback other

    ...

    View full-size slide

  17. ...
    # pc/smartphone

    key device
    pattern ^(pc|sp)$
    append_to_tag true

    ...

    View full-size slide

  18. ...
    # user/guest

    key loggedIn
    pattern ^(user|guest)$
    append_to_tag true


    View full-size slide

  19. filtered.book.pc.user => { ... }
    filtered.book.sp.guest => { ... }
    filtered.read.pc.guest => { ... }
    ... uriGroup:book device:pc loggedIn:user ...
    ... uriGroup:book device:sp loggedIn:guest ...
    ... uriGroup:read device:pc loggedIn:guest ...
    Raw Log Lines
    Filtered Key/Values

    View full-size slide

  20. Problem
    Hard to track results
    continuously

    View full-size slide

  21. class Foo {
    function hoge () {
    $this->profiler->start(‘method-hoge’);
    // ... do something ...
    $this->profiler->end(‘method-hoge’);
    }
    function fuga () {
    $this->profiler->start(‘method-fuga-foo’);
    // ... do something ...
    $this->profiler->end(‘method-fuga-foo’);
    $this->profiler->start(‘method-fuga-bar’);
    // ... do something ...
    $this->profiler->end(‘method-fuga-bar’);
    }
    }

    View full-size slide

  22. public function dispatchLoopShutdown()
    {
    $profile_result = Model_Measure::dump();
    if (APPLICATION_ENV !== 'production') {
    Pb_Logger::debug($profile_result);
    }
    Pb_Util::httpd_note('profile', json_encode($profile_result));
    }

    View full-size slide

  23. static public function httpd_note($name, $value = '')
    {
    if (function_exists('apache_note')) {
    if ($value) {
    return apache_note($name, $value);
    } else {
    // do nothing...
    }
    }
    }

    View full-size slide

  24. ... profile:%{profile}n" ...
    profile:{\"book.index.get_book\":0.1010639667511,
    \"book.index.get_author\":0.2032630443573,
    \"book.index.get_chapters\":0.23988509178162,
    \"book.index.get_version\":0.039833068847656,
    \"book.index.check\":0.00014090538024902,
    \"book.index.get_pager\":0.00022792816162109}
    Log Format
    Raw Log Line

    View full-size slide


  25. type flatten
    key foo
    add_tag_prefix flattened.
    remove_tag_prefix test.
    inner_key value_for_flat_key

    "test" => {
    "foo" => '{"bar" : {"qux" : "quux", "hoe" : "poe" }, "baz" : "bazz" }',
    "hoge" => "fuga"
    }
    "flattened.foo.bar.qux" => { "value_for_flat_key" => "quux" }
    "flattened.foo.bar.hoe" => { "value_for_flat_key" => "poe" }
    "flattened.foo.baz" => { "value_for_flat_key" => "bazz" }
    flattened

    View full-size slide


  26. type flatten
    key profile
    inner_key response_time
    add_tag_prefix flattened.
    remove_tag_prefix app.httpd.access.

    View full-size slide

  27. out_extract_query_params

    View full-size slide


  28. type extract_query_params
    key url
    add_tag_prefix extracted.
    only foo, baz

    "test" => {
    "url" => "http://example.com/?foo=bar&baz=qux&hoge=fuga"
    }
    "extracted.test" => {
    "url" => "http://example.com/?foo=bar&baz=qux&hoge=fuga"
    "foo" => "bar",
    "baz" => "qux"
    }
    Extract
    Params

    View full-size slide


  29. type extract_query_params
    key path
    add_tag_prefix extracted.
    only hoge

    method:%m path:%U%q version:%H
    #=> method:GET path:/foo/bar?campaign_id=1 version:HTTP1.1
    extracted.access_log => {
    "method" : "GET",
    "path" : "/foo/bar?campaign_id=1",
    "version" : "HTTP1.1",
    "campaign_id" : "1"
    }
    Extract
    Params
    Works
    fine with
    LTSV

    View full-size slide

  30. • Adopt LTSV asap
    • Log things as much
    as possible
    • My plugins help you

    View full-size slide