Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Logging: Your new best friend

Logging: Your new best friend

Logs are not just used when things go wrong. They also help you keep track of what is going on within your app. We will look at how you can add helpful messages throughout your codebase and leave them there, even in production! We will cover common logging strategies, log aggregation, and how to efficiently work with your logs to get the data back out. We will also look at Graphite, which can help work out what actually happened by correlating logs with peaks/drops in other systems.

Michael Heap

May 25, 2016
Tweet

More Decks by Michael Heap

Other Decks in Technology

Transcript

  1. Logging: Your new best friend
    Michael Heap (@mheap)
    Developer at DataSift
    Presented at php|tek, May 2016

    View full-size slide

  2. Me!
    I’m Michael
    I’m @mheap
    Developer at DataSift

    View full-size slide

  3. @mheap
    https://joind.in/17042

    View full-size slide

  4. 1. Logging
    2. Getting started
    3. The ELK stack
    4. Logs and dashboards
    5. Log management
    6. Supporting services
    7. Conclusion

    View full-size slide

  5. 1. Logging
    2. Getting started
    3. The ELK stack
    4. Logs and dashboards
    5. Log management
    6. Supporting services
    7. Conclusion

    View full-size slide

  6. 1. Logging
    2. Getting started
    3. The ELK stack
    4. Logs and dashboards
    5. Log management
    6. Supporting services
    7. Conclusion

    View full-size slide

  7. 1. Logging
    2. Getting started
    3. The ELK stack
    4. Logs and dashboards
    5. Log management
    6. Supporting services
    7. Conclusion

    View full-size slide

  8. 1. Logging
    2. Getting started
    3. The ELK stack
    4. Logs and dashboards
    5. Log management
    6. Supporting services
    7. Conclusion

    View full-size slide

  9. 1. Logging
    2. Getting started
    3. The ELK stack
    4. Logs and dashboards
    5. Log management
    6. Supporting services
    7. Conclusion

    View full-size slide

  10. 1. Logging
    2. Getting started
    3. The ELK stack
    4. Logs and dashboards
    5. Log management
    6. Supporting services
    7. Conclusion

    View full-size slide

  11. 1. Logging
    2. Getting started
    3. The ELK stack
    4. Logs and dashboards
    5. Log management
    6. Supporting services
    7. Conclusion

    View full-size slide

  12. 1. Logging
    2. Getting started
    3. The ELK stack
    4. Logs and dashboards
    5. Log management
    6. Supporting services
    7. Conclusion

    View full-size slide

  13. What went wrong?
    (Error log)

    View full-size slide

  14. Who visited us?
    (Access log)

    View full-size slide

  15. Who enabled ?
    (Audit log)

    View full-size slide

  16. Runtime documentation
    (Application log)

    View full-size slide

  17. Can I have it for free?

    View full-size slide

  18. Actually, yes!

    View full-size slide

  19. But that doesn’t help
    my application

    View full-size slide

  20. Two types of log

    View full-size slide

  21. Human readable

    View full-size slide

  22. Machine readable

    View full-size slide

  23. We should log both

    View full-size slide

  24. What is an application log?

    View full-size slide

  25. Debug information

    View full-size slide

  26. Narrative information

    View full-size slide

  27. Business information

    View full-size slide

  28. “An application log
    signposts every twist and
    turn through the code”

    View full-size slide

  29. 1. Logging
    2. Getting started
    3. The ELK stack
    4. Logs and dashboards
    5. Log management
    6. Supporting services
    7. Conclusion

    View full-size slide

  30. Getting started

    View full-size slide

  31. function countConsonants($str){
    $c = strlen(str_replace(['a','e','i','o','u'],'', $str));
    error_log("Consonants in {$str}: {$c}");
    return $c;
    }
    echo countConsonants("Michael");

    View full-size slide

  32. function countConsonants($str){
    $c = strlen(str_replace(['a','e','i','o','u'],'', $str));
    error_log("Consonants in {$str}: {$c}");
    return $c;
    }
    echo countConsonants("Michael");

    View full-size slide

  33. ini_set("error_log", "/var/log/casino-app.log");
    function countConsonants($str){
    $c = strlen(str_replace(['a','e','i','o','u'],'', $str));
    error_log("Consonants in {$str}: {$c}");
    return $c;
    }
    echo countConsonants("Michael");

    View full-size slide

  34. Pros
    ✴ It’s built in

    View full-size slide

  35. Cons
    ✴ Is it semantically correct?
    ✴ Errors mixed with informational logs
    ✴ It’s not very powerful

    View full-size slide

  36. Logging frameworks

    View full-size slide

  37. 1) Monolog
    2) Everything else

    View full-size slide

  38. require_once 'vendor/autoload.php';
    $log = new Monolog\Logger('casino-app');
    $log->pushHandler(new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog
    \Logger::DEBUG));
    function countConsonants($str, $log){
    $c = strlen(str_replace(['a','e','i','o','u'],'', $str));
    $log->info("Consonants in {$str}: {$c}");
    return $c;
    }
    echo countConsonants("Michael", $log);

    View full-size slide

  39. require_once 'vendor/autoload.php';
    $log = new Monolog\Logger('casino-app');
    $log->pushHandler(new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog
    \Logger::DEBUG));
    function countConsonants($str, $log){
    $c = strlen(str_replace(['a','e','i','o','u'],'', $str));
    $log->info("Consonants in {$str}: {$c}");
    return $c;
    }
    echo countConsonants("Michael", $log);

    View full-size slide

  40. require_once 'vendor/autoload.php';
    $log = new Monolog\Logger('casino-app');
    $log->pushHandler(new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog
    \Logger::DEBUG));
    function countConsonants($str, $log){
    $c = strlen(str_replace(['a','e','i','o','u'],'', $str));
    $log->info("Consonants in {$str}: {$c}");
    return $c;
    }
    echo countConsonants("Michael", $log);

    View full-size slide

  41. require_once 'vendor/autoload.php';
    $log = new Monolog\Logger('casino-app');
    $log->pushHandler(new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog
    \Logger::DEBUG));
    function countConsonants($str, $log){
    $c = strlen(str_replace(['a','e','i','o','u'],'', $str));
    $log->info("Consonants in {$str}: {$c}");
    return $c;
    }
    echo countConsonants("Michael", $log);

    View full-size slide

  42. require_once 'vendor/autoload.php';
    $log = new Monolog\Logger('casino-app');
    $log->pushHandler(new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog
    \Logger::DEBUG));
    function countConsonants($str, $log){
    $c = strlen(str_replace(['a','e','i','o','u'],'', $str));
    $log->info("Consonants in {$str}: {$c}");
    return $c;
    }
    echo countConsonants("Michael", $log);

    View full-size slide

  43. [2016-05-25 03:56:01] casino-app.INFO: Consonants in Michael: 4 [] []

    View full-size slide

  44. FingersCrossedHandler

    View full-size slide

  45. $log = new Monolog\Logger('casino-app');
    $streamHandler = new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog
    \Logger::DEBUG);
    $fcHandler = new Monolog\Handler\FingersCrossedHandler($streamHandler, Monolog
    \Logger::ERROR);
    $log->pushHandler($fcHandler);
    function countConsonants($str, $log){
    $c = strlen(str_replace(['a','e','i','o','u'],'', $str));
    $log->info("Consonants in {$str}: {$c}");
    return $c;
    }
    echo countConsonants("Michael", $log);

    View full-size slide

  46. $log = new Monolog\Logger('casino-app');
    $streamHandler = new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog
    \Logger::DEBUG);
    $fcHandler = new Monolog\Handler\FingersCrossedHandler($streamHandler, Monolog
    \Logger::ERROR);
    $log->pushHandler($fcHandler);
    function countConsonants($str, $log){
    $c = strlen(str_replace(['a','e','i','o','u'],'', $str));
    $log->info("Consonants in {$str}: {$c}");
    return $c;
    }
    echo countConsonants("Michael", $log);

    View full-size slide

  47. $log = new Monolog\Logger('casino-app');
    $streamHandler = new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog
    \Logger::DEBUG);
    $fcHandler = new Monolog\Handler\FingersCrossedHandler($streamHandler, Monolog
    \Logger::ERROR);
    $log->pushHandler($fcHandler);
    function countConsonants($str, $log){
    $c = strlen(str_replace(['a','e','i','o','u'],'', $str));
    $log->info("Consonants in {$str}: {$c}");
    return $c;
    }
    echo countConsonants("Michael", $log);

    View full-size slide

  48. $log = new Monolog\Logger('casino-app');
    $streamHandler = new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog
    \Logger::DEBUG);
    $fcHandler = new Monolog\Handler\FingersCrossedHandler($streamHandler, Monolog
    \Logger::ERROR);
    $log->pushHandler($fcHandler);
    function countConsonants($str, $log){
    $c = strlen(str_replace(['a','e','i','o','u'],'', $str));
    $log->info("Consonants in {$str}: {$c}");
    $log->error("Something bad happened");
    return $c;
    }
    echo countConsonants("Michael", $log);

    View full-size slide

  49. Pros
    ✴ It’s an object! Dependency injection FTW
    ✴ Supports multiple log writers
    ✴ Log level support

    View full-size slide

  50. Cons
    ✴ Instantiating an instance can be complicated

    View full-size slide

  51. Error Levels

    View full-size slide

  52. 0. Emergency System is unusable
    1. Alert Should be corrected immediately
    2. Critical Critical conditions
    3. Error Error conditions
    4. Warning May indicate that an error will occur if action is not taken.
    5. Notice Events that are unusual, but not error conditions.
    6. Informational Normal operational messages that require no action.
    7. Debug Information useful to developers for debugging the application.
    Syslog (RFC 5424)

    View full-size slide

  53. 0. Emergency System is unusable
    1. Alert Should be corrected immediately
    2. Critical Critical conditions
    3. Error Error conditions
    4. Warning May indicate that an error will occur if action is not taken.
    5. Notice Events that are unusual, but not error conditions.
    6. Informational Normal operational messages that require no action.
    7. Debug Information useful to developers for debugging the application.
    PSR3

    View full-size slide

  54. 1. Logging
    2. Getting started
    3. The ELK stack
    4. Logs and dashboards
    5. Log management
    6. Supporting services
    7. Conclusion

    View full-size slide

  55. Everything is on fire

    View full-size slide

  56. The ELK Stack

    View full-size slide

  57. Elasticsearch
    Logstash
    Kibana

    View full-size slide

  58. Logstash
    Elasticsearch
    Kibana

    View full-size slide

  59. Beats
    CouchDB_Changes
    Drupal_DBLog
    Elasticsearch
    Exec
    Event log
    File
    Ganglia
    Gelf
    Generator
    Graphite
    Github
    Heartbeat
    Heroku
    HTTP
    HTTP_Poller
    IRC
    IMAP
    JDBC
    JMX

    Kafka
    Log4J
    Lumberjack
    Meetup
    Pipe
    Puppet_Facter
    Relp
    RSS
    Backspace
    RabbitMQ
    Redis
    Salesforce
    SNMPTrap
    Stdin
    sqlite
    S3
    SQS
    Stomp
    Syslog
    TCP
    Twitter
    Unix
    UDP
    Varnishlog
    WMI
    Web socket
    XMPP
    Zenoss
    ZeroMQ
    Inputs

    View full-size slide

  60. filter {
    json {
    source => "message"
    add_field => [ “my_field", "tek_%{host}" ]
    }
    }

    View full-size slide

  61. filter {
    kv {
    default_keys => [ "from", "[email protected]",
    "to", "[email protected]" ]
    }
    }

    View full-size slide

  62. Accepted publickey for root from 172.14.183.11 port 22 ssh2

    View full-size slide

  63. Accepted publickey for root from 172.14.183.11 port 22 ssh2

    View full-size slide

  64. filter {
    grok {
    match => { "message" => "Accepted
    %{WORD:auth_method} for %{USER:username} from
    %{IP:src_ip} port %{INT:src_port} ssh2" }
    }
    }
    Accepted publickey for root from 172.14.183.11 port 22 ssh2

    View full-size slide

  65. filter {
    grok {
    match => { "message" => "Accepted
    %{WORD:auth_method} for %{USER:username} from
    %{IP:src_ip} port %{INT:src_port} ssh2" }
    }
    }
    Accepted publickey for root from 172.14.183.11 port 22 ssh2

    View full-size slide

  66. http://grokdebug.herokuapp.com/

    View full-size slide

  67. Boundary
    Circus
    CSV
    Cloud watch
    Datadog
    Datadog_Metrics
    Email
    Elastic search
    Exec
    File
    Google BigQuery
    Google Cloud Storage
    Ganglia
    Gelf
    Graphtastic
    Graphite
    Hipchat
    HTTP
    IRC
    InfluxDB
    Juggernaut
    Jira
    Kafka
    Lumberjack
    Librato
    Loggly
    MongoDB
    MetricCatcher
    Nagios
    Null
    OpenTSDB
    Pagerduty
    Pipe
    Riemann
    Redmine
    Rackspace
    RabbitMQ
    Redis
    Riak
    S3
    SQS
    Stomp
    StatsD
    Solr
    SNS
    Syslog
    Stdout
    TCP
    UDP
    WebHDFS
    Websocket
    XMPP
    Outputs
    Zabbix
    ZeroMQ

    View full-size slide

  68. Input -> Filter -> Output

    View full-size slide

  69. Logstash is slow(ish)

    View full-size slide

  70. Elasticsearch

    View full-size slide

  71. 1. Logging
    2. Getting started
    3. The ELK stack
    4. Logs and dashboards
    5. Log management
    6. Supporting services
    7. Conclusion

    View full-size slide

  72. 1. Logging
    2. Getting started
    3. The ELK stack
    4. Logs and dashboards
    5. Log management
    6. Supporting services
    7. Conclusion

    View full-size slide

  73. Asimov’s Law

    View full-size slide

  74. “A robot may not injure a human being
    or, through inaction, allow a human being
    to come to harm.”

    View full-size slide

  75. @mheap’s Law

    View full-size slide

  76. “An application log may not injure a an
    application’s performance or readability”

    View full-size slide

  77. Plan for bursts of data

    View full-size slide

  78. Index management

    View full-size slide

  79. Ship what’s relevant

    View full-size slide

  80. Devs create dashboards

    View full-size slide

  81. Unique request IDs

    View full-size slide

  82. Normalise timezones

    View full-size slide

  83. No really.
    Normalise timezones

    View full-size slide

  84. 1. Logging
    2. Getting started
    3. The ELK stack
    4. Logs and dashboards
    5. Log management
    6. Supporting services
    7. Conclusion

    View full-size slide

  85. 1. Logging
    2. Getting started
    3. The ELK stack
    4. Logs and dashboards
    5. Log management
    6. Supporting services
    7. Conclusion

    View full-size slide

  86. Logging is required

    View full-size slide

  87. Developers are empowered

    View full-size slide

  88. Logging isn’t free

    View full-size slide

  89. “Would you rather fly slowly
    or fly blind?”

    View full-size slide

  90. Thanks!
    I’ve been @mheap, you’ve been awesome.
    Please leave feedback on Joind.in
    https://joind.in/17042

    View full-size slide