$30 off During Our Annual Pro Sale. View Details »

Metrics and an application log: Your new best f...

Metrics and an application log: Your new best friends

Do you remember the time you spent an afternoon putting print statements in your app trying to debug an issue and removed them before shipping the fix, only to add them back in a day later to work on another issue? Wouldn't it be great if those debug statements could just stay in your code forever? Like a little gift that keeps on giving, not just for you, but for everyone else on your team too.

That's what an application log is for! Logs aren't just for when things go wrong. They're for helping you to keep track of what's going on within your application.

We take a look at how you can add helpful messages throughout your codebase and leave them there, even in production! We'll cover common logging strategies, log aggregation and how to efficiently work with your logs to get the data back out again.

We'll also take a look at metrics solutions such as Graphite that can help augment your logs to help work out what was going on by correlating event logs with peaks/drops in other monitoring systems.

Michael Heap

October 13, 2016
Tweet

More Decks by Michael Heap

Other Decks in Technology

Transcript

  1. 1. Logging 2. Getting started 3. The ELK stack 4.

    Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
  2. 1. Logging 2. Getting started 3. The ELK stack 4.

    Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
  3. 1. Logging 2. Getting started 3. The ELK stack 4.

    Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
  4. 1. Logging 2. Getting started 3. The ELK stack 4.

    Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
  5. 1. Logging 2. Getting started 3. The ELK stack 4.

    Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
  6. 1. Logging 2. Getting started 3. The ELK stack 4.

    Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
  7. 1. Logging 2. Getting started 3. The ELK stack 4.

    Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
  8. 1. Logging 2. Getting started 3. The ELK stack 4.

    Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
  9. 1. Logging 2. Getting started 3. The ELK stack 4.

    Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
  10. 1. Logging 2. Getting started 3. The ELK stack 4.

    Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
  11. Cons ✴ Is it semantically correct? ✴ Errors mixed with

    informational logs ✴ It’s not very powerful
  12. <?php require_once 'vendor/autoload.php'; $log = new Monolog\Logger('my-app'); $log->pushHandler(new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog

    \Logger::DEBUG)); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael", $log);
  13. <?php require_once 'vendor/autoload.php'; $log = new Monolog\Logger('my-app'); $log->pushHandler(new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog

    \Logger::DEBUG)); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael", $log);
  14. <?php require_once 'vendor/autoload.php'; $log = new Monolog\Logger('my-app'); $log->pushHandler(new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog

    \Logger::DEBUG)); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael", $log);
  15. <?php require_once 'vendor/autoload.php'; $log = new Monolog\Logger('my-app'); $log->pushHandler(new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog

    \Logger::DEBUG)); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael", $log);
  16. <?php require_once 'vendor/autoload.php'; $log = new Monolog\Logger('my-app'); $log->pushHandler(new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog

    \Logger::DEBUG)); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael", $log);
  17. $log = new Monolog\Logger('my-app'); $streamHandler = new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog \Logger::DEBUG);

    $fcHandler = new Monolog\Handler\FingersCrossedHandler($streamHandler, Monolog \Logger::ERROR); $log->pushHandler($fcHandler); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael", $log);
  18. $log = new Monolog\Logger('my-app'); $streamHandler = new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog \Logger::DEBUG);

    $fcHandler = new Monolog\Handler\FingersCrossedHandler($streamHandler, Monolog \Logger::ERROR); $log->pushHandler($fcHandler); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael", $log);
  19. $log = new Monolog\Logger('my-app'); $streamHandler = new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog \Logger::DEBUG);

    $fcHandler = new Monolog\Handler\FingersCrossedHandler($streamHandler, Monolog \Logger::ERROR); $log->pushHandler($fcHandler); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); return $c; } echo countConsonants("Michael", $log);
  20. $log = new Monolog\Logger('my-app'); $streamHandler = new Monolog\Handler\StreamHandler('/tmp/app.log', Monolog \Logger::DEBUG);

    $fcHandler = new Monolog\Handler\FingersCrossedHandler($streamHandler, Monolog \Logger::ERROR); $log->pushHandler($fcHandler); function countConsonants($str, $log){ $c = strlen(str_replace(['a','e','i','o','u'],'', $str)); $log->info("Consonants in {$str}: {$c}"); $log->error("Something bad happened"); return $c; } echo countConsonants("Michael", $log);
  21. Pros ✴ It’s an object! Dependency injection FTW ✴ Supports

    multiple log writers ✴ Log level support
  22. 0. Emergency System is unusable 1. Alert Should be corrected

    immediately 2. Critical Critical conditions 3. Error Error conditions 4. Warning May indicate that an error will occur if action is not taken. 5. Notice Events that are unusual, but not error conditions. 6. Informational Normal operational messages that require no action. 7. Debug Information useful to developers for debugging the application. Syslog (RFC 5424)
  23. 0. Emergency System is unusable 1. Alert Should be corrected

    immediately 2. Critical Critical conditions 3. Error Error conditions 4. Warning May indicate that an error will occur if action is not taken. 5. Notice Events that are unusual, but not error conditions. 6. Informational Normal operational messages that require no action. 7. Debug Information useful to developers for debugging the application. PSR3
  24. 1. Logging 2. Getting started 3. The ELK stack 4.

    Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
  25. Beats CouchDB_Changes Drupal_DBLog Elasticsearch Exec Event log File Ganglia Gelf

    Generator Graphite Github Heartbeat Heroku HTTP HTTP_Poller IRC IMAP JDBC JMX
 Kafka Log4J Lumberjack Meetup Pipe Puppet_Facter Relp RSS Backspace RabbitMQ Redis Salesforce SNMPTrap Stdin sqlite S3 SQS Stomp Syslog TCP Twitter Unix UDP Varnishlog WMI Web socket XMPP Zenoss ZeroMQ Inputs
  26. filter { json { source => "message" add_field => [

    “my_field", “mheap_%{host}” ] } }
  27. filter { grok { match => { "message" => "Accepted

    %{WORD:auth_method} for %{USER:username} from %{IP:src_ip} port %{INT:src_port} ssh2" } } } Accepted publickey for root from 172.14.183.11 port 22 ssh2
  28. filter { grok { match => { "message" => "Accepted

    %{WORD:auth_method} for %{USER:username} from %{IP:src_ip} port %{INT:src_port} ssh2" } } } Accepted publickey for root from 172.14.183.11 port 22 ssh2
  29. Boundary Circus CSV Cloud watch Datadog Datadog_Metrics Email Elastic search

    Exec File Google BigQuery Google Cloud Storage Ganglia Gelf Graphtastic Graphite Hipchat HTTP IRC InfluxDB Juggernaut Jira Kafka Lumberjack Librato Loggly MongoDB MetricCatcher Nagios Null OpenTSDB Pagerduty Pipe Riemann Redmine Rackspace RabbitMQ Redis Riak S3 SQS Stomp StatsD Solr SNS Syslog Stdout TCP UDP WebHDFS Websocket XMPP Outputs Zabbix ZeroMQ
  30. 1. Logging 2. Getting started 3. The ELK stack 4.

    Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
  31. 1. Logging 2. Getting started 3. The ELK stack 4.

    Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
  32. “A robot may not injure a human being or, through

    inaction, allow a human being to come to harm.”
  33. 1. Logging 2. Getting started 3. The ELK stack 4.

    Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion
  34. 1. Logging 2. Getting started 3. The ELK stack 4.

    Logs and dashboards 5. Log management 6. Supporting services 7. Conclusion