Expect the un-expected: How to handle errors gracefully

Expect the un-expected: How to handle errors gracefully

Even though you tested your application perfectly errors and bugs will still happen in production, especially if other services or databases go down or are under high load. Thus it is very important to see errors happing and to be able to react to them quickly. In this talk I’ll introduce you to efficient ways for monitoring and logging for errors and show how you can handle them if they happen, covering deployment strategies, using intelligent circuit breakers and gracefully reducing functionality.

Ded87c77266697ee6981c2277bb97633?s=128

Bastian Hofmann

April 08, 2017
Tweet

Transcript

  1. 2.
  2. 3.
  3. 15.
  4. 27.
  5. 30.

    bool error_log ( string $message [, int $message_type = 0

    [, string $destination [, string $extra_headers ]]] )
  6. 32.
  7. 33.
  8. 35.
  9. 40.

    set_error_handler( function($errno, $msg, $file, $line) { $e = new \Exception();

    $error = [ 'type' => $errno, 'message' => $msg, 'file' => $file, 'line' => $line, 'trace' => $e->getTrace(), ]; error_log(json_encode($error)); return true; });
  10. 42.

    set_error_handler( function($errno, $msg, $file, $line) { switch ($errno) { case

    E_RECOVERABLE_ERROR: case E_USER_ERROR: throw new \ErrorException($msg, null, $errno, $file, $line); case E_WARNING: case E_USER_WARNING: case E_CORE_WARNING: case E_COMPILE_WARNING: throw new WarningException($msg, null, $errno, $file, $line); case E_NOTICE: case E_USER_NOTICE: throw new NoticeException($msg, null, $errno, $file, $line); case E_STRICT: throw new StrictException($msg, null, $errno, $file, $line); } return true; }); $a = []; try { $b = $a['doesNotExist']; } catch (NoticeException $e) { }
  11. 50.

    web server http service http service http service http service

    create unique trace_id for request user request trace_id trace_id trace_id trace_id log log log log log
  12. 53.
  13. 54.
  14. 55.
  15. 67.

    input { file { type => "error" path => [

    "/var/logs/php/*.log" ] add_field => [ "severity", "error" ] } }
  16. 71.

    input {
 rabbitmq {
 queue => "logs"
 host => "amqp.host"


    exchange => "ls_exchange"
 exclusive => false
 }
 }
  17. 72.

    output {
 elasticsearch {
 embedded => false
 bind_host => "localhost"


    bind_port => "9305"
 host => "localhost"
 cluster => "logs"
 }
 }
  18. 74.
  19. 77.
  20. 80.
  21. 81.
  22. 83.
  23. 89.
  24. 90.
  25. 91.
  26. 92.
  27. 93.
  28. 96.
  29. 98.
  30. 99.
  31. 100.
  32. 101.
  33. 102.
  34. 103.
  35. 104.
  36. 115.
  37. 116.
  38. 118.
  39. 124.
  40. 129.
  41. 131.
  42. 146.

    Service A Service B Error Circuit Breaker Status: -> open

    Error rate: > threshold Test if still failing
  43. 147.

    Service A Service B 200 OK Circuit Breaker Status: ->

    close Error rate: 0 Test if still failing
  44. 152.
  45. 154.
  46. 155.
  47. 159.
  48. 161.
  49. 166.