How to get to zero unhandled exceptions in production

7a0e72a6f55811246bb5d9a946fd2e49?s=47 Radoslav Stankov
September 06, 2019
160

How to get to zero unhandled exceptions in production

Practical tips and tricks about how to deal with exceptions in Ruby on Rails applications.

Video of the talk 👉https://www.youtube.com/watch?v=btUnSR-NGV0

7a0e72a6f55811246bb5d9a946fd2e49?s=128

Radoslav Stankov

September 06, 2019
Tweet

Transcript

  1. 3.
  2. 4.
  3. 5.
  4. 6.
  5. 7.
  6. 8.
  7. 10.
  8. 11.
  9. 12.
  10. 13.
  11. 14.
  12. 15.
  13. 16.
  14. 17.

    Happy Friday # $ Fix bugs % Goodies features &

    Pay technical dept ' Catchup on projects ( Fix exceptions
  15. 18.

    Happy Friday # $ Fix bugs % Goodies features &

    Pay technical dept ' Catchup on projects ( Fix exceptions
  16. 19.
  17. 24.
  18. 28.

    “Be explicit around the exceptions. Handle specific errors and have

    explanations of why they happen.” 
 ) Tip 

  19. 30.
  20. 31.
  21. 32.
  22. 33.
  23. 34.

    Raven.configure do |config| # Note(rstankov): Exclude unactionable errors config.excluded_exceptions =

    [ 'Rack::Timeout::RequestExpiryError', 'Rack::Timeout::RequestTimeoutException', 'ActionController::RoutingError', 'ActionController::InvalidAuthenticityToken', 'ActionDispatch::ParamsParser::ParseError', 'Sidekiq::Shutdown', ] end
  24. 35.
  25. 37.
  26. 38.
  27. 39.

    # NOTE(rstankov): Fix invalid byte sequence in UTF-8. More info:

    # - https://robots.thoughtbot.com/fight-back-utf-8-invalid-byte-sequences module Handle::InvalidByteSequence extend self def call(string) return if string.nil? string.encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '' ) end end
  28. 41.
  29. 42.
  30. 45.

    ✅ Check for other accounts without a subscription ✅ Find

    out why some accounts don't have a subscription ✅ Fix the root problem ✅ Add missing subscriptions to accounts , Steps to fix -
  31. 46.
  32. 47.
  33. 51.
  34. 52.
  35. 54.

    class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
  36. 55.

    class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
  37. 56.

    class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
  38. 57.
  39. 58.

    “Invest in your monitoring and exception traceability. When you have

    a hard time racing an exception. Ask yourself - what more information I need? . Then add it.” 
 ) Tip 

  40. 60.
  41. 68.

    module Handle::RaceCondition extend self UNIQUE_ACTIVE_RECORD_ERROR = 'has already been taken'.freeze

    def call retries ||= 2 yield rescue ActiveRecord::RecordNotUnique, PG::UniqueViolation retries -= 1 raise unless retries.nonzero? retry rescue ActiveRecord::RecordInvalid => e raise unless e.message.include? UNIQUE_ACTIVE_RECORD_ERROR retries -= 1 raise unless retries.nonzero? retry end end
  42. 71.
  43. 77.

    module Handle::NetworkErrors extend self ERRORS = [ EOFError, Errno::ECONNRESET, Errno::EINVAL,

    Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Net::ProtocolError, Timeout::Error, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
  44. 82.
  45. 83.

    0

  46. 84.

    module Handle::NetworkErrors extend self ERRORS = [ # ... Faraday::ConnectionFailed,

    Faraday::TimeoutError, RestClient::BadGateway, RestClient::BadRequest, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
  47. 85.
  48. 86.
  49. 87.
  50. 88.
  51. 89.
  52. 90.

    ) Have process around exceptions.
 ) Be explicit around the

    exceptions
 ) Reduce noise
 ) Don't hide exceptions
 ) Invest in your monitoring
 ) Have tooling around handling common exceptions 
 1 Recap 

  53. 91.
  54. 92.
  55. 93.