Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to get to zero unhandled exceptions in production

Radoslav Stankov
September 06, 2019
360

How to get to zero unhandled exceptions in production

Practical tips and tricks about how to deal with exceptions in Ruby on Rails applications.

Video of the talk 👉https://www.youtube.com/watch?v=btUnSR-NGV0

Radoslav Stankov

September 06, 2019
Tweet

Transcript

  1. Happy Friday # $ Fix bugs % Goodies features &

    Pay technical dept ' Catchup on projects ( Fix exceptions
  2. Happy Friday # $ Fix bugs % Goodies features &

    Pay technical dept ' Catchup on projects ( Fix exceptions
  3. “Be explicit around the exceptions. Handle specific errors and have

    explanations of why they happen.” 
 ) Tip 

  4. Raven.configure do |config| # Note(rstankov): Exclude unactionable errors config.excluded_exceptions =

    [ 'Rack::Timeout::RequestExpiryError', 'Rack::Timeout::RequestTimeoutException', 'ActionController::RoutingError', 'ActionController::InvalidAuthenticityToken', 'ActionDispatch::ParamsParser::ParseError', 'Sidekiq::Shutdown', ] end
  5. # NOTE(rstankov): Fix invalid byte sequence in UTF-8. More info:

    # - https://robots.thoughtbot.com/fight-back-utf-8-invalid-byte-sequences module Handle::InvalidByteSequence extend self def call(string) return if string.nil? string.encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '' ) end end
  6. ✅ Check for other accounts without a subscription ✅ Find

    out why some accounts don't have a subscription ✅ Fix the root problem ✅ Add missing subscriptions to accounts , Steps to fix -
  7. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
  8. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
  9. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
  10. “Invest in your monitoring and exception traceability. When you have

    a hard time racing an exception. Ask yourself - what more information I need? . Then add it.” 
 ) Tip 

  11. module Handle::RaceCondition extend self UNIQUE_ACTIVE_RECORD_ERROR = 'has already been taken'.freeze

    def call retries ||= 2 yield rescue ActiveRecord::RecordNotUnique, PG::UniqueViolation retries -= 1 raise unless retries.nonzero? retry rescue ActiveRecord::RecordInvalid => e raise unless e.message.include? UNIQUE_ACTIVE_RECORD_ERROR retries -= 1 raise unless retries.nonzero? retry end end
  12. module Handle::NetworkErrors extend self ERRORS = [ EOFError, Errno::ECONNRESET, Errno::EINVAL,

    Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Net::ProtocolError, Timeout::Error, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
  13. 0

  14. module Handle::NetworkErrors extend self ERRORS = [ # ... Faraday::ConnectionFailed,

    Faraday::TimeoutError, RestClient::BadGateway, RestClient::BadRequest, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
  15. ) Have process around exceptions.
 ) Be explicit around the

    exceptions
 ) Reduce noise
 ) Don't hide exceptions
 ) Invest in your monitoring
 ) Have tooling around handling common exceptions 
 1 Recap