How to get to zero unhandled exceptions in production

How to get to zero unhandled exceptions in production Radoslav
Stankov 07/09/2019

Radoslav Stankov @rstankov blog.rstankov.com github.com/rstankov  twitter.com/rstankov

  ! Product Hunt Engineering Team "  

Happy Friday # $ Fix bugs % Goodies features &
Pay technical dept ' Catchup on projects ( Fix exceptions

“Have process around exceptions.”   ) Tip  

Back to basics

  http://exceptionalruby.com/ 

def perform do_something rescue end

def perform do_something rescue SpecificError end

def perform do_something rescue SpecificError # NOTE(rstankov): Reason to return
nil nil end Your name, not mine *

def perform do_something rescue Timeout::Error # NOTE(rstankov): WiFi Sucks nil
end

“Be explicit around the exceptions. Handle speciﬁc errors and have
explanations of why they happen.”   ) Tip  

Monitoring

Raven.configure do |config| # Note(rstankov): Exclude unactionable errors config.excluded_exceptions =
[ 'Rack::Timeout::RequestExpiryError', 'Rack::Timeout::RequestTimeoutException', 'ActionController::RoutingError', 'ActionController::InvalidAuthenticityToken', 'ActionDispatch::ParamsParser::ParseError', 'Sidekiq::Shutdown', ] end

ArgumentError: invalid byte sequence in UTF-8 (

# NOTE(rstankov): Fix invalid byte sequence in UTF-8. More info:
# - https://robots.thoughtbot.com/fight-back-utf-8-invalid-byte-sequences module Handle::InvalidByteSequence extend self def call(string) return if string.nil? string.encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '' ) end end

“Reduce noise. See only exceptional errors in your tracker.”  
) Tip  

account.subscription.status

account.subscription&.status

✅ Check for other accounts without a subscription ✅ Find
out why some accounts don't have a subscription ✅ Fix the root problem ✅ Add missing subscriptions to accounts , Steps to ﬁx -

if account.ship_subscription.blank? Raven.capture_message "Missing ship subscription", extra: { account_id: account.id
} return :no_subscription end

“Don't hide exceptions. Fix root causes.”   ) Tip  

  https://graphql.org/ 

Which query causes this issue? .

class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:
Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end

“Invest in your monitoring and exception traceability. When you have
a hard time racing an exception. Ask yourself - what more information I need? . Then add it.”   ) Tip  

  https://sidekiq.org/ 

Use action Achievements::Job Achievement  (unique)

class Achievements::Job < ApplicationJob def perform(achievement, user) Achievements::Create.call achievement, user
end end

Use action Achievements::Job Achievement  (unique) Use action Achievements::Job Achievement  (unique)

Use action Achievements::Job Achievement  (unique) Use action Achievements::Job Achievement  (unique)
(

Use action Achievements::Job Achievement  (unique) Use action Achievements::Job PG::UniqueViolation (

class Achievements::Job < ApplicationJob def perform(achievement, user) Achievements::Create.call achievement, user
end end

class Achievements::Job < ApplicationJob def perform(achievement, user) Handle::RaceCondition.call do Achievements::Create.call
achievement, user end end end

module Handle::RaceCondition extend self UNIQUE_ACTIVE_RECORD_ERROR = 'has already been taken'.freeze
def call retries ||= 2 yield rescue ActiveRecord::RecordNotUnique, PG::UniqueViolation retries -= 1 raise unless retries.nonzero? retry rescue ActiveRecord::RecordInvalid => e raise unless e.message.include? UNIQUE_ACTIVE_RECORD_ERROR retries -= 1 raise unless retries.nonzero? retry end end

class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event) end
end

Errno::ECONNRESET (

class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event) end
end

class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)  rescue
Errno::ECONNRESET  retry_job end end

Errno::ECONNRESET, EOFError  retry_job end end

Errno::ECONNRESET, EOFError, Timeout::Error  retry_job end end

Errno::ECONNRESET, EOFError, Timeout::Error, ... /  retry_job end end

module Handle::NetworkErrors extend self ERRORS = [ EOFError, Errno::ECONNRESET, Errno::EINVAL,
Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Net::ProtocolError, Timeout::Error, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end

class Notifications::Deliver < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)  rescue
Handle::NetworkErrors retry_job end end

class Notifications::Deliver::Job < ApplicationJob retry_on(*::Handle::NetworkErrors::ERRORS, wait: 2.minutes, attempts: 5) queue_as
:notifications def perform(event) Notifications::Deliver.call(event) end end

class Notifications::Deliver::Job < ApplicationJob include Handle::Job::NetworkErrors queue_as :notifications def perform(event)
Notifications::Deliver.call(event) end end

module Handle::Job::NetworkErrors def self.included(job) job.retry_on(*::Handle::NetworkErrors::ERRORS, wait: 2.minutes, attempts: 5) end
end

module Handle::NetworkErrors extend self ERRORS = [ # ... Faraday::ConnectionFailed,
Faraday::TimeoutError, RestClient::BadGateway, RestClient::BadRequest, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end

“Have tooling around handling common exceptions. Make it a no-brainer
to process everyday errors.”   ) Tip  

) Have process around exceptions.  ) Be explicit around the
exceptions  ) Reduce noise  ) Don't hide exceptions  ) Invest in your monitoring  ) Have tooling around handling common exceptions   1 Recap  

Thanks 2

https://speakerdeck.com/rstankov

How to get to zero unhandled exceptions in production

How to get to zero unhandled exceptions in production

More Decks by Radoslav Stankov

Featured

Transcript