How to get to zero unhandled exceptions in production

Practical tips and tricks about how to deal with exceptions in Ruby on Rails applications.

Video of the talk 👉https://www.youtube.com/watch?v=btUnSR-NGV0

Transcript

1. How to get to zero unhandled exceptions in production Radoslav Stankov 07/09/2019
2. Radoslav Stankov @rstankov blog.rstankov.com github.com/rstankov
 twitter.com/rstankov
9. 
 ! Product Hunt Engineering Team " 

17. Happy Friday # $ Fix bugs % Goodies features & Pay technical dept ' Catchup on projects ( Fix exceptions
18. Happy Friday # $ Fix bugs % Goodies features & Pay technical dept ' Catchup on projects ( Fix exceptions
20. “Have process around exceptions.” 
 ) Tip 

21. Back to basics
22. 
 http://exceptionalruby.com/

23. def perform do_something rescue end
25. def perform do_something rescue SpecificError end
26. def perform do_something rescue SpecificError # NOTE(rstankov): Reason to return nil nil end Your name, not mine *
27. def perform do_something rescue Timeout::Error # NOTE(rstankov): WiFi Sucks nil end
28. “Be explicit around the exceptions. Handle specific errors and have explanations of why they happen.” 
 ) Tip 

29. Monitoring
34. Raven.configure do |config| # Note(rstankov): Exclude unactionable errors config.excluded_exceptions = [ 'Rack::Timeout::RequestExpiryError', 'Rack::Timeout::RequestTimeoutException', 'ActionController::RoutingError', 'ActionController::InvalidAuthenticityToken', 'ActionDispatch::ParamsParser::ParseError', 'Sidekiq::Shutdown', ] end
36. ArgumentError: invalid byte sequence in UTF-8 (
39. # NOTE(rstankov): Fix invalid byte sequence in UTF-8. More info: # - https://robots.thoughtbot.com/fight-back-utf-8-invalid-byte-sequences module Handle::InvalidByteSequence extend self def call(string) return if string.nil? string.encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '' ) end end
40. “Reduce noise. See only exceptional errors in your tracker.” 
 ) Tip 

43. account.subscription.status
44. account.subscription&.status
45. ✅ Check for other accounts without a subscription ✅ Find out why some accounts don't have a subscription ✅ Fix the root problem ✅ Add missing subscriptions to accounts , Steps to fix -
48. if account.ship_subscription.blank? Raven.capture_message "Missing ship subscription", extra: { account_id: account.id } return :no_subscription end
49. “Don't hide exceptions. Fix root causes.” 
 ) Tip 

50. 
 https://graphql.org/

53. Which query causes this issue? .
54. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json: Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
55. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json: Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
56. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json: Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
58. “Invest in your monitoring and exception traceability. When you have a hard time racing an exception. Ask yourself - what more information I need? . Then add it.” 
 ) Tip 

59. 
 https://sidekiq.org/

61. Use action Achievements::Job Achievement
 (unique)
62. class Achievements::Job < ApplicationJob def perform(achievement, user) Achievements::Create.call achievement, user end end
63. Use action Achievements::Job Achievement
 (unique) Use action Achievements::Job Achievement
 (unique)
64. Use action Achievements::Job Achievement
 (unique) Use action Achievements::Job Achievement
 (unique) (
65. Use action Achievements::Job Achievement
 (unique) Use action Achievements::Job PG::UniqueViolation (
66. class Achievements::Job < ApplicationJob def perform(achievement, user) Achievements::Create.call achievement, user end end
67. class Achievements::Job < ApplicationJob def perform(achievement, user) Handle::RaceCondition.call do Achievements::Create.call achievement, user end end end
68. module Handle::RaceCondition extend self UNIQUE_ACTIVE_RECORD_ERROR = 'has already been taken'.freeze def call retries ||= 2 yield rescue ActiveRecord::RecordNotUnique, PG::UniqueViolation retries -= 1 raise unless retries.nonzero? retry rescue ActiveRecord::RecordInvalid => e raise unless e.message.include? UNIQUE_ACTIVE_RECORD_ERROR retries -= 1 raise unless retries.nonzero? retry end end
69. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event) end end
70. Errno::ECONNRESET (
72. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event) end end
73. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue Errno::ECONNRESET
 retry_job end end
74. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue Errno::ECONNRESET, EOFError
 retry_job end end
75. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue Errno::ECONNRESET, EOFError, Timeout::Error
 retry_job end end
76. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue Errno::ECONNRESET, EOFError, Timeout::Error, ... /
 retry_job end end
77. module Handle::NetworkErrors extend self ERRORS = [ EOFError, Errno::ECONNRESET, Errno::EINVAL, Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Net::ProtocolError, Timeout::Error, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
78. class Notifications::Deliver < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue Handle::NetworkErrors retry_job end end
79. class Notifications::Deliver::Job < ApplicationJob retry_on(*::Handle::NetworkErrors::ERRORS, wait: 2.minutes, attempts: 5) queue_as :notifications def perform(event) Notifications::Deliver.call(event) end end
80. class Notifications::Deliver::Job < ApplicationJob include Handle::Job::NetworkErrors queue_as :notifications def perform(event) Notifications::Deliver.call(event) end end
81. module Handle::Job::NetworkErrors def self.included(job) job.retry_on(*::Handle::NetworkErrors::ERRORS, wait: 2.minutes, attempts: 5) end end
84. module Handle::NetworkErrors extend self ERRORS = [ # ... Faraday::ConnectionFailed, Faraday::TimeoutError, RestClient::BadGateway, RestClient::BadRequest, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
85. “Have tooling around handling common exceptions. Make it a no-brainer to process everyday errors.” 
 ) Tip 

88. Recap
90. ) Have process around exceptions.
 ) Be explicit around the exceptions
 ) Reduce noise
 ) Don't hide exceptions
 ) Invest in your monitoring
 ) Have tooling around handling common exceptions 
 1 Recap 

93. Thanks 2
94. https://speakerdeck.com/rstankov