How to get to zero unhandled exceptions in production

7a0e72a6f55811246bb5d9a946fd2e49?s=47 Radoslav Stankov
September 06, 2019
190

How to get to zero unhandled exceptions in production

Practical tips and tricks about how to deal with exceptions in Ruby on Rails applications.

Video of the talk 👉https://www.youtube.com/watch?v=btUnSR-NGV0

7a0e72a6f55811246bb5d9a946fd2e49?s=128

Radoslav Stankov

September 06, 2019
Tweet

Transcript

  1. How to get to zero unhandled exceptions in production Radoslav

    Stankov 07/09/2019
  2. Radoslav Stankov @rstankov blog.rstankov.com github.com/rstankov
 twitter.com/rstankov

  3. None
  4. None
  5. None
  6. None
  7. None
  8. None
  9. 
 ! Product Hunt Engineering Team " 


  10. None
  11. None
  12. None
  13. None
  14. None
  15. None
  16. None
  17. Happy Friday # $ Fix bugs % Goodies features &

    Pay technical dept ' Catchup on projects ( Fix exceptions
  18. Happy Friday # $ Fix bugs % Goodies features &

    Pay technical dept ' Catchup on projects ( Fix exceptions
  19. None
  20. “Have process around exceptions.” 
 ) Tip 


  21. Back to basics

  22. 
 http://exceptionalruby.com/


  23. def perform do_something rescue end

  24. None
  25. def perform do_something rescue SpecificError end

  26. def perform do_something rescue SpecificError # NOTE(rstankov): Reason to return

    nil nil end Your name, not mine *
  27. def perform do_something rescue Timeout::Error # NOTE(rstankov): WiFi Sucks nil

    end
  28. “Be explicit around the exceptions. Handle specific errors and have

    explanations of why they happen.” 
 ) Tip 

  29. Monitoring

  30. None
  31. None
  32. None
  33. None
  34. Raven.configure do |config| # Note(rstankov): Exclude unactionable errors config.excluded_exceptions =

    [ 'Rack::Timeout::RequestExpiryError', 'Rack::Timeout::RequestTimeoutException', 'ActionController::RoutingError', 'ActionController::InvalidAuthenticityToken', 'ActionDispatch::ParamsParser::ParseError', 'Sidekiq::Shutdown', ] end
  35. None
  36. ArgumentError: invalid byte sequence in UTF-8 (

  37. None
  38. None
  39. # NOTE(rstankov): Fix invalid byte sequence in UTF-8. More info:

    # - https://robots.thoughtbot.com/fight-back-utf-8-invalid-byte-sequences module Handle::InvalidByteSequence extend self def call(string) return if string.nil? string.encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '' ) end end
  40. “Reduce noise. See only exceptional errors in your tracker.” 


    ) Tip 

  41. None
  42. None
  43. account.subscription.status

  44. account.subscription&.status

  45. ✅ Check for other accounts without a subscription ✅ Find

    out why some accounts don't have a subscription ✅ Fix the root problem ✅ Add missing subscriptions to accounts , Steps to fix -
  46. None
  47. None
  48. if account.ship_subscription.blank? Raven.capture_message "Missing ship subscription", extra: { account_id: account.id

    } return :no_subscription end
  49. “Don't hide exceptions. Fix root causes.” 
 ) Tip 


  50. 
 https://graphql.org/


  51. None
  52. None
  53. Which query causes this issue? .

  54. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
  55. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
  56. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
  57. None
  58. “Invest in your monitoring and exception traceability. When you have

    a hard time racing an exception. Ask yourself - what more information I need? . Then add it.” 
 ) Tip 

  59. 
 https://sidekiq.org/


  60. None
  61. Use action Achievements::Job Achievement
 (unique)

  62. class Achievements::Job < ApplicationJob def perform(achievement, user) Achievements::Create.call achievement, user

    end end
  63. Use action Achievements::Job Achievement
 (unique) Use action Achievements::Job Achievement
 (unique)

  64. Use action Achievements::Job Achievement
 (unique) Use action Achievements::Job Achievement
 (unique)

    (
  65. Use action Achievements::Job Achievement
 (unique) Use action Achievements::Job PG::UniqueViolation (

  66. class Achievements::Job < ApplicationJob def perform(achievement, user) Achievements::Create.call achievement, user

    end end
  67. class Achievements::Job < ApplicationJob def perform(achievement, user) Handle::RaceCondition.call do Achievements::Create.call

    achievement, user end end end
  68. module Handle::RaceCondition extend self UNIQUE_ACTIVE_RECORD_ERROR = 'has already been taken'.freeze

    def call retries ||= 2 yield rescue ActiveRecord::RecordNotUnique, PG::UniqueViolation retries -= 1 raise unless retries.nonzero? retry rescue ActiveRecord::RecordInvalid => e raise unless e.message.include? UNIQUE_ACTIVE_RECORD_ERROR retries -= 1 raise unless retries.nonzero? retry end end
  69. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event) end

    end
  70. Errno::ECONNRESET (

  71. None
  72. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event) end

    end
  73. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue

    Errno::ECONNRESET
 retry_job end end
  74. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue

    Errno::ECONNRESET, EOFError
 retry_job end end
  75. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue

    Errno::ECONNRESET, EOFError, Timeout::Error
 retry_job end end
  76. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue

    Errno::ECONNRESET, EOFError, Timeout::Error, ... /
 retry_job end end
  77. module Handle::NetworkErrors extend self ERRORS = [ EOFError, Errno::ECONNRESET, Errno::EINVAL,

    Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Net::ProtocolError, Timeout::Error, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
  78. class Notifications::Deliver < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue

    Handle::NetworkErrors retry_job end end
  79. class Notifications::Deliver::Job < ApplicationJob retry_on(*::Handle::NetworkErrors::ERRORS, wait: 2.minutes, attempts: 5) queue_as

    :notifications def perform(event) Notifications::Deliver.call(event) end end
  80. class Notifications::Deliver::Job < ApplicationJob include Handle::Job::NetworkErrors queue_as :notifications def perform(event)

    Notifications::Deliver.call(event) end end
  81. module Handle::Job::NetworkErrors def self.included(job) job.retry_on(*::Handle::NetworkErrors::ERRORS, wait: 2.minutes, attempts: 5) end

    end
  82. None
  83. 0

  84. module Handle::NetworkErrors extend self ERRORS = [ # ... Faraday::ConnectionFailed,

    Faraday::TimeoutError, RestClient::BadGateway, RestClient::BadRequest, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
  85. “Have tooling around handling common exceptions. Make it a no-brainer

    to process everyday errors.” 
 ) Tip 

  86. None
  87. None
  88. Recap

  89. None
  90. ) Have process around exceptions.
 ) Be explicit around the

    exceptions
 ) Reduce noise
 ) Don't hide exceptions
 ) Invest in your monitoring
 ) Have tooling around handling common exceptions 
 1 Recap 

  91. None
  92. None
  93. Thanks 2

  94. https://speakerdeck.com/rstankov