Zero Exceptions in Production

Zero Exceptions in Production

Presented at Balkan Ruby 2018

In the talk, I'm explaining how to categorize exceptions and their level of impact. Present use cases and code samples of common problems in a Rails application. How to make sure your background workers run without issues and how to debug exceptions.

Links:

- 
http://exceptionalruby.com/
- https://sentry.io/
- 
https://graphql.org/
- 
https://sidekiq.org/

7a0e72a6f55811246bb5d9a946fd2e49?s=128

Radoslav Stankov

May 24, 2018
Tweet

Transcript

  1. How to get to zero unhandled exceptions in production Radoslav

    Stankov 26/05/2018
  2. Radoslav Stankov @rstankov blog.rstankov.com github.com/rstankov
 twitter.com/rstankov

  3. None
  4. None
  5. None
  6. None
  7. None
  8. None
  9. None
  10. None
  11. None
  12. None
  13. Happy Friday ! " Fix bugs # Goodies features $

    Pay technical dept % Catchup on projects & Fix exceptions
  14. Happy Friday ! " Fix bugs # Goodies features $

    Pay technical dept % Catchup on projects & Fix exceptions
  15. None
  16. Back to basics

  17. 
 http://exceptionalruby.com/


  18. def perform do_something rescue end

  19. def perform do_something rescue end

  20. def perform do_something rescue end

  21. def perform do_something rescue SpecificError end

  22. def perform do_something rescue SpecificError # NOTE(rstankov): Reason to return

    nil nil end
  23. def perform do_something rescue SpecificError # NOTE(rstankov): Reason to return

    nil nil end Your name, not mine '
  24. def perform do_something rescue SpecificError # NOTE(rstankov): Reason to return

    nil nil end
  25. def perform do_something rescue NetworkErrors # NOTE(rstankov): WiFi Sucks nil

    end
  26. module NetworkErrors extend self ERRORS = [ EOFError, Errno::ECONNRESET, Errno::EINVAL,

    Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Net::ProtocolError, Timeout::Error, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
  27. – saw it in a tweet once “A mature Rails

    project is one, that uses every popular (or not so popular) Ruby networking gem.” (
  28. – saw it in a tweet once “A mature Rails

    project is one, that uses every popular (or not so popular) Ruby networking gem.” (
  29. module NetworkErrors extend self ERRORS = [ # ... Faraday::ConnectionFailed,

    Faraday::TimeoutError, RestClient::BadGateway, RestClient::BadRequest, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
  30. if user.ship_subscription.blank? Raven.capture_message "Missing ship subscription", extra: { user_id: user.id

    } return :no_subscription end
  31. Monitoring

  32. None
  33. None
  34. None
  35. None
  36. Raven.configure do |config| # Note(rstankov): Exclude un actionable errors config.excluded_exceptions

    = [ 'Rack::Timeout::RequestExpiryError', 'Rack::Timeout::RequestTimeoutException', 'ActionController::RoutingError', 'ActionController::InvalidAuthenticityToken', 'ActionDispatch::ParamsParser::ParseError', 'Sidekiq::Shutdown', ] end
  37. None
  38. None
  39. None
  40. None
  41. account.subscription.status

  42. account.subscription&.status

  43. account.subscription&.status

  44. Steps to fix )

  45. 0. Check for other accounts without subscription Steps to fix

    )
  46. 0. Check for other accounts without subscription 1. Find out

    why account doesn't have a subscription Steps to fix )
  47. 0. Check for other accounts without subscription 1. Find out

    why account doesn't have a subscription 2. Fix the problem Steps to fix )
  48. 0. Check for other accounts without subscription 1. Find out

    why account doesn't have a subscription 2. Fix the problem 3. Add missing subscriptions to this accounts Steps to fix )
  49. None
  50. None
  51. – Brian Kernighan and Rob Pike, The Practice of Programming

    “Exceptions shouldn't be expected” Use exceptions only for exceptional situations. […] Exceptions are often overused. Because they distort the flow of control, they can lead to convoluted constructions that are prone to bugs. It is hardly exceptional to fail to open a file; generating an exception in this case strikes us as over-engineering.
  52. ArgumentError: invalid byte sequence in UTF-8

  53. None
  54. # NOTE(rstankov): Fix invalid byte sequence in UTF-8. More info:

    # - https://robots.thoughtbot.com/fight-back-utf-8-invalid-byte-sequences module Utf8Sanitize extend self def call(string) return if string.nil? string.encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '' ) end end
  55. 
 https://graphql.org/


  56. None
  57. None
  58. Which query causes this issue? *

  59. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(e) if Rails.env.development? logger.error e.message logger.error e.backtrace.join("\n") render json: { message: e.message, backtrace: e.backtrace }, status: 500 elsif Rails.env.test? p e.message # rubocop:disable Rails/Output p e.backtrace # rubocop:disable Rails/Output else Raven.capture_exception(e, extra: { query: query }) end end end
  60. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(e) if Rails.env.development? logger.error e.message logger.error e.backtrace.join("\n") render json: { message: e.message, backtrace: e.backtrace }, status: 500 elsif Rails.env.test? p e.message # rubocop:disable Rails/Output p e.backtrace # rubocop:disable Rails/Output else Raven.capture_exception(e, extra: { query: query }) end end end
  61. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(e) if Rails.env.development? logger.error e.message logger.error e.backtrace.join("\n") render json: { message: e.message, backtrace: e.backtrace }, status: 500 elsif Rails.env.test? p e.message # rubocop:disable Rails/Output p e.backtrace # rubocop:disable Rails/Output else Raven.capture_exception(e, extra: { query: query }) end end end
  62. None
  63. None
  64. Graph::Query = GraphQL::ObjectType.define do name 'Query' field :post do type

    !Graph::Types::PostType argument :id, !types.ID resolve ->(_obj, inputs, _ctx) { Post.find(inputs[:id]) } end end
  65. Graph::Query = GraphQL::ObjectType.define do name 'Query' field :post do type

    Graph::Types::PostType argument :id, !types.ID resolve ->(_obj, inputs, _ctx) { Post.find_by(id: inputs[:id]) } end end
  66. class Graph::Mutations::CollectionAddPost < Graph::Resolvers::Mutation node :collection, type: Collection input :post_id,

    !types.ID authorize: Authorization::MANAGE returns Graph::Types::PostType def perform post = Post.find inputs[:post_id] Collections.add_post collection, post post end end
  67. class Graph::Resolvers::Mutation # ... def call pre_proccess handle_result perform rescue

    ActiveRecord::RecordInvalid, MiniForm::InvalidForm => e errors_from_record(e.record) rescue ActiveRecord::RecordNotFound error :base, :record_not_found rescue CanCan::AccessDenied error_access_denied end # ... end
  68. 
 https://sidekiq.org/


  69. None
  70. class Achievements::Worker < ApplicationJob def perform(achievement, user)
 HandleRaceCondition.call do Achievements::Create.call

    achievement, user end end end
  71. module HandleRaceCondition extend self UNIQUE_ACTIVE_RECORD_ERROR = 'has already been taken'.freeze

    def call retries ||= 2 yield rescue ActiveRecord::RecordNotUnique, PG::UniqueViolation retries -= 1 raise unless retries.nonzero? retry rescue ActiveRecord::RecordInvalid => e raise unless e.message.include? UNIQUE_ACTIVE_RECORD_ERROR retries -= 1 raise unless retries.nonzero? retry end end
  72. Notifications::FanOutWorker Notifications::ScheduleWorker Notifications::DeliveryWorker Notifications.notify_about :kind, object

  73. Notifications::FanOutWorker Notifications::ScheduleWorker Notifications::DeliveryWorker Notifications.notify_about :kind, object kind, object

  74. Notifications::FanOutWorker Notifications::ScheduleWorker Notifications::DeliveryWorker Notifications.notify_about :kind, object kind, object kind, object,

    subscriber
  75. Notifications::FanOutWorker Notifications::ScheduleWorker Notifications::DeliveryWorker Notifications.notify_about :kind, object kind, object kind, object,

    subscriber kind, object, subscriber, channel
  76. class Notifications::ScheduleWorker < ApplicationJob include ActiveJobHandleDeserializationError queue_as :notifications def perform(kind:,

    object:, subscriber_id:) Notifications::Schedule.call( kind: kind, object: object, subscriber_id: subscriber_id, ) end end
  77. module ActiveJobHandleDeserializationError include ActiveJobRetriesCount extend ActiveSupport::Concern included do rescue_from ActiveJob::DeserializationError

    do retry_job wait: 5.minutes if retries_count.zero? end end end
  78. module ActiveJobHandleDeserializationError include ActiveJobRetriesCount extend ActiveSupport::Concern included do rescue_from ActiveJob::DeserializationError

    do retry_job wait: 5.minutes if retries_count.zero? end end end
  79. module ActiveJobRetriesCount extend ActiveSupport::Concern included do attr_accessor :retries_count end def

    initialize(*arguments) super @retries_count ||= 0 end def deserialize(job_data) super @retries_count = job_data['retries_count'] || 0 end def serialize hash = super.merge('retries_count' => retries_count || 0) # NOTE(rstankov): Workaround for Rails bug # When arguments raise `ActiveJob::DeserializationError` arguments are blank hash['arguments'] ||= @serialized_arguments if @serialized_arguments hash end def retry_job(options) @retries_count = (retries_count || 0) + 1 super(options) end end
  80. class Notifications::Deliver::Worker < ApplicationJob include ActiveJobHandleNetworkErrors queue_as :notifications def perform(event)

    Notifications::Deliver.call(event) end end
  81. module ActiveJobHandleNetworkErrors include ActiveJobRetriesCount extend ActiveSupport::Concern included do rescue_from(NetworkErrors) do

    if retries_count <= 10 retry_job wait: 5.minutes else
 raise e end end end end
  82. None
  83. None
  84. None
  85. Thanks +