Zero Exceptions in Production

Presented at Balkan Ruby 2018

In the talk, I'm explaining how to categorize exceptions and their level of impact. Present use cases and code samples of common problems in a Rails application. How to make sure your background workers run without issues and how to debug exceptions.

Links:

- 
http://exceptionalruby.com/

- https://sentry.io/
- 
https://graphql.org/

- 
https://sidekiq.org/


Transcript

1. How to get to zero unhandled exceptions in production Radoslav Stankov 26/05/2018
2. Radoslav Stankov @rstankov blog.rstankov.com github.com/rstankov
 twitter.com/rstankov
13. Happy Friday ! " Fix bugs # Goodies features $ Pay technical dept % Catchup on projects & Fix exceptions
14. Happy Friday ! " Fix bugs # Goodies features $ Pay technical dept % Catchup on projects & Fix exceptions
16. Back to basics
17. 
 http://exceptionalruby.com/

18. def perform do_something rescue end
19. def perform do_something rescue end
20. def perform do_something rescue end
21. def perform do_something rescue SpecificError end
22. def perform do_something rescue SpecificError # NOTE(rstankov): Reason to return nil nil end
23. def perform do_something rescue SpecificError # NOTE(rstankov): Reason to return nil nil end Your name, not mine '
24. def perform do_something rescue SpecificError # NOTE(rstankov): Reason to return nil nil end
25. def perform do_something rescue NetworkErrors # NOTE(rstankov): WiFi Sucks nil end
26. module NetworkErrors extend self ERRORS = [ EOFError, Errno::ECONNRESET, Errno::EINVAL, Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Net::ProtocolError, Timeout::Error, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
27. – saw it in a tweet once “A mature Rails project is one, that uses every popular (or not so popular) Ruby networking gem.” (
28. – saw it in a tweet once “A mature Rails project is one, that uses every popular (or not so popular) Ruby networking gem.” (
29. module NetworkErrors extend self ERRORS = [ # ... Faraday::ConnectionFailed, Faraday::TimeoutError, RestClient::BadGateway, RestClient::BadRequest, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
30. if user.ship_subscription.blank? Raven.capture_message "Missing ship subscription", extra: { user_id: user.id } return :no_subscription end
31. Monitoring
36. Raven.configure do |config| # Note(rstankov): Exclude un actionable errors config.excluded_exceptions = [ 'Rack::Timeout::RequestExpiryError', 'Rack::Timeout::RequestTimeoutException', 'ActionController::RoutingError', 'ActionController::InvalidAuthenticityToken', 'ActionDispatch::ParamsParser::ParseError', 'Sidekiq::Shutdown', ] end
41. account.subscription.status
42. account.subscription&.status
43. account.subscription&.status
44. Steps to fix )
45. 0. Check for other accounts without subscription Steps to fix )
46. 0. Check for other accounts without subscription 1. Find out why account doesn't have a subscription Steps to fix )
47. 0. Check for other accounts without subscription 1. Find out why account doesn't have a subscription 2. Fix the problem Steps to fix )
48. 0. Check for other accounts without subscription 1. Find out why account doesn't have a subscription 2. Fix the problem 3. Add missing subscriptions to this accounts Steps to fix )
51. – Brian Kernighan and Rob Pike, The Practice of Programming “Exceptions shouldn't be expected” Use exceptions only for exceptional situations. […] Exceptions are often overused. Because they distort the flow of control, they can lead to convoluted constructions that are prone to bugs. It is hardly exceptional to fail to open a file; generating an exception in this case strikes us as over-engineering.
52. ArgumentError: invalid byte sequence in UTF-8
54. # NOTE(rstankov): Fix invalid byte sequence in UTF-8. More info: # - https://robots.thoughtbot.com/fight-back-utf-8-invalid-byte-sequences module Utf8Sanitize extend self def call(string) return if string.nil? string.encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '' ) end end
55. 
 https://graphql.org/

58. Which query causes this issue? *
59. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json: Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(e) if Rails.env.development? logger.error e.message logger.error e.backtrace.join("\n") render json: { message: e.message, backtrace: e.backtrace }, status: 500 elsif Rails.env.test? p e.message # rubocop:disable Rails/Output p e.backtrace # rubocop:disable Rails/Output else Raven.capture_exception(e, extra: { query: query }) end end end
60. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json: Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(e) if Rails.env.development? logger.error e.message logger.error e.backtrace.join("\n") render json: { message: e.message, backtrace: e.backtrace }, status: 500 elsif Rails.env.test? p e.message # rubocop:disable Rails/Output p e.backtrace # rubocop:disable Rails/Output else Raven.capture_exception(e, extra: { query: query }) end end end
61. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json: Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(e) if Rails.env.development? logger.error e.message logger.error e.backtrace.join("\n") render json: { message: e.message, backtrace: e.backtrace }, status: 500 elsif Rails.env.test? p e.message # rubocop:disable Rails/Output p e.backtrace # rubocop:disable Rails/Output else Raven.capture_exception(e, extra: { query: query }) end end end
64. Graph::Query = GraphQL::ObjectType.define do name 'Query' field :post do type !Graph::Types::PostType argument :id, !types.ID resolve ->(_obj, inputs, _ctx) { Post.find(inputs[:id]) } end end
65. Graph::Query = GraphQL::ObjectType.define do name 'Query' field :post do type Graph::Types::PostType argument :id, !types.ID resolve ->(_obj, inputs, _ctx) { Post.find_by(id: inputs[:id]) } end end
66. class Graph::Mutations::CollectionAddPost < Graph::Resolvers::Mutation node :collection, type: Collection input :post_id, !types.ID authorize: Authorization::MANAGE returns Graph::Types::PostType def perform post = Post.find inputs[:post_id] Collections.add_post collection, post post end end
67. class Graph::Resolvers::Mutation # ... def call pre_proccess handle_result perform rescue ActiveRecord::RecordInvalid, MiniForm::InvalidForm => e errors_from_record(e.record) rescue ActiveRecord::RecordNotFound error :base, :record_not_found rescue CanCan::AccessDenied error_access_denied end # ... end
68. 
 https://sidekiq.org/

70. class Achievements::Worker < ApplicationJob def perform(achievement, user)
 HandleRaceCondition.call do Achievements::Create.call achievement, user end end end
71. module HandleRaceCondition extend self UNIQUE_ACTIVE_RECORD_ERROR = 'has already been taken'.freeze def call retries ||= 2 yield rescue ActiveRecord::RecordNotUnique, PG::UniqueViolation retries -= 1 raise unless retries.nonzero? retry rescue ActiveRecord::RecordInvalid => e raise unless e.message.include? UNIQUE_ACTIVE_RECORD_ERROR retries -= 1 raise unless retries.nonzero? retry end end
72. Notifications::FanOutWorker Notifications::ScheduleWorker Notifications::DeliveryWorker Notifications.notify_about :kind, object
73. Notifications::FanOutWorker Notifications::ScheduleWorker Notifications::DeliveryWorker Notifications.notify_about :kind, object kind, object
74. Notifications::FanOutWorker Notifications::ScheduleWorker Notifications::DeliveryWorker Notifications.notify_about :kind, object kind, object kind, object, subscriber
75. Notifications::FanOutWorker Notifications::ScheduleWorker Notifications::DeliveryWorker Notifications.notify_about :kind, object kind, object kind, object, subscriber kind, object, subscriber, channel
76. class Notifications::ScheduleWorker < ApplicationJob include ActiveJobHandleDeserializationError queue_as :notifications def perform(kind:, object:, subscriber_id:) Notifications::Schedule.call( kind: kind, object: object, subscriber_id: subscriber_id, ) end end
77. module ActiveJobHandleDeserializationError include ActiveJobRetriesCount extend ActiveSupport::Concern included do rescue_from ActiveJob::DeserializationError do retry_job wait: 5.minutes if retries_count.zero? end end end
78. module ActiveJobHandleDeserializationError include ActiveJobRetriesCount extend ActiveSupport::Concern included do rescue_from ActiveJob::DeserializationError do retry_job wait: 5.minutes if retries_count.zero? end end end
79. module ActiveJobRetriesCount extend ActiveSupport::Concern included do attr_accessor :retries_count end def initialize(*arguments) super @retries_count ||= 0 end def deserialize(job_data) super @retries_count = job_data['retries_count'] || 0 end def serialize hash = super.merge('retries_count' => retries_count || 0) # NOTE(rstankov): Workaround for Rails bug # When arguments raise `ActiveJob::DeserializationError` arguments are blank hash['arguments'] ||= @serialized_arguments if @serialized_arguments hash end def retry_job(options) @retries_count = (retries_count || 0) + 1 super(options) end end
80. class Notifications::Deliver::Worker < ApplicationJob include ActiveJobHandleNetworkErrors queue_as :notifications def perform(event) Notifications::Deliver.call(event) end end
81. module ActiveJobHandleNetworkErrors include ActiveJobRetriesCount extend ActiveSupport::Concern included do rescue_from(NetworkErrors) do if retries_count <= 10 retry_job wait: 5.minutes else
 raise e end end end end
85. Thanks +