Zero Exceptions in Production

Zero Exceptions in Production

Presented at Balkan Ruby 2018

In the talk, I'm explaining how to categorize exceptions and their level of impact. Present use cases and code samples of common problems in a Rails application. How to make sure your background workers run without issues and how to debug exceptions.

Links:

- 
http://exceptionalruby.com/
- https://sentry.io/
- 
https://graphql.org/
- 
https://sidekiq.org/

7a0e72a6f55811246bb5d9a946fd2e49?s=128

Radoslav Stankov

May 24, 2018
Tweet

Transcript

  1. 3.
  2. 4.
  3. 5.
  4. 6.
  5. 7.
  6. 8.
  7. 9.
  8. 10.
  9. 11.
  10. 12.
  11. 13.

    Happy Friday ! " Fix bugs # Goodies features $

    Pay technical dept % Catchup on projects & Fix exceptions
  12. 14.

    Happy Friday ! " Fix bugs # Goodies features $

    Pay technical dept % Catchup on projects & Fix exceptions
  13. 15.
  14. 26.

    module NetworkErrors extend self ERRORS = [ EOFError, Errno::ECONNRESET, Errno::EINVAL,

    Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Net::ProtocolError, Timeout::Error, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
  15. 27.

    – saw it in a tweet once “A mature Rails

    project is one, that uses every popular (or not so popular) Ruby networking gem.” (
  16. 28.

    – saw it in a tweet once “A mature Rails

    project is one, that uses every popular (or not so popular) Ruby networking gem.” (
  17. 29.

    module NetworkErrors extend self ERRORS = [ # ... Faraday::ConnectionFailed,

    Faraday::TimeoutError, RestClient::BadGateway, RestClient::BadRequest, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
  18. 32.
  19. 33.
  20. 34.
  21. 35.
  22. 36.

    Raven.configure do |config| # Note(rstankov): Exclude un actionable errors config.excluded_exceptions

    = [ 'Rack::Timeout::RequestExpiryError', 'Rack::Timeout::RequestTimeoutException', 'ActionController::RoutingError', 'ActionController::InvalidAuthenticityToken', 'ActionDispatch::ParamsParser::ParseError', 'Sidekiq::Shutdown', ] end
  23. 37.
  24. 38.
  25. 39.
  26. 40.
  27. 46.

    0. Check for other accounts without subscription 1. Find out

    why account doesn't have a subscription Steps to fix )
  28. 47.

    0. Check for other accounts without subscription 1. Find out

    why account doesn't have a subscription 2. Fix the problem Steps to fix )
  29. 48.

    0. Check for other accounts without subscription 1. Find out

    why account doesn't have a subscription 2. Fix the problem 3. Add missing subscriptions to this accounts Steps to fix )
  30. 49.
  31. 50.
  32. 51.

    – Brian Kernighan and Rob Pike, The Practice of Programming

    “Exceptions shouldn't be expected” Use exceptions only for exceptional situations. […] Exceptions are often overused. Because they distort the flow of control, they can lead to convoluted constructions that are prone to bugs. It is hardly exceptional to fail to open a file; generating an exception in this case strikes us as over-engineering.
  33. 53.
  34. 54.

    # NOTE(rstankov): Fix invalid byte sequence in UTF-8. More info:

    # - https://robots.thoughtbot.com/fight-back-utf-8-invalid-byte-sequences module Utf8Sanitize extend self def call(string) return if string.nil? string.encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '' ) end end
  35. 56.
  36. 57.
  37. 59.

    class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(e) if Rails.env.development? logger.error e.message logger.error e.backtrace.join("\n") render json: { message: e.message, backtrace: e.backtrace }, status: 500 elsif Rails.env.test? p e.message # rubocop:disable Rails/Output p e.backtrace # rubocop:disable Rails/Output else Raven.capture_exception(e, extra: { query: query }) end end end
  38. 60.

    class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(e) if Rails.env.development? logger.error e.message logger.error e.backtrace.join("\n") render json: { message: e.message, backtrace: e.backtrace }, status: 500 elsif Rails.env.test? p e.message # rubocop:disable Rails/Output p e.backtrace # rubocop:disable Rails/Output else Raven.capture_exception(e, extra: { query: query }) end end end
  39. 61.

    class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(e) if Rails.env.development? logger.error e.message logger.error e.backtrace.join("\n") render json: { message: e.message, backtrace: e.backtrace }, status: 500 elsif Rails.env.test? p e.message # rubocop:disable Rails/Output p e.backtrace # rubocop:disable Rails/Output else Raven.capture_exception(e, extra: { query: query }) end end end
  40. 62.
  41. 63.
  42. 64.

    Graph::Query = GraphQL::ObjectType.define do name 'Query' field :post do type

    !Graph::Types::PostType argument :id, !types.ID resolve ->(_obj, inputs, _ctx) { Post.find(inputs[:id]) } end end
  43. 65.

    Graph::Query = GraphQL::ObjectType.define do name 'Query' field :post do type

    Graph::Types::PostType argument :id, !types.ID resolve ->(_obj, inputs, _ctx) { Post.find_by(id: inputs[:id]) } end end
  44. 66.

    class Graph::Mutations::CollectionAddPost < Graph::Resolvers::Mutation node :collection, type: Collection input :post_id,

    !types.ID authorize: Authorization::MANAGE returns Graph::Types::PostType def perform post = Post.find inputs[:post_id] Collections.add_post collection, post post end end
  45. 67.

    class Graph::Resolvers::Mutation # ... def call pre_proccess handle_result perform rescue

    ActiveRecord::RecordInvalid, MiniForm::InvalidForm => e errors_from_record(e.record) rescue ActiveRecord::RecordNotFound error :base, :record_not_found rescue CanCan::AccessDenied error_access_denied end # ... end
  46. 69.
  47. 71.

    module HandleRaceCondition extend self UNIQUE_ACTIVE_RECORD_ERROR = 'has already been taken'.freeze

    def call retries ||= 2 yield rescue ActiveRecord::RecordNotUnique, PG::UniqueViolation retries -= 1 raise unless retries.nonzero? retry rescue ActiveRecord::RecordInvalid => e raise unless e.message.include? UNIQUE_ACTIVE_RECORD_ERROR retries -= 1 raise unless retries.nonzero? retry end end
  48. 76.

    class Notifications::ScheduleWorker < ApplicationJob include ActiveJobHandleDeserializationError queue_as :notifications def perform(kind:,

    object:, subscriber_id:) Notifications::Schedule.call( kind: kind, object: object, subscriber_id: subscriber_id, ) end end
  49. 79.

    module ActiveJobRetriesCount extend ActiveSupport::Concern included do attr_accessor :retries_count end def

    initialize(*arguments) super @retries_count ||= 0 end def deserialize(job_data) super @retries_count = job_data['retries_count'] || 0 end def serialize hash = super.merge('retries_count' => retries_count || 0) # NOTE(rstankov): Workaround for Rails bug # When arguments raise `ActiveJob::DeserializationError` arguments are blank hash['arguments'] ||= @serialized_arguments if @serialized_arguments hash end def retry_job(options) @retries_count = (retries_count || 0) + 1 super(options) end end
  50. 82.
  51. 83.
  52. 84.
  53. 85.