Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Zero Exceptions in Production

Zero Exceptions in Production

Presented at Balkan Ruby 2018

In the talk, I'm explaining how to categorize exceptions and their level of impact. Present use cases and code samples of common problems in a Rails application. How to make sure your background workers run without issues and how to debug exceptions.

Links:

- 
http://exceptionalruby.com/
- https://sentry.io/
- 
https://graphql.org/
- 
https://sidekiq.org/

Radoslav Stankov

May 24, 2018
Tweet

More Decks by Radoslav Stankov

Other Decks in Technology

Transcript

  1. How to get to zero
    unhandled exceptions
    in production
    Radoslav Stankov 26/05/2018

    View Slide

  2. Radoslav Stankov
    @rstankov

    blog.rstankov.com

    github.com/rstankov

    twitter.com/rstankov

    View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. View Slide

  12. View Slide

  13. Happy Friday !
    " Fix bugs
    # Goodies features
    $ Pay technical dept
    % Catchup on projects
    & Fix exceptions

    View Slide

  14. Happy Friday !
    " Fix bugs
    # Goodies features
    $ Pay technical dept
    % Catchup on projects
    & Fix exceptions

    View Slide

  15. View Slide

  16. Back to basics

    View Slide


  17. http://exceptionalruby.com/


    View Slide

  18. def perform
    do_something
    rescue
    end

    View Slide

  19. def perform
    do_something
    rescue
    end

    View Slide

  20. def perform
    do_something
    rescue
    end

    View Slide

  21. def perform
    do_something
    rescue SpecificError
    end

    View Slide

  22. def perform
    do_something
    rescue SpecificError
    # NOTE(rstankov): Reason to return nil
    nil
    end

    View Slide

  23. def perform
    do_something
    rescue SpecificError
    # NOTE(rstankov): Reason to return nil
    nil
    end
    Your name, not mine '

    View Slide

  24. def perform
    do_something
    rescue SpecificError
    # NOTE(rstankov): Reason to return nil
    nil
    end

    View Slide

  25. def perform
    do_something
    rescue NetworkErrors
    # NOTE(rstankov): WiFi Sucks
    nil
    end

    View Slide

  26. module NetworkErrors
    extend self
    ERRORS = [
    EOFError,
    Errno::ECONNRESET,
    Errno::EINVAL,
    Net::HTTPBadResponse,
    Net::HTTPHeaderSyntaxError,
    Net::ProtocolError,
    Timeout::Error,
    # ...
    ]
    def ===(error)
    ERRORS.any? { |error_class| error_class === error }
    end
    end

    View Slide

  27. – saw it in a tweet once
    “A mature Rails project is one, that uses every
    popular (or not so popular) Ruby networking gem.”
    (

    View Slide

  28. – saw it in a tweet once
    “A mature Rails project is one, that uses every
    popular (or not so popular) Ruby networking gem.”
    (

    View Slide

  29. module NetworkErrors
    extend self
    ERRORS = [
    # ...
    Faraday::ConnectionFailed,
    Faraday::TimeoutError,
    RestClient::BadGateway,
    RestClient::BadRequest,
    # ...
    ]
    def ===(error)
    ERRORS.any? { |error_class| error_class === error }
    end
    end

    View Slide

  30. if user.ship_subscription.blank?
    Raven.capture_message "Missing ship subscription", extra: { user_id: user.id }
    return :no_subscription
    end

    View Slide

  31. Monitoring

    View Slide

  32. View Slide

  33. View Slide

  34. View Slide

  35. View Slide

  36. Raven.configure do |config|
    # Note(rstankov): Exclude un actionable errors
    config.excluded_exceptions = [
    'Rack::Timeout::RequestExpiryError',
    'Rack::Timeout::RequestTimeoutException',
    'ActionController::RoutingError',
    'ActionController::InvalidAuthenticityToken',
    'ActionDispatch::ParamsParser::ParseError',
    'Sidekiq::Shutdown',
    ]
    end

    View Slide

  37. View Slide

  38. View Slide

  39. View Slide

  40. View Slide

  41. account.subscription.status

    View Slide

  42. account.subscription&.status

    View Slide

  43. account.subscription&.status

    View Slide

  44. Steps to fix )

    View Slide

  45. 0. Check for other accounts without subscription
    Steps to fix )

    View Slide

  46. 0. Check for other accounts without subscription
    1. Find out why account doesn't have a subscription
    Steps to fix )

    View Slide

  47. 0. Check for other accounts without subscription
    1. Find out why account doesn't have a subscription
    2. Fix the problem
    Steps to fix )

    View Slide

  48. 0. Check for other accounts without subscription
    1. Find out why account doesn't have a subscription
    2. Fix the problem
    3. Add missing subscriptions to this accounts
    Steps to fix )

    View Slide

  49. View Slide

  50. View Slide

  51. – Brian Kernighan and Rob Pike, The Practice of Programming
    “Exceptions shouldn't be expected”
    Use exceptions only for exceptional situations. […]
    Exceptions are often overused.
    Because they distort the flow of control, they can
    lead to convoluted constructions that are prone to
    bugs. It is hardly exceptional to fail to open a file;
    generating an exception in this case strikes us as
    over-engineering.

    View Slide

  52. ArgumentError: invalid byte sequence in UTF-8

    View Slide

  53. View Slide

  54. # NOTE(rstankov): Fix invalid byte sequence in UTF-8. More info:
    # - https://robots.thoughtbot.com/fight-back-utf-8-invalid-byte-sequences
    module Utf8Sanitize
    extend self
    def call(string)
    return if string.nil?
    string.encode('UTF-8', 'binary',
    invalid: :replace,
    undef: :replace,
    replace: ''
    )
    end
    end

    View Slide


  55. https://graphql.org/


    View Slide

  56. View Slide

  57. View Slide

  58. Which query causes
    this issue? *

    View Slide

  59. class Frontend::GraphqlController < Frontend::BaseController
    before_action :ensure_query
    def index
    render json: Graph::Schema.execute(query, variables: variables, context: context)
    rescue => e
    handle_error e
    end
    private
    # ...
    def handle_error(e)
    if Rails.env.development?
    logger.error e.message
    logger.error e.backtrace.join("\n")
    render json: { message: e.message, backtrace: e.backtrace }, status: 500
    elsif Rails.env.test?
    p e.message # rubocop:disable Rails/Output
    p e.backtrace # rubocop:disable Rails/Output
    else
    Raven.capture_exception(e, extra: { query: query })
    end
    end
    end

    View Slide

  60. class Frontend::GraphqlController < Frontend::BaseController
    before_action :ensure_query
    def index
    render json: Graph::Schema.execute(query, variables: variables, context: context)
    rescue => e
    handle_error e
    end
    private
    # ...
    def handle_error(e)
    if Rails.env.development?
    logger.error e.message
    logger.error e.backtrace.join("\n")
    render json: { message: e.message, backtrace: e.backtrace }, status: 500
    elsif Rails.env.test?
    p e.message # rubocop:disable Rails/Output
    p e.backtrace # rubocop:disable Rails/Output
    else
    Raven.capture_exception(e, extra: { query: query })
    end
    end
    end

    View Slide

  61. class Frontend::GraphqlController < Frontend::BaseController
    before_action :ensure_query
    def index
    render json: Graph::Schema.execute(query, variables: variables, context: context)
    rescue => e
    handle_error e
    end
    private
    # ...
    def handle_error(e)
    if Rails.env.development?
    logger.error e.message
    logger.error e.backtrace.join("\n")
    render json: { message: e.message, backtrace: e.backtrace }, status: 500
    elsif Rails.env.test?
    p e.message # rubocop:disable Rails/Output
    p e.backtrace # rubocop:disable Rails/Output
    else
    Raven.capture_exception(e, extra: { query: query })
    end
    end
    end

    View Slide

  62. View Slide

  63. View Slide

  64. Graph::Query = GraphQL::ObjectType.define do
    name 'Query'
    field :post do
    type !Graph::Types::PostType
    argument :id, !types.ID
    resolve ->(_obj, inputs, _ctx) { Post.find(inputs[:id]) }
    end
    end

    View Slide

  65. Graph::Query = GraphQL::ObjectType.define do
    name 'Query'
    field :post do
    type Graph::Types::PostType
    argument :id, !types.ID
    resolve ->(_obj, inputs, _ctx) { Post.find_by(id: inputs[:id]) }
    end
    end

    View Slide

  66. class Graph::Mutations::CollectionAddPost < Graph::Resolvers::Mutation
    node :collection, type: Collection
    input :post_id, !types.ID
    authorize: Authorization::MANAGE
    returns Graph::Types::PostType
    def perform
    post = Post.find inputs[:post_id]
    Collections.add_post collection, post
    post
    end
    end

    View Slide

  67. class Graph::Resolvers::Mutation
    # ...
    def call
    pre_proccess
    handle_result perform
    rescue ActiveRecord::RecordInvalid, MiniForm::InvalidForm => e
    errors_from_record(e.record)
    rescue ActiveRecord::RecordNotFound
    error :base, :record_not_found
    rescue CanCan::AccessDenied
    error_access_denied
    end
    # ...
    end

    View Slide


  68. https://sidekiq.org/


    View Slide

  69. View Slide

  70. class Achievements::Worker < ApplicationJob
    def perform(achievement, user)

    HandleRaceCondition.call do
    Achievements::Create.call achievement, user
    end
    end
    end

    View Slide

  71. module HandleRaceCondition
    extend self
    UNIQUE_ACTIVE_RECORD_ERROR = 'has already been taken'.freeze
    def call
    retries ||= 2
    yield
    rescue ActiveRecord::RecordNotUnique, PG::UniqueViolation
    retries -= 1
    raise unless retries.nonzero?
    retry
    rescue ActiveRecord::RecordInvalid => e
    raise unless e.message.include? UNIQUE_ACTIVE_RECORD_ERROR
    retries -= 1
    raise unless retries.nonzero?
    retry
    end
    end

    View Slide

  72. Notifications::FanOutWorker
    Notifications::ScheduleWorker
    Notifications::DeliveryWorker
    Notifications.notify_about :kind, object

    View Slide

  73. Notifications::FanOutWorker
    Notifications::ScheduleWorker
    Notifications::DeliveryWorker
    Notifications.notify_about :kind, object
    kind, object

    View Slide

  74. Notifications::FanOutWorker
    Notifications::ScheduleWorker
    Notifications::DeliveryWorker
    Notifications.notify_about :kind, object
    kind, object
    kind, object, subscriber

    View Slide

  75. Notifications::FanOutWorker
    Notifications::ScheduleWorker
    Notifications::DeliveryWorker
    Notifications.notify_about :kind, object
    kind, object
    kind, object, subscriber
    kind, object, subscriber, channel

    View Slide

  76. class Notifications::ScheduleWorker < ApplicationJob
    include ActiveJobHandleDeserializationError
    queue_as :notifications
    def perform(kind:, object:, subscriber_id:)
    Notifications::Schedule.call(
    kind: kind,
    object: object,
    subscriber_id: subscriber_id,
    )
    end
    end

    View Slide

  77. module ActiveJobHandleDeserializationError
    include ActiveJobRetriesCount
    extend ActiveSupport::Concern
    included do
    rescue_from ActiveJob::DeserializationError do
    retry_job wait: 5.minutes if retries_count.zero?
    end
    end
    end

    View Slide

  78. module ActiveJobHandleDeserializationError
    include ActiveJobRetriesCount
    extend ActiveSupport::Concern
    included do
    rescue_from ActiveJob::DeserializationError do
    retry_job wait: 5.minutes if retries_count.zero?
    end
    end
    end

    View Slide

  79. module ActiveJobRetriesCount
    extend ActiveSupport::Concern
    included do
    attr_accessor :retries_count
    end
    def initialize(*arguments)
    super
    @retries_count ||= 0
    end
    def deserialize(job_data)
    super
    @retries_count = job_data['retries_count'] || 0
    end
    def serialize
    hash = super.merge('retries_count' => retries_count || 0)
    # NOTE(rstankov): Workaround for Rails bug
    # When arguments raise `ActiveJob::DeserializationError` arguments are blank
    hash['arguments'] ||= @serialized_arguments if @serialized_arguments
    hash
    end
    def retry_job(options)
    @retries_count = (retries_count || 0) + 1
    super(options)
    end
    end

    View Slide

  80. class Notifications::Deliver::Worker < ApplicationJob
    include ActiveJobHandleNetworkErrors
    queue_as :notifications
    def perform(event)
    Notifications::Deliver.call(event)
    end
    end

    View Slide

  81. module ActiveJobHandleNetworkErrors
    include ActiveJobRetriesCount
    extend ActiveSupport::Concern
    included do
    rescue_from(NetworkErrors) do
    if retries_count <= 10
    retry_job wait: 5.minutes
    else

    raise e
    end
    end
    end
    end

    View Slide

  82. View Slide

  83. View Slide

  84. View Slide

  85. Thanks +

    View Slide