Zero Exceptions in Production

Presented at Balkan Ruby 2018

In the talk, I'm explaining how to categorize exceptions and their level of impact. Present use cases and code samples of common problems in a Rails application. How to make sure your background workers run without issues and how to debug exceptions.

Links:

- 
http://exceptionalruby.com/

- https://sentry.io/
- 
https://graphql.org/

- 
https://sidekiq.org/


Transcript

1. How to get to zero
   unhandled exceptions
   in production
   Radoslav Stankov 26/05/2018
   

   View Slide

2. Radoslav Stankov
   @rstankov
   
   blog.rstankov.com
   
   github.com/rstankov

   twitter.com/rstankov
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. View Slide

12. View Slide

13. Happy Friday !
    " Fix bugs
    # Goodies features
    $ Pay technical dept
    % Catchup on projects
    & Fix exceptions
    

    View Slide

14. Happy Friday !
    " Fix bugs
    # Goodies features
    $ Pay technical dept
    % Catchup on projects
    & Fix exceptions
    

    View Slide

15. View Slide

16. Back to basics
    

    View Slide

17. 

    http://exceptionalruby.com/

    

    View Slide

18. def perform
    do_something
    rescue
    end
    

    View Slide

19. def perform
    do_something
    rescue
    end
    

    View Slide

20. def perform
    do_something
    rescue
    end
    

    View Slide

21. def perform
    do_something
    rescue SpecificError
    end
    

    View Slide

22. def perform
    do_something
    rescue SpecificError
    # NOTE(rstankov): Reason to return nil
    nil
    end
    

    View Slide

23. def perform
    do_something
    rescue SpecificError
    # NOTE(rstankov): Reason to return nil
    nil
    end
    Your name, not mine '
    

    View Slide

24. def perform
    do_something
    rescue SpecificError
    # NOTE(rstankov): Reason to return nil
    nil
    end
    

    View Slide

25. def perform
    do_something
    rescue NetworkErrors
    # NOTE(rstankov): WiFi Sucks
    nil
    end
    

    View Slide

26. module NetworkErrors
    extend self
    ERRORS = [
    EOFError,
    Errno::ECONNRESET,
    Errno::EINVAL,
    Net::HTTPBadResponse,
    Net::HTTPHeaderSyntaxError,
    Net::ProtocolError,
    Timeout::Error,
    # ...
    ]
    def ===(error)
    ERRORS.any? { |error_class| error_class === error }
    end
    end
    

    View Slide

27. – saw it in a tweet once
    “A mature Rails project is one, that uses every
    popular (or not so popular) Ruby networking gem.”
    (
    

    View Slide

28. – saw it in a tweet once
    “A mature Rails project is one, that uses every
    popular (or not so popular) Ruby networking gem.”
    (
    

    View Slide

29. module NetworkErrors
    extend self
    ERRORS = [
    # ...
    Faraday::ConnectionFailed,
    Faraday::TimeoutError,
    RestClient::BadGateway,
    RestClient::BadRequest,
    # ...
    ]
    def ===(error)
    ERRORS.any? { |error_class| error_class === error }
    end
    end
    

    View Slide

30. if user.ship_subscription.blank?
    Raven.capture_message "Missing ship subscription", extra: { user_id: user.id }
    return :no_subscription
    end
    

    View Slide

31. Monitoring
    

    View Slide

32. View Slide

33. View Slide

34. View Slide

35. View Slide

36. Raven.configure do |config|
    # Note(rstankov): Exclude un actionable errors
    config.excluded_exceptions = [
    'Rack::Timeout::RequestExpiryError',
    'Rack::Timeout::RequestTimeoutException',
    'ActionController::RoutingError',
    'ActionController::InvalidAuthenticityToken',
    'ActionDispatch::ParamsParser::ParseError',
    'Sidekiq::Shutdown',
    ]
    end
    

    View Slide

37. View Slide

38. View Slide

39. View Slide

40. View Slide

41. account.subscription.status
    

    View Slide

42. account.subscription&.status
    

    View Slide

43. account.subscription&.status
    

    View Slide

44. Steps to fix )
    

    View Slide

45. 0. Check for other accounts without subscription
    Steps to fix )
    

    View Slide

46. 0. Check for other accounts without subscription
    1. Find out why account doesn't have a subscription
    Steps to fix )
    

    View Slide

47. 0. Check for other accounts without subscription
    1. Find out why account doesn't have a subscription
    2. Fix the problem
    Steps to fix )
    

    View Slide

48. 0. Check for other accounts without subscription
    1. Find out why account doesn't have a subscription
    2. Fix the problem
    3. Add missing subscriptions to this accounts
    Steps to fix )
    

    View Slide

49. View Slide

50. View Slide

51. – Brian Kernighan and Rob Pike, The Practice of Programming
    “Exceptions shouldn't be expected”
    Use exceptions only for exceptional situations. […]
    Exceptions are often overused.
    Because they distort the flow of control, they can
    lead to convoluted constructions that are prone to
    bugs. It is hardly exceptional to fail to open a file;
    generating an exception in this case strikes us as
    over-engineering.
    

    View Slide

52. ArgumentError: invalid byte sequence in UTF-8
    

    View Slide

53. View Slide

54. # NOTE(rstankov): Fix invalid byte sequence in UTF-8. More info:
    # - https://robots.thoughtbot.com/fight-back-utf-8-invalid-byte-sequences
    module Utf8Sanitize
    extend self
    def call(string)
    return if string.nil?
    string.encode('UTF-8', 'binary',
    invalid: :replace,
    undef: :replace,
    replace: ''
    )
    end
    end
    

    View Slide

55. 

    https://graphql.org/

    

    View Slide

56. View Slide

57. View Slide

58. Which query causes
    this issue? *
    

    View Slide

59. class Frontend::GraphqlController < Frontend::BaseController
    before_action :ensure_query
    def index
    render json: Graph::Schema.execute(query, variables: variables, context: context)
    rescue => e
    handle_error e
    end
    private
    # ...
    def handle_error(e)
    if Rails.env.development?
    logger.error e.message
    logger.error e.backtrace.join("\n")
    render json: { message: e.message, backtrace: e.backtrace }, status: 500
    elsif Rails.env.test?
    p e.message # rubocop:disable Rails/Output
    p e.backtrace # rubocop:disable Rails/Output
    else
    Raven.capture_exception(e, extra: { query: query })
    end
    end
    end
    

    View Slide

60. class Frontend::GraphqlController < Frontend::BaseController
    before_action :ensure_query
    def index
    render json: Graph::Schema.execute(query, variables: variables, context: context)
    rescue => e
    handle_error e
    end
    private
    # ...
    def handle_error(e)
    if Rails.env.development?
    logger.error e.message
    logger.error e.backtrace.join("\n")
    render json: { message: e.message, backtrace: e.backtrace }, status: 500
    elsif Rails.env.test?
    p e.message # rubocop:disable Rails/Output
    p e.backtrace # rubocop:disable Rails/Output
    else
    Raven.capture_exception(e, extra: { query: query })
    end
    end
    end
    

    View Slide

61. class Frontend::GraphqlController < Frontend::BaseController
    before_action :ensure_query
    def index
    render json: Graph::Schema.execute(query, variables: variables, context: context)
    rescue => e
    handle_error e
    end
    private
    # ...
    def handle_error(e)
    if Rails.env.development?
    logger.error e.message
    logger.error e.backtrace.join("\n")
    render json: { message: e.message, backtrace: e.backtrace }, status: 500
    elsif Rails.env.test?
    p e.message # rubocop:disable Rails/Output
    p e.backtrace # rubocop:disable Rails/Output
    else
    Raven.capture_exception(e, extra: { query: query })
    end
    end
    end
    

    View Slide

62. View Slide

63. View Slide

64. Graph::Query = GraphQL::ObjectType.define do
    name 'Query'
    field :post do
    type !Graph::Types::PostType
    argument :id, !types.ID
    resolve ->(_obj, inputs, _ctx) { Post.find(inputs[:id]) }
    end
    end
    

    View Slide

65. Graph::Query = GraphQL::ObjectType.define do
    name 'Query'
    field :post do
    type Graph::Types::PostType
    argument :id, !types.ID
    resolve ->(_obj, inputs, _ctx) { Post.find_by(id: inputs[:id]) }
    end
    end
    

    View Slide

66. class Graph::Mutations::CollectionAddPost < Graph::Resolvers::Mutation
    node :collection, type: Collection
    input :post_id, !types.ID
    authorize: Authorization::MANAGE
    returns Graph::Types::PostType
    def perform
    post = Post.find inputs[:post_id]
    Collections.add_post collection, post
    post
    end
    end
    

    View Slide

67. class Graph::Resolvers::Mutation
    # ...
    def call
    pre_proccess
    handle_result perform
    rescue ActiveRecord::RecordInvalid, MiniForm::InvalidForm => e
    errors_from_record(e.record)
    rescue ActiveRecord::RecordNotFound
    error :base, :record_not_found
    rescue CanCan::AccessDenied
    error_access_denied
    end
    # ...
    end
    

    View Slide

68. 

    https://sidekiq.org/

    

    View Slide

69. View Slide

70. class Achievements::Worker < ApplicationJob
    def perform(achievement, user)

    HandleRaceCondition.call do
    Achievements::Create.call achievement, user
    end
    end
    end
    

    View Slide

71. module HandleRaceCondition
    extend self
    UNIQUE_ACTIVE_RECORD_ERROR = 'has already been taken'.freeze
    def call
    retries ||= 2
    yield
    rescue ActiveRecord::RecordNotUnique, PG::UniqueViolation
    retries -= 1
    raise unless retries.nonzero?
    retry
    rescue ActiveRecord::RecordInvalid => e
    raise unless e.message.include? UNIQUE_ACTIVE_RECORD_ERROR
    retries -= 1
    raise unless retries.nonzero?
    retry
    end
    end
    

    View Slide

72. Notifications::FanOutWorker
    Notifications::ScheduleWorker
    Notifications::DeliveryWorker
    Notifications.notify_about :kind, object
    

    View Slide

73. Notifications::FanOutWorker
    Notifications::ScheduleWorker
    Notifications::DeliveryWorker
    Notifications.notify_about :kind, object
    kind, object
    

    View Slide

74. Notifications::FanOutWorker
    Notifications::ScheduleWorker
    Notifications::DeliveryWorker
    Notifications.notify_about :kind, object
    kind, object
    kind, object, subscriber
    

    View Slide

75. Notifications::FanOutWorker
    Notifications::ScheduleWorker
    Notifications::DeliveryWorker
    Notifications.notify_about :kind, object
    kind, object
    kind, object, subscriber
    kind, object, subscriber, channel
    

    View Slide

76. class Notifications::ScheduleWorker < ApplicationJob
    include ActiveJobHandleDeserializationError
    queue_as :notifications
    def perform(kind:, object:, subscriber_id:)
    Notifications::Schedule.call(
    kind: kind,
    object: object,
    subscriber_id: subscriber_id,
    )
    end
    end
    

    View Slide

77. module ActiveJobHandleDeserializationError
    include ActiveJobRetriesCount
    extend ActiveSupport::Concern
    included do
    rescue_from ActiveJob::DeserializationError do
    retry_job wait: 5.minutes if retries_count.zero?
    end
    end
    end
    

    View Slide

78. module ActiveJobHandleDeserializationError
    include ActiveJobRetriesCount
    extend ActiveSupport::Concern
    included do
    rescue_from ActiveJob::DeserializationError do
    retry_job wait: 5.minutes if retries_count.zero?
    end
    end
    end
    

    View Slide

79. module ActiveJobRetriesCount
    extend ActiveSupport::Concern
    included do
    attr_accessor :retries_count
    end
    def initialize(*arguments)
    super
    @retries_count ||= 0
    end
    def deserialize(job_data)
    super
    @retries_count = job_data['retries_count'] || 0
    end
    def serialize
    hash = super.merge('retries_count' => retries_count || 0)
    # NOTE(rstankov): Workaround for Rails bug
    # When arguments raise `ActiveJob::DeserializationError` arguments are blank
    hash['arguments'] ||= @serialized_arguments if @serialized_arguments
    hash
    end
    def retry_job(options)
    @retries_count = (retries_count || 0) + 1
    super(options)
    end
    end
    

    View Slide

80. class Notifications::Deliver::Worker < ApplicationJob
    include ActiveJobHandleNetworkErrors
    queue_as :notifications
    def perform(event)
    Notifications::Deliver.call(event)
    end
    end
    

    View Slide

81. module ActiveJobHandleNetworkErrors
    include ActiveJobRetriesCount
    extend ActiveSupport::Concern
    included do
    rescue_from(NetworkErrors) do
    if retries_count <= 10
    retry_job wait: 5.minutes
    else

    raise e
    end
    end
    end
    end
    

    View Slide

82. View Slide

83. View Slide

84. View Slide

85. Thanks +
    

    View Slide