How to get to zero unhandled exceptions in production

Transcript

1. How to get to zero unhandled exceptions in production Radoslav

   Stankov 07/09/2019

2. Radoslav Stankov @rstankov blog.rstankov.com github.com/rstankov
 twitter.com/rstankov

3. None
4. None
5. None
6. None
7. None
8. None

9. 
 ! Product Hunt Engineering Team " 


10. None
11. None
12. None
13. None
14. None
15. None
16. None

17. Happy Friday # $ Fix bugs % Goodies features &

    Pay technical dept ' Catchup on projects ( Fix exceptions

18. Happy Friday # $ Fix bugs % Goodies features &

    Pay technical dept ' Catchup on projects ( Fix exceptions
19. None

20. “Have process around exceptions.” 
 ) Tip 


21. Back to basics

22. 
 http://exceptionalruby.com/


23. def perform do_something rescue end

24. None

25. def perform do_something rescue SpecificError end

26. def perform do_something rescue SpecificError # NOTE(rstankov): Reason to return

    nil nil end Your name, not mine *

27. def perform do_something rescue Timeout::Error # NOTE(rstankov): WiFi Sucks nil

    end

28. “Be explicit around the exceptions. Handle specific errors and have

    explanations of why they happen.” 
 ) Tip 


29. Monitoring

30. None
31. None
32. None
33. None

34. Raven.configure do |config| # Note(rstankov): Exclude unactionable errors config.excluded_exceptions =

    [ 'Rack::Timeout::RequestExpiryError', 'Rack::Timeout::RequestTimeoutException', 'ActionController::RoutingError', 'ActionController::InvalidAuthenticityToken', 'ActionDispatch::ParamsParser::ParseError', 'Sidekiq::Shutdown', ] end
35. None

36. ArgumentError: invalid byte sequence in UTF-8 (

37. None
38. None

39. # NOTE(rstankov): Fix invalid byte sequence in UTF-8. More info:

    # - https://robots.thoughtbot.com/fight-back-utf-8-invalid-byte-sequences module Handle::InvalidByteSequence extend self def call(string) return if string.nil? string.encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '' ) end end

40. “Reduce noise. See only exceptional errors in your tracker.” 


    ) Tip 

41. None
42. None

43. account.subscription.status

44. account.subscription&.status

45. ✅ Check for other accounts without a subscription ✅ Find

    out why some accounts don't have a subscription ✅ Fix the root problem ✅ Add missing subscriptions to accounts , Steps to fix -
46. None
47. None

48. if account.ship_subscription.blank? Raven.capture_message "Missing ship subscription", extra: { account_id: account.id

    } return :no_subscription end

49. “Don't hide exceptions. Fix root causes.” 
 ) Tip 


50. 
 https://graphql.org/


51. None
52. None

53. Which query causes this issue? .

54. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end

55. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end

56. class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
57. None

58. “Invest in your monitoring and exception traceability. When you have

    a hard time racing an exception. Ask yourself - what more information I need? . Then add it.” 
 ) Tip 


59. 
 https://sidekiq.org/


60. None

61. Use action Achievements::Job Achievement
 (unique)

62. class Achievements::Job < ApplicationJob def perform(achievement, user) Achievements::Create.call achievement, user

    end end

63. Use action Achievements::Job Achievement
 (unique) Use action Achievements::Job Achievement
 (unique)

64. Use action Achievements::Job Achievement
 (unique) Use action Achievements::Job Achievement
 (unique)

    (

65. Use action Achievements::Job Achievement
 (unique) Use action Achievements::Job PG::UniqueViolation (

66. class Achievements::Job < ApplicationJob def perform(achievement, user) Achievements::Create.call achievement, user

    end end

67. class Achievements::Job < ApplicationJob def perform(achievement, user) Handle::RaceCondition.call do Achievements::Create.call

    achievement, user end end end

68. module Handle::RaceCondition extend self UNIQUE_ACTIVE_RECORD_ERROR = 'has already been taken'.freeze

    def call retries ||= 2 yield rescue ActiveRecord::RecordNotUnique, PG::UniqueViolation retries -= 1 raise unless retries.nonzero? retry rescue ActiveRecord::RecordInvalid => e raise unless e.message.include? UNIQUE_ACTIVE_RECORD_ERROR retries -= 1 raise unless retries.nonzero? retry end end

69. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event) end

    end

70. Errno::ECONNRESET (

71. None

72. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event) end

    end

73. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue

    Errno::ECONNRESET
 retry_job end end

74. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue

    Errno::ECONNRESET, EOFError
 retry_job end end

75. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue

    Errno::ECONNRESET, EOFError, Timeout::Error
 retry_job end end

76. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue

    Errno::ECONNRESET, EOFError, Timeout::Error, ... /
 retry_job end end

77. module Handle::NetworkErrors extend self ERRORS = [ EOFError, Errno::ECONNRESET, Errno::EINVAL,

    Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Net::ProtocolError, Timeout::Error, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end

78. class Notifications::Deliver < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue

    Handle::NetworkErrors retry_job end end

79. class Notifications::Deliver::Job < ApplicationJob retry_on(*::Handle::NetworkErrors::ERRORS, wait: 2.minutes, attempts: 5) queue_as

    :notifications def perform(event) Notifications::Deliver.call(event) end end

80. class Notifications::Deliver::Job < ApplicationJob include Handle::Job::NetworkErrors queue_as :notifications def perform(event)

    Notifications::Deliver.call(event) end end

81. module Handle::Job::NetworkErrors def self.included(job) job.retry_on(*::Handle::NetworkErrors::ERRORS, wait: 2.minutes, attempts: 5) end

    end
82. None

83. 0

84. module Handle::NetworkErrors extend self ERRORS = [ # ... Faraday::ConnectionFailed,

    Faraday::TimeoutError, RestClient::BadGateway, RestClient::BadRequest, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end

85. “Have tooling around handling common exceptions. Make it a no-brainer

    to process everyday errors.” 
 ) Tip 

86. None
87. None

88. Recap

89. None

90. ) Have process around exceptions.
 ) Be explicit around the

    exceptions
 ) Reduce noise
 ) Don't hide exceptions
 ) Invest in your monitoring
 ) Have tooling around handling common exceptions 
 1 Recap 

91. None
92. None

93. Thanks 2

94. https://speakerdeck.com/rstankov