Living Without Exceptions

Transcript

1. Living without Exceptions Radoslav Stankov 07/09/2019
2. !
3. Radoslav Stankov @rstankov rstankov.com
4. https://tips.rstankov.com
14. " Happy Friday # Bug Duty ⚔ Strike Team 4 8 20 developers count
15. 1 developers count " Happy Friday
16. % Fix bugs & Fix exceptions ' Bump dependancies ( Pay technical dept ) Catchup on projects " Happy Friday
17. % Fix bugs & Fix exceptions ' Bump dependancies ( Pay technical dept ) Catchup on projects " Happy Friday
18. % Fix bugs & Fix exceptions ' Bump dependancies ( Pay technical dept ) Catchup on projects " Happy Friday
20. “Have process around exceptions.” * Rado's tip
21. Back to basics
22. http://exceptionalruby.com/
23. def perform do_something rescue end
24. def perform do_something rescue end
26. def perform do_something rescue SpecificError end
27. def perform do_something rescue SpecificError # NOTE(rstankov): Reason to return nil nil end
28. def perform do_something rescue SpecificError # NOTE(rstankov): Reason to return nil nil end Your name, not mine +
29. def perform do_something rescue SpecificError # NOTE(rstankov): Reason to return nil nil end
30. def perform do_something rescue Timeout::Error # NOTE(rstankov): WiFi Sucks nil end
31. “Be explicit around the exceptions. Handle specific errors and have explanations of why they happen.” * Rado's tip
32. Monitoring
37. “Split web and background processes errors into separate projects” * Rado's tip
38. ＃engineering-exceptions
39. “Have a Slack channel where every new exception is being logged. Log deployments in this channel as well, so you can correlate.” * Rado's tip
40. – Brian Kernighan and Rob Pike, The Practice of Programming “Exceptions shouldn't be expected” Use exceptions only for exceptional situations. […] Exceptions are often overused. Because they distort the flow of control, they can lead to convoluted constructions that are prone to bugs. It is hardly exceptional to fail to open a file; generating an exception in this case strikes us as over-engineering.
41. Sentry.configure do |config| # Note(rstankov): Exclude unactionable errors config.excluded_exceptions = [ 'Rack::Timeout::RequestExpiryError', 'Rack::Timeout::RequestTimeoutException', 'ActionController::RoutingError', 'ActionController::InvalidAuthenticityToken', 'ActionDispatch::ParamsParser::ParseError', 'Sidekiq::Shutdown', ] end
42. “Reduce noise. See only exceptional errors in your tracker.” * Rado's tip
45. ArgumentError: invalid byte sequence in UTF-8 &
48. # NOTE(rstankov): Fix invalid byte sequence in UTF-8. More info: # - https://robots.thoughtbot.com/fight-back-utf-8-invalid-byte-sequences module Handle::InvalidByteSequence extend self def call(string) return if string.nil? string.encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '' ) end end
49. “I put most of the code related to exceptions in a single module named Handle” * Rado's tip
53. account.subscription.status
54. account.subscription&.status
55. account.subscription&.status
56. , Steps to fix -
57. ✅ Check for other accounts without a subscription , Steps to fix -
58. ✅ Check for other accounts without a subscription , Steps to fix -
59. ✅ Check for other accounts without a subscription ✅ Find out why those accounts don't have a subscription , Steps to fix -
60. ✅ Check for other accounts without a subscription ✅ Find out why those accounts don't have a subscription , Steps to fix -
61. ✅ Check for other accounts without a subscription ✅ Find out why those accounts don't have a subscription ✅ Fix this / , Steps to fix -
62. ✅ Check for other accounts without a subscription ✅ Find out why those accounts don't have a subscription ✅ Fix this / , Steps to fix -
63. ✅ Check for other accounts without a subscription ✅ Find out why those accounts don't have a subscription ✅ Fix this / ✅ Add missing subscriptions to accounts , Steps to fix -
66. “Don't hide exceptions. Fix root causes.” * Rado's tip
67. if account.ship_subscription.blank? ErrorReporting.capture "Missing ship subscription", account_id: account.id return :no_subscription end
68. “I have a module named ErrorReporting, which helps me capture errors that won't get to the user and provides a centralized place for error tracking.” * Rado's tip
69. https://graphql.org/
73. Which query causes this issue? 0
74. class Frontend::GraphqlController < Frontend::BaseController before_action :authenticate def index render json: Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private def authenticate # ... 
 ErrorReporting.assign_user(@user) end
 def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else ErrorReporting.capture(e, query: query) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
75. class Frontend::GraphqlController < Frontend::BaseController before_action :authenticate def index render json: Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private def authenticate # ... 
 ErrorReporting.assign_user(@user) end
 def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else ErrorReporting.capture(e, query: query) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
76. class Frontend::GraphqlController < Frontend::BaseController before_action :authenticate def index render json: Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private def authenticate # ... 
 ErrorReporting.assign_user(@user) end
 def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else ErrorReporting.capture(e, query: query) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
77. class Frontend::GraphqlController < Frontend::BaseController before_action :authenticate def index render json: Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private def authenticate # ... 
 ErrorReporting.assign_user(@user) end
 def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else ErrorReporting.capture(e, query: query) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
78. “Invest in your monitoring and exception traceability. When you have a hard time racing an exception. Ask yourself - what more information I need? 0 Then add it.” * Tip
79. https://sidekiq.org/
81. Use action
82. Use action Achievements::Job
83. Use action Achievements::Job Achievement
 (unique)
84. class Achievements::Job < ApplicationJob def perform(achievement, user) Achievements::Create.call achievement, user end end
85. Use action Achievements::Job Achievement
 (unique) Use action Achievements::Job Achievement
 (unique)
86. Use action Achievements::Job Achievement
 (unique) Use action Achievements::Job Achievement
 (unique) &
87. Use action Achievements::Job Achievement
 (unique) Use action Achievements::Job PG::UniqueViolation &
88. class Achievements::Job < ApplicationJob def perform(achievement, user) Achievements::Create.call achievement, user end end
89. class Achievements::Job < ApplicationJob def perform(achievement, user) Handle::RaceCondition.call do Achievements::Create.call achievement, user end end end
90. module Handle::RaceCondition extend self UNIQUE_ACTIVE_RECORD_ERROR = 'has already been taken'.freeze def call retries ||= 2 yield rescue ActiveRecord::RecordNotUnique, PG::UniqueViolation retries -= 1 raise unless retries.nonzero? retry rescue ActiveRecord::RecordInvalid => e raise unless e.message.include? UNIQUE_ACTIVE_RECORD_ERROR retries -= 1 raise unless retries.nonzero? retry end end
91. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event) end end
92. Errno::ECONNRESET &
94. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event) end end
95. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue Errno::ECONNRESET
 retry_job end end
96. Errno:: EOFError &
98. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue Errno::ECONNRESET, EOFError
 retry_job end end
99. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue Errno::ECONNRESET, EOFError, Timeout::Error
 retry_job end end
100. class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue Errno::ECONNRESET, EOFError, Timeout::Error, ... 1
 retry_job end end
101. module Handle::NetworkErrors extend self ERRORS = [ EOFError, Errno::ECONNRESET, Errno::EINVAL, Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Net::ProtocolError, Timeout::Error, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
102. class Notifications::Deliver < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue Handle::NetworkErrors retry_job end end
103. class Notifications::Deliver::Job < ApplicationJob retry_on(*::Handle::NetworkErrors::ERRORS, wait: 2.minutes, attempts: 5) queue_as :notifications def perform(event) Notifications::Deliver.call(event) end end
104. class Notifications::Deliver::Job < ApplicationJob include Handle::Job::NetworkErrors queue_as :notifications def perform(event) Notifications::Deliver.call(event) end end
105. module Handle::Job::NetworkErrors def self.included(job) job.retry_on( *::Handle::NetworkErrors::ERRORS, wait: 2.minutes, attempts: 50 ) do |job, exception| ErrorReporting.job_discarded(job, exception) end end end
106. module Handle::Job::NetworkErrors def self.included(job) job.retry_on( *::Handle::NetworkErrors::ERRORS, wait: 2.minutes, attempts: 50 ) do |job, exception| ErrorReporting.job_discarded(job, exception) end end end
109. module Handle::NetworkErrors extend self ERRORS = [ # ... Faraday::ConnectionFailed, Faraday::TimeoutError, RestClient::BadGateway, RestClient::BadRequest, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
110. ActiveJob::DeserializationError &
112. class Notifications::ScheduleJob < ApplicationJob queue_as :notifications def perform(kind:, object:, subscriber_id:) Notifications::Schedule.call( kind: kind, object: object, subscriber_id: subscriber_id, ) end end
113. class Notifications::ScheduleJob < ApplicationJob include Handle::Job::DeserializationError queue_as :notifications def perform(kind:, object:, subscriber_id:) Notifications::Schedule.call( kind: kind, object: object, subscriber_id: subscriber_id, ) end end
114. “Have tooling around handling common exceptions. Make it a no-brainer to process everyday errors.” * Rado's tip
117. Recap
118. * Be explicit around the exceptions * Reduce noise * Don't hide exceptions * Invest in your monitoring * Have tooling around handling exceptions 3 Recap
119. https://tips.rstankov.com/p/tips-exceptions