How to get to zero unhandled exceptions in production

Transcript

1. 1.

   How to get to zero unhandled exceptions in production Radoslav

   Stankov 07/09/2019
2. 2.

   Radoslav Stankov @rstankov blog.rstankov.com github.com/rstankov
 twitter.com/rstankov

3. 3.

   None
4. 4.

   None
5. 5.

   None
6. 6.

   None
7. 7.

   None
8. 8.

   None
9. 9.

   
 ! Product Hunt Engineering Team " 


10. 10.

    None
11. 11.

    None
12. 12.

    None
13. 13.

    None
14. 14.

    None
15. 15.

    None
16. 16.

    None
17. 17.

    Happy Friday # $ Fix bugs % Goodies features &

    Pay technical dept ' Catchup on projects ( Fix exceptions
18. 18.

    Happy Friday # $ Fix bugs % Goodies features &

    Pay technical dept ' Catchup on projects ( Fix exceptions
19. 19.

    None
20. 20.

    “Have process around exceptions.” 
 ) Tip 


21. 21.

    Back to basics

22. 22.

    
 http://exceptionalruby.com/


23. 23.

    def perform do_something rescue end

24. 24.

    None
25. 25.

    def perform do_something rescue SpecificError end

26. 26.

    def perform do_something rescue SpecificError # NOTE(rstankov): Reason to return

    nil nil end Your name, not mine *
27. 27.

    def perform do_something rescue Timeout::Error # NOTE(rstankov): WiFi Sucks nil

    end
28. 28.

    “Be explicit around the exceptions. Handle specific errors and have

    explanations of why they happen.” 
 ) Tip 

29. 29.

    Monitoring

30. 30.

    None
31. 31.

    None
32. 32.

    None
33. 33.

    None
34. 34.

    Raven.configure do |config| # Note(rstankov): Exclude unactionable errors config.excluded_exceptions =

    [ 'Rack::Timeout::RequestExpiryError', 'Rack::Timeout::RequestTimeoutException', 'ActionController::RoutingError', 'ActionController::InvalidAuthenticityToken', 'ActionDispatch::ParamsParser::ParseError', 'Sidekiq::Shutdown', ] end
35. 35.

    None
36. 36.

    ArgumentError: invalid byte sequence in UTF-8 (

37. 37.

    None
38. 38.

    None
39. 39.

    # NOTE(rstankov): Fix invalid byte sequence in UTF-8. More info:

    # - https://robots.thoughtbot.com/fight-back-utf-8-invalid-byte-sequences module Handle::InvalidByteSequence extend self def call(string) return if string.nil? string.encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '' ) end end
40. 40.

    “Reduce noise. See only exceptional errors in your tracker.” 


    ) Tip 

41. 41.

    None
42. 42.

    None
43. 43.

    account.subscription.status

44. 44.

    account.subscription&.status

45. 45.

    ✅ Check for other accounts without a subscription ✅ Find

    out why some accounts don't have a subscription ✅ Fix the root problem ✅ Add missing subscriptions to accounts , Steps to fix -
46. 46.

    None
47. 47.

    None
48. 48.

    if account.ship_subscription.blank? Raven.capture_message "Missing ship subscription", extra: { account_id: account.id

    } return :no_subscription end
49. 49.

    “Don't hide exceptions. Fix root causes.” 
 ) Tip 


50. 50.

    
 https://graphql.org/


51. 51.

    None
52. 52.

    None
53. 53.

    Which query causes this issue? .

54. 54.

    class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
55. 55.

    class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
56. 56.

    class Frontend::GraphqlController < Frontend::BaseController before_action :ensure_query def index render json:

    Graph::Schema.execute(query, variables: variables, context: context) rescue => e handle_error e end private # ... def handle_error(error) if Rails.env.development? logger.error error.message logger.error error.backtrace.join("\n") render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 elsif Rails.env.test? p error.message p error.backtrace render json: { error: { message: error.message, backtrace: error.backtrace } }, status: 500 else Raven.capture_exception(e, extra: { query: query }) render json: { error: { message: 'SERVER_ERROR' }, data: {} }, status: 500 end end end
57. 57.

    None
58. 58.

    “Invest in your monitoring and exception traceability. When you have

    a hard time racing an exception. Ask yourself - what more information I need? . Then add it.” 
 ) Tip 

59. 59.

    
 https://sidekiq.org/


60. 60.

    None
61. 61.

    Use action Achievements::Job Achievement
 (unique)

62. 62.

    class Achievements::Job < ApplicationJob def perform(achievement, user) Achievements::Create.call achievement, user

    end end
63. 63.

    Use action Achievements::Job Achievement
 (unique) Use action Achievements::Job Achievement
 (unique)

64. 64.

    Use action Achievements::Job Achievement
 (unique) Use action Achievements::Job Achievement
 (unique)

    (
65. 65.

    Use action Achievements::Job Achievement
 (unique) Use action Achievements::Job PG::UniqueViolation (

66. 66.

    class Achievements::Job < ApplicationJob def perform(achievement, user) Achievements::Create.call achievement, user

    end end
67. 67.

    class Achievements::Job < ApplicationJob def perform(achievement, user) Handle::RaceCondition.call do Achievements::Create.call

    achievement, user end end end
68. 68.

    module Handle::RaceCondition extend self UNIQUE_ACTIVE_RECORD_ERROR = 'has already been taken'.freeze

    def call retries ||= 2 yield rescue ActiveRecord::RecordNotUnique, PG::UniqueViolation retries -= 1 raise unless retries.nonzero? retry rescue ActiveRecord::RecordInvalid => e raise unless e.message.include? UNIQUE_ACTIVE_RECORD_ERROR retries -= 1 raise unless retries.nonzero? retry end end
69. 69.

    class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event) end

    end
70. 70.

    Errno::ECONNRESET (

71. 71.

    None
72. 72.

    class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event) end

    end
73. 73.

    class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue

    Errno::ECONNRESET
 retry_job end end
74. 74.

    class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue

    Errno::ECONNRESET, EOFError
 retry_job end end
75. 75.

    class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue

    Errno::ECONNRESET, EOFError, Timeout::Error
 retry_job end end
76. 76.

    class Notifications::DeliverJob < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue

    Errno::ECONNRESET, EOFError, Timeout::Error, ... /
 retry_job end end
77. 77.

    module Handle::NetworkErrors extend self ERRORS = [ EOFError, Errno::ECONNRESET, Errno::EINVAL,

    Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Net::ProtocolError, Timeout::Error, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
78. 78.

    class Notifications::Deliver < ApplicationJob queue_as :notifications def perform(event) Notifications::Deliver.call(event)
 rescue

    Handle::NetworkErrors retry_job end end
79. 79.

    class Notifications::Deliver::Job < ApplicationJob retry_on(*::Handle::NetworkErrors::ERRORS, wait: 2.minutes, attempts: 5) queue_as

    :notifications def perform(event) Notifications::Deliver.call(event) end end
80. 80.

    class Notifications::Deliver::Job < ApplicationJob include Handle::Job::NetworkErrors queue_as :notifications def perform(event)

    Notifications::Deliver.call(event) end end
81. 81.

    module Handle::Job::NetworkErrors def self.included(job) job.retry_on(*::Handle::NetworkErrors::ERRORS, wait: 2.minutes, attempts: 5) end

    end
82. 82.

    None
83. 83.

    0

84. 84.

    module Handle::NetworkErrors extend self ERRORS = [ # ... Faraday::ConnectionFailed,

    Faraday::TimeoutError, RestClient::BadGateway, RestClient::BadRequest, # ... ] def ===(error) ERRORS.any? { |error_class| error_class === error } end end
85. 85.

    “Have tooling around handling common exceptions. Make it a no-brainer

    to process everyday errors.” 
 ) Tip 

86. 86.

    None
87. 87.

    None
88. 88.

    Recap

89. 89.

    None
90. 90.

    ) Have process around exceptions.
 ) Be explicit around the

    exceptions
 ) Reduce noise
 ) Don't hide exceptions
 ) Invest in your monitoring
 ) Have tooling around handling common exceptions 
 1 Recap 

91. 91.

    None
92. 92.

    None
93. 93.

    Thanks 2

94. 94.

    https://speakerdeck.com/rstankov